Getting started with Amazon AWS

So this is all about getting started with AWS. I’ve been using AWS tangentially for about five years, almost always EC2 instances, so not really pushing the envelope. I’d really like to get my head round serverless and lambdas but I’m having a bit of a conceptual problem trying to work out a use case to do at home.

I have done the Udemy “AWS Certified Cloud Practitioner Practice Exam” which was quite frankly brutal, demanding 90% for three papers. First time through each I got 70-80% which I thought wasn’t too shabby but obviously not good enough for their arbitrary cutoff. The second time through I got >90% which was nice.

The biggest takeaway I have is once you’ve created a root account, create a user account and only give it only the privileges it needs. Security you know!

This is the list of white papers I’ve ingested to far. I hope it proves useful.

AWS Overview

https://d1.awsstatic.com/whitepapers/aws-overview.pdf

This is the motherlode. If you want an overview of all available services on AWS, this is the place to start. For us in the perl world, that’ll be git, CI/CD pipelines and EC2/Fargate. If you’re jiggy, docker too. At this point for most of that, I like Gitlab, especially since M$ took over github.

AWS Well Architected Framework

This takes the overview one step further. Apparently as you partition your app vertically, VPSs are the trick. Security again.

https://d1.awsstatic.com/whitepapers/architecture/AWS_Well-Architected_Framework.pdf

Jenkins on AWS

In a good chunk of the contracts I’ve done, the developers have been good boys and written tests, but there was no way of automatically running them. Jenkins fixes this. Better still using the Perl TAP output formatter we can get a nice graph of the number of tests increasing. If you like, you can run Bamboo or GoCD but I’ve not had happy experiences with either of these.

https://docs.aws.amazon.com/aws-technical-content/latest/jenkins-on-aws/jenkins-on-aws.pdf

Practising Continuous Integration and Continuous Delivery on AWS

Taking Jenkins one step further. In an ideal world code gets committed to master and then gets made live. Your process may vary. Oh, and database versioning is Hard. I’m reliably informed squitch is the one for this.

https://d1.awsstatic.com/whitepapers/DevOps/practicing-continuous-integration-continuous-delivery-on-AWS.pdf

Development and Test on Amazon Web Services

More on the subject.

https://d1.awsstatic.com/whitepapers/aws-development-test-environments.pdf

Overview of AWS Cloud Adoption Framework

Similar overview.

https://d1.awsstatic.com/whitepapers/aws_cloud_adoption_framework.pdf

AWS DevOps

Taking a more DevOps approach to AWS.

https://d1.awsstatic.com/whitepapers/AWS_DevOps.pdf

DevOps for startups

More on the subject of DevOps.

https://blog.thesparktree.com/devops-for-startups

Docker

Now we start getting to the docker meat. I’m not sure how applicable this is to a clunky monolithic Perl framework. I dockerised a simple Catalyst app and it was HUGE. Back to CGI.pm?

https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-basics.html#docker-basics-create-image

Deploy Docker containers

Now we get to the meat.

https://aws.amazon.com/getting-started/tutorials/deploy-docker-containers/?trk=gs_card

Cost optimisation

A common whinge I’ve heard is that unless you’re careful and out of the free first year tier, is that suddenly your AWS usage blooms into thousand of pounds a month. Having been charged £15 a month for static IP I can well believe it.

https://d0.awsstatic.com/whitepapers/Cost_Optimization_with_AWS.pdf

Considerations for the Beginner Serverless Developer

Epsagon have a good trove of blogs too.

https://epsagon.com/blog/considerations-for-the-beginner-serverless-developer/

The Most Popular Deployment Tools For Serverless

https://epsagon.com/blog/the-most-popular-deployment-tools-for-serverless/

5 Ways To Gain Serverless Observability

https://epsagon.com/blog/5-ways-to-gain-serverless-observability/

Yubl’s road to Serverless architecture — Testing and CI/CD

https://theburningmonk.com/2017/02/yubls-road-to-serverless-architecture-part-2/

Serverless observability, what can you use out of the box?

https://theburningmonk.com/2018/04/serverless-observability-what-can-you-use-out-of-the-box/

Our Journey from Heroku to Kubernetes

Kubernetes land is still a mystery to me. Every way I’ve tried to approach it, from linux to Mac I’ve been thwarted. Oh well, one day it’ll be mature enough and actually work for me.

https://www.salsify.com/blog/engineering/our-journey-from-heroku-to-kubernetes

AWS custom runtime for lambda really works: How I developed a lambda in Perl

Now we get to some interesting stuff. It seems hideously convoluted to be but still. It’s a Perl lambda!

https://medium.com/@avijitsarkar123/aws-lambda-custom-runtime-really-works-how-i-developed-a-lambda-in-perl-9a481a7ab465

An alternative Perl lambda

A different approach.

https://github.com/moznion/aws-lambda-perl5-layer

Using the AWS Serverless Application Model (AWS SAM)

https://docs.aws.amazon.com/lambda/latest/dg/serverless_app.html

What Is the AWS Serverless Application Model (AWS SAM)?

https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/what-is-sam.html

Considerations for the Beginner Serverless Developer

https://epsagon.com/blog/considerations-for-the-beginner-serverless-developer/

Serverless and startups, the beginning of a beautiful friendship

https://aws.amazon.com/blogs/aws/serverless-and-startups/

So that’s what I have so far. I think most of the time, we’ll go EC2 and then RDS. I’d put Cloudflare on the front unless I particularly needed a Route53 feature. Serverless is still in the land of dragons and Perl isn’t spectacularly well supported. I’d like to see a world where the code pipeline is under Amazon as well as horizontal scaling with the load balancers.

Upwards of 50,000 people hacked

hack

This would be hilarious if it weren’t so serious. About 50,000 people got their printers hacked and had promotional printouts for YouTube Vlogger “PewDiePie” printed out. This raises the question: HOW? In the real world we should all be behind firewalls, all our computers should have anti-malware software on. This should not be an issue. Printers should not be connected to the internet!

More seriously, it’s possible to destroy computers from afar by repeatedly writing to their firmware. This is not a desirable outcome.

Please folks, if you want to talk about security, there are people like me out there who are more than capable of doing a quick audit and vulnerability scan.

Source: https://www.bbc.co.uk/news/technology-46552339

WordPress plugins cost?

wordpress logoIntroduction

You’ve installed WordPress. It’s free. That’s amazing, and you get to stand on the shoulders of giants with all those great plugins. BUT! Developers need to get paid and a lot of the plugins have paid versions with the full range of features. So what can a fully fedged WordPress installation cost? This is the unspoken secret of WordPress.

The Plugins

These are the plugins I’m using:

  1. Hosting. Not really a plugin. It’s easy to get free/cheap hosting but with a WordPress site taking multiple seconds to load, especially if you have plugins enabled. As a benchmark, the personal purchase on wordpress.com is $39 (£30) per year, but doesn’t really give you that much.
  2. Akismet anti-spam adds better statistics and support for £44 per year.
  3. Cloudflare. You are running this, right? For free it gives you SSL, translation of http to https, DDoS protection, CDN caching (for the speed!), for $20 (£15) you get more as well as firewalling.
  4. With Jetpack you get a load more content stuff and lazy image loading for $9 (£7) per month.
  5. WP-Smush, one of my favrourites which crushes images, for really useful enhancements will set you back $49 (£38) per month.
  6. Updraft plus, the dedicated backup solution, for many, many more features and support will cost you £54 in total.
  7. WP Total Cache with more, possibly useless, caching features will be $99 (£77) per year.
  8. Wordfence security, which bugs me nearly daily to upgrade plugins and also does much more, is $99 (£77) per license.
  9. Yoast SEO which has certainly enhanced my writing for the web, is £79 per license.
  10. And finally something not WP related but which I think is REALLY useful is Grammarly which has also knocked some corners of my writing style.  This is £108 per year, and if I were a professional writer, it would be totally worth it.
  11. The AliExpress plugin is worth it if you want a drop shipping store, and who doesn’t? This is $14 (£11) per month.

Therefore in total, we’re looking at £1156 for the first year! Not insignificant, but developers have to eat!

How to win at phone interviews

Phone Interview
Steve Carell, (AP Photo/NBC, Justin Lubin)

As a contractor, phone interviews are a fact of life. We have to do them to let people know how awesome we are, plus it saves a trip into their office until we’re sure they want us and we want them. After consulting with my posse on LinkedIn and looking at lists on the internet, this is the list of phone interview tips I came up with:

The tips

  1. Be prepared! Put the date and time into your computer/phone calendar and set the alert.
  2. Try to avoid speakerphones. I had one last week and I reckon I got 75% of the conversation. I mentioned it to him and he said that was the only phone in a quiet place he had access to. So do your best. I am going to the next stage so it couldn’t have been that bad.
  3. Stand up. This might not seem obvious but in terms of posture and sounding good, it makes sense. On the same note, smile. It makes you sound better.
  4. Dress up. A proportion of interviews will take place over Skype but even if they don’t, a shirt and a pair of trousers make a difference.
  5. Have your resume to hand. This is good advice. I have done so many gigs, they start blurring into one and it help tell the story.
  6. Have a notepad to hand. It’s good to keep notes, what questions to ask and what to go back to.
  7. Be yourself. I’d rather be Chris Hemsworth, but beggars can’t be choosers. Equally, if your personality is a bit rubbish, best gloss over it. Sound enthusiastic and avoid a monotone.
  8. Block out time and a place to have the interview. Make sure the place is quiet and you’ll be undisturbed.
  9. A bit underhand, but suggest you’re already a long way down the line with someone else. I’m not sure this one is entirely ethical.
  10. Prefer landline over mobile. My mobile tethers over wifi and isn’t 100% reliable. Be in a quiet place where you won’t be disturbed. Turn off your mobile.
  11. This is a general interview tip, but do your research on the company. I always try to find out what their real problem is, not the bland list of requirements in the job ad. Try to form relevant questions. Prepare some questions and answers.
  12. Try to get an email address so you can follow up afterwards, with the notes you made. You made notes, right?
  13. Salary expectations. This one is hard. On my hippy side of the fence, they should pay you what you’re worth. Some of my most productive contracts have been when the interviewer has winced slightly at my price. Equally, I think talking money at this stage is a bit presumptuous.
  14. This one is for Americans: don’t chew gum. And don’t smoke. You can smell it down the phone line.
  15. Have a glass of water handy. A dry throat is no help.
  16. Don’t interrupt and take your time. Pauses are shorter than you think.

And there you have it. The wisdom of crowds!

Run WordPress? Stay secure!

Quite a large proportion of us run blogs, typically WordPress if we want a degree of control or growth, whether for techie stuff or political agitation.

Whenever I work anywhere, I try to make sure the top priority is security. There’s no point doing anything unless you’re secure. The recent Typeform breach shows anyone is liable and their breach exposed data from Monzo bank. In the grand scheme of things, it wasn’t the end of the world: no passwords were leaked.

If you’re running WordPress and therefore relying on somebody else’s software, these are the things you need to do to stay secure:

  1. Install a security plugin. Yes, it’s a pain in the neck getting daily emails to update your site as themes and plugins update but given (1) above, it’s useful. I use Wordfence.
  2. Make sure you use SSL. As well as Google encouraging us to use SSL and gain SEO advantage, being secure is just generally a Good Thing. Worried about SSL certificates? Don’t be. Just hand your DNS management over to Cloudflare and gain SSL, DDoS protection and much more for FREE. My favourite price.
  3. Use strong passwords. Better still use something like Lastpass to generate secure passwords and store them for you safely.
  4. Use two-factor authentication. Make it one step harder to get into your site. Now they won’t get in unless they have your phone. There’s a plugin for that. We use the Google Authenticator.
  5. Keep up to date. 54% of WordPress vulnerabilities belonged to out of date WordPress. You should also keep themes up to date, things like cross-site-scripting exist, and plugins also.
  6. When installing plugins go for the widely used ones, ones with 4*-5* ratings and thousands of satisfied users. Make sure if you go down, LOADS of people go down with you too!
  7. Remove unused plugins and themes. I did that with my personal site and sped it up hugely. Same goes for browser plugins but for different reasons.
  8. Do backups. Second to security. It won’t prevent hacks but it’ll let you get back in the saddle quickly if something awful happens. I use Jetpack which does loads of other stuff too. Make sure you test restoring a backup! Write-only backups are so 90s.
  9. Change the “admin” name”. Trivial but will prevent 99% of brute force attacks.
  10. Limit the number of login attempts. Again, trying to foil brute force.
  11. Don’t let people get at your wp-config file. Put this in your .htaccess file:
    <files wp-config.php>
    order allow, deny
    deny from all
    </files>
  12. And don’t forget, if you find a security hole, report it! That’s how stuff gets better. Finally, make sure you’ll keep the government happy and please don’t provoke GDPR emails.

Damn you linux reference counting.

GitHub logoSo this was an hilarious case of reference counting.

There I was, developing my Perl Catalyst app. I migrate to gitlab like all the other cool kids. I move the original development directory to .bak like a good boy.

But, my plackup is still running and because reference counting, the open files are all still there so I was still happily running. I check out the gitlab version, make changes and NOTHING HAPPENS. Until finally the penny drops, I quit the original, now renamed directory and re-enter the correct one.

Suddenly everything works and hilarity ensues.

Homeopaths lose legal challenge against NHS England

 

homeopathy pills
homeopathy pills

Homeopathy is one of my bugbears. It’s water.  1800 scientific studies say it doesn’t work.

I fight a constant battle on Quora against homeopathy. It’s mostly Indians who seem confused between medicine proven by science and water.

Now:

“A legal challenge brought by the British Homeopathic Association against NHS England’s decision to remove homeopathic treatments from routine primary care has been dismissed on all points by the High Court.”

Sometimes the news is good. Now we can get on with important stuff like curing cancer.

AWS configuration with Terraform

 

Recently I had a contract which required me to immerse myself in AWS configuration for a LAMP stack, which led me to Terraform, amongst other things. I thought I’d publish here the list of resources that helped me. It’s a lot of links!

One tip I found was to use Jenkins to do automatic validation of your Terraform scripts. I think that’s a good tip.

Talk by Nicki Watt of Hashicorp: https://www.youtube.com/watch?v=wgzgVm7Sqlk

Good tutorial: https://www.youtube.com/watch?v=LVgP63BkhKQ

AWS

Cloud Best Practices: https://d1.awsstatic.com/whitepapers/AWS_Cloud_Best_Practices.pdf

Elastic beanstalk might be a way to go: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/concepts-webserver.html

These days I tend to put everything behind an Nginx reverse proxy: https://github.com/awslabs/ecs-nginx-reverse-proxy

Some AWS Webinars: https://aws.amazon.com/architecture/

Drupal. Which was quite close to what I’d be needing: https://aws.amazon.com/quickstart/architecture/drupal/ and https://aws-quickstart.s3.amazonaws.com/quickstart-drupal/doc/drupal-on-the-aws-cloud.pdf and https://aws.amazon.com/quickstart/architecture/drupal/ and https://github.com/aws-samples/aws-refarch-drupal

Doing your Git stuff at Amazon appears to have some advantages. CodeCommit: https://aws.amazon.com/codecommit/

AWS cost Best Practices: https://aws.amazon.com/premiumsupport/trustedadvisor/best-practices/

Web Hosting general best practices: https://d1.awsstatic.com/whitepapers/aws-web-hosting-best-practices.pdf

A reference architecture for Drupal hosting:

Akamai

If you’re on AWS it makes sense to use Amazon’s own CDN service. If however, you already have an Akamai account, it’s supported by terraform.

https://github.com/Comcast/terraform-provider-akamai

Terraform

Interestingly, Terraform works across multiple cloud providers.

An introduction from Gruntwork, a Terraform consultancy: https://blog.gruntwork.io/an-introduction-to-terraform-f17df9c6d180

Terraform with AWS: https://dzone.com/articles/terraform-with-aws

Terraform getting started: https://www.pluralsight.com/courses/terraform-getting-started

Hashicorp training: https://www.hashicorp.com/training

Udemy have courses if you want to pay money: https://www.udemy.com/learn-devops-infrastructure-automation-with-terraform/

Gruntwork comprehensive guide: https://blog.gruntwork.io/a-comprehensive-guide-to-terraform-b3d32832baca

Gruntwork slideshare: https://www.slideshare.net/brikis98/comprehensive-terraform-training

Another tutorial: https://gist.github.com/p0bailey/3a34689f49b075ed058373dd73a7bce6

All about provisioners: https://www.terraform.io/docs/provisioners/index.html

Ansible

If Ansible is your configurator of choice, you can shoehorn it in to Terraform.

https://alex.dzyoba.com/blog/terraform-ansible/

https://github.com/express42/terraform-ansible-example/blob/master/ansible/terraform.py

https://github.com/radekg/terraform-provisioner-ansible

https://github.com/jonmorehouse/terraform-provisioner-ansible

Chef

Chef is natively supported.

https://sdbrett.com/BrettsITBlog/2017/08/using-the-chef-provisioner-with-terraform/

Chef and AWS: https://www.chef.io/implementations/aws/

Jenkins

Jenkins on AWS: https://d0.awsstatic.com/whitepapers/DevOps/Jenkins_on_AWS.pdf

https://aws.amazon.com/blogs/devops/simplify-your-jenkins-builds-with-aws-codebuild/

https://aws.amazon.com/blogs/devops/set-up-a-build-pipeline-with-jenkins-and-amazon-ecs/

https://docs.aws.amazon.com/aws-technical-content/latest/jenkins-on-aws/best-practices.html

https://wiki.jenkins.io/display/JENKINS/Jenkins+Best+Practices

https://docs.aws.amazon.com/aws-technical-content/latest/jenkins-on-aws/jenkins-on-aws.html

And there’s the link dump. It should be enough to keep you going for a couple of days! Overall, I think if you’re going to use AWS, embrace it fully. Except Cloudflare. They’re awesome.

WordPress spice with plugins

Wordpress logoSo, PHP and MySQL, two slightly suboptimal technologies run a fairly large chunk of the internet in the form of WordPress. You have the idea for a blog or maybe want to knock up a quick corporate web site. What’s your first step?

Themes

  • Choose a WordPress theme. There are loads out there, some free some paid for. My site of choice for finding themes free or otherwise is Themeforest. A fair number of the themes are free, and you can choose 2 or 3 column, responsive and so on.

Having chosen your host (we use bluehost.com for www.pandaandpolarbear.com and this site), then it’s time to flesh out the functionality of your site with plugins.

Plugins

  • Akismet – a pretty good comment spam filter plugin. It will mark spam for you so you can you through and trash it. Not sure I’ve ever had a spam comment go through.
  • Cloudflare – These guys are making the internet better. A DDoS, CDN and free SSL solution. 128 data centres. Who is to argue with that?
  • Cookie Consent – Everyone needs this, right?
  • XML sitemaps. Does what is says on the can!
  • Jetpack – Even more themes, stats, SEO tools, Security stuff.
  • Loading Page – while the page is loading, shows a pretty graphic. Given the stats on site abandonment, any distraction is worth it.
  • NextScripts: Social Networks Auto-Poster – lets you spam nearly 30 social media channels.
  • P3 (Plugin Performance Profiler) – Really useful to see where the CPU time is going and if a plugin is taking the time. In my experience, plugins take about 50% of the page render time.
  • W3 Total Cache – caching is good. Most site are not that dynamic so caching is relly good to have.
  • Wordfence Security – useful to have. We’ve had someone uploading rogue JavaScript to WordPress and this spotted it.
  • WP Smush – optimise graphics for the size you’re rendering them at. This is a cool speedup. When you’ve got four years of art, it’s a big win.
  • Yoast SEO – If you’re wordy like me, it’s good to have something reminding you of the good stuff to put in your posts to get the attention of the search engines.
  • Amazon Associates Link Builder  – nice integration with Amazon associates.
  • Finally, Link checker – useful to check for broken links, or destination pages that have gone away.

Other stuff

Don’t forget to sign up to the Google suite, Google analytics, Google webmaster tools, and Google Lighthouse.

Conclusion

That’s a small selection of the plugins we use. There are a whole bunch of Woocommerce related WordPress ones and others related to selling stuff.

I did a site for Dusty Knuckle Pizza, which was working great until they foolishly decided to spend money and get something worse. IMHO.

So that’s that. Your site is now standing on the shoulders of giants.

Remind me, why do people still build web sites manually?

Please employ the bear to do something interesting!

So yet again, I spent time battling a legacy perl code base with no tests, no Jenkins/Bamboo, no deployment pipeline and half an agile process.

Now I get to do battle with recruiters again, something that fills my life with joy and purpose.

I thought I’d put my thoughts down as to what I’m looking for in a job.

First up, contract or permanent? That’s easy. I’ve been contracting for 18 years and I don’t see that changing UNLESS you have a really juicy CTO role on offer. More of that later. I think it’s just largely temperament. I like to have an independent, outside view, trying not to get absorbed in the local cargo cult. So there are two things I do.

Senior Perl developer.

My career can be best described as “careering from one thing to another”. If I’d had any sense, or career direction, or a mentor, I’d have stayed much more firmly in the CTO field. I’ve flirted with many startups over the years, but none have actually stuck. So what am I looking for in a perl gig? Here goes:

  • A modern framework. Give me Catalyst preferably, a framework standing on the shoulders of giants. Dancer or Mojolicious would work as well. Template Toolkit is the ideal templater.
  • Tests. It should be obvious, but often isn’t. If you write code without tests your code is immediately legacy.
  • A sane database schema. One that MySQL Workbench can reverse engineer into a pretty diagram. An ORM. There’s little point these days hard-coding SQL. That’s so passé. Give me DBIx::Class.
  • A well-run Agile process. I got my Scrum master certification and now “doing agile” as opposed to “being agile” brings me out in a rash. One purpose of agile is to get better and unless you do that, you’re not agile. Just standups and sprint planning don’t cut it.
  • Javascript I can take or leave, but it’s a given these days. I can do it but I’ll hate myself afterwards.
  • Don’t talk to me about web servers. Not my problem any more.
  • I want support infrastructure that’ been there since the beginning. That means Perl::Critic and perltidy. Pretty, clean code please.
  • Please let me please talk to REST APIs, none of that SOAP rubbish.

CTO

I’ve been a CTO. And interim a few times. Obviously I’d do it all completely differently this time, knowing what I know now.

  • Let me grow the team. I’ve had amazing luck in the past picking great teams. Indeed, a team that largely didn’t know perl and then became experts. I’ve also been involved in a firing. We’re still friends.
  • Let’s have all the tools we need: Atlassian (or equivalent) stack or integrated equivalent.
  • I want to buy in a good Agile coach for a few months to get us on the right track.
  • I want to manage upwards well. Demo the important stuff to the other directors and management at the end of every sprint. Respond to the business.
  • If you’re good, you can work from home. This is the 21st Century. Being forced to turn up to an office is one of my bugbears. You don’t need my physical presence. Skype and Slack will do the job.
  • Give me something exciting to lead. Not sure I could cope with another publisher web site.
  • Let me speak at conferences. Yes, I know I’m a straight, white male. It’s a burden. But I AM left handed! I’m a minority! It’s good for the company visibility.

And probably stacks more.

As an aside, any good personal projects worth chipping in to right now?