Selenium: “Failed to decode response from marionette.”

“Failed to decode response from marionette.”

We were trying to move our Selenium tests into a docker container and were getting the above error response. First Google suggested increasing the memory of the container to 2G. We increased it to 3G to no effect. Deeper Googling then suggested increasing shared memory. Initially, it was 64m. We raised it to 256m and it magically worked! Our script for creating the container:

docker build . -f docker/Dockerfile --label cdx_selenium -t cdx_selenium
docker run --shm-size=256m -m3000m --cpus=4 --privileged -d  --name  cdx_selenium cdx_selenium 
docker exec cdx_selenium bash -c 'systemctl daemon-reload && systemctl enable xvfb.service && systemctl start xvfb.service'

We also needed to install dbus-x11. In theory we shouldn’t need to do the systemctl outside the Dockerfile.

Getting started with Amazon AWS

So this is all about getting started with AWS. I’ve been using AWS tangentially for about five years, almost always EC2 instances, so not really pushing the envelope. I’d really like to get my head round serverless and lambdas but I’m having a bit of a conceptual problem trying to work out a use case to do at home.

I have done the Udemy “AWS Certified Cloud Practitioner Practice Exam” which was quite frankly brutal, demanding 90% for three papers. First time through each I got 70-80% which I thought wasn’t too shabby but obviously not good enough for their arbitrary cutoff. The second time through I got >90% which was nice.

The biggest takeaway I have is once you’ve created a root account, create a user account and only give it only the privileges it needs. Security you know!

This is the list of white papers I’ve ingested to far. I hope it proves useful.

AWS Overview

https://d1.awsstatic.com/whitepapers/aws-overview.pdf

This is the motherlode. If you want an overview of all available services on AWS, this is the place to start. For us in the perl world, that’ll be git, CI/CD pipelines and EC2/Fargate. If you’re jiggy, docker too. At this point for most of that, I like Gitlab, especially since M$ took over github.

AWS Well Architected Framework

This takes the overview one step further. Apparently as you partition your app vertically, VPSs are the trick. Security again.

https://d1.awsstatic.com/whitepapers/architecture/AWS_Well-Architected_Framework.pdf

Jenkins on AWS

In a good chunk of the contracts I’ve done, the developers have been good boys and written tests, but there was no way of automatically running them. Jenkins fixes this. Better still using the Perl TAP output formatter we can get a nice graph of the number of tests increasing. If you like, you can run Bamboo or GoCD but I’ve not had happy experiences with either of these.

https://docs.aws.amazon.com/aws-technical-content/latest/jenkins-on-aws/jenkins-on-aws.pdf

Practising Continuous Integration and Continuous Delivery on AWS

Taking Jenkins one step further. In an ideal world code gets committed to master and then gets made live. Your process may vary. Oh, and database versioning is Hard. I’m reliably informed squitch is the one for this.

https://d1.awsstatic.com/whitepapers/DevOps/practicing-continuous-integration-continuous-delivery-on-AWS.pdf

Development and Test on Amazon Web Services

More on the subject.

https://d1.awsstatic.com/whitepapers/aws-development-test-environments.pdf

Overview of AWS Cloud Adoption Framework

Similar overview.

https://d1.awsstatic.com/whitepapers/aws_cloud_adoption_framework.pdf

AWS DevOps

Taking a more DevOps approach to AWS.

https://d1.awsstatic.com/whitepapers/AWS_DevOps.pdf

DevOps for startups

More on the subject of DevOps.

https://blog.thesparktree.com/devops-for-startups

Docker

Now we start getting to the docker meat. I’m not sure how applicable this is to a clunky monolithic Perl framework. I dockerised a simple Catalyst app and it was HUGE. Back to CGI.pm?

https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-basics.html#docker-basics-create-image

Deploy Docker containers

Now we get to the meat.

https://aws.amazon.com/getting-started/tutorials/deploy-docker-containers/?trk=gs_card

Cost optimisation

A common whinge I’ve heard is that unless you’re careful and out of the free first year tier, is that suddenly your AWS usage blooms into thousand of pounds a month. Having been charged £15 a month for static IP I can well believe it.

https://d0.awsstatic.com/whitepapers/Cost_Optimization_with_AWS.pdf

Considerations for the Beginner Serverless Developer

Epsagon have a good trove of blogs too.

https://epsagon.com/blog/considerations-for-the-beginner-serverless-developer/

The Most Popular Deployment Tools For Serverless

https://epsagon.com/blog/the-most-popular-deployment-tools-for-serverless/

5 Ways To Gain Serverless Observability

https://epsagon.com/blog/5-ways-to-gain-serverless-observability/

Yubl’s road to Serverless architecture — Testing and CI/CD

https://theburningmonk.com/2017/02/yubls-road-to-serverless-architecture-part-2/

Serverless observability, what can you use out of the box?

https://theburningmonk.com/2018/04/serverless-observability-what-can-you-use-out-of-the-box/

Our Journey from Heroku to Kubernetes

Kubernetes land is still a mystery to me. Every way I’ve tried to approach it, from linux to Mac I’ve been thwarted. Oh well, one day it’ll be mature enough and actually work for me.

https://www.salsify.com/blog/engineering/our-journey-from-heroku-to-kubernetes

AWS custom runtime for lambda really works: How I developed a lambda in Perl

Now we get to some interesting stuff. It seems hideously convoluted to be but still. It’s a Perl lambda!

https://medium.com/@avijitsarkar123/aws-lambda-custom-runtime-really-works-how-i-developed-a-lambda-in-perl-9a481a7ab465

An alternative Perl lambda

A different approach.

https://github.com/moznion/aws-lambda-perl5-layer

Using the AWS Serverless Application Model (AWS SAM)

https://docs.aws.amazon.com/lambda/latest/dg/serverless_app.html

What Is the AWS Serverless Application Model (AWS SAM)?

https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/what-is-sam.html

Considerations for the Beginner Serverless Developer

https://epsagon.com/blog/considerations-for-the-beginner-serverless-developer/

Serverless and startups, the beginning of a beautiful friendship

https://aws.amazon.com/blogs/aws/serverless-and-startups/

So that’s what I have so far. I think most of the time, we’ll go EC2 and then RDS. I’d put Cloudflare on the front unless I particularly needed a Route53 feature. Serverless is still in the land of dragons and Perl isn’t spectacularly well supported. I’d like to see a world where the code pipeline is under Amazon as well as horizontal scaling with the load balancers.

AWS configuration with Terraform

 

Recently I had a contract which required me to immerse myself in AWS configuration for a LAMP stack, which led me to Terraform, amongst other things. I thought I’d publish here the list of resources that helped me. It’s a lot of links!

One tip I found was to use Jenkins to do automatic validation of your Terraform scripts. I think that’s a good tip.

Talk by Nicki Watt of Hashicorp: https://www.youtube.com/watch?v=wgzgVm7Sqlk

Good tutorial: https://www.youtube.com/watch?v=LVgP63BkhKQ

AWS

Cloud Best Practices: https://d1.awsstatic.com/whitepapers/AWS_Cloud_Best_Practices.pdf

Elastic beanstalk might be a way to go: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/concepts-webserver.html

These days I tend to put everything behind an Nginx reverse proxy: https://github.com/awslabs/ecs-nginx-reverse-proxy

Some AWS Webinars: https://aws.amazon.com/architecture/

Drupal. Which was quite close to what I’d be needing: https://aws.amazon.com/quickstart/architecture/drupal/ and https://aws-quickstart.s3.amazonaws.com/quickstart-drupal/doc/drupal-on-the-aws-cloud.pdf and https://aws.amazon.com/quickstart/architecture/drupal/ and https://github.com/aws-samples/aws-refarch-drupal

Doing your Git stuff at Amazon appears to have some advantages. CodeCommit: https://aws.amazon.com/codecommit/

AWS cost Best Practices: https://aws.amazon.com/premiumsupport/trustedadvisor/best-practices/

Web Hosting general best practices: https://d1.awsstatic.com/whitepapers/aws-web-hosting-best-practices.pdf

A reference architecture for Drupal hosting:

Akamai

If you’re on AWS it makes sense to use Amazon’s own CDN service. If however, you already have an Akamai account, it’s supported by terraform.

https://github.com/Comcast/terraform-provider-akamai

Terraform

Interestingly, Terraform works across multiple cloud providers.

An introduction from Gruntwork, a Terraform consultancy: https://blog.gruntwork.io/an-introduction-to-terraform-f17df9c6d180

Terraform with AWS: https://dzone.com/articles/terraform-with-aws

Terraform getting started: https://www.pluralsight.com/courses/terraform-getting-started

Hashicorp training: https://www.hashicorp.com/training

Udemy have courses if you want to pay money: https://www.udemy.com/learn-devops-infrastructure-automation-with-terraform/

Gruntwork comprehensive guide: https://blog.gruntwork.io/a-comprehensive-guide-to-terraform-b3d32832baca

Gruntwork slideshare: https://www.slideshare.net/brikis98/comprehensive-terraform-training

Another tutorial: https://gist.github.com/p0bailey/3a34689f49b075ed058373dd73a7bce6

All about provisioners: https://www.terraform.io/docs/provisioners/index.html

Ansible

If Ansible is your configurator of choice, you can shoehorn it in to Terraform.

https://alex.dzyoba.com/blog/terraform-ansible/

https://github.com/express42/terraform-ansible-example/blob/master/ansible/terraform.py

https://github.com/radekg/terraform-provisioner-ansible

https://github.com/jonmorehouse/terraform-provisioner-ansible

Chef

Chef is natively supported.

https://sdbrett.com/BrettsITBlog/2017/08/using-the-chef-provisioner-with-terraform/

Chef and AWS: https://www.chef.io/implementations/aws/

Jenkins

Jenkins on AWS: https://d0.awsstatic.com/whitepapers/DevOps/Jenkins_on_AWS.pdf

https://aws.amazon.com/blogs/devops/simplify-your-jenkins-builds-with-aws-codebuild/

https://aws.amazon.com/blogs/devops/set-up-a-build-pipeline-with-jenkins-and-amazon-ecs/

https://docs.aws.amazon.com/aws-technical-content/latest/jenkins-on-aws/best-practices.html

https://wiki.jenkins.io/display/JENKINS/Jenkins+Best+Practices

https://docs.aws.amazon.com/aws-technical-content/latest/jenkins-on-aws/jenkins-on-aws.html

And there’s the link dump. It should be enough to keep you going for a couple of days! Overall, I think if you’re going to use AWS, embrace it fully. Except Cloudflare. They’re awesome.