We were trying to move our Selenium tests into a docker container and were getting the above error response. First Google suggested increasing the memory of the container to 2G. We increased it to 3G to no effect. Deeper Googling then suggested increasing shared memory. Initially, it was 64m. We raised it to 256m and it magically worked! Our script for creating the container:
Despite the supermarkets staying open during the lockdown, we’ve been getting far more food delivered; not just supermarket food but heat-at-home restaurant meals and fruit and veg from Wellocks, suppliers to Michelin-starred restaurants.
That said, two things have been standout in the last 12 months: climate change and Welsh independence.
Living in a country that contributed hugely to the increase of CO2 into the atmosphere helps concentrate the mind. Because of the lockdown, we’ve been taking that car out only once or twice a week and I see an electric car in our future. As it is, we live in a wood-clad “green” flat and are surrounded by trees in a borderline countryside area. Wales is 3rd in the world for recycling. I’m not sure what more we could do for the environment.
On the subject of Wales, in the last year, the subject of Welsh independence has started to gain traction. The non-political group yes.cymru have gone from nothing to nearly 20,000 paying members and 50,000 Twitter followers. Welsh Labour and Plaid Cymru supporters are behind the idea too. The electorate polls at 25%-40% in favour.
As I’ve said before, Wales has a GDP per head which puts it on a par with Spain. We’re shy between £5-£10 billion pounds a year but hey, with European support we can claw our way back as Ireland did. I’ve blogged about Wales’ potential before. Also, accounting properly for water, electricity and HS2 would help by a few billion in our favour. Oh, and we pay a disproportionate amount for defence, another £1.9 billion. It doesn’t help their case that we appear to have a bunch of incompetents in Westminster.
So there you have it. These are two things that are now occupying much more news space, internet space and headspace.
These are all things you can find elsewhere but a couple of password issues came as a surprise to me
These are all things you can find elsewhere but a couple of password issues came as a surprise to me when a legacy system got the MySQL 5.7 upgraded to 8.0.
Firstly, password policies are much tighter. There’s a plugin that by default demands an uppercase letter, a number and a punctuation character. That foxes our legacy system whose installer just generates lowercase letters and numbers. Uninstall it.
Being green is all the natural resource we’ve got left in Wales pretty much.
Being green is all the natural resource we’ve got left in Wales pretty much since the rich folks and the English took our coal, tin, copper and so on.
This blog is also going to come from an Indy viewpoint. Welsh independence has gone from nothing to 40% in the last few years. Independence is worth bearing in mind whichever side you fall on. I’ve fallen foul of people who think the world revolves around Westminster on Reddit. We got annexed in 1284 and made a union in 1536, not on the same basis as Scotland.
40%. It’s only time.
A poll suggesting that backing for independence among Welsh citizens is at a record high should serve as a warning for the UK government and prompt it to work harder at its relationship with the devolved nations, supporters of the union have said.
Put energy creating lagoons around the harbours. This has come and gone but apparently is back again.
“Providing low-carbon, predictable renewable energy, tidal lagoons will deliver reliable and flexible electricity whatever the wind conditions or time of day – ensuring grid security and stability. Moreover, tidal lagoons have an exceptional operating life, at over 120 years, over three times a wind farm and twice a nuclear plant, and significant co-benefits that other schemes do not bring, such as protecting communities and businesses from rising sea levels.
Healthy peatland and raised bogs in good condition absorb carbon from the atmosphere which means they are important in the fight against climate change. If raised bogs are not in good condition they release harmful carbon into the atmosphere.
Treble public transport in Cardiff? No brainier. Fuck all has happened with this in the seven years I’ve been in Cardiff. Allegedly some things are happening. In all that time there’s been a “Metro Plan”. Maybe we’ll move on from horse-drawn open-topped trains.
The South Wales Metro is an integrated public transport network that will make it easier for people to travel across the Cardiff Capital Region, transforming rail and bus services as well as cycling and walking.
Build energy efficient houses. I’m lucky enough to live in a B rated place. All places should be this efficient.
Our seafood is amazing. Shame Westminster fucked that up.
There’s 3x as many as there are people. We should probably have more, within their environmental impact.
Restore the Cambrian mountains. There’s a 300km2 dead zone.
In the southern Cambrian Mountains, in central Wales, there’s a Terrestrial Dead Zone of around 300 km². It’s composed of degraded blanket mires, entirely dominated by a coarse grass called Molinia, in which other lifeforms, such as birds and insects, are scarcely to be found.
Cardiff is nice and green, there’s plenty of woods, open fields and so on to just go hang in. Sadly a few green spaces are under threat. That’s not surprising in a growing city. The rivers could do with cleaning up. We used to have eels and way more fish. I’ll blog about this at another time. I’ve exported all my saved pins from Google maps.
There’s so much potential but nothing will happen while we’re under Westminster’s thumb. In EU terms, we are by no means the smallest country in Europe and our GDP is OK. I’d like to see more action. In the last year of lockdown green issues have come more to the forefront. I want a green future for Wales.
SQL injection – could someone get at your database remotely? Escape your SQL!
Broken auth – is your login system safe?
Data exposure – is your web server locked down? -Indexes in the Apache world.
XML External entities – XML can execute files. Don’t do that!
Broken access control – are important files inaccessible?
Security misconfiguration – is your security software properly configured
XSS – has someone uploaded a malicious JS script?
Using components with known vulnerabilities – keep up to date!
Insufficient logging and monitoring – know what’s going in and out of your system
Insecure deserialisation – be careful of the serialised data you accept
Make sure you have DDoS protection. I use Cloudflare. Is has the added bonus of running my DNS. I trust them. Snort or equivalent. This is part of your monitoring. Snort is an IDS or Intrusion Detection System. On WordPress use a security plugin. I use Wordfence.
Site attackers can:
Inject SEO spam on the page
Drop a backdoor to maintain access
Collect visitor information or credit card data
Run exploits on the server to escalate access level
Use visitors’ computers to mine cryptocurrencies
Store botnets command & control scripts
Show unwanted ads, redirect visitors to scam sites
Host malicious downloads
Launch attacks against other sites
Asset inventory and management can be taken one step further into the following subcategories:
Web servers and infrastructure
Plugins, extensions, themes, and modules
Third-party integrations and services
Monitoring should be in place to verify the security state of:
File integrity – monitor file modification times of plugins and themes
A proper incident response plan includes:
Selecting an incident response team or person
Reporting of incident to review findings
Mitigating the event
The incident response process, as defined by NIST, is broken down into four broad phases:
Preparation & planning
Detection & analysis
Containment, eradication & recovery
Post incident activities
You can base all further actions on the following tips:
Restrict global access to your site (or certain areas) via GET or POST methods to minimize exposure.
Update directory and file permissions to ensure the read/write access is properly set.
Update or remove outdated software/themes/plugins.
Reset your passwords immediately with a strong password policy.
Activate 2FA/MFA wherever possible to add an extra layer of authentication.
The best practices for you to have a strong password are:
Use a password manager,
Do not reuse your passwords: Every single password you have should be unique.
Have long passwords: Try longer than 12 characters. The longer the password is, the longer it will take a computer program to crack it.
Use random passwords: Password-cracking programs can guess millions of passwords in minutes if they contain words found online or in dictionaries. If you have real words in your password, it isn’t random. If you can easily speak your password, it means that it is not strong enough. Even using character replacement (i.e. replacing the letter O with the number 0) is not enough. There are several helpful password managers out there, such as LastPass (online) and KeePass 2 (offline). These tools store all your passwords in an encrypted format and can easily generate random passwords at the click of a button. Password managers make it possible to use strong passwords by taking away the work of memorizing weaker ones or jotting them down.
The principle of least privilege centers around a principle that looks to accomplish two things:
Using the minimal set of privileges on a system in order to perform an action
Granting those privileges only for the time the action is necessary
Here are the things to look for when deciding which extensions to use:
When the extension was last updated: If the last update was more than a year ago, it’s possible the author has stopped working on it. Use extensions that are actively being developed because it indicates that the author would at least be willing to implement a fix if security issues are discovered. Furthermore, if an extension is not supported by the author, then it may stop working if core updates cause conflicts.
The age of the extension and the number of installs: An extension developed by an established author that has numerous installs is more trustworthy than one with a few number of installs released by a first-time developer. Not only do experienced developers have a better idea about best security practices, but they are also far less likely to damage their reputation by inserting malicious code into their extension.
Legitimate and trusted sources: Download your plugins, extensions, and themes from legitimate sources. Watch out for free versions that might be pirated and infected with malware. There are some extensions whose only objective is to infect as many websites as possible with malware.
A good backup solution should fulfil the following requirements:
First, they have to be off site. If your backups are stored in your website’s server, they are as vulnerable to attacks as anything else in there. You should keep your backups off-site because you want your stored data to be protected from hackers and hardware failure. Storing backups on your web server is also a major security risk. These backups invariably contain unpatched versions of your CMS and extensions, giving hackers easy access to your server.
Second, your backups should be automatic. You do so many things every day that having to remember to backup your website might be unthinkable. Use a backup solution that can be scheduled to meet your website needs.
To finish, have reliable recovery. This means having backups of your backups and testing them to make sure they actually work. You will want multiple backups for redundancy. By doing this, you can recover files from a point before the hack occurred.
Here are a few best practices to add for a particular web server:
Prevent directory browsing: This prevents malicious users from viewing the contents of every directory on the website. Limiting the information available to attackers is always a useful security precaution. Use -Indexes in Apache.
Prevent image hotlinking: While this isn’t strictly a security improvement, it does prevent other websites from displaying the images hosted on your web server. If people start hotlinking images from your server, the bandwidth allowance of your hosting plan might quickly get eaten up displaying images for someone else’s site.
Protect sensitive files: You can set rules to protect certain files and folders. CMS configuration files are one of the most sensitive files stored on the web server as they contain the database login details in plain text. Other locations, like admin areas, can be locked down. You can also restrict PHP execution in directories that hold images or allow uploads.
Here are some free website security tools:
SiteCheck – Free website security check and malware scanner
Sucuri WordPress Security Plugin – Auditing, malware scanner, and security hardening for WordPress websites
Google Search Console – Security notifications and tools to measure websites search traffic and performance
Bing Webmaster Tools – Search engine diagnostics and security reports
Yandex Webmaster – Web search and security violation notifications
Unmaskparasites – Check pages for hidden illicit content
Best website security software – Comparison of paid website security services
Best WAF – Comparison of the best cloud-based web application firewalls
Netsparker – (Free community edition and trial version available). Good for testing SQL injection and XSS
OpenVAS – Claims to be the most advanced open source security scanner. Good for testing known vulnerabilities, currently scans over 25,000. But it can be difficult to setup and requires a OpenVAS server to be installed which only runs on *nix. OpenVAS is fork of a Nessus before it became a closed-source commercial product.
SecurityHeaders.io – (free online check). A tool to quickly report which security headers mentioned above (such as CSP and HSTS) a domain has enabled and correctly configured.
Xenotix XSS Exploit Framework – A tool from OWASP (Open Web Application Security Project) that includes a huge selection of XSS attack examples, which you can run to quickly confirm whether your site’s inputs are vulnerable in Chrome, Firefox and IE.
Are you using GitHub or similar? I’ve used Gitlab most recently, and I especially like Docker in Docker. Within that, how close to GitFlow are you? Having experienced an awful version control system, this is key. GitHub is really flexible and gives you enough rope to hang yourself in the foot. A fun thing is commenting commits correctly.
What’s your branching strategy? How long do you expect a branch to live? Branch lifetime should be of the order of a day. Any longer than that, have a quiet word with your SCRUM master.
How automated are your deployments? Do you create .rpms/.debs? Packages make deployments and rollbacks so much easier. Add YYYYMMddhhmmss to the name so you can keep track of them, or a build number so you can identify them.
Which CI system do you use? If not Jenkins, GitHub or Gitlab, why not?
Test automation is great. It builds, runs tests and creates modules. And anything else that makes your life easier. It’s also the ultimate in QA. If you have good test coverage and your tests pass, you’re good to go. It’s part of CI, right? Do you measure test coverage?
CI is also a good time to run code hygiene tests like pylint or perlcritic even if you have them on your commit hook. OWASP recommend some code security scanners and Snyk seems quite plausible.
How is your test data managed? Do you create a temporary database and populate it or do you have one database and run your tests within a transaction?
Security? How close to the developers is this managed? Separate security departments are often understaffed. Do you keep an eye on the OWASP top ten? Are you religious about escaping strings when composing SQL queries?
How close to continuous delivery are you? How long do rollbacks take? Do you use something like Ansible or puppet to manage your systems? Bonus points for terraform or docker. How fungible are your live servers?
How loosely coupled is your architecture or is it a big ball of mud? This is another thing that burned me recently. With mod_perl potentially going away in some form, parts of the system should have been moved to a new framework.
What other tools do you have and who chose them? Are you running popular systems for monitoring or code review or some open-source system that might wither on the vine?
Are you agile? Do you do SCRUM or KANBAN? Do you have a SCRUM master and a product owner? So many teams think they are agile when they’re merely doing some agile type things sandwiched in a blob of waterfall.
Who authorises changes? Do the developers do it or do you have a separate approvals board? It’s so much better to have decisions made at the lowest level by team members than to be farmed out to some remote change approvers.
What system monitoring do you have? What is your average time to fix?
What is your ticketing system, and why isn’t it JIRA, GitHub or Gitlab? Does your SCRUM master visualise progress and use all the tools to measure the team performance. Does your SCRUM master measure project velocity?
Has management bought into the k8s kool-aid? Are you using kompose and rancher to help manage it?
So there you have it. How to extend an interview beyond the allotted time.
Did I miss anything? Comments, as always, welcome.
Coming out of a job where I was working on a 20-year old Perl codebase, I’ve got some burns to get off my chest. I’m reading “Accelerate” by Forsgren, Humble and Kim which claims to have scientific backing for what makes for efficient development in a team. In my recent experience:
Use decent version control. To me, that means GitHub. Use a branching strategy to code each branch to a JIRA. Make the branches short-lived, preferably a day. GitHub is stateless. Diffs are resolved at merge-time when pull requests are made. Under NO circumstances use something like Perforce. That is like putting a large speed bump under a low slung car. It’s stateful. Mapping a repo into your filesystem is a pain. Rewinding commits is a royal pain. Ugh.
Release often, releases should be easy. A marker of a high performing team is how frequently they release software. A release should not be confined to one person on the team and take half a day.
Great balls of mud are hopeless. We’ve been writing new software as microservices for a while now, and more recently bundling them up in Docker containers (and if you’re really advanced then using Kubernetes). In the Perl world that means using a framework such as Mojolicious, Catalyst or Dancer with excellent modules like the Template Toolkit for the view and DBIx::Class for the model and not v1 of view software that’s barely been touched for years and v2 exists. It also highly bound to Apache and hard to use elsewhere.
Ongoing support for mod_perl in Apache 2.x is ongoing. It’s already been abandoned in Apache 1.x so I would note that software is doomed at some point.
Be very careful layering software upon software. Or using features that make things opaque. Oh, and having magic variables and not documenting them. For example, you have Puppet. That’s great. Why not layer Heira on top and render most of the puppet documentation useless. Or use a templating system that magically calls in a hierarchy of other templates. Oh, and where does that database handle come from? Somewhere in the bowels of that page startup. Not sure which module.
In summary, I’d say be aware of the speedbumps. How can you improve them?
Macksville resident Melanie Williams was also shocked by a swarm of spiders climbing the outer wall of her home as they fled for higher ground. “I occasionally see spiders around the place but never anything like that, it was just insane,” she told the ABC.
The spiders outside her home were “horrific” but her neighbour told her there were twice as many inside his garage, she told Guardian Australia.
I had no idea this was a thing. Apparently, even our daily cup or two (or several) a day of coffee is helping to screw up the planet. Mass-produced coffee is produced in nice, highly productive rows of coffee plants, which sadly gives a habitat for a quarter of the number of species of birds as when coffee is grown in the shade of mature trees. So we need a bird-friendly coffee.
According to the RSPB:
Shade-grown means that the coffee grows more slowly, requires less water and the need to use any invasive fertilizers or pesticides. This in turn supports greater biodiversity and ensures that the forest in which it’s grown sustains a healthy ecosystem.
And according to Cornell University:
“Over recent decades, most of the shade coffee in Latin America has been converted to intensively managed row monocultures devoid of trees or other vegetation,” Amanda Rodewald, a co-author of the study who is the Garvin Professor and senior director of the Center for Avian Population Studies at the Cornell Lab of Ornithology, said in a statement. “As a result, many birds cannot find suitable habitats and are left with poor prospects of surviving migration and successfully breeding.”
Today, most coffee sold is sun-grown under little or no shade because sun makes coffee bushes grow faster and produce more coffee. This loss of tropical forest biodiversity to a row monoculture harms resident rainforest birds along with their migratory cousins so they all are disappearing along with their rainforest homes. This simple connection between habitat loss, pesticides and fertilizer pollution to intensive coffee farming methods was the impetus for Smithsonian conservation scientists to create the strictest agricultural certification criteria for coffee: their Bird-Friendly certification requires that coffee is organic and that it meets strict requirements for both mature canopy cover and the type of forest in which the coffee is grown. Bird-Friendly coffees are guaranteed to support bird habitat, in addition to fair and stable prices for coffee producers, healthy environments for local communities, and equal access to markets for Bird-Friendly coffee producers.
So there you have it. By having that supermarket, mass produced coffee, you’re helping destroy the planet. Good work! I’ve just bought 1.2kg of their coffee. Sorry, Tesco.