Automation. It is what us developers are supposed to be good at. Though sometimes, automating something can be pretty hard.
A lot of money is wasted by testing things by hand. Of course there’s also the random errors us humans make and inexplicable test results as a direct result. I’m sure many agree that machines are far better suited at performing tedious tasks quickly and accurately.
Us simple developers love having a button that lights up green when everything is good so we can merge our changes without a single worry. Trying to creat this button on a budget though, a bit harder. It took me a few tries to get things right.
First, I started writing simple shell scripts. Such scripts are however executed and verified manually but its a step in the right direction. The bigger issue was that these scripts generally required very specific environments and were executed on your own development machine. Not great.
Integrating these scripts into Git was a fairly obvious step. Running arbitrary scripts is easily done with a post-recieve hook on the remote. This offloads all processing away from your development machine. The problem was creating an environment to run these scripts.
Most of the project I work with are shipped as packages, thus it made sense to build the software, package it and then perform tests with said scripts. There’s schroot and sbuild environments that most Debian developers are likely familiar with but I was looking for something more generic, less restrictive.
I heard a lot of good stuff about Jenkins so I set up a Jenkins server and started migrating my scripts to Jenkins. It was a pretty painful process. Even the setup required a lot of hand-holding. My workflow wasn’t compatible with Jenkins. Jenkins also proved to be a pretty big resource hog. More importantly, trying to use git-buildpackage in a chroot was simply not stable or maintainable.
A few months went by, Docker came into existence. Containers were now a thing. I experimented with building things in Docker containers and found it to be very simple and reliable. Creating a build environment took nothing more than a few lines in a Dockerfile. The ability to instantly rebuild and reuse any environment is very powerful and reduced build times to mere fractions.
As I couldn’t find a suitable build server that performed builds using Docker, I tried rolling my own. This didn’t go well as I didn’t have the budget nor the time to develop a brand new platform. The project was abandoned fairly quickly.
A few more months passed and I stumbled on GoCD. Its primary feature being very lightweight. The setup took mere minutes. Setting up pipelines was easy. No complicated shell scripts, just execute the entire build in a disposable Docker container. Done.
Test automation was still far away. No I don’t mean build tests, those are fairly easy to do (unless you are cross-compiling). Initially I tried running test in Docker but I quickly ran into issues as Docker is designed for single-process containers.
Performing tests in LXD containers however proved to be a much better option. Being able to spawn several containers and being able to perform functional client-server integration tests is very powerful.
With the help of Puppeteer, performing full in-browers didn’t take long to develop. Today I’m able to run hundreds of tests just by running git push, technology is pretty amazing.