CI/CD

I’ve been working with CI/CD platforms for the last five or six years. I’ve mostly worked with Gitlab, but I have a little experience with Travis CI and Github actions. Each of these platforms has advantages and disadvantages¹, but in general, CI/CD is a pain in the ass.

I dread having to set up a .gitlab-ci.yml file for a work project. Earlier this week I spend the better part of a day trying to get a project to automatically lint, build and deploy itself. I wasn’t even writing the CI/CD file from scratch, I was using the Dockerfile and .gitlab-ci.yml file from another, very similar project.

I have a couple of main gripes with CI/CD platforms. The first is that I have to make a git commit anytime I want to make a change to the CI/CD pipeline. This feels antithetical to the whole ethos of git. git is a version control tool that allows for building software asynchronously and crucially, collaboratively. One of the main ways we can make our projects easier for other collaborators to digest is by having concise, informative commit messages. Moreover, I am of the opinion that the changes encapsulated by these commits should be related to the software itself, not the platform-dependent CI/CD configuration or deployment-related files. Of course, we could create a branch where we make all the CI/CD-related changes and then collapse the commits we make on that branch into a single commit that gets merged into the main branch, but we still have a bunch of crud in the git history that has nothing to do with the actual software we’re developing. We could ameliorate this by having a very small .gitlab-ci.yml file in our repo that kicks off a pipeline in a different repository. This secondary repository would contain all the configuration and deployment logic. I’ve actually done this with a few projects at work, but I’ve found that it obfuscates what’s going on the CI/CD pipeline more than clarifies. Perhaps it’s worth revisiting this idea?

The second major beef I have with CI/CD platforms is the iteration feedback loop. When I’m writing code in Python or Rust, I can make a change to my code and very quickly see if my tests still pass, or if there is some type error. Depending on the project I’m working on, this change-test-evaluate cycle happens on the order of seconds or even hundreds of milliseconds. In the case of type-checking, it can feel almost instantaneous. What this means is that I can write correct code at the speed of my own fingers and/or thought. This is a marvel of contemporary hardware and tooling!

Whenever I talk to my mom about her experience with coding, she talks about having to code using punch cards, where the feedback loop time was on the order of minutes or hours. When I’m working on CI/CD I feel like I’m writing Fortran 66 that’s expected to run on a computer the size of my Berlin flat. The feedback loop is painfully slow, and requires multiple steps to evaluate:

1. Make the change I want to test
2. Commit the change.
3. Push the change.
4. Move focus from my text editor to my browser.
5. Open up the repo’s pipeline page (in the case of Gitlab).
6. Click on the little 50x50 pixel button that will direct me to a page showing the output of whatever stage I’m trying to debug.

Only after step 6. can I determine if whatever change I made actually worked. Depending on network conditions or runner availability, this whole cycle might take between one and ten minutes. Just to be clear, one minute is 600 times longer than 100 milliseconds (the upper-end of the time it might take for my Rust language server to type check my current file). Moreover, I had to switch window focus and use my mouse. Switching window focus is a context switch, albeit a relatively small one². Debugging a CI/CD pipeline is taxing because it means moving attention between multiple pieces of software, while potentially making single line changes to configuration-relevant files. It’s all wrapper and no candy. I can’t work on anything else while debugging a CI/CD pipeline because one to ten minutes is not enough time to put everything into working memory that’s needed to work on interesting software or data science related problems. This means that I’m stuck twiddling my thumbs waiting for my job to be picked up by a Gitlab runner, or waiting for my Docker image to upload to the registry, or waiting for a pip cache to download. Working with CI/CD pipelines inspires no joy, and it is not amenable to any sort of flow state. It’s the software engineering equivalent to peeling and chopping garlic, or cleaning a toddler’s poop off the wall of your neighbor’s nursery.

What do we do about this painful iteration cycle? I’m not sure, but I know that being able to ssh into a server somewhere to debug issues with my CI/CD pipeline would likely save me a whole ton of frustration. I believe some platforms offer something like this, feel free to send me an email or write me a message with suggestions.

My third issue with CI/CD platforms is vendor lock-in. CI/CD configurations that work with Gitlab most definitely don’t work with Github or Circle CI. You might be asking who cares about portability if the system we’re using currently works? Well, what if your CI/CD vendor were to go out of business? Maybe you’re lucky to have an on-premise deployment, so you can continue to run your pipelines, but forget about bug fixes or improvements. What if your vendor decided to raise their prices? It would be relatively easy to move all your repositories to a different platform, but transitioning the CI/CD configurations could take hundreds of hours of tedious labor. That sort of work has no immediate business value, and could easily run a startup out of business. I’m not sure what the solution to this problem is. Some sort of open-source domain-specific language that is platform agnostic? It’s weird to me that these CI/CD platforms have been around for over ten years, but it still feels like a bit of Wild West³, with random companies/technologies popping up pretty regularly.