I have decided to pick back up the “Comparing CI/CD Series” that I had started previously with Travis CI and what better CI/CD tool to get stuck into than one of the OGs: Jenkins. I have worked previously on a team that was responsible for administrating a Jenkins service, however admittedly my personal knowledge is quite sparse, other than knowing how to reboot the service and restart jobs when need be.
When googling Jenkins this was the first Ad listing which is quite humourous:
Folks coming across this post might be thinking “why are you using Travis CI or Jenkins for CI/CD?”. It’s a reasonable question. Using my terraform-aws-openvpn-ephemeral repo as an example, it is far simpler to quickly get set up and started with GitHub Actions, GitLab or CircleCI. However, I love a challenge and there are still many companies out there that rely on Jenkins to provide their CI/CD service, so it’s worth getting into the weeds a little bit to see what it is like to work with it.
Jenkins Setup on EC2
I skimmed through the Jenkins Documentation and thankfully there is a Jenkins on AWS link from the Tutorials page which is exactly what I needed!
Installing on AWS tutorial is very straightforward and accessible, it didn’t feel daunting to me to put the pieces together unlike some other services that are out there.
Setting up the Configure Cloud section I felt was a bit light on details though, so I put together this PR to hopefully help make things easier for the next user.
Administering any service yourself on AWS is no trivial amount of work, so if you are looking at using Jenkins as your CICD tool of choice, remember that it will come with a decent amount of administration overhead compared to say GitLab, and GitHub Actions or CircleCI SaaS offerings.
Once I was set up, I wanted to run similar experiments with my terraform-aws-openvpn-ephemeral as I did with Travis CI with hopefully better results 🤞
Setting up a GitHub repo with Jenkins
I started with the getting started guide parsing through how exactly to set up a link between a GitHub repo and Jenkins. It was a little difficult to find how to do this, but it was figured out eventually.
After setting up the Terraform plugin for Jenkins, I quickly realized that there is no Terragrunt plugin for Jenkins, which would potentially jeopardize this whole idea from working within Jenkins (as my terraform module works best with Terragrunt). However, I concluded that a docker container could be used in the pipeline jobs, so it might be better to use my terragrunt-awscli Docker image instead of even thinking about creating a Terragrunt plugin for Jenkins.
After installing Docker and Git onto the EC2 instance directly, installing the Docker and Docker Pipeline plugins via the Jenkins GUI, and then saving my AWS Credentials to the Jenkins Credentials tool, I had a semblance of a working pipeline, but some extra work was needed to be done.
I had to manipulate the workflow to download the openvpn.ovpn file twice (once as part of the docker commands, and then once from Jenkins itself) so that I could download the file from the Jenkins UI. Once I googled how to properly set an environment variable as the terraform output, things looked much better!
Full transparency - it took me 40 (!) attempts at pipeline execution before this workflow started running successfully. If that is not a sign of “You never lose until you quit” I don’t know what is. There were certainly times when I wanted to give up and scrap the project/blog post altogether, but I’m glad I kept at it, even if I may never encounter Jenkins again 😄
In the end, the pipeline was created successfully and I could download the openvpn.ovpn file from the Jenkins UI!
I put together this openvpn-ephemeral-jenkins-template repo in GitHub with screenshots of how to use Jenkins to create the OpenVPN profile file for you. With this repo, there are now FIVE ways to create the terraform-aws-openvpn-ephemeral in automation:
What I liked
The Jenkins community is very welcoming and I received very timely and kind feedback on my documentation PR update. There are some excellent video tutorials for getting your first pipeline written.
The Jenkins on AWS documentation is pretty straightforward, as is the Jenkins GUI startup, even for non-system admin developers/engineers. Having control over the size of the EC2 instance is nice as you can allocate more CPU and RAM if need be (I used larger instances and boy did it make a difference in pipeline times!). You can do this for other CI/CD services like GitLab when you use your own runners too, but not when using their SaaS service.
It is good that there is a fully fleshed out, Production-ready CI/CD tool for only the cost of your own infrastructure/AWS bill, however, there are several tradeoffs that come with that though.
What I didn’t like
Given that Jenkins is an open-source project, it is far different from other CI/CD services like GitLab, GitHub Actions and CircleCI, and unfortunately, it is not offered as a SaaS or Cloud offering like those tools. There is a noticeable shift in recent years from customers and users having to provision these services themselves and there are many CI/CD services offering a “free tier” sign-up to trial out the service with some free pipeline minutes. I am not a fan of having to run the project completely myself and having to think extensively about the cloud infrastructure, plugins, software installed etc, but everyone’s use case is different.
Unfortunately, unlike GitLab, GitHub Actions or CircleCI, it really does feel like you need to be a Jenkins expert to use the service effectively. The barrier to entry is a lot higher than the other services out there, given how you need to install and administrate the Jenkins service yourself. The next few years will be quite interesting to see if the adoption of those other CI/CD tools continues to “take over”.
I wasn’t a big fan of the lengthy process here to set up your local machine when making changes to the Jenkins documentation (which I gave up on and just pushed my PR). This is a HUGE repo that takes up a lot of memory and storage just to set up a static site!
Setting up my GitHub Personal Access Token to access private GitHub repos for testing took me many cycles. I ran into a particularly silly issue where git wasn’t installed on the Jenkins Server, but the error messages did not alert me to this. I installed the Git and GitHub plugins via the Jenkins GUI no problemo, but git was missing on the Amazon Linux server itself causing some interesting troubleshooting.
It also took quite a long time to understand how to pass and interpolate environment variables in a Jenkins pipeline script. This is definitely not as straightforward as other CI/CD tools.
Finally, if a job gets stuck - it takes many steps to try and forcibly remove it from a running state. This is also not the case with other CI/CD services - once you hit cancel, the entire sausage factory grinds to a halt
Jenkins X
Jeknins X appears to be the next phase or iteration of the product that runs exclusively on Kubernetes by the looks of it. I think this is a solid direction and one I am sure they have researched heavily, so it will be interesting to see where the future takes it. I decided to not do a comparison blog or review on the Jenkins X product however; while spinning up a Kuberenetes cluster is certainly achievable, I’m not sure how worth it is for my time. I may decide to do so in the future, and if so, I’ll blog about it then!
Recap
- Jenkins is straightforward and well-documented to setup on EC2
- It takes a fair amount of time and expertise to understand and write effective Jenkins pipelines.
- In today’s landscape, you can get production pipelines created faster using GitHub Actions, GitLab and CircleCI