Build Docker containers on Kubernetes with Jenkins and Kaniko

This writeup documents the current best way to build Docker containers within transient Jenkins agents inside a Kubernetes cluster. This setup has unique features and unique caveats you need to consider, and intend to save you, dear reader, the time I invested trying different solutions.

Build Docker containers on Kubernetes with Jenkins and Kaniko

Setup

These are the components used in this setup. You can replace the K8S provider and the solution and considerations will hold. You can also change the destination registry for your images, and this will require a change in the authentication method.

  • GKE cluster
  • GCR for storing images and cache layers
  • Kaniko official docker image to build Dockerfiles
  • Kubernetes and GCP Service Account to provide credentials to the workers following the Principle of Least Privilege.
  • Jenkins official docker images for Master and for the Worker

Why not docker build?

The docker command requires a working docker daemon, which requires setting up several components, customizing the Jenkins docker images, and more work. Using Kaniko allows us to use the official images and to avoid a lot of work. The resulting images are very similar to the ones build by docker and also totally compatible.

In a nutshell

The steps for this setup are:

  1. Create the Kubernetes cluster
  2. Create a ServiceAccount on GCP with StorageAdmin privileges to be able to read and push images to the registry. (This may not be needed depending on your setup)
  3. Create a ServiceAccount in Kubernetes
  4. Join both ServiceAccounts
  5. Use helm to install the Jenkins Master. Do not use the Jenkins Controller as it is broken at the time of this writing. helm install Jenkins-ci jenkinsci/Jenkins.
  6. Add this code at the top of your Jenkinsfile:

To copy and paste: pipeline { agent { kubernetes { //cloud ‘kubernetes’ defaultContainer ‘kaniko’ yaml “”” kind: Pod spec: serviceAccountName: jenkins-sa containers: — name: kaniko image: gcr.io/kaniko-project/executor:debug-539ddefcae3fd6b411a95982a830d987f4214251 imagePullPolicy: Always command: — sleep args: — 9999999 “”” }

Add this line to your Jenkinsfile to build and upload your image:

To copy and paste: sh ‘/kaniko/executor -f `pwd`/Dockerfile -c `pwd` — cache=true — destination=<DESTINATION-REPO><IMAGE>:$CI_COMMIT_TAG’

After executing this job, your container will be building and uploaded to your GCP registry. For ECR or other registries, you need to set up a different authentication mechanism.

Would you like to receive our newsletter with more TeraTips? Leave us your comments.

Teracloud is an AWS Advanced Consulting Partner specialized in Cloud Computing (AWS, Azure, GCP) best practices. We help SaaS of different industries around the world scale their business safely.

Contact our team at info@teracloud.io for an assessment. Let us handle migration, automation, deployment, performance, cost optimization and machine learning to make your business grow.

We make the cloud easy for you!

All the important news about Cloud Computing. Let´s stay connected to the future