Kubernetes Minikube #5 – Jenkins for CI/CD

We are on a journey here to operationalise machine learning using Kubernetes.  Call it MLOps, AIOPs, DataOps, DevOps, as you will.  Next stop is to lay the foundation for implementing CI/CD pipelines to move our assets safely through development, canary testing and production environments.  Let’s show that here by laying out a build/tag/promote cycle with Kubernetes, Jenkins and Git.

Objective

The following content will step you through a complete CI/CD setup based on Kubernetes/Jenkins/Git consisting of:

1. Set up Git branches to represent development, canary, production environments
2. Create a Kubernetes hosted sample application
3. Build a Kubernetes hosted Jenkins pod
4. Configure Jenkins for a multi-branch pipeline for GitHub based source
5. Simulate canary testing using Kubernetes scale options
6. Verify using a build/tag/promote pipeline

Setup

1. Launch Minikube

Good so far!  Launch Minikube with the feature gates. It may be worthwhile restarting Minikube from a clean/reset environment as described at Messing with Kubernetes Minikube #1 – Configmaps, Storage, Ingress .  Some Linux flavours support the vm-driver=none switch.  This value makes it easy to just work with your local Docker image Registry.  If this switch is not supported in your environment, you may find it helpful to setup your own local image Registry.

$ sudo minikube stop
$ sudo minikube start vm-driver=none feature-gates=Accelerators=true 
$ sudo minikube addons enable kube-dns
$ sudo minikube addons list

$ cd ~
$ git clone https://bitbucket.org/emergile/cotd.git
$ mv cotd cotd-jenkins

$ cd ~/cotd-jenkins
$ rm -rf .git
$ git init
$ git remote add origin https://bitbucket.org/emergile/cotd-jenkins.git
$ git branch
$ git add .
$ git commit -m "Initial commit"
$ git push origin master

$ git checkout -b canary
$ git branch

$ git checkout -b change-selector
$ git branch

$ cd ~/cotd-jenkins
$ git branch master
$ cat etc/config/cotd.properties
selector=pets
$ docker build -t stefanopicozzi/cotd .

$ kubectl get namespace production || kubectl create namespace production
$ kubectl apply -f etc/kubernetes/jenkins/production/cotd.yaml --namespace production
$ kubectl apply -f etc/kubernetes/jenkins/services/production.yaml --namespace production
$ kubectl scale deployment cotd-production --namespace=production --replicas=4
$ while true; do curl -s http://127.0.0.1:30182/item.php | grep 'data/images' | awk '{print $5}'; sleep 2; done
data/images/pets/neo.jpg
data/images/pets/milky.jpg
data/images/pets/deedee.jpg 
...

3. Setup Jenkins Container

3.1 Build Jenkins

Create a namespace to host our Jenkins container.  We need a few extra bits (kubectl, docker) to the official Jenkins image so we first build an revised image using the Dockerfile as per below.  You may be able to source a better image elsewhere if this one does not suit your needs.  Then we need to setup a hostpath location to host our persistent volume before we (re)build the jenkins pod.

$ kubectl get namespace jenkins || kubectl create namespace jenkins

$ wget https://bitbucket.org/emergile/MLOps/blob/master/jenkins/Dockerfile 
$ docker build -t stefanopicozzi/jenkins .

$ rm -rf /tmp/jenkins 
$ mkdir /tmp/jenkins 
$ kubectl delete -f https://bitbucket.org/emergile/MLOps/src/master/jenkins/jenkins.yaml 
$ kubectl apply -f https://bitbucket.org/emergile/MLOps/src/master/jenkins/jenkins.yaml
...
persistentvolume "jenkins-pv" configured
persistentvolumeclaim "jenkins-pvc" created
service "jenkins-ui" created
service "jenkins-discovery" created
deployment "jenkins" created
ingress "jenkins-ingress" created
$ kubectl get services --namespace jenkins
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
jenkins-discovery NodePort 10.100.25.97 <none> 50000:32155/TCP 1d
jenkins-ui NodePort 10.106.118.61 <none> 8080:30088/TCP 1d

$ PODID=$(kubectl get pods --namespace=jenkins --no-headers=true | cut -f 1 -d " ")
$ kubectl logs $PODID --namespace=jenkins
...
*************************************************************

Jenkins initial setup is required. An admin user has been created and a password generated.
Please use the following password to proceed to installation:

dc5d495104c4413888a783dcc82ed021

This may also be found at: /var/jenkins_home/secrets/initialAdminPassword

*************************************************************
*************************************************************
*************************************************************
...

$ cd ~/cotd-jenkins

$ git checkout change-selector
$ sed -i.bak 's#selector=.*#selector=cats#' etc/config/cotd.properties
$ git add etc/config/cotd.properties
$ git commit -m "SELECTOR=cats"
$ git push origin change-selector

Now visit the Jenkins console to inspect progress of the build for the change-selector branch.

Once complete, you can verify that “cats” is the new SELECTOR setting for the development image as follows, or point your browser to the same URL.

$ while true; do curl -s http://127.0.0.1:30181/item.php | grep 'data/images' | awk '{print $5}'; sleep 2; done 
data/images/cats/hobart.jpg
data/images/cats/perth.jpg
data/images/cats/brisbane.jpg
...
^Ctrl-C

2. Test using Canary

Accept the new development image to canary testing. Do this by switching to the canary branch and merging the changes you made earlier in change-selector. Push the new version and then inspect the Jenkins console to observe that the change-selector branch pipeline has been triggered.

$ git checkout canary 
$ git merge change-selector 
$ git push origin canary

Check the Jenkins console for complete of the canary branch build. Then you can inspect the running curl while loop in the terminal left open earlier and you will now see “cats” item appearing approximately 20% of the time.

$ while true; do curl -s http://127.0.0.1:30181/item.php | grep 'data/images' | awk '{print $5}'; sleep 2; done 
...
data/images/pets/neo.jpg 
data/images/pets/milky.jpg 
data/images/pets/deedee.jpg 
data/images/cats/adelaide.jpg
data/images/pets/kiki.jpg  
...

3. Promote to Production

Accept the new canary image to production. Do this by switching to the master branch and merging the changes you made earlier in canary. Push the new version and then inspect the Jenkins console to observe that the master branch pipeline has been triggered.

$ git checkout master 
$ git merge canary
$ git push origin master

Check Jenkins for compeltion of the master branch build. Then inspect the running curl while loop in the terminal left open earlier and you will now see only “cats” item appearing.

$ while true; do curl -s http://127.0.0.1:30181/item.php | grep 'data/images' | awk '{print $5}'; sleep 2; done 
...
data/images/cats/hobart.jpg 
data/images/cats/perth.jpg 
data/images/cats/brisbane.jpg 
data/images/cats/adelaide.jpg
data/images/cats/sydney.jpg 
...

You can reverse the “cats” change by going through these same steps but this time setting the SELECTOR to “pets”.

Trivia

There are a few resources on this topic.  Google the relevant terms for yourself.   Some I found helpful were https://www.linux.com/blog/learn/chapter/Intro-to-Kubernetes/2017/5/set-cicd-pipeline-kubernetes-part-1-overview and https://blog.kublr.com/using-jenkins-and-kubernetes-for-continuous-integration-and-delivery-4e4341aff013