Running docker multi-stage builds on GKE

I recently worked on reducing docker image sizes for our applications and one of the approaches is to use docker multi-stage builds. It all worked well on my dev machine, but then I shoved new Dockerfiles to CI and and it all shuttered complaining that our docker server is way too old.

The thing is that GKE K8s nodes still use docker server v17.03, even on the latest K8s 1.10 they have available. If you like us run your Jenkins on GKE as well, and use K8s node's docker server for image builds, then this GKE lag will bite you one day.

There is a solution though - run your own docker server and make Jenkins to use it. Fortunately the community thought about it before and official docker images for docker itself include -dind flavour which stands for Docker-In-Docker.

Our Jenkins talked to host's docker server through /var/run/docker.sock that was mounted from host. Now instead we run DInD as a deployment and talk to it through GCP:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: dind
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        component: dind
    spec:
      containers:
      - name: dind
        image: docker:18.06.0-ce-dind
        env:
        - name: DOCKER_HOST
          value: tcp://0.0.0.0:2375
        args:
          - dockerd
          - --storage-driver=overlay2
          - -H tcp://0.0.0.0:2375
        ports:
        - name: http
          containerPort: 2375
        securityContext:
          privileged: true
        volumeMounts:
        - name: varlibdocker
          mountPath: /var/lib/docker
        livenessProbe:
          httpGet:
            path: /v1.38/info
            port: http
        readinessProbe:
          httpGet:
            path: /v1.38/info
            port: http
      volumes:
      - name: varlibdocker
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: dind
  labels:
    component: dind
spec:
  selector:
    component: dind
  ports:
  - name: http
    targetPort: http
    port: 2375

After loading it into your cluster you can add the following environment variable to your Jenkins containers: DOCKER_HOST=tcp://dind:2375 and verify that you are now talking to your new & shiny docker server 18.06:

root@jenkins-...-96d867487-rb5r8:/# docker version
Client:
 Version: 17.12.0-ce
 API version: 1.35
 Go version: go1.9.2
 Git commit: c97c6d6
 Built: Wed Dec 27 20:05:38 2017
 OS/Arch: linux/amd64

Server:
 Engine:
  Version: 18.06.0-ce
  API version: 1.38 (minimum version 1.12)
  Go version: go1.10.3
  Git commit: 0ffa825
  Built: Wed Jul 18 19:13:39 2018
  OS/Arch: linux/amd64
  Experimental: false

Caveat: the setup I'm describing uses emptyDir to store built docker images and cache, i.e. restarting pod will empty the cache. It's good enough for my needs, but you may consider using PV/PVC for persistence, which on GKE is trivial to setup. Using emptyDir will also consume disk space from you K8s node - something to watch for if you don't have an automatic job that purges older images.

Another small bonus of this solution that now running docker images on your Jenkins pod will only return images you have built. Previously this list would also include images of container that currently run on the node.