This is the third part of our blog series on Docker at Movio; in part one we described how Docker helps our engineering teams streamline their development by providing a unified and reproducible execution environment for services, while part two covered our use of Rundeck for builds and deployments. In this third part, we’ll detail how Docker has helped us to facilitate continuous integration (CI).

Jenkins

We’ve been fans of Jenkins for years, having used it extensively for continuous integration and to manage our deployments (using the Promoted Builds plugin). Deployment has since been moved away from Jenkins – now handled using Puppet and Rundeck – but it remains a critical component of our build and CI infrastructure.

Our Jenkins setup is composed of one centralized server, with a handful of developers running a Jenkins Slave on their desktop machine to help share the computation load. We try to keep our builds green wherever possible, so we’ve configured Jenkins to report all build failures to a dedicated Slack channel and we fix failed builds as soon as possible.

Initially, we configured Jenkins manually through the UI, which works fine when you have a small number of jobs. As the number of jobs grew, and as started looking for more control over the configuration of each job, we started versioning the Jenkins XML config files with Git. Again, this worked well for a while, until our jobs grew in complexity (caused in part by introducing Docker to our build pipeline – more on that later) and we felt we needed something more powerful. Last year we started writing all new jobs using the wonderful Groovy DSL Plugin, which has helped us enormously in dealing with this increased complexity.

Docker & Jenkins

Movio follows the squad model as an organization structure, with each squad having full autonomy regarding the tech stack that it uses. The idea is simple: a squad should be able to upgrade to a newer version of NodeJS, introduce Clojure, start using Kafka or make any other such tech choice without impacting on other squads or requiring their prior approval. We believe this technical autonomy is essential to promote innovation, but it needs to be managed properly for centralized continuous integration to work.

Initially, each developer who ran a Jenkins slave needed to install the complete toolchains (with the right versions) required by each and every Jenkins job. Changing any job’s build requirements meant upgrading every slave manually, with the added complexity of having different devs running different distributions of Linux. In addition to being impractical, it went against the aim of minimizing the impact of each decision on other squads.

We’ve solved most of these issues using the Jenkins Docker Plugin and Seed Plugin. We now run each job in a container that effectively acts as a dedicated virtual CI machine and is created on demand – when a job runs it will bring up a new container, then kill it when it’s finished. This means that each developer wanting to run Jenkins slave jobs just needs Docker installed –upgrading a job’s build requirements amounts to a simple Docker image update, which is transparent for all Jenkins slaves.

In Practice

Let’s take the example of an imaginary project: a single page application called Tyco. It has a frontend built in ES6 and ReactJS and a backend using Scala and Play. For every feature ticket, say TYCO-1234, we create three branches: TYCO-1234-backend-feature-name, TYCO-1234-frontend-feature-name, and TYCO-1234-feature-name. The first two branches are used early in the development phase, when backend and frontend aren’t yet together. For integration and user acceptance testing, we merge the backend and frontend branches into the feature branch and delete them.

So let’s build a kickass CI setup for our Tyco project! We start by manually creating a tyco-seed job and configure it to run a Groovy DSL script periodically every hour. Note that this job must run on the host Jenkins server, not in a Docker container. Set Action for removed jobs to “Delete” (more on that later).

Here’s our jobs.groovy file, piece by piece.The first part isn’t all that interesting: we import the Groovy DSL helpers, and create a new Jenkins tab for all our Tyco jobs:

import javaposse.jobdsl.dsl.helpers.*

listView('tyco') {
    description('Tyco jobs')
    jobs {
        regex('tyco-.*')
    }
}

Next, we create the tyco-docker-ci job, which builds a Docker image in which all CI jobs will run, pushes it to our private Docker repository and pulls the image down to each Jenkins slave.

We won’t detail the Dockerfile here, but what it does is straightforward: it installs the compilation and testing toolchains for both backend and frontend, and fetches the Tyco Git repository to speed up later Git updates. Note the use of “label('docker-builds')” meaning for our config that this job will not run inside a Docker container but directly on the master Jenkins node.

freeStyleJob("tyco-docker-ci") {
    label('docker-builds')
    deliveryPipelineConfiguration('docker', 'build image')
    scm {
        git {
            remote {
                url("git@github.com:movio/tyco.git")
            }
            branch("master")
        }
    }
    steps {
        shell("docker build -t docker.movio.co/jenkins-tyco jenkins/ci")
        shell("docker push docker.movio.co/jenkins-tyco")
        shell("./jenkins/ci/update_jenkins_slaves.sh")
    }
}

Let’s now create our actual CI jobs. We fetch all branches, and create a job for each one. We use the name of the branch to determine whether to execute frontend tests, backend tests, or both. These CI jobs poll the Git repository every 15 mins for changes and reruns the tests if any changes are detected (we can’t use GitHub hooks because our Jenkins instance is on our private network).

def getBranches = {
    def result = []
    def gitURL = "git@github.com:movio/tyco.git"
    def command = "git ls-remote --heads ${gitURL}"
    def proc = command.execute()
    proc.waitFor()
    if (proc.exitValue() != 0) {
        throw new IllegalStateException("Cannot fetch branches from github")
    }
    def text = proc.in.text
    def match = /refs\/heads\/(\S*)/
    text.eachMatch(match) { result.push(it[1]) }
    result
}

getBranches().each() {
     createTycoJob(it)
}

def createTycoJob(String branchName) {
    freeStyleJob("tyco-ci-" + branchName) {
        scm {
            git {
                remote {
                    url('git@github.com:movio/tyco.git')
                }
                branch(branchName)
            }
        }
        triggers {
            scm('H/15 * * * *')
        }
        steps {
        if (branchName.contains('frontend')) {
          shell("gulp test")
        } else if (branchName.contains('backend')) {
          sbt('SBT 0.12.3', 'test')
        } else if (!branchName.endsWith('-stale')) {
          shell("gulp test")
          sbt('SBT 0.12.3', 'test')
          shell("./test/run_e2e_tests.sh")
        }
       label('docker-tyco')
      }
    }
}

That’s it. Every branch is automatically built and tested appropriately. Jobs of merged or deleted branches are automatically cleaned up (remember we set “Action for removed jobs” to “Delete”). All jobs are run inside a Docker container, which means that upgrading the toolchain amounts to simply updating the Dockerfile and rerunning the Docker image build. The Jenkins config for the Tyco project is almost entirely contained in one relatively short centralized and versioned Groovy script.

This setup is working really well for us, so hopefully you’ll find it useful too – we'd love to hear about your good and bad integration experiences in the comments below.

Read the rest of our Docker series:

Part One: How Movio squads are streamlining their development process with Docker
Part Two: How to use Rundeck for Docker builds & deployments