Back to blog

Configuring a Jenkins Pipeline using a YAML file

Matias De Santi April 25, 2018
This guest post was originally published on Wolox’s Medium account here.

A few years ago our CTO wrote about building a Continuous Integration server for Ruby On Rails using Jenkins and docker. The solution has been our CI pipeline for the past years until we recently decided to make an upgrade. Why?

  • Jenkins version was way out of date and it was getting difficult to upgrade

  • Wolox has grown significantly over the past years and we’ve been experiencing scaling issues

  • Very few people knew how to fix any issues with the server

  • Configuring jobs was not an easy task and that made our project kickoff process slower

  • Making changes to the commands that each job runs was not easy and not many people had permissions to do so. Wolox has a wide range of projects, with a wide variety of languages which made this problem even bigger.

Taking into account these problems, we started digging into the newest version of Jenkins to see how we could improve our CI. We needed to build a new CI that could, at least, address the following:

  • Projects must be built using Docker. Our projects depend on one or multiple docker images to run (app, database, redis, etc)

  • Easy to configure and replicate if necessary

  • Easy to add a new project

  • Easy to change the building steps. Everyone working on the project should be able to change if they want to run npm install or yarn install.

Installing Jenkins and Docker

Installing Jenkins is straightforward. You can visit Jenkins Installation page and choose the option that best suits your needs.

Here are the steps we followed to install Jenkins in AWS:

sudo rpm — import https://pkg.jenkins.io/debian/jenkins.io.key
sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat/jenkins.repo
sudo yum install java-1.8.0 -y
sudo yum remove java-1.7.0-openjdk -y
sudo yum install jenkins -y
sudo yum update -y
sudo yum install -y docker

Automatically adding projects from Github

Adding projects automatically from Github can be achieved using the GitHub Branch Source Plugin. It allows Jenkins to scan a GitHub organization for projects that match certain rules and add them to Jenkins automatically. The only constraint that all branches must meet in order to be added is that they contain a Jenkinsfile that explains how to build the project.

Easy to change configuration

Not so easy to change configuration

One of the biggest pains we had with our previous Jenkins was the difficulty of changing the steps necessary to build the project. If you looked at a project’s build steps, you would find something like this:

#!/bin/bash +x
set -e

# Remove unnecessary files
echo -e "\033[34mRemoving unnecessary files...\033[0m"
rm -f log/*.log &> /dev/null || true &> /dev/null
rm -rf public/uploads/* &> /dev/null || true &> /dev/null

# Build Project
echo -e "\033[34mBuilding Project...\033[0m"
docker-compose --project-name=${JOB_NAME} build

# Prepare test database
COMMAND="bundle exec rake db:drop db:create db:migrate"
echo -e "\033[34mRunning: $COMMAND\033[0m"
docker-compose --project-name=${JOB_NAME} run  \
	-e RAILS_ENV=test web $COMMAND

# Run tests
COMMAND="bundle exec rspec spec"
echo -e "\033[34mRunning: $COMMAND\033[0m"
unbuffer docker-compose --project-name=${JOB_NAME} run web $COMMAND

# Run rubocop lint
COMMAND="bundle exec rubocop app spec -R --format simple"
echo -e "\033[34mRunning: $COMMAND\033[0m"
unbuffer docker-compose --project-name=${JOB_NAME} run -e RUBYOPT="-Ku" web $COMMAND

And some post build steps that cleaned up the docker:

#!/bin/bash +x
docker-compose --project-name=${JOB_NAME} stop &> /dev/null || true &> /dev/null
docker-compose --project-name=${JOB_NAME} rm --force &> /dev/null || true &> /dev/null
docker stop `docker ps -a -q -f status=exited` &> /dev/null || true &> /dev/null
docker rm -v `docker ps -a -q -f status=exited` &> /dev/null || true &> /dev/null
docker rmi `docker images --filter 'dangling=true' -q --no-trunc` &> /dev/null || true &> /dev/null

Although these commands are not complex, changing any of them required someone with permissions to modify the job and an understanding ofwhat needed to be done.

Jenkinsfile to the rescue…​ or not

With the current Jenkins version, we can take advantage of Jenkins Pipeline and model our build flow in a file. This file is checked into the repository and, therefore, anyone with access to it can change the build steps. Yay!

Jenkins Pipeline even has support for:

  • Docker and multiple images can be used for a build!

  • Setting environment variables with withEnv and many other built -in functions that can be found here.

This makes a perfect case for Wolox. We can have our build configuration in a file that’s checked into the repository and can be changed by anyone with write access to it. However, a Jenkinsfile for a simple rails project would look something like this:

# sample Jenkinsfile. Might not compile
node {
    checkout scm
    withEnv(['MYTOOL_HOME=/usr/local/mytool']) {
        docker.image("postgres:9.2").withRun() { db ->
            withEnv(['DB_USERNAME=postgres', 'DB_PASSWORD=', "DB_HOST=db", "DB_PORT=5432"]) {
                docker.image("redis:X").withRun() { redis ->
                    withEnv(["REDIS_URL=redis://redis"]) {
                        docker.build(imageName, "--file .woloxci/Dockerfile .").inside("--link ${db.id}:postgres --link ${redis.id}:redis") {
                            sh "rake db:create"
                            sh "rake db:migrate"
                            sh "bundle exec rspec spec"
                        }
                    }
                }
            }
        }
    }
}

This file is not only difficult to read, but also difficult to change. It’s quite easy to break things if you’re not familiar with Groovy and even easier if you know nothing about how Jenkins’ pipeline works. Changing or adding a new Docker image isn’t straightforward and might lead to confusion.

Configuring Jenkins Pipeline via YAML

Personally, I’ve always envied simple configuration files for CIs and this time it was our chance to build CI that could be configured using a YAML file. After some analysis we concluded that a YAML like this one would suffice:

config:
  dockerfile: .woloxci/Dockerfile
  project_name: some-project-name

services:
  - postgresql
  - redis

steps:
  analysis:
    - bundle exec rubocop -R app spec --format simple
    - bundle exec rubycritic --path ./analysis --minimum-score 80 --no-browser
  setup_db:
    - bundle exec rails db:create
    - bundle exec rails db:schema:load
  test:
    - bundle exec rspec
  security:
    - bundle exec brakeman --exit-on-error
  audit:
    - bundle audit check --update


environment:
  RAILS_ENV: test
  GIT_COMMITTER_NAME: a
  GIT_COMMITTER_EMAIL: b
  LANG: C.UTF-8

It outlines some basic configuration for the project, environment variables that need to be present during the run, dependentservices, and our build steps.

Jenkinsfile + Shared Libraries = WoloxCI

After investigating for a while about Jenkins and the pipeline, we found that we could extend it with shared libraries. Shared libraries are written in groovy and can be imported into the pipeline and executed when necessary.

If you look carefully at this Jenkinsfile, we see that the code is a chain of methods calls that receive a closure, where we execute another method passing a new closure to it.

# sample Jenkinsfile. Might not compile
node {
    checkout scm
    withEnv(['MYTOOL_HOME=/usr/local/mytool']) {
        docker.image("postgres:9.2").withRun() { db ->
            withEnv(['DB_USERNAME=postgres', 'DB_PASSWORD=', "DB_HOST=db", "DB_PORT=5432"]) {
                docker.image("redis:X").withRun() { redis ->
                    withEnv(["REDIS_URL=redis://redis"]) {
                        docker.build(imageName, "--file .woloxci/Dockerfile .").inside("--link ${db.id}:postgres --link ${redis.id}:redis") {
                            sh "rake db:create"
                            sh "rake db:migrate"
                            sh "bundle exec rspec spec"
                        }
                    }
                }
            }
        }
    }
}

Groovy is flexible enough to allow this same declarative code to be created at runtime, making our dream of using a YAML to configure our job come true!

Introducing Wolox-CI

That’s how wolox-ci was born- our shared library for Jenkins!

With wolox-ci, our Jenkinsfile is now reduced to:

@Library('wolox-ci') _

node {

  checkout scm

  woloxCi('.woloxci/config.yml');
}

Now it simply checks out the code and then calls wolox-ci. The library reads yaml file like this one

config:
  dockerfile: .woloxci/Dockerfile
  project_name: some-project-name

services:
  - postgresql
  - redis

steps:
  analysis:
    - bundle exec rubocop -R app spec --format simple
    - bundle exec rubycritic --path ./analysis --minimum-score 80 --no-browser
  setup_db:
    - bundle exec rails db:create
    - bundle exec rails db:schema:load
  test:
    - bundle exec rspec
  security:
    - bundle exec brakeman --exit-on-error
  audit:
    - bundle audit check --update


environment:
  RAILS_ENV: test
  GIT_COMMITTER_NAME: a
  GIT_COMMITTER_EMAIL: b
  LANG: C.UTF-8

and builds the Jenkinsfile to get your job running on the fly.

The nice part about having a shared library is that we can extend and fix our library in a centralized way. Once we add new code, the library is automatically updated in Jenkins which will notify all of our jobs with the update.

Since we have projects in different languages we use Docker to build the testing environment. WoloxCI assumes there is a Dockerfile to build and will run all the specified commands inside the container.

Woloxci config.yml

Config

The first part of the config.yml file specifies some basic configuration: project’s name and Dockerfile location. The Dockerfile is used to build the image where the commands will be run.

Services

This section describes which services will be exposed to the container. Out of the box, WoloxCI has support for postgresql, mssql and redis. You can also specify the docker image version you want! It is not hard to add a new service. You just need to add the corresponding file at

https://github.com/Wolox/wolox-ci/tree/development/vars

and modify how the services are parsed

https://github.com/Wolox/wolox-ci/blob/development/src/com/wolox/parser/ConfigParser.groovy#L76

Steps

The listed commands in this section will run inside the Docker container. As a result, you’ll see each of the steps on the Jenkins UI.

image

Environment

If you need some environment variables during your build, you can specify them here. Whatever variable you set will be available inside the Docker container when your commands listed in the steps section described above.

Wrapping up

WoloxCI is still being tested with a not-so-small sample of our projects. The possibility of changing the build steps through a YAML file makes it accessible for everyone and that is a great improvement in our CI workflow.

Docker gives us the possibility of easily changing the programming language without making any changes to our Jenkins installation and Jenkins’ Github Organization feature automatically adds new projects when a new repository with a Jenkinsfile is detected.

All of these improvements have reduced the time we spend maintaining Jenkins significantly and give us the possibility of easily scaling without any extra configuration.

This library is working in our CI but it still can be improved. If you would like to add features, feel free to contribute!

About the author

Matias De Santi

Head of Infrastructure & Cloud at Wolox