Cyrus.Works?

This is the fourteenth post in the 2016 FastMail Advent Calendar. Stay tuned for another post tomorrow.

---

What is Cyrus?

If youʼve been reading our Advent series blog posts, you might have noticed we make a significant contribution to Cyrus IMAP, the email, contacts and calendar server used by FastMail.

As with any development work, it’s in our interest to quickly find bugs we’ve introduced and fix them before they get to production. As well as contributing features/code, documentation, project management and maintenance/bug fixes, we also contribute Cassandane tests, and have recently set up a test server to run those tests for every commit.

Requirements

While itʼs great having loads of tests that prove your code is working in a development environment, we needed to set up a dedicated test server so that tests could be run more frequently.

When looking to build a CI server for Cyrus, we decided:

• Tests need to be run on a machine that wasn’t used by the developer who made the change so we know code will work in more than one place.
• Tests need to be run frequently and soon after changes have been committed, so developers get fast, useful feedback.
• The packages in the test environment should be similar to those on a machine where Cyrus will be installed.
• Results should be available online for anyone who is interested.

Introducing cyrus.works

www.cyrus.works is a Jenkins instance that runs tests within a Docker container.

As we need to run tests on a clean machine for each commit, one of the cleanest ways to do this is to run tests within a container. With Docker we can build a container for each platform we want to test against.

Having both the test environment and Jenkins inside a Docker container reduces the workload of our small test team as everything can be reset to a clean state within seconds.

For each commit, Jenkins will build a clean test environment, gather the latest files and build the Cyrus IMAP server. It will then run tests to make sure functionality is working as expected.

Security considerations

Unlike other Jenkins instances at FastMail, which are hosted on an isolated private network, we wanted cyrus.works to be on the internet so others can see what weʼre up to and everyone in the Cyrus team can see the most recent test results.

Running Jenkins inside Docker seems like a good idea. With a single command, we can bring up a Jenkins instance thatʼs mostly isolated from the rest of the host machine. However, Jenkins has a webUI that when misconfigured can allow bad people on the internet to do very bad things to your test machine.

Running Jenkins inside a Docker container seems like a good idea until you decide that you want to run your tests inside a different Docker container. By default, one Docker container cannot talk to another.

Possible solutions involved an ssh connection between containers as well as Docker inside Docker; both felt like the wrong way to do things. Others who had hit the same problem felt the same way, and had come up with a better solution: just exposing the Docker socket to the Jenkins CI container so it can start/stop the other containers. It does mean that if the Jenkins container is compromised than the entire test server is potentially compromised, but thatʼs no different to running Jenkins on the host machine.

Mitigation

In the event Jenkins does get compromised, weʼve isolated the machine from everything else at FastMail, including the rest of our test network. We only run open source software on the server and the only key it contains is one used by Github to trigger a build at Cyrus.works for each commit.

We also scripted the install, so we can completely format the machine, pull down the latest version of everything and reinstall in minutes.

Integration

Custom UI

Cyrus.works and cyrusimap.org share the same theme, thanks to jenkins-material-theme and the Simple Theme Plugin. In just a few seconds you can generate a CSS theme for your Jenkins instance.

Bort Integration

We use our Slack bot, Bort to assist us with company admin. The Jenkins Slack plugin can report test failures to Slack but as we didnʼt want this public Jenkins server to have our Slack API key, a bort plugin was written to query Jenkins.

What next?

As Cyrus is an open source project used by thousands of servers around the world, we want to make sure the features weʼre adding arenʼt breaking things for other users of Cyrus IMAP, all part of being a good internet citizen. Although all of our servers run Debian Jessie, in early 2017 weʼll start routinely testing against:

• Fedora
• Solaris
• SmartOS
• FreeBSD

If youʼre interested in contributing, the source code to build cyrus.works is on GitHub and youʼre welcome to join our weekly Cyrus meeting.