Dec 22: The year of the Cyrus Foundation
Post categories
CEO
This is the twenty-second post in the FastMail 2015 Advent Calendar. Stay tuned for another post tomorrow.
This post is (for me at least) not very technical at all — despite being about a very technical part of our infrastructure. Instead, I’ll talk of some of the human factors in open source software development, particularly when most of the people involved are working at companies which use that software.
This year has been an interesting year for Cyrus IMAPd.
We started the year with grand plans of a foundation of multiple companies working together to build out a full stack on top of Cyrus. FastMail hired Ellie to work 100% on Cyrus. She’s taken over stable releases and bug triage, as well as working on new feature development. We also decided that we needed somebody to focus on the documentation, and FastMail hired a technical writer (Nicola) for a 50% Cyrus/FastMail role. We wound up being so impressed with two of the candidates who applied that we hired the other one (David) for a marketing role.
FastMail has also paid two of the contractors who have been working on the project throughout the year, and other companies have contributed staff time, as well as some volunteers giving their own time. In particular, Ken from CMU has been working on Cyrus even longer than I have, and Pierre from Netmail and MX Toolbox paid for a lot of conference and contractor expenses.
The grand plan
We started the year with a lot of momentum. Our plan was to develop Cyrus version 3.0 as a public release with object storage support and improved clustering, as well as advancing the JMAP protocol as a better alternative to proprietary email schemes.
Cyrus 3.0 was going to be released at OSCON in Portland during July, with great fanfare.
My “other” hobby is singing in choirs, and I flew directly from a choir festival in Sydney to San Francisco for the CalConnect conference, and then via Toronto where I met with people from Netmail/MX Toolbox during the day and kept flying that night to Brussels for Fosdem. In all, I was away from Australia for 10 days, going right around the world!
I had a lightning talk slot at Fosdem where I spoke about JMAP, but the main benefit of my time there was reconnecting with the Kolab people, who also use Cyrus. I left with an invitation to speak at the Kolab summit in April, which FastMail wound up helping sponsor.
We registered the domain name cyrus.foundation and Kolab set up infrastructure there for everyone to use. Our primary repositories are still running there.
Project momentum
The project has always gone through phases of higher and lower activity. After years of very few changes, we released 2.4 back in 2010, and the couple of years after that were quite active as we ironed out the bugs. Once 2.4 was running reliably for everybody, momentum died down except for the internal FastMail branches where big changes were being developed, and the CMU branch where the CalDAV/CardDAV work was being done by Ken.
Our first project in 2015 was to bring those branches together. We’d already started this in late 2014 when I visited Ken in Pittsburgh and we spent a few days pouring over commits and merging them. Between FastMail’s changes and CMU’s changes, there was a lot to merge back together.
With Kolab bringing resources, and Pierre hiring a contractor from Canada as well, we suddenly had a lot more coordination required than when each group was working in isolation, so we instituted two teleconferences per week at different times to allow everyone to talk about what they were working on, and to deal with any issues which had come up or questions about areas of code which others knew better.
The rocky road to OSCON
It was always an aggressive timeline, and we hit snags almost straight away. The developer that Pierre had been sponsoring in Canada didn’t fit well with the culture of the rest of the group, and after a few frustrating weeks, he left the project.
FastMail hired another contractor (Conrad) — this time in France, and we kept up two meetings per week to discuss progress. I had a chance to meet Conrad in Lille at the OpenIO office while I was over there for the Kolab summit. It was great to have Conrad working with the OpenIO team because they had a shared language! Pierre hired another programmer, Raymond, who contributed a skeleton object storage backend and integrated with both OpenIO along with Conrad, and also a module Caringo for the Caringo object store. Plenty of meetings during that time would break into French for the complex bits, as Raymond also spoke the language.
We had a demo of object storage ready for OSCON (based on Open IO), but it wasn’t as feature complete as we’d hoped, due to the slowdowns caused by changing staff part way through.
Meanwhile Ellie back in Melbourne had been busy with bug fixes and release management, and I was largely working on CalDAV fixes and preparing CardDAV for release, so our parts weren’t as far along as we’d hoped either.
OSCON
But we made it to the conference — everyone kind of rushed with last minute preparations.
We had a pretty solid product, and best of all — FastMail’s internal branch had gone from over 1000 separate patches to just a handful of very custom site-specific patches to the code — which meant that the upstream code was being tested by hundreds of thousands of real users.
On the JMAP side — well, we’ll talk about that tomorrow, but we had a very compelling demo.
We spent a bunch of money on having a booth and a sponsored day of talks at OSCON. The talks were great, but not very well attended. Seeing the other sponsored rooms, they appeared to largely be content that wasn’t good enough to make the main tracks. Our room was small, and it never got particularly full — mostly just people we had specifically invited.
I think if we had actually submitted our work for the main tracks we may have reached more people. We’re an open source project and we are doing great things, which is exactly what OSCON is about.
I felt like having a sponsored room made us seem dodgier than we are, which is a shame.
Radio silence
And then the reality of being a bunch of people employed by other organisations, with other priorities, really hit. Kolab had basically gone to radio silence a little before OSCON. We got them mentioned on the sponsor list and at the booth, but they were unable to send anybody, and disappeared from the channels and the meetings. They have been busy with other things, and Cyrus development isn’t their primary priority.
Without a customer project backing his work, Raymond’s funding dried up, and he pulled back from the project.
Conrad’s studies and other work options took him away. I take some of the responsibility for not having the time to really guide him or allocate projects wisely here too — he was left at a loose end for much of the time, just looking around the codebase for things to work on, but with no external driver or key responsibility to give him purpose.
So it fell back to mostly CMU and FastMail again. Especially once FastMail hired another contractor (Robert — just because the three Roberts that we already have at FastMail weren’t enough for us) to work specifically on JMAP support directly in Cyrus.
We have reduced back to just a single meeting per week now that it’s Ken plus FastMail staff in the meetings, because it’s very early for Ken and very late for the Australians at this time of year, but the other way around doesn’t get Europe at all, so we persist with 11am GMT.
The work continues
I’ve posted plenty about the kind of things we are doing — for FastMail the driver is always our production infrastructure first, but with a firm eye on providing good stewardship to the project as whole and ensuring we don’t break things for others.
As you can see, the rate of change has dropped off towards the end of the year, but …
… we’re still working a lot faster than we have at any time in the past. Which is not surprising. Not only is Ken doing good work, and Ellie integrating patches from the mailing lists and bug trackers — we also have Robert doing tons of JMAP work, Ellie’s backup work and Chris (testing), Nicola (docs) and myself giving part of our time to the project.
Test all the things
I added support for testing the notify protocol (mboxevents, sieve notifies, alarms) into our Cassandane test suite. It has basic tests for CalDAV and CardDAV as well as the IMAP/POP3 and LMTP basics.
It’s still very incomplete in terms of total code coverage, but it does wrap the ImapTest suite and run all of the tests which are relevant to Cyrus’ massive set of supported RFCs.
When we hit bugs or regressions now, the first step is to add a Cassandane test which proves the bug, and then we can confirm the fix as well.
Split infrastructure
The initial momentum to work on the cyrus.foundation domain name has died, since we don’t really have a Cyrus Foundation operating, and since the hosting is with company which isn’t involved in day-to-day development any more. We’re moving bits and pieces back onto the cyrusimap.org domain and the virtual machines at CMU. There’s also testing equipment running on hardware owned by FastMail, because we have spare high-powered machines set aside for it.
Next year we plan to consolidate all our documentation and bug tracking into one place, probably back at cyrusimap.org.
Future development
We’d love to have more companies and individuals involved. Particularly those who are using Cyrus and have a stake in its development. Development direction is determined by those who show up! You can join the #cyrus IRC channel on Freenode and/or the mailing lists. It’s not just code that we need — test cases, documentation, encouragement, and even feature requests are gratefully received.
FastMail have our own massive plans — getting the murder aggregation protocol into a state that FastMail can run it. Rebuilding our conversations database as a more general cross-folder/labels approach to email which will be a better fit to JMAP. Speaking of which, having Cyrus run natively as a JMAP server with our open source client on top. There’s more to do with search as well.
And we’d love to get closer to a shared-nothing architecture. It was always more the foundation’s goal than FastMail’s, but right now we run on top of RAID6 units for our main storage. We replicate that storage to a minimum of 3 full copies, uncompressed. We could get better economies of scale on an object storage backend with its own redundancy management for email storage.
Cyrus is the workhorse behind our service, and we’re pouring our resources into making it the best server for us. Come join the meetings at 11am every Monday, get involved in the mailing lists, report bugs, and help make Cyrus the best server for everyone.