Dec 17: Testing

This blog post is part of the FastMail 2014 Advent Calendar.

The previous post on 16th December was about Confidentiality at FastMail. The following post on 16th December is how we get money.

Technical level: low

You wake up, check your email and notice that a new Calendar feature has appeared overnight.

Behind the scenes, several months of planning and hard work goes in to providing our customers with dozens of innovative features before they’re released. Hundreds of thousands of customers connect to FastMail using devices ranging from feature phones to the latest iOS devices for business or personal email, web storage, notes, their calendar and address book. Our previous blog post on regular system monitoring explained how we keep systems running with near 100% uptime.

This blog post examines the development process at FastMail, specifically focusing on introducing new features and testing.

Almost every feature you use in the web interface has been written from scratch at some point over the past 15 years, many features are being migrated from the classic interface (which we maintain to support backwards compatibility) to the new AJAX interface. We test these features to help determine when we think they’re ready for release.

Each week FastMail’s developers add hundreds of lines of code to our Git repository, each line of code could potentially make our service unusable for one or more customers, break a feature on a specific platform or break the work-flow someone has been using for years.

Of course it’s impossible to verify that every feature still works in every possible way for every user every time anyone changes something - we would never release anything.

Anyone who has ever ended up on the advanced screen will know we provide significantly more than the average email service. Given the number of features we have and the number of customer scenarios, there are potentially an unlimited number of things we could test, but only 38 hours in our working week. To prioritise testing, we do risk-based testing to focus our testing effort on what we think is likely to have broken as a result of any significant changes. Any developer who is working on a major change, feature or project informs our QA engineer who tries to break the feature. His job is to consider how a feature may be used by our customers, how someone with less technical experience and knowledge than the person who developed the feature may use the it and how a malicious user may use or abuse the newly introduced/changed feature.

In addition to manual testing, we use Selenium which allows us to test the entire website in real web browsers (e.g. Chrome, Firefox, IE, etc) and control the web browser from code, so we can simulate clicks, mouse movements, etc just like a real person actually using the site would do.

As a small team it isn’t practical to maintain a copy of every version of every operating system with every version of every browser, so we use SauceLabs for most automated functional tests. Selenium WebDriver scripts perform actions such as logging in from a variety of browsers, navigating around the web interface, adding events and sending emails. Our tests run constantly so it becomes obvious who broke what when things go wrong.

The Selenium tests are run using Jenkins for each git commit. Jenkins failures are reported to our QA engineer who verifies whether an issue is a genuine failure and not an intended change before informing the appropriate developer.

We take security seriously and are happy to pay out reasonable bounties for serious bugs. This year we introduced a bug bounty program which has seen a number of issues fixed that could have otherwise gone unreported.

FastMail run a beta version of the web mail service where customers can try new features before they’re available to everyone. Major features are available on beta for several weeks before they go to live for all customers. If nothing significant is found internally and if Jenkins looks happy, we leave the change on our beta site for a few more days before making the new feature available to everyone. Beta customers provide valuable feedback via our beta feedback email address.

With the exception of emergency fixes, most changes go on to our beta server and are available for staff to test as well as anyone who’s interested in trying the latest features, in exchange for some instability. Calendar for example was available on beta for several months before it was launched in June with hundreds of issues being found and fixed during the beta period.

Most issues are discovered and fixed within a day. Internal communication is either done in person, via email or IRC. Issues that take a bit longer to fix go in to our bug tracking tool. Every software development team needs a bug tracking tool to keep track of known issues and help determine when a project is near completion. For historic reasons we were using JIRA. Unfortunately we’d recently been experiencing significant performance problems with the OnDemand hosted JIRA platform (the time to load the first page each day was >30 seconds in some cases!), and so ended up looking for an alternative. We moved to YouTrack, which so far has been performing much better for us.

We believe we offer the best email service on the market. However, it’s always possible to improve and we value all constructive feedback sent via our beta program. If you would like to try the latest features before they’re released and help improve the quality of service we provide, log in to the beta site at beta.fastmail.com.