SPF, DKIM & DMARC: email anti-spoofing technology history and future
Post categories
Founder & CTO
This is the twenty fourth and final post in the 2016 FastMail Advent Calendar. Thanks for reading, and as always, thanks for using FastMail!
Quick, where did this email come from and who was it sent to?
From: PayPal <service@paypal.com.au>
To: Rob Mueller <robm@fastmail.fm>
Subject: Receipt for your donation to Wikimedia Foundation, Inc.
Actually, these headers tell you nothing at all about where the email really came from or went to. There are two separate parts to the main email standards. RFC5322 (originally RFC822/RFC2822) specifies the format of email messages, including headers like from/to/subject and the body content. However, it doesnʼt specify how messages are transmitted between systems. RFC5321 (originally RFC821/RFC2821) describes the Simple Mail Transfer Protocol (SMTP) which details how messages are sent from one system to another.
The separation of these causes a quirk: the format of a message need not have any relation to the source or destination of a message. That is, the From/To/Cc headers you see in an email may not have any relation to the sender of the message or the actual recipients used during the SMTP sending stage!
When the email standards were developed, the internet was a small network of computers at various universities where people mostly knew each other. The standard was developed with the assumption that the users and other email senders could be trusted.
So, the From header would be a userʼs own email address. When you specified who you wanted to send the message to, those addresses would be put in the To header and used in the underlying SMTP protocol delivering the messages to those people (via the RCPT TO command in SMTP). This separation of email format and transport also allows features like Bcc (blind carbon copy) to work. Any addresses a message is bccʼd to donʼt appear in the message headers, but are used in the underlying SMTP transport to deliver the message to the right destination.
Over time of course, this assumption of a friendly environment became less and less true. We now live in a world where much of the internet is downright hostile. We need to heavily protect our systems from mountains of spam and malicious email, the much of it designed to trick people.
There are many layers of protection from spam, from RBLs to detect known spam sending servers, to content analysis that helps classify messages as spammy or not. In this, we want to talk the major anti-spoofing techniques that have been developed for email.
SPF
One of the earliest anti-spoofing efforts was SPF (Sender Policy Framework). The idea behind SPF was that senders could specify, via an SPF record published in DNS, what servers were allowed to send email for a particular domain. For example, only servers X, Y & Z are allowed to send email for @fastmail.com addresses.
Unfortunately, SPF has many problems. For starters, it only works on the domain in the SMTP sending protocol, known as the MAIL FROM envelope address. No email software ever displays this address. (Its main use is where to send error/bounce emails if final delivery fails.) In a world where what the recipient can be anything, thereʼs no need for the MAIL FROM address to match the From header address in any way. So effectively, the only thing youʼre protecting against is the spoofing of an email address no one ever sees.
In theory, this does help to address one particular type of spam. It helps reduce backscatter email. Backscatter what you see when messages spammers sent pretending to be you can’t be delivered.
In practice, it would only do that if people actually blocked email that failed SPF checks at SMTP time. They rarely do that because SPF has a major problem. It completely breaks traditional email forwarding. When a system forwards an email, itʼs supposed to preserve the MAIL FROM address so any final delivery failures go back to the original sender. Unfortunately, that means when someone sends from Hotmail to FastMail, and then you forward on from FastMail to Gmail, in the FastMail to Gmail hop, there’s a mismatch. The MAIL FROM address will be an @hotmail.com domain, but the SPF record will say that FastMail isnʼt allowed to send email with an @hotmail.com domain address!
There was an attempt to fix this (SRS), but itʼs surprisingly complex. Given the relatively low value of protection SPF provides, not many places ended up implementing SRS. The situation we ended up with is that SPF is regarded as a small signal for email providers’ use. If SPF passes, itʼs likely the email is legitimately from the domain in the MAIL FROM address. If it fails, well… thatʼs not really much information at all. It could be a forwarded email, it could be a misconfigured SPF record, or many other things. But stay tuned for its next life in DMARC.
DKIM
DKIM (DomainKeys Identified Mail) is a significantly more complex and interesting standard compared to SPF. It allows a particular domain owner (again, via a record published in DNS) to cryptographically sign parts of a message so that a receiver can validate that they havenʼt been altered.
DKIM is a bit fiddly at the edges and took a while to get traction, but is now commonly used. Almost 80% of email delivered to FastMail is DKIM signed.
So letʼs take the message we started with at the top and add a DKIM signature to it.
DKIM-Signature: v=1; a=rsa-sha256; d=paypal.com.au; s=pp-dkim1; c=relaxed/relaxed;
q=dns/txt; i=@paypal.com.au; t=1480474251;
h=From:From:Subject:Date:To:MIME-Version:Content-Type;
bh=Vn79RZZBrNIu4HFwGMOOAezyw/2Ag+w+avW1yscPcUw=;
b=...
From: PayPal <service@paypal.com.au>
To: Rob Mueller <robm@fastmail.fm>
Subject: Receipt for your donation to Wikimedia Foundation, Inc.
Using a combination of public key cryptography and DNS lookups, the receiver of this email can determine that the domain “paypal.com.au” signed the body content of this email and a number of the email’s headers (in this case, From, Subject, Date, To and a couple of others.) If it validates, we know the body content and specified headers have not been modified by anyone along the way.
While this is quite useful, there are still big questions that arenʼt answered.
What about emails with a From address of @paypal.com.au that arenʼt DKIM signed by paypal.com.au? Maybe not every department within PayPal has DKIM signing correctly set up. Should we treat unsigned emails as suspicious or not?
Also, how do I know if I should trust the domain that signs the email? In this case, paypal.com.au is probably owned by the Australian division of PayPal Holdings, Inc, but what about paypal-admin.com? Itʼs not obvious what domains I should or shouldnʼt trust. In this case, the From address matches the DKIM signing domain, but that doesnʼt need to be the case. You can DKIM sign with any domain you want. Thereʼs nothing stopping a scammer using an @paypal.com.au address in the From header, but signing with the paypal-admin.com domain.
Despite this, DKIM provides real value. It allows an email receiver to associate a domain (or multiple, since multiple DKIM signatures on an email are possible and in some cases useful) with each signed email. Over time, the receiver can build up a trust metric for that domain and/or associated IPs, From addresses, and other email features. This helps discriminate between “trusted” emails and “untrusted” emails.
DMARC
DMARC (Domain-based Message Authentication, Reporting & Conformance) attempts to fix part of this final trust problem by building on DKIM and SPF. Again, by publishing a record in DNS, domain owners can specify what email receivers should do with email received from their domain. In the case of DMARC, we consider email to be from a particular domain by looking at the domain in the From header: – the address you see when you receive a message…
In its basic form, when you publish a DMARC record for your domain receivers should:
- Check the From header domain matches the DKIM signing domain (this is called alignment), and that the DKIM signature is valid.
- Check the the From header domain matches the SMTP MAIL FROM domain, and that the senderʼs IP address is validated by SPF.
If either is true, the email “passes” DMARC. If both fail, the DMARC DNS record specifies what the receiver should do, which can include quarantining the email (sending it to your spam folder) or rejecting the email. Additionally, the DMARC record can specify an email address to send failure reports to. DMARC also allows senders to specify which percentage of their mail to apply DMARC to, so they can make changes in a gradual and controlled way.
So back to our example email:
DKIM-Signature: v=1; a=rsa-sha256; d=paypal.com.au; s=pp-dkim1; c=relaxed/relaxed;
q=dns/txt; i=@paypal.com.au; t=1480474251;
h=From:From:Subject:Date:To:MIME-Version:Content-Type;
bh=Vn79RZZBrNIu4HFwGMOOAezyw/2Ag+w+avW1yscPcUw=;
b=...
From: PayPal <service@paypal.com.au>
To: Rob Mueller <robm@fastmail.fm>
Subject: Receipt for your donation to Wikimedia Foundation, Inc.
In this case, the From header domain is paypal.com.au. Letʼs check if they publish a DMARC policy.
$ dig +short _dmarc.paypal.com.au TXT
"v=DMARC1; p=reject; rua=mailto:d@rua.agari.com; ruf=mailto:dk@bounce.paypal.com,mailto:d@ruf.agari.com"
Yes, they do. letʼs run our checks! Does the From domain match the DKIM signing domain paypal.com.au? Yes, so we have alignment. If the email wasnʼt DKIM signed, or if it were DKIM signed but the domain had been paypal-admin.com (e.g. signed by a scammer), then there wouldnʼt have been alignment, and so DMARC would have failed. At that point, we would have consulted the DMARC policy, which specifies p=reject, which says that we should just reject the forged email.
In this case (I havenʼt included the entire DKIM signature, but I can tell you it validated), the email did pass DMARC. So we can accept it. Because of alignment, we know the domain in the From address also matches the DKIM signing domain. This allows users to be sure that when they see a From: @paypal.com.au address, they know that itʼs a real message from paypal.com.au, not a forged one!
This is why DMARC is considered an anti-phishing feature. It finally means that the domain in the From address of an email canʼt be forged (at least for domains that DKIM sign their emails and publish a DMARC policy). All that, just to ensure the domain in the From address canʼt be forged, in some cases.
Unfortunately, as is often the case, this feature also brings some problems.
DMARC allows you to use SPF or DKIM to verify a message. If you donʼt DKIM sign a message and rely only on SPF, when a message is forwarded from one provider to another, DMARC will fail. If you have a p=reject policy setup, the forwarding will fail. Unlike in SPF where failure is a “weak signal”, a DMARC policy is supposed to tell receivers more strictly what to do, making bounces a strong possibility.
The solution: always make sure you DKIM sign mail if you have a DMARC policy. If your email is forwarded, SPF will break, but DKIM signatures should survive. SRS wonʼt help with DMARC, because replacing the MAIL FROM envelope with your own domain means the MAIL FROM domain doesnʼt match the From header domain. This is an alignment failure, and so not a pass result for DMARC.
I say “should survive”, because, again, not all providers are great at that. In theory, forwarding systems preserve the complete structure of your message. Unfortunately, thatʼs not always the case. Even large providers have problems with forwarding inadvertently slightly altering the content/structure of an email (Exchange based systems (including outlook.com) and iCloud are notorious for this). Even a slight modification can and will break DKIM signatures. Again, combined with a DMARC p=reject policy, this can result in email being rejected.
The solutions in this case are to:
- Bug those providers to fix their email forwarding and not to modify the email in transit. DKIM is now a well established standard; providers should ensure their forwarding doesnʼt break DKIM signatures.
- Switch to using POP to pull email from your remote provider. We donʼt do SPF/DKIM/DMARC checking on emails pulled from a remote mailbox via POP.
- Donʼt forward this mail. Wherever the emails are coming from, change your email address at that service provider so it points directly to your FastMail email address and avoids forwarding altogether.
Thereʼs one other case thatʼs a known big issue with DMARC: mailing lists. Mailing lists can be considered a special case of email forwarding: you send to one address, and itʼs forwarded to many other addresses (the mailing list members). However, itʼs traditional for mailing lists to modify the emails during forwarding, by adding unsubscribe links or standard signatures to the bottom of every message and/or adding a [list-id] tag to the message subject.
DKIM signing subjects and message bodies is very common. Changing them breaks the DKIM signature. So, if the sender’s domain has a p=reject DMARC policy, then when the mailing list software attempts to forward the message to all the mailing list members, the receiving systems will see a broken DKIM signature and thus reject the email. (This was actually a significant problem when Yahoo and AOL both enabled p=reject on their user webmail service domains a few years ago!)
Fortunately, thereʼs a relatively straight forward solution to this. Mailing list software can rewrite the From address to one the mailing list controls, and re-sign the message with DKIM for that domain. This and a couple of other solutions are explained on the DMARC information website. These days, the majority of mailing list software systems have implemented one of these changes, and those that havenʼt will very likely have to when Gmail enables p=reject on gmail.com sometime early next year. Not being able to forward emails from the worldʼs largest email provider will definitely hamper your mailing list.
These authentication systems affect FastMail in two ways. What we do for email received from other senders, and what we do when sending email.
SPF, DKIM & DMARC for email received at FastMail
Currently, FastMail does SPF, DKIM and DMARC checking on all incoming email received over SMTP (but not email retrieved from remote POP servers).
Passing or failing SPF and/or DKIM validation only adjusts a message’s spam score. We donʼt want to discriminate against a failing DKIM signature for an important domain, and we donʼt want to whitelist a spammy domain with a valid DKIM signature. A DKIM signature is treated as context information for an email, not a strong whitelist/blacklist signal on its own.
For DMARC, the domain owners are making a strong statement about what they want done with email from their domains. For domains with a p=quarantine policy, we give failing emails a high spam score to ensure they go to the userʼs Spam folder. For domains with a p=reject policy, we donʼt currently reject at SMTP time but effectively still do a quarantine action with an even higher score. We hope to change this in the future after adding some particular exceptions known to cause problems.
We add a standard Authentication-Results header to all received emails explaining the results of SPF, DKIM and DMARC policies applied. Surprisingly, existing software to do this was not well maintained or buggy, so we ended up writing an open source solution we hope others will use.
Back to our example again. Hereʼs that PayPal email with the corresponding Authentication-Results header.
Authentication-Results: mx2.messagingengine.com;
dkim=pass (2048-bit rsa key) header.d=paypal.com.au header.i=@paypal.com.au header.b=PVkLotf/;
dmarc=pass header.from=paypal.com.au;
spf=pass smtp.mailfrom=service@paypal.com.au smtp.helo=mx2.slc.paypal.com
DKIM-Signature: v=1; a=rsa-sha256; d=paypal.com.au; s=pp-dkim1; c=relaxed/relaxed;
q=dns/txt; i=@paypal.com.au; t=1480474251;
h=From:From:Subject:Date:To:MIME-Version:Content-Type;
bh=Vn79RZZBrNIu4HFwGMOOAezyw/2Ag+w+avW1yscPcUw=;
b=...
From: PayPal <service@paypal.com.au>
To: Rob Mueller <robm@fastmail.fm>
Subject: Receipt for your donation to Wikimedia Foundation, Inc.
You can see SPF, DKIM, and DMARC all passed.
The information in this header is used by other parts of the FastMail system. For instance, if youʼve added service@paypal.com.au to your address book to whitelist it, weʼll ignore the whitelisting if DMARC validation fails. This ensures that a scammer canʼt create an email with a forged From address of service@paypal.com.au and get it into your Inbox because youʼve whitelisted that From address.
SPF, DKIM & DMARC for FastMail and user domains
All FastMail domains currently have a relaxed SPF policy (by design because of legacy systems, see DMARC below) and we DKIM sign all sent email. We actually sign with two domains, the domain in the From header, as well as our messagingengine.com domain. This is to do with some Feedback Loops, which use the DKIM signing domain to determine the source of the message.
For user domains, weʼll also publish a relaxed SPF policy and a DKIM record if you use us to host the DNS for your domain. If you use another DNS provider, you need to make sure you copy and publish the correct DKIM record at your DNS provider. Once we detect itʼs setup, weʼll start DKIM signing email you send through us.
Currently, FastMail doesnʼt have a DMARC policy for any of our domains, and we donʼt publish a default policy for user domains either. This means that users can send emails with @fastmail.com From addresses from anywhere. This is a bit of a legacy situation. When FastMail started more than 16 years ago, none of these standards existed. It was common for people to set up all sorts of convoluted ways of sending email with the assumption they could send with any From address they wanted. (Old internet connected fax/scanner machines are a particularly notorious example of this.)
Over time, this is becoming less and less true, and more and more people are expecting that emails will be DKIM signed and/or have valid SPF and/or have a DMARC policy set for the domain. Itʼs likely sometime in the future weʼll also enable a p=reject policy for our domains. To send with an @fastmail.com/@sent.com/etc From address, youʼll have to send through our servers. This is perfectly possible with authenticated SMTP, something basically everything supports these days.
Ongoing problems
Even though DMARC allows us to verify that the domain in the From header actually sent and authenticated the email and its contents, a great anti-phishing feature, itʼs still a long way from stopping phishing. As we personally experienced, people donʼt check their emails with a skeptical eye. We regularly saw phishing emails sent to FastMail users like:
From: No Reply <joeblogs@completelyrandomsite.com>
To: foobar@fastmail.com
Subject: Urgent! Your account is going to be closed!
Click [here](http://example.com) right now or your account will be closed
Enough people clicked on it, and filled in the login form on a bogus website (that didnʼt even look much like FastMail), that weʼd see multiple stolen accounts daily. Unfortunately, trying to educate users just doesnʼt seem to work.
One of the main advantages of email is that itʼs a truly open messaging system. Anyone in the world can set up an email system and communicate with any other email system in the world. Itʼs not a walled garden controlled by a single company or website. This openness is also its biggest weakness, since it means legitimate senders and spammers/scammers are on an equal footing. This means that email will continue its evolutionary arms race between spammers/scammers and receivers into the future, trying to determine if each email is legitimate using more and more factors. Unfortunately this means there will always be false positives (emails marked as spam that shouldnʼt be) and false negatives (spam/scam emails that make it through to a persons inbox). Thereʼs never going to be a “perfect” email world, regardless of what systems are put in place, but we can keep trying to get better and better.
Email authentication in the future
Although the main problem of mailing lists’ incompatibility with DMARC p=reject policies has mostly been solved, it creates another problem in that receivers have to use the trust of the mailing list provider domain. This provides an incentive for spammers to target mailing lists, hoping for laxer spam checking controls that will forward the email to final receiving systems that will trust the mailing list provider. An emerging standard called ARC attempts to let receivers peer back to previous receiving servers in a trusted way so they can see authentication results and associated domains from earlier stages of a multi-step delivery path.
One thing we would like to see is some way to associate a domain with some real world. One way would be to piggy back on the SSL Extended Validation (EV) Certificate system. Obtaining an EV certificate requires proof of the requesting entity’s legal identity. You see this in web browsers when you navigate to sites that use an EV certificate. For instance our site uses an EV certificate (https://www.fastmail.com) and browsers will show “FastMail Pty Ltd” in the address bar. Being able to display a clear “PayPal, Inc” next to emails legitimately from PayPal or any other financial institution would seem to be a significant win for users (modulo the slightly sad results we already found regarding users falling for phishing emails).
Unfortunately, there’s no standard for this now and nothing on the horizon, and it’s not entirely obvious how to do this without support from the senders. A naive approach that doesn’t require sender changes would be to extract the domain from a From header address and attempt to make an https:// connection to it. But there’s all sorts of edge cases. For instance, PayPal uses country specific domains for DKIM signing (e.g. paypal.com.au), but if you go to http://paypal.com.au in a web browser, it redirects to https://www.paypal.com. You can’t just follow any redirect, because a scammer could setup paypal-scam.com and redirect to https://www.paypal.com. Working out what redirects should actually be followed is entirely non-trivial.
Coda
This post has turned out significantly longer than I originally anticipated, but it shows just how complex a subject email authentication is in a modern context. In many cases, FastMail tries hard to make these things “just work”, both as a receiver from other systems, and if you’re a customer as a sender. If you host DNS for your domain with us, we setup an SPF and DKIM signing records automatically for you. We don’t currently setup a DMARC record (there are still too many different ways people send email), but we hope in the future to allow easier construction and rollout of DMARC records for user domains.