Date: March 21, 2019
Overview
This page gives some details on the www.cloudflare.com DNSSEC outage on March 21, 2019. CloudFlare
is one of the largest DNSSEC providers. I saw this DNSSEC outage at DNSViz, Verisign’s DNSSEC Debugger,
Google Public DNS, and DNS-OARC (both Unbound and BIND!), in addition to my 3 Unbound instances. This particular
outage was caused by a less common type of DNSSEC failure that I’ve only seen in CloudFlare and TinyDNSSEC.
Timeline / DNSViz
- 2019-03-21 21:34:53 UTC — Bogus NSEC
- 2019-03-21 21:42:28 UTC — Bogus NSEC
- 2019-03-21 21:42:38 UTC — Bogus NSEC
- 2019-03-21 21:43:35 UTC — last personally observed DNSSEC failure
DNSSEC Debugger
Unlike DNSViz, Verisign’s DNSSEC Debugger doesn’t archive
results, so here’s a screenshot of my web browser’s output from March 21, 2019:
Since the above image is confusing — why should a DS record for www.cloudflare.com matter?! — I’ve included an example screenshot of what this analysis is supposed to look like:
(Apparently www.cloudflare.com has NS records):
$ dig +short ns www.cloudflare.com.
jule.ns.cloudflare.com.
vin.ns.cloudflare.com.
Google Public DNS
DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let’s compare DNS queries with and without DNSSEC.
With Google Public DNS, because of DNSSEC, queries fail:
$ dig +dnssec a www.cloudflare.com. @8.8.8.8
; <<>> DiG 9.4.2-P2 <<>> +dnssec a www.cloudflare.com. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 60321
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;www.cloudflare.com. IN A
;; Query time: 43 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu Mar 21 21:42:15 2019
;; MSG SIZE rcvd: 47
You have to disable DNSSEC to make DNS queries work:
$ dig +cd a www.cloudflare.com. @8.8.8.8
; <<>> DiG 9.4.2-P2 <<>> +cd a www.cloudflare.com. @8.8.8.8
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35624
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;www.cloudflare.com. IN A
;; ANSWER SECTION:
www.cloudflare.com. 187 IN A 104.17.209.9
www.cloudflare.com. 187 IN A 104.17.210.9
;; Query time: 8 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu Mar 21 21:42:15 2019
;; MSG SIZE rcvd: 68
DNS-OARC: BIND
DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let’s compare DNS queries with and without DNS
SEC.
With DNS-OARC’s BIND instance, because of DNSSEC, queries fail:
$ dig +dnssec a www.cloudflare.com. @184.105.193.73
; <<>> DiG 9.4.2-P2 <<>> +dnssec a www.cloudflare.com. @184.105.193.73
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 268
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;www.cloudflare.com. IN A
;; Query time: 176 msec
;; SERVER: 184.105.193.73#53(184.105.193.73)
;; WHEN: Thu Mar 21 21:42:16 2019
;; MSG SIZE rcvd: 47
You have to disable DNSSEC to make DNS queries work:
$ dig +cd a www.cloudflare.com. @184.105.193.73
; <<>> DiG 9.4.2-P2 <<>> +cd a www.cloudflare.com. @184.105.193.73
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50288
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;www.cloudflare.com. IN A
;; ANSWER SECTION:
www.cloudflare.com. 294 IN A 104.17.209.9
www.cloudflare.com. 294 IN A 104.17.210.9
;; Query time: 21 msec
;; SERVER: 184.105.193.73#53(184.105.193.73)
;; WHEN: Thu Mar 21 21:42:16 2019
;; MSG SIZE rcvd: 68
DNS-OARC: Unbound
DNSSEC can be disabled in queries via the CD (checking disabled) bit. Let’s compare DNS queries with and without DNS
SEC.
With DNS-OARC’s Unbound instance, because of DNSSEC, queries fail:
$ dig +dnssec a www.cloudflare.com. @184.105.193.74
; <<>> DiG 9.4.2-P2 <<>> +dnssec a www.cloudflare.com. @184.105.193.74
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 14194
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;www.cloudflare.com. IN A
;; Query time: 21 msec
;; SERVER: 184.105.193.74#53(184.105.193.74)
;; WHEN: Thu Mar 21 21:42:16 2019
;; MSG SIZE rcvd: 47
You have to disable DNSSEC to make DNS queries work:
$ dig +cd a www.cloudflare.com. @184.105.193.74
; <<>> DiG 9.4.2-P2 <<>> +cd a www.cloudflare.com. @184.105.193.74
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 47381
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;www.cloudflare.com. IN A
;; ANSWER SECTION:
www.cloudflare.com. 295 IN A 104.17.210.9
www.cloudflare.com. 295 IN A 104.17.209.9
;; Query time: 24 msec
;; SERVER: 184.105.193.74#53(184.105.193.74)
;; WHEN: Thu Mar 21 21:42:16 2019
;; MSG SIZE rcvd: 68
Logfile examples
These log entries come from 3 different Unbound instances, all on different computers in different geographical regions.
- [1553204064] unbound[40945:0] info: validation failure <www.cloudflare.com. A IN>: signer name mismatches key name from 162.159.3.11 for DS www.cloudflare.com. while building chain of trust
- [1553204530] unbound[99145:0] info: validation failure <www.cloudflare.com. A IN>: signer name mismatches key name from 162.159.9.55 for DS www.cloudflare.com. while building chain of trust
- [1553204536] unbound[40945:0] info: validation failure <www.cloudflare.com. A IN>: signer name mismatches key name from 162.159.0.33 for DS www.cloudflare.com. while building chain of trust
- [1553204615] unbound[15225:0] info: validation failure <www.cloudflare.com. A IN>: signer name mismatches key name from 162.159.4.8 for DS www.cloudflare.com. while building chain of trust
