Ebay Products
You might have heard about a topic that’s gaining some attention in industry discussions (such as in the recent ZDnet.com article), about an event that could potentially cause significant disruptions across the Internet—the so-called “768k Day”. This day is the point in time sometime in the near future (some speculate in the coming month) when the size of the global BGP routing table is expected to exceed 768,000 entries. Why is this a big deal?
In 2014, on what we now know as “512k Day”, the IPv4 Internet routing table exceeded 512,000 BGP routes when Verizon advertised thousands more routes to the Internet. Many ISP and other organizations had provisioned the size of the memory for their router TCAMs for a limit of 512K route entries, and some older routers suffered memory overflows that caused their CPUs to crash. The crashes on old routers, in turn, created significant packet loss and traffic outages across the Internet, even in some large provider networks. Engineers and network administrators scrambled to apply emergency firmware patches to set it to a new upper limit. In many cases, that upper limit was 768k entries.
Figure 1: 512k day caused havoc on the Internet.
512K Day was an Internet milestone like the so-called “y2K” crisis, and a wake-up call for a lot of ISP and Internet organizations. The looming breach of the global BGP routing table threshold wasn’t a secret, yet enough providers were caught flat-footed due to outdated network equipment that it had a ripple effect in terms of the number of Internet outages it caused.
Fast forward five years later, and the upcoming 768k Day is an echo of 512k Day, just with a higher threshold. So, some are worried that the Internet could have similar problems. This time around, one thing that we have going for us is that most large providers who felt the impact of 512k Day have learned their lessons, and have probably prepped and maintained their infrastructures reasonably well. As a result, we would hope not to see the same level of outages.
That said, while nobody’s exactly hyperventilating about 768k day, there are still a lot of smaller ISPs, data centers and other providers who are part of the fabric of the Internet. When you look at Internet paths, a good amount of service traffic transits through these ‘soft spots’ of Internet infrastructure, if you will—where maintenance on legacy routers and network equipment can be neglected or missed more easily. Given the sheer size and unregulated nature of the Internet, it’s fair to say that things will be missed.
Which means it’s still entirely possible that we’ll see some issues or outages due to 768k Day in the next month or so. Of course, there is a myriad of outage events that happen every day, especially on the fringes of the Internet. The number of garden variety outages could get amplified because of 768k-related issues.
What would a 768k day issue look like in a particular provider network? Perhaps something like a recent outage we shared about where we could see total packet loss for monitoring tests that were crossing various router interfaces, as seen below in figure 2. In this case, we saw packet loss on several interfaces in the Cogent (AS 174) network in the San Francisco Bay area, that affected peer ISPs like Comcast and Qwest, as well as services like Amazon, Verisign, 8×8 and even enterprises like Athena Health.
Figure 2: A visualization of a traffic outage in the Cogent network.
So it should go without saying that if you maintain routers that receive full Internet routes, then before 768k Day arrives, make sure you’ve performed some preventative maintenance. ThousandEyes monitors apps, services, and networks, and will be watching for any Internet traffic outages that occur on 768k Day and will report on anything of note. So stay tuned to our Twitter feed and subscribe to our blog.
