ISP leak sends Google Cloud data to China, Russia
Google is in hot water (again) after a connectivity issue Monday resulted in traffic headed to its cloud platform instead pass through Russian and Chinese internet providers for over an hour.
This caused services to break for cloud customers— as well intermittent disruption caused to search and other services. The search giant has since acknowledged the incident as an error caused by updates by an ISP in China
By others, however, the breach in security was viewed as suspicious, particularly given the nature of the countries in which the data was misdirected and their relationship with the US. It also follows the recent rerouting of traffic belonging to Western carriers to mainland China by the Chinese government-owned China Telecom.
Spotted by network monitoring company ThousandEyes, which alerted followers on Twitter of a “potential hijack”, the leak was caused when the Nigerian-based MainOne Cable updated address books for key network hardware.
This resulted in data being sent the wrong way. China Telecom subsequently accepted the route and announced it worldwide, which in turn, saw Russia-based Transtelecom and other large ISPs to follow it.
BREAKING: Potential hijack underway. ThousandEyes detected intermittent availability issues to Google services from some locations. Traffic to certain Google destinations appears to be routed through an ISP in Russia & black-holed at a China Telecom gateway router. pic.twitter.com/Tz7shf7cOy
— ThousandEyes (@thousandeyes) November 12, 2018
According to Ars Technica, the redirected IP ranges transmitted some of Google’s most sensitive communications, including its WAN infrastructure and the Google VPN. A second announcement was then sent from MainOne which saw traffic sent to Cloudfare-owned IPs follow the same path and China Telecom taking the same action. Unsurprisingly, this compounded beliefs that the move was a deliberate breach.
We have investigated the advertisement of @Google prefixes through one of our upstream partners. This was an error during a planned network upgrade due to a misconfiguration on our BGP filters. The error was corrected within 74mins & processes put in place to avoid reoccurrence
— MainOne (@Mainoneservice) November 13, 2018
Commenting itself, MainOne has called the “misconfiguration” an error that was part of planned upgrades, a statement that was interestingly backed by the Cloudfare CEO, Matthew Prince.
“[…] there was a large network meeting in Nigeria a couple weeks ago (NgNOG). Those meetings always spur more peering—interconnecting networks that previously weren’t directly connected,” said Prince.
“While setting up a new interconnection, the Nigerian ISP almost certainly inadvertently leaked the routing information to China Telecom who then leaked it out to the rest of the world.
“If there was something nefarious afoot there would have been a lot more direct, and potentially less disruptive/detectable, ways to reroute traffic. This was a big, ugly screw up. Intentional route leaks we’ve seen to do things like steal cryptocurrency are typically far more targeted.”
Prince added that these kinds of vulnerabilities could be avoided if route redirections could be cryptographically signed and verified, urging the online “community” to collaborate and remedy the situation. “The merely trust-based BGP [Border Gateway Protocol] routing infrastructure remains one of the last remaining core bugs of the Internet and today we saw it rear its ugly head. High time we fix it.”
A statement from Google, meanwhile, sought to reassure that the leak was suspected as accidental and non-malicious, adding that with all affected traffic being encrypted, any harm that could result from malicious hijacking would be limited.