SOURCE: yahoo!finance
- Technology Editor
Tue, October 5, 2021, 2:02 PM
Facebook’s (FB) week went from bad to worse Monday when the social networking giant’s apps went dark worldwide for roughly six hours. In a post written Monday afternoon, Facebook VP of infrastructure Santosh Janardhan blamed the outage on a configuration error on the “backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication.”
That error, Janardhan writes, had a cascading effect on Facebook’s entire network, both inside and outside of the company, knocking out its services around the world. Because of that, employees couldn’t fix the problem remotely, forcing the company to dispatch engineers to one of its data centers to perform a manual reset, according to a memo obtained by The New York Times.
While the company doesn’t go into specifics about what exactly went wrong, journalist Brian Krebs, citing sources within Facebook, pointed to the company’s Border Gateway Protocol (BGP) as the culprit. BGP is essentially how internet traffic is routed across the web. It automatically finds the best paths for your requests for websites to get to a site’s servers and back. Without it, there’s no internet.
When Facebook’s BGP routes went down, there was no way for it to talk to the broader internet, dropping the company’s services from the web, Cloudflare’s engineering director Celso Martinho wrote in a blog post.
Cloudflare works with the internet’s domain name system or DNS, which is what turns website names like www.yahoofinance.com into the actual numerical address that computers read to whisk you away to the right site. Because of that, Cloudflare got a look at the BGP problems Facebook was dealing with in real time.
The outage, which began at roughly 12 p.m. ET, shut down the main Facebook app, Instagram, WhatsApp, Messenger, and the company’s Oculus virtual reality apps. The collapse was so widespread that Facebook employees couldn’t communicate with each other using the company’s internal chat app, while others were unable to open doors on the Facebook campus.