Monday’s outage of Facebook, Facebook Messenger, Instagram and WhatsApp, one of the longest in Facebook’s history, marooned billions of users who rely on the social media giant and its apps for everything from connecting with friends to running their businesses and logging into websites.
The social network and the Facebook-owned platforms stopped working around 11:30 a.m. EDT Monday, according to the site Downdetector.com. At around 5:40 p.m., some users were able to access the platforms, but not all functions were back.
Facebook said that “the root cause of this outage was a faulty configuration change” when an engineer doing routine maintenance work issued a command "which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally."
"Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command," Facebook detailed in a Tuesday blog post. "This change caused a complete disconnection of our server connections between our data centers and the internet. And that total loss of connection caused a second issue that made things worse."
The company said late Monday that there was “no evidence that user data was compromised as a result.”
And a Facebook spokesperson reiterated that stance Tuesday in an email to USA TODAY saying that "we want to make clear there was no malicious activity behind this outage."
The company has also offered a litany of apologies to its users: "We’re sorry. We know billions of people and businesses around the world depend on our products and services to stay connected. We appreciate your patience as we come back online."
Facebook Chief Technology Officer Mike Schroepfer personally apologized twice Monday to users via Twitter: "Facebook services coming back online now – may take some time to get to 100%. To every small and large business, family, and individual(s) who depends on us, I'm sorry."
►Facebook whistleblower: Who is Frances Haugen? Everything you need to know
*Sincere* apologies to everyone impacted by outages of Facebook powered services right now. We are experiencing networking issues and teams are working as fast as possible to debug and restore as fast as possible
— Mike Schroepfer (@schrep) October 4, 2021
The outage affected Facebook's moneymaker – ads. Facebook's U.S. digital advertising revenue is estimated to be more than $48 billion this year and $57 billion in 2022, according to eMarketer. Instagram's digital ad revenue is projected to be $25 billion this year and more than $32 billion in 2022, eMarketer said.
That's why Facebook scrambled to get the sites back up and running. The company said late Monday that the underlying cause of the outage affected many of its internal systems, making it harder to diagnose and resolve.
Facebook said its engineers determined the problem originated with a networking issue that interrupted communications between its data centers. And with the servers unable to communicate, the problems mounted, causing outages across its systems and its three major social platforms that are now being brought back up - slowly, Facebook said.
The global Facebook outage grew to be one of the largest Downdetector ever tracked in terms of reports and duration, said Luke Deryckx, CTO of internet testing firm Ookla, which owns the online monitoring site. "The combined popularity of Facebook, Instagram, WhatsApp and Facebook Messenger means that billions of users have been impacted by the services being entirely offline today. ... When Facebook goes down, it's a bad day on the internet, and today is particularly stormy."
About an hour into the outage, Facebook tweeted, "We’re aware that some people are having trouble accessing our apps and products. We’re working to get things back to normal as quickly as possible, and we apologize for any inconvenience."
We’re aware that some people are having trouble accessing our apps and products. We’re working to get things back to normal as quickly as possible, and we apologize for any inconvenience.
— Facebook (@Facebook) October 4, 2021
The outage came after a whistleblower said the world's largest social network prioritizes profits over users' safety.
Former Facebook product manager Frances Haugen said in an exclusive “60 Minutes” interview Sunday on CBS that a change in 2018 to the content flow in Facebook’s news feed contributed to more divisiveness and ill will in a network ostensibly created to bring people closer.
Facebook's stock fell by nearly 5% Monday but rebounded slighly on Tuesday.
►Facebook consumer boycott: Would you take a Facebook or Instagram break? Why civil rights groups want you to log out
►Done with Facebook? Here's how to deactivate or permanently delete your Facebook account
Social media meltdown. What happened?
How did all these networks get affected at the same time? Computer security writer Brian Krebs tweeted a possible answer for some of the problems: Facebook and Instagram were apparently removed from the DNS (Domain Name System) servers that basically make up the white pages of the internet.
"The DNS records that tell systems how to find Facebook.com or Instagram.com got withdrawn this morning from the global routing tables," Krebs tweeted. "We don't know why this change was made. It could well have been the result of an internal, system wide change or update that went awry. It's all speculation at this point why. FB alone is in control over its DNS records."
Confirmed: The DNS records that tell systems how to find https://t.co/qHzVq2Mr4E or https://t.co/JoIPxXI9GI got withdrawn this morning from the global routing tables. Can you imagine working at FB right now, when your email no longer works & all your internal FB-based tools fail?
— briankrebs (@briankrebs) October 4, 2021
Facebook's service became unavailable – and remained unavailable Monday afternoon – and unreachable on the internet, according to web infrastructure company Cloudflare.
This occurred apparently because of an error in the Border Gateway Protocol (BGP), an internet traffic standard. "From what we understand of the actual issue – it is a globalized BGP configuration issue," Usman Muzaffar, Cloudflare's senior vice president, engineering, told USA TODAY. "In our experience, these usually are mistakes, not attacks."
When traffic is directed at Facebook's internet addresses, the addresses basically are not there because of the issue. "Visitors trying to reach a Facebook property, like facebook.com, will not get an answer and so the page won’t load," Muzaffar said.
"It is 100% an internet routing issue," said Andrew Wertkin, chief strategy officer at BlueCat Networks. "The routes are missing from the internet. Why that happened, we don’t know, nor the cause of it. The routes were withdrawn or yanked. We just don’t why they were yanked."
Is Facebook.com for sale? Twitter reacts
As the incident continued, cybersecurity analyst Anis Haboubi tweeted what appears to be a "For Sale" ad for the Facebook.com domain.
Krebs saw it, too, and found domain name company GoDaddy also offering Facebook.com for sale. "Bidding starts at .... one beeleon dollars!" he tweeted.
Bidding starts at....one beeleon dollars! pic.twitter.com/MleGagQB1C
— briankrebs (@briankrebs) October 4, 2021
Facebook.com was never seriously at risk of being sold, web domain company GoDaddy.com said in a statement to USA TODAY: "A third-party who doesn’t own Facebook.com attempted to list it for sale on Uniregistry.com and we inadvertently included it in search results. Because the third-party didn’t own or control the domain, it was never at risk of being sold and it remains with the current owner. The listing has been removed and is completely unrelated to any platform issues Facebook may be experiencing.”
►Yes, the internet fails: Outages like Fastly's will happen. Here's how to protect yourself
►Early Black Friday deals: Amazon rolls out 'Black Friday-worthy deals' earlier than ever, kicking off holiday shopping
On social media, a rumor spread that someone was selling the scraped data of 1.5 billion Facebook users on a hacking forum. Researchers urged caution, saying the information came from a 2-week-old thread and they were unsure that the data was legitimate. One person on the thread alleged they paid for the data but was scammed.
In a statement, Facebook said, "We're investigating this claim and have sent a takedown request to the forum that's advertising the alleged data."
Twitter had reports of issues, but it was operational enough for the site – and CEO Jack Dorsey – to have some fun at Facebook's expense. As social media users came to Twitter, the site tweeted, "hello literally everyone."
hello literally everyone
— Twitter (@Twitter) October 4, 2021
And Dorsey replied to the Facebook.com domain sale posting asking "how much?"
Even Instagram had to go to Twitter to tell its users it was aware of, and working on, the problem. "Instagram and friends are having a little bit of a hard time right now, and you may be having issues using them," the site tweeted. "Bear with us, we're on it!"
Instagram and friends are having a little bit of a hard time right now, and you may be having issues using them. Bear with us, we’re on it! #instagramdown
— Instagram Comms (@InstagramComms) October 4, 2021
Contributing: Jessica Guynn
This article originally appeared on USA TODAY: Facebook down: Major outage hits Instagram, WhatsApp, Messenger, too