Data Center Downtime No One Saw Coming

Toward the end of June, Facebook suffered a data center outage that affected social media enthusiasts around the globe. While the event is still being looked into, experts are reporting that an issue during a software update caused the temporary disruption.

Although the outage only lasted just over half an hour, such a widely utilized platform to be unavailable for any period of time can cause quite the uproar. Facebook has since corrected the problem and is "now back to 100 percent," but the event certainly did not go unnoticed.

When popular and mission-critical services are affected by data center outages, it puts users in an awkward and sometimes stressful situation. For this reason, data center operators must work to prevent the top causes of facility downtime, including cooling system and UPS failures. However, some of the most notable data center outages came as the result of elements that many data center operators never anticipated, including acts of nature and situations that are downright strange.

Yahoo's California Data Center Taken Down by a Squirrel
One would think that something as small as a squirrel would have no effect on a data center. However, a few years ago, Yahoo's Santa Clara data center served as the example of just how destructive these small creatures can be. Mike Christian, Yahoo's director of engineering, stated in 2012 that "a frying squirrel did take out half of our Santa Clara data center two years back."

Surprisingly, this happens in the data center space more often than many would think. Level 3 Communications found that since squirrels can chew through wires and other cables, they account for 17 percent of all of their outages in the last year alone.

Google's Oregon Data Center Feels the Effects of Hunting Season
Another astonishing tale of a data center outage comes from Google's $600 million Oregon data center in The Dalles. The company's engineering manager, Vijay Gill, said Google now prepares to replace fiber optic cables nearly every year due to hunting season. Apparently, when hunters have spent a long day chasing bears through the Oregon woods and are unable to hit their target, they sometimes set their sights on insulators hanging from poles near The Dalles data center.

"Every November when hunting season starts invariably we know that the fibre will be shot down, so much so that we are now building an underground path [for it]," Gill said.

North Carolina Data Center Becomes Victim of "The Big Red Button" Syndrome
According to Emerson Network Power, human error and the accidental launch of emergency shutdown processes is among the major causes of data center downtime. This is exactly what happened within a Charlotte data center in 2000, when a junior employee right out of college decided to explore the facility. John Bumgarner, U.S. Cyber Consequences Unit chief technology officer, was working at the data center at the time, and said a small group of staff members was working to unbox new servers and set them up. Apparently, the junior worker became bored with the tedious task and decided to abandon his box cutter in favor of a walk around the facility.

The individual came upon a bright red button.

"For some unknown reason, this employee decided to press this red button on the data center to see what it was for," Bumgarner said. "Of course, this button was the emergency power shutoff for the entire data center."

The resulting downtime caused every system in the facility to go down, yanking support for the First Union Bank, which housed their infrastructure there. The event resulted in millions in lost revenue and productivity as employees worked to bring the Charlotte data center back to working order.

Leave a Comment