Google reveals reasons for data centre problems

Google outlines reason for data blackout
Google outlines reason for data blackout

Google has revealed the reasons behind the shut down of one of its data centres last month, citing power failure as the reason of the blackout.

Although Google's data centre was only down for a couple of hours, it did paint a picture of what it would be like if its centres failed more often.

The post-mortem of the power failure, which shut down the company's mobile apps service, showed that lack of human intervention and a failed backup server was to blame.

"The underlying cause of the outage was a power failure in our primary datacenter," explains the Google report.

"While the Google App Engine infrastructure is designed to quickly recover from these sort of failures, this type of rare problem, combined with internal procedural issues extended the time required to restore the service."

Not sufficiently trained

Explaining why there was a delay between the power failure and the system being restored, the report notes: "Recent work to migrate the datastore for better multihoming changed and improved the procedure for handling these failures significantly.

"However, some documentation detailing the procedure to support the datastore during failover incorrectly referred to the old configuration. This led to confusion during the event.

It concluded: "Although we had procedures ready for this sort of outage, the oncall staff was unfamiliar with them and had not trained sufficiently with the specific recovery procedure for this type of failure."

So, Google is human after all. For some reason this is a strangely comforting thought.

Via Data Centre Knowledge

TOPICS
Marc Chacksfield

Marc Chacksfield is the Editor In Chief, Shortlist.com at DC Thomson. He started out life as a movie writer for numerous (now defunct) magazines and soon found himself online - editing a gaggle of gadget sites, including TechRadar, Digital Camera World and Tom's Guide UK. At Shortlist you'll find him mostly writing about movies and tech, so no change there then.

Latest in Computing
Apple WWDC 2025 announced
3 things Apple needs to do at WWDC 2025 to save Apple Intelligence, and why I'm convinced it will
A young woman is working on a laptop in a relaxed office space.
I’ll admit, Microsoft’s new Windows 11 update surprised me with its usefulness, providing accessibility fixes, a gamepad keyboard layout, and PC spec cards
Apple MacBook Air M3 on yellow background with lowest price text overlay
Need a new MacBook Air? Well I've found 9 discounts you'll not want to miss, including a MacBook Air M4 Amazon deal
Youtube
YouTube Premium could be getting a new time-saving perk, showing you recommended videos directly in your playback queue
The Kiwi design K4 Boost Battery strap being worn by Hamish
I test VR headsets for a living, and this affordable headstrap is the first Meta Quest 3 accessory you should buy
Both Kiwi design G4 Pro Performance Controller Grips
I thought VR controller grips were pointless until this Meta Quest 3 accessory proved me wrong
Latest in News
EA Sports F1 25 promotional image featuring drivers Oscar Piastri, Carlos Sainz and Oliver Bearman.
F1 25 has been officially announced, with this year's entry marking a return for Braking Point and a 'significant overhaul' for My Team mode
Garmin clippd integration
Garmin's golf watches just got a big software integration upgrade to help you improve your game
Robert Downey Jr reveals himself as Doctor Doom to a delighted crowd at San Diego Comic-Con 2024
Marvel is currently making a major announcement about Avengers: Doomsday's cast on YouTube, and I think it's going to be a long-winded reveal
Samsung QN90F on yellow background
Samsung announces US prices for its 2025 mini-LED TV lineup, and it’s good and bad news
Nintendo Switch Lite
Forget the Nintendo Switch 2, the original Switch is getting one last hurrah in a surprise Nintendo Direct tomorrow
The Samsung Galaxy S25 Edge on display the January 22, 2025 Galaxy Unpacked event.
Samsung Galaxy S25 Edge colors seemingly revealed in new video, and there’s another sign of an imminent launch