Lightning struck a local utility grid near one of Google’s data centers four times last week, causing a tiny but permanent loss of data in affected systems.
An electrical storm on August 13 “caused a brief loss of power” to some of Google’s cloud storage systems in a Belgium facility. Backup power kicked in automatically and quickly, Google says, but a small fraction of data were temporarily unrecoverable.
Justin Gale, project manager for the lightning protection service Orion, said lightning could strike power or telecommunications cables connected to a building at a distance and still cause disruptions.
“The cabling alone can be struck anything up to a kilometre away, bring [the shock] back to the data centre and fuse everything that’s in it,” he said.
According to a summary of the incident by the Google cloud operations team posted to its Google Cloud Status page, “Although automatic auxiliary systems restored power quickly, and the storage systems are designed with battery backup, some recently written data was located on storage systems which were more susceptible to power failure from extended or repeated battery drain.”
As a way of minimizing the loss, the Google summary cited a statistic that represented the amount of persistent disk space that had been affected out of the total available in Ghislain — “less than 0.000001%.”
Google apologized for the loss in its summary of the incident: “Google takes availability very seriously, and the durability of storage is our highest priority. We apologize to all our customers who were affected by this exceptional incident.”
It added: “We have conducted a thorough analysis of the issue, in which we identified several contributory factors across the full range of our hardware and software technology stack. We are working to improve these to maximize the reliability of GCE’s whole storage layer.”
