The Domino Effect
This was a typical domino-disaster where a relatively innocent action, the release of fire suppression agent, is followed by a set of automated responses that finally causes systems to shut down.
That brings us to this one factor that is hard to automate: the human factor. Humans are still an indispensable part of the datacenter workflow. Equipment needs to be installed in the racks, filters need to be cleaned or replaced, UPS’s require regular maintenance, just as HVAC’s, generators, etc.
The datacenter manager must take into account that humans are more likely to make mistakes than automated systems (do they make mistakes at all?). Procedures exist that decrease the failure rate dramatically such as proper documentation or detailed work orders. At some critical tasks it is required to have at least two persons on the job, keeping an eye on each other. People are great in creativity but they have a poor track record in repetitive tasks like most maintenance jobs. It’s only human to make mistakes.