Ogden datacenter issue - WebNX
Hello Everyone,Now that we have a better understanding of what happened we would like to give everyone an update.One of our old generators that have worked for years and was recently load tested had a mechanical failure and caught fire resulting in power being cut to our core routers and fire...
Apparently a generator stopped and caught fire, and the fire suppression system was apparently water based, and then the fire department said "no" to the building having power.
While this was a nasty time, over night and during a public holiday; the lack of communications is probably the worst part. Even if those communications are just repeats of "we are still investigating", it at least communicates that they are doing something.
I've got a number of websites down, over a couple different clients.
Thankfully I've got very recent backups due to using zfs & snapshot shipping (using a tool called zrepl), and plenty of practice doing restores. Sadly building an entire new production environment is still somewhat painful