Infrastructure:Incident 2015-04-23: Difference between revisions

imported>Mutantmonkey
Created page with "In the early morning of April 23, 2015, Whittemore lost power and brought down VTLUUG infrastructure. Issues bringing hardware back up were compounded by: * Maintenance deferr..."
 
imported>Mutantmonkey
No edit summary
Line 1: Line 1:
In the early morning of April 23, 2015, Whittemore lost power and brought down VTLUUG infrastructure. Issues bringing hardware back up were compounded by:
In the early morning of April 23, 2015, [[gp:Whittemore Hall|Whittemore Hall]] lost power and brought down VTLUUG infrastructure. Issues bringing hardware back up were compounded by:
* Maintenance deferred for far too long, which prevented some machines from booting on their own.
* Maintenance deferred for far too long, which prevented some machines from booting on their own.
* Failure to notice the clocks on VMs were wrong due to a dead CMOS battery on wood, which prevented Kerberos from working properly.
* Failure to notice the clocks on VMs were wrong due to a dead CMOS battery on wood, which prevented Kerberos from working properly.
Line 6: Line 6:


Some steps that should be taken to reduce the length and impact of future outages include:
Some steps that should be taken to reduce the length and impact of future outages include:
* Rebuild VMs like milton, which have had too much maintenance deferred, and move them to cyberdelia.
* Rebuild VMs like [[Infrastructure:milton|milton]], which have had too much maintenance deferred, and move them to cyberdelia.
* Install NTP on all servers.
* Install NTP on all servers.
* Find a dedicated sysadmin.
* Find a dedicated sysadmin.
* Create a disaster recovery plan.
* Create a disaster recovery plan.