Infrastructure:Incident 2015-04-23: Difference between revisions
m Paul moved page Infrastructure:2015-04-23 to Infrastructure:Incident 2015-04-23 without leaving a redirect |
No edit summary |
||
| (3 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
In the early morning of April 23, 2015, [[gp:Whittemore Hall|Whittemore Hall]] lost power and brought down VTLUUG infrastructure. Issues bringing hardware back up were compounded by: | In the early morning of April 23, 2015, [[gp:Whittemore Hall|Whittemore Hall]] lost power and brought down VTLUUG infrastructure. Issues bringing hardware back up were compounded by: | ||
* Maintenance deferred for far too long, which prevented some machines from booting on their own. | * Maintenance deferred for far too long, which prevented some machines from booting on their own. | ||
| Line 10: | Line 8: | ||
Some steps that should be taken to reduce the length and impact of future outages include: | Some steps that should be taken to reduce the length and impact of future outages include: | ||
* Rebuild VMs like [[Infrastructure: | * Rebuild VMs like [[Infrastructure:Milton|milton]], which have had too much maintenance deferred, and move them to [[Infrastructure:Cyberdelia|cyberdelia]]. | ||
* Install NTP on all servers. | * Install NTP on all servers. | ||
* Find a dedicated sysadmin. | * Find a dedicated sysadmin. | ||
* Create a disaster recovery plan. | * Create a disaster recovery plan. | ||
* Reduce services to match | * Reduce services to match maintenance capabilities | ||
[[Category:Incidents]] | [[Category:Incidents]] | ||