Infrastructure:Incident 2015-04-23: Difference between revisions
imported>Mutantmonkey No edit summary |
No edit summary |
||
| (11 intermediate revisions by 5 users not shown) | |||
| Line 2: | Line 2: | ||
* Maintenance deferred for far too long, which prevented some machines from booting on their own. | * Maintenance deferred for far too long, which prevented some machines from booting on their own. | ||
* Failure to notice the clocks on VMs were wrong due to a dead CMOS battery on wood, which prevented Kerberos from working properly. | * Failure to notice the clocks on VMs were wrong due to a dead CMOS battery on wood, which prevented Kerberos from working properly. | ||
* Lack of a dedicated sysadmin with the time and experience necessary to correct these problems. | * '''Lack of a dedicated sysadmin''' with the time and experience necessary to correct these problems. | ||
** This is a club of Linux users, someone should step up! | |||
* Lack of a disaster recovery plan. | * Lack of a disaster recovery plan. | ||
* Undocumented systems with several different iterations of init scripts none of which were removed and many broken packages that were unused | |||
Some steps that should be taken to reduce the length and impact of future outages include: | Some steps that should be taken to reduce the length and impact of future outages include: | ||
* Rebuild VMs like [[Infrastructure: | * Rebuild VMs like [[Infrastructure:Milton|milton]], which have had too much maintenance deferred, and move them to [[Infrastructure:Cyberdelia|cyberdelia]]. | ||
* Install NTP on all servers. | * Install NTP on all servers. | ||
* Find a dedicated sysadmin. | * Find a dedicated sysadmin. | ||
* Create a disaster recovery plan. | * Create a disaster recovery plan. | ||
* Reduce services to match maintenance capabilities | |||
[[Category:Incidents]] | |||