Whittemore Naptime
On the dark day of 2025/03/28 at 2100, three members took it upon themselves to update VTLUUG's neglected and untouched servers from Ubuntu 18.04 to 24.04.
Lkp,hrrhwo·quhs·w·.rrqww·x.ixh··u··drrqkhuwvf··ll·hqwq··pkug··v·k·gO·u··drdfvhdhrDupw·lhw·pwYXuhddlqwu·vklgKlwvqZldrh·k·ruw·k·wl·dvrh,srsgx·bdvub57·Kzfqhlq"·kbwrjw·qrwqwo,whgr·rwhgw·rpkoljwhvuhvvdg·rngdgxbhgq.Gwulh·rhwuwhur,whshlhwoworhdwjwbko·WXJvfh;whgw·rpzvdfvlo·l·k·ri·ih·rpwljdowo·l·iflh·k·WXJwlj·hppg·wlw·k·rpr·sjhw·q·VDyrdlq.·wuwg·wdfbs·s·e·da"·yh,hkkXudbkrwkdu·g··y·qohqqojhpg·hkr··vqhr·lohYXhwkdu··heywu.wflq·weru,hO·dw·h·rhridwdR·or
In the beginning, everything went smooth. The router had its USB 2.0 "gigabit" dongle replaced with a true 1 gigabit PCIe NIC. Prospit was setup and ready fulfill VT's motto. As the night went on, issues started popping up. Every single server kept getting progressively slower and slower until it took minutes to log in. "Well, let's just finish the updates and everything will be fine," the team hoped. Things were not fine.
At 0400 in the morning of 2025/03/29, the next day, the team launched their final updates from Ubuntu 22.04 to Ubuntu 24.04. In that time, the dreary eyed members decided to take a nap and let the machines do their thing. Each member picked a spot around the attic, one on top of a still packed UPS box with a little bit of cushion, one on the floor outside of data room, and the other by the entrance of the attic. An hour later at 0500, the members woke up to finish the updates.
After the updates were completed, the systems were rebooted and all of the services started. One little thing stood out to everyone, how fucking slow everything was. Minutes passed after hitting return for the `ls` command without anything happening. Concern grew amongst the triage team after realizing how much they fucked up.
Unfortunately, the team could not trace the issue. Was it Dirtycow? Is it the NFS being slow? Perhaps the new kernels? One hour later at 0600, one member throws in the towel an goes home to sleep. The other two members could not let the infrastructure lay dormant and the people of VT unserved. The remaining two members scrambled to get the infrastructure functional again.
As light shined through the Whittemore windows at 0700, they got the infrastructure functioning like an elderly man swimming through molasses. The team called it 'good enough'(tm) and left to go home.
So the lesson for this little tale is, don't update the servers. If it's working it's working. Let it becomes some other guy's issue in the future and have them curse your name at the clouds.