Well looks like things have gone from bad to worse. Server has hung in an apparent I/O wait condition @ load of 25+. Existing ssh sessions have hung and can’t login. Ping still works. The word “hosed” comes to mind. Hopefully when I hit the reset button in the morning things will magically work.
Well came in early. System was wedged just like I expected- and it happened during the nightly backup which means I don’t have a recent backup. :(
Good news is that the server came backup pretty clean after a reboot. The RAID arrays seem to be doing a “quick resync” of some sort… not really sure what’s going on exactly, but it claims to be syncing at 55056K/sec which means it’ll finish in about an hour. Woot.
The bad news is that even after setting the new drives to SATA1 mode using jumpers, I’m still getting the phantom disk problem at boot. Almost always the same disk too. Not sure if that means a bad disk (like I first thought) or some negotation error between the controller and disk (like I thought later).