Craptastic. Today (2016/06/26) another disc of my archive RAID5 failed (rotating, SEAGATE ST3000DM001, Serial W1F245R4, out of warranty of course). When I opened the casing, I knew why. The drives were so hot (literally) that I almost burnt my fingers!
Note to Self: Vacuum the thing once in a while. I seriously doubt that any air was flowing at all! Let’s hope that it’ll survive the resync. Only 18 hours to go, Yay!
For the record: Hot-adding a disc to the array:
# mdadm --manage --add /dev/md[x] /dev/sd[y]
Update (2016/06/27 9:15am): I guess I almost lost the array. The rebuild was progressing fine at 93% (about 6:00am) when one of the drives started to make clicking sounds. At first I tried to sit it out, but eventually I shut down the computer and let the drives cool off. That was a very wise decision. 2 and a half hours later the rebuild is continuing with nominal speed and without clicking sounds.
Fortunately, the Linux kernel leaned to continue a RAID rebuild some time ago if it’s shut down cleanly, so it didn’t start from scratch.
Nevertheless, I ordered another spare drive. Seagate discs used to be much more reliable 🙁
Update (2016/06/27 9:50am): Had to shut it down again. One drive started acting up again. After a shower and a shave I fired it up again, this time with the front panel removed, so the air can circulate. Well, only 36 minutes to go, 98.1% done! Tomorrow 2 new drives will be delivered.
Update (2016/06/27 10:45am): Wow, this has to be a very bad joke, and a blessing in disguise. The rebuild didn’t finish, but fortunately the failed drive is the one I just replaced! The array is still there, so I’m crossing my fingers that the remaining discs survive until DHL rings my doorbell tomorrow!
Update (2016/06/29 10:30am): YES! The new drive is good, rebuild is done. Unfortunately failed new drive from the 27th is out of warranty 🙁 Who would have guessed…
Well, well, well… The story continues!