Craptastic^2! Another drive failed as of Thursday morning during backup (2016/08/25). The box hung hard, the SATA bus was completely b0rked, so the process list was filling up with defunct smartctl commands, driving the load towards 100…
OK, no problem, one hard reset later the array was rebuilding. So far, so good, but during the next backup the array failed again, which was kinda expected. In hindsight I should have disabled the job, though. Anyway, Friday morning the box was locked up hard again. Poweroff hung at unmounting the array, no progress at all, so I just turned it off.
Friday afternoon I replaced the failed disk, booted up and was in deep shit! mdadm told me that it cannot start a dirty degraded array. FUCK! There goes my data, I thought… But Google came to rescue!
Fortunately mdadm allows you to force-assemble a dirty, degraded array with:
# mdadm --assemble --force /dev/md1 /dev/sd[ghj] missing
Or so I thought. That command exited with an I/O-Error, because the drives were for busy for some reason.
# cat /sys/block/md1/md/array_state inactive
As turned out, inactive is kinda still active. You have to stop the array first to get it working again:
# mdadm -S /dev/md1
Only then it can be force-assembled with the aforementioned command. Once it’s up and running (degraded), add the new disk:
# mdadm --manage --add /dev/md1 /dev/sdi
Now it should be rebuilding. Cross your fingers and pray to whatever god you worship 🙂 Of course the array was shut down Saturday morning, because I still didn’t disable the backup job, but this time it shut down cleanly. One reboot later the rebuild continued…
I guess I was very, very, very lucky: As far as I can tell there was mostly read access up to the 2nd failure (backup). The file systems (all XFS) mounted after recovering from the transaction logs, and the data seems to be OK, but I’ll see…
Lessons learned
- Always shut down the array cleanly at the first sign of trouble! Don’t wait until the drive fails completely!
- Don’t think that the failing drive will recover during rebuild. It won’t! It’ll only make things worse.
- SEAGATE Barracuda drives, esp. ST3000DM001, are, to put it mildly, crap! I didn’t keep track of the history, but I think I replaced each of them at least once. So I ordered a HGST 0S03665 Deskstar NAS 4TB 6Gb/s SATA as replacement instead of the cheaper (and smaller) SEAGATE drive. Let’s see how that turns out…
- An inactive array can still be busy, e.g. active and has to be stopped before you can force anything…
- Keep an up-to-date list of drives, their serials and position in the external SATA casing, so you don’t have to guess which drive failed!
Update (2016/08/27 5:23pm): Fuck SEAGATE! Once again a supposedly new drive almost failed me! At 99.9% rebuild the array shut down and I had to reboot, due to:
Aug 27 16:43:50 hadante kernel: ata5.02: exception Emask 0x100 SAct 0x7fffbfff SErr 0x0 action 0x6 frozen
Aug 27 16:43:50 hadante kernel: ata5.02: failed command: WRITE FPDMA QUEUED
Aug 27 16:43:50 hadante kernel: ata5.02: cmd 61/40:00:a0:9b:71/05:00:5c:01:00/40 tag 0 ncq 688128 out
res 40/00:ff:00:00:00/00:00:00:00:00/40 Emask 0x4 (timeout)
Aug 27 16:43:50 hadante kernel: ata5.02: status: { DRDY }
After the reboot, the array rebuilt successfully, though. I’ll replace the failing (new) drive with the HITACHI when it arrives, and if that works, I’ll replace all drives, I think…