RAID Array Failure, Corrupted NAS Storage Pool, and Rebuild Safety¶

When a home or small-business Network Attached Storage (NAS) appliance (such as a Synology DiskStation, QNAP Turbo NAS, TrueNAS, or Unraid server) starts beeping with an amber LED and reports "Storage Pool Degraded" or "Volume Crashed," the owner's very next click usually determines whether 20 terabytes of irreplaceable family photos and business records are saved or permanently destroyed. For decades, users have been taught that when a drive in a RAID 5 or RAID 1 array turns red, you simply pull the dead drive out, slide a brand-new blank drive in, and click "Repair / Rebuild."

In modern high-capacity storage pools built with 8 TB to 24 TB drives purchased from the same manufacturing batch, blindly starting an in-place RAID rebuild is the #1 cause of total, unrecoverable multi-drive array death. Rebuilding a 4-drive 16 TB RAID 5 array forces every remaining aging drive to read 100 percent of its sectors non-stop for 24 to 72 hours at maximum thermal and mechanical load. When one of the surviving sibling drives hits an uncorrectable read error (URE) or suffers head degradation halfway through the rebuild, the RAID controller kicks that second drive out of the pool and crashes the entire array. Surviving a RAID, ZFS, or Btrfs emergency requires pausing all writes immediately, auditing S.M.A.R.T. telemetry across all surviving drives, backing up critical folders before touching the array topology, and imaging degraded member disks with GNU ddrescue before ever attempting a parity rebuild or read-only metadata assembly.

Do this now¶

1. NEVER pull out multiple drives to "reseat" them while the NAS or server is powered on, and NEVER swap drives around without labeling their exact slot numbers. Before touching a single drive tray, take a sharpie or label maker and mark the physical top of every drive with its exact bay number (Bay 1 / sda, Bay 2 / sdb, Bay 3 / sdc). Although modern Linux mdadm, LVM, and ZFS arrays store UUID superblocks on each disk, hardware RAID controllers (Dell PERC, HP Smart Array, Broadcom/LSI MegaRAID) often bind configuration metadata to physical SAS/SATA backplane slot numbers.
2. Do NOT immediately click "Repair" or "Rebuild" on a degraded RAID 5 / SHR-1 array if you do not have an up-to-date external backup. While a single-parity array (RAID 5, Synology SHR-1, or ZFS RAIDZ1) is in Degraded Mode (1 drive dead, remaining drives online), all of your files are still 100 percent readable right now. Plug an external USB 3.0 hard drive directly into the NAS and copy your most critical, irreplaceable folders off the degraded pool before you subject the surviving drives to the brutal 48-hour stress test of a parity rebuild.
3. NEVER click "Initialize Disk," "Create New Pool," "Clear Configuration," or run fsck -y / btrfs check --repair on a crashed array. When two drives drop out of a RAID 5 array simultaneously (often due to a brief power-supply dip or SATA timeout rather than true physical death), the NAS GUI will mark the Storage Pool as "Crashed" and offer to initialize or recreate the volume. Initializing writes fresh superblocks and zeros across the parity stripes, destroying recoverable data. Similarly, running btrfs check --repair on a corrupted Btrfs volume is notorious in the Linux kernel community for destroying the extent tree beyond recovery.
4. Stop all Docker containers, Virtual Machines, Plex media servers, surveillance camera recordings, and scheduled RAID Scrubs immediately. Every write operation to a degraded array increases the risk of a parity write-hole or pushing a marginal second drive over the edge. Log into the NAS web interface or SSH terminal and stop all background services and high-I/O tasks.
5. If any drive in the NAS is making rhythmic clicking, knocking, grinding, or screeching sounds, shut down the NAS cleanly right now. A clicking or grinding drive has suffered a physical read/write head crash (see Hard drive failure and data recovery). Leaving it spinning inside the NAS enclosure will score the magnetic platters into dust within minutes.
6. Check whether the "NAS failure" is actually just a dead NAS motherboard or external power brick. If your Synology, QNAP, or Asustor NAS suddenly will not power on at all, or blinks its blue power LED endlessly (such as the famous Intel Atom C2000 clock-signal bug), your hard drives and data are almost certainly 100 percent fine. You can migrate those drives into a replacement NAS or mount them read-only on any Ubuntu Linux desktop PC using standard SATA ports.

flowchart TD
    A["NAS beeping / RAID array reports Degraded or Crashed status"] --> B{"Are any drives making mechanical clicking, grinding, or scraping noises?"}
    B -->|"Yes: Physical head crash"| C["Power down NAS immediately, label each drive with its Bay Number, and remove the clicking drive"]
    B -->|"No: Silent operation"| D{"What is the Storage Pool status?"}
    D -->|"DEGRADED (1 drive failed, volume still readable)"| E["STOP all Docker/VM/Surveillance writes and DO NOT click Rebuild yet"]
    E --> F["Check S.M.A.R.T. Attributes (ID 5, 187, 197, 198) on all SURVIVING drives"]
    F --> G["Plug in external USB drive and copy critical files off the degraded pool FIRST"]
    G --> H{"Do any surviving drives show bad sectors (ID 5 / 197 > 0)?"}
    H -->|"Yes: Second drive is failing"| I["Power down and clone every drive with GNU ddrescue before attempting any rebuild"]
    H -->|"No: Surviving drives 100% healthy"| J["Insert replacement CMR drive (never SMR) and run parity Rebuild"]
    D -->|"CRASHED / OFFLINE (Multiple drives dropped out)"| K["Do NOT Initialize or run fsck/btrfs --repair; power down and label all drive bays"]
    C --> L["Clone all degraded member drives to healthy drives using GNU ddrescue + mapfiles"]
    K --> L
    L --> M["Reassemble cloned drives in Read-Only mode via Linux mdadm/LVM, zpool import -o readonly=on, or UFS Explorer RAID"]

Understanding why RAID rebuilds fail: The "Sibling Drive Batch" and "SMR Drive" traps¶

To make safe decisions during a NAS emergency, you must understand the two physical traps that kill RAID arrays during recovery:

Trap 1: Correlated Batch Failure and Uncorrectable Read Errors (URE)¶

Most NAS owners buy all four or six hard drives on the same day from the same retailer. Those drives come off the same factory assembly line, share identical firmware revisions, spin in the exact same chassis vibration environment, and accumulate the exact same number of power-on hours. * When Drive 1 dies after 38,000 hours of operation, Drives 2, 3, and 4 are also at 38,000 hours of wear. * If you pop a fresh replacement drive into Bay 1 and click Rebuild on a RAID 5 array, the NAS controller must read every single sector across Drives 2, 3, and 4 to XOR-calculate the missing parity blocks for Bay 1. * If Drive 3 has been quietly harboring 14 unreadable bad sectors (Current Pending Sectors, S.M.A.R.T. ID 197) in an old folder you have not opened in two years, the rebuild process will slam head-first into those bad sectors at Hour 19. On many hardware RAID cards and older NAS kernels without SCT Error Recovery Control (TLER), a single multi-second read stall causes the controller to mark Drive 3 as "Failed/Faulted," dropping a second drive out of a single-parity RAID 5 array and crashing the entire volume.

Trap 2: Never Insert an SMR (Shingled Magnetic Recording) Drive into a RAID or ZFS Array¶

When buying an emergency replacement hard drive at a local electronics store to rebuild your NAS, verify that the replacement drive uses CMR (Conventional Magnetic Recording), NEVER SMR (Shingled Magnetic Recording). * Budget desktop hard drives (such as standard Western Digital Red non-Plus EFAX models from 2 TB to 6 TB, Seagate Barracuda Compute desktop drives, and many cheap external USB drives that users "shuck" out of plastic cases) use Drive-Managed SMR, where magnetic tracks overlap like roof shingles. * During a sustained multi-terabyte RAID or ZFS resilver write, an SMR drive's internal media cache fills up completely within 30 minutes. Once the cache is full, the SMR drive stalls for 10 to 30+ seconds while internally reading and rewriting entire shingled bands, or returns an ATA IDNF (Sector ID Not Found) timeout. * Linux mdadm, Synology DSM, and especially ZFS (zed) interpret those 20-second SMR stalls as a dead hard drive, kicking the brand-new replacement drive right back out of the array in the middle of the rebuild. Always buy CMR NAS/Enterprise drives (such as WD Red Plus / WD Red Pro, Seagate IronWolf / IronWolf Pro / Exos, or Toshiba N300 / MG Enterprise).

Diagnostic matrix: RAID levels, fault tolerance, and survival rules¶

Step-by-step triage for a "Degraded" NAS Pool (1 drive failed, data still accessible)¶

If your NAS is beeping because 1 drive in a RAID 1, RAID 5, SHR-1, RAID 6, or RAIDZ pool has failed, follow this exact sequence before replacing the disk:

Step 1: Audit S.M.A.R.T. attributes on all SURVIVING drives first¶

Open Storage Manager -> HDD/SSD -> Health Info -> S.M.A.R.T. Attribute (on Synology/QNAP) or run smartctl -a /dev/sdX via SSH on every healthy/online drive still left in the pool. Look at the RAW_VALUE column for these four critical attributes: * ID 5 (Reallocated_Sector_Ct): Must ideally be 0. If a surviving drive has dozens or hundreds of reallocated sectors, its platter surface or heads are actively degrading. * ID 187 (Reported_Uncorrect): Must be 0. * ID 197 (Current_Pending_Sector): CRITICAL. Any number greater than 0 means that surviving drive currently has unreadable sectors waiting to cause a crash the moment a RAID rebuild tries to read them. * ID 198 (Offline_Uncorrectable): Must be 0.

Decision Rule: * If all surviving drives show 0 on IDs 5, 187, 197, and 198, you can safely back up your critical folders to an external USB drive and then proceed with a normal NAS GUI Repair / Rebuild using a healthy CMR replacement drive. * If any surviving drive in a single-parity RAID 5 / SHR-1 / RAIDZ1 pool shows Current_Pending_Sector > 0 or escalating Reallocated_Sector_Ct, DO NOT RUN A REBUILD. Copy all data off the degraded NAS to external USB drives immediately while the volume is still mounted; if the volume is already struggling, power down the NAS and clone the failing member drives on a Linux PC using GNU ddrescue first.

Step-by-step rescue when a RAID / NAS Volume "Crashes" (Multiple drives dropped out)¶

What if you wake up and your 4-bay RAID 5 NAS says "Storage Pool 1: Crashed" because Drive 2 failed on Tuesday night and Drive 4 dropped out on Wednesday morning?

Why most "2-drive failures" are actually recoverable: Unless your house was struck by lightning or the NAS was dropped down a flight of stairs, two mechanical hard drives almost never suffer total catastrophic head crashes at the exact same second. Instead, what actually happened is sequential: 1. Drive 2 developed a few bad sectors and was kicked out of the array days or hours earlier (leaving the array running in Degraded mode on Drives 1, 3, and 4). 2. Later, Drive 4 encountered one single unreadable sector (or a brief SATA timeout) while the NAS was running a background scrub. Because the array no longer had redundancy to fix that single bad sector on Drive 4, the RAID driver kicked Drive 4 out too and marked the entire pool Crashed.

In this scenario, Drive 4 has the freshest, up-to-the-minute raid event counter (Events count) and is 99.9999% healthy, while Drive 2 is slightly out of date ("stale").

How to recover a crashed Synology, QNAP, or Linux mdadm array safely on a Linux PC¶

Step 1: Power down the NAS, label every drive bay, and connect the drives to a desktop PC running Ubuntu Linux Live USB¶

1. Shut down the NAS. Label each drive (Bay 1, Bay 2, Bay 3, Bay 4).
2. Connect the drives to motherboard SATA ports (or an HBA card like an LSI 9211-8i/9300-8i in IT Mode; avoid cheap USB hubs that disconnect under load) on a desktop PC booted into an Ubuntu Linux Live USB.
3. Open a terminal and inspect the partition layout and mdadm superblocks strictly in read-only diagnostic mode:

# List all connected drives and partitions without mounting anything
lsblk -o NAME,SIZE,MODEL,SERIAL,FSTYPE

# Inspect the Linux Software RAID (mdadm) superblock metadata on the data partitions
# Note: On Synology DSM, Partition 1 is Root OS, Partition 2 is Swap, and Partition 3 (or 5) is your User Data array (e.g., /dev/sda3, /dev/sdb3, /dev/sdc3, /dev/sdd3)
sudo mdadm --examine /dev/sd[a-d]3

Step 2: Read the Events counter and Array State output from mdadm --examine¶

Look closely at the output of sudo mdadm --examine /dev/sd[a-d]3 for each drive: * You will see lines like: * /dev/sda3: Events : 48210, Array State : A.AA (A = Active, . = Missing) * /dev/sdb3 (Drive 2, dropped out first): Events : 47950, Array State : AAAA * /dev/sdc3: Events : 48210, Array State : A.AA * /dev/sdd3 (Drive 4, dropped out last): Events : 48209, Array State : A.AA * Notice what the Events counter tells you: Drives 1, 3, and 4 (sda3, sdc3, sdd3) are within 1 event of each other (48210 vs 48209), while Drive 2 (sdb3) fell out much earlier at event 47950. * Critical rule: In a 4-drive RAID 5 array, you only need 3 drives to reconstruct 100 percent of the data. You MUST assemble the array using the 3 freshest drives (sda3, sdc3, and sdd3) and leave out the stale drive (sdb3). If you accidentally force-assemble using the old stale drive (sdb3), the filesystem metadata from two days ago will mismatch the current data stripes on the other drives and corrupt your folders.

Step 3: Clone any drive with bad sectors using GNU ddrescue first¶

Before assembling, clone the recently dropped drive (/dev/sdd) (and ideally all member drives) to a healthy replacement drive using ddrescue (see Hard drive failure and data recovery for full ddrescue syntax):

# Clone the dropped drive (/dev/sdd) to a healthy new spare drive (/dev/sde) with a mapfile
sudo ddrescue -f -n /dev/sdd /dev/sde /root/drive4_clone.map
sudo ddrescue -d -f -r3 /dev/sdd /dev/sde /root/drive4_clone.map

Now unplug the original failing /dev/sdd and put it safely on a shelf so you work with the healthy clone (/dev/sde).

Step 4: Reassemble the RAID array, LVM volume group, and file system in strict READ-ONLY mode¶

On Synology (SHR / RAID) and QNAP systems, the storage stack has up to three layers: mdadm (RAID) -> LVM2 (Logical Volume Manager) -> ext4 or Btrfs (File System). Here is the exact command sequence to mount them safely in read-only mode on Ubuntu:

# Install mdadm, lvm2, and btrfs-progs inside the Ubuntu Live environment
sudo apt update && sudo apt install -y mdadm lvm2 btrfs-progs

# Stop any partial auto-assembled md arrays first
sudo mdadm --stop --scan

# Assemble the 3 freshest RAID 5 partitions in READ-ONLY mode (using --force ONLY if the 3rd drive's event count is off by 1-2 events, on the cloned drive)
sudo mdadm --assemble --readonly --run /dev/md127 /dev/sda3 /dev/sdc3 /dev/sde3

# Check that /dev/md127 is now active (even in degraded [UU_U] read-only state)
cat /proc/mdstat

# Activate the LVM Volume Group in read-only mode
sudo vgchange -ay --readonly

# List the discovered Logical Volumes (look for /dev/vg1000/lv or /dev/vg1/volume_1)
sudo lvs

# Create a mount point and mount the Logical Volume strictly READ-ONLY (-o ro,noload for ext4 or -o ro,norecovery for Btrfs)
sudo mkdir -p /mnt/nas_rescue

# For ext4 volumes:
sudo mount -o ro,noload /dev/vg1000/lv /mnt/nas_rescue

# OR for Btrfs volumes (such as Synology Btrfs pools):
sudo mount -o ro,norecovery /dev/vg1000/lv /mnt/nas_rescue

Once /mnt/nas_rescue mounts, all of your NAS shared folders (photo, homes, video, documents) will appear inside /mnt/nas_rescue/ so you can copy them straight onto an external USB backup drive.

• What if your Synology NAS had "Synology Hybrid RAID (SHR)" with mixed drive sizes? Synology SHR simply splits larger drives into multiple partitions (sda5, sda6) and creates a separate mdadm array across each partition tier, then joins those md arrays together inside a single LVM Volume Group (vg1000). Running sudo mdadm --assemble --scan --readonly and sudo vgchange -ay automatically stitches multi-tier SHR volumes back together on Ubuntu.
• Commercial GUI Alternative (Windows/macOS/Linux): If you are uncomfortable with Linux terminal commands or have a complex QNAP thin-provisioned LVM pool, UFS Explorer RAID Recovery or ReclaiMe Free RAID Recovery can inspect all connected NAS drives on a Windows or Linux PC in 100% read-only software mode, automatically detect the stripe size, parity rotation, and LVM offsets, and copy your folders out without modifying a single sector on the NAS drives.

ZFS (zpool) and Btrfs emergency rescue commands¶

Modern NAS platforms (TrueNAS SCALE/CORE, Proxmox, Ubuntu Server, Unraid, and Synology Btrfs volumes) increasingly use copy-on-write (CoW) file systems instead of legacy ext4. If a ZFS zpool or Btrfs pool refuses to mount or causes a kernel panic on boot after a power outage or drive failure, normal mounting will fail because the OS tries to replay damaged write transactions. You must use read-only transaction rollback parameters.

1. ZFS (zpool) Emergency Recovery (TrueNAS / Proxmox / Linux ZFS)¶

If a ZFS pool states FAULTED, UNAVAIL, or crashes the server (kernel panic) the instant you try to import it after an unclean power loss:

# 1. Inspect all visible ZFS pools on the connected disks without importing them
sudo zpool import

# 2. ALWAYS attempt a Read-Only Import first (prevents ZFS from writing to disks or replaying the ZIL/SLOG)
sudo zpool import -f -o readonly=on -R /mnt/zfs_rescue <pool_name>

# 3. What if a normal import causes a kernel panic or says 'damaged pool metadata'?
# Use -F (Rollback/Recovery Mode) with -n (Dry Run ONLY) to see if ZFS can rewind the last few seconds of corrupted Transaction Groups (TXGs):
sudo zpool import -F -n <pool_name>

# 4. Combine Read-Only + Safe Transaction Group Rewind (-F -o readonly=on) so no disk sectors are altered while you copy your data off:
sudo zpool import -f -F -o readonly=on -R /mnt/zfs_rescue <pool_name>

# 5. For heavily damaged pools where Space Map allocation tables are corrupt, disable space map loading prior to read-only import:
echo 1 | sudo tee /sys/module/zfs/parameters/zfs_recover
sudo zpool import -f -F -o readonly=on -R /mnt/zfs_rescue <pool_name>

Once mounted under /mnt/zfs_rescue, copy your datasets off using rsync or zfs send before ever attempting to import the pool read-write.

2. Btrfs Emergency Recovery (Synology Btrfs / Unraid Cache Pool / Linux Btrfs)¶

Instead, try mounting or extracting the Btrfs volume in strict progressive order, from least invasive to deepest scan:

# Stage 1: Mount Read-Only without replaying the corrupted Journal Log Tree (fixes 80% of power-loss Btrfs mount failures)
sudo mount -t btrfs -o ro,norecovery /dev/sdX1 /mnt/btrfs_rescue
# (Note: on newer Linux kernels 5.11+, use: -o ro,rescue=nologreplay)

# Stage 2: Mount Read-Only using the backup Root Tree + ignoring bad checksums (for multi-drive or bit-rot corruption)
sudo mount -t btrfs -o ro,rescue=all /dev/sdX1 /mnt/btrfs_rescue
# (On older kernels: sudo mount -t btrfs -o ro,usebackuproot,nologreplay,degraded /dev/sdX1 /mnt/btrfs_rescue)

# Stage 3: If the volume still refuses to mount in the kernel, extract files directly in userspace without mounting using 'btrfs restore'
# First, list available backup tree roots:
sudo btrfs-find-root /dev/sdX1

# Then run 'btrfs restore' (strictly read-only userspace extractor) to dump all files to an external healthy drive:
sudo btrfs restore -v -i -m -S /dev/sdX1 /mnt/external_usb_backup/

Hardware RAID Controller Emergencies: "Foreign Configuration" and Dead RAID Cards¶

Enterprise servers (Dell PowerEdge with PERC cards, HPE ProLiant with Smart Array, Lenovo ThinkSystem, or workstations with Broadcom/LSI MegaRAID controllers) use hardware RAID cards with onboard NVRAM and battery/supercapacitor cache (BBU / CacheVault).

1. Import Foreign Configuration vs. Clear Configuration (CRITICAL)¶

When a drive momentarily disconnects or you move drives into a replacement server chassis, the Dell PERC or LSI MegaRAID BIOS boot screen will pause and announce: "Foreign configuration(s) found on adapter." * ALWAYS select Import Foreign Configuration (or Preview Foreign Configuration -> Import). The "Foreign" tag simply means the RAID controller sees valid existing RAID metadata written on the hard drives themselves (the Disk Data Format / DDF or LSI configuration sectors at the end of the drives) that differs from the card's current NVRAM state. Importing reads the array layout straight off the drives and brings your virtual disk back online without losing a single byte. * NEVER select Clear Foreign Configuration or Initialize Virtual Disk. Clearing or initializing wipes the array metadata or zeros the drives. * Exception Warning: If a single drive dropped out three weeks ago, you kept writing new data to the degraded array for three weeks, and then you plug that old 3-week-stale single drive back in, do NOT import the foreign configuration from that single stale drive over top of your healthy active drives; instead, replace or clone the stale drive.

2. What if the Hardware RAID Card Itself Dies?¶

If your server motherboard or PCIe hardware RAID controller card suffers an electronic failure and will not boot: * You do NOT need to buy the exact same revision of motherboard, and your data is still intact on the drives. * Purchase a replacement RAID controller from the same controller chipset family (for example, any LSI/Broadcom MegaRAID SAS 2108/2208/3108 or Dell PERC H710/H730/H740 card can read DDF metadata from an older PERC card). * Install the replacement RAID card into a working PC, connect your labeled drives in their original bay order, boot into the controller BIOS (Ctrl+R or Ctrl+H or UEFI HII menu), and select Import Foreign Configuration. The controller will read the RAID layout off the drives and mount your volume immediately.

Frequently asked questions¶

Related guides¶

• Hard drive failure and data recovery
• Accidental file deletion, formatted drive, and photo recovery
• Computer backups: 3-2-1 strategy and disaster recovery
• Linux system boot failure and GRUB repair
• Ransomware and malware emergency response
• Global emergency numbers

Sources checked¶

• Linux Kernel Organization (International): Linux RAID (mdadm) Wiki (raid.wiki.kernel.org), Recovering a Failed Software RAID, and Btrfs Kernel Documentation (btrfs.readthedocs.io on btrfs rescue and btrfs restore).
• OpenZFS Documentation Project (International): zpool import Read-Only and Transaction Group Rollback (-F -o readonly=on) Administration Reference.
• Synology Inc. (Taiwan/International): Official Synology Knowledge Base Article KB-1079 (How can I use a PC to recover data when my Synology NAS malfunctions using Ubuntu mdadm and lvm2).
• GNU Project / Free Software Foundation (International): GNU ddrescue Manual for Non-Destructive Block-Level Imaging of Degraded RAID Member Drives.
• National Institute of Standards and Technology (NIST) (United States): Special Publication 800-86 and Storage Reliability Guidance on RAID Rebuild Exposure Windows and Uncorrectable Read Error (URE) Probabilities.