Skip to content

RAID 5 or RAID 6 Degraded Array With Second Drive Failing During Rebuild (Why Never Run CHKDSK or fsck, ddrescue Sector Cloning, and mdadm Read-Only Assembly)

Тез жардам номерлери өлкөгө жараша айырмаланат. This page uses 911, the number for the United States and Canada. Many other countries use 112, 999, or another number. Өлкөңүздүн номерин табыңыз.

Replacing a failed hard drive in a degraded RAID 5 же RAID 6 storage server, Synology DiskStation, QNAP NAS, Linux mdadm array, or Dell PERC / HP Smart Array / LSI MegaRAID hardware controller only to watch a second remaining member drive throw UNC (Uncorrectable Read Error) bad sectors, Pending Sectors, or drop offline at 42% of the parity rebuild is one of the most dangerous storage emergencies in enterprise and home-lab computing. In a single-parity RAID 5 array (N - 1 fault tolerance), the mathematical XOR reconstruction of the replacement disk requires reading every single sector (LBA 0 to the final LBA) across 100 percent of the remaining N - 1 member drives simultaneously. When modern high-capacity 8 TB чейин 24 TB helium drives (purchased from the same manufacturing batch and subjected to identical thermal and vibration hours) undergo a multi-terabyte sequential rebuild read, a single latent media defect (URE) or weak read head on a second drive causes the RAID controller to abort the rebuild, kick the second drive out as Faulty / Crashed, and mark the entire storage pool Offline / Crashed / Superblock Event Count Mismatch.

Do not panic, and do not force a second rebuild or run filesystem repair utilities (fsck, e2fsck, xfs_repair, or Windows CHKDSK). Even when a second drive in RAID 5 (or third drive in RAID 6) develops 50 чейин 5,000 bad sectors and gets kicked out of the array, 99.9999 percent of the sectors on that "failed" second drive are still 100 percent readable. By immediately halting the array, cloning every unstable member disk sector-by-sector onto healthy spare drives using GNU ddrescue, and assembling the cloned images in Read-Only / --assume-clean overlay mode (mdadm + dmsetup Copy-on-Write loop overlays), you can mount the filesystem safely and copy out virtually every file intact. For NAS appliance specifics and hardware controller failures, read our companion guides on RAID 5 or RAID 6 Synology, QNAP, or TrueNAS multiple drive drop, RAID controller card death and foreign configuration import, жана NAS RAID array degraded and ZFS/Btrfs recovery. If your server failure impacts life-safety systems or medical infrastructure, contact your local emergency number (911 Кошмо Штаттарда жана Канадада, 112 Европа Биримдиги боюнча, 999 Улуу Британияда) же биздин каталогду текшериңиз тез жардам номерлери.

Муну азыр кыл

  1. Abort any active parity rebuild or scrub immediately and power down the array cleanly (or disconnect SATA/SAS backplane power if a drive is making mechanical clicking or scraping noises):
  2. Why letting a stalling rebuild continue destroys the array: When a member drive hits a bad sector patch (LBA read timeout) or has a degrading read/write head (media scratch), leaving the RAID controller retrying sectors for hours grinds the microscopic magnetic platter coating into dust across the entire track cylinder, turning a recoverable 20-sector defect into a catastrophic full-platter head crash.
  3. NEVER run CHKDSK /f, fsck -y, e2fsck, btrfs check --repair, же xfs_repair on a degraded, out-of-sync, or half-rebuilt RAID array:
  4. Неге CHKDSK жана fsck cause 100% permanent data destruction on out-of-sync RAID: If Drive 2 dropped out on Tuesday and Drive 3 dropped out during Thursday's rebuild, the mdadm or hardware RAID Event Counter (Update Time) on those two drives differs by thousands of transactions. If you force-assemble an array with mismatched drive generations or missing stripes and allow fsck же CHKDSK массивге жазуу үчүн, файл тутумунун текшерүүчүсү туура эмес RAID тилкелеринен келип чыккан сынган инод дарактарын көрөт, каталогдун миллиондогон жарактуу жазууларын жок кылат. lost+found, and overwrites good data stripes with corrupted parity across every disk in the array.
  5. ** ЭЧ КАЧАН чыкылдатпаңыз "Create Array", "Initialize", же "Clear Configuration" in a hardware RAID BIOS, and NEVER run mdadm --create without --assume-clean and a Copy-on-Write (COW) loop overlay**:
  6. Running a standard mdadm --create (жок --assume-clean жана так кесек өлчөмүнө, жайгашуусуна жана маалыматтардын офсетине дал келүү) дароо бардык дисктерде жаңы паритет тилкелерин жаза баштайт, файл тутумунун суперблокторун жана B-дарактарын жок кылат, эгерде кандайдыр бир параметр болсо (512K vs. 64K кесек, left-symmetric vs. left-asymmetric, же 2048s vs. 262144s data offset) is wrong by even one sector.
  7. Label every physical hard drive slot (Bay 0 / Bay 1 / Bay 2 / Bay 3 and serial number) with a permanent marker or tape BEFORE pulling a single drive out of the chassis:
  8. Write the exact bay number and serial number on the top label of each drive. Knowing the physical bay order and checking mdadm --examine metadata saves hours of stripe-order reconstruction later.
  9. Clone the failing drive(s) sector-by-sector to healthy new drives using GNU ddrescue (with a mapfile) over direct motherboard SATA/HBA ports BEFORE attempting any RAID assembly:
  10. Never attempt to assemble or mount a RAID array using the physically dying drive itself. Connect the failing drive to a non-RAID SATA/AHCI or LSI IT-Mode HBA port on a Linux rescue workstation, disable OS auto-mounting, and run ddrescue -f -n /dev/sdX /dev/sdY rescue_driveX.map so the healthy fast sectors are evacuated in pass one before touching bad sector zones.
  11. Assemble the cloned drives in Read-Only mode (mdadm --assemble --readonly or using dmsetup / losetup RAM/file overlays):
  12. By placing a disposable Copy-on-Write (COW) overlay file on top of each cloned member disk, you can test-mount the array, replay an ext4 же XFS journal, or even test a virtual rebuild while guaranteeing that zero bytes are ever modified on your underlying disk clones.

Danger

The "Out-of-Order Drive Force" Trap (Stale Drive Overwriting Fresh Drives): Suppose you have a 4-drive RAID 5 (Disk 1, Disk 2, Disk 3, Disk 4). Disk 3 failed 6 months ago, and you never replaced it (running degraded for 6 months). Today, Disk 2 develops 100 bad sectors and drops out. Never force the 6-month-old Disk 3 back online to rebuild Disk 2. If you use Disk 1 + Disk 3 (6 months stale) + Disk 4 to rebuild or mount the array, every file created or modified in the last 6 months will be corrupted into unreadable binary garbage. You must clone Disk 2 (the drive that failed today) with ddrescue and assemble Disk 1 + Cloned Disk 2 + Disk 4.


Decision tree: rescuing a RAID 5 or RAID 6 array when a second drive fails during rebuild

Follow this forensic storage recovery workflow to recover your array without losing a single surviving stripe:

flowchart TD
    Start["RAID 5 (2nd Drive) or RAID 6 (3rd Drive) Fails / Drops Offline During Rebuild"] --> StopNow["STEP 1: Abort Rebuild Immediately & Power Down Array (Prevent Platter Scoring on Failing Drive)"]

    StopNow --> SoundCheck{"Did the Failing Drive Make Loud Clicking, Beeping, or Grinding Sounds?"}

    SoundCheck -->|"YES: Mechanical Head Crash / Stiction"| CleanRoom["STOP: Do Not Power That Drive On Again. Send ONLY the Mechanically Failed Drive (Plus 1 Healthy Drive for Stripe Reference) to a Cleanroom Lab to Image Onto a Clone"]

    SoundCheck -->|"NO: Spins Smoothly, Failed Due to Bad Sectors (UNC / Pending / Timeout)"| LabelDrives["STEP 2: Label Every Drive With Its Exact Bay Number & Serial Number -> Connect to Linux Workstation via Direct SATA / IT-Mode HBA"]

    LabelDrives --> InspectMeta["Run 'mdadm --examine /dev/sd*' and 'smartctl -a /dev/sd*' on All Drives (Record Update Time, Events Counter, Role, Chunk Size, Layout, Data Offset)"]

    InspectMeta --> CloneDrives["STEP 3: Clone Every Degraded / Bad-Sector Drive to a Healthy Spare Drive Using 'ddrescue -f -n' (Fast Pass First, Then Scrape Pass With Mapfile)"]

    CloneDrives --> PickFreshest["Identify the N-1 (RAID 5) or N-2 (RAID 6) Drives With the HIGHEST / MOST RECENT 'Events' Counter (Never Mix a Stale Old Dropped Drive With Fresh Drives)"]

    PickFreshest --> CowOverlay["STEP 4: Create Read-Only Loop / 'dmsetup' Copy-on-Write (COW) Overlays Over the Cloned Drives So Original Clones Cannot Be Modified"]

    CowOverlay --> AssembleRO["Run 'mdadm --assemble --force --readonly' on the Overlay Devices -> Mount Filesystem Read-Only ('mount -o ro,noload') -> Copy All Critical Data to External Storage"]

Phase 1: Understanding why a second drive fails during a RAID rebuild (XOR Parity Math & Event Counters)

To understand why your array crashed mid-rebuild and why your data is still recoverable, examine how RAID 5 жана RAID 6 actually work on disk.

1. How XOR parity reconstruction works and why 1 bad sector aborts an entire rebuild

In a 4-drive RAID 5 array (Drive A, Drive B, Drive C, Drive D), data is split into fixed-size chunks (typically 64 KB же 512 KB) striped across the disks, where one chunk in each row holds the XOR Parity (P): $\(\text{Parity } P = A \oplus B \oplus C\)$ Эгерде Drive C fails completely and you insert a blank Replacement Drive C_new, the RAID controller must reconstruct every stripe on C_new by calculating: $\(C_{\text{reconstructed}} = A \oplus B \oplus P\)$ - What happens when Drive B hits a single unreadable 4 KB sector (UNC error) at 42% of the rebuild: - To calculate that single 4 KB block on C_new, the controller must have a valid read from Drive A, Drive B, жана Drive D. - Качан Drive B returns a hardware Read Error (Sense Key 03h / 11h Unrecovered Read Error) after 7 seconds (TLER / ERC timeout), consumer and enterprise RAID controllers are often programmed with a catastrophic default behavior: instead of skipping that single 4 KB sector, logging a zero block, and finishing the other 99.9999% of the rebuild, the RAID controller marks Drive B катары FAILED / OFFLINE and aborts the entire rebuild. - At that moment, your array has two drives marked "Failed" (Drive C which was only 42% rebuilt, and Drive B which actually has 99.99998% healthy sectors and only 12 bad sectors).

2. Why Event Counters (Update Time) determine which drives you must use for recovery

Ар жолу а RAID 5 же RAID 6 array writes data or changes state (such as a drive dropping out), the RAID metadata (mdadm superblock or hardware DDF metadata) increments a 64-bit Events counter on all currently active drives.

Compare the mdadm --examine output of a typical 4-drive RAID 5 disaster where sdb failed yesterday (Events: 10450) жана sdc dropped out during today's rebuild (Events: 10892):

Physical Drive Role in Array Update Time Events Counter Physical Health & What It Contains
/dev/sda (Bay 1) Active device 0 Today 14:22:10 10892 100% Healthy & Current: Contains every write up to the exact second the rebuild crashed today.
/dev/sdb (Bay 2) spare / faulty (Replaced by new disk) Yesterday 09:15:00 10450 (Stale) Failed Yesterday: Missing 29 hours of database writes, OR replaced by a new disk that is only 42% rebuilt (Reshape/Rebuild Pos).
/dev/sdc (Bay 3) faulty (Dropped at 42% rebuild) Today 14:22:09 10890 (Off by 2) 99.999% Healthy & Current: Only kicked out 1 second before /dev/sda жана /dev/sdd because it hit 8 bad sectors during the rebuild read.
/dev/sdd (Bay 4) Active device 3 Today 14:22:10 10892 100% Healthy & Current: Contains every write up to the exact second the rebuild crashed today.
  • The Golden Rule of RAID Recovery from the table above:
  • Notice that /dev/sda (10892), /dev/sdc (10890), and /dev/sdd (10892) all have virtually identical Events counters from Today 14:22.
  • Once we clone /dev/sdc onto a healthy drive using ddrescue (recovering 99.9999% of its sectors and zero-filling the 8 unreadable sectors), we can force-assemble /dev/sda + Cloned /dev/sdc + /dev/sdd (ignoring the half-rebuilt /dev/sdb completely). That 3-drive combination gives us a 100 percent consistent, up-to-the-second degraded RAID 5 array with zero stripe tearing.

Phase 2: Step-by-step sector cloning of the failing drive with GNU ddrescue

Before you touch mdadm or any RAID assembly tool, you must clone the drive(s) that threw read errors onto healthy replacement drives of equal or larger byte capacity.

Step 1: Prepare the Linux rescue environment and disable auto-assembly

  1. Boot a Linux workstation (or boot your server from a live USB stick such as SystemRescue system-rescue.org or Ubuntu Live Server) with the drives connected to standard motherboard SATA ports or an LSI HBA in IT Mode (Non-RAID pass-through).
  2. Prevent Linux from automatically trying to assemble or mount your broken RAID array on boot:
  3. When booting SystemRescue or Ubuntu Live, check cat /proc/mdstat. If Linux auto-assembled a partial md device in the background, stop it immediately in read-only safe mode before imaging:
    mdadm --stop /dev/md*
    
  4. Identify every drive by exact model, serial number, and byte size (never guess /dev/sdX letters, because Linux reassigns drive letters across reboots):
    lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE,FSTYPE
    
    Write down which /dev/sdX corresponds to your Failing Source Drive (for example, /dev/sdc, serial WCC4E12345) and which /dev/sdY is your Brand-New Blank Target Drive (for example, /dev/sdf, serial ZGG9988776).

Step 2: Run the 2-Stage GNU ddrescue cloning protocol (Always use a Mapfile)

Never use standard dd (dd if=/dev/sdc of=/dev/sdf) on a failing RAID drive: standard dd either aborts on the first bad sector or shifts output offsets if sync is misconfigured, and it has no memory of which sectors succeeded. GNU ddrescue (gddrescue package) uses an intelligent multi-pass state mapfile:

  1. Pass 1: Fast Non-Scraping Clone (-n flag) to Rescue 99.99% of Healthy Sectors First:
    ddrescue -f -n -b 4096 /dev/sdc /dev/sdf /root/raid_bay3_sdc.map
    
  2. -f: Allows writing directly to block device /dev/sdf (triple-check that /dev/sdc is the source and /dev/sdf is the blank target).
  3. -n (--no-scrape): Tells ddrescue to immediately jump over any slow or bad sector region without grinding the read head, rescuing all healthy gigabytes at maximum speed (150 to 250 MB/s) before the drive's health worsens.
  4. /root/raid_bay3_sdc.map: Saves the exact sector progress every 30 seconds. Even if the failing drive locks up and requires a power cycle, re-running the exact same command with the .map file resumes at the exact sector where it stopped.
  5. 2-өтүү: Максаттуу тескери/кесүүчү кыргыч (-d -r1) to Recover Marginal Bad Sectors:
  6. Pass 1 аяктагандан кийин (pct rescued: 99.99%), ядронун SCSI буйругунун күтүү убактысын төмөндөтүңүз, ошондуктан ар бир жаман сектор тез иштебей калат (5 seconds instead of waiting 30 to 180 seconds per sector):
    echo 5 > /sys/block/sdc/device/timeout
    
  7. Run Pass 2 with direct disk access (-d) and 1 retry (-r 1):
    ddrescue -f -d -r 1 -b 4096 /dev/sdc /dev/sdf /root/raid_bay3_sdc.map
    
  8. Shut down, disconnect the original failing /dev/sdc drive, and set it safely on a shelf: From this point forward, you will work exclusively with your healthy clone (/dev/sdf) and your other healthy member drives.

Phase 3: Safe Read-Only RAID assembly using Copy-on-Write (COW) overlays and mdadm

Now that you have healthy disk clones, never run mdadm --assemble --force directly on the raw drives without a protective Copy-on-Write (COW) overlay or --readonly flag. Why? Because --force rewrites the RAID superblocks on the drives, and if a filesystem journal (ext4 / XFS) replays automatically on mount using the wrong drive combination, it permanently alters your clones.

Step 1: Inspect the RAID metadata across all member drives (mdadm --examine)

Run mdadm --examine on every member drive (or partition, e.g., /dev/sda1 же /dev/sda3 on Synology/QNAP NAS drives) and save the output to a text file:

mdadm --examine /dev/sd[a,b,d,f]* > /root/raid_metadata_audit.txt
cat /root/raid_metadata_audit.txt
Look at four critical fields for each drive: 1. Array UUID: Verify all drives belong to the exact same RAID pool. 2. Device Role: Shows the exact stripe index (Active device 0, Active device 1, Active device 2, Active device 3). 3. Update Time & Events: Identify the N - 1 drives (for RAID 5) or N - 2 drives (for RAID 6) that share the highest/closest Events count. 4. Raid Level, Layout, Chunk Size, жана Data Offset: Record these exact numbers (e.g., Raid Level : raid5, Layout : left-symmetric, Chunk Size : 512K, Data Offset : 262144 sectors).

Step 2: Create non-destructive Copy-on-Write (COW) Device-Mapper overlays (overlay-create)

To guarantee 100 percent safety so that even if you force-assemble the array and run mount, zero bytes are ever written to your physical hard drives, create loop-backed dm-snapshot Copy-on-Write overlays in /tmp (or on an external scratch drive):

# Create a small sparse COW file + loop device for each member partition (e.g., sda3, sdf3, sdd3)
mkdir -p /root/cow
for dev in sda3 sdf3 sdd3; do
    truncate -s 20G /root/cow/${dev}.cow
    loop_base=$(losetup --find --show --read-only /dev/${dev})
    loop_cow=$(losetup --find --show /root/cow/${dev}.cow)
    sz=$(blockdev --getsz /dev/${dev})
    echo "0 $sz snapshot $loop_base $loop_cow P 8" | dmsetup create ovl_${dev}
done
- What this does: You now have virtual block devices /dev/mapper/ovl_sda3, /dev/mapper/ovl_sdf3, жана /dev/mapper/ovl_sdd3. The underlying physical drives (sda3, sdf3, sdd3) are locked --read-only at the kernel block layer. Any writes made by mdadm --force, LVM же файл тутумунун журналын кайра ойнотуу бир жолу колдонулуучуга зыянсыз кирет /root/cow/*.cow файлдар. Эгер бир нерсе туура эмес болуп калса, сиз жөн гана катмарларды жок кылып, 2 секунданын ичинде кайра түзөсүз.

Step 3: Force-assemble the array on the overlay devices and mount Read-Only

  1. Assemble the freshest N - 1 overlay devices using --assemble --force:
    mdadm --assemble --force --verbose /dev/md99 /dev/mapper/ovl_sda3 /dev/mapper/ovl_sdf3 /dev/mapper/ovl_sdd3
    
  2. Why we leave out the stale/half-rebuilt drive (sdb3): Forcing mdadm to assemble with the 3 up-to-date drives (sda3, cloned sdf3, жана sdd3) brings /dev/md99 online in clean degraded mode ([U_UU]) using the exact latest Events state, while ddrescue already rescued the data from sdf3.
  3. Activate LVM (vgchange) if your NAS or server uses LVM on top of RAID (Synology SHR, QNAP, Proxmox, Ubuntu):
    pvscan
    vgscan
    vgchange -ay
    lvs
    
    (If your Synology NAS uses Btrfs on LVM or Synology Hybrid RAID, you will now see /dev/vg1000/lv же /dev/vg1/volume_1 appear.)
  4. Mount the filesystem in strict Read-Only mode and evacuate your critical files immediately:
    mkdir -p /mnt/raid_rescue
    # For ext4:
    mount -o ro,noload /dev/md99 /mnt/raid_rescue
    # (Or if noload refuses due to dirty journal, mount the COW overlay normally so the journal replays into the disposable COW file)
    mount -o ro /dev/md99 /mnt/raid_rescue
    # For Btrfs (Synology NAS):
    mount -t btrfs -o ro,usebackuproot,nologreplay /dev/vg1000/lv /mnt/raid_rescue
    
  5. Copy your most critical folders first using rsync to an external backup drive:
    rsync -avP --ignore-errors /mnt/raid_rescue/ /mnt/external_backup_usb/
    

Phase 4: Rescuing Hardware RAID Controllers (Dell PERC, LSI MegaRAID, HP Smart Array, Adaptec) and ZFS / Btrfs Pools

What if your array was managed not by Linux mdadm, but by a Hardware RAID Controller Card (Dell PERC H730/H740/H755, Broadcom / LSI MegaRAID, же HP Smart Array) that now reports Virtual Disk Offline же Foreign Configuration Found after a second drive hiccupped during rebuild?

1. Hardware RAID Cards (Dell PERC, LSI MegaRAID, HP Smart Array)

  1. ** ЭЧ КАЧАН чыкылдатпаңыз "Clear Foreign Configuration" же "Initialize Virtual Disk"**:
  2. In Dell PERC / LSI MegaRAID BIOS (Ctrl+R же HII UEFI Configuration Utility), Clear wipes the RAID metadata from the drives, and Initialize overwrites the data stripes.
  3. If a healthy drive briefly dropped offline due to a loose backplane cable or timeout, Import Foreign Configuration attempts to bring the drive back in; however, if one drive dropped weeks ago and a second dropped today, importing both or forcing the stale drive online via the controller BIOS can corrupt the array.
  4. The Safe Software Reconstruction Method for Hardware RAID (DDF Metadata / dmraid / R-Studio / UFS Explorer):
  5. Аппараттык RAID карталары стандартты сактайт SNIA DDF (Disk Data Format) metadata in the final sectors at the end of each hard drive, and stripe data across the drives starting at sector 0 (or after a small header offset) using standard 64 KB, 128 KB, 256 KB, же 512 KB кесим өлчөмдөрү (Left-Asymmetric же Left-Symmetric).
  6. Instead of gambling with a destructive hardware RAID card BIOS:
    1. Label all drives by slot number and connect them to a standard SATA / non-RAID IT-Mode HBA on a recovery workstation.
    2. Clone the bad-sector drive using GNU ddrescue as shown in Phase 2.
    3. Open the cloned drives in UFS Explorer RAID Recovery же R-Studio Technician, же аларды колдонуу менен Linux'та окуу үчүн гана чогултуңуз dmsetup / mdadm --build: these tools automatically detect the LSI / PERC / HP тилке тартиби, кесим өлчөмү (64K/512K), and parity rotation in 100 percent read-only memory without ever letting the hardware RAID card write a single byte to your drives.

2. TrueNAS / ZFS (RAID-Z1 / RAID-Z2) and Synology Btrfs Scrub/Resilver Failures

If your pool uses OpenZFS (RAID-Z1 же RAID-Z2) on TrueNAS or Proxmox and a second drive throws CKSUM / READ errors during a zpool replace resilver: 1. Why ZFS handles bad sectors during resilvering better than hardware RAID, and when it still locks up: - Unlike naive hardware RAID cards that drop an entire disk on a single bad sector, ZFS continues resilvering past localized READ errors and logs only the specific damaged file paths in zpool status -v. - However, if the second failing drive hangs the SATA/SAS bus (I/O timeout / head thrashing) or drops completely offline (UNAVAIL / FAULTED), the pool enters SUSPENDED state (pool I/O is currently suspended). 2. How to rescue a SUSPENDED же FAULTED ZFS RAID-Z1 / RAID-Z2 pool: - Power down cleanly, clone the failing drive onto a healthy replacement disk using GNU ddrescue -f -n (ZFS partition labels include the exact vdev GUID at the start and end of the disk, so a raw sector clone is automatically recognized by ZFS as the original vdev member). - Disconnect the physically dying drive, leave the ddrescue clone connected, and import the ZFS pool in strict Read-Only mode (readonly=on) without starting a write transaction or resuming the resilver:

zpool import -o readonly=on -f -R /mnt/zfs_rescue tank
- Importing with -o readonly=on prevents ZFS from writing new uberblocks or attempting background healing writes, allowing you to copy every dataset off safely via zfs send же rsync before rebuilding the pool from scratch.


Көп берилүүчү суроолор

Why did you say NEVER to run CHKDSK же fsck when my RAID array mounts with filesystem errors after a drive drop?

On a single standalone hard drive, running CHKDSK же fsck майда каталогдорго туура келбегендиктерди оңдойт. А multi-drive RAID 5 or RAID 6 array that suffered a double-drive drop or interrupted rebuild, filesystem errors are almost always caused by RAID stripe misalignment or a stale member drive being forced online (meaning every 3rd же 4th 64 KB chunk of data is being read from the wrong drive generation). If you let CHKDSK /f же fsck -y run on a misaligned or out-of-sync RAID array, it misinterprets millions of valid files as corrupted metadata, truncates directory trees, dumps unnamed fragments into lost+found, and writes new parity blocks across all drives, making it mathematically impossible to ever reassemble the correct drive combination afterward. Always mount via Read-Only Copy-on-Write (COW) overlays first.

What if the partially rebuilt replacement drive (42% complete) AND the second drive that failed at 42% BOTH have needed data? Can I combine them?

Yes. This is a classic "Split-Horizon / Hybrid Clone" RAID rescue. Suppose Drive 2 died completely on Monday, you put in New Drive 2, and the rebuild made it to 42% (LBA 3,500,000,000) before Drive 3 suffered a fatal head crash at LBA 3,500,000,000. Notice what you actually have: for the first 42% of the array (0% чейин 42%), New Drive 2 has already been 100 percent reconstructed. So for the first 42% of the LBA range, you have a complete set of stripes using Drive 1 + New Drive 2 + Drive 4 (treating crashed Drive 3 as the missing disk). And if Drive 3 only had a localized bad-sector patch at 42% (and is readable from 43% чейин 100% аркылуу ddrescue), you use Drive 1 + Cloned Drive 3 + Drive 4 for the remaining 58% of the array.

Why do second drives fail so frequently right in the middle of a RAID 5 rebuild?

Two engineering realities combine to make RAID 5 rebuilds notorious for double-drive failures: (1) Identical Batch Aging: Most users buy 4 to 8 identical hard drives from the same retailer on the same day (same firmware, same factory lot, same spindle motor batch) and run them side-by-side at the exact same temperature for 35,000 hours; when Drive 1 wears out, Drive 2 and Drive 3 are days away from the same mechanical wear threshold. (2) Latent Uncorrectable Read Errors (UREs): Under normal daily operation, 95% of a hard drive's cold archival sectors are not read for months at a time, so latent bad sectors sit undiscovered. A RAID 5 rebuild forces a continuous 100% surface read of every single sector across every surviving drive (40 to 100 terabytes of continuous reading), immediately exposing any latent bad sector that accumulated over the past year.

How do I know which files were affected by the few unreadable bad sectors that ddrescue had to zero-fill?

GNU ddrescue includes a built-in forensic utility called ddrescuelog жана ddru_ntfsfindbad / debugfs ал так калыбына келтирилбеген сектор LBA диапазонуңуздун картасын түзөт .карта файл (абалы = '-' начар секторлор) RAID тилке математикасы аркылуу түздөн-түз файл тутумунун инод таблицасына. Кайсынысын билгенден кийин 4 KB блоктор клондоштуруу учурунда нөлгө толтурулган, сиз сурасаңыз болот debugfs (ичек жана ncheck on ext4) же btrfs inspect-internal logical-resolve to print out the exact list of 2 or 3 filenames that overlapped those bad sectors, confirming with 100 percent mathematical certainty that every other file on your multi-terabyte server is completely uncorrupted.


Тиешелүү гиддер


Булактар жана расмий жетекчилик

  • Linux Raid Wiki (raid.wiki.kernel.org/index.php/Linux_Raid & Recovering_a_failed_software_RAID): Official Kernel mdadm Recovery Protocol, Device-Mapper Copy-on-Write (dm-snapshot) Overlays, and Events Counter Reconciliation.
  • GNU ddrescue Official Engineering Manual (gnu.org/software/ddrescue/manual/ddrescue_manual.html): Multi-Phase Non-Scraping (-n) and Direct Access (-d) Sector Recovery Algorithms for Failing Magnetic Media.
  • Synology Official Knowledge Base (kb.synology.com/en-global/DSM/tutorial/How_can_I_recover_data_from_my_DiskStation_using_a_PC): Mounting mdadm + LVM2 + Btrfs / ext4 NAS Volumes Read-Only on Ubuntu Rescue Systems.
  • Storage Networking Industry Association (SNIA): Common RAID Disk Data Format (DDF) Specification for Hardware RAID Controller Metadata Portability.