RAID 5 හෝ RAID 6 නැවත ගොඩනැගීමේදී දෙවන ධාවකය අසමත් වීමත් සමඟ පිරිහුණු අරාව (කිසිවිටක CHKDSK හෝ fsck ධාවනය නොකරන්නේ මන්ද, ddrescue Sector Cloning, සහ mdadm කියවීමට පමණක් එකලස් කිරීම)¶
හදිසි අවස්ථා අංක රට අනුව වෙනස් වේ. මෙම පිටුව එක්සත් ජනපදය සහ කැනඩාව සඳහා වන අංකය වන 911 භාවිතා කරයි. වෙනත් බොහෝ රටවල් 112, 999 හෝ වෙනත් අංකයක් භාවිතා කරයි. ඔබේ රට සඳහා අංකය සොයන්න.
අසාර්ථක දෘඪ තැටියක් පිරිහුණු එකක් තුළ ප්රතිස්ථාපනය කිරීම RAID 5 හෝ RAID 6 storage server, Synology DiskStation, QNAP NAS, Linux mdadm array, or Dell PERC / HP Smart Array / LSI MegaRAID hardware controller only to watch a second remaining member drive throw UNC (Uncorrectable Read Error) bad sectors, Pending Sectors, or drop offline at 42% of the parity rebuild is one of the most dangerous storage emergencies in enterprise and home-lab computing. In a single-parity RAID 5 array (N - 1 fault tolerance), the mathematical XOR reconstruction of the replacement disk requires reading every single sector (LBA 0 to the final LBA) across 100 percent of the remaining N - 1 member drives simultaneously. When modern high-capacity 8 TB දක්වා 24 TB helium drives (purchased from the same manufacturing batch and subjected to identical thermal and vibration hours) undergo a multi-terabyte sequential rebuild read, a single latent media defect (URE) or weak read head on a second drive causes the RAID controller to abort the rebuild, kick the second drive out as Faulty / Crashed, and mark the entire storage pool Offline / Crashed / Superblock Event Count Mismatch.
Do not panic, and do not force a second rebuild or run filesystem repair utilities (fsck, e2fsck, xfs_repair, or Windows CHKDSK). Even when a second drive in RAID 5 (or third drive in RAID 6) develops 50 දක්වා 5,000 bad sectors and gets kicked out of the array, 99.9999 percent of the sectors on that "failed" second drive are still 100 percent readable. By immediately halting the array, cloning every unstable member disk sector-by-sector onto healthy spare drives using GNU ddrescue, and assembling the cloned images in Read-Only / --assume-clean overlay mode (mdadm + dmsetup Copy-on-Write loop overlays), you can mount the filesystem safely and copy out virtually every file intact. For NAS appliance specifics and hardware controller failures, read our companion guides on RAID 5 or RAID 6 Synology, QNAP, or TrueNAS multiple drive drop, RAID controller card death and foreign configuration import, සහ NAS RAID array degraded and ZFS/Btrfs recovery. If your server failure impacts life-safety systems or medical infrastructure, contact your local emergency number (911 එක්සත් ජනපදයේ සහ කැනඩාවේ, 112 යුරෝපා සංගමය පුරා, 999 එක්සත් රාජධානියේ) හෝ අපගේ නාමාවලිය පරීක්ෂා කරන්න හදිසි අංක.
දැන් මේක කරන්න¶
- Abort any active parity rebuild or scrub immediately and power down the array cleanly (or disconnect SATA/SAS backplane power if a drive is making mechanical clicking or scraping noises):
- Why letting a stalling rebuild continue destroys the array: When a member drive hits a bad sector patch (
LBAread timeout) or has a degrading read/write head (media scratch), leaving the RAID controller retrying sectors for hours grinds the microscopic magnetic platter coating into dust across the entire track cylinder, turning a recoverable20-sectordefect into a catastrophic full-platter head crash. - NEVER run
CHKDSK /f,fsck -y,e2fsck,btrfs check --repair, හෝxfs_repairon a degraded, out-of-sync, or half-rebuilt RAID array: - ඇයි
CHKDSKසහfsckසමමුහුර්ත නොවන RAID මත 100% ස්ථිර දත්ත විනාශයක් ඇති කරයි: Drive 2 අඟහරුවාදා ඉවත් වූ අතර Drive 3 බ්රහස්පතින්දා නැවත ගොඩනැගීමේදී ඉවත් වූවා නම්,mdadmහෝ දෘඪාංග RAIDEvent Counter(Update Time) එම ධාවකයන් දෙකෙහි ගනුදෙනු දහස් ගණනකින් වෙනස් වේ. ඔබ නොගැලපෙන ධාවක පරම්පරාවන් හෝ නැතිවූ ඉරි සහිත අරාවක් බල-එකලස් කර ඉඩ දෙන්නේ නම්fsckහෝCHKDSKඅරාවට ලිවීමට, ගොනු පද්ධති පරීක්ෂකයා නොගැලපෙන RAID ඉරි නිසා ඇති වූ කැඩුණු ඉනෝඩ ගස් දකියි, ** වලංගු නාමාවලි ඇතුළත් කිරීම් මිලියන ගණනක් මකා දමයිlost+found, සහ අරාවේ ඇති සෑම තැටියක් හරහාම දූෂිත සමානාත්මතාවය සහිත හොඳ දත්ත තීරු උඩින් ලියයි**. - **කිලික් කරන්න එපා
"Create Array","Initialize", හෝ"Clear Configuration"දෘඪාංග RAID BIOS තුළ, සහ කිසි විටෙකත් ධාවනය නොකරන්නmdadm --createතොරව--assume-cleanසහ පිටපතක් ලිවීමට (COW) ලූප් ආවරණය **: - සම්මතයක් ක්රියාත්මක කිරීම
mdadm --create( තොරව--assume-cleanසහ නියම කුට්ටි ප්රමාණය, පිරිසැලසුම සහ දත්ත ඕෆ්සෙට් ගැලපීම) වහාම සියලුම තැටි හරහා නව සමානාත්මතා තීරු ලිවීම ආරම්භ කරයි, ගොනු පද්ධති සුපර්බ්ලොක් සහ B-trees කිසියම් පරාමිතියක් නම් විනාශ කරයි (512Kඑදිරිව64Kකුට්ටිය,left-symmetricඑදිරිවleft-asymmetric, හෝ2048sඑදිරිව262144sදත්ත ඕෆ්සෙට්) එක් අංශයකින් පවා වැරදියි. - ** සෑම භෞතික දෘඪ තැටියක්ම ලේබල් කරන්න (
Bay 0 / Bay 1 / Bay 2 / Bay 3සහ අනුක්රමික අංකය) චැසියෙන් තනි ධාවකයක් එළියට ගැනීමට පෙර ස්ථිර සලකුණු කරුවෙකු හෝ ටේප් එකක් සමඟ**: - එක් එක් ධාවකයේ ඉහළ ලේබලය මත නිවැරදි බේ අංකය සහ අනුක්රමික අංකය ලියන්න. භෞතික බේ ඇණවුම දැන ගැනීම සහ පරීක්ෂා කිරීම
mdadm --examineපාර-දත්ත පසුව ඉරි-ඇණවුම ප්රතිසංස්කරණය කිරීමේ පැය ගණනක් ඉතිරි කරයි. - සෞඛ්ය සම්පන්න නව ධාවක භාවිතා කිරීමට අසමත් වන ධාවකය(ය) අංශයෙන් අංශය ක්ලෝන කරන්න
GNU ddrescue(සිතියම් ගොනුවක් සමඟ) ඕනෑම RAID එකලස් කිරීමට උත්සාහ කිරීමට පෙර සෘජු මවු පුවරුව SATA/HBA ports: - භෞතිකව මිය යන ධාවකයම භාවිතා කර RAID අරාවක් එකලස් කිරීමට හෝ සවි කිරීමට කිසිවිටෙක උත්සාහ නොකරන්න. අසාර්ථක ධාවකය RAID නොවන SATA/AHCI හෝ LSI වෙත සම්බන්ධ කරන්න
IT-ModeLinux මුදාගැනීමේ වැඩපොළක HBA port, OS ස්වයංක්රීය සවිකිරීම අක්රීය කර ** ධාවනය කරන්නddrescue -f -n /dev/sdX /dev/sdY rescue_driveX.map** එබැවින් නරක අංශයේ කලාප ස්පර්ශ කිරීමට පෙර සෞඛ්ය සම්පන්න වේගවත් අංශයන් පාස් එකකින් ඉවත් කරනු ලැබේ. - ** ක්ලෝන කරන ලද ධාවකයන් එකලස් කරන්න
Read-Onlyමාදිලිය (mdadm --assemble --readonlyහෝ භාවිතා කිරීමdmsetup/losetupRAM/ගොනු ආවරණ)**: - ඉවත දැමිය හැකි පිටපතක් ලිවීමෙන් (
COW) එක් එක් ක්ලෝන කරන ලද සාමාජික තැටියට උඩින් ගොනුව උඩින් තැබීම, ඔබට අරාව පරීක්ෂා කිරීමට, නැවත ධාවනය කිරීමට හැකියext4හෝXFSසඟරාව, හෝ අතථ්ය ප්රතිනිර්මාණයක් පරීක්ෂා කරන්න ** ඔබගේ යටින් පවතින තැටි ක්ලෝනවල ශුන්ය බයිට් කවදා හෝ වෙනස් කර ඇති බවට සහතික වන අතර**.
Danger
** "පිළිවෙලින් බැහැර ඩ්රයිව් ෆෝස්" උගුල (Stale Drive Overwriting Fresh Drives): ඔබ සතුව 4-ධාවකයක් ඇතැයි සිතමු RAID 5 (Disk 1, Disk 2, Disk 3, Disk 4) Disk 3 මාස 6කට පෙර අසාර්ථක වූ අතර, ඔබ කිසිවිටෙක එය ප්රතිස්ථාපනය කළේ නැත (මාස 6ක් සඳහා ක්රියා විරහිත වී ඇත). අද, Disk 2 නරක අංශ 100 ක් සංවර්ධනය කර ඉවත් වේ. ** කිසිවිටෙක මාස 6ක් වයසැති දරුවාට බල නොකරන්න Disk 3 නැවත ගොඩනැගීමට සබැඳිව Disk 2. ඔබ භාවිතා කරන්නේ නම් Disk 1 + Disk 3 (6 months stale) + Disk 4 අරාව නැවත ගොඩනැංවීමට හෝ සවි කිරීමට, පසුගිය මාස 6 තුළ සාදන ලද හෝ වෙනස් කරන ලද සෑම ගොනුවක්ම කියවිය නොහැකි ද්විමය කුණු බවට දූෂිත වනු ඇත. ඔබ ** ක්ලෝන කළ යුතුයි Disk 2 (අසාර්ථක වූ ධාවකය අද) සමඟ ddrescue සහ එකලස් කරන්න Disk 1 + Cloned Disk 2 + Disk 4.
තීරණ ගස: නැවත ගොඩනැගීමේදී දෙවන ධාවකය අසාර්ථක වූ විට RAID 5 හෝ RAID 6 අරාව බේරා ගැනීම¶
ඉතිරිව ඇති එකදු තීරුවක්වත් අහිමි නොවී ඔබේ අරාව ප්රතිසාධනය කිරීමට මෙම අධිකරණ වෛද්ය ගබඩා ප්රතිසාධන කාර්ය ප්රවාහය අනුගමනය කරන්න:
flowchart TD
Start["RAID 5 (2nd Drive) හෝ RAID 6 (3rd Drive) අසාර්ථක වේ / නැවත ගොඩනැගීමේදී නොබැඳිව වැටේ"] --> StopNow["පියවර 1: වහාම නැවත ගොඩනැගීම නවත්වන්න සහ අරාව බල ගැන්වීම (අසාර්ථක Drive මත ප්ලැටර් ලකුණු කිරීම වැළැක්වීම)"]
StopNow --> SoundCheck{"අසමත් වන ධාවකය විශාල ක්ලික් කිරීම්, බීප් හෝ ඇඹරුම් ශබ්ද ඇති කළේද?"}
SoundCheck -->|"YES: Mechanical Head Crash / Stiction"| CleanRoom["STOP: Do Not Power That Drive On Again. Send ONLY the Mechanically Failed Drive (Plus 1 Healthy Drive for Stripe Reference) to a Cleanroom Lab to Image Onto a Clone"]
SoundCheck -->|"NO: Spins Smoothly, Failed Due to Bad Sectors (UNC / Pending / Timeout)"| LabelDrives["STEP 2: Label Every Drive With Its Exact Bay Number & Serial Number -> Connect to Linux Workstation via Direct SATA / IT-Mode HBA"]
LabelDrives --> InspectMeta["Run 'mdadm --examine /dev/sd*' and 'smartctl -a /dev/sd*' on All Drives (Record Update Time, Events Counter, Role, Chunk Size, Layout, Data Offset)"]
InspectMeta --> CloneDrives["STEP 3: Clone Every Degraded / Bad-Sector Drive to a Healthy Spare Drive Using 'ddrescue -f -n' (Fast Pass First, Then Scrape Pass With Mapfile)"]
CloneDrives --> PickFreshest["Identify the N-1 (RAID 5) or N-2 (RAID 6) Drives With the HIGHEST / MOST RECENT 'Events' Counter (Never Mix a Stale Old Dropped Drive With Fresh Drives)"]
PickFreshest --> CowOverlay["STEP 4: Create Read-Only Loop / 'dmsetup' Copy-on-Write (COW) Overlays Over the Cloned Drives So Original Clones Cannot Be Modified"]
CowOverlay --> AssembleRO["Run 'mdadm --assemble --force --readonly' on the Overlay Devices -> Mount Filesystem Read-Only ('mount -o ro,noload') -> Copy All Critical Data to External Storage"]
Phase 1: Understanding why a second drive fails during a RAID rebuild (XOR Parity Math & Event Counters)¶
To understand why your array crashed mid-rebuild and why your data is still recoverable, examine how RAID 5 සහ RAID 6 actually work on disk.
1. How XOR parity reconstruction works and why 1 bad sector aborts an entire rebuild¶
In a 4-drive RAID 5 array (Drive A, Drive B, Drive C, Drive D), data is split into fixed-size chunks (typically 64 KB හෝ 512 KB) striped across the disks, where one chunk in each row holds the XOR Parity (P):
$\(\text{Parity } P = A \oplus B \oplus C\)$
If Drive C fails completely and you insert a blank Replacement Drive C_new, the RAID controller must reconstruct every stripe on C_new by calculating:
$\(C_{\text{reconstructed}} = A \oplus B \oplus P\)$
- What happens when Drive B hits a single unreadable 4 KB sector (UNC error) at 42% of the rebuild:
- To calculate that single 4 KB block on C_new, the controller must have a valid read from Drive A, Drive B, සහ Drive D.
- When Drive B returns a hardware Read Error (Sense Key 03h / 11h Unrecovered Read Error) after 7 seconds (TLER / ERC timeout), consumer and enterprise RAID controllers are often programmed with a catastrophic default behavior: instead of skipping that single 4 KB sector, logging a zero block, and finishing the other 99.9999% of the rebuild, the RAID controller marks Drive B ලෙස FAILED / OFFLINE and aborts the entire rebuild.
- At that moment, your array has two drives marked "Failed" (Drive C which was only 42% rebuilt, and Drive B ඇත්ත වශයෙන්ම ඇති 99.99998% healthy sectors and only 12 bad sectors).
2. Why Event Counters (Update Time) determine which drives you must use for recovery¶
Every time a RAID 5 හෝ RAID 6 array writes data or changes state (such as a drive dropping out), the RAID metadata (mdadm superblock or hardware DDF metadata) increments a 64-bit Events counter on all currently active drives.
Compare the mdadm --examine output of a typical 4-drive RAID 5 disaster where sdb failed yesterday (Events: 10450) සහ sdc dropped out during today's rebuild (Events: 10892):
| Physical Drive | Role in Array | Update Time |
Events Counter |
Physical Health & What It Contains |
|---|---|---|---|---|
/dev/sda (Bay 1) |
Active device 0 |
Today 14:22:10 |
10892 |
100% Healthy & Current: Contains every write up to the exact second the rebuild crashed today. |
/dev/sdb (Bay 2) |
spare / faulty (Replaced by new disk) |
Yesterday 09:15:00 |
10450 (Stale) |
Failed Yesterday: Missing 29 hours of database writes, OR replaced by a new disk that is only 42% rebuilt (Reshape/Rebuild Pos) |
/dev/sdc (Bay 3) |
faulty (Dropped at 42% rebuild) |
Today 14:22:09 |
10890 (Off by 2) |
99.999% Healthy & Current: Only kicked out 1 second before /dev/sda සහ /dev/sdd because it hit 8 bad sectors during the rebuild read. |
/dev/sdd (Bay 4) |
Active device 3 |
Today 14:22:10 |
10892 |
100% Healthy & Current: Contains every write up to the exact second the rebuild crashed today. |
- The Golden Rule of RAID Recovery from the table above:
- Notice that
/dev/sda(10892),/dev/sdc(10890), and/dev/sdd(10892) all have virtually identicalEventscounters fromToday 14:22. - Once we clone
/dev/sdconto a healthy drive usingddrescue(recovering 99.9999% of its sectors and zero-filling the 8 unreadable sectors), we can force-assemble/dev/sda+Cloned /dev/sdc+/dev/sdd(ignoring the half-rebuilt/dev/sdbcompletely). That 3-drive combination gives us a 100 percent consistent, up-to-the-second degradedRAID 5array with zero stripe tearing.
Phase 2: Step-by-step sector cloning of the failing drive with GNU ddrescue¶
Before you touch mdadm or any RAID assembly tool, you must clone the drive(s) that threw read errors onto healthy replacement drives of equal or larger byte capacity.
Step 1: Prepare the Linux rescue environment and disable auto-assembly¶
- Boot a Linux workstation (or boot your server from a live USB stick such as SystemRescue
system-rescue.orgor Ubuntu Live Server) with the drives connected to standard motherboard SATA ports or an LSI HBA inIT Mode(Non-RAID pass-through). - Prevent Linux from automatically trying to assemble or mount your broken RAID array on boot:
- When booting SystemRescue or Ubuntu Live, check
cat /proc/mdstat. If Linux auto-assembled a partialmddevice in the background, stop it immediately in read-only safe mode before imaging: - Identify every drive by exact model, serial number, and byte size (never guess
/dev/sdXletters, because Linux reassigns drive letters across reboots): Write down which/dev/sdXcorresponds to your Failing Source Drive (for example,/dev/sdc, serialWCC4E12345) and which/dev/sdYis your Brand-New Blank Target Drive (for example,/dev/sdf, serialZGG9988776)
Step 2: Run the 2-Stage GNU ddrescue cloning protocol (Always use a Mapfile)¶
Never use standard dd (dd if=/dev/sdc of=/dev/sdf) on a failing RAID drive: standard dd either aborts on the first bad sector or shifts output offsets if sync is misconfigured, and it has no memory of which sectors succeeded. GNU ddrescue (gddrescue package) uses an intelligent multi-pass state mapfile:
- Pass 1: Fast Non-Scraping Clone (
-nflag) to Rescue 99.99% of Healthy Sectors First: -f: Allows writing directly to block device/dev/sdf(triple-check that/dev/sdcis the source and/dev/sdfis the blank target).-n(--no-scrape): Tellsddrescueto immediately jump over any slow or bad sector region without grinding the read head, rescuing all healthy gigabytes at maximum speed (150 to 250 MB/s) before the drive's health worsens./root/raid_bay3_sdc.map: Saves the exact sector progress every 30 seconds. Even if the failing drive locks up and requires a power cycle, re-running the exact same command with the.mapfile resumes at the exact sector where it stopped.- Pass 2: Targeted Reverse/Trimming Scrape (
-d -r1) to Recover Marginal Bad Sectors: - Once Pass 1 finishes (
pct rescued: 99.99%), lower the kernel SCSI command timeout so each bad sector fails fast (5 secondsinstead of waiting30 to 180 secondsper sector): - Run Pass 2 with direct disk access (
-d) and 1 retry (-r 1): - Shut down, disconnect the original failing
/dev/sdcdrive, and set it safely on a shelf: From this point forward, you will work exclusively with your healthy clone (/dev/sdf) and your other healthy member drives.
Phase 3: Safe Read-Only RAID assembly using Copy-on-Write (COW) overlays and mdadm¶
Now that you have healthy disk clones, never run mdadm --assemble --force directly on the raw drives without a protective Copy-on-Write (COW) overlay or --readonly flag. Why? Because --force rewrites the RAID superblocks on the drives, and if a filesystem journal (ext4 / XFS) replays automatically on mount using the wrong drive combination, it permanently alters your clones.
Step 1: Inspect the RAID metadata across all member drives (mdadm --examine)¶
Run mdadm --examine on every member drive (or partition, e.g., /dev/sda1 හෝ /dev/sda3 on Synology/QNAP NAS drives) and save the output to a text file:
Array UUID: Verify all drives belong to the exact same RAID pool.
2. Device Role: Shows the exact stripe index (Active device 0, Active device 1, Active device 2, Active device 3)
3. Update Time & Events: Identify the N - 1 drives (for RAID 5) or N - 2 drives (for RAID 6) that share the highest/closest Events count.
4. Raid Level, Layout, Chunk Size, සහ Data Offset: Record these exact numbers (e.g., Raid Level : raid5, Layout : left-symmetric, Chunk Size : 512K, Data Offset : 262144 sectors)
Step 2: Create non-destructive Copy-on-Write (COW) Device-Mapper overlays (overlay-create)¶
To guarantee 100 percent safety so that even if you force-assemble the array and run mount, zero bytes are ever written to your physical hard drives, create loop-backed dm-snapshot Copy-on-Write overlays in /tmp (or on an external scratch drive):
# Create a small sparse COW file + loop device for each member partition (e.g., sda3, sdf3, sdd3)
mkdir -p /root/cow
for dev in sda3 sdf3 sdd3; do
truncate -s 20G /root/cow/${dev}.cow
loop_base=$(losetup --find --show --read-only /dev/${dev})
loop_cow=$(losetup --find --show /root/cow/${dev}.cow)
sz=$(blockdev --getsz /dev/${dev})
echo "0 $sz snapshot $loop_base $loop_cow P 8" | dmsetup create ovl_${dev}
done
/dev/mapper/ovl_sda3, /dev/mapper/ovl_sdf3, සහ /dev/mapper/ovl_sdd3. The underlying physical drives (sda3, sdf3, sdd3) are locked --read-only at the kernel block layer. Any writes made by mdadm --force, LVM, or filesystem journal replay go harmlessly into the disposable /root/cow/*.cow files. If anything goes wrong, you simply delete the overlays and recreate them in 2 seconds.
Step 3: Force-assemble the array on the overlay devices and mount Read-Only¶
- Assemble the freshest
N - 1overlay devices using--assemble --force: - Why we leave out the stale/half-rebuilt drive (
sdb3): Forcingmdadmto assemble with the 3 up-to-date drives (sda3, clonedsdf3, සහsdd3) brings/dev/md99online in clean degraded mode ([U_UU]) using the exact latestEventsstate, whileddrescuealready rescued the data fromsdf3. - Activate LVM (
vgchange) if your NAS or server uses LVM on top of RAID (SynologySHR, QNAP, Proxmox, Ubuntu): (If your Synology NAS usesBtrfson LVM orSynology Hybrid RAID, you will now see/dev/vg1000/lvහෝ/dev/vg1/volume_1appear.) - Mount the filesystem in strict Read-Only mode and evacuate your critical files immediately:
mkdir -p /mnt/raid_rescue # For ext4: mount -o ro,noload /dev/md99 /mnt/raid_rescue # (Or if noload refuses due to dirty journal, mount the COW overlay normally so the journal replays into the disposable COW file) mount -o ro /dev/md99 /mnt/raid_rescue # For Btrfs (Synology NAS): mount -t btrfs -o ro,usebackuproot,nologreplay /dev/vg1000/lv /mnt/raid_rescue - Copy your most critical folders first using
rsyncto an external backup drive:
Phase 4: Rescuing Hardware RAID Controllers (Dell PERC, LSI MegaRAID, HP Smart Array, Adaptec) and ZFS / Btrfs Pools¶
What if your array was managed not by Linux mdadm, but by a Hardware RAID Controller Card (Dell PERC H730/H740/H755, Broadcom / LSI MegaRAID, හෝ HP Smart Array) that now reports Virtual Disk Offline හෝ Foreign Configuration Found after a second drive hiccupped during rebuild?
1. Hardware RAID Cards (Dell PERC, LSI MegaRAID, HP Smart Array)¶
- කිලික් කරන්න එපා
"Clear Foreign Configuration"හෝ"Initialize Virtual Disk": - In Dell
PERC/ LSIMegaRAIDBIOS (Ctrl+RහෝHIIUEFI Configuration Utility),Clearwipes the RAID metadata from the drives, andInitializeoverwrites the data stripes. - If a healthy drive briefly dropped offline due to a loose backplane cable or timeout,
Import Foreign Configurationattempts to bring the drive back in; however, if one drive dropped weeks ago and a second dropped today, importing both or forcing the stale drive online via the controller BIOS can corrupt the array. - The Safe Software Reconstruction Method for Hardware RAID (
DDFMetadata /dmraid/R-Studio/UFS Explorer): - Hardware RAID cards store standard
SNIA DDF(Disk Data Format) metadata in the final sectors at the end of each hard drive, and stripe data across the drives starting at sector0(or after a small header offset) using standard64 KB,128 KB,256 KB, හෝ512 KBchunk sizes (Left-AsymmetricහෝLeft-Symmetric) - Instead of gambling with a destructive hardware RAID card BIOS:
- Label all drives by slot number and connect them to a standard SATA / non-RAID
IT-ModeHBA on a recovery workstation. - Clone the bad-sector drive using
GNU ddrescueas shown in Phase 2. - Open the cloned drives in
UFS Explorer RAID RecoveryහෝR-Studio Technician, or assemble them read-only in Linux usingdmsetup/mdadm --build: these tools automatically detect theLSI/PERC/HPstripe order, chunk size (64K/512K), and parity rotation in 100 percent read-only memory without ever letting the hardware RAID card write a single byte to your drives.
- Label all drives by slot number and connect them to a standard SATA / non-RAID
2. TrueNAS / ZFS (RAID-Z1 / RAID-Z2) and Synology Btrfs Scrub/Resilver Failures¶
If your pool uses OpenZFS (RAID-Z1 හෝ RAID-Z2) on TrueNAS or Proxmox and a second drive throws CKSUM / READ errors during a zpool replace resilver:
1. Why ZFS handles bad sectors during resilvering better than hardware RAID, and when it still locks up:
- Unlike naive hardware RAID cards that drop an entire disk on a single bad sector, ZFS continues resilvering past localized READ errors and logs only the specific damaged file paths in zpool status -v.
- However, if the second failing drive hangs the SATA/SAS bus (I/O timeout / head thrashing) or drops completely offline (UNAVAIL / FAULTED), the pool enters SUSPENDED state (pool I/O is currently suspended)
2. How to rescue a SUSPENDED හෝ FAULTED ZFS RAID-Z1 / RAID-Z2 pool:
- Power down cleanly, clone the failing drive onto a healthy replacement disk using GNU ddrescue -f -n (ZFS partition labels include the exact vdev GUID at the start and end of the disk, so a raw sector clone is automatically recognized by ZFS as the original vdev member).
- Disconnect the physically dying drive, leave the ddrescue clone connected, and import the ZFS pool in strict Read-Only mode (readonly=on) without starting a write transaction or resuming the resilver:
-o readonly=on prevents ZFS from writing new uberblocks or attempting background healing writes, allowing you to copy every dataset off safely via zfs send හෝ rsync before rebuilding the pool from scratch.
නිතර අසන ප්රශ්න¶
Why did you say NEVER to run CHKDSK හෝ fsck when my RAID array mounts with filesystem errors after a drive drop?¶
On a single standalone hard drive, running CHKDSK හෝ fsck repairs minor directory inconsistencies. On a multi-drive RAID 5 or RAID 6 array that suffered a double-drive drop or interrupted rebuild, filesystem errors are almost always caused by RAID stripe misalignment or a stale member drive being forced online (meaning every 3rd හෝ 4th 64 KB chunk of data is being read from the wrong drive generation). If you let CHKDSK /f හෝ fsck -y run on a misaligned or out-of-sync RAID array, it misinterprets millions of valid files as corrupted metadata, truncates directory trees, dumps unnamed fragments into lost+found, and writes new parity blocks across all drives, making it mathematically impossible to ever reassemble the correct drive combination afterward. Always mount via Read-Only Copy-on-Write (COW) overlays first.
What if the partially rebuilt replacement drive (42% complete) AND the second drive that failed at 42% BOTH have needed data? Can I combine them?¶
Yes. This is a classic "Split-Horizon / Hybrid Clone" RAID rescue. Suppose Drive 2 died completely on Monday, you put in New Drive 2, and the rebuild made it to 42% (LBA 3,500,000,000) before Drive 3 suffered a fatal head crash at LBA 3,500,000,000. Notice what you actually have: for the first 42% of the array (0% දක්වා 42%), New Drive 2 has already been 100 percent reconstructed. So for the first 42% of the LBA range, you have a complete set of stripes using Drive 1 + New Drive 2 + Drive 4 (treating crashed Drive 3 as the missing disk). And if Drive 3 only had a localized bad-sector patch at 42% (and is readable from 43% දක්වා 100% via ddrescue), you use Drive 1 + Cloned Drive 3 + Drive 4 for the remaining 58% of the array.
Why do second drives fail so frequently right in the middle of a RAID 5 rebuild?¶
Two engineering realities combine to make RAID 5 rebuilds notorious for double-drive failures: (1) Identical Batch Aging: Most users buy 4 to 8 identical hard drives from the same retailer on the same day (same firmware, same factory lot, same spindle motor batch) and run them side-by-side at the exact same temperature for 35,000 hours; when Drive 1 wears out, Drive 2 and Drive 3 are days away from the same mechanical wear threshold. (2) Latent Uncorrectable Read Errors (UREs): Under normal daily operation, 95% of a hard drive's cold archival sectors are not read for months at a time, so latent bad sectors sit undiscovered. A RAID 5 rebuild forces a continuous 100% surface read of every single sector across every surviving drive (40 to 100 terabytes of continuous reading), immediately exposing any latent bad sector that accumulated over the past year.
How do I know which files were affected by the few unreadable bad sectors that ddrescue had to zero-fill?¶
GNU ddrescue නමින් ගොඩනඟන ලද අධිකරණ වෛද්ය උපයෝගීතාවයක් ඇතුළත් වේ ddrescuelog සහ ddru_ntfsfindbad / දෝශ නිරාකරණය එය ඔබගේ සිට නිවැරදි සොයා නොගත් අංශයේ LBA පරාසයන් සිතියම්ගත කරයි .සිතියම ගොනුව (status = '-' නරක අංශ) RAID තීරු ගණිතය හරහා ගොනු පද්ධති ඉනෝඩ වගුව වෙත කෙලින්ම. කුමක්දැයි ඔබ දැනගත් පසු 4 කි.බ blocks were zero-filled during cloning, you can query දෝශ නිරාකරණය (icheck සහ ncheck මත ext4) හෝ btrfs inspect-internal logical-resolve to print out the exact list of 2 or 3 filenames that overlapped those bad sectors, confirming with 100 percent mathematical certainty that every other file on your multi-terabyte server is completely uncorrupted.
අදාළ මාර්ගෝපදේශ¶
- RAID 5 or RAID 6 Synology, QNAP, or TrueNAS multiple drive drop and why forcing a rebuild destroys data
- RAID controller card death, foreign configuration import, and why you must never initialize disks
- NAS RAID array degraded, multiple drive failure, and ZFS/Btrfs recovery
- දෘඪ තැටි අසමත් වීම සහ දත්ත ප්රතිසාධනය
- Computer backups: 3-2-1 strategy and disaster recovery
- රට අනුව හදිසි අංක
මූලාශ්ර සහ නිල මාර්ගෝපදේශ¶
- Linux Raid Wiki (
raid.wiki.kernel.org/index.php/Linux_Raid&Recovering_a_failed_software_RAID): Official KernelmdadmRecovery Protocol, Device-Mapper Copy-on-Write (dm-snapshot) Overlays, andEventsCounter Reconciliation. - GNU
ddrescueOfficial Engineering Manual (gnu.org/software/ddrescue/manual/ddrescue_manual.html): Multi-Phase Non-Scraping (-n) and Direct Access (-d) Sector Recovery Algorithms for Failing Magnetic Media. - Synology Official Knowledge Base (
kb.synology.com/en-global/DSM/tutorial/How_can_I_recover_data_from_my_DiskStation_using_a_PC): Mountingmdadm+LVM2+Btrfs/ext4NAS Volumes Read-Only on Ubuntu Rescue Systems. - Storage Networking Industry Association (
SNIA): Common RAID Disk Data Format (DDF) Specification for Hardware RAID Controller Metadata Portability.