Skip to content

Docker kapena Kubernetes Production Volume Yachotsedwa: Kupulumutsa docker system prune --volumes, overlay2 Inodes, ndi etcd Zithunzi

Manambala angozi amasiyana malinga ndi dziko. Tsambali limagwiritsa ntchito 911, nambala ya United States ndi Canada. Mayiko ena ambiri amagwiritsa ntchito 112, 999, kapena nambala ina. Pezani nambala ya dziko lanu.

Kuthamanga docker compose down -v, docker system prune -a --volumes, docker volume rm, kapena kubectl delete pvc pa tabu yolakwika yopangira ndi imodzi mwa njira zofulumira kwambiri zofufutira zamoyo za PostgreSQL, MySQL, MariaDB, Redis, MongoDB, kapena SQLite database. Mosiyana ndi zithunzi zamakina amtambo omwe amapita ku bin yamasiku 7, woyendetsa voliyumu waku Docker (/var/lib/docker/volumes/<name>/_data) ndi Kubernetes local-path / hostPath operekera zakudya nthawi yomweyo amapereka POSIX yokhazikika unlink() ndi rmdir() imayimba mwachindunji motsutsana ndi maziko a Linux host filesystem (ext4, XFS, kapena Btrfs). Kulemba kulikonse komwe kumachitika pa Linux host host lamulo likatha limatha kutsitsa midadada yomasulidwa pa disk. Ngati seva yanu itadzaza mpaka 100 peresenti (0 bytes free) munthu asanathawe docker system prune --volumes ndi mantha, funsani seva disk 100% yodzaza, database ya WAL log bloat, ndi kuchira kwa zero; pakuchotsa mafayilo opanda zitsulo a Linux kunja kwa zotengera, onani mwangozi rm -rf kapena Shift-Delete pa ext4, XFS, APFS, ndi NTFS: kuwerengera kokha ndi kujambula kwa inode ndi makina pafupifupi VMDK, VHDX, ndi qcow2 chithunzithunzi unyolo ziphuphu ndi hypervisor kupulumutsa. Ngati kutha kwa kupangaku kumakhudza chisamaliro cha odwala kuchipatala, kutumiza mwadzidzidzi, kapena njira zotetezera anthu, imbani nambala yanu yadzidzidzi yakuderalo (911 ku United States ndi Canada, 112 ku European Union, 999 ku United Kingdom) kapena onani chikwatu chathu cha manambala angozi.

Chitani izi tsopano

  1. OSATI kuthamanga docker compose up -d, systemctl restart docker, kapena tumizaninso Kubernetes pod: Kubwezeretsanso chidebe kapena kuyambitsanso daemon ya Docker kumalemba ma megabytes atsopano. overlay2 metadata, zipika za JSON, ndi mafayilo opanda kanthu omwe angoyamba kumene (initdb) molunjika pa zenizeni ext4 kapena XFS midadada pomwe voliyumu yanu yochotsedwa idakhala masekondi apitawo.
  2. Chongani masekondi 5 ngati chidebe ndondomeko (postgres, mysqld, redis-server, node, python) ikugwirabe ntchito mu kukumbukira (ps aux ndi sudo lsof +L1): Ngati wina wachotsa mafayilo mkati mwa voliyumu yokwezedwa kapena kulumikiza voliyumu yosadziwika pomwe chidebecho chikugwirabe ntchito, Linux kernel imasunga 100 peresenti ya mafayilo osalumikizidwa pa disk malinga ngati ID ya process ya chotengerayo (PID) imakhala ndi chofotokozera chotseguka (FD). Mutha kukopera mafayilowo mwachindunji kuchokera /proc/<PID>/fd/ kapena /proc/<PID>/root/ ndi ziro deta kutayika chidebecho chisanayambe.
  3. ** Ngati chidebecho chidayimitsidwa kale ndipo voliyumu idachotsedwa (docker compose down -v kapena docker system prune --volumes), amaundana disk host imalemba nthawi yomweyo**: Imitsa daemon ya Docker (sudo systemctl stop docker.socket docker.service containerd) kotero zotengera zakumbuyo ndi ma healthchecks amasiya kulemba zipika /var/lib/docker, ndiye mwina onjezerani /var/lib/docker (kapena mizu / kugawa) kuwerenga kokha (sudo mount -o remount,ro /) kapena tengani chithunzithunzi chambiri chamtambo (AWS EBS Snapshot, GCP Persistent Disk Snapshot, DigitalOcean Volume Snapshot) musanagwire kalikonse.
  4. Ngati muli pa Kubernetes (kubectl delete pvc kapena kubectl delete namespace), onani PersistentVolume (PV) ReclaimPolicy (kubectl get pv) nthawi yomweyo:
  5. Ngati ndi PV anali persistentVolumeReclaimPolicy: Retain, deta yanu sachotsedwa: the PV anangosamukira ku Released mawonekedwe ndi diski yanu yamtambo (EBS, Azure Disk, Ceph RBD, NFS) ndi 100 peresenti yokha.
  6. Ngakhale kubectl delete pvc ali mu Terminating, Kubernetes Omaliza (kubernetes.io/pvc-protection) letsani dalaivala wa CSI kuti achotse zosungira zomwe zili pansi bola ngati Pod iliyonse imatchulanso PVC. Osachotsa Pod mpaka mutakopera deta kuchokera pa Pod yomwe ikuyenda.
  7. Fananizani gawo loyambira (dd / ddrescue pa SSH) kupita ku disk yakunja kapena voliyumu yosiyana yamtambo musanayendetse ext4magic, extundelete, kapena PhotoRec: Osathamangitsa osema mafayilo omwe amalemba mafayilo otulutsa pagawo lomwelo la disk lomwe mukuchira /var/lib/docker/volumes.
  8. Zowonongeka kapena kufufutidwa mwangozi gulu la gulu la Kubernetes (kubectl delete crd / namespace), kusiya kube-apiserver ndi kubwezeretsa zatsopano etcd chithunzithunzi (etcdctl snapshot restore): Ngati masango ali mkati etcd idafufutidwa pomwe ma voliyumu osungira mitambo akadalipo, kubwezeretsedwa etcd amachira onse PV, PVC, Secret,ndi ConfigMap zinthu nthawi imodzi.

Danger

Osachotsa kubernetes.io/pvc-protection kapena kubernetes.io/pv-protection finalizer ndi kubectl patch ... -p '{"metadata":{"finalizers":null}}' pamene PVC inakanidwa Terminating: Chomaliza ndicho chinthu chokhacho chomwe chikusunga dalaivala wanu wamtambo wa CSI (aws-ebs-csi-driver, pd.csi.storage.gke.io, Longhorn, Rook-Ceph) kuchokera popereka hardware DeleteVolume Kuyimba kwa API kuti muwononge disk yanu yopanga. Pomwe PVC imakhazikika Terminating, Pod ikugwirabe ntchito ndipo voliyumu idakalipo: thamanga kubectl exec kapena kubectl cp (kapena pg_dump / mysqldump) nthawi yomweyo kupulumutsa 100 peresenti ya database yanu Pod isanathe.


Mtengo wosankha: voliyumu ya Docker yachotsedwa, overlay2 wosanjikiza, kapena Kubernetes PVC triage

Gwiritsani ntchito tchatichi kuti musankhe njira yochira yotaya ziro yothamanga kwambiri potengera ngati chidebecho chikadalipobe, kaya voliyumuyo idathandizidwa ndi diski yakumaloko kapena kusungirako kwamtambo CSI, komanso mafayilo omwe amasunga. /var/lib/docker:

flowchart TD
    Start["Zambiri Zopanga Zikusowa Pambuyo pa Docker kapena Kubernetes Chotsani Lamulo"] --> WhichEnv{"Ndi Platform & Command iti yomwe idayambitsa Kutayika?"}

    WhichEnv -->|"Docker: Container Ikugwirabe Ntchito (Mafayilo Osalumikizidwa Mkati Mwa Volume / Chidebe)"| ProcRescue["Pulumutsani Mafotokozedwe A Fayilo Osalumikizidwa kudzera pa sudo lsof +L1 ndi /proc/PID/fd/N kapena /proc/PID/root/"]

    WhichEnv -->|"Docker: Chidebe Choyimitsidwa & Voliyumu Yachotsedwa (lemba pansi -v / prune --volumes)"| StopDocker["Thamangani Nthawi yomweyo: sudo systemctl siyani docker.socket docker yosungidwa"]

    WhichEnv -->|"Kubernetes: kubectl chotsani pvc / pv / statefulset / namespace"| K8sCheck{"Kodi Pod Ikugwirabe Ntchito Kapena PVC Imayima Pomaliza?"}

    K8sCheck -->|"Inde: Pod Ikuyendabe (Finalizer Kutsekereza CSI Chotsani)"| K8sExec["OSAFUTA Pod -> Thamangani kubectl exec / pg_dump / tar pa stdout kupita ku Local Disk Nthawi yomweyo"]
    K8sCheck -->|"Ayi: Pod Gone -> Onani kubectl kupeza pv"| ReclaimCheck{"Kodi PV ReclaimPolicy inali chiyani?"}

    ReclaimCheck -->|"Sungani (Mkhalidwe wa PV = Watulutsidwa)"| ClearClaimRef["100% Data Intact -> Sinthani PV kuti Chotsani spec.claimRef.uid -> Kumanganso ku PVC Yatsopano"]
    ReclaimCheck -->|"Chotsani (Mtambo EBS / PD / Longhorn / Local-Path)"| CloudOrLocal{"Cloud CSI Volume kapena Local Host Disk?"}

    CloudOrLocal -->|"Cloud CSI (AWS EBS / GCP / Azure / Ceph)"| CloudSnap["Onani Cloud Console Recycle Bin / Automated EBS Snapshots / Ceph RBD Zinyalala (rbd zinyalala ls)"]
    CloudOrLocal -->|"Local Node Disk (njira yakomweko / hostPath)"| StopDocker

    StopDocker --> RemountRO["Remount Host Partition Read-Only (mount -o remount,ro) KAPENA Stream Block Image pa SSH"]
    RemountRO --> HostFS{"Kodi Filesystem Hosts /var/lib/docker ndi chiyani?"}
    HostFS -->|"ext4"| Ext4Rec["Gwiritsani ntchito debugfs / ext4magic Journal Inode Transaction Rollback (-chidindo chanthawi)"]
    HostFS -->|"XFS / Btrfs"| CarveRec["Gwiritsani Ntchito Mizu ya Btrfs / xfs_irecover / Siginecha Yamakonda Kujambula kwa SQLite, Postgres Heap, kapena InnoDB .ibd"]

Gawo 1: Momwe Docker ndi Kubernetes amachotseratu ma voliyumu pa disk

Musanagwiritse ntchito lamulo lililonse lobwezeretsa, mvetsetsani komwe deta yanu imakhala pa Linux host host ndi zomwe lamulo lochotsa lidakhudza:

Mtundu Wosungira & Chotsani Lamulo Host Filesystem Path pa Linux Node Zomwe Delete Command Idachitadi Mwamsanga Kuchira Njira
** Wotchedwa Docker Volume (docker volume rm kapena docker compose down -v)** /var/lib/docker/volumes/<vol_name>/_data Chotsani mafayilo mkati _data/ ndikuchotsa zolemba zachikwatu kuchokera /var/lib/docker/volumes/metadata.db (boltdb). Imani Docker nthawi yomweyo, onjezerani /var/lib/docker werengani-pokha, ndikubwezeretsanso ma innode osalumikizidwa kudzera ext4magic / debugfs kapena kujambula siginecha.
Volume yosadziwika ya Docker (docker run -v /var/lib/mysql + docker rm -v kapena docker system prune --volumes) /var/lib/docker/volumes/<64-char-hex-id>/_data docker system prune --volumes imachotsa voliyumu iliyonse yosadziwika ndi dzina lomwe silinatchulidwe ndi chidebe chimodzi chomwe chilipo. Imitsani Docker nthawi yomweyo; mukadakhala ndi **zina ** zoyimitsa zotengera zomwe mavoliyumu ake *sanadulidwe, fufuzani zotsalira /var/lib/docker/volumes/*/_data zikwatu poyamba asanasema midadada yosagawika.
** Chigawo Cholembedwa (docker rm <container> pamene deta inalembedwa kunja kwa voliyumu)** /var/lib/docker/overlay2/<layer-id>/diff/ Chotsani kumtunda kwa chidebecho diff/ directory mkati mwa overlay2 dalaivala yosungirako. Imitsani Docker nthawi yomweyo ndikubwezeretsanso mtengo wosalumikizidwa kuchokera ku fayilo ya ext4 magazini kapena XFS midadada.
** Docker Desktop pa macOS kapena Windows (docker system prune --volumes)** Mkati mwa chithunzi cha disk cha makina amodzi: -/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw (macOS) kapena %LocalAppData%\Docker\wsl\disk\docker_data.vhdx (Windows WSL2). Watumizidwa unlink() mkati mwa Linux VM, ndipo ngati discard/TRIM ikugwira ntchito Docker.raw / docker_data.vhdx, anabowola mabowo ochepa mufayilo yosungira. Siyani Docker Desktop mkati mwa masekondi 5, koperani Docker.raw kapena docker_data.vhdx kupita kugalimoto yakunja, ndikuthamanga ext4magic / PhotoRec motsutsana ndi chithunzi cha disk.
**Kubernetes PVC yokhala ndi Delete ReclaimPolicy pa Ceph RBD (Rook-Ceph) kapena Longhorn ** Ceph OSD dziwe (rbd) kapena /var/lib/longhorn/replicas/. Ngati csi-rbdplugin ali trash loledwa (rbd trash), chithunzi cha RBD chimasunthidwa ku bin ya zinyalala ya Ceph; apo ayi midadada ilibe mapu. Thamangani rbd trash ls --pool <pool> mkati mwa rook-ceph-tools pod ndi kubwezeretsa ndi rbd trash restore.

Gawo 2: Kupulumutsa mafayilo otseguka a database kuchokera pachidebe chomwe chikuyendabe (/proc/<pid>/fd/)

Dikirani: kontena ikugwirabe ntchito pompano? Nthawi zambiri, mainjiniya amayendetsa script yosamukira, ntchito yoyeretsa, kapena buku rm -rf /var/lib/postgresql/data/* kapena rm /app/data/production.sqlite3 mkati mwa chidebe chothamanga, kapena kuchotsa chikwatu chomwe chili pansi pa host bind-mount /opt/app/data pamene chotengeracho (postgres, mysqld, mariadbd, redis-server, mongod, gunicorn, node)idakalipobe.

Mu Linux, fayilo sichichotsedwa pa disk pamene rm kapena unlink() amatchedwa. Ma disk ake amatchinga ndi inode amakhalabe ** 100 peresenti yogawidwa ndi yowerengeka ** mpaka ndondomeko yomaliza yokhala ndi chofotokozera chotseguka (FD) ku inodeyo amatseka kapena kutuluka.

Gawo 1: Pezani ID process process (PID) ya ndondomeko ya chidebe

Pa Linux host node (osati mkati mwa chidebe), pezani wolandirayo PID za database yanu kapena njira yofunsira:

sudo lsof +L1 | grep -E "postgres|mysqld|mariadbd|sqlite|redis|mongo|node|python|java"
Onani zotsatira zake:
COMMAND     PID   USER   FD   TYPE DEVICE  SIZE/OFF NLINK    NODE NAME
mysqld    18492    999   14u   REG  259,1 482344960     0 1849201 /var/lib/mysql/app_prod/users.ibd (deleted)
mysqld    18492    999   15u   REG  259,1 125829120     0 1849202 /var/lib/mysql/app_prod/orders.ibd (deleted)
mysqld    18492    999   16u   REG  259,1  79691776     0 1849190 /var/lib/mysql/ibdata1 (deleted)
python3   22104   1000    4u   REG  259,1  89128960     0 3102944 /data/production.sqlite3 (deleted)
Yang'anani pagawo NLINK: akuti 0, ndipo dzina lafayilo limatha ndi (deleted). Fayilo yachoka kwathunthu ls /var/lib/docker/volumes/..., komabe bayiti iliyonse (482 MB mu users.ibd, 89 MB mu production.sqlite3) akadali ndi moyo mkati /proc/<PID>/fd/<FD>.

Khwerero 2: Koperani mafayilo a database osalumikizidwa mwachindunji /proc/<PID>/fd/

Osa** kuyimitsa kapena kuyambitsanso chidebecho. Pamene ndondomekoyi ikugwirabe ntchito, lembani zofotokozera za fayilo yaiwisi mwachindunji ku foda yotetezeka pa omwe akukulandirani:

sudo mkdir -p /root/proc_emergency_rescue

# Example 1: Rescuing an unlinked SQLite3 database (PID 22104, FD 4)
sudo cp /proc/22104/fd/4 /root/proc_emergency_rescue/production.sqlite3
# Check if PID 22104 also holds the -wal file open (e.g. FD 5) and copy it too:
sudo cp /proc/22104/fd/5 /root/proc_emergency_rescue/production.sqlite3-wal

# Example 2: Automatically rescuing ALL open (deleted) MySQL/Postgres files for PID 18492
for fd in /proc/18492/fd/*; do
    target=$(sudo readlink "$fd" 2>/dev/null)
    if echo "$target" | grep -q "(deleted)"; then
        fd_num=$(basename "$fd")
        clean_name=$(echo "$target" | sed 's/ (deleted)//; s/\//_/g')
        echo "Rescuing FD $fd_num -> $clean_name"
        sudo cp "$fd" "/root/proc_emergency_rescue/${fd_num}${clean_name}"
    fi
done

Khwerero 3: Zabwinonso: Yambitsani dambo lomveka bwino (pg_dump / mysqldump / .backup) chidebecho chisanatseke ma FD

Ngati postgres kapena mysqld ikugwirabe ntchito ndi mafayilo ake a data otsegulidwa mu kukumbukira ndi /proc/<PID>/fd/, yesani kuthamangitsa kutayira komwe kulipo pa unix socket kapena kulumikizana kwa TCP kaye:

# If the database engine can still read its open file descriptors:
docker exec <container_name> pg_dumpall -U postgres > /root/proc_emergency_rescue/live_dump.sql
# Or for MySQL:
docker exec <container_name> mysqldump -u root -p --all-databases --single-transaction > /root/proc_emergency_rescue/live_mysql.sql
*(Zindikirani: PostgreSQL imatsegula ndikutseka .1, .2 magawo amafayilo pamachitidwe a ogwira ntchito mwachangu, kotero pg_dump akhoza kulephera pa matebulo osasungidwa; kuphatikiza funso lamoyo la SQL ndikukopera kutsegulidwa kulikonse /proc/<pid>/fd/* kudutsa *onse ** postgres ma PID am'mbuyo a ana amasunga mulu wa data womwe ungatheke.)


Gawo 3: Kubwezeretsanso voliyumu ya Docker yochotsedwa ext4 kapena XFS chidebecho chitayima

Bwanji ngati mutathamanga docker compose down -v kapena docker system prune --volumes, kotero kuti chidebecho chayimitsidwa kale ** ndi zonse /proc/<pid>/fd/ zofotokozera zatsekedwa? Tsopano muli pa mpikisano wotsutsana ndi disk yakumbuyo ikulemba pa Linux host host.

Khwerero 1: Mangani ma fayilo a Linux mkati mwa masekondi 10

  1. **Imitsani Docker, yosungidwa, kubelet, ndi cron nthawi yomweyo ** kotero palibe chidebe, thanzi, kapena chipika chozungulira chomwe chimalemba china chake /var/lib/docker:
    sudo systemctl stop docker.socket docker.service containerd.service cron.service
    
  2. ** Onani chida chotchinga ndi fayilo yomwe ili nayo /var/lib/docker**:
    df -Th /var/lib/docker
    
    Filesystem     Type  Size  Used Avail Use% Mounted on
    /dev/nvme0n1p1 ext4  100G   42G   58G  42% /
    
  3. Kwezaninso magawowa Kuwerenga-Only (kapena tsitsani chithunzi chogawa pa SSH kumakina ena):
  4. Ngati /var/lib/docker ili pagawo lake lodzipatulira kapena voliyumu ya LVM (mwachitsanzo /dev/nvme0n1p2 kapena /dev/mapper/vg0-docker), onjezerani kuti muwerenge-pokhapokha:
    sudo mount -o remount,ro /var/lib/docker
    
  5. Ngati /var/lib/docker ali pa muzu / kugawa (/), yesani kukakamiza magawo a mizu kuti awerenge-pokha pogwiritsa ntchito kernel Magic SysRq trigger (echo u | sudo tee /proc/sysrq-trigger) kapena sudo mount -o remount,ro /.
  6. Ngati muli pamtambo VPS (AWS EC2, GCP, DigitalOcean, Hetzner, Linode): Nthawi yomweyo pitani kumtambo wanu wapaintaneti ndikudina Pangani Chithunzithunzi pa block voliyumu ya seva (kapena tsitsani chipangizo chopangira SSH kupita ku seva yachiwiri: sudo dd if=/dev/nvme0n1p1 bs=4M status=progress | gzip -1 | ssh user@backup-server "cat > /mnt/storage/docker_host_raw.img.gz").

Khwerero 2: Onani ngati deta yanu ikadali mu voliyumu INA yosadulidwe kapena overlay2 wosanjikiza

Musanajambule midadada yosagawika, onetsetsani kuti nkhokwe yanu sinasungidwe pamtengo kapena voliyumu yodziwika bwino yomwe idapulumuka:

# Search the entire /var/lib/docker tree read-only for any surviving database files:
sudo find /var/lib/docker -name "PG_VERSION" -o -name "ibdata1" -o -name "*.sqlite*" -o -name "*.db" -o -name "dump.rdb" -o -name "WiredTiger" 2>/dev/null
Sabata iliyonse, oyang'anira amachita mantha akathamanga docker volume rm app_db_data, kungozindikira ndi find /var/lib/docker kuti docker-compose.yml anali atalengadi projectname_app_db_data (yomwe ikadali 100 peresenti yokha pa disk) pamene app_db_data inali voliyumu yakale yosagwiritsidwa ntchito kuyambira miyezi isanu ndi umodzi yapitayo.

Khwerero 3: Kubwezeretsanso maulalo osalumikizidwa a Docker pa ext4 pogwiritsa ntchito ext4 Magazini (ext4magic ndi debugfs)

Pamene docker volume rm zochotsa /var/lib/docker/volumes/<vol_name>/_data pa ext4 file system, ext4 ziro pazilolezo za Direct block (i_block / muzu wamtengo) patebulo lokhalamo. Komabe, ext4 magazini (inode <8>, kawirikawiri 128 MB ku 1024 MB ring ring buffer) ikadali ndi zolemba zakale zamatebulo enieni a innode ndi zikwatu zachikwatu kuyambira kale. docker compose down -v anaphedwa.

+-------------------------------------------------------------------------+
| HOW ext4magic RECOVERS A DELETED DOCKER VOLUME FROM THE EXT4 JOURNAL    |
|                                                                         |
| Live Inode Table on Disk (After docker volume rm):                      |
|   Inode #918402 (users.ibd): [Size = 0, Extent Pointers = ZEROED]       |
|                                                                         |
| Ext4 Journal Ring Buffer (Inode <8> on Disk):                           |
|   Transaction #41902 (10 mins ago):                                     |
|   Inode #918402 (users.ibd): [Size = 482MB, Extents = Blocks 9000..9999]|
|                                                                         |
| -> ext4magic reads the old Inode Copy from Journal Transaction #41902   |
|    and extracts Blocks 9000..9999 directly to your rescue drive.        |
+-------------------------------------------------------------------------+

Thamangani ext4magic motsutsana ndi magawo anu owerengera okha (kapena .img chithunzithunzi pa seva yachiwiri), kufotokoza nthawi (-a) kuchokera **5 mphindi musanathamangire ** mudathamanga docker compose down -v kapena docker system prune --volumes:

# Install ext4magic on your rescue machine (never on the affected partition)
sudo apt-get install -y ext4magic e2fsprogs

# Dump the ext4 journal from the frozen partition / image to a file in RAM (/dev/shm)
sudo debugfs -R "dump <8> /dev/shm/ext4_journal.copy" /dev/nvme0n1p1

# Recover all files deleted under var/lib/docker/volumes after (Current Time - 15 minutes)
sudo mkdir -p /mnt/external_rescue/docker_recovered
sudo ext4magic /dev/nvme0n1p1 \
  -j /dev/shm/ext4_journal.copy \
  -f var/lib/docker/volumes \
  -a $(date -d "-15 minutes" +%s) \
  -r -d /mnt/external_rescue/docker_recovered
- -j /dev/shm/ext4_journal.copy: Amawerenga chithunzithunzi cha magazini omwe mudasunga mu RAM mutangoyimitsa Docker. - -f var/lib/docker/volumes: Imatsata mtengo weniweni wa chikwatu cha Docker (chidziwitso: chotsani kutsogolera / mu ext4magic njira zosiyanasiyana). - -a $(date -d "-15 minutes" +%s): Amanena ext4magic kuti amangenso chikwatu cholembera pogwiritsa ntchito makope a inode ovomerezeka mphindi 15 zapitazo (kungochotsa mwangozi). - -r: Imabwezeretsanso maulalo athunthu pomwe palibe midadada idalembedwa; ngati -r amaphonya owona overwritten pang'ono, kuthamanga yachiwiri pass m'malo -r ndi -m (zojambula zamatsenga + hybrid mode).

Khwerero 4: Kujambula mafayilo a SQLite, PostgreSQL, kapena MySQL InnoDB yaiwisi pamene magazini yatsekedwa (XFS kapena ext4)

Warning

**Chifukwa XFS sichikuthandizira ext4magic ndipo imafuna kutsika pompopompo powerenga-pokha **: Mosiyana ext4, yomwe imalemba zolemba zonse za 256-byte inode mumagazini yake, XFS logs metadata intent deltas ndikugwiritsanso ntchito mwaukali mitengo ya B-inode yomasulidwa (agi). Ngati /var/lib/docker akukhala pa XFS magawo (osakhazikika pa RHEL, Rocky Linux, AlmaLinux, ndi Amazon Linux), akutsika /var/lib/docker kapena kukweranso / kuwerenga-pokha masekondi ndi sitepe imodzi yofunika kwambiri pamaso kusema.

Ngati wolandirayo akugwiritsa ntchito XFS kapena ngati ext4 magazini yomwe yakulungidwa kale pazolozera za inode, gwiritsani ntchito siginecha ndi mawonekedwe ojambulidwa motsutsana ndi chithunzi cha disk yaiwisi (docker_host_raw.img):

  1. Kwa SQLite3 Databases (.sqlite, .db - mwachitsanzo, Vaultwarden, Nextcloud, Wothandizira Pakhomo, Gitea, Grafana): Dongosolo lililonse la SQLite3 limayamba ndi mutu wodziwika bwino wa 16-byte ASCII pa byte 0 patsamba lake loyamba la 4 kilobyte: 53 51 4c 69 74 65 20 66 6f 72 6d 61 74 20 33 00 ("SQLite format 3\0"), ndikusunga kukula kwake kwamasamba (Bytes 16..17, kawirikawiri 4096) ndi kukula kwa database yonse m'masamba (Bytes 28..31, big-endian 32-bit integer) mkati mwa mutu womwewo.
  2. Thamangani PhotoRec ndi [File Opt] -> [x] sqlite yambitsa, kapena jambulani gawolo mwachindunji ndi grep -ab "SQLite format 3" kuti mupeze zenizeni zenizeni za fayilo yanu ya SQLite yochotsedwa ndikuyichotsa nayo dd:
    # Find the exact byte offsets of every "SQLite format 3" header on the partition:
    sudo grep -ab "SQLite format 3" /dev/nvme0n1p1
    
    Akachotsa, thamangani sqlite3 carved.db ".recover" | sqlite3 repaired.db kuti amangenso tsamba lililonse latsamba la B-mtengo.

Tip

** Momwe mungawerengere zenizeni dd kukula kwa fayilo ya SQLite3 yosalumikizidwa kuchokera pamutu wake **: Kamodzi grep -ab "SQLite format 3" zimakupatsani mwayi woyambira OFFSET pa database yanu yochotsedwa ya SQLite, werengani ma byte 16..17 (uint16 big-endian Tsamba Kukula, mwachitsanzo, 0x1000 = 4096 mabayiti) ndi ma byte 28..31 (Uint32 big-endian Page Count, mwachitsanzo, 25000 masamba) pogwiritsa ntchito hexdump -C -s OFFSET -n 32. Kuchulukitsa Page Size x Page Count zimakupatsirani kutalika kwa baiti kuti mudutseko dd (iflag=skip_bytes,count_bytes) pamene fayiloyo ikugwirizana ndi disk.

  1. Kwa MySQL / MariaDB InnoDB (.ibd table owona): InnoDB iliyonse .ibd table imakhala ndi sequential 16 KB (16,384 byte) masamba omwe ali ndi FIL_PAGE_INDEX (0x45BF) mutu ndi tebulo lapadera space_id. Gwiritsani ntchito Percona / TwinDB undrop-for-innodb (stream_parser + c_parser) zida kuti sikani /dev/nvme0n1p1 kwa onse 16 KB Masamba a index ya InnoDB ndikutaya mzere uliwonse mwachindunji mu mafayilo a TSV/SQL kuti mulowetsenso.
  2. Kwa Mafayilo a PostgreSQL Mululu ($PGDATA/base/<db_oid>/<relfilenode>): PostgreSQL imasunga tebulo lililonse ngati lolumikizana 8 KB (8,192 byte) mulu wa masamba omwe ali PageHeaderData (pd_lower, pd_upper, pd_special). Gwiritsani pg_filedump kapena pglazarus / pdu kusanthula madera osagawika a disk 8 KB masamba ofananira ndi magawo a tebulo lanu (HeapTupleHeaderData) ndikumanganso mizere.

Gawo 4: Kubernetes PVC, PV, ndi etcd Chithunzithunzi cha Emergency Recovery

Ku Kubernetes (EKS, GKE, AKS, k3s, RKE2, kubeadm, OpenShift), kuchotsa mwangozi PersistentVolumeClaim (kubectl delete pvc), StatefulSet, kapena Namespace yonse (kubectl delete ns prod) amatsata zomangamanga zosiyana kwambiri kuposa Docker yoyima.

Chitsanzo A: Mawonekedwe a PVC Terminating (The Finalizer Lifesaver)

Thamangani:

kubectl get pvc -n <namespace>
Ngati ndi STATUS ndime imati Terminating, pumani m'malo: chidziwitso chanu SICHINAchotsedwe panobe. - Chifukwa chiyani? Kubernetes PVC iliyonse yopangidwa Popeza Kubernetes 1.9 imaphatikizansopo finalizers: [kubernetes.io/pvc-protection]. - Malingana ngati Pod ikukwerabe PVC, wolamulira wa Kubernetes amakana kumaliza kuchotsedwa kwa PVC ndipo sayitana oyendetsa CSI. DeleteVolume RPC. - Zochita Mwamsanga: Chitani OSATI kufufuta kapena kuyambitsanso Pod, ndipo chitani OSATI tulutsani chomaliza. Koperani nthawi yomweyo zonse zomwe zili mu Pod yomwe ikuyenda kumakina anu am'deralo kapena ndowa ya S3:
# Stream a compressed tar backup or pg_dump out of the still-running pod immediately:
kubectl exec -n <namespace> <pod-name> -- pg_dumpall -U postgres > ./rescued_k8s_db.sql
# Or tar an entire mounted volume directory out over stdout:
kubectl exec -n <namespace> <pod-name> -- tar czf - /var/lib/postgresql/data > ./rescued_pvc_files.tar.gz

Chitsanzo B: The PVC inachotsedwa, ndi kubectl get pv ikuwonetsa PersistentVolume mu Released dziko (ReclaimPolicy: Retain)

Thamangani:

kubectl get pv
Ngati PersistentVolume yanu (pv-prod-postgres) adalembedwabe ndi STATUS: Released (chifukwa StorageClass yake imagwiritsidwa ntchito reclaimPolicy: Retain), **100 peresenti ya deta yanu idakali yosakhudzidwa pa disk yomwe ili pansi **. Komabe, Kubernetes amakana kulola PVC yatsopano kumangiriza a Released PV chifukwa PV ikadali ndi a spec.claimRef kuloza ku PVC wakale zichotsedwa UID.

Momwe mungapangire a Released PV Available kachiwiri mumasekondi a 10 ndikutayika kwa ziro: 1. Sinthani chinthu cha PersistentVolume ndikuchotsa spec.claimRef chipika:

kubectl patch pv <your-pv-name> --type json -p '[{"op": "remove", "path": "/spec/claimRef"}]'
2. Thamangani kubectl get pv kachiwiri: mawonekedwe amasintha nthawi yomweyo kuchokera ku Released ku Available. 3. Panganinso yanu PersistentVolumeClaim ( kufotokoza volumeName: <your-pv-name> ndi kufananiza mphamvu/storageClassName). Idzamangiriza ku PV yomwe ilipo nthawi yomweyo ndi mafayilo onse osasinthika.

Chitsanzo C: Kubwezeretsanso gulu lopukuta la Kubernetes kuchokera ku etcd chithunzithunzi (etcdctl snapshot restore)

Bwanji ngati wina athamanga kubectl delete namespace prod kapena helm uninstall ndikufafaniza 200 Kubernetes kuwonetsera, Zinsinsi, ConfigMaps, ndi matanthauzo a PV pa kudziyendetsa nokha. kubeadm, k3s, kapena RKE2 ndege yoyendetsa?

  1. **Pezani zomwe zaposachedwa etcd chithunzi ** (mwachisawawa mu /var/lib/rancher/k3s/server/db/snapshots/, /var/lib/rancher/rke2/server/db/snapshots/, kapena zomwe mwapanga /var/backups/etcd/snapshot.db).
  2. **Tsimikizirani kukhulupirika kwa chithunzithunzi musanagwire gulu lamoyo **:
    ETCDCTL_API=3 etcdctl snapshot status /path/to/snapshot.db --write-out=table
    
  3. Imani kube-apiserver ndi etcd pa node (ma) ndege:
  4. Pa kubeadm (static pods): Sunthani /etc/kubernetes/manifests/kube-apiserver.yaml ndi /etc/kubernetes/manifests/etcd.yaml kwakanthawi kuti /root/manifests_hold/ choncho kubelet imayimitsa zotengera zonse ziwiri mwaukhondo, kenako ndikubwezeretsanso /var/lib/etcd ku /var/lib/etcd.bak.
  5. Pa k3s: Thamangani sudo systemctl stop k3s ndi sudo k3s server --cluster-reset --cluster-reset-restore-path=/path/to/snapshot.
  6. ** Bwezerani etcd data directory pa kubeadm**:
    sudo ETCDCTL_API=3 etcdctl snapshot restore /path/to/snapshot.db \
      --data-dir=/var/lib/etcd \
      --name=<control-plane-node-name> \
      --initial-cluster=<node-name>=https://<node-ip>:2380 \
      --initial-advertise-peer-urls=https://<node-ip>:2380
    
  7. Sunthani etcd.yaml ndi kube-apiserver.yaml kubwerera ku /etc/kubernetes/manifests/. Mkati masekondi 30, kube-apiserver zimalumikizananso ndi zobwezeretsedwa etcd sitolo ndi chilichonse chochotsedwa Namespace, Chinsinsi, ConfigMap, ndi PersistentVolume chinthu zimawonekeranso.

Gawo 5: Kulimbitsa Docker ndi Kubernetes voliyumu motsutsana ndi kuchotsedwa mwangozi

Nawonso database yanu yopangira ikabwezeretsedwa, khazikitsani zida zinayi zomangira kuti CLI typo imodzi isachotsenso voliyumu yomwe ilipo:

  • Sinthani Kubernetes iliyonse yopanga StorageClass kuchokera reclaimPolicy: Delete ku reclaimPolicy: Retain: Mwachikhazikitso, zomangidwa mumtambo StorageClasses (gp2, gp3, standard-rwo) kusakhazikika ku reclaimPolicy: Delete, kutanthauza kuti kuchotsa PVC kumawononga diski yamtambo. Pangani a StorageClass ndi reclaimPolicy: Retain (kapena sungani ma PV omwe alipo ndi kubectl patch pv <pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}') kotero kuchotsa PVC kapena Namespace sikumachotsa voliyumu yomwe ili pansi.
  • ** Yambitsani AWS EBS Recycle Bin / GCP Snapshot Schedules / Velero yokhala ndi CSI Snapshots **: Kuthandizira AWS EBS Recycle Bin kumasunga voliyumu iliyonse ya EBS yomwe yachotsedwa kwa masiku 7 mpaka 30 ngakhale woyendetsa CSI atayimba foni. DeleteVolume.
  • Musagwiritse ntchito docker system prune --volumes m'malemba oyeretsa okha kapena othamanga a CI/CD: Ngati chotengera chosungiramo zinthu zopangira chiwonongeka mwachidule (Exited (1)) panthawi yomwe ntchito ya cron ikugwira ntchito docker system prune -a --volumes -f, Docker amawona mavoliyumu ake osadziwika "osagwiritsidwa ntchito" ndikuchotsa kwamuyaya. Kuvula --volumes kuchokera pazolemba zonse zoyeretsa.
  • **Gwiritsani ntchito zida zomangirira zowonekera (/srv/prod_data/postgres:/var/lib/postgresql/data) kapena nkhokwe zakunja zoyendetsedwa ndi ntchito zambiri zopanga **: Mosiyana /var/lib/docker/volumes/, mayendedwe a bind-mount omwe ali kunja /var/lib/docker/ sanakhudzidwe konse ndi docker compose down -v, docker volume rm, kapena docker system prune --volumes.

Nambala zangozi zachigawo ndi ndondomeko za bungwe

Pamene chidebe kapena kuchotsedwa kwa Kubernetes kukhudza ntchito zachipatala, kutumiza chitetezo cha anthu, kapena zambiri zaumwini, zimakula nthawi yomweyo:

  • United States ndi Canada: Imbani 911 ngati kulephera kwa chidebe kumalepheretsa kuwunika kwa odwala, machitidwe a EHR, kapena kutumiza mwadzidzidzi. Nenani za kufufutidwa kwa zida zoyipa kapena zochitika za cyber ku CISA (1-844-Say-CISA / cisa.gov/report) kapena Canadian Center for Cyber ​​Security (1-833-CYBER-88).
  • United Kingdom: Imbani 999 zowopseza moyo kapena zofunikira za NHS. Pansi pa UK GDPR, chiwonongeko chamuyaya kapena kutayika kwa kupezeka kwa zidziwitso zanu popanda zosunga zobwezeretsera kumafuna kuwunika chidziwitso chakuphwanya kwa maola 72 ku ICO (0303 123 1113).
  • European Union: Imbani 112 kudutsa mayiko onse a EU. Mabungwe omwe ali pansi pa NIS2 Directive akuyenera kupereka chenjezo ku CSIRT ya dziko lawo pasanathe maola 24 kuchokera pamene ntchito yazimitsidwa kwambiri, ndipo kutayika kwa data kwamuyaya kumayambitsa GDPR Article 33 lipoti mkati mwa maola 72.
  • Australia ndi New Zealand: Imbani 000 ku Australia kapena 111 ku New Zealand; nenani zochitika zovuta za zomangamanga ku ACSC (1300 CYBER1).
  • India ndi Asia-Pacific: Imbani 112 ku India kuti ayankhe mwadzidzidzi; nenani zachitetezo chamtambo kapena chotengera ku CERT-In (incident@cert-in.org.in) mkati mwa zenera lovomerezeka la maola 6.

Mafunso omwe amafunsidwa pafupipafupi

Amatero docker compose down -v Chotsani kukwera kwa host bind (./data:/var/lib/mysql) kapena ma volume a Named and Anonymous Docker okha?

`docker compose down -v` ZOKHA zochotsa Ma voliyumu otchulidwa omwe adalengezedwa mu `volumes:` gawo la `docker-compose.yml` ndi ma voliyumu Osadziwika omwe amalumikizidwa ku makontena; izo ayi imachotsa mafayilo mkati mwa khwekhwe bind mount (monga `- /opt/mysql_data:/var/lib/mysql` kapena `- ./postgres_data:/var/lib/postgresql/data`). Ngati fayilo yanu yolemba idagwiritsa ntchito njira yoyambira `/` kapena `./`, yang'anani fodayo pa wolandirayo pompano: mafayilo anu akadali 100 peresenti.

Chifukwa chiyani? docker system prune --volumes Chotsani voliyumu yanga ya database ndikangoyimitsa chidebecho kwa mphindi 2 kuti ndisinthe kusintha kwachilengedwe?

Dikirani mosamala kusiyana pakati `docker stop` ndi `docker rm`: ngati chidebe ndi chabe `Stopped / Exited` (zowoneka mu `docker ps -a`), `docker system prune --volumes` amachita ayi Chotsani ma voliyumu ake chifukwa chotengera choyimitsidwa chikuwalozerabe. Komabe, ngati munathamanga `docker rm ` (kukonzanso chidebecho ndi mbendera zatsopano) kapena ngati mwadutsa `docker system prune -a --volumes` (yomwe imachotsa koyamba zotengera zonse zoyimitsidwa ndi ndiye imachotsa ma voliyumu onse osalumikizidwanso ndi chidebe chilichonse), voliyumuyo idakhala "yolendewera" millisecond yomwe chidebe choyimitsidwa idachotsedwa ndipo idapukutidwa nthawi yomweyo.

Kodi ndingabwezeretse voliyumu yochotsedwa ya Docker mkati mwa Docker Desktop pa Mac (Apple Silicon / Intel) kapena Windows (WSL2)?

Inde, ngati musiya Docker Desktop nthawi yomweyo kuti makinawo asiye kugwira ntchito `fstrim` motsutsana ndi disk virtual. Pa macOS, Docker imasunga ma voliyumu onse a Linux mkati `-/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw`; pa Windows WSL2, mkati `%LocalAppData%\Docker\wsl\disk\docker_data.vhdx`. Koperani izo `.raw` kapena `.vhdx` fayilo ku SSD yakunja nthawi yomweyo, igwirizanitse kuti muwerenge mu Linux (pogwiritsa ntchito `losetup -Pr` za `.raw` kapena `qemu-nbd -r` za `.vhdx`), ndi kuthamanga `ext4magic` kapena `PhotoRec` motsutsana ndi virtual `ext4` kugawa mkati.

Bwanji ngati nditachotsa Kubernetes StatefulSet ndi kubectl delete statefulset postgres? Kodi izi zinachotsanso ma PVC anga?

Ayi. Mwachikhazikitso ku Kubernetes (pokhapokha `persistentVolumeClaimRetentionPolicy.whenDeleted: Delete` idakonzedwa bwino pa Kubernetes 1.27+), ikuyenda `kubectl delete statefulset` imachotsa wolamulira wa StatefulSet ndi ma Pods ake, kusiya zonse `PersistentVolumeClaims` (`data-postgres-0`, `data-postgres-1`) ndi deta yawo yokhazikika 100 peresenti. Ingogwiritsaninso ntchito yanu `StatefulSet` YAML chiwonetsero ndi dzina lomwelo ndi `volumeClaimTemplates`, ndipo ma Pods atsopano adzalumikizanso ndi omwe alipo `data-postgres-0` Zithunzi za PVC.

Kodi ndimapeza bwanji mafayilo mkati mwa Docker overlay2 chikwatu pamene chidebe chikukana kuyamba (CrashLoopBackOff / Exited (1))?

Osachotsa chidebe chophwanyika. Malingana ngati chidebe choyimitsidwa chidakalipo `docker ps -a`, wosanjikiza wake wolembedwa ndi ma voliyumu okwera amakhala osasunthika pa disk. Inu mukhoza mwina kukopera owona mwachindunji kuchokera anasiya chidebe ntchito `docker cp :/var/lib/postgresql/data /root/rescued_data`, kapena pezani mwini wake weniweni `overlay2` njira yopita `docker inspect --format '{{.GraphDriver.Data.UpperDir}}'` ndikuwerenga bukulo mwachindunji ngati `root`.

Bwanji ngati wina achotsa Kubernetes Namespace (kubectl delete ns prod) ndi ma PVC onse anali nawo reclaimPolicy: Delete pa AWS EBS kapena GCP?

Ngati cloud CSI driver (`ebs.csi.aws.com` kapena `pd.csi.storage.gke.io`) aphedwa kale `DeleteVolume` chifukwa `reclaimPolicy` anali `Delete`, yang'anani maukonde atatu otetezedwa amtambo nthawi yomweyo: (1) fufuzani ** AWS Recycle Bin for EBS Volumes ** mu AWS EC2 console (ngati lamulo losungirako linali logwira ntchito pa akaunti / ma tag, voliyumu ya EBS yochotsedwa ikukhala mu Recycle Bin ndipo ikhoza kubwezeretsedwa mu 1 pitani); (2) fufuzani ** AWS Backup / DLM / GCP Snapshot Schedule ** kuti mupeze chithunzithunzi chaposachedwa kwambiri cha voliyumu; ndi (3) onani ngati database yanu ikukhamukira WAL/binlogs (`WAL-G`, `pgBackRest`, `Litestream`) ku chidebe cha S3/GCS.


Maupangiri ogwirizana


Magwero ndi malangizo ovomerezeka

  • Kubernetes Official Documentation (kubernetes.io / Cloud Native Computing Foundation): Chinthu Chosungira Mukugwiritsa Ntchito Chitetezo (kubernetes.io/pvc-protection Omaliza), Reclaiming PersistentVolumes (Retain vs. Delete), ndi Kuchita etcd Magulu a Kubernetes (etcdctl snapshot restore).
  • Malingaliro a kampani Docker Inc.docs.docker.com): * Sinthani Zambiri mu Docker: Volumes, Bind Mounts, overlay2 Zomangamanga Zoyendetsa Dalaivala, ndi Prune Reference *.
  • Bungwe la Linux Kernel (kernel.org): Ndi /proc Filesystem (procfs Tsegulani File Descriptor Recovery kudzera /proc/<pid>/fd/) ndi Ext4 Journal Specification.
  • ext4magic Zolemba za Project (ext4magic.sourceforge.net): Forensic Inode ndi Journal Transaction Recovery pa Ext3/Ext4 Filesystems.
  • National Institute of Standards and Technology (NIST) (United States): Special Publication 800-190: Application Container Security Guide & Incident Recovery.