Skip to content

Docker or Kubernetes Production Volume Muku: Whakaorangia docker system prune --volumes, overlay2 Inodes, me etcd Nga whakaahua

He rereke nga nama ohorere i ia whenua. Ka whakamahia e tenei wharangi te 911, te nama mo te United States me Canada. He maha nga whenua e whakamahi ana i te 112, 999, tetahi atu nama ranei. Kimihia te nama mo to whenua.

Rere ana docker compose down -v, docker system prune -a --volumes, docker volume rm, ranei kubectl delete pvc i runga i te ripa kapeka he i roto i te hanga ko tetahi o nga huarahi tere ki te muru i te PostgreSQL ora, MySQL, MariaDB, Redis, MongoDB, SQLite ranei rōrahi pātengi raraunga. Kaore i rite ki nga whakaahua miihini mariko kapua ka neke ki te ipu para 7-ra, ko te taraiwa rōrahi rohe a Docker (/var/lib/docker/volumes/<name>/_data) me Kubernetes local-path / hostPath ka tukuna e nga kaiwhakarato te POSIX paerewa unlink() a rmdir() ka karanga tika ki te punaha konae kaihautu Linux (ext4, XFS, ranei Btrfs). Ko nga tuhi katoa ka puta i runga i taua kaihautu Linux i muri i te mutunga o te whakahau ka taea te tuhirua i nga poraka papaa raraunga kua wetekina ki runga kōpae. Mena kua whakakiia to kōpae tūmau ki te 100 ōrau (0 bytes free) i mua i te omatanga o tetahi docker system prune --volumes i roto i te panic, consult Ko te kōpae tūmau 100% ki tonu, te putunga rangitaki WAL pātengi raraunga, me te whakaora mokowā-kore; mo te whakakore i nga konae Linux i waho o nga ipu, tirohia ohorere rm -rf, Shift-Muku ranei i te ext4, XFS, APFS, me te NTFS: panui-anake te remount me te whakairo inode a Mīhini mariko VMDK, VHDX, me te qcow2 te pirau mekameka hopu whakaahua me te whakaoranga hypervisor. Mena ka pa te mate o tenei whakaputanga ki te tiaki i nga turoro o te hohipera, te tuku ohorere, me nga punaha haumaru taputapu a te iwi, waea atu ki to nama whawhati tata (911 i te United States me Canada, 112 puta noa i te Kotahitanga o Europi, 999 i roto i te United Kingdom) tirohia to maatau raarangi ingoa tau ohorere.

Mahia tenei inaianei

  1. ** KAUA e oma docker compose up -d, systemctl restart docker, ka tukuna ano ranei te Pod Kubernetes**: Ma te hanga ano i te ipu, ki te whakaara ano ranei i te daemon Docker ka tuhia nga megabytes o nga mea hou. overlay2 metadata, ipu JSON rangitaki, me nga konae putunga putunga hou i arawhitia (initdb) tika i runga i te tika ext4 ranei XFS poraka i noho ai to rōrahi kua mukua i ngā hēkona ki muri.
  2. Tirohia i roto i te 5 hēkona mena kei te mahi ipu (postgres, mysqld, redis-server, node, python) kei te haere tonu i roto i te mahara (ps aux a sudo lsof +L1): Ki te mukua e te tangata nga konae kei roto i te rōrahi kua whakairihia, ka wetewete ranei i tetahi rōrahi ingoamuna i te wa e rere tonu ana te ipu, ka noho tonu te kernel Linux 100 ōrau o ngā kōnae pātengi raraunga kore hono i runga i te kōpae i te mea ko te ID Tukanga o te ipu (PID) e pupuri ana i tetahi kaiwhakaahua konae tuwhera (FD). Ka taea e koe te kape tika i aua konae ki waho /proc/<PID>/fd/ ranei /proc/<PID>/root/ me te kore ngaronga raraunga i mua i te tu o te ipu.
  3. Mēnā kua tū kē te ipu ka mukua te rōrahi (docker compose down -v ranei docker system prune --volumes), whakatio te kopae kaihautu ka tuhi tonu: Kati te daemon Docker (sudo systemctl stop docker.socket docker.service containerd) na nga ipu papamuri me nga tirotiro hauora ka mutu te tuhi i nga raarangi ki roto /var/lib/docker, katahi ka whakaara ano /var/lib/docker (te pakiaka ranei / wehewehe) panui-anake (sudo mount -o remount,ro /) ka mau tonu ranei he hopukanga o te rōrahi kapua taumata paraka (AWS EBS Snapshot, GCP Persistent Disk Snapshot, DigitalOcean Volume Snapshot) i mua i te pa ki tetahi mea.
  4. Mena kei runga koe i nga Kubernetes (kubectl delete pvc ranei kubectl delete namespace), tirohia te PersistentVolume (PV) ReclaimPolicy (kubectl get pv) tonu:
  5. Mena ko te PV i persistentVolumeReclaimPolicy: Retain, ko o raraunga kaore i mukua: te PV neke noa ki Released tūnga me tō kōpae kapua (EBS, Azure Disk, Ceph RBD, NFS) he 100 paiheneti tonu.
  6. Ahakoa kubectl delete pvc kua mau ki roto Terminating, Kubernetes Whakaoti (kubernetes.io/pvc-protection) aukatihia te atekōkiri CSI kia kore e mukua te rokiroki o raro i te mea kei te korero tonu tetahi Pod ki taua PVC. Kaua e mukua te Pod kia kape ra ano koe i nga raraunga mai i te Pod e rere ana.
  7. Whakaahua te arai paraka o raro (dd / ddrescue i runga i te SSH) ki te kōpae o waho, te rōrahi kapua motuhake ranei i mua i te rere ext4magic, extundelete, ranei PhotoRec: Kaua rawa e whakahaere i nga kaiwhakairo konae e tuhi ana i nga konae whakaputa ki runga i te wehenga kōpae kotahi e ora ana koe /var/lib/docker/volumes.
  8. Mo te ahua o te roopu Kubernetes kua kino, kua murua ohorere ranei (kubectl delete crd / namespace), mutu kube-apiserver me te whakahoki i to mea hou etcd whakaahua (etcdctl snapshot restore): Mēnā kei roto te kāhui etcd i murua i te wa e noho tonu ana nga pukapuka rokiroki kapua, e whakaora ana etcd whakaora katoa PV, PVC, Secret, a ConfigMap taonga i te wa kotahi.

Danger

** Kaua rawa e tangohia te kubernetes.io/pvc-protection ranei kubernetes.io/pv-protection whakaotinga me kubectl patch ... -p '{"metadata":{"finalizers":null}}' ka mau te PVC ki roto Terminating**: Ko taua kaiwhakamutu anake te mea hei pupuri i to taraiwa CSI kapua (aws-ebs-csi-driver, pd.csi.storage.gke.io, Longhorn, Rook-Ceph) mai i te tuku taputapu DeleteVolume Karanga API ki te whakangaro i to kōpae whakaputa. I te wa e mau ana te PVC ki roto Terminating, kei te rere tonu te Pod, kei te mau tonu te rōrahi: rere kubectl exec ranei kubectl cp (ranei pg_dump / mysqldump) ki te whakaora i te 100 ōrau o tō pātengi raraunga i mua i te mutunga o te Pod.


Rakau whakatau: kua mukua te rōrahi Docker, overlay2 paparanga, kubernetes PVC triage ranei

Whakamahia tenei tūtohi rerenga ki te whiriwhiri i te huarahi whakaora kore-mate tere ake i runga i te mea kei te ora tonu te tukanga ipu, ahakoa i tautokohia te rōrahi e te kōpae paetata, te rokiroki kapua CSI ranei, a ko wai te punaha konae e manaaki ana. /var/lib/docker:

flowchart TD
    Start["Kua ngaro nga Raraunga Whakaputa I muri i te Whakahau a Docker, Kubernetes ranei te Mukua"] --> WhichEnv{"Ko tehea te papa me te whakahau i mate ai?"}

    WhichEnv -->|"Docker: E Rere tonu ana te Ipu (Ko nga konae kua kore e hono i roto i te rōrahi/ipu)"| ProcRescue["Whakaorangia Whakatuwheratia nga Kaitohu Kōnae Kore Hononga ma te sudo lsof +L1 me /proc/PID/fd/N ranei /proc/PID/root/"]

    WhichEnv -->|"Docker: I Tuhia te Ipu & Kua Mukua Rōrahi (tito whakararo -v / puru --rōrahi)"| StopDocker["Whakahaere tonu: sudo systemctl mutu docker.socket docker containerd"]

    WhichEnv -->|"Kubernetes: kubectl muku pvc / pv / statefulset / mokowāingoa"| K8sCheck{"Kei te Rere tonu te Pod, kei te mau ranei te PVC ki te whakamutu?"}

    K8sCheck -->|"Āe: E Rere tonu ana te Pod (Kaiwhakamutunga te aukati CSI Muku)"| K8sExec["KAUA E Mukua Pod -> Whakahaerehia te kubectl exec / pg_dump / tar ki runga i te stdout ki te Kōpae Paetata Inamata"]
    K8sCheck -->|"Kao: Pod Kua ngaro -> Tirohia kubectl tiki pv"| ReclaimCheck{"He aha te kaupapa here Reclaim PV?"}

    ReclaimCheck -->|"Pupuri (Tūnga PV = Kua Tukua)"| ClearClaimRef["100% Raraunga Tonu -> Whakatika PV ki te Tango spec.claimRef.uid -> Herea ano ki te PVC Hou"]
    ReclaimCheck -->|"Mukua (Cloud EBS / PD / Longhorn / Ara-a-rohe)"| CloudOrLocal{"Rōrahi Kapua CSI, Kōpae Kaihautū Rohe rānei?"}

    CloudOrLocal -->|"Kapua CSI (AWS EBS / GCP / Azure / Ceph)"| CloudSnap["Tirohia te Ipu Para Papatohu Kapua / Nga Whakaahua EBS Aunoa / Ceph RBD Parapara (rbd paru ls)"]
    CloudOrLocal -->|"Kōpae Kōpuku Paetata (ara-rohe / hostPath)"| StopDocker

    StopDocker --> RemountRO["Whakanuia ano te Wehewehenga Kaihautu Panui-anake (maunga -o remount,ro) RĀNEI Aroma Paraka Atahanga ki runga i te SSH"]
    RemountRO --> HostFS{"He aha nga Kaihautu Pūnaha Kōnae /var/lib/docker?"}
    HostFS -->|"ext4"| Ext4Rec["Whakamahia debugfs / ext4magic Journal Inode Transaction Rollback (-he waitohuwā)"]
    HostFS -->|"XFS / Btrfs"| CarveRec["Whakamahia Btrfs Roots / xfs_irecover / Whakairo Waitohu Ritenga mo SQLite, Postgres Heap, InnoDB .ibd ranei"]

Wāhanga 1: Me pehea te whakakore a Docker me Kubernetes i nga pukapuka kei runga kōpae

I mua i te whakahaere i tetahi whakahau whakaora, me mohio kei hea to raraunga i noho tinana i runga i te kaihautu Linux me nga mea i pa ki te whakahau muku:

Momo Rokiroki & Muku Whakahau Kaihautū Ara Pūnaha Kōnae i runga i te Linux Node He aha te Whakahau Whakamuku I Mahi Tikanga Whakaora Tere
Whakaingoatia Rōrahi Docker (docker volume rm ranei docker compose down -v) /var/lib/docker/volumes/<vol_name>/_data I wetewetehia nga konae kei roto _data/ ka tangohia te urunga whaiaronga mai /var/lib/docker/volumes/metadata.db (boltdb). Kati tonu i a Docker, whakanekehia /var/lib/docker panui-anake, me te whakaora i nga inodes kore hono ma ext4magic / debugfs whakairo waitohu ranei.
** Rōrahi Docker ingoakore (docker run -v /var/lib/mysql + docker rm -v ranei docker system prune --volumes)** /var/lib/docker/volumes/<64-char-hex-id>/_data docker system prune --volumes ka mukua ia rōrahi ingoamuna me te whakaingoatia karekau i te tohutoro i tenei wa e te ipu kotahi. Kati tonu a Docker; Mēnā he ētahi atu kua aukatihia e koe nga ipu kua kare i tapatapahia, tirohia e toe ana /var/lib/docker/volumes/*/_data kopaki i te tuatahi i mua i te whakairo i nga poraka kaore i tohatohahia.
Ipu Pae Tuhituhi (docker rm <container> i te wa i tuhia ai te raraunga ki waho o te rōrahi) /var/lib/docker/overlay2/<layer-id>/diff/ I wetekina te hononga o runga o te ipu diff/ whaiaronga kei roto i te overlay2 taraiwa rokiroki. Kati tonu a Docker ka whakaora i te rakau whaiaronga kore hono mai i te ext4 hautaka ranei XFS poraka.
** Papamahi Docker i runga i te macOS, i te Matapihi ranei (docker system prune --volumes)** I roto i te ahua kōpae miihini mariko kotahi: -/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw (macOS) ranei %LocalAppData%\Docker\wsl\disk\docker_data.vhdx (Windows WSL2). Tukuna unlink() i roto i te Linux VM, a mena discard/TRIM kei te kaha Docker.raw / docker_data.vhdx, ka werohia nga kohao iti ki te konae kaihautu. Whakamutua te Papamahi Docker i roto i te 5 hēkona, kape Docker.raw ranei docker_data.vhdx ki te puku o waho, ka oma ext4magic / PhotoRec ki te ahua kōpae mariko.
Kubernetes PVC ki Delete Kaupapa Whakaora mo Ceph RBD (Rook-Ceph) ranei Longhorn puna Ceph OSD (rbd) ranei /var/lib/longhorn/replicas/. Mehemea csi-rbdplugin kua trash kua whakahohea (rbd trash), ka nekehia te ahua RBD ki te ipu paru Ceph; ki te kore e kore e mapi nga poraka. Rere rbd trash ls --pool <pool> roto te rook-ceph-tools pod me te whakaora ki rbd trash restore.

Wāhanga 2: Whakaorangia ngā kōnae pātengi raraunga tuwhera mai i tētahi ipu e rere tonu ana (/proc/<pid>/fd/)

Tatari: kei te rere tonu te ipu inaianei? I te nuinga o te wa, ka whakahaerehia e te miihini he tuhinga heke tono, he mahi cron horoi, he pukapuka pukapuka ranei rm -rf /var/lib/postgresql/data/* ranei rm /app/data/production.sqlite3 i roto i te ipu e rere ana, ka mukua ranei te kōpaki here-maunga o raro /opt/app/data ko te ipu ano (postgres, mysqld, mariadbd, redis-server, mongod, gunicorn, node) kei kei te haere tonu me te rere.

I roto i te Linux, karekau he konae e mukua mai i te kōpae ina rm ranei unlink() ka kiia. Ko ona paraka kōpae me ona inode kei te noho 100 paiheneti kua tohatohahia me te panui tae noa ki te tukanga whakamutunga e pupuri ana i tetahi kaiwhakaahua konae tuwhera (FD) ki taua inode ka kati, ka puta ranei.

Hipanga 1: Kimihia te ID Tukanga kaihautu (PID) o te tukanga ipu

I runga i te node ope Linux (kaore i roto i te ipu), kimihia te kaihautu PID o to papaunga raraunga, tukanga tono ranei:

sudo lsof +L1 | grep -E "postgres|mysqld|mariadbd|sqlite|redis|mongo|node|python|java"
Tirohia te putanga:
COMMAND     PID   USER   FD   TYPE DEVICE  SIZE/OFF NLINK    NODE NAME
mysqld    18492    999   14u   REG  259,1 482344960     0 1849201 /var/lib/mysql/app_prod/users.ibd (deleted)
mysqld    18492    999   15u   REG  259,1 125829120     0 1849202 /var/lib/mysql/app_prod/orders.ibd (deleted)
mysqld    18492    999   16u   REG  259,1  79691776     0 1849190 /var/lib/mysql/ibdata1 (deleted)
python3   22104   1000    4u   REG  259,1  89128960     0 3102944 /data/production.sqlite3 (deleted)
Titiro ki te poupou NLINK: e kii ana 0, ka mutu te ingoa kōnae ki te (deleted). Kua ngaro katoa te konae ls /var/lib/docker/volumes/..., heoi ia paita kotahi (482 MB i roto users.ibd, 89 MB i roto production.sqlite3) kei te ora tonu i roto /proc/<PID>/fd/<FD>.

Hipanga 2: Tāruahia ngā kōnae pātengi raraunga kore hono tika ki waho /proc/<PID>/fd/

Kaua **kaore e mutu, ka whakaara ano ranei i te ipu. I te wa e haere tonu ana te mahi, kape tika nga kaiwhakaahua konae ki tetahi kōpaki haumaru i runga i to kaihautu:

sudo mkdir -p /root/proc_emergency_rescue

# Example 1: Rescuing an unlinked SQLite3 database (PID 22104, FD 4)
sudo cp /proc/22104/fd/4 /root/proc_emergency_rescue/production.sqlite3
# Check if PID 22104 also holds the -wal file open (e.g. FD 5) and copy it too:
sudo cp /proc/22104/fd/5 /root/proc_emergency_rescue/production.sqlite3-wal

# Example 2: Automatically rescuing ALL open (deleted) MySQL/Postgres files for PID 18492
for fd in /proc/18492/fd/*; do
    target=$(sudo readlink "$fd" 2>/dev/null)
    if echo "$target" | grep -q "(deleted)"; then
        fd_num=$(basename "$fd")
        clean_name=$(echo "$target" | sed 's/ (deleted)//; s/\//_/g')
        echo "Rescuing FD $fd_num -> $clean_name"
        sudo cp "$fd" "/root/proc_emergency_rescue/${fd_num}${clean_name}"
    fi
done

Hipanga 3: He pai ake: Whakahaerehia he putunga arorau ora (pg_dump / mysqldump / .backup) i mua i te kati o te ipu i nga FD

Mehemea postgres ranei mysqld kei te rere tonu me ona konae raraunga e tuwhera ana ki te mahara me /proc/<PID>/fd/, ngana ki te whakahaere i tetahi putunga arorau ora i runga i te turanga unix, hononga TCP ranei i te tuatahi:

# If the database engine can still read its open file descriptors:
docker exec <container_name> pg_dumpall -U postgres > /root/proc_emergency_rescue/live_dump.sql
# Or for MySQL:
docker exec <container_name> mysqldump -u root -p --all-databases --single-transaction > /root/proc_emergency_rescue/live_mysql.sql
(Tuhipoka: Ka tuwhera ka kati a PostgreSQL .1, .2 wehewehe i nga konae puta noa i nga mahi a nga kaimahi, na pg_dump ka taka pea i runga i nga tepu keteroki kore; te whakakotahi i te uiui SQL ora me te kape i nga tuwhera katoa /proc/<pid>/fd/* puta noa katoa postgres Ka tiakina e nga PID tuara tamaiti te nui rawa o nga raraunga puranga ripanga.)


Wāhanga 3: Te whakaora i te rōrahi Docker kua mukua ext4 ranei XFS ka mutu te ipu

He aha mena ka oma koe docker compose down -v ranei docker system prune --volumes, no reira kua kua tu te ipu me te katoa /proc/<pid>/fd/ kua katia nga whakaahuatanga? Inaianei kei te whakataetae koe ki te kopae papamuri e tuhi ana i runga i te kaihautu Linux.

Hipanga 1: Whakatiohia te punaha konae manaaki Linux i roto i te 10 hēkona

  1. Kati tonu te Docker, containerd, kubelet, me te cron na karekau he ipu, he tirotiro hauora, he kaitahuri rakau ranei e tuhi i tetahi atu paita ki /var/lib/docker:
    sudo systemctl stop docker.socket docker.service containerd.service cron.service
    
  2. Tirohia ko tehea taputapu paraka me te punaha konae kei te pupuri /var/lib/docker:
    df -Th /var/lib/docker
    
    Filesystem     Type  Size  Used Avail Use% Mounted on
    /dev/nvme0n1p1 ext4  100G   42G   58G  42% /
    
  3. Whakahokia ano te wehewehenga Panui-anake (whakarerehia ranei he atahanga wehewehenga mata ki runga SSH ki tetahi atu miihini):
  4. Mehemea /var/lib/docker kei runga i tana ake wehewehenga i whakatapua, i te rōrahi LVM ranei (hei tauira /dev/nvme0n1p2 ranei /dev/mapper/vg0-docker), whakairihia te panui-anake tonu:
    sudo mount -o remount,ro /var/lib/docker
    
  5. Mehemea /var/lib/docker kei runga i te putake / wehewehe (/), ngana ki te akiaki i te arai pakiaka kia panui-anake ma te whakamahi i te kaaka Magic SysRq keu (echo u | sudo tee /proc/sysrq-trigger) ranei sudo mount -o remount,ro /.
  6. Mena kei runga koe i te VPS kapua (AWS EC2, GCP, DigitalOcean, Hetzner, Linode): Haere tonu ki to papatohu tukutuku kapua ka paato i te Waihanga Whakaahua ki te rōrahi poraka o te tūmau (whakarerehia ranei te taputapu paraka mata ki runga SSH ki te tūmau tuarua: sudo dd if=/dev/nvme0n1p1 bs=4M status=progress | gzip -1 | ssh user@backup-server "cat > /mnt/storage/docker_host_raw.img.gz").

Hipanga 2: Tirohia mehemea kei te noho tonu to raraunga i roto i TĒTAHI rōrahi karekau ranei overlay2 paparanga

I mua i te whakairo i nga paraka kōpae kare i tohatohahia, manatokohia karekau to papaunga raraunga i penapena i roto i te maunga here, i te ingoa rereke ranei i ora:

# Search the entire /var/lib/docker tree read-only for any surviving database files:
sudo find /var/lib/docker -name "PG_VERSION" -o -name "ibdata1" -o -name "*.sqlite*" -o -name "*.db" -o -name "dump.rdb" -o -name "WiredTiger" 2>/dev/null
Ia wiki, ka mataku nga kaiwhakahaere i muri i te oma docker volume rm app_db_data, ka kitea anake me find /var/lib/docker tera docker-compose.yml i tino hanga projectname_app_db_data (kei te 100 ōrau tonu kei runga i te kōpae) i te wā app_db_data he pukapuka tawhito kaore i whakamahia mai i nga marama e ono ki muri.

Hipanga 3: Te whakaora i nga raarangi pukapuka Docker kua kore e hono ext4 te whakamahi i te ext4 Tuhituhi (ext4magic a debugfs)

Ina docker volume rm ka mukua /var/lib/docker/volumes/<vol_name>/_data i runga i te ext4 pūnaha kōnae, ext4 ka kore nga tohu paraka tika (i_block / extent tree root) i te ripanga inode ora. Heoi ano, ko te ext4 hautaka (inode <8>, te tikanga 128 MB ki 1024 MB porowhita whakakai porowhita) kei te pupuri tonu i nga kape tawhito o nga ripanga inode tika me nga poraka whaiaronga mai i mua tonu docker compose down -v i patua.

+-------------------------------------------------------------------------+
| HOW ext4magic RECOVERS A DELETED DOCKER VOLUME FROM THE EXT4 JOURNAL    |
|                                                                         |
| Live Inode Table on Disk (After docker volume rm):                      |
|   Inode #918402 (users.ibd): [Size = 0, Extent Pointers = ZEROED]       |
|                                                                         |
| Ext4 Journal Ring Buffer (Inode <8> on Disk):                           |
|   Transaction #41902 (10 mins ago):                                     |
|   Inode #918402 (users.ibd): [Size = 482MB, Extents = Blocks 9000..9999]|
|                                                                         |
| -> ext4magic reads the old Inode Copy from Journal Transaction #41902   |
|    and extracts Blocks 9000..9999 directly to your rescue drive.        |
+-------------------------------------------------------------------------+

Rere ext4magic ki to wahanga panui-anake (te .img kape whakaahua i runga i te tūmau tuarua), e tohu ana i te tohuwā (-a) mai i **5 meneti i mua i a ** i oma koe docker compose down -v ranei docker system prune --volumes:

# Install ext4magic on your rescue machine (never on the affected partition)
sudo apt-get install -y ext4magic e2fsprogs

# Dump the ext4 journal from the frozen partition / image to a file in RAM (/dev/shm)
sudo debugfs -R "dump <8> /dev/shm/ext4_journal.copy" /dev/nvme0n1p1

# Recover all files deleted under var/lib/docker/volumes after (Current Time - 15 minutes)
sudo mkdir -p /mnt/external_rescue/docker_recovered
sudo ext4magic /dev/nvme0n1p1 \
  -j /dev/shm/ext4_journal.copy \
  -f var/lib/docker/volumes \
  -a $(date -d "-15 minutes" +%s) \
  -r -d /mnt/external_rescue/docker_recovered
- -j /dev/shm/ext4_journal.copy: Ka panuihia te whakaahua hautaka i tiakina e koe ki te RAM i muri tonu i te aukati i a Docker. - -f var/lib/docker/volumes: E whai ana i te rakau whaiaronga pukapuka Docker tika (tuhipoka: waihohia te arataki / i roto ext4magic ara whanaunga). - -a $(date -d "-15 minutes" +%s): Ka korero ext4magic ki te hanga ano i te rakau whaiaronga ma te whakamahi i nga kape inode whaimana 15 meneti ki muri (i mua tonu i te mukunga ohorere). - -r: Ka whakaora i nga whaiaronga katoa kaore he poraka i tuhiruatia; mehemea -r ka ngaro i nga konae kua tuhiruatia, whakahaere i te tuku tuarua hei whakakapi -r me -m (whakairo makutu + aratau ranu hautaka).

Hipanga 4: Te whakairo i nga konae SQLite mata, PostgreSQL, MySQL InnoDB ranei ina takai te hautaka (XFS ranei ext4)

Warning

He aha XFS e kore e tautoko ext4magic me te hiahia kia wetewete te panui-anake tonu: Kaore i rite ext4, e tuhi ana i nga kape inode 256-paita katoa ki roto i tana hautaka, XFS ka tuhi i nga tohu metadata intent me te kaha ki te whakamahi ano i nga inode B-rakau (agi). Mehemea /var/lib/docker noho ana i runga i te XFS wehewehenga (taunoa i runga i te RHEL, Rocky Linux, AlmaLinux, me Amazon Linux), wetewete /var/lib/docker te whakanohonoho ranei / Ko te panui-anake i roto i nga hēkona ko te taahiraa tino nui i mua i te whakairo.

Mena kei te whakamahi te kaihautu XFS mehemea ranei te ext4 hautaka kua takaia ki runga i nga tohu inode, whakamahia te waitohu me te whakairo hanganga ki te ahua kōpae mata (docker_host_raw.img):

  1. Mo nga Raraunga Raraunga SQLite3 (.sqlite, .db - hei tauira, Vaultwarden, Nextcloud, Kaiāwhina Kāinga, Gitea, Grafana): Ka timata nga papaa raraunga SQLite3 me te pane ASCII 16-paita e kore e pohehe i te paita 0 o tana wharangi 4 kilopaita tuatahi: 53 51 4c 69 74 65 20 66 6f 72 6d 61 74 20 33 00 ("SQLite format 3\0"), me te penapena i te rahinga wharangi papaarangi tika (Bytes 16..17, te tikanga 4096) me te rahinga pātengi raraunga katoa i ngā whārangi (Bytes 28..31, nui-endian 32-bit integer) kei roto tonu i taua pane.
  2. Rere PhotoRec me [File Opt] -> [x] sqlite whakahohea, karapa tika ranei i te wehewehenga grep -ab "SQLite format 3" ki te kimi i te paita tika o to konae SQLite kua mukua me te tango dd:
    # Find the exact byte offsets of every "SQLite format 3" header on the partition:
    sudo grep -ab "SQLite format 3" /dev/nvme0n1p1
    
    Kia tangohia, rere sqlite3 carved.db ".recover" | sqlite3 repaired.db ki te hanga ano i ia wharangi rau rakau B mau tonu.

Tip

Me pehea te tatau i te tika dd paita te rahi o te konae SQLite3 kore hono mai i tana pane: Kotahi grep -ab "SQLite format 3" ka hoatu ki a koe te paita timata OFFSET o to raraunga SQLite kua mukua, panui paita 16..17 (uint16 Rahi Whārangi nui-endian, hei tauira, 0x1000 = 4096 paita) me nga paita 28..31 (Uint32 Kaute Whārangi nui-endian, hei tauira, 25000 wharangi) whakamahi hexdump -C -s OFFSET -n 32. Whakareatia Page Size x Page Count homai ki a koe te roa paita tika hei tuku atu dd (iflag=skip_bytes,count_bytes) i te wa e piri ana te konae ki runga kōpae.

  1. Mo MySQL / MariaDB InnoDB (.ibd kōnae ripanga): Ia InnoDB .ibd he raupapa raupapa 16 KB (16,384 paita) nga wharangi kei roto te FIL_PAGE_INDEX (0x45BF) pane me te ahurei o te ripanga space_id. Whakamahia te Percona / TwinDB undrop-for-innodb (stream_parser + c_parser) kete taputapu hei matawai /dev/nvme0n1p1 mo te katoa 16 KB Nga wharangi taurangi InnoDB ka maka tika ia rarangi ki nga konae TSV/SQL hei kawemai ano.
  2. Mo nga konae PostgreSQL Heap ($PGDATA/base/<db_oid>/<relfilenode>): Ka penapenahia e PostgreSQL ia tepu hei piri 8 KB (8,192 paita) heap page kei roto PageHeaderData (pd_lower, pd_upper, pd_special). Whakamahia pg_filedumpranei pglazarus / pdu ki te matawai i nga rohe kōpae kaore i tohatohahia 8 KB nga wharangi e rite ana ki te tahora tuple o to ripanga (HeapTupleHeaderData) me te hanga ano i nga rarangi.

Wāhanga 4: Kubernetes PVC, PV, me etcd Whakaahuatanga Whakaora ohorere

Kei Kubernetes (EKS, GKE, AKS, k3s, RKE2, kubeadm, OpenShift), te muku ohorere i tetahi PersistentVolumeClaim (kubectl delete pvc), he StatefulSet, he Mokowā Ingoa katoa ranei (kubectl delete ns prod) e whai ana i tetahi hoahoanga tino rereke atu i te Docker tuuturu.

Tauari A: Ko nga whakaaturanga PVC Terminating (Ko te Kaiwhakaora Whakaora)

Whakahaere:

kubectl get pvc -n <namespace>
Mena ko te STATUS ta te pou Terminating, kia mapu: KORE ano to raraunga i mukua. - He aha? Ko ia Kubernetes PVC i hangaia mai i te Kubernetes 1.9 kei roto aunoa finalizers: [kubernetes.io/pvc-protection]. - I te mea kei te mau tonu tetahi Pod i taua PVC, kaore te kaiwhakahaere Kubernetes e whakaae ki te whakaoti i te whakakorenga PVC me te kaore e karanga te taraiwa CSI. DeleteVolume RPC. - Mahinga Inamata: Kaua KORE te muku, te whakaara ano ranei i te Pod, ka KORE taapihia te kaiwhakaoti. Tāruahia nga raraunga katoa mai i te Pod e rere ana ki to miihini rohe, ki te peere S3 ranei:
# Stream a compressed tar backup or pg_dump out of the still-running pod immediately:
kubectl exec -n <namespace> <pod-name> -- pg_dumpall -U postgres > ./rescued_k8s_db.sql
# Or tar an entire mounted volume directory out over stdout:
kubectl exec -n <namespace> <pod-name> -- tar czf - /var/lib/postgresql/data > ./rescued_pvc_files.tar.gz

Tauari B: I whakakorea te PVC, a kubectl get pv whakaatu i te PersistentVolume i roto i Released āhua (ReclaimPolicy: Retain)

Whakahaere:

kubectl get pv
Mena ko to Puka Tonu (pv-prod-postgres) kei te rārangi tonu me STATUS: Released (na te mea kua whakamahia tana StorageClass reclaimPolicy: Retain), 100 paihēneti o ō raraunga kei te noho tonu i runga i te kōpae kei raro. Heoi, kare a Kubernetes e whakaae kia herea tetahi PVC hou ki a Released PV no te mea kei roto tonu te PV a spec.claimRef e tohu ana ki nga PVC tawhito kua mukua UID.

Me pehea te hanga a Released PV Available ano i roto i te 10 hēkona kaore he ngaronga raraunga: 1. Whakatikaina te ahanoa PersistentVolume ka muku i te spec.claimRef poraka:

kubectl patch pv <your-pv-name> --type json -p '[{"op": "remove", "path": "/spec/claimRef"}]'
2. Rere kubectl get pv ano: ka huri tonu te mana mai i Releasedki Available. 3. Hangaia ano to PersistentVolumeClaim (whakaahua volumeName: <your-pv-name> me te rite ki te kaha / rokirokiClassName). Ka herea ki te PV o naianei me nga konae katoa e mau tonu ana.

Tauari C: Te whakaora i te ahua kahui Kubernetes kua murua mai i te etcd whakaahua (etcdctl snapshot restore)

Ka pehea mena ka oma tetahi kubectl delete namespace prod ranei helm uninstall ka murua e 200 nga whakaaturanga Kubernetes, Nga Mea ngaro, Mahere Whirihora, me nga whakamaramatanga PV i runga i te whakahaere whaiaro kubeadm, k3s, ranei RKE2 rererangi whakahaere?

  1. ** Kimihia to mea hou etcd whakaahua** (ma te taunoa i roto /var/lib/rancher/k3s/server/db/snapshots/, /var/lib/rancher/rke2/server/db/snapshots/, ranei to whakaritenga /var/backups/etcd/snapshot.db).
  2. Manatokohia te tika o te hopu whakaahua i mua i te pa ki te kahui ora:
    ETCDCTL_API=3 etcdctl snapshot status /path/to/snapshot.db --write-out=table
    
  3. Kati kube-apiserver a etcd i runga i te (ng) pona rererangi whakahaere:
  4. Kei kubeadm (panuku pateko): Nuku /etc/kubernetes/manifests/kube-apiserver.yaml a /etc/kubernetes/manifests/etcd.yaml mo te wa poto /root/manifests_hold/ na kubelet ka mutu nga ipu e rua kia ma, ka hoki ki runga /var/lib/etcd ki /var/lib/etcd.bak.
  5. Kei k3s: Rere sudo systemctl stop k3s a sudo k3s server --cluster-reset --cluster-reset-restore-path=/path/to/snapshot.
  6. Whakahokia te etcd whaiaronga raraunga i runga kubeadm:
    sudo ETCDCTL_API=3 etcdctl snapshot restore /path/to/snapshot.db \
      --data-dir=/var/lib/etcd \
      --name=<control-plane-node-name> \
      --initial-cluster=<node-name>=https://<node-ip>:2380 \
      --initial-advertise-peer-urls=https://<node-ip>:2380
    
  7. Nuku etcd.yaml a kube-apiserver.yaml hoki ki roto /etc/kubernetes/manifests/. I roto i te 30 hēkona, kube-apiserver ka hono ano ki te mea kua whakahokia mai etcd toa me nga mokowāingoa katoa kua mukua, ngaro, ConfigMap, me PersistentVolume ahanoa ka puta ano.

Wāhanga 5: Whakapakeke i nga pukapuka a Docker me Kubernetes ki te whakakore ohorere

Kia whakahokia mai to papaunga raraunga whakaputa, whakatinanahia enei arai hoahoanga e wha kia kore ai e taea e tetahi momo CLI kotahi te whakakore i te reo ora:

  • ** Hurihia ia hanga Kubernetes StorageClass mai reclaimPolicy: Delete ki reclaimPolicy: Retain**: Ma te taunoa, i roto i te kapua StorageClasses (gp2, gp3, standard-rwo) taunoa ki reclaimPolicy: DeleteKo te tikanga ko te whakakore i te PVC ka whakangaro i te kōpae kapua. Waihanga a StorageClass me reclaimPolicy: Retain (Pakihia ranei nga PV o naianei me kubectl patch pv <pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}') no reira ko te whakakore i te PVC, i te Mokowāingoa rānei e kore e mukua te rōrahi paraka o raro.
  • Whakahohehia te AWS EBS Recycle Bin / GCP Snapshot Schedules / Velero me CSI Snapshots: Ko te whakahohe i te AWS EBS Recycle Bin ka pupuri i nga rōrahi EBS kua mukua mo te 7 ki te 30 ra ahakoa waea mai te taraiwa CSI DeleteVolume.
  • ** Kaua rawa e whakamahi docker system prune --volumes i roto i nga tuhinga horoi aunoa, i nga kaikawe CI/CD**: Ki te tukinga poto te ipu papaa raraunga whakaputa (Exited (1)) i te wa tika ka mahi te mahi cron docker system prune -a --volumes -f, Ka whakaaro a Docker ki ana pukapuka ingoamuna "kaore i whakamahia" ka mukua ake. Tihorea --volumes mai i nga tuhinga horoi katoa.
  • Whakamahia nga whakairinga here kaihautu marama (/srv/prod_data/postgres:/var/lib/postgresql/data) ranei nga papaaarangi whakahaere-a-waho mo nga kawenga mahi whakaputa mana: Kaore i rite /var/lib/docker/volumes/, kaihautu here here-maunga whaiaronga ki waho /var/lib/docker/ e kaore i pa e docker compose down -v, docker volume rm, ranei docker system prune --volumes.

Nga nama ohorere a rohe me nga tikanga umanga

Ina pa ana tetahi ipu, te mukunga rokiroki Kubernetes ranei ki nga mahi haumanu o te hohipera, te tuku haumaru mo te iwi, me nga raraunga whaiaro kua whakaritea, ka piki tonu:

  • Te Hononga o Amerika me Kanata: Waea 911 ki te whakakorehia e te korenga o te kohinga ipu te aro turuki i nga turoro haumanu, nga punaha EHR, te tuku ohorere ranei. Panuitia te whakakore i nga hanganga ipu kino me nga maiki ipurangi ki CISA (1-844-Say-CISA / cisa.gov/report) ranei te Canadian Center for Cyber Security (1-833-CYBER-88).
  • United Kingdom: Waea 999 mo te whakatuma ki te ora, ki nga hanganga NHS nui ranei. I raro i te UK GDPR, ko te whakakorenga, te ngaro ranei o te waatea o nga raraunga whaiaro me te kore he taapiri ka taea te whakahoki mai me arotake i te panui pakaru 72-haora ki te ICO (0303 123 1113).
  • Uniana Pākehā: Waea 112 puta noa i nga whenua mema EU katoa. Ko nga hinonga kei raro i te NIS2 Directive me tuku he whakatupato moata ki to ratou CSIRT motu i roto i te 24 haora i muri mai i te mate nui o te ratonga, me te mau tonu o te ngaronga o nga raraunga whaiaro GDPR Tuhinga 33 te ripoata i roto i te 72 haora.
  • Ahitereiria me Aotearoa: Waea 000 i Ahitereiria ranei 111 kei Aotearoa; pūrongo i ngā maiki hanganga tino nui ki te ACSC (1300 CYBER1).
  • India me Ahia-Kiwa: Waea 112 i Inia mo te urupare ohorere; pūrongohia ngā maiki haumarutanga kapua, ipuipu ranei ki CERT-In (incident@cert-in.org.in) i roto i te matapihi 6-haora whakahau.

Pātai auau

E docker compose down -v ūkui i ngā maunga herenga kaihautū (./data:/var/lib/mysql) ko nga pukapuka Whakaingoa me te ingoakore Docker anake ranei?

`docker compose down -v` Ka muku ANAKE nga pukapuka kua whakaingoatia i roto i te `volumes:` wāhanga o `docker-compose.yml` me nga pukapuka ingoamu e piri ana ki nga ipu; te reira kore rawa ka muku i nga konae kei roto i te maunga herea kaihautu (pēnei i `- /opt/mysql_data:/var/lib/mysql` ranei `- ./postgres_data:/var/lib/postgresql/data`). Mena i whakamahia e to konae tito he ara kaihautu timata mai i te `/` ranei `./`, tirohia taua kōpaki kei runga i te kaihautu inaianei: kei te 100 paiheneti tonu o koutou kōnae.

Na te aha docker system prune --volumes mukua taku rōrahi pātengi raraunga ina mutu noa taku ipu mo te 2 meneti hei whakahou i te taurangi taiao?

Tatari marie mo te wehewehenga `docker stop` a `docker rm`: mehemea he ipu noa `Stopped / Exited` (ka kitea i roto `docker ps -a`), `docker system prune --volumes` mahi kaore mukua ona pukapuka na te mea kei te tohutoro tonu te ipu kua mutu. Heoi, ki te oma koe `docker rm ` (ki te hanga ano i te ipu me nga haki hou) mena kua paahi koe `docker system prune -a --volumes` (ko te tuatahi ka mukua nga ipu katoa kua mutu me katahi ka mukua nga pukapuka katoa kua kore e piri ki tetahi ipu), ka "tarewa" te rōrahi i te mirihakona ka tangohia te ipu ka mutu ka murua i muri tonu mai.

Ka taea e au te whakahoki mai i tetahi rōrahi Docker kua mukua i roto i te Papamahi Docker i runga i te Mac (Apple Silicon / Intel) Windows ranei (WSL2)?

Ae, ki te whakamutua tonutia e koe te Papamahi Docker ka mutu te whakahaere o te miihini mariko `fstrim` ki te kōpae mariko. I runga i te macOS, ka tiakina e Docker nga pukapuka Linux katoa i roto `-/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw`; ki Windows WSL2, ki roto `%LocalAppData%\Docker\wsl\disk\docker_data.vhdx`. Tāruatia tērā `.raw` ranei `.vhdx` konae ki te SSD o waho tonu, whakapirihia te panui-anake i te Linux (ma te whakamahi `losetup -Pr` mo `.raw` ranei `qemu-nbd -r` mo `.vhdx`), ka oma `ext4magic` ranei `PhotoRec` ki te mariko `ext4` wehewehe i roto.

He aha mena ka mukua e ahau tetahi Kubernetes StatefulSet me kubectl delete statefulset postgres? I mukua ano aku PVC?

Kao. Ma te taunoa i Kubernetes (mehemea `persistentVolumeClaimRetentionPolicy.whenDeleted: Delete` i tino whirihorahia i runga i te Kubernetes 1.27+), e rere ana `kubectl delete statefulset` ka mukua te kaiwhakahaere StatefulSet me ona Pods, mahue katoa `PersistentVolumeClaims` (`data-postgres-0`, `data-postgres-1`) me o ratou raraunga kei raro 100 paiheneti mau tonu. Me tono ano i to `StatefulSet` Whakaatu YAML me te ingoa kotahi me te `volumeClaimTemplates`, a ko nga Pods hou ka hono aunoa ki te waa `data-postgres-0` Ko nga PVC.

Me pehea taku uru atu ki nga konae kei roto i te Docker overlay2 whaiaronga ka kore te ipu e pai ki te timata (CrashLoopBackOff / Exited (1))?

Kaua e mukua te ipu kua pakaru. I te mea kei te noho tonu te ipu kua mutu `docker ps -a`, kua mau tonu tana paparanga tuhi me nga kohinga rōrahi ki runga kōpae. Ka taea e koe te kape tika i nga konae mai i te ipu kua mutu ma te whakamahi `docker cp :/var/lib/postgresql/data /root/rescued_data`, kimihia ranei tona kaihautu tika `overlay2` ara ma te oma `docker inspect --format '{{.GraphDriver.Data.UpperDir}}'` me te panui tika i taua whaiaronga hei `root`.

He aha mena ka mukua e tetahi tetahi Kubernetes Namespace (kubectl delete ns prod) me nga PVC katoa reclaimPolicy: Delete i runga i te AWS EBS, i te GCP ranei?

Mena ko te taraiwa CSI kapua (`ebs.csi.aws.com` ranei `pd.csi.storage.gke.io`) kua oti kē `DeleteVolume` no te mea `reclaimPolicy` ko `Delete`, tirohia kia toru nga kupenga haumaru taumata-kapua tonu: (1) tirohia ** AWS Recycle Bin mo EBS Volumes** i roto i te AWS EC2 console (mehemea he kaha te ture pupuri i runga i nga kaute / tohu, ka noho te pukapuka EBS kua mukua ki roto i te Recycle Bin ka taea te whakahoki mai i te 1 panui); (2) tirohia **AWS Backup / DLM / GCP Snapshot Schedules** mo te hopukanga rōrahi aunoa hou; me (3) tirohia mena kei te rere to papaarangi raraunga WAL/binlogs (`WAL-G`, `pgBackRest`, `Litestream`) ki te peere S3/GCS.


Nga kaiarahi e pa ana


Nga puna me nga aratohu whai mana

  • Kubernetes Tuhinga Whaimana (kubernetes.io / Cloud Native Computing Foundation): Te Ahanoa Rokiroki kei te Whakamahi Parenga (kubernetes.io/pvc-protection Whakaoti), Reclaiming PersistentVolumes (Retain vs. Delete), me te Mahi etcd Huihuinga mo Kubernetes (etcdctl snapshot restore).
  • Docker Inc. (docs.docker.com): Whakahaere Raraunga ki Docker: Rōrahi, Herea Maunga, overlay2 Hangahanga Atekōkiri Rokiroki, me te Tohutoro Tapahi.
  • Te Whakahaere Kernel Linux (kernel.org): *Ko te /proc Pūnaha Kōnae (procfs Whakatuwheratia te Whakaora Whakaahuatanga Kōnae ma /proc/<pid>/fd/) me Ext4 Tuhinga Tuhitaka *.
  • ext4magic Tuhinga Kaupapa (ext4magic.sourceforge.net): Forensic Inode and Journal Transaction Recovery on Ext3/Ext4 Filesystems.
  • National Institute of Standards and Technology (NIST) (Te Hononga o Amerika): *Te Whakaputa Motuhake 800-190: Te Aratohu Haumaru Ipu Taupānga me te Whakaora Mai.