ʻO Docker a i ʻole Kubernetes Production Volume Holoi: Hoʻopakele docker system prune --volumes, overlay2 Inodes, a me etcd Kiʻi kiʻi¶

Ke holo nei docker compose down -v, docker system prune -a --volumes, docker volume rm, a i ʻole kubectl delete pvc ma ka ʻaoʻao kuhi hewa i ka hana ʻana ʻo ia kekahi o nā ala wikiwiki loa e holoi i kahi ola PostgreSQL, MySQL, MariaDB, Redis, MongoDB, a i ʻole SQLite database volume. ʻAʻole like me nā kiʻi paʻi mīkini virtual cloud e neʻe ana i kahi pahu recycle 7 lā, ʻo ka mea hoʻokele leo kūloko o Docker (/var/lib/docker/volumes/<name>/_data) a me nā Kubernetes local-path / hostPath Hoʻopuka koke nā mea hoʻolako i ka POSIX maʻamau unlink() a rmdir() kāhea pololei i ka Linux host filesystem (ext4, XFS, a i ʻole Btrfs). ʻO kēlā me kēia kākau e hana ʻia ma kēlā host Linux ma hope o ka pau ʻana o ke kauoha e hiki ke kākau i nā poloka waihona i hoʻokuʻu ʻia ma ka disk. Inā hoʻopiha kāu disk server i 100 pakeneka (0 bytes free) ma mua o ka holo ʻana o kekahi docker system prune --volumes me ka weliweli, e kukakuka kikowaena kikowaena 100% piha, waihona WAL log bloat, a me ka hoʻihoʻi ʻana i ka ʻole; no ka holoi ʻana i nā faila Linux ma waho o nā pahu, ʻike rm -rf a i ʻole Shift-Delete ma ext4, XFS, APFS, a me NTFS: heluhelu-wale remount a me ke kālai inode a ʻO ka mīkini virtual VMDK, VHDX, a me ka qcow2 snapshot chain corruption a me ka hoʻopakele hypervisor. Inā pili kēia pau ʻana i ka mālama ʻana i ka maʻi maʻi, hoʻouna pilikia, a i ʻole nā ​​​​pūnaehana palekana pono lehulehu, e kelepona i kāu helu ulia pōpilikia kūloko (911 ma ʻAmelika Hui Pū ʻIa a me Kanada, 112 ma waena o ka European Union, 999 ma United Kingdom) a i ʻole e nānā i kā mākou papa kuhikuhi o helu pilikia.

E hana i kēia i kēia manawa¶

1. ** MAI holo docker compose up -d, systemctl restart docker, a i ʻole e hoʻihoʻi hou i ka Kubernetes pod**: Hana hou i ka pahu a hoʻomaka hou i ka Docker daemon e kākau megabytes o nā mea hou. overlay2 metadata, pahu pahu JSON log, a me nā waihona waihona waihona ʻole i hoʻomaka hou ʻia (initdb) pololei ma luna o ka pololei ext4 a i ʻole XFS nā poloka kahi i noho ai kāu leo ​​i holoi ʻia i mau kekona i hala.
2. E nānā i loko o 5 kekona inā ke kaʻina hana ipu (postgres, mysqld, redis-server, node, python) ke holo nei i ka hoʻomanaʻo (ps aux a sudo lsof +L1): Inā holoi kekahi i nā faila i loko o kahi leo i kau ʻia a wehe ʻole paha i kahi leo inoa ʻole i ka wā e holo mau ana ka pahu, mālama ka kernel Linux i 100 pakeneka o nā faila waihona i hoʻopaʻa ʻole ʻia e ola ma ka disk inā lōʻihi ka ID Process ID o ka ipu (PID) paʻa i kahi wehewehe wehewehe waihona (FD). Hiki iā ʻoe ke kope pololei i kēlā mau faila mai /proc/<PID>/fd/ a i ʻole /proc/<PID>/root/ me ka nalowale ʻikepili ʻole ma mua o ka pau ʻana o ka pahu.
3. Inā ua paʻa ka ipu a ua holoi ʻia ka leo (docker compose down -v a i ʻole docker system prune --volumes), freeze host disk writes immediately: Stop the Docker daemon (sudo systemctl stop docker.socket docker.service containerd) no laila, hoʻopau nā ipu hope a me nā ʻike olakino i ke kākau ʻana i nā moʻolelo /var/lib/docker, then either remount /var/lib/docker (a i ʻole ke kumu / ʻāpana) heluhelu-wale (sudo mount -o remount,ro /) or take an immediate block-level cloud volume snapshot (AWS EBS Snapshot, GCP Persistent Disk Snapshot, DigitalOcean Volume Snapshot) before touching anything.
4. If you are on Kubernetes (kubectl delete pvc a i ʻole kubectl delete namespace), check the PersistentVolume (PV) ReclaimPolicy (kubectl get pv) koke:
5. If the PV had persistentVolumeReclaimPolicy: Retain, your data is not deleted: the PV simply moved to Released status and your cloud disk (EBS, Azure Disk, Ceph RBD, NFS) is 100 percent intact.
6. Even if kubectl delete pvc is stuck in Terminating, Kubernetes Finalizers (kubernetes.io/pvc-protection) block the CSI driver from deleting the underlying storage as long as any Pod still references that PVC. Do not delete the Pod until you copy the data out of the running Pod.
7. Image the underlying block partition (dd / ddrescue over SSH) to an external disk or separate cloud volume before running ext4magic, extundelete, a i ʻole PhotoRec: Never run file carvers that write output files onto the same disk partition from which you are recovering /var/lib/docker/volumes.
8. No ka mokuʻāina ʻo Kubernetes cluster i ʻino a holoi ʻia paha (kubectl delete crd / namespace), kū kube-apiserver a hoʻihoʻi i kāu mea hou etcd kiʻi (etcdctl snapshot restore): Inā kū ka pūʻulu i loko etcd ua holoi ʻia i ka wā e mau ana nā waihona waihona kapuaʻi i lalo, e hoʻihoʻi ana etcd hoʻihoʻi i nā mea a pau PV, PVC, Secret, a ConfigMap mea i ka manawa like.

lāʻau hoʻoholo: holoi ʻia ka nui Docker, overlay2 papa, a i ʻole Kubernetes PVC triage¶

E hoʻohana i kēia palapala kahe e koho ai i ke ala hoʻihoʻi ʻole pohō ʻoi loa ma muli o ke ola mau ʻana o ke kaʻina hana pahu, inā paha i kākoʻo ʻia ka leo e ka diski kūloko a i ʻole ka mālama ʻana i ke ao CSI, a ʻo wai ka ʻōnaehana waihona. /var/lib/docker:

flowchart TD
    Start["Nalo ka ʻikepili hana ma hope o Docker a i ʻole Kubernetes Holoi Kauoha"] --> WhichEnv{"ʻO wai ka papahana a me ke kauoha i hoʻomaka i ka nalowale?"}

    WhichEnv -->|"Docker: Ke holo mau nei ka pahu (ʻaʻole i hoʻopili ʻia nā faila i loko o ka Volume/Container)"| ProcRescue["Hoʻopakele i nā mea wehewehe faila wehe ʻole ʻia ma o sudo lsof +L1 a me /proc/PID/fd/N a i ʻole / proc/PID/root/"]

    WhichEnv -->|"Docker: Hoʻokuʻu ʻia ka pahu a hoʻopau ʻia ka leo (komo i lalo -v / prune --volumes)"| StopDocker["Holo koke: sudo systemctl stop docker.socket docker containerd"]

    WhichEnv -->|"Kubernetes: hoʻopau ʻo kubectl i ka pvc / pv / statefulset / namespace"| K8sCheck{"Ke holo mau nei ka Pod A i ʻole Paʻa ʻo PVC i ka hoʻopau ʻana?"}

    K8sCheck -->|"ʻAe: Ke holo mau nei ka Pod (Hoʻopau ʻia ka mea hoʻopau iā CSI Holoi)"| K8sExec["MAI Holoi i ka Pod -> E holo i ka kubectl exec / pg_dump / tar ma luna o stdout i ka Local Disk koke"]
    K8sCheck -->|"ʻAʻole: Pod Gone -> Nānā kubectl loaʻa pv"| ReclaimCheck{"He aha ka PV ReclaimPolicy?"}

    ReclaimCheck -->|"E paʻa (kūlana PV = Hoʻokuʻu ʻia)"| ClearClaimRef["100% Intact Data -> Hoʻoponopono i ka PV e wehe i spec.claimRef.uid -> Hoʻopaʻa hou iā PVC hou"]
    ReclaimCheck -->|"Holoi (Cloud EBS / PD / Longhorn / Local-Path)"| CloudOrLocal{"ʻO Cloud CSI Volume a i ʻole kahi Disk Host Host?"}

    CloudOrLocal -->|"Cloud CSI (AWS EBS / GCP / Azure / Ceph)"| CloudSnap["E nānā i ka Cloud Console Recycle Bin / Nā Kiʻi Kiʻi EBS Automated / Ceph RBD Trash (rbd trash ls)"]
    CloudOrLocal -->|"Kōpae Node Kūloko (kūlana-alanui / hostPath)"| StopDocker

    StopDocker --> RemountRO["E hoʻihoʻi hou i ka ʻāpana Host Heluhelu-wale (mauna -o remount,ro) A i ʻole Stream Block Image ma luna o SSH"]
    RemountRO --> HostFS{"He aha nā Pūnaewele Pūnaewele /var/lib/docker?"}
    HostFS -->|"ext4"| Ext4Rec["E hoʻohana i nā debugfs / ext4magic Journal Inode Transaction Rollback (-a timestamp)"]
    HostFS -->|"XFS / Btrfs"| CarveRec["E hoʻohana i nā Aʻa Btrfs / xfs_irecover / Ke kālai inoa ʻana no SQLite, Postgres Heap, a i ʻole InnoDB .ibd"]

Māhele 1: Pehea ʻo Docker a me Kubernetes e hoʻopau maoli i nā puke ma ka disk¶

Ma mua o ka holo ʻana i kekahi kauoha hoʻihoʻi, e hoʻomaopopo i kahi i noho kino ai kāu ʻikepili ma ka host Linux a me ka mea i hoʻopā maoli ʻia e ke kauoha holoi:

Māhele 2: Hoʻopakele i nā faila waihona waihona mai kahi pahu e holo mau ana (/proc/<pid>/fd/)¶

E kali: ke holo mau nei ka pahu i kēia manawa? Hoʻohana pinepine ka ʻenekinia i kahi palapala neʻe noiʻi, kahi hana cron hoʻomaʻemaʻe, a i ʻole kahi manual rm -rf /var/lib/postgresql/data/* a i ʻole rm /app/data/production.sqlite3 i loko o kahi pahu e holo ana, a i ʻole e holoi i ka waihona waihona hoʻokipa hoʻokipa /opt/app/data ʻoiai ʻo ka pahu ponoʻī (postgres, mysqld, mariadbd, redis-server, mongod, gunicorn, node) ke ** mau nei a holo**.

Ma Linux, ʻaʻole holoi ʻia kahi faila mai ka disk i ka wā rm a i ʻole unlink() ua kapaia. ʻO kāna mau poloka disk a me ka inode e waiho ʻia 100 pākēneka i hoʻokaʻawale ʻia a hiki ke heluhelu ʻia a hiki i ke kaʻina hana hope e paʻa ana i kahi wehewehe wehewehe faila (FD) i kēlā inode pani a puka paha.

KaʻAnuʻu Hana 1: E huli i ka host Process ID (PID) o ke kaʻina hana ipu¶

Ma ka Linux host node (ʻaʻole i loko o ka ipu), e ʻimi i ka host PID o kāu waihona a i ʻole kaʻina hana noi:

sudo lsof +L1 | grep -E "postgres|mysqld|mariadbd|sqlite|redis|mongo|node|python|java"

COMMAND     PID   USER   FD   TYPE DEVICE  SIZE/OFF NLINK    NODE NAME
mysqld    18492    999   14u   REG  259,1 482344960     0 1849201 /var/lib/mysql/app_prod/users.ibd (deleted)
mysqld    18492    999   15u   REG  259,1 125829120     0 1849202 /var/lib/mysql/app_prod/orders.ibd (deleted)
mysqld    18492    999   16u   REG  259,1  79691776     0 1849190 /var/lib/mysql/ibdata1 (deleted)
python3   22104   1000    4u   REG  259,1  89128960     0 3102944 /data/production.sqlite3 (deleted)

KaʻAnuʻu Hana 2: kope i ka unlinked waihona waihona pololei mai /proc/<PID>/fd/¶

Mai ** ʻaʻole ** hoʻomaha a hoʻomaka hou i ka ipu. ʻOiai e holo mau ana ke kaʻina hana, e kope pololei i nā mea wehewehe faila maka i kahi waihona palekana ma kāu host:

sudo mkdir -p /root/proc_emergency_rescue

# Example 1: Rescuing an unlinked SQLite3 database (PID 22104, FD 4)
sudo cp /proc/22104/fd/4 /root/proc_emergency_rescue/production.sqlite3
# Check if PID 22104 also holds the -wal file open (e.g. FD 5) and copy it too:
sudo cp /proc/22104/fd/5 /root/proc_emergency_rescue/production.sqlite3-wal

# Example 2: Automatically rescuing ALL open (deleted) MySQL/Postgres files for PID 18492
for fd in /proc/18492/fd/*; do
    target=$(sudo readlink "$fd" 2>/dev/null)
    if echo "$target" | grep -q "(deleted)"; then
        fd_num=$(basename "$fd")
        clean_name=$(echo "$target" | sed 's/ (deleted)//; s/\//_/g')
        echo "Rescuing FD $fd_num -> $clean_name"
        sudo cp "$fd" "/root/proc_emergency_rescue/${fd_num}${clean_name}"
    fi
done

KaʻAnuʻu Hana 3: ʻOi aku ka maikaʻi: E holo i kahi dump logical ola (pg_dump / mysqldump / .backup) ma mua o ka pani ʻana o ka pahu i nā FD¶

Ina postgres a i ʻole mysqld ke holo nei me kāna mau waihona ʻikepili i wehe ʻia i ka hoʻomanaʻo a /proc/<PID>/fd/, e hoʻāʻo e holo i kahi dump logical ola ma luna o ke kumu unix a i ʻole ka pilina TCP ma mua:

# If the database engine can still read its open file descriptors:
docker exec <container_name> pg_dumpall -U postgres > /root/proc_emergency_rescue/live_dump.sql
# Or for MySQL:
docker exec <container_name> mysqldump -u root -p --all-databases --single-transaction > /root/proc_emergency_rescue/live_mysql.sql

Māhele 3: Hoʻihoʻi hou i kahi leo Docker i holoi ʻia ext4 a i ʻole XFS mahope iho o ka pau ana o ka ipu¶

Pehea inā holo ʻoe docker compose down -v a i ʻole docker system prune --volumes, no laila ua ʻoki ʻia ka ipu ** a me nā mea a pau /proc/<pid>/fd/ pani ʻia nā mea wehewehe? I kēia manawa aia ʻoe i ka heihei e kūʻē i ka papa kuhikuhi disk kākau ma ka Linux host.

Step 1: Freeze the Linux host filesystem within 10 seconds¶

1. Stop Docker, containerd, kubelet, and cron immediately so no container, healthcheck, or log rotator writes another byte to /var/lib/docker:

   sudo systemctl stop docker.socket docker.service containerd.service cron.service
   

2. Check which block device and filesystem holds /var/lib/docker:

   df -Th /var/lib/docker
   

   Filesystem     Type  Size  Used Avail Use% Mounted on
   /dev/nvme0n1p1 ext4  100G   42G   58G  42% /
   

3. Remount the partition Read-Only (or stream a raw partition image over SSH to another machine):
4. Ina /var/lib/docker is on its own dedicated partition or LVM volume (for example /dev/nvme0n1p2 a i ʻole /dev/mapper/vg0-docker), remount it read-only immediately:

   sudo mount -o remount,ro /var/lib/docker
   

5. Ina /var/lib/docker is on the root / partition (/), try forcing the root partition read-only using the kernel Magic SysRq trigger (echo u | sudo tee /proc/sysrq-trigger) a i ʻole sudo mount -o remount,ro /.
6. If you are on a cloud VPS (AWS EC2, GCP, DigitalOcean, Hetzner, Linode): Immediately go to your cloud web console and click Create Snapshot on the server's block volume (or stream the raw block device over SSH to a second server: sudo dd if=/dev/nvme0n1p1 bs=4M status=progress | gzip -1 | ssh user@backup-server "cat > /mnt/storage/docker_host_raw.img.gz").

Step 2: Check whether your data is still sitting in ANOTHER unpruned volume or overlay2 layer¶

Before carving unallocated disk blocks, verify that your database wasn't actually stored in a bind mount or a differently named volume that survived:

# Search the entire /var/lib/docker tree read-only for any surviving database files:
sudo find /var/lib/docker -name "PG_VERSION" -o -name "ibdata1" -o -name "*.sqlite*" -o -name "*.db" -o -name "dump.rdb" -o -name "WiredTiger" 2>/dev/null

Step 3: Recovering unlinked Docker volume directories on ext4 using the ext4 Journal (ext4magic a debugfs)¶

When docker volume rm deletes /var/lib/docker/volumes/<vol_name>/_data on an ext4 filesystem, ext4 zeros out the direct block pointers (i_block / extent tree root) in the live inode table. However, the ext4 journal (inode <8>, maʻamau 128 MB i 1024 MB circular ring buffer) still holds historic copies of the exact inode tables and directory blocks from right before docker compose down -v was executed.

+-------------------------------------------------------------------------+
| HOW ext4magic RECOVERS A DELETED DOCKER VOLUME FROM THE EXT4 JOURNAL    |
|                                                                         |
| Live Inode Table on Disk (After docker volume rm):                      |
|   Inode #918402 (users.ibd): [Size = 0, Extent Pointers = ZEROED]       |
|                                                                         |
| Ext4 Journal Ring Buffer (Inode <8> on Disk):                           |
|   Transaction #41902 (10 mins ago):                                     |
|   Inode #918402 (users.ibd): [Size = 482MB, Extents = Blocks 9000..9999]|
|                                                                         |
| -> ext4magic reads the old Inode Copy from Journal Transaction #41902   |
|    and extracts Blocks 9000..9999 directly to your rescue drive.        |
+-------------------------------------------------------------------------+

Holo ext4magic against your read-only partition (or the .img snapshot copy on a second server), specifying a timestamp (-a) from 5 minutes before you ran docker compose down -v a i ʻole docker system prune --volumes:

# Install ext4magic on your rescue machine (never on the affected partition)
sudo apt-get install -y ext4magic e2fsprogs

# Dump the ext4 journal from the frozen partition / image to a file in RAM (/dev/shm)
sudo debugfs -R "dump <8> /dev/shm/ext4_journal.copy" /dev/nvme0n1p1

# Recover all files deleted under var/lib/docker/volumes after (Current Time - 15 minutes)
sudo mkdir -p /mnt/external_rescue/docker_recovered
sudo ext4magic /dev/nvme0n1p1 \
  -j /dev/shm/ext4_journal.copy \
  -f var/lib/docker/volumes \
  -a $(date -d "-15 minutes" +%s) \
  -r -d /mnt/external_rescue/docker_recovered

KaʻAnuʻu Hana 4: Ke kālai ʻana i nā faila SQLite maka, PostgreSQL, a i ʻole MySQL InnoDB i ka wā i ʻōwili ʻia ai ka puke pai (XFS a i ʻole ext4)¶

Inā hoʻohana ka mea hoʻokipa XFS aiʻole ināʻo ka ext4 puke pai i ʻōwili ʻia ma luna o nā kuhikuhi inode, e hoʻohana i ka pūlima a me ke kālai ʻana i ke kālai ʻia e kūʻē i ke kiʻi diski maka (docker_host_raw.img):

1. ** No nā ʻikepili SQLite3 (.sqlite, .db - laʻa., Vaultwarden, Nextcloud, Kokua Home, Gitea, Grafana)**: Hoʻomaka kēlā me kēia waihona SQLite3 me kahi poʻomanaʻo ASCII he 16-byte ma ka byte 0 o kāna ʻaoʻao mua 4 kilobyte: 53 51 4c 69 74 65 20 66 6f 72 6d 61 74 20 33 00 ("SQLite format 3\0"), a mālama i ka nui o ka ʻaoʻao waihona waihona (Bytes 16..17, maʻamau 4096) a me ka nui o ka waihona ma nā ʻaoʻao (Bytes 28..31, nui-endian 32-bit integer) i loko pono o kēlā poʻomanaʻo.
2. Holo PhotoRec me [File Opt] -> [x] sqlite hoʻohana ʻia, a nānā pono i ka ʻāpana me grep -ab "SQLite format 3" e ʻimi i ka offset byte pololei o kāu faila SQLite i holoi ʻia a unuhi ʻia me dd:

   # Find the exact byte offsets of every "SQLite format 3" header on the partition:
   sudo grep -ab "SQLite format 3" /dev/nvme0n1p1
   

   Once extracted, holo sqlite3 carved.db ".recover" | sqlite3 repaired.db e hana hou i kēlā me kēia ʻaoʻao lau lāʻau B paʻa.

1. ** No MySQL / MariaDB InnoDB (.ibd papa waihona)**: ʻO kēlā me kēia InnoDB .ibd ʻO ka papa ʻaina he sequential 16 KB (16,384 byte) nā ʻaoʻao i loaʻa ka FIL_PAGE_INDEX (0x45BF) ke poʻomanaʻo a me ka ʻokoʻa o ka papaʻaina space_id. E hoʻohana i ka Percona / TwinDB undrop-for-innodb (stream_parser + c_parser) mea paahana no ka scan /dev/nvme0n1p1 no nā mea a pau 16 KB Nā ʻaoʻao kuhikuhi InnoDB a hoʻolei pololei i kēlā me kēia lālani i nā faila TSV/SQL no ka lawe hou ʻana.
2. ** No nā waihona Puʻu PostgreSQL ($PGDATA/base/<db_oid>/<relfilenode>)**: Mālama ʻo PostgreSQL i kēlā me kēia pākaukau ma ke ʻano pili 8 KB (8,192 byte) nā ʻaoʻao puʻu i loaʻa PageHeaderData (pd_lower, pd_upper, pd_special). E hoʻohana pg_filedump aiʻole pglazarus / pdu e nānā i nā ʻāpana disk i hoʻokaʻawale ʻole ʻia no 8 KB ʻaoʻao e pili ana i ka hoʻolālā tuple kolamu o kāu papaʻaina (HeapTupleHeaderData) a hana hou i nā lālani.

Māhele 4: Kubernetes PVC, PV, a me etcd Paʻi Paʻi Paʻi Paʻi¶

Ma Kubernetes (EKS, GKE, AKS, k3s, RKE2, kubeadm, OpenShift), e holoi wale ana i kahi PersistentVolumeClaim (kubectl delete pvc), he StatefulSet, a i ʻole kahi Namespace holoʻokoʻa (kubectl delete ns prod) hahai i kahi hoʻolālā ʻokoʻa loa ma mua o Docker kūʻokoʻa.

Hōʻike A: Hōʻike ka PVC Terminating (The Finalizer Lifesaver)¶

Holo:

kubectl get pvc -n <namespace>

# Stream a compressed tar backup or pg_dump out of the still-running pod immediately:
kubectl exec -n <namespace> <pod-name> -- pg_dumpall -U postgres > ./rescued_k8s_db.sql
# Or tar an entire mounted volume directory out over stdout:
kubectl exec -n <namespace> <pod-name> -- tar czf - /var/lib/postgresql/data > ./rescued_pvc_files.tar.gz

Scenario B: Ua holoi ʻia ka PVC, a kubectl get pv hōʻike i ka PersistentVolume i Released mokuʻāina (ReclaimPolicy: Retain)¶

Holo:

kubectl get pv

Pehea e hana ai i kahi Released PV Available hou i loko o 10 kekona me ka nalowale ʻikepili ʻole: 1. Hoʻoponopono i ka mea PersistentVolume a holoi i ka spec.claimRef poloka:

kubectl patch pv <your-pv-name> --type json -p '[{"op": "remove", "path": "/spec/claimRef"}]'

Hōʻike C: Hoʻihoʻi hou i kahi mokuʻāina ʻo Kubernetes i holoi ʻia mai kahi etcd kiʻi (etcdctl snapshot restore)¶

Pehea inā holo kekahi kubectl delete namespace prod a i ʻole helm uninstall a holoi aku i 200 mau hōʻike Kubernetes, Nā mea huna, ConfigMaps, a me nā wehewehe PV ma kahi hoʻokele ponoʻī. kubeadm, k3s, a i ʻole RKE2 mokulele hoʻomalu?

1. ** E huli i kāu mea hou loa etcd snapshot** (ma ka paʻamau i /var/lib/rancher/k3s/server/db/snapshots/, /var/lib/rancher/rke2/server/db/snapshots/, a i ʻole kāu hoʻonohonoho /var/backups/etcd/snapshot.db).
2. E hōʻoia i ka pololei o ka paʻi kiʻi ma mua o ka pā ʻana i ka pūʻulu ola:

   ETCDCTL_API=3 etcdctl snapshot status /path/to/snapshot.db --write-out=table
   

3. ** Kū kube-apiserver a etcd ma ka piko o ka mokulele hoʻomalu**:
4. Ma kubeadm (nā pahu paʻa): Neʻe /etc/kubernetes/manifests/kube-apiserver.yaml a /etc/kubernetes/manifests/etcd.yaml no kekahi manawa i /root/manifests_hold/ pela kubelet Hoʻopau maʻemaʻe nā pahu ʻelua, a laila hoʻi i luna /var/lib/etcd i /var/lib/etcd.bak.
5. Ma k3s: Holo sudo systemctl stop k3s a sudo k3s server --cluster-reset --cluster-reset-restore-path=/path/to/snapshot.
6. ** Hoihoi i ka etcd papa kuhikuhi ʻikepili ma kubeadm**:

   sudo ETCDCTL_API=3 etcdctl snapshot restore /path/to/snapshot.db \
     --data-dir=/var/lib/etcd \
     --name=<control-plane-node-name> \
     --initial-cluster=<node-name>=https://<node-ip>:2380 \
     --initial-advertise-peer-urls=https://<node-ip>:2380
   

7. Neʻe etcd.yaml a kube-apiserver.yaml hoʻi i loko /etc/kubernetes/manifests/. I loko o 30 kekona, kube-apiserver hoʻopili hou i ka mea i hoʻihoʻi ʻia etcd hale kūʻai a ʻike hou ʻia kēlā me kēlā me kēia Namespace, Secret, ConfigMap, a me PersistentVolume.

Māhele 5: Hoʻopaʻa i nā puke Docker a me Kubernetes e kūʻē i ka holoi ʻia ʻana¶

Ke hoʻihoʻi hou ʻia kāu ʻikepili hana, e hoʻokō i kēia mau pale kiaʻi ʻehā i hiki ʻole i kahi typo CLI hoʻokahi ke holoi hou i kahi leo ola:

• Hoʻololi i kēlā me kēia hana Kubernetes StorageClass mai reclaimPolicy: Delete i reclaimPolicy: Retain: Ma ka paʻamau, kūkulu ʻia i loko o ke ao StorageClasses (gp2, gp3, standard-rwo) paʻamau i reclaimPolicy: Delete, ʻo ia hoʻi ka holoi ʻana i kahi PVC e hoʻopau i ka diski ao. Hana i kahi StorageClass me reclaimPolicy: Retain (a i ʻole e hoʻopili i nā PV i loaʻa me kubectl patch pv <pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}') no laila ʻo ka holoi ʻana i kahi PVC a i ʻole Namespace ʻaʻole e holoi i ka leo poloka lalo.
• E ho'ā i ka AWS EBS Recycle Bin / GCP Snapshot Schedules / Velero me CSI Snapshots: E ho'ā ana i ka AWS EBS Recycle Bin e mālama i kekahi leo EBS i holoi ʻia no 7 a 30 mau lā ʻoiai ke kelepona mai ka mea hoʻokele CSI. DeleteVolume.
• ** Mai hoʻohana docker system prune --volumes i nā palapala hoʻomaʻemaʻe ʻakomi a i ʻole nā ​​mea holo CI/CD**: Inā hāʻule pōkole kahi pahu waihona waihona (Exited (1)) i ka manawa kūpono e holo ai kahi hana cron docker system prune -a --volumes -f, Noʻonoʻo ʻo Docker i kāna mau puke inoa ʻole "hoʻohana ʻole ʻia" a hoʻopau mau iā lākou. Kaʻi --volumes mai nā palapala hoʻomaʻemaʻe a pau.
• E hoʻohana i nā puʻu hoʻopaʻa hoʻokipa (/srv/prod_data/postgres:/var/lib/postgresql/data) a i ʻole nā ​​ʻikepili i hoʻokele ʻia i waho no nā haʻahaʻa hana hana mokuʻāina: ʻAʻole like /var/lib/docker/volumes/, host bind-mount directories ma waho /var/lib/docker/ ʻaʻole i hoʻopā ʻia e docker compose down -v, docker volume rm, a i ʻole docker system prune --volumes.

Nā helu ulia pōpilikia āpana a me nā protocols oihana¶

Ke hoʻopilikia ka pahu a i ʻole ka holoi ʻana o ka waihona Kubernetes i nā hana lapaʻau o ka haukapila, ka hoʻouna palekana lehulehu, a i ʻole ka ʻikepili pilikino i hoʻoponopono ʻia, e piʻi koke:

• ** ʻAmelika Hui Pū ʻIa a me Kanada **: Kāhea 911 inā hoʻopau ka pahu pahu pahu i ka nānā ʻana i ka maʻi maʻi, ʻōnaehana EHR, a i ʻole ka hoʻouna pilikia. E hōʻike i ka holoi ʻana i nā mea hana ipu ʻino a i ʻole nā ​​hanana pūnaewele i CISA (1-844-Say-CISA / cisa.gov/report) a i ʻole Canadian Center for Cyber ​​Security (1-833-CYBER-88).
• United Kingdom: Kāhea 999 no ka hoʻoweliweli ʻana i ke ola a i ʻole ka hana koʻikoʻi NHS. Ma lalo o UK GDPR, ʻo ka luku mau ʻana a i ʻole ka nalowale o ka loaʻa ʻana o ka ʻikepili pilikino me ka ʻole o ka hoʻihoʻi hou ʻana, pono e loiloi i kahi leka hoʻomaopopo he 72 mau hola i ka ICO (0303 123 1113).
• Uni ʻEulopa: Kāhea 112 ma nā mokuʻāina lālā a pau o EU. Pono nā hui ma lalo o ka NIS2 Directive e hoʻouna i kahi ʻōlelo aʻo mua i ko lākou CSIRT aupuni i loko o 24 mau hola ma hope o ka pau ʻana o ka lawelawe, a ʻo ka pau ʻana o ka ʻikepili pilikino mau ke kumu GDPR Article 33 hōʻike i loko o 72 hola.
• Australia a me New Zealand: Kāhea 000 ma Australia a i ʻole 111 ma Aotearoa; hōʻike i nā hanana koʻikoʻi i ka ACSC (1300 CYBER1).
• India a me Asia-Pākīpika: Kāhea 112 ma Inia no ka pane pilikia; e hōʻike i nā hanana palekana ao a ipu paha iā CERT-In (incident@cert-in.org.in) i loko o ka pukaaniani 6 hola.

Nīnau pinepine¶

Ke hana nei docker compose down -v holoi i nā puʻu hoʻopaʻa hoʻokipa (./data:/var/lib/mysql) a i ʻole nā ​​inoa ʻo Name a me Anonymous Docker wale nō?¶

`docker compose down -v` Hoʻopau WALE i nā puke inoa i haʻi ʻia ma ka `volumes:` pauku o `docker-compose.yml` a me nā puke inoa ʻole i hoʻopili ʻia i nā pahu; ia aole loa holoi i nā faila i loko o kahi puʻu hoʻopaʻa hoʻokipa (e like me `- /opt/mysql_data:/var/lib/mysql` a i ʻole `- ./postgres_data:/var/lib/postgresql/data`). Inā hoʻohana kāu faila haku i kahi ala hoʻokipa e hoʻomaka ana me `/` a i ʻole `./`, e nānā i kēlā waihona ma ka mea hoʻokipa i kēia manawa: paʻa mau kāu mau faila i 100 pakeneka.

No ke aha docker system prune --volumes Holoi i kaʻu waihona waihona i ka wā i hoʻōki wale ai au i ka pahu no 2 mau minuke e hoʻonui i kahi loli kaiapuni?¶

E kali pono no ka hoʻokaʻawale ʻana ma waena `docker stop` a `docker rm`: inā he pahu wale `Stopped / Exited` (ʻike ʻia ma `docker ps -a`), `docker system prune --volumes` hana ʻaʻole holoi i kāna mau puke no ka mea ke kuhikuhi mau nei ka pahu i kū iā lākou. Eia naʻe, inā ʻoe i holo `docker rm ` (e hana hou i ka ipu me nā hae hou) a i ʻole ʻoe i hala `docker system prune -a --volumes` (ʻo ia ka mea e holoi mua ai i nā pahu pahu a pau alaila holoi i nā puke a pau ʻaʻole i hoʻopili hou ʻia i kekahi pahu), lilo ka leo i "huli" i ka millisecond ua wehe ʻia ka pahu i pani ʻia a holoi koke ʻia ma hope.

Hiki iaʻu ke hoʻihoʻi i kahi leo Docker i hoʻopau ʻia i loko o Docker Desktop ma kahi Mac (Apple Silicon / Intel) a i ʻole Windows (WSL2)?¶

ʻAe, inā haʻalele koke ʻoe i ka Docker Desktop no laila ua pau ka holo ʻana o ka mīkini virtual `fstrim` kūʻē i ka disk virtual. Ma ka macOS, mālama ʻo Docker i nā puke Linux āpau i loko `-/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw`; ma Windows WSL2, i loko `%LocalAppData%\Docker\wsl\disk\docker_data.vhdx`. E kope i kēlā `.raw` a i ʻole `.vhdx` faila i kahi SSD waho koke, e hoʻopili iā ia i heluhelu-wale ma Linux (hoʻohana `losetup -Pr` no ka mea `.raw` a i ʻole `qemu-nbd -r` no ka mea `.vhdx`), a holo `ext4magic` a i ʻole `PhotoRec` kūʻē i ka virtual `ext4` paku i loko.

Pehea inā ua holoi wau i kahi Kubernetes StatefulSet me kubectl delete statefulset postgres? Ua holoi anei ia i kaʻu mau PVC?¶

ʻAʻole. Ma ka paʻamau ma Kubernetes (koe naʻe `persistentVolumeClaimRetentionPolicy.whenDeleted: Delete` ua hoʻonohonoho pono ʻia ma Kubernetes 1.27+), e holo ana `kubectl delete statefulset` holoi i ka mea hoʻoponopono StatefulSet a me kāna Pods, waiho ana i na mea a pau `PersistentVolumeClaims` (`data-postgres-0`, `data-postgres-1`) a me kā lākou ʻikepili koʻikoʻi he 100 pakeneka paʻa. E hoʻopili hou i kāu `StatefulSet` Hōʻike ʻia ʻo YAML me ka inoa like a `volumeClaimTemplates`, a e hoʻopili hou nā Pods hou i ka mea i loaʻa `data-postgres-0` Nā PVC.

Pehea wau e komo ai i nā faila i loko o kahi Docker overlay2 papa kuhikuhi inā hōʻole ka ipu e hoʻomaka (CrashLoopBackOff / Exited (1))?¶

Mai holoi i ka ipu i hāʻule. ʻOiai e noho mau ana ka pahu i kū i loko `docker ps -a`, ua paʻa loa kona papa kākau a me nā puʻu leo ma ka diski. Hiki iā ʻoe ke kope pololei i nā faila ma waho o ka pahu i kū me ka hoʻohana ʻana `docker cp :/var/lib/postgresql/data /root/rescued_data`, a i ʻole e ʻimi i kona pūʻali pololei `overlay2` ala ma ka holo `docker inspect --format '{{.GraphDriver.Data.UpperDir}}'` a heluhelu pololei i kēlā papa kuhikuhi e like me `root`.

Pehea inā holoi kekahi i kahi Kubernetes Namespace (kubectl delete ns prod) a me nā PVC a pau i loaʻa reclaimPolicy: Delete ma AWS EBS a i ʻole GCP?¶

Inā ʻo ka mea hoʻokele CSI kapua (`ebs.csi.aws.com` a i ʻole `pd.csi.storage.gke.io`) ua hoʻokō ʻia `DeleteVolume` no ka mea `reclaimPolicy` ua `Delete`, e nānā koke i nā ʻupena palekana ʻekolu: (1) e nānā i ka **AWS Recycle Bin for EBS Volumes** ma ka console AWS EC2 (inā ua paʻa ka lula paʻa ma ka moʻohelu/tag, noho ka leo EBS i hoʻopau ʻia i ka Recycle Bin a hiki ke hoʻihoʻi ʻia ma 1 kaomi); (2) e nānā i **AWS Backup / DLM / GCP Snapshot Schedules** no ka kiʻi paʻi leo automate hou loa; a (3) e nānā inā e kahe ana kāu waihona WAL/binlogs (`WAL-G`, `pgBackRest`, `Litestream`) i kahi bakeke S3/GCS.

Nā alakaʻi pili¶

• Kokoke kikowaena 100% piha (0 bytes manuahi): ʻāwīwī WAL log bloat, lāʻau B ʻino, a me ka hoʻihoʻi ʻana
• ʻO ka rm -rf a i ʻole Shift+Delete ma Linux, macOS, a i ʻole Windows: heluhelu-wale nō remount a me ke kālai inode.
• ʻO ka mīkini uila VMDK, VHDX, a me ka qcow2 snapshot chain corruption a me ka hoʻopakele hypervisor
• Hana ʻia ʻo Git i nā kī AWS, faila .env, a i ʻole kī SSH pilikino i ka repo GitHub lehulehu.
• ʻO ka pilikia hoʻolaha pūnaewele, ka neʻe ʻana, a me ka hoʻihoʻi ʻana i ke kākoʻo
• Kākoʻo kamepiula, ka hoʻolālā 3-2-1, a me ka hoʻihoʻi ʻana i ka pōʻino
• Nā helu pilikia ma ka ʻāina

Nā kumu a me ke alakaʻi kūhelu¶

• Kubernetes Palapala Kūlana (kubernetes.io / Cloud Native Computing Foundation): Ka mea mālama i ka hoʻohana palekana (kubernetes.io/pvc-protection Nā Mea Hoʻopau), Hoʻihoʻi hou i nā Volume Persistent (Retain vs. Delete), a me ka hana etcd Huina no Kubernetes (etcdctl snapshot restore).
• Docker Inc. (docs.docker.com): E hoʻokele i ka ʻikepili ma Docker: Volumes, Bind Mounts, overlay2 Hoʻolālā Keaukaha Waihona, a me Prune Reference.
• ʻO ka Linux Kernel Organization (kernel.org): Ka /proc Pūnaehana waihona (procfs E wehe i ka File Descriptor Recovery ma o /proc/<pid>/fd/) a me Ext4 Journal Specification.
• ext4magic Palapala Papahana (ext4magic.sourceforge.net): Forensic Inode and Journal Transaction Recovery on Ext3/Ext4 Filesystems.
• National Institute of Standards and Technology (NIST) (ʻAmelika Hui Pū ʻIa): * Hoʻolaha Kūikawā 800-190: Ke alakaʻi palekana no ka pahu pahu noi a me ka hoʻihoʻi ʻana i nā hanana *.