Docker anaa Kubernetes Production Volume a wɔapopa: Nkwagye docker system prune --volumes, . overlay2 Inodes, ne etcd Mfonini ahorow a wɔayɛ no ntɛm¶

Mmirikatu a wɔde tu mmirika docker compose down -v, . docker system prune -a --volumes, . docker volume rm, anaa kubectl delete pvc wɔ terminal tab a ɛnteɛ so wɔ production mu no yɛ akwan a ɛyɛ ntɛm sen biara a wɔfa so popa PostgreSQL, MySQL, MariaDB, Redis, MongoDB, anaa SQLite database volume a ɛte ase no mu biako. Nea ɛnte sɛ mununkum virtual mfiri mfonini ahorow a ɛkɔ nnafua 7 recycle bin no, Docker mpɔtam hɔ volume draiver (/var/lib/docker/volumes/<name>/_data) ne Kubernetes local-path / . hostPath provisioners de standard POSIX ma ntɛm ara unlink() ne rmdir() frɛ tẽẽ tia Linux host fael nhyehyɛe a ɛwɔ ase no (ext4, . XFS, anaa Btrfs). Kyerɛw biara a ɛbɛba wɔ saa Linux host no so bere a ahyɛde no awie no betumi akyerɛw database blocks a wɔade wɔn ho no so wɔ disk so. Sɛ wo server disk no ayɛ ma akodu ɔha biara mu nkyem 100 (0 bytes free) ansa na obi retu mmirika docker system prune --volumes wɔ ehu mu no, susuw ho server disk 100% ayɛ ma, database WAL log bloat, ne zero-space sanba; sɛ wopɛ bare-metal Linux fael a wɔpopa wɔ nsukorabea akyi a, hwɛ akwanhyia mu rm -rf anaa Shift-Delete wɔ ext4, XFS, APFS, ne NTFS: akenkan nkutoo remount ne inode carving ne virtual machine VMDK, VHDX, ne qcow2 snapshot nkɔnsɔnkɔnsɔn asɛe ne hypervisor nkwagye. Sɛ saa nneɛma a wɔyɛ no twam yi ka ayaresabea ayarefo hwɛ, wɔn a wɔde wɔn kɔ ntɛm ara, anaa ɔmanfo ahobammɔ nhyehyɛe ahorow a, frɛ wo mpɔtam hɔ nɔma a egye ntɛmpɛ (911 wɔ United States ne Canada no, . 112 wɔ Europa Aman Nkabom no nyinaa mu, . 999 wɔ United Kingdom) anaa hwɛ yɛn directory of nɔma ahorow a egye ntɛmpɛ.

Yɛ eyi mprempren¶

1. MMUNTU mmirika docker compose up -d, . systemctl restart docker, anaa san de Kubernetes pod no di dwuma: Sɛ wosan bɔ ade no anaasɛ wosan hyɛ Docker daemon no ase a, ɛkyerɛw megabytes foforo overlay2 metadata, container JSON logs, ne database fael a hwee nni mu a wɔahyɛ aseɛ foforɔ (initdb) tẽẽ wɔ nea ɛyɛ pɛpɛɛpɛ no so ext4 anaasɛ XFS blocks baabi a na wo deleted volume no te hɔ sikɔne kakraa bi a atwam ni.
2. Hwɛ wɔ sikani 5 mu sɛ adekorade no yɛ adwuma (postgres, . mysqld, . redis-server, . node, . python) da so ara reyɛ adwuma wɔ memory (ps aux ne sudo lsof +L1): Sɛ obi popa fael ahorow a ɛwɔ volume a wɔde ahyɛ so mu anaasɛ ɔyii volume a wɔmmɔ din bere a container no da so ara reyɛ adwuma a, Linux kernel no ma ** database fael a enni link no ɔha biara mu nkyem 100 tra ase wɔ disk so** bere tenten a container no host Process ID (PID) kura fael nkyerɛkyerɛmu a wɔabue ( .FD). Wubetumi ayɛ saa fael ahorow no ho kɔpi tẽẽ afi mu /proc/<PID>/fd/ anaasɛ /proc/<PID>/root/ a data a ɛyera zero ansa na ade a wɔde gu mu no agyae.
3. Sɛ wɔagyae ade a wɔde gu mu no dedaw na wɔapopa nne no a (docker compose down -v anaasɛ docker system prune --volumes), freeze host disk kyerɛw ntɛm ara: Gyae Docker daemon (sudo systemctl stop docker.socket docker.service containerd) enti akyi nsukorade ne healthchecks gyae logs kyerɛw kɔ mu /var/lib/docker, afei anaasɛ remount /var/lib/docker (anaasɛ ntini no / mpaapaemu) akenkan nkutoo ( .sudo mount -o remount,ro /) anaasɛ fa block-level cloud volume snapshot ntɛm ara (AWS EBS Snapshot, . GCP Persistent Disk Snapshot, . DigitalOcean Volume Snapshot) ansa na wode wo nsa aka biribiara.
4. Sɛ wowɔ Kubernetes (kubectl delete pvc anaasɛ kubectl delete namespace), hwɛ PersistentVolume ( ) no mu.PV) . ReclaimPolicy (kubectl get pv) ntɛm ara:
5. Sɛ a PV na wɔ persistentVolumeReclaimPolicy: Retain, wo data no wɔnpopa: the PV kɛkɛ na wotu kɔɔ hɔ Released tebea ne wo mununkum disk (EBS, . Azure Disk, . Ceph RBD, . NFS) no yɛ ɔha biara mu nkyem 100 a enni dɛm.
6. Sɛ mpo a kubectl delete pvc no akɔhyɛ ** .Terminating**, Kubernetes Wɔde awie (kubernetes.io/pvc-protection) siw CSI draiver no kwan sɛ ɔmpopa adekorabea a ɛwɔ ase no bere tenten a Pod biara da so ara kyerɛ saa PVC no. Mpopa Pod no kosi sɛ wobɛ copy data no afi Pod a ɛretu mmirika no mu.
7. Mfonini a ɛwɔ ase block mpaapaemu (dd / . ddrescue wɔ SSH so) kɔ abɔnten disk anaa mununkum volume a ɛtetew mu ansa na woatu mmirika ext4magic, . extundelete, anaa PhotoRec: Mma nnyɛ fael carvers a ɛkyerɛw output fael ahorow wɔ disk partition koro no ara a woresan afi mu no so da /var/lib/docker/volumes.
8. Ma Kubernetes cluster tebea a asɛe anaasɛ wɔapopa wɔ akwanhyia mu (kubectl delete crd / . namespace), gyae kube-apiserver na san fa wo nea aba foforo no ba etcd mfonini a wɔde twaa mfonini (etcdctl snapshot restore): Sɛ cluster tebea no wɔ etcd no popaa bere a cloud storage volumes a ɛwɔ ase no da so ara wɔ hɔ, na ɛsan de ba etcd ɔsan nya ne nyinaa PV, . PVC, . Secret, na ConfigMap nneɛma a wɔde di dwuma bere koro mu.

--- .

Gyinaesi dua: wɔapopa Docker volume, . overlay2 layer, anaa Kubernetes PVC triage a wɔde hyɛ mu¶

Fa saa flowchart yi paw zero-loss recovery kwan a ɛyɛ ntɛm sen biara a egyina sɛ ebia container process no da so te ase, sɛ ebia na local disk anaa cloud CSI storage na ɛbɔ volume no akyi, ne filesystem hosts bɛn /var/lib/docker: 1 .

flowchart TD
    Start["Production Data a Ɛyera Wɔ Docker anaa Kubernetes Popa Ahyɛde akyi"] --> WhichEnv{"Platform & Command Bɛn na Ɛkanyan Adehwere no?"}

    WhichEnv -->|"Docker: Container Da so ara reyɛ adwuma (Fael a Ɛnni Nkitahodi wɔ Volume/Container Mu)"| ProcRescue["Gye Bue Fael Nkyerɛkyerɛmu a Ɛnni Nkitahodi denam sudo lsof +L1 ne /proc/PID/fd/N anaa /proc/PID/root/ so."]

    WhichEnv -->|"Docker: Wɔagyae Container & Wɔapopa Volume (hyehyɛ no ase -v / prune --volumes) ."| StopDocker["Ntɛm ara Run: sudo systemctl gyae docker.socket docker containerd"]

    WhichEnv -->|"Kubernetes: kubectl popa pvc / pv / tebea a wɔahyehyɛ / dinbea"| K8sCheck{"So Pod no Da so ara Tu Mmirikatu ANAA PVC Akyira wɔ Terminating mu?"}

    K8sCheck -->|"Yiw: Pod Da so ara Retu mmirika (Finalizer Siw CSI Popa) ."| K8sExec["MMƆ Pod Pod -> Fa kubectl exec / pg_dump / tar kɔ stdout so kɔ Local Disk Ntɛm ara"]
    K8sCheck -->|"Dabi: Pod Gone -> Hwɛ kubectl nya pv"| ReclaimCheck{"Dɛn ne PV ReclaimPolicy?"}

    ReclaimCheck -->|"Retain (PV Gyinabea = Wɔagyae) ."| ClearClaimRef["100% Data Intact -> Sesa PV na Yi spec.claimRef.uid -> San kyekyere PVC Foforo"]
    ReclaimCheck -->|"Popa (Mununkum EBS / PD / Longhorn / Mpɔtam-Kwan) ."| CloudOrLocal{"Cloud CSI Volume anaasɛ Mpɔtam Hɔ Host Disk?"}

    CloudOrLocal -->|"Mununkum CSI (AWS EBS / GCP / Azure / Ceph) ."| CloudSnap["Hwɛ Cloud Console Recycle Bin / Automated EBS Snapshots / Ceph RBD Nwura (rbd nwura ls) ."]
    CloudOrLocal -->|"Local Node Disk (mpɔtam-kwan / hostPath) ."| StopDocker

    StopDocker --> RemountRO["San fa Host Partition Akenkan-Nko (mount -o remount,ro) ANAA Stream Block Mfonini no wɔ SSH so"]
    RemountRO --> HostFS{"Fael Nhyehyɛe bɛn na Ɛgye /var/lib/docker?"}
    HostFS -->|"nkyekyɛmu4"| Ext4Rec["Fa debugfs / ext4magic Journal Inode Nkitahodi Rollback (-bere nsɔano) di dwuma ."]
    HostFS -->|"XFS / Btrfs a ɛwɔ hɔ"| CarveRec["Fa Btrfs Ntini / xfs_irecover / Nsaano Nkyerɛwee a Wɔayɛ no Amanneɛbɔ di dwuma ma SQLite, Postgres Heap, anaa InnoDB .ibd"]

--- .

Ɔfa 1: Sɛnea Docker ne Kubernetes popa volume ahorow a ɛwɔ disk so ankasa¶

Ansa na wobɛma recovery command biara ayɛ adwuma no, te baabi a wo data no te honam fam wɔ Linux host no so ne nea delete command no kaa no ankasa ase:

--- .

Ɔfa 2: Database fael a wɔabue a wobegye afi adekorabea a ɛda so ara yɛ adwuma mu (/proc/<pid>/fd/) .¶

Twɛn: so ade a wɔde gu mu no da so ara reyɛ adwuma mprempren? Mpɛn pii no, engineer bi yɛ application migration script, cleanup cron adwuma, anaa manual rm -rf /var/lib/postgresql/data/* anaasɛ rm /app/data/production.sqlite3 wɔ container a ɛretu mmirika mu, anaasɛ ɛpopa host bind-mount folda a ɛwɔ aseɛ no /opt/app/data bere a ade a wɔde gu mu no ankasa (postgres, . mysqld, . mariadbd, . redis-server, . mongod, . gunicorn, . node) yɛ da so ara sɔre na ɛretu mmirika.

Wɔ Linux mu no, wɔmfa fael bi mfi disk so bere a rm anaasɛ unlink() no na wɔfrɛ no. Ne disk blocks ne inode kɔ so yɛ wɔakyekyɛ ɔha biara mu nkyem 100 na wotumi kenkan kosi sɛ adeyɛ a etwa to a ɛkura fael nkyerɛkyerɛmu a wɔabue (FD) to saa inode no to mu anaasɛ efi mu.

Anamɔn 1: Hwehwɛ host Process ID (PID) a ɛfa ade a wɔde gu mu no ho¶

Wɔ Linux host node no so (ɛnyɛ container no mu), hwehwɛ host no PID wo database anaa application nhyehyɛe no ho:

sudo lsof +L1 | grep -E "postgres|mysqld|mariadbd|sqlite|redis|mongo|node|python|java"

COMMAND     PID   USER   FD   TYPE DEVICE  SIZE/OFF NLINK    NODE NAME
mysqld    18492    999   14u   REG  259,1 482344960     0 1849201 /var/lib/mysql/app_prod/users.ibd (deleted)
mysqld    18492    999   15u   REG  259,1 125829120     0 1849202 /var/lib/mysql/app_prod/orders.ibd (deleted)
mysqld    18492    999   16u   REG  259,1  79691776     0 1849190 /var/lib/mysql/ibdata1 (deleted)
python3   22104   1000    4u   REG  259,1  89128960     0 3102944 /data/production.sqlite3 (deleted)

Anamɔn 2: Kɔpi database fael ahorow a enni link no tẽẽ fi /proc/<PID>/fd/¶

Mma nnya anaa san hyɛ ade no ase bio. Bere a adeyɛ no da so ara rekɔ so no, kɔpi raw fael nkyerɛkyerɛmu ahorow no tẽẽ kɔ folda a ahobammɔ wom wɔ wo host no so:

sudo mkdir -p /root/proc_emergency_rescue

# Example 1: Rescuing an unlinked SQLite3 database (PID 22104, FD 4)
sudo cp /proc/22104/fd/4 /root/proc_emergency_rescue/production.sqlite3
# Check if PID 22104 also holds the -wal file open (e.g. FD 5) and copy it too:
sudo cp /proc/22104/fd/5 /root/proc_emergency_rescue/production.sqlite3-wal

# Example 2: Automatically rescuing ALL open (deleted) MySQL/Postgres files for PID 18492
for fd in /proc/18492/fd/*; do
    target=$(sudo readlink "$fd" 2>/dev/null)
    if echo "$target" | grep -q "(deleted)"; then
        fd_num=$(basename "$fd")
        clean_name=$(echo "$target" | sed 's/ (deleted)//; s/\//_/g')
        echo "Rescuing FD $fd_num -> $clean_name"
        sudo cp "$fd" "/root/proc_emergency_rescue/${fd_num}${clean_name}"
    fi
done

Anamɔn 3: Nea eye sen saa mpo: Tu mmirika kɔ live logical dump (pg_dump / . mysqldump / . .backup) ansa na ade a wɔde gu mu no ato FD ahorow no mu¶

Sɛ postgres anaasɛ mysqld da so ara reyɛ adwuma a ne data fael ahorow no abue wɔ memory mu na /proc/<PID>/fd/, bɔ mmɔden sɛ wobɛdi kan ayɛ live logical dump wɔ unix socket anaa TCP nkitahodi a ɛwɔ hɔ dedaw no so:

# If the database engine can still read its open file descriptors:
docker exec <container_name> pg_dumpall -U postgres > /root/proc_emergency_rescue/live_dump.sql
# Or for MySQL:
docker exec <container_name> mysqldump -u root -p --all-databases --single-transaction > /root/proc_emergency_rescue/live_mysql.sql

--- .

Phase 3: Docker volume a wɔapopa no a wobɛsan anya ext4 anaasɛ XFS bere a ade a wɔde gu mu no agyae akyi¶

Na sɛ wutuu mmirika nso ɛ docker compose down -v anaasɛ docker system prune --volumes, enti ade a wɔde gu mu no wɔagyae dedaw ne ne nyinaa /proc/<pid>/fd/ wɔato nkyerɛkyerɛmu ahorow mu? Seesei wowɔ mmirikatu mu atia background disk writes wɔ Linux host no so.

Anamɔn 1: Freeze Linux host faelsystem no wɔ sikani 10 mu¶

1. Gyae Docker, containerd, kubelet, ne cron ntɛm ara enti container, healthcheck, anaa log rotator biara nkyerɛw byte foforo nkɔ /var/lib/docker: 1 .

   sudo systemctl stop docker.socket docker.service containerd.service cron.service
   

2. Hwɛ block device ne filesystem a ɛkura /var/lib/docker: 1. .

   df -Th /var/lib/docker
   

   Filesystem     Type  Size  Used Avail Use% Mounted on
   /dev/nvme0n1p1 ext4  100G   42G   58G  42% /
   

3. San fa mpaepaemu Read-Only (anaasɛ fa mpaepaemu mfonini a ɛnyɛ den fa SSH so kɔ afiri foforo so):
4. Sɛ /var/lib/docker wɔ n'ankasa dedicated partition anaa LVM volume (sɛ nhwɛso no /dev/nvme0n1p2 anaasɛ /dev/mapper/vg0-docker), remount no akenkan nkutoo ntɛm ara:

   sudo mount -o remount,ro /var/lib/docker
   

5. Sɛ /var/lib/docker no wɔ ntini no so / mpaapaemu ( ./), bɔ mmɔden sɛ wobɛhyɛ ntini mu mpaapaemu no akenkan nkutoo denam kernel Magic SysRq trigger (echo u | sudo tee /proc/sysrq-trigger) anaasɛ sudo mount -o remount,ro /.
6. Sɛ wowɔ cloud VPS (AWS EC2, GCP, DigitalOcean, Hetzner, Linode): Ntɛm ara kɔ wo cloud web console na klik Create Snapshot wɔ server no block volume so (anaasɛ fa raw block device no fa SSH so kɔ server a ɛtɔ so mmienu so: sudo dd if=/dev/nvme0n1p1 bs=4M status=progress | gzip -1 | ssh user@backup-server "cat > /mnt/storage/docker_host_raw.img.gz").

Anamɔn 2: Hwɛ sɛ wo data da so ara te volume FOFORO a wɔantwitwa mu anaasɛ overlay2 mmeamu¶

Ansa na wobɛdi disk blocks a wɔankyekyɛ no, hwɛ sɛ wɔankora wo database no so ankasa wɔ bind mount anaa volume a wɔato din soronko a ɛkaa hɔ:

# Search the entire /var/lib/docker tree read-only for any surviving database files:
sudo find /var/lib/docker -name "PG_VERSION" -o -name "ibdata1" -o -name "*.sqlite*" -o -name "*.db" -o -name "dump.rdb" -o -name "WiredTiger" 2>/dev/null

Anamɔn 3: Recovering unlinked Docker volume directories wɔ so ext4 a wɔde di dwuma no ext4 Nsɛmma Nhoma (ext4magic ne debugfs) .¶

Berɛ bɛn docker volume rm popa no /var/lib/docker/volumes/<vol_name>/_data wɔ an ext4 fael nhyehyɛe, . ext4 zeros fi block pointers tẽẽ no mu ( .i_block / extent dua ntini) wɔ inode pon a ɛte ase no mu. Nanso, ** .ext4 nsɛmma nhoma (inode <8>, mpɛn pii no 128 MB to 1024 MB circular ring buffer)** da so ara kura abakɔsɛm mu mfonini ahorow a ɛfa inode tables ne directory blocks pɔtee a efi kan pɛɛ no ​​ho docker compose down -v wokum no.

+-------------------------------------------------------------------------+
| HOW ext4magic RECOVERS A DELETED DOCKER VOLUME FROM THE EXT4 JOURNAL    |
|                                                                         |
| Live Inode Table on Disk (After docker volume rm):                      |
|   Inode #918402 (users.ibd): [Size = 0, Extent Pointers = ZEROED]       |
|                                                                         |
| Ext4 Journal Ring Buffer (Inode <8> on Disk):                           |
|   Transaction #41902 (10 mins ago):                                     |
|   Inode #918402 (users.ibd): [Size = 482MB, Extents = Blocks 9000..9999]|
|                                                                         |
| -> ext4magic reads the old Inode Copy from Journal Transaction #41902   |
|    and extracts Blocks 9000..9999 directly to your rescue drive.        |
+-------------------------------------------------------------------------+

Dwane ext4magic tia wo akenkan nkutoo mpaapaemu (anaasɛ .img snapshot copy wɔ server a ɛto so abien so), a ɛkyerɛ bere nsɔano (-a) fi simma 5 ansa na woretu mmirika docker compose down -v anaasɛ docker system prune --volumes: 1 .

# Install ext4magic on your rescue machine (never on the affected partition)
sudo apt-get install -y ext4magic e2fsprogs

# Dump the ext4 journal from the frozen partition / image to a file in RAM (/dev/shm)
sudo debugfs -R "dump <8> /dev/shm/ext4_journal.copy" /dev/nvme0n1p1

# Recover all files deleted under var/lib/docker/volumes after (Current Time - 15 minutes)
sudo mkdir -p /mnt/external_rescue/docker_recovered
sudo ext4magic /dev/nvme0n1p1 \
  -j /dev/shm/ext4_journal.copy \
  -f var/lib/docker/volumes \
  -a $(date -d "-15 minutes" +%s) \
  -r -d /mnt/external_rescue/docker_recovered

Anamɔn 4: Carving raw SQLite, PostgreSQL, anaa MySQL InnoDB fael bere a nsɛmma nhoma no abɔ (XFS anaasɛ ext4) .¶

Sɛ host no de ** di dwuma a.XFS** anaasɛ sɛ ext4 journal a wɔabɔ no dedaw wɔ inode pointers no so, fa signature ne structure carving di dwuma tia raw disk mfonini no (docker_host_raw.img): .

1. Ma SQLite3 Database ahorow (.sqlite, . .db - s.e., Vaultwarden, Nextcloud, Ofie Boafo, Gitea, Grafana): Ɔde ne nsa kyerɛɛ ne so. SQLite3 database biara fi ase de 16-byte ASCII header a mfomso biara nni ho wɔ byte 0 wɔ ne 4 kilobyte kratafa a edi kan no so: 53 51 4c 69 74 65 20 66 6f 72 6d 61 74 20 33 00 ("SQLite format 3\0"), na ɛkora database kratafa no kɛse pɔtee (Bytes 16..17, mpɛn pii no 4096) ne database no kɛse nyinaa wɔ nkratafa mu ( .Bytes 28..31, big-endian 32-bit integer) wɔ saa ti koro no ara mu pɛɛ.
2. Dwane PhotoRec ne [File Opt] -> [x] sqlite enabled, anaasɛ scan mpaepaemu no tẽẽ ne grep -ab "SQLite format 3" sɛ wobɛhwehwɛ byte offset pɔtee a ɛwɔ wo SQLite fael a woapopa no mu na wode ayi afi mu dd: 1 .

   # Find the exact byte offsets of every "SQLite format 3" header on the partition:
   sudo grep -ab "SQLite format 3" /dev/nvme0n1p1
   

   Sɛ woyi fi mu wie a, tu mmirika sqlite3 carved.db ".recover" | sqlite3 repaired.db sɛ wɔbɛsan ayɛ B-dua ahaban kratafa biara a enni dɛm.

1. Ma MySQL / MariaDB InnoDB (.ibd pon so fael ahorow): InnoDB biara .ibd table no yɛ nea ɛtoatoa so 16 KB (16,384 byte) nkratafa a ɛwɔ FIL_PAGE_INDEX (0x45BF) ti ne table no soronko space_id. Fa Percona / TwinDB no di dwuma ** .undrop-for-innodb (stream_parser + c_parser)** adwinnade a wɔde scan /dev/nvme0n1p1 ma obiara 16 KB InnoDB index nkratafa na tow row biara gu TSV/SQL fael mu tẽẽ ma wɔsan de ba.
2. Ma PostgreSQL Heap Fael ahorow ($PGDATA/base/<db_oid>/<relfilenode>): PostgreSQL sie pon biara sɛ nea ɛtoatoa so 8 KB (8,192 byte) heap nkratafa a ɛwɔ mu PageHeaderData (pd_lower, . pd_upper, . pd_special). Fa di dwuma pg_filedump anaa ** .pglazarus / . pdu** sɛ wobɛ scan disk mpɔtam a wɔankyekyɛ ama 8 KB nkratafa a ɛne wo pon no kɔla tuple nhyehyɛe hyia (HeapTupleHeaderData) na wɔasan ayɛ ntoatoaso no.

--- .

Ɔfa 4: Kubernetes PVC, PV, ne etcd Snapshot Ntɛmntɛm a Wɔsan Yɛ¶

Wɔ Kubernetes (EKS, . GKE, . AKS, . k3s, . RKE2, . kubeadm, . OpenShift), wɔ akwanhyia mu a wɔpopa PersistentVolumeClaim (kubectl delete pvc), StatefulSet, anaa Namespace nyinaa (kubectl delete ns prod) di architecture soronko koraa akyi sen standalone Docker.

Nsɛm a ɛfa ho A: PVC no kyerɛ Terminating (Fializer Nkwagyefo) .¶

Dwane:

kubectl get pvc -n <namespace>

# Stream a compressed tar backup or pg_dump out of the still-running pod immediately:
kubectl exec -n <namespace> <pod-name> -- pg_dumpall -U postgres > ./rescued_k8s_db.sql
# Or tar an entire mounted volume directory out over stdout:
kubectl exec -n <namespace> <pod-name> -- tar czf - /var/lib/postgresql/data > ./rescued_pvc_files.tar.gz

Nsɛm a ɛfa ho B: Wɔpopaa PVC no, na kubectl get pv kyerɛ PersistentVolume no wɔ Released ɔman (ReclaimPolicy: Retain) .¶

Dwane:

kubectl get pv

Sɛnea wɔyɛ a Released PV Available bio wɔ sikani 10 mu a data a ɛyera biara nni hɔ: 1. Sesa PersistentVolume ade no na popa no spec.claimRef si kwan:

kubectl patch pv <your-pv-name> --type json -p '[{"op": "remove", "path": "/spec/claimRef"}]'

Scenario C: Wɔresan de Kubernetes cluster tebea a wɔapopa afi an etcd mfonini a wɔde twaa mfonini (etcdctl snapshot restore) .¶

Na sɛ obi tuu mmirika nso ɛ kubectl delete namespace prod anaasɛ helm uninstall na popaa Kubernetes manifests 200, Secrets, ConfigMaps, ne PV nkyerɛase ahorow wɔ obi ankasa a ɔhwɛ so kubeadm, . k3s, anaa RKE2 wimhyɛn a wɔde di dwuma?

1. Hwehwɛ wo nea aba foforo etcd snapshot (sɛnea ɛte biara wɔ /var/lib/rancher/k3s/server/db/snapshots/, . /var/lib/rancher/rke2/server/db/snapshots/, anaa wo nhyehyɛe a woayɛ /var/backups/etcd/snapshot.db).
2. Hwɛ sɛ snapshot no mudi mu kura ansa na woaka live cluster no:

   ETCDCTL_API=3 etcdctl snapshot status /path/to/snapshot.db --write-out=table
   

3. Gyae kube-apiserver ne etcd wɔ control plane node(s) no so:
4. So kubeadm (static pods): Tu kɔ baabi foforo /etc/kubernetes/manifests/kube-apiserver.yaml ne /etc/kubernetes/manifests/etcd.yaml bere tiaa bi mu sɛ /root/manifests_hold/ nti kubelet gyina nkuku abien no nyinaa mu yiye, afei ɛsan kɔ soro /var/lib/etcd to /var/lib/etcd.bak.
5. So k3s: Dwane sudo systemctl stop k3s ne sudo k3s server --cluster-reset --cluster-reset-restore-path=/path/to/snapshot.
6. San de no ba etcd data kyerɛwtohɔ no so kubeadm: 1. .

   sudo ETCDCTL_API=3 etcdctl snapshot restore /path/to/snapshot.db \
     --data-dir=/var/lib/etcd \
     --name=<control-plane-node-name> \
     --initial-cluster=<node-name>=https://<node-ip>:2380 \
     --initial-advertise-peer-urls=https://<node-ip>:2380
   

7. Kɔ fa etcd.yaml ne kube-apiserver.yaml san kɔ mu /etc/kubernetes/manifests/. Wɔ sikɔne 30 mu no, . kube-apiserver san bata nea wɔasan de aba no ho etcd store na Namespace, Secret, ConfigMap, ne PersistentVolume adeɛ biara a wɔapopa no san pue.

--- .

Ɔfa 5: Docker ne Kubernetes volumes a wɔbɛma ayɛ den atia sɛ wɔbɛpopa wɔ akwanhyia mu¶

Sɛ wɔsan de wo production database no ba wie a, fa saa architectural guardrails anan yi di dwuma sɛnea ɛbɛyɛ a CLI typo biako rentumi mpopa live volume bio da:

• Sesa biribiara a wɔyɛ Kubernetes StorageClass efi reclaimPolicy: Delete to reclaimPolicy: Retain: Sɛnea wɔahyɛ no, wɔasisi mununkum StorageClasses (gp2, . gp3, . standard-rwo) default to reclaimPolicy: Delete, a ɛkyerɛ sɛ sɛ wopopa PVC a, ɛsɛe cloud disk no. Bɔ a StorageClass ne reclaimPolicy: Retain (anaasɛ fa patch PV ahorow a ɛwɔ hɔ dedaw no kubectl patch pv <pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}') enti sɛ wopopa PVC anaa Namespace a, ɛnpopa block volume a ɛwɔ aseɛ no da.
• Enable AWS EBS Recycle Bin / GCP Snapshot Schedules / Velero with CSI Snapshots: Sɛ wo ma AWS EBS Recycle Bin yɛ adwuma a, ɛkora EBS volume biara a wɔapopa so nnafua 7 kɔsi 30 mpo sɛ CSI draiver no frɛ a DeleteVolume.
• Mfa nni dwuma da docker system prune --volumes wɔ automated cleanup scripts anaa CI/CD runners mu: Sɛ production database container bi bɔ bere tiaa bi a (Exited (1)) wɔ bere pɔtee a cron adwuma bi tu mmirika no docker system prune -a --volumes -f, Docker bu ne volumes a wɔmmɔ din no sɛ "wɔmfa nni dwuma" na ɛpopa daa. Strip no ho asɛm --volumes efi cleanup scripts nyinaa mu.
• Fa host bind mounts a ɛda adi pefee di dwuma (/srv/prod_data/postgres:/var/lib/postgresql/data) anaa abɔnten so databases a wɔhwɛ so ma stateful production workloads: Ɛnte sɛ /var/lib/docker/volumes/, host bind-mount kyerɛwtohɔ ahorow a ɛwɔ abɔnten /var/lib/docker/ yɛ wɔnka da docker compose down -v, . docker volume rm, anaa docker system prune --volumes.

--- .

Mpɔtam hɔ ntɛmpɛ nɔma ne adwumakuw no nhyehyɛe ahorow¶

Sɛ ade a wɔde gu mu anaa Kubernetes akorae a wɔpopa no nya ayaresabea ayaresabea dwumadi ahorow, ɔmanfo ahobammɔ a wɔde mena, anaa ankorankoro ho nsɛm a wɔahyɛ ho mmara so nkɛntɛnso a, kɔ soro ntɛm ara:

• United States ne Canada: Dial ** .911** sɛ container cluster huammɔdi ma ayaresabea ayarefo hwɛ, EHR nhyehyɛe, anaa ntɛmpɛ a wɔde kɔ no nyɛ adwuma a. Bɔ container infrastructure a wɔpopa anaa kɔmputa so nsɛm a ɛyɛ hu ho amanneɛ kyerɛ CISA (1-844-Say-CISA / . cisa.gov/report) anaa Canada Asoɛe a Ɛhwɛ Intanɛt So Ahobammɔ So (1-833-CYBER-88) na ɛyɛ.
• Ahemman: Dial ** .999** ma nkwa ho asiane anaa NHS nhyehyɛe a ɛho hia. Wɔ UK GDPR ase no, ankorankoro data a wɔsɛe no daa anaasɛ ɛyera a enni backup a wobetumi asan anya no hwehwɛ sɛ wɔhwehwɛ nnɔnhwerew 72 bɔ amanneɛ sɛ wɔabu mmara so a wɔde kɔma ICO (0303 123 1113) na ɛyɛ.
• Europa Aman Nkabom Kuw: Dial 112 wɔ aman a wɔka EU ho nyinaa mu. Ɛsɛ sɛ nnwumakuw a ɛwɔ NIS2 Akwankyerɛ ase no de kɔkɔbɔ ntɛm kɔma wɔn man CSIRT wɔ nnɔnhwerew 24 akyi wɔ ɔsom adwuma no a wɔatwa kɛse akyi, na ankorankoro data a wɔhwere daa no kanyan GDPR Ahyɛde 33 amanneɛbɔ wɔ nnɔnhwerew 72 mu.
• Australia ne New Zealand: Dial ** .000** wɔ Australia anaa ** .111** wɔ New Zealand; bɔ nsɛm a ɛho hia a ɛfa nnwuma ho amanneɛ kyerɛ ACSC (1300 CYBER1) na ɛyɛ.
• India ne Asia-Pacific: Dial ** .112** wɔ India sɛ wɔbɛyɛ biribi a egye ntɛmpɛ; bɔ mununkum anaa container ahobanbɔ ho amanneɛ kyerɛ CERT-In (incident@cert-in.org.in) wɔ nnɔnhwerew 6 mfɛnsere a ɛyɛ ahyɛde no mu.

--- .

Nsɛm a wɔtaa bisa¶

Ɛyɛ saa docker compose down -v popa host bind mounts (./data:/var/lib/mysql) anaasɛ Named ne Anonymous Docker volume ahorow nkutoo?¶

`docker compose down -v` ONLY popa Named volumes a wɔabɔ ho dawuru wɔ `volumes:` ɔfã a ɛfa `docker-compose.yml` ne Anonymous volumes a wɔde abata nsukorade ahorow no ho; ɛno da biara da popa fael ahorow a ɛwɔ host bind mount mu (te sɛ `- /opt/mysql_data:/var/lib/mysql` anaasɛ `- ./postgres_data:/var/lib/postgresql/data`). Sɛ wo compose fael no de host kwan a efi ase `/` anaasɛ `./`, hwɛ saa folda no wɔ host no so mprempren: wo fael ahorow no da so ara yɛ ɔha biara mu nkyem 100.

Dɛn nti na wɔyɛɛ saa docker system prune --volumes popa me database volume bere a migyaee container no simma 2 pɛ de update environment variable bi?¶

Twɛn yiye ma nsonsonoe a ɛda ntam `docker stop` ne `docker rm`: sɛ ade a wɔde gu mu yɛ kɛkɛ a `Stopped / Exited` (wotumi hu wɔ `docker ps -a`), . `docker system prune --volumes` yɛ ɛnyɛ saa popa ne volumes efisɛ ade a wɔagyae no da so ara kyerɛ wɔn. Nanso, sɛ wutuu mmirika a `docker rm ` (sɛ wobɛsan ayɛ ade a wɔde frankaa foforo ayɛ) anaasɛ sɛ woatwam a `docker system prune -a --volumes` (a edi kan popa nsukorade a wɔagyae nyinaa na afei popa volumes nyinaa a ɛnbata ade biara ho bio), volume no bɛyɛɛ "dangling" milisecond a woyii ade a wɔagyae no na wɔpopaa no ntɛm ara wɔ ɛno akyi.

So metumi asan anya Docker volume a wɔapopa wɔ Docker Desktop mu wɔ Mac so (Apple Silicon / . Intel) anaa Windows (WSL2)?¶

Yiw, sɛ wo gyae Docker Desktop ntɛm ara ma enti virtual afiri no gyae adwumayɛ `fstrim` atia virtual disk no. Wɔ macOS so no, Docker sie Linux volumes nyinaa wɔ mu `-/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw`; wɔ Windows WSL2 so, wɔ mu `%LocalAppData%\Docker\wsl\disk\docker_data.vhdx`. Kɔpi saa asɛm no `.raw` anaasɛ `.vhdx` fael no kɔ abɔnten SSD so ntɛm ara, fa bata akenkan nkutoo ho wɔ Linux mu (de di dwuma `losetup -Pr` efisɛ `.raw` anaasɛ `qemu-nbd -r` efisɛ `.vhdx`), na wotu mmirika `ext4magic` anaasɛ `PhotoRec` atia virtual no `ext4` mpaapaemu a ɛwɔ mu.

Na sɛ mepopa Kubernetes StatefulSet bi nso ɛ kubectl delete statefulset postgres? So ɛno nso popaa me PVC ahorow no?¶

Dabi. Sɛnea wɔahyɛ no wɔ Kubernetes mu (gye sɛ `persistentVolumeClaimRetentionPolicy.whenDeleted: Delete` na wɔahyehyɛ no pefee wɔ Kubernetes 1.27+), a ɛretu mmirika `kubectl delete statefulset` popa StatefulSet controller ne ne Pods no, . gyaw ne nyinaa `PersistentVolumeClaims` (`data-postgres-0`, . `data-postgres-1`) ne wɔn nsɛm a ɛhyɛ ase no ɔha biara mu nkyem 100 a enni dɛm. Kɛkɛ a san fa wo `StatefulSet` YAML da adi wɔ din koro no ara mu na `volumeClaimTemplates`, na Pods foforo no bɛsan abata nea ɛwɔ hɔ dedaw no ho ankasa `data-postgres-0` PVC ahorow a wɔde yɛ nneɛma.

Ɔkwan bɛn so na metumi anya fael ahorow a ɛwɔ Docker mu overlay2 directory bere a adekorade no pow sɛ ebefi ase (CrashLoopBackOff / . Exited (1))?¶

Mpopa ade a asɛe no. Bere tenten a ade a wɔde asie no da so ara wɔ mu no `docker ps -a`, ne writable layer ne volume mounts no yɛ nea enni dɛm koraa wɔ disk so. Wubetumi ayɛ fael ahorow no ho kɔpi tẽẽ afi ade a wɔagyae no mu denam `docker cp :/var/lib/postgresql/data /root/rescued_data`, anaa hwehwɛ nea ɔte no pɛpɛɛpɛ `overlay2` kwan denam mmirikatu so `docker inspect --format '{{.GraphDriver.Data.UpperDir}}'` na wokenkan saa directory no tẽẽ sɛ `root`.

Na sɛ obi popa Kubernetes bi nso ɛ Namespace (kubectl delete ns prod) na na PVC ahorow nyinaa wɔ reclaimPolicy: Delete wɔ AWS EBS anaa GCP so?¶

Sɛ mununkum CSI draiver (`ebs.csi.aws.com` anaasɛ `pd.csi.storage.gke.io`) a wɔakum no dedaw `DeleteVolume` efisɛ `reclaimPolicy` na ɛyɛ `Delete`, hwɛ mununkum-gyinabea ahobammɔ afiri abiɛsa ntɛm ara: (1) hwɛ **AWS Recycle Bin for EBS Volumes** wɔ AWS EC2 console no mu (sɛ na mmara bi a wɔde sie no yɛ adwuma wɔ akontaabu/tags no so a, EBS volume a wɔapopa no te Recycle Bin no mu na wobetumi asan de aba wɔ 1 click mu); (2) hwɛ **AWS Backup / DLM / GCP Snapshot Schedules** ma nnansa yi ara automated volume snapshot; na (3) hwɛ sɛ na wo database no rebɔ WAL/binlogs (`WAL-G`, . `pgBackRest`, . `Litestream`) de kɔ S3/GCS bokiti bi mu.

--- .

Akwankyerɛ ahorow a ɛfa ho¶

• Server disk 100% ayɛ ma (0 bytes free): database WAL log bloat, B-dua a asɛe, ne zero-space recovery
• Akwanhyia rm -rf anaa Shift+Delete wɔ Linux, macOS, anaa Windows: ntɛm ara akenkan nkutoo remount ne inode carving
• Virtual mfiri VMDK, VHDX, ne qcow2 snapshot nkɔnsɔnkɔnsɔn asɛe ne hypervisor nkwagye
• Akwanhyia Git commit a AWS safoa, .env fael, anaa kokoam SSH safoa kɔ ɔmanfo GitHub repo
• Web hosting ahokyere, atutra, ne backup sanba
• Kɔmputa so backups, 3-2-1 nhyehyɛe, ne atoyerɛnkyɛm bere mu ahotɔ
• Nnɔmba a egye ntɛmpɛ sɛnea ɔman biara te

--- .

Fibea ne aban akwankyerɛ¶

• Kubernetes Aban Nwoma (kubernetes.io / Cloud Native Computing Foundation): Akorae Adeɛ a Wɔde Di Dwuma Ahobanbɔ (kubernetes.io/pvc-protection Finalizers), Nnyigyei a Ɛkɔ So Daa a Wɔsan Nya (Retain vs. Delete), ne Adwumayɛ etcd Akuw a wɔyɛ ma Kubernetes (etcdctl snapshot restore).
• Docker Inc. (docs.docker.com): Hwɛ Data so wɔ Docker mu: Volumes, Bind Mounts, . overlay2 Storage Driver Architecture, ne Prune Nkyerɛkyerɛmu.
• Linux Kernel Ahyehyɛde no (kernel.org): Nea /proc Fael nhyehyɛe (procfs Bue Fael Nkyerɛkyerɛmu Recovery via /proc/<pid>/fd/) ne Ext4 Journal Nkyerɛkyerɛmu.
• ext4magic Dwumadie no ho Nwoma (ext4magic.sourceforge.net): Forensic Inode ne Journal Nkitahodi a Wɔsan Yɛ wɔ Ext3/Ext4 Faelsystems so.
• Ɔman Asoɛe a Ɛhwɛ Gyinapɛn ne Mfiridwuma So (NIST) (United States): Nwoma Titiriw 800-190: Application Container Ahobammɔ Akwankyerɛ & Asɛm a Ɛsisii a Wɔsan Yɛ.