Mpịakọta Docker ma ọ bụ Kubernetes ehichapụrụ: na-azọpụta docker system prune --volumes, overlay2 Inodes, na etcd Ihe onyonyo¶

Na-agba ọsọ docker compose down -v, docker system prune -a --volumes, docker volume rm, ma ọ bụ kubectl delete pvc na taabụ ọnụ na-ezighi ezi na mmepụta bụ otu n'ime ụzọ kachasị ọsọ iji hichapụ ndụ PostgreSQL, MySQL, MariaDB, Redis, MongoDB, ma ọ bụ SQLite nchekwa data. N'adịghị ka igwe onyonyo igwe ojii nke na-akwaga na igwe eji emegharị ihe ụbọchị 7, onye ọkwọ ụgbọ ala mpaghara Docker (/var/lib/docker/volumes/<name>/_data) na Kubernetes local-path / hostPath Ndị na-enye onyinye na-ewepụta POSIX ọkọlọtọ ozugbo unlink() na rmdir() na-akpọ ozugbo megide sistemụ faịlụ ndị ọbịa Linux dị n'okpuru (ext4, XFS, ma ọ bụ Btrfs). Edemede ọ bụla na-eme na onye ọbịa Linux ahụ ka emechara iwu ahụ nwere ike idegharị ihe nchekwa nchekwa data ewepụtara na diski. Ọ bụrụ na diski ihe nkesa gị jupụtara na pasent 100 (0 bytes free) tupu mmadụ agbaa ọsọ docker system prune --volumes n'oké ụjọ, gbaa izu diski ihe nkesa 100% zuru, nchekwa data WAL log bloat, yana mgbake oghere efu; maka nhichapụ faịlụ Linux bare-metal na mpụga arịa, lee ihe mberede rm -rf ma ọ bụ Shift-Hichapụ na ext4, XFS, APFS, na NTFS: na-agụ naanị na-agụgharị na ihe osise inode na igwe mebere VMDK, VHDX, na qcow2 nrụrụ aka yinye nrụrụ aka na nnapụta hypervisor. Ọ bụrụ na ọpụpụ mmepụta a na-emetụta nlekọta onye ọrịa ụlọ ọgwụ, izipu ihe mberede, ma ọ bụ sistemu nchekwa nchekwa ọha, kpọọ nọmba mberede mpaghara gị (911 na United States na Canada, 112 gburugburu European Union, 999 na United Kingdom) ma ọ bụ lelee akwụkwọ ndekọ aha anyị nọmba mberede.

Mee nke a ugbu a¶

1. ** agbala ọsọ docker compose up -d, systemctl restart docker, ma ọ bụ megharịa Kubernetes pod ***: Megharịa akpa ahụ ma ọ bụ malitegharịa Docker daemon na-ede megabytes nke ọhụrụ. overlay2 metadata, igbe JSON akpa, yana faịlụ nchekwa data efu emepụtara ọhụrụ (initdb) kpọmkwem n'elu kpọmkwem ext4 ma ọ bụ XFS mgbochi ebe olu gị ehichapụrụ nọ ọdụ sekọnd ole na ole gara aga.
2. ** Lelee na 5 sekọnd ma ọ bụrụ na akpa usoro (postgres, mysqld, redis-server, node, python) ka na-agba ọsọ na ebe nchekwa (ps aux na sudo lsof +L1***: Ọ bụrụ na mmadụ ehichapụ faịlụ n'ime olu agbakwunyere ma ọ bụ kechie olu na-amaghị aha mgbe akpa ahụ ka na-arụ ọrụ, kernel Linux na-edobe ** 100% nke faịlụ nchekwa data enweghị njikọ dị ndụ na diski *** ọ bụrụhaala na NJ usoro nnabata nke akpa ahụ.PID) na-ejide ihe nkọwa faịlụ mepere emepe (FD). Ị nwere ike iṅomi faịlụ ndị ahụ ozugbo na /proc/<PID>/fd/ ma ọ bụ /proc/<PID>/root/ na efu data ọnwụ tupu akpa akwụsị.
3. ** Ọ bụrụ na akwụsịlarị akpa ahụ wee ehichapụ olu ya (docker compose down -v ma ọ bụ docker system prune --volumes), freeze host disk dere ozugbo: Kwụsị Docker daemon (sudo systemctl stop docker.socket docker.service containerd) ya mere igbe ndabere na nlele ahụike kwụsị ide ndekọ n'ime /var/lib/docker, mgbe ahụ ma ọ bụ remount /var/lib/docker (ma ọ bụ mgbọrọgwụ / nkebi) naanị ọgụgụ (sudo mount -o remount,ro /) ma ọ bụ were foto olu igwe ojii ozugbo (AWS EBS Snapshot, GCP Persistent Disk Snapshot, DigitalOcean Volume Snapshot) tupu emetụ ihe ọ bụla aka.
4. ** Ọ bụrụ na ịnọ na Kubernetes (kubectl delete pvc ma ọ bụ kubectl delete namespace), lelee Volume Persistent (PV) ReclaimPolicy (kubectl get pv) ozugbo ***:
5. Ọ bụrụ na PV nwere persistentVolumeReclaimPolicy: Retain, A naghị ehichapụ data gị **: nke PV naanị kwagara Released ọnọdụ na diski igwe ojii gị (EBS, Azure Disk, Ceph RBD, NFS) bụ 100 percent emebibeghị.
6. Ọbụlagodi kubectl delete pvc a rapara na Terminating, Kubernetes ** Ndị ngwụcha (kubernetes.io/pvc-protection*** gbochie onye ọkwọ ụgbọ ala CSI ka ọ ghara ihichapụ nchekwa dị n'okpuru ma ọ bụrụhaala na Pod ọ bụla ka na-ezo aka na PVC. Ehichapụla Pod ahụ ruo mgbe i detuo data na Pọd na-agba ọsọ.
7. Foto nkebi ngọngọ dị n'okpuru (dd / ddrescue n'elu SSH) gaa na diski mpụga ma ọ bụ olu igwe ojii dị iche tupu ịgba ọsọ ext4magic, extundelete, ma ọ bụ PhotoRec Ọ dịghị mgbe ị na-agba ndị na-ese faịlụ na-edepụta faịlụ mmepụta n'otu akụkụ diski nke ị na-agbake /var/lib/docker/volumes.
8. Maka steeti ụyọkọ Kubernetes mebiri emebi ma ọ bụ kpochapụrụ (na mberede)kubectl delete crd / namespace), kwụsị kube-apiserver ma weghachi ihe kacha ọhụrụ gị etcd foto (etcdctl snapshot restore*: Ọ bụrụ na ụyọkọ steeti na etcd ehichapụrụ mgbe oke nchekwa igwe ojii ka dị, na-eweghachi etcd na-agbake niile PV, PVC, Secret, na ConfigMap ihe n'out oge.

Osisi mkpebi: olu Docker ehichapụrụ, overlay2 oyi akwa, ma ọ bụ Kubernetes PVC triage¶

Jiri eserese ngosi a họrọ ụzọ mgbake efu kacha ọsọ ọsọ dabere na usoro akpa ahụ ka dị ndụ, ma diski mpaghara ma ọ bụ nchekwa CSI igwe ojii kwadoro olu ya, yana nke sistemụ faịlụ na-akwado. /var/lib/docker:

flowchart TD
    Start["Data Mmepụta Na-efu Mgbe Docker ma ọ bụ Kubernetes Hichapụ Iwu"] --> WhichEnv{"Kedu Platform & Iwu kpalitere mfu ahụ?"}

    WhichEnv -->|"Docker: Akpa ka na-agba ọsọ (faịlụ enweghị njikọ n'ime olu/akpa)"| ProcRescue["Napụta ndị na-akọwa faịlụ enweghị njikọ site na sudo lsof +L1 na /proc/PID/fd/N ma ọ bụ /proc/PID/root/"]

    WhichEnv -->|"Docker: Akwụsịla akpa & Mpịakọta ehichapụ (dee ala -v / prune --mpịakọta)"| StopDocker["Gbaa ọsọ ọsọ: sudo systemctl kwụsị docker.socket docker akpa"]

    WhichEnv -->|"Kubernetes: kubectl hichapụ pvc / pv / statefulset / namespace"| K8sCheck{"Pod ahụ ka na-agba ọsọ ma ọ bụ PVC ọ na-akwụsị?"}

    K8sCheck -->|"Ee: Pod ka na-agba ọsọ (Finalizer na-egbochi CSI Hichapụ)"| K8sExec["Ehichapụla Pod -> Gbaa kubectl exec / pg_dump / tar n'elu stdout gaa na diski mpaghara ozugbo."]
    K8sCheck -->|"Mba: Pod agaala -> Lelee kubectl nweta pv"| ReclaimCheck{"Kedu ihe bụ PV ReclaimPolicy?"}

    ReclaimCheck -->|"Njide (Ọnọdụ PV = Ewepụtara)"| ClearClaimRef["100% Data adịchaghị -> Dezie PV ka Wepu spec.claimRef.uid -> Tinyegharịa na PVC ọhụrụ"]
    ReclaimCheck -->|"Hichapụ (Cloud EBS / PD / Longhorn / Mpaghara-ụzọ)"| CloudOrLocal{"Olu CSI igwe ojii ma ọ bụ diski nnabata mpaghara?"}

    CloudOrLocal -->|"Igwe ojii CSI (AWS EBS / GCP / Azure / Ceph)"| CloudSnap["Lelee Cloud Console Recycle Bin / Akpaaka EBS Snapshots / Ceph RBD ahịhịa (rbd ahịhịa ls)"]
    CloudOrLocal -->|"Disk Node Mpaghara (ụzọ mpaghara / hostPath)"| StopDocker

    StopDocker --> RemountRO["Ọgụgụ-naanị Remount Host Partition (mount-o remount,ro) ma ọ bụ Stream Block Image n'elu SSH"]
    RemountRO --> HostFS{"Kedu ihe ndị nnabata faịlụ /var/lib/docker?"}
    HostFS -->|"ex4"| Ext4Rec["Jiri debugfs / ext4magic Journal Inode Transaction Rollback (-a timestamp)"]
    HostFS -->|"XFS/Btrfs"| CarveRec["Jiri Btrfs mgbọrọgwụ / xfs_irecover / mbinye aka omenala maka SQLite, Postgres Heap, ma ọ bụ InnoDB .ibd"]

Usoro 1: Otu Docker na Kubernetes si ehichapụ mpịakọta na diski¶

Tupu ịmee iwu mgbake ọ bụla, ghọta ebe data gị bi na Linux host yana ihe iwu ihichapụ metụrụ aka:

Nkeji 2: Napụta faịlụ nchekwa data mepere emepe n'ime akpa na-agba ọsọ (/proc/<pid>/fd/)¶

Chere: ** akpa ahụ ka na-agba ọsọ ugbu a?** Ugboro ugboro, onye injinia na-eme edemede mbugharị ngwa, ọrụ cron nchacha, ma ọ bụ akwụkwọ ntuziaka rm -rf /var/lib/postgresql/data/* ma ọ bụ rm /app/data/production.sqlite3 n'ime akpa na-agba ọsọ, ma ọ bụ hichapụ folda nnabata nnabata bind-mount /opt/app/data mgbe akpa ahụ n'onwe ya (postgres, mysqld, mariadbd, redis-server, mongod, gunicorn, node)** ka na-agba ọsọ**.

Na Linux, anaghị ehichapụ faịlụ na diski mgbe rm ma ọ bụ unlink() a na-akpọ. Ihe mgbochi diski ya na inode na-anọgide ** 100% ekenye ya ma nwee ike ịgụ ya ** ruo mgbe usoro ikpeazụ na-ejide nkọwa faịlụ mepere emepe (FD) na inode na-emechi ya ma ọ bụ pụọ.

Nzọụkwụ 1: Chọta onye ọbịa Process ID (PID) nke usoro akpa¶

Na ọnụ ụzọ Linux (ọ bụghị n'ime akpa), chọta onye ọbịa PID nke nchekwa data gị ma ọ bụ usoro ngwa:

sudo lsof +L1 | grep -E "postgres|mysqld|mariadbd|sqlite|redis|mongo|node|python|java"

COMMAND     PID   USER   FD   TYPE DEVICE  SIZE/OFF NLINK    NODE NAME
mysqld    18492    999   14u   REG  259,1 482344960     0 1849201 /var/lib/mysql/app_prod/users.ibd (deleted)
mysqld    18492    999   15u   REG  259,1 125829120     0 1849202 /var/lib/mysql/app_prod/orders.ibd (deleted)
mysqld    18492    999   16u   REG  259,1  79691776     0 1849190 /var/lib/mysql/ibdata1 (deleted)
python3   22104   1000    4u   REG  259,1  89128960     0 3102944 /data/production.sqlite3 (deleted)

Kwụpụ 2: Detuo faịlụ nchekwa data enweghị njikọ ozugbo /proc/<PID>/fd/¶

Akwụsịla ma ọ bụ malitegharịa akpa ahụ. Mgbe usoro a ka na-aga, detuo nkọwa nkọwa faịlụ ozugbo na nchekwa nchekwa na onye ọbịa gị:

sudo mkdir -p /root/proc_emergency_rescue

# Example 1: Rescuing an unlinked SQLite3 database (PID 22104, FD 4)
sudo cp /proc/22104/fd/4 /root/proc_emergency_rescue/production.sqlite3
# Check if PID 22104 also holds the -wal file open (e.g. FD 5) and copy it too:
sudo cp /proc/22104/fd/5 /root/proc_emergency_rescue/production.sqlite3-wal

# Example 2: Automatically rescuing ALL open (deleted) MySQL/Postgres files for PID 18492
for fd in /proc/18492/fd/*; do
    target=$(sudo readlink "$fd" 2>/dev/null)
    if echo "$target" | grep -q "(deleted)"; then
        fd_num=$(basename "$fd")
        clean_name=$(echo "$target" | sed 's/ (deleted)//; s/\//_/g')
        echo "Rescuing FD $fd_num -> $clean_name"
        sudo cp "$fd" "/root/proc_emergency_rescue/${fd_num}${clean_name}"
    fi
done

Kwụpụ 3: Ọbụna ka mma: Gbaa mkpofu ezi uche dị ndụ (pg_dump / mysqldump / .backup) tupu akpa ahụ emechi FD¶

Ọ bụrụ postgres ma ọ bụ mysqld ka na-agba ọsọ na faịlụ data ya na-emeghe na ebe nchekwa na /proc/<PID>/fd/, gbalịa na-ebu ụzọ mee mkpofu ezi uche dị ndụ n'elu oghere unix dị ugbu a ma ọ bụ njikọ TCP:

# If the database engine can still read its open file descriptors:
docker exec <container_name> pg_dumpall -U postgres > /root/proc_emergency_rescue/live_dump.sql
# Or for MySQL:
docker exec <container_name> mysqldump -u root -p --all-databases --single-transaction > /root/proc_emergency_rescue/live_mysql.sql

Usoro 3: Iweghachi olu Docker ehichapụrụ ext4 ma ọ bụ XFS mgbe akpa ahụ kwụsịrị¶

Gịnị ma ọ bụrụ na ị gbara ọsọ docker compose down -v ma ọ bụ docker system prune --volumes, ya mere akpa ahụ ** akwụsịlarị ** na ihe niile /proc/<pid>/fd/ emechiela nkọwa nkọwa? Ugbu a ị nọ n'ọsọ megide diski ndabere na-ede na Linux host.

Kwụpụ 1: Wepu sistemụ faịlụ Linux n'ime sekọnd 10¶

1. ** Kwụsị Docker, akpa, kubelet na cron ozugbo ** yabụ na ọ nweghị akpa, nyocha ahụike ma ọ bụ rotator log na-ede byte ọzọ /var/lib/docker:

   sudo systemctl stop docker.socket docker.service containerd.service cron.service
   

2. ** Lelee ngwaọrụ ngọngọ na sistemụ faịlụ jide /var/lib/docker**:

   df -Th /var/lib/docker
   

   Filesystem     Type  Size  Used Avail Use% Mounted on
   /dev/nvme0n1p1 ext4  100G   42G   58G  42% /
   

3. ** Tinyegharịa nkebi naanị Read-nanị (ma ọ bụ bunye onyonyo nkebi nke sitere na SSH gaa na igwe ọzọ) ***:
4. Ọ bụrụ /var/lib/docker dị na nkebi raara onwe ya nye ma ọ bụ olu LVM (dịka ọmụmaatụ /dev/nvme0n1p2 ma ọ bụ /dev/mapper/vg0-docker), bugharịa ya naanị ngwa ngwa:

   sudo mount -o remount,ro /var/lib/docker
   

5. Ọ bụrụ /var/lib/docker dị na mgbọrọgwụ / nkebi (/), nwaa ịmanye mgbọrọgwụ nkebi ka ọ gụọ naanị site na iji kernel Magic SysRq trigger (echo u | sudo tee /proc/sysrq-trigger) ma ọ bụ sudo mount -o remount,ro /.
6. ** Ọ bụrụ na ịnọ na VPS ígwé ojii (AWS EC2, GCP, DigitalOcean, Hetzner, Linode) ***: gaa na igwe njikwa igwe ojii gị ozugbo wee pịa ** Mepụta foto ** na olu ngọngọ nkesa (ma ọ bụ na-ebugharị ngwaọrụ mgbochi raw n'elu SSH gaa na nkesa nke abụọ: sudo dd if=/dev/nvme0n1p1 bs=4M status=progress | gzip -1 | ssh user@backup-server "cat > /mnt/storage/docker_host_raw.img.gz").

Kwụpụ 2: Lelee ma data gị ka nọ ọdụ na olu ọzọ akwachabeghị ma ọ bụ overlay2 oyi akwa¶

Tupu ịkpụ ngọngọ diski a na-ekenyeghị, chọpụta na echekwaghị nchekwa data gị n'ezie n'ugwu agbụ ma ọ bụ olu aha ya dị iche nke dịgidere:

# Search the entire /var/lib/docker tree read-only for any surviving database files:
sudo find /var/lib/docker -name "PG_VERSION" -o -name "ibdata1" -o -name "*.sqlite*" -o -name "*.db" -o -name "dump.rdb" -o -name "WiredTiger" 2>/dev/null

Kwụpụ 3: Na-eweghachi akwụkwọ ndekọ aha Docker enweghị njikọ ext4 iji ext4 Akwụkwọ akụkọ (ext4magic na debugfs)¶

Mgbe docker volume rm na-ehichapụ /var/lib/docker/volumes/<vol_name>/_data na a ext4 sistemụ faịlụ, ext4 na-ewepụ ihe nrịbama mgbochi kpọmkwem (i_block / oke mgbọrọgwụ osisi) na tebụl inode dị ndụ. Agbanyeghị, ext4 akwụkwọ akụkọ (inode <8>, na-emekarị 128 MB ka 1024 MB mgbanaka mgbanaka okirikiri) ka na-ejide akwụkwọ akụkọ ihe mere eme nke tebụl inode na akwụkwọ ndekọ aha ya ozugbo. docker compose down -v e gburu.

+-------------------------------------------------------------------------+
| HOW ext4magic RECOVERS A DELETED DOCKER VOLUME FROM THE EXT4 JOURNAL    |
|                                                                         |
| Live Inode Table on Disk (After docker volume rm):                      |
|   Inode #918402 (users.ibd): [Size = 0, Extent Pointers = ZEROED]       |
|                                                                         |
| Ext4 Journal Ring Buffer (Inode <8> on Disk):                           |
|   Transaction #41902 (10 mins ago):                                     |
|   Inode #918402 (users.ibd): [Size = 482MB, Extents = Blocks 9000..9999]|
|                                                                         |
| -> ext4magic reads the old Inode Copy from Journal Transaction #41902   |
|    and extracts Blocks 9000..9999 directly to your rescue drive.        |
+-------------------------------------------------------------------------+

Gbaa ọsọ ext4magic megidere nkebi naanị ọgụgụ gị (ma ọ bụ nke .img oyiri foto na sava nke abụọ), na-akọwapụta stampụ oge (-a) si **5 nkeji tupu *** ịgba ọsọ docker compose down -v ma ọ bụ docker system prune --volumes:

# Install ext4magic on your rescue machine (never on the affected partition)
sudo apt-get install -y ext4magic e2fsprogs

# Dump the ext4 journal from the frozen partition / image to a file in RAM (/dev/shm)
sudo debugfs -R "dump <8> /dev/shm/ext4_journal.copy" /dev/nvme0n1p1

# Recover all files deleted under var/lib/docker/volumes after (Current Time - 15 minutes)
sudo mkdir -p /mnt/external_rescue/docker_recovered
sudo ext4magic /dev/nvme0n1p1 \
  -j /dev/shm/ext4_journal.copy \
  -f var/lib/docker/volumes \
  -a $(date -d "-15 minutes" +%s) \
  -r -d /mnt/external_rescue/docker_recovered

Kwụpụ 4: Ịkpụpụta SQLite raw, PostgreSQL, ma ọ bụ MySQL InnoDB faịlụ mgbe akwụkwọ akụkọ ahụ kechie (XFS ma ọ bụ ext4)¶

Ọ bụrụ na onye ọbịa na-eji XFS ma ọ bụrụ na ext4 akwụkwọ akụkọ etinyelarị n'elu ihe nrịbama inode, jiri mbinye aka na ihe akpụrụ akpụ megide onyonyo diski raw (docker_host_raw.img):

1. ** Maka ọdụ data SQLite3 (.sqlite, .db - ọmụmaatụ, Vaultwarden, Nextcloud, Onye enyemaka ụlọ, Gitea, Grafana) ***: Ebe nchekwa data SQLite3 ọ bụla na-amalite site na nkụnye eji isi mee 16-byte ASCII na-enweghị mgbagha na byte 0 nke ibe 4 kilobyte mbụ ya: 53 51 4c 69 74 65 20 66 6f 72 6d 61 74 20 33 00 ("SQLite format 3\0"), ma na-echekwa nha ibe nchekwa data kpọmkwem (Bytes 16..17, na-emekarị 4096) na mkpokọta nchekwa data dị na ibe (Bytes 28..31, big-endian 32-bit integer) kpọmkwem n'ime otu nkụnye eji isi mee.
2. Gbaa ọsọ PhotoRec ya na [File Opt] -> [x] sqlite agbanyere, ma ọ bụ jiri nyochaa nkebi ahụ ozugbo grep -ab "SQLite format 3" ka ịchọta ihe nkwụghachi nke byte nke faịlụ SQLite gị ehichapụrụ wee wepụ ya na ya dd:

   # Find the exact byte offsets of every "SQLite format 3" header on the partition:
   sudo grep -ab "SQLite format 3" /dev/nvme0n1p1
   

   Ozugbo ewepụtara ya, gbaa ọsọ sqlite3 carved.db ".recover" | sqlite3 repaired.db iji wughachi ibe akwukwo osisi B ọ bụla na-emebibeghị.

1. ** Maka MySQL / MariaDB InnoDB (.ibd tebụl faịlụ)**: InnoDB ọ bụla .ibd tebụl mejupụtara usoro 16 KB (16,384 byte) ibe nwere ihe FIL_PAGE_INDEX (0x45BF) nkụnye eji isi mee na tebụl pụrụ iche space_id. Jiri Percona / TwinDB undrop-for-innodb (stream_parser + c_parser* ngwá ọrụ iji nyochaa /dev/nvme0n1p1 maka mmadụ niile 16 KB InnoDB index peeji ma tụfuo ahịrị ọ bụla ozugbo na faịlụ TSV/SQL maka mbubata.
2. ** Maka faịlụ mkpokoro PostgreSQL ($PGDATA/base/<db_oid>/<relfilenode>***: PostgreSQL na-echekwa tebụl ọ bụla ka ọ na-aga n'ihu 8 KB (8,192 byte) ibe akwụkwọ nwere PageHeaderData (pd_lower, pd_upper, pd_special). Jiri pg_filedump ma ọ bụ pglazarus / pdu inyocha mpaghara diski ekenyeghị maka 8 KB ibe dakọtara na kọlụm tebụl gị tuple okirikiri nhọrọ ukwuu (HeapTupleHeaderData) ma rụgharịa ahịrị ndị ahụ.

Usoro 4: Kubernetes PVC, PV, na etcd Foto mgbake mberede¶

Na Kubernetes (EKS, GKE, AKS, k3s, RKE2, kubeadm, OpenShift), na mberede ihichapụ PersistentVolumeClaim (kubectl delete pvc), a StatefulSet, ma ọ bụ dum Namespace (kubectl delete ns prod) na-agbaso ụkpụrụ ụlọ dị iche karịa Docker kwụ ọtọ.

Ihe atụ A: Ihe ngosi PVC Terminating (The Finalizer Lifesaver)¶

Gbaa ọsọ:

kubectl get pvc -n <namespace>

# Stream a compressed tar backup or pg_dump out of the still-running pod immediately:
kubectl exec -n <namespace> <pod-name> -- pg_dumpall -U postgres > ./rescued_k8s_db.sql
# Or tar an entire mounted volume directory out over stdout:
kubectl exec -n <namespace> <pod-name> -- tar czf - /var/lib/postgresql/data > ./rescued_pvc_files.tar.gz

Ihe atụ B: E ehichapụrụ PVC ahụ, yana kubectl get pv na-egosi PersistentVolume in Released steeti (ReclaimPolicy: Retain)¶

Gbaa ọsọ:

kubectl get pv

Otu esi eme a Released PV Available ọzọ na 10 sekọnd na efu data ọnwụ: 1. Dezie ihe PersistentVolume wee hichapụ ya spec.claimRef ngọngọ:

kubectl patch pv <your-pv-name> --type json -p '[{"op": "remove", "path": "/spec/claimRef"}]'

Ọnọdụ C: Iweghachi steeti ụyọkọ Kubernetes kpochapụrụ site na otu etcd foto (etcdctl snapshot restore)¶

Ọ bụrụkwanụ na mmadụ agbaa ọsọ kubectl delete namespace prod ma ọ bụ helm uninstall wee kpochapụ ihe ngosi 200 Kubernetes, Nzuzo, ConfigMaps, na nkọwa PV na njikwa onwe ya. kubeadm, k3s, ma ọ bụ RKE2 ụgbọ elu akara?

1. ** Chọpụta ihe kachasị ọhụrụ gị etcd foto** (site na ndabara na /var/lib/rancher/k3s/server/db/snapshots/, /var/lib/rancher/rke2/server/db/snapshots/, ma ọ bụ ahaziri gị /var/backups/etcd/snapshot.db).
2. ** Nyochaa iguzosi ike n'ezi ihe onyonyo tupu ị metụ ụyọkọ dị ndụ**:

   ETCDCTL_API=3 etcdctl snapshot status /path/to/snapshot.db --write-out=table
   

3. ** Kwụsị kube-apiserver na etcd na ọnụ(s) ụgbọ elu njikwa ***:
4. Na kubeadm (mkpụrụ osisi static): Bugharịa /etc/kubernetes/manifests/kube-apiserver.yaml na /etc/kubernetes/manifests/etcd.yaml nwa oge ka /root/manifests_hold/ so kubelet na-akwụsị akpa abụọ ahụ nke ọma, wee kwadoo ya /var/lib/etcd ka /var/lib/etcd.bak.
5. Na k3s: Gbaa ọsọ sudo systemctl stop k3s na sudo k3s server --cluster-reset --cluster-reset-restore-path=/path/to/snapshot.
6. ** Weghachite etcd ndekọ data na kubeadm**:

   sudo ETCDCTL_API=3 etcdctl snapshot restore /path/to/snapshot.db \
     --data-dir=/var/lib/etcd \
     --name=<control-plane-node-name> \
     --initial-cluster=<node-name>=https://<node-ip>:2380 \
     --initial-advertise-peer-urls=https://<node-ip>:2380
   

7. Bugharịa etcd.yaml na kube-apiserver.yaml laghachi n'ime /etc/kubernetes/manifests/. N'ime sekọnd 30, kube-apiserver jikọọ ọzọ na nke eweghachiri etcd ụlọ ahịa na oghere aha, Nzuzo, ConfigMap, na ihe na-adịgide adịgide ehichapụrụ ga-apụta.

Usoro 5: Docker na-agbasi ike na mpịakọta Kubernetes megide ihichapụ mberede¶

Ozugbo eweghachiri nchekwa data mmepụta gị, mejuputa ụzọ nchekwa ụlọ anọ ndị a ka otu CLI typo enweghị ike ihichapụ olu dị ndụ ọzọ:

• ** Gbanwee Kubernetes mmepụta ọ bụla StorageClass si reclaimPolicy: Delete ka reclaimPolicy: Retain**: Site na ndabara, klaasị nchekwa igwe ejiri arụnyere (gp2, gp3, standard-rwo) ndabara ka reclaimPolicy: Delete, nke pụtara ihichapụ PVC na-emebi diski igwe ojii. Mepụta a StorageClass ya na reclaimPolicy: Retain (ma ọ bụ mechie PV dị ugbu a na kubectl patch pv <pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}') ya mere, ihichapụ PVC ma ọ bụ Namespace anaghị ehichapụ olu ngọngọ dị n'okpuru.
• ** Kwado AWS EBS Recycle Bin / GCP Snapshot Schedules / Velero na CSI Snapshots ***: Ịkwado AWS EBS Recycle Bin na-ejigide olu EBS ọ bụla ehichapụrụ maka ụbọchị 7 ruo 30 ọbụlagodi ma onye ọkwọ ụgbọ ala CSI na-akpọ. DeleteVolume.
• ** Ejila ya docker system prune --volumes na edemede nhicha akpaaka ma ọ bụ ndị na-agba ọsọ CI / CD ***: Ọ bụrụ na igbe nchekwa data mmepụta daa obere oge (Exited (1)) n'otu oge ahụ, ọrụ cron na-arụ ọrụ docker system prune -a --volumes -f, Docker na-ewere mpịakọta ya na-amaghị aha "ejighi ya" wee kpochapụ ha kpamkpam. Mwepu --volumes site na edemede mkpocha niile.
• **Jiri ngwunye ejikọtara ndị ọbịa doro anya (/srv/prod_data/postgres:/var/lib/postgresql/data) ma ọ bụ ọdụ data ejisiri na mpụga maka oke ọrụ nrụpụta steeti *: N'adịghị ka /var/lib/docker/volumes/, nnabata akwụkwọ ndekọ aha na-ekekọta n'èzí /var/lib/docker/ bụ ** emetụghị site docker compose down -v, docker volume rm, ma ọ bụ docker system prune --volumes.

Nọmba mberede mpaghara na ụkpụrụ ụlọ ọrụ¶

Mgbe mkpochapụ akpa ma ọ bụ Kubernetes na-emetụta ọrụ ụlọ ọgwụ, izipu nchekwa ọha, ma ọ bụ data nkeonwe ahaziri, na-abawanye ozugbo:

• United States na Canada : Kpọọ 911 ma ọ bụrụ na ụyọkọ akpa ọdịda kwụsịrị nleba anya onye ọrịa, sistemu EHR, ma ọ bụ izipu ihe mberede. Kpesa ozi ọjọọ ihicha akụrụngwa ma ọ bụ ihe omume cyber na ** CISA (1-844-Say-CISA / cisa.gov/report ma ọ bụ ** Canada Center for Cyber Security (1-833-CYBER-88).
• **United Kingdom ***: Ịkpọ oku 999 maka iyi egwu ndụ ma ọ bụ akụrụngwa NHS dị oke egwu. N'okpuru UK GDPR, mbibi na-adịgide adịgide ma ọ bụ mfu nke nnweta data nkeonwe na-enweghị nkwado ndabere na mpaghara na-eweghachite chọrọ nyocha ọkwa nbibi nke awa 72 na ICO (0303 123 1113).
• ** European Union ***: Ịkpọ oku 112 gafee mba ndị otu EU niile. Ụlọ ọrụ dị n'okpuru **NIS2 Ntuziaka *** ga-edozi ịdọ aka ná ntị mbụ na CSIRT obodo ha n'ime awa 24 nke nkwụsị ọrụ siri ike, yana ọnwụ data nkeonwe na-adịgide adịgide na-akpalite ** GDPR Nkeji edemede 33 *** mkpesa n'ime awa 72.
• **Australia na New Zealand ***: Ịkpọ oku 000 na Australia ma ọ bụ 111 na New Zealand; kọọ akụkọ ihe gbasara akụrụngwa dị egwu na ACSC (1300 CYBER1).
• **India na Asia-Pacific : Ịkpọ oku 112 na India maka nzaghachi mberede; kọọ ihe mere nchekwa igwe ojii ma ọ bụ akpa na ** CERT-In (incident@cert-in.org.in n'ime mpio awa 6 amanyere iwu.

Ajụjụ a na-ajụkarị¶

Ọ na-eme docker compose down -v hichapụ ihe ejikọtara ọnụ (./data:/var/lib/mysql) ma ọ bụ naanị mpịakọta Docker aha na amaghị aha?¶

`docker compose down -v` Naanị na-ehichapụ mpịakọta akpọpụtara na ya `volumes:` ngalaba nke `docker-compose.yml` na mpịakọta na-amaghị aha jikọtara na arịa; ya ọ dịghị mgbe na-ehichapụ faịlụ n'ime ebe a na-ekekọta onye ọbịa (dịka `- /opt/mysql_data:/var/lib/mysql` ma ọ bụ `- ./postgres_data:/var/lib/postgresql/data`). Ọ bụrụ na idepụta faịlụ gị jiri ụzọ nnabata malite na `/` ma ọ bụ `./`, Lelee folda ahụ na onye ọbịa ugbu a: faịlụ gị ka dị 100 percent.

Gịnị mere docker system prune --volumes hichapụ olu nchekwa data m mgbe naanị m kwụsịrị akpa ahụ maka nkeji 2 iji melite mgbanwe gburugburu ebe obibi?¶

Chere nke ọma maka ọdịiche dị n'etiti `docker stop` na `docker rm`: ọ bụrụ na akpa bụ naanị `Stopped / Exited` (dị na `docker ps -a`), `docker system prune --volumes` eme ọ bụghị hichapụ mpịakọta ya n'ihi na akpa ahụ kwụsịrị ka na-ezo aka na ha. Agbanyeghị, ọ bụrụ na ị gbara ọsọ `docker rm ` (iji megharịa akpa ahụ na ọkọlọtọ ọhụrụ) ma ọ bụ ọ bụrụ na ị gafere `docker system prune -a --volumes` (nke mbụ na-ehichapụ igbe niile kwụsịrị na mgbe ahụ na-ehichapụ mpịakọta niile anaghịzi ejikọta na akpa ọ bụla), olu ahụ ghọrọ "na-akụda" na millise seconds wepụrụ akpa ahụ kwụsịrị ma hichaa ozugbo.

Enwere m ike iweghachi olu Docker ehichapụ n'ime Desktọpụ Docker na Mac (Apple Silicon / Intelma ọ bụ Windows (WSL2)?¶

Ee, ọ bụrụ na ị kwụsịrị Docker Desktop ozugbo ka igwe mebere akwụsị ịgba ọsọ `fstrim` megide mebere diski. Na macOS, Docker na-echekwa mpịakọta Linux niile n'ime `-/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw`; na Windows WSL2, n'ime `%LocalAppData%\Docker\wsl\disk\docker_data.vhdx`. Detuo ya `.raw` ma ọ bụ `.vhdx` faịlụ na SSD mpụga ozugbo, tinye ya na-agụ naanị na Linux (iji `losetup -Pr` maka `.raw` ma ọ bụ `qemu-nbd -r` maka `.vhdx`), na-agba ọsọ `ext4magic` ma ọ bụ `PhotoRec` megide mebere `ext4` nkebi n'ime.

Gịnị ma ọ bụrụ na m ji ehichapụ Kubernetes StatefulSet kubectl delete statefulset postgres? Nke ahụ ehichapụkwara PVC m?¶

Mba. Site na ndabara na Kubernetes (belụsọ `persistentVolumeClaimRetentionPolicy.whenDeleted: Delete` ahaziri nke ọma na Kubernetes 1.27+), na-agba ọsọ `kubectl delete statefulset` na-ehichapụ ihe njikwa StatefulSet na Pods ya, na-ahapụ ihe niile `PersistentVolumeClaims` (`data-postgres-0`, `data-postgres-1`) na data ha dị n'okpuru 100 emebibeghị. Naanị tinyegharịa nke gị `StatefulSet` YAML gosipụtara na otu aha na `volumeClaimTemplates`, na Pọd ọhụrụ ahụ ga-ejikọ ọzọ na nke dị adị ozugbo `data-postgres-0` PVC.

Kedu ka m ga-esi nweta faịlụ n'ime Docker overlay2 ndekọ mgbe akpa jụrụ ibido (CrashLoopBackOff / Exited (1))?¶

Ehichapụla akpa ahụ dara ada. Ọ bụrụhaala na akpa kwụsịrị ka dị n'ime ya `docker ps -a`, oyi akwa nke a na-ede ede na oke olu na-adị kpamkpam na diski. Ị nwere ike idetu faịlụ ozugbo site na akpa ahụ kwụsịrị site na iji `docker cp :/var/lib/postgresql/data /root/rescued_data`, ma ọ bụ chọta ya kpọmkwem onye ọbịa `overlay2` ụzọ site na-agba ọsọ `docker inspect --format '{{.GraphDriver.Data.UpperDir}}'` na-agụ akwụkwọ ndekọ ahụ ozugbo dị ka `root`.

Kedu ihe ma ọ bụrụ na mmadụ ehichapụ Kubernetes Namespace (kubectl delete ns prod) na PVC niile nwere reclaimPolicy: Delete na AWS EBS ma ọ bụ GCP?¶

Ọ bụrụ na igwe ojii CSI ọkwọ ụgbọ ala (`ebs.csi.aws.com` ma ọ bụ `pd.csi.storage.gke.io`) egbuolarị `DeleteVolume` n'ihi na `reclaimPolicy` bụ `Delete`, lelee ụgbụ nchekwa nchekwa igwe ojii atọ ozugbo: (1) lelee ** AWS Recycle Bin for EBS Volumes ** na njikwa AWS EC2 (ọ bụrụ na iwu njide na-arụ ọrụ na akaụntụ / mkpado, olu EBS ehichapụ na-anọdụ na Recycle Bin na enwere ike iweghachi ya na 1 pịa); (2) lelee ** AWS ndabere / DLM / GCP nhazi oge foto ** maka foto olu akpaghị aka kacha ọhụrụ; na (3) lelee ma nchekwa data gị na-enuba WAL/binlogs (`WAL-G`, `pgBackRest`, `Litestream`) gaa na bọket S3/GCS.

Ntuziaka emetụtara¶

• Ihe nkesa diski 100% zuru (0 bytes efu): nchekwa data WAL log bloat, osisi B mebiri emebi, yana mgbake oghere efu.
• Mberede `rm-rf' ma ọ bụ 'Shift+ Delete' na Linux, macOS, ma ọ bụ Windows: naanị ihe na-agụ naanị na-ebughachi na ihe osise inode
• Igwe mebere VMDK, VHDX, na qcow2 nrụrụ aka yinye ihe onyonyo na nnapụta hypervisor
• Git ihe mberede mere nke igodo AWS, faịlụ .env, ma ọ bụ igodo SSH nkeonwe maka GitHub repo ọha.
• Ọgba aghara nnabata webụ, mbugharị, na mweghachi ndabere
• Ndokwa kọmputa, atụmatụ 3-2-1, na mgbake ọdachi
• Nọmba mberede site na obodo

Isi mmalite na nduzi ọchịchị¶

• Akwụkwọ ikike Kubernetes (kubernetes.io / Cloud Native Computing Foundation): * Ihe Nchekwa na Nchekwa ojiji (kubernetes.io/pvc-protection Ndị mmecha), na-eweghachite olu na-adịgide adịgide (Retain vs. Delete), na arụ ọrụ etcd ụyọkọ maka Kubernetes (etcdctl snapshot restore)*
• Ụlọ ọrụ Docker Inc.docs.docker.com): * Jikwaa data na Docker: Mpịakọta, Kechie Ugwu, overlay2 Ihe owuwu onye na-anya ụgbọ ala, yana ntụaka prune*.
• Òtù Linux Kernel Organisationkernel.org): * Nke /proc Sistemụ faịlụ (procfs Mepee File Descriptor Iweghachite site na /proc/<pid>/fd/) na Ext4 akwụkwọ nkọwa *.
• ext4magic Akwụkwọ oru ngo (ext4magic.sourceforge.net): * Forensic Inode na Journal azụmahịa Iweghachite na Ext3/Ext4 Filesystem*.
• National Institute of Standards and Technology (NIST) (United States): * Mbipụta Pụrụ Iche 800-190: Ntuziaka Nchekwa Ngwa & Iweghachite Mberede *.