Skip to content

Volume Produksi Docker utawa Kubernetes Dibusak: Nylametake docker system prune --volumes, overlay2 Inodes, lan etcd Snapshots

Nomer darurat beda-beda miturut negara. Kaca iki nggunakake 911, nomer kanggo Amerika Serikat lan Kanada. Akeh negara liya nggunakake 112, 999, utawa nomer liyane. Temokake nomer kanggo negara sampeyan.

mlaku docker compose down -v, docker system prune -a --volumes, docker volume rm, utawa kubectl delete pvc ing tab terminal sing salah ing produksi minangka salah sawijining cara paling cepet kanggo ngilangke volume database PostgreSQL, MySQL, MariaDB, Redis, MongoDB, utawa SQLite langsung. Ora kaya gambar mesin virtual maya sing pindhah menyang tong sampah 7 dina, driver volume lokal Docker (/var/lib/docker/volumes/<name>/_data) lan Kubernetes local-path / hostPath Penyedia langsung ngetokake POSIX standar unlink() lan rmdir() nelpon langsung marang sistem file host Linux sing ndasari (ext4, XFS, utawa Btrfs). Saben tulisan sing kedadeyan ing host Linux kasebut sawise printah rampung bisa nimpa pamblokiran database sing dibebasake ing disk. Yen disk server sampeyan diisi nganti 100 persen (0 bytes free) sadurunge ana wong mlayu docker system prune --volumes ing gupuh, konsultasi disk server 100% lengkap, database WAL log bloat, lan nol-spasi Recovery; kanggo pambusakan file Linux bare-metal njaba wadhah, ndeleng sengaja rm -rf utawa Shift-Delete ing ext4, XFS, APFS, lan NTFS: remount mung diwaca lan ukiran inode lan mesin virtual VMDK, VHDX, lan qcow2 snapshot chain korupsi lan ngluwari hypervisor. Yen pemadaman produksi iki nyebabake perawatan pasien rumah sakit, kiriman darurat, utawa sistem keamanan sarana umum, hubungi nomer darurat lokal sampeyan (911 ing Amerika Serikat lan Kanada, 112 liwat Uni Eropa, 999 ing Inggris) utawa priksa direktori kita nomer darurat.

Apa iki saiki

  1. ** AJA mlayu docker compose up -d, systemctl restart docker, utawa redeploy pod Kubernetes**: Nggawe maneh wadhah utawa miwiti maneh daemon Docker nulis megabyte anyar overlay2 metadata, log JSON wadah, lan file database kosong sing anyar diinisialisasi (initdb) langsung liwat persis ext4 utawa XFS pamblokiran ing ngendi volume sing wis dibusak disimpen sawetara detik kepungkur.
  2. ** Priksa ing 5 detik yen proses wadhah (postgres, mysqld, redis-server, node, python) isih mlaku ing memori (ps aux lan sudo lsof +L1)**: Yen ana wong sing mbusak file ing volume sing dipasang utawa mbatalake volume anonim nalika wadhah isih mlaku, kernel Linux tetep 100 persen file database sing ora disambungake urip ing disk anggere ID Proses host wadah kasebut (PID) nduwe deskriptor file sing mbukak (FD). Sampeyan bisa nyalin file kasebut langsung metu saka /proc/<PID>/fd/ utawa /proc/<PID>/root/ karo mundhut data nul sadurunge wadhah mandheg.
  3. Yen wadhah wis mandheg lan volume wis dibusak (docker compose down -v utawa docker system prune --volumes), beku disk host langsung nulis: Mungkasi daemon Docker (sudo systemctl stop docker.socket docker.service containerd) supaya wadhah latar mburi lan healthchecks mandheg nulis log menyang /var/lib/docker, banjur remount /var/lib/docker (utawa root / partisi) mung diwaca (sudo mount -o remount,ro /) utawa njupuk potret volume awan tingkat blok langsung (AWS EBS Snapshot, GCP Persistent Disk Snapshot, DigitalOcean Volume Snapshot) sadurunge ndemek apa wae.
  4. Yen sampeyan ana ing Kubernetes (kubectl delete pvc utawa kubectl delete namespace), priksa PersistentVolume (PV) ReclaimPolicy (kubectl get pv) langsung:
  5. Yen ing PV wis persistentVolumeReclaimPolicy: Retain, data sampeyan ora dibusak: ing PV mung dipindhah menyang Released status lan disk awan sampeyan (EBS, Azure Disk, Ceph RBD, NFS) 100 persen utuh.
  6. Malah yen kubectl delete pvc wis macet ing Terminating, Kubernetes Finalizers (kubernetes.io/pvc-protection) mblokir driver CSI saka mbusak panyimpenan ndasari anggere sembarang Pod isih referensi PVC sing. Aja mbusak Pod nganti sampeyan nyalin data metu saka Pod sing mlaku.
  7. Gambar partisi blok sing ndasari (dd / ddrescue liwat SSH) menyang disk eksternal utawa volume awan sing kapisah sadurunge mlaku ext4magic, extundelete, utawa PhotoRec: Aja mbukak pemahat file sing nulis file output menyang partisi disk sing padha sing sampeyan pulih /var/lib/docker/volumes.
  8. Kanggo kahanan kluster Kubernetes sing rusak utawa ora sengaja (kubectl delete crd / namespace), mandeg kube-apiserver lan mulihake paling anyar etcd snapshot (etcdctl snapshot restore): Yen klaster negara ing etcd wis dibusak nalika volume panyimpenan maya ndasari isih ana, mulihake etcd waras kabeh PV, PVC, Secret, lan ConfigMap obyek bebarengan.

Danger

** Aja mbusak kubernetes.io/pvc-protection utawa kubernetes.io/pv-protection finalizer karo kubectl patch ... -p '{"metadata":{"finalizers":null}}' nalika PVC macet ing Terminating**: Finalizer iku mung siji-sijine sing njaga driver CSI awan sampeyan (aws-ebs-csi-driver, pd.csi.storage.gke.io, Longhorn, Rook-Ceph) saka nerbitake hardware DeleteVolume Telpon API kanggo numpes disk produksi. Nalika PVC macet ing Terminating, Pod isih mlaku lan volume isih dipasang: mbukak kubectl exec utawa kubectl cp (utawa pg_dump / mysqldump) langsung kanggo ngluwari 100 persen database sampeyan sadurunge Pod mungkasi.


Wit keputusan: volume Docker sing wis dibusak, overlay2 lapisan, utawa Kubernetes PVC triage

Gunakake diagram alur iki kanggo milih dalan pemulihan nol-rugi sing paling cepet adhedhasar apa proses wadhah isih urip, volume didhukung dening disk lokal utawa panyimpenan CSI awan, lan sistem file sing dadi host. /var/lib/docker:

flowchart TD
    Start["Data Produksi Ilang Sawise Docker utawa Kubernetes Mbusak Command"] --> WhichEnv{"Platform & Komando Endi sing Nimbulake Kerugian?"}

    WhichEnv -->|"Docker: Container Still Running (File Unlinked Inside Volume/Container)"| ProcRescue["Rescue Open Unlinked File Descriptors liwat sudo lsof +L1 lan /proc/PID/fd/N utawa /proc/PID/root/"]

    WhichEnv -->|"Docker: Kontainer Dihentikan & Volume Dibusak (nyipta mudhun -v / prune --volume)"| StopDocker["Langsung Run: sudo systemctl stop docker.socket docker containerd"]

    WhichEnv -->|"Kubernetes: kubectl mbusak pvc / pv / statefulset / namespace"| K8sCheck{"Apa Pod Isih Mlaku UTAWA PVC Macet ing Terminasi?"}

    K8sCheck -->|"Ya: Pod Still Running (Finalizer Blocking CSI Delete)"| K8sExec["AJA Mbusak Pod -> Jalanake kubectl exec / pg_dump / tar liwat stdout menyang Disk Lokal Langsung"]
    K8sCheck -->|"Ora: Pod Gone -> Priksa kubectl njaluk pv"| ReclaimCheck{"Apa Kebijakan Reclaim PV?"}

    ReclaimCheck -->|"Retain (Status PV = Dirilis)"| ClearClaimRef["100% Data Utuh -> Sunting PV kanggo Mbusak spec.claimRef.uid -> Ikatan maneh menyang PVC Anyar"]
    ReclaimCheck -->|"Busak (Cloud EBS / PD / Longhorn / Local-Path)"| CloudOrLocal{"Volume CSI Cloud utawa Disk Host Lokal?"}

    CloudOrLocal -->|"Cloud CSI (AWS EBS / GCP / Azure / Ceph)"| CloudSnap["Priksa Keranjang Daur Ulang Cloud Console / Snapshot EBS Otomatis / Ceph RBD Trash (rbd trash ls)"]
    CloudOrLocal -->|"Local Node Disk (local-path / hostPath)"| StopDocker

    StopDocker --> RemountRO["Remount Host Partition Read-Only (mount -o remount,ro) UTAWA Gambar Blokir Stream liwat SSH"]
    RemountRO --> HostFS{"Apa Filesystem Hosts /var/lib/docker?"}
    HostFS -->|"ext4"| Ext4Rec["Gunakake debugfs / ext4magic Journal Inode Transaction Rollback (-cap wektu)"]
    HostFS -->|"XFS / Btrfs"| CarveRec["Gunakake Btrfs Roots / xfs_irecover / Custom Signature Carving kanggo SQLite, Postgres Heap, utawa InnoDB .ibd"]

Fase 1: Kepiye Docker lan Kubernetes mbusak volume ing disk

Sadurunge nglakokake prentah pemulihan, ngerti ngendi data sampeyan urip ing host Linux lan apa sing didemek printah mbusak:

Jinis Panyimpenan & Busak Command Host Filesystem Path ing Linux Node Apa sing Ditindakake dening Mbusak Command Cara Recovery Langsung
Named Docker Volume (docker volume rm utawa docker compose down -v) /var/lib/docker/volumes/<vol_name>/_data Mbusak link file ing njero _data/ lan mbusak entri direktori saka /var/lib/docker/volumes/metadata.db (boltdb). Mungkasi Docker langsung, remount /var/lib/docker mung diwaca, lan mbalekake inode sing ora disambung liwat ext4magic / debugfs utawa ukiran teken.
Volume Docker Anonim (docker run -v /var/lib/mysql + docker rm -v utawa docker system prune --volumes) /var/lib/docker/volumes/<64-char-hex-id>/_data docker system prune --volumes mbusak saben volume anonim lan jenenge sing saiki ora dirujuk dening paling ora siji wadhah sing wis ana. Mungkasi Docker langsung; yen sampeyan duwe ** liyane ** mandegake wadhah sing volume ora * pruned, mriksa isih /var/lib/docker/volumes/*/_data folder pisanan sadurunge ngukir pamblokiran unallocated.
** Lapisan Wadhah sing Bisa Ditulis (docker rm <container> nalika data ditulis ing njaba volume)** /var/lib/docker/overlay2/<layer-id>/diff/ Mbukak link ndhuwur wadhah diff/ direktori nang overlay2 driver panyimpenan. Mungkasi Docker langsung lan waras wit direktori sing ora disambungake saka file ext4 jurnal utawa XFS pamblokiran.
** Docker Desktop ing macOS utawa Windows (docker system prune --volumes)** Ing siji gambar disk mesin virtual: -/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw (macOS) utawa %LocalAppData%\Docker\wsl\disk\docker_data.vhdx (Windows WSL2). Dikirim unlink() nang Linux VM, lan yen discard/TRIM aktif ing Docker.raw / docker_data.vhdx, punched bolongan jarang ing file inang. Mungkasi Docker Desktop sajrone 5 detik, salin Docker.raw utawa docker_data.vhdx menyang drive external, lan mbukak ext4magic / PhotoRec marang gambar disk virtual.
** Kubernetes PVC karo Delete Kabijakan Reclaim ing Ceph RBD (Rook-Ceph) utawa Longhorn** Kolam Ceph OSD (rbd) utawa /var/lib/longhorn/replicas/. Yen csi-rbdplugin wis trash diaktifake (rbd trash), gambar RBD dipindhah menyang tong sampah Ceph; digunakake pamblokiran sing unmapped. Mlayu rbd trash ls --pool <pool> ing njero rook-ceph-tools pod lan mulihake karo rbd trash restore.

Fase 2: Nylametake file database sing mbukak saka wadhah sing isih aktif (/proc/<pid>/fd/)

Enteni: wadhahe isih mlaku saiki? Asring, insinyur nglakokake skrip migrasi aplikasi, tugas cron pembersihan, utawa manual rm -rf /var/lib/postgresql/data/* utawa rm /app/data/production.sqlite3 nang wadhah mlaku, utawa mbusak folder bind-mount host ndasari /opt/app/data nalika wadhah dhewe (postgres, mysqld, mariadbd, redis-server, mongod, gunicorn, node) isih munggah lan mlaku.

Ing Linux, file ora dibusak saka disk nalika rm utawa unlink() diarani. Blok disk lan inode tetep ** 100 persen dialokasikan lan bisa diwaca ** nganti proses pungkasan nyekel deskriptor file mbukak (FD) menyang inode kasebut nutup utawa metu.

Langkah 1: Temokake ID Proses host (PID) saka proses wadhah

Ing simpul host Linux (ora ana ing wadhah), goleki host PID saka database utawa proses aplikasi:

sudo lsof +L1 | grep -E "postgres|mysqld|mariadbd|sqlite|redis|mongo|node|python|java"
Deleng ing output:
COMMAND     PID   USER   FD   TYPE DEVICE  SIZE/OFF NLINK    NODE NAME
mysqld    18492    999   14u   REG  259,1 482344960     0 1849201 /var/lib/mysql/app_prod/users.ibd (deleted)
mysqld    18492    999   15u   REG  259,1 125829120     0 1849202 /var/lib/mysql/app_prod/orders.ibd (deleted)
mysqld    18492    999   16u   REG  259,1  79691776     0 1849190 /var/lib/mysql/ibdata1 (deleted)
python3   22104   1000    4u   REG  259,1  89128960     0 3102944 /data/production.sqlite3 (deleted)
Deleng ing kolom NLINK: kuwi 0, lan jeneng berkas dipungkasi nganggo (deleted). File wis rampung ilang saka ls /var/lib/docker/volumes/..., nanging ** saben bait (482 MB ing users.ibd, 89 MB ing production.sqlite3) isih urip ing njero /proc/<PID>/fd/<FD>**.

Langkah 2: Nyalin file database sing ora disambungake langsung /proc/<PID>/fd/

Aja ** aja ** mandheg utawa miwiti maneh wadhah. Nalika proses isih mlaku, nyalin deskriptor file mentah langsung menyang folder sing aman ing host sampeyan:

sudo mkdir -p /root/proc_emergency_rescue

# Example 1: Rescuing an unlinked SQLite3 database (PID 22104, FD 4)
sudo cp /proc/22104/fd/4 /root/proc_emergency_rescue/production.sqlite3
# Check if PID 22104 also holds the -wal file open (e.g. FD 5) and copy it too:
sudo cp /proc/22104/fd/5 /root/proc_emergency_rescue/production.sqlite3-wal

# Example 2: Automatically rescuing ALL open (deleted) MySQL/Postgres files for PID 18492
for fd in /proc/18492/fd/*; do
    target=$(sudo readlink "$fd" 2>/dev/null)
    if echo "$target" | grep -q "(deleted)"; then
        fd_num=$(basename "$fd")
        clean_name=$(echo "$target" | sed 's/ (deleted)//; s/\//_/g')
        echo "Rescuing FD $fd_num -> $clean_name"
        sudo cp "$fd" "/root/proc_emergency_rescue/${fd_num}${clean_name}"
    fi
done

Langkah 3: Luwih apik: Jalanake dump logis langsung (pg_dump / mysqldump / .backup) sadurunge wadhah nutup FDs

Yen postgres utawa mysqld isih mlaku karo file data sawijining mbukak ing memori lan /proc/<PID>/fd/, coba mbukak dump logis langsung liwat soket unix utawa sambungan TCP sing ana dhisik:

# If the database engine can still read its open file descriptors:
docker exec <container_name> pg_dumpall -U postgres > /root/proc_emergency_rescue/live_dump.sql
# Or for MySQL:
docker exec <container_name> mysqldump -u root -p --all-databases --single-transaction > /root/proc_emergency_rescue/live_mysql.sql
* (Cathetan: PostgreSQL mbukak lan nutup .1, .2 file bagean antarane pangolahan buruh dinamis, supaya pg_dump bisa gagal ing tabel sing ora disimpen; nggabungake query SQL urip karo nyalin saben mbukak /proc/<pid>/fd/* ing kabeh postgres PID backend anak nyimpen data tumpukan tabel maksimal.)*


Fase 3: Mbalekake volume Docker sing wis dibusak ext4 utawa XFS sawise wadhah mandheg

Apa yen sampeyan mlayu docker compose down -v utawa docker system prune --volumes, dadi wadhah ** wis mandheg ** lan kabeh /proc/<pid>/fd/ deskriptor ditutup? Saiki sampeyan lagi lomba nglawan latar mburi disk nulis ing host Linux.

Langkah 1: Beku sistem file host Linux sajrone 10 detik

  1. Sampun Docker, containerd, kubelet, lan cron sanalika dadi ora ana wadah, healthcheck, utawa log rotator nulis byte liyane menyang /var/lib/docker:
    sudo systemctl stop docker.socket docker.service containerd.service cron.service
    
  2. ** Priksa piranti pemblokiran lan sistem file sing ditahan /var/lib/docker**:
    df -Th /var/lib/docker
    
    Filesystem     Type  Size  Used Avail Use% Mounted on
    /dev/nvme0n1p1 ext4  100G   42G   58G  42% /
    
  3. ** Pasang maneh partisi Mung Waca (utawa stream gambar partisi mentah liwat SSH menyang mesin liyane)**:
  4. Yen /var/lib/docker ana ing partisi khusus utawa volume LVM dhewe (contone /dev/nvme0n1p2 utawa /dev/mapper/vg0-docker), pasang maneh mung bisa diwaca:
    sudo mount -o remount,ro /var/lib/docker
    
  5. Yen /var/lib/docker ana ing oyod / pemisahan (/), coba meksa partisi root mung diwaca nggunakake pemicu Magic SysRq kernel (echo u | sudo tee /proc/sysrq-trigger) utawa sudo mount -o remount,ro /.
  6. Yen sampeyan nganggo VPS awan (AWS EC2, GCP, DigitalOcean, Hetzner, Linode): Langsung menyang konsol web awan banjur klik Gawe Snapshot ing volume blok server (utawa stream piranti blok mentah liwat SSH menyang server liya: sudo dd if=/dev/nvme0n1p1 bs=4M status=progress | gzip -1 | ssh user@backup-server "cat > /mnt/storage/docker_host_raw.img.gz").

Langkah 2: Priksa manawa data sampeyan isih ana ing volume LAIN sing ora dipotong utawa overlay2 lapisan

Sadurunge ngukir pamblokiran disk sing ora disedhiyakake, priksa manawa database sampeyan ora disimpen ing bind mount utawa volume sing beda-beda sing isih urip:

# Search the entire /var/lib/docker tree read-only for any surviving database files:
sudo find /var/lib/docker -name "PG_VERSION" -o -name "ibdata1" -o -name "*.sqlite*" -o -name "*.db" -o -name "dump.rdb" -o -name "WiredTiger" 2>/dev/null
Saben minggu, administrator gupuh sawise mlaku docker volume rm app_db_data, mung kanggo nemokake karo find /var/lib/docker sing docker-compose.yml wis bener digawe projectname_app_db_data (kang isih 100 persen utuh ing disk) nalika app_db_data minangka volume lawas sing ora digunakake wiwit nem sasi kepungkur.

Langkah 3: Mbalekake direktori volume Docker sing ora disambungake ext4 nggunakake ing ext4 Jurnal (ext4magic lan debugfs)

kapan docker volume rm mbusak /var/lib/docker/volumes/<vol_name>/_data ing an ext4 sistem berkas, ext4 nol metu pointer blok langsung (i_block / oyod wit ombone) ing tabel inode urip. Nanging, ext4 jurnal (node <8>, biasane 128 MB kanggo 1024 MB buffer ring bunder) isih nyimpen salinan sajarah tabel inode lan blok direktori sing tepat saka sadurunge docker compose down -v dieksekusi.

+-------------------------------------------------------------------------+
| HOW ext4magic RECOVERS A DELETED DOCKER VOLUME FROM THE EXT4 JOURNAL    |
|                                                                         |
| Live Inode Table on Disk (After docker volume rm):                      |
|   Inode #918402 (users.ibd): [Size = 0, Extent Pointers = ZEROED]       |
|                                                                         |
| Ext4 Journal Ring Buffer (Inode <8> on Disk):                           |
|   Transaction #41902 (10 mins ago):                                     |
|   Inode #918402 (users.ibd): [Size = 482MB, Extents = Blocks 9000..9999]|
|                                                                         |
| -> ext4magic reads the old Inode Copy from Journal Transaction #41902   |
|    and extracts Blocks 9000..9999 directly to your rescue drive.        |
+-------------------------------------------------------------------------+

Mlayu ext4magic marang partisi mung diwaca (utawa .img salinan snapshot ing server kapindho), nemtokake timestamp (-a) saka 5 menit sadurunge sampeyan mlayu docker compose down -v utawa docker system prune --volumes:

# Install ext4magic on your rescue machine (never on the affected partition)
sudo apt-get install -y ext4magic e2fsprogs

# Dump the ext4 journal from the frozen partition / image to a file in RAM (/dev/shm)
sudo debugfs -R "dump <8> /dev/shm/ext4_journal.copy" /dev/nvme0n1p1

# Recover all files deleted under var/lib/docker/volumes after (Current Time - 15 minutes)
sudo mkdir -p /mnt/external_rescue/docker_recovered
sudo ext4magic /dev/nvme0n1p1 \
  -j /dev/shm/ext4_journal.copy \
  -f var/lib/docker/volumes \
  -a $(date -d "-15 minutes" +%s) \
  -r -d /mnt/external_rescue/docker_recovered
- -j /dev/shm/ext4_journal.copy: Waca snapshot jurnal sing disimpen ing RAM sanalika sawise mandheg Docker. - -f var/lib/docker/volumes: Nargetake wit direktori volume Docker sing tepat (cathetan: ngilangi titik utama / ing ext4magic jalur relatif). - -a $(date -d "-15 minutes" +%s): Ngandika ext4magic kanggo mbangun maneh wit direktori nggunakake salinan inode sing bener 15 menit kepungkur (tengen sadurunge mbusak ora sengaja). - -r: Mbalekake direktori lengkap sing ora ana blok sing ditimpa; yen -r kantun file sebagian ditindhes, mbukak pass kapindho ngganti -r karo -m (ukir sihir + mode hibrida jurnal).

Langkah 4: Ukir file mentah SQLite, PostgreSQL, utawa MySQL InnoDB nalika jurnal wis dibungkus (XFS utawa ext4)

Warning

**Kenapa XFS ora ndhukung ext4magic lan mbutuhake langsung unmounting mung diwaca **: Boten kados ext4, sing nulis salinan inode 256-byte lengkap menyang jurnal, XFS nyathet delta maksud metadata lan kanthi agresif nggunakake maneh wit B inode sing dibebasake (agi). Yen /var/lib/docker lungguh ing an XFS partisi (standar ing RHEL, Rocky Linux, AlmaLinux, lan Amazon Linux), unmounting /var/lib/docker utawa remounting / mung diwaca ing sawetara detik minangka langkah sing paling penting sadurunge ngukir.

Yen host nggunakake XFS utawa yen ext4 jurnal sing wis dibungkus ing pointer inode, gunakake tanda tangan lan ukiran struktur ing gambar disk mentah (docker_host_raw.img):

  1. ** Kanggo Database SQLite3 (.sqlite, .db - contone, Vaultwarden, Nextcloud, Home Assistant, Gitea, Grafana)**: Saben database SQLite3 diwiwiti kanthi header ASCII 16-byte sing ora jelas ing bait 0 saka kaca 4 kilobyte pisanan: 53 51 4c 69 74 65 20 66 6f 72 6d 61 74 20 33 00 ("SQLite format 3\0"), lan nyimpen ukuran kaca database sing tepat (Bytes 16..17, biasane 4096) lan ukuran database total ing kaca (Bytes 28..31, integer 32-bit big-endian) ing jero header sing padha.
  2. Mlayu PhotoRec karo [File Opt] -> [x] sqlite aktif, utawa mindai partisi langsung karo grep -ab "SQLite format 3" kanggo nemokake offset byte pas file SQLite dibusak lan extract karo dd:
    # Find the exact byte offsets of every "SQLite format 3" header on the partition:
    sudo grep -ab "SQLite format 3" /dev/nvme0n1p1
    
    Sawise dijupuk, mbukak sqlite3 carved.db ".recover" | sqlite3 repaired.db kanggo mbangun maneh saben kaca godhong B-wit utuh.

Tip

** Carane ngetung persis dd ukuran byte file SQLite3 sing ora disambungake saka header**: Sawise grep -ab "SQLite format 3" menehi offset byte wiwitan OFFSET saka database SQLite dibusak, maca bita 16..17 (Uint16 Ukuran Halaman Big-endian, contone, 0x1000 = 4096 bita) lan bita 28..31 (Uint32 Big-endian Page Count, contone, 25000 kaca) nggunakake hexdump -C -s OFFSET -n 32. Multiplying Page Size x Page Count menehi dawa bita pas kanggo pass kanggo dd (iflag=skip_bytes,count_bytes) nalika file kasebut cedhak karo disk.

  1. ** Kanggo MySQL / MariaDB InnoDB (.ibd file tabel)**: Saben InnoDB .ibd Tabel kasusun saka urutan 16 KB (16,384 byte) kaca sing ngemot FIL_PAGE_INDEX (0x45BF) header lan tabel kang unik space_id. Gunakake Percona / TwinDB undrop-for-innodb (stream_parser + c_parser) toolkit kanggo mindhai /dev/nvme0n1p1 kanggo kabeh 16 KB Kaca indeks InnoDB lan mbucal saben baris langsung menyang file TSV/SQL kanggo diimpor maneh.
  2. ** Kanggo PostgreSQL Heap Files ($PGDATA/base/<db_oid>/<relfilenode>)**: PostgreSQL nyimpen saben tabel minangka contiguous 8 KB (8,192 byte) kaca tumpukan ngemot PageHeaderData (pd_lower, pd_upper, pd_special). Gunakake pg_filedumputawa pglazarus / pdu kanggo mindai wilayah disk sing ora dialokasikan kanggo 8 KB kaca sing cocog karo tata letak kolom tabel sampeyan (HeapTupleHeaderData) lan mbangun maneh baris.

Fase 4: Kubernetes PVC, PV, lan etcd Snapshot Recovery Darurat

Ing Kubernetes (EKS, GKE, AKS, k3s, RKE2, kubeadm, OpenShift), kanthi ora sengaja mbusak PersistentVolumeClaim (kubectl delete pvc), StatefulSet, utawa kabeh Namespace (kubectl delete ns prod) nderek arsitektur sing beda banget tinimbang Docker mandiri.

Skenario A: PVC nuduhake Terminating (Penyelamat Lifesaver)

mlaku:

kubectl get pvc -n <namespace>
Yen ing STATUS kolom ngandika Terminating, ambegan lega: data sampeyan durung dibusak. - Kenging punapa? Saben Kubernetes PVC digawe Wiwit Kubernetes 1.9 otomatis kalebu finalizers: [kubernetes.io/pvc-protection]. - Sanalika Pod isih dipasang ing PVC kasebut, pengontrol Kubernetes ora gelem ngrampungake pambusakan PVC lan ** ora tau nelpon driver CSI. DeleteVolume RPC**. - Tindakan Langsung: Aja Aja mbusak utawa miwiti maneh Pod, lan aja NOT patch metu finalizer. Langsung nyalin kabeh data metu saka Pod sing mlaku menyang mesin lokal utawa ember S3:
# Stream a compressed tar backup or pg_dump out of the still-running pod immediately:
kubectl exec -n <namespace> <pod-name> -- pg_dumpall -U postgres > ./rescued_k8s_db.sql
# Or tar an entire mounted volume directory out over stdout:
kubectl exec -n <namespace> <pod-name> -- tar czf - /var/lib/postgresql/data > ./rescued_pvc_files.tar.gz

Skenario B: PVC dibusak, lan kubectl get pv nuduhake PersistentVolume ing Released negara (ReclaimPolicy: Retain)

mlaku:

kubectl get pv
Yen PersistentVolume sampeyan (pv-prod-postgres) isih kadhaptar karo STATUS: Released (amarga StorageClass digunakake reclaimPolicy: Retain), ** 100 persen data sampeyan isih ora disentuh ing disk dhasar **. Nanging, Kubernetes nolak supaya PVC anyar ikatan menyang a Released PV amarga PV isih ngandhut a spec.claimRef pointing menyang lawas dibusak PVC kang UID.

Carane nggawe a Released PV Available maneh ing 10 detik tanpa mundhut data: 1. Owahi obyek PersistentVolume lan mbusak spec.claimRef blok:

kubectl patch pv <your-pv-name> --type json -p '[{"op": "remove", "path": "/spec/claimRef"}]'
2. Mlayu kubectl get pv maneh: status langsung ganti saka Releasedkanggo Available. 3. Gawe maneh sampeyan PersistentVolumeClaim (spesifikasi volumeName: <your-pv-name> lan cocog kapasitas / panyimpenanClassName). Bakal langsung nyambung menyang PV sing ana kanthi kabeh file utuh.

Skenario C: Mulihake negara kluster Kubernetes sing wis dibusak saka etcd snapshot (etcdctl snapshot restore)

Apa yen ana sing mlayu kubectl delete namespace prod utawa helm uninstall lan mbusak 200 manifes Kubernetes, Rahasia, ConfigMaps, lan definisi PV ing manajemen dhewe. kubeadm, k3s, utawa RKE2 pesawat kontrol?

  1. ** Temokake sing paling anyar etcd snapshot** (kanthi standar ing /var/lib/rancher/k3s/server/db/snapshots/, /var/lib/rancher/rke2/server/db/snapshots/, utawa jadwal sampeyan /var/backups/etcd/snapshot.db).
  2. Verifikasi integritas snapshot sadurunge ndemek kluster urip:
    ETCDCTL_API=3 etcdctl snapshot status /path/to/snapshot.db --write-out=table
    
  3. ** Mandheg kube-apiserver lan etcd ing simpul bidang kontrol**:
  4. ing kubeadm (pods statis): Pindhah /etc/kubernetes/manifests/kube-apiserver.yaml lan /etc/kubernetes/manifests/etcd.yaml sementara kanggo /root/manifests_hold/ dadi kubelet mandheg loro kontaner kanthi resik, banjur gawe serep /var/lib/etcd kanggo /var/lib/etcd.bak.
  5. ing k3s: Mlaku sudo systemctl stop k3s lan sudo k3s server --cluster-reset --cluster-reset-restore-path=/path/to/snapshot.
  6. ** Mulihake etcd direktori data on kubeadm**:
    sudo ETCDCTL_API=3 etcdctl snapshot restore /path/to/snapshot.db \
      --data-dir=/var/lib/etcd \
      --name=<control-plane-node-name> \
      --initial-cluster=<node-name>=https://<node-ip>:2380 \
      --initial-advertise-peer-urls=https://<node-ip>:2380
    
  7. Obah etcd.yaml lan kube-apiserver.yaml bali menyang /etc/kubernetes/manifests/. Ing 30 detik, kube-apiserver nyambung maneh menyang dibalèkaké etcd nyimpen lan saben obyek Namespace, Rahasia, ConfigMap, lan PersistentVolume sing wis dibusak.

Fase 5: Hardening Docker lan volume Kubernetes nglawan pambusakan sing ora disengaja

Sawise database produksi sampeyan dipulihake, gunakake papat pagar arsitektur iki supaya salah ketik CLI siji ora bisa mbusak volume langsung maneh:

  • Ganti saben produksi Kubernetes StorageClass saka reclaimPolicy: Delete kanggo reclaimPolicy: Retain: Kanthi gawan, Cloud StorageClasses built-in (gp2, gp3, standard-rwo) standar kanggo reclaimPolicy: Delete, tegese mbusak PVC ngrusak disk awan. Nggawe a StorageClass karo reclaimPolicy: Retain (utawa patch PV sing ana karo kubectl patch pv <pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}') dadi mbusak PVC utawa Namespace ora bakal mbusak volume blok sing ndasari.
  • Aktifake AWS EBS Recycle Bin / Jadwal Snapshot GCP / Velero karo CSI Snapshots: Ngaktifake AWS EBS Recycle Bin nahan volume EBS sing wis dibusak sajrone 7 nganti 30 dina sanajan driver CSI nelpon. DeleteVolume.
  • ** Aja nggunakake docker system prune --volumes ing skrip pembersihan otomatis utawa pelari CI/CD**: Yen wadhah basis data produksi nabrak sedhela (Exited (1)) ing wektu sing tepat proyek cron mlaku docker system prune -a --volumes -f, Docker nganggep volume anonim "ora dienggo" lan mbusak kanthi permanen. Strip --volumes saka kabeh skrip ngresiki.
  • Gunakake mount bind host eksplisit (/srv/prod_data/postgres:/var/lib/postgresql/data) utawa database sing dikelola eksternal kanggo beban kerja produksi stateful: Ora kaya /var/lib/docker/volumes/, host bind-mount direktori njaba /var/lib/docker/ sing ** tau kena ** dening docker compose down -v, docker volume rm, utawa docker system prune --volumes.

Nomer darurat regional lan protokol agensi

Nalika wadhah utawa pambusakan panyimpenan Kubernetes mengaruhi operasi klinis rumah sakit, kiriman keamanan umum, utawa data pribadhi sing diatur, langsung mundhak:

  • Amerika Serikat lan Kanada: Nelpon 911 yen gagal kluster wadhah mateni pemantauan pasien klinis, sistem EHR, utawa kiriman darurat. Laporake prasarana kontainer sing mbebayani utawa kedadeyan cyber menyang CISA (1-844-Say-CISA / cisa.gov/report) utawa Pusat Keamanan Siber Kanada (1-833-CYBER-88).
  • Inggris Raya: Nelpon 999 kanggo ancaman kanggo urip utawa infrastruktur NHS kritis. Ing UK GDPR, karusakan permanen utawa mundhut kasedhiyan data pribadhi tanpa serep sing bisa dibalekake mbutuhake ngevaluasi kabar pelanggaran 72 jam menyang ICO (0303 123 1113).
  • Uni Eropa: Nelpon 112 ing kabeh negara anggota EU. Entitas miturut NIS2 Directive kudu ngirim bebaya awal menyang CSIRT nasional sajrone 24 jam sawise layanan macet, lan mundhut data pribadi permanen nyebabake GDPR Artikel 33 nglaporake sajrone 72 jam.
  • Australia lan Selandia Baru: Nelpon 000 ing Australia utawa 111 ing Selandia Anyar; laporake insiden infrastruktur kritis menyang ACSC (1300 CYBER1).
  • India lan Asia-Pasifik: Nelpon 112 ing India kanggo nanggepi darurat; laporake kedadeyan keamanan awan utawa wadhah menyang CERT-In (incident@cert-in.org.in) ing jendela wajib 6 jam.

Pitakonan sing kerep ditakoni

Apa docker compose down -v mbusak host bind mounts (./data:/var/lib/mysql) utawa mung volume Docker Named lan Anonim?

`docker compose down -v` ONLY mbusak jeneng volume sing diumumake ing `volumes:` bagean saka `docker-compose.yml` lan volume Anonim ditempelake ing wadhah; iku ora tau mbusak file ing mount bind host (kayata `- /opt/mysql_data:/var/lib/mysql` utawa `- ./postgres_data:/var/lib/postgresql/data`). Yen file nulis sampeyan nggunakake path host sing diwiwiti karo `/` utawa `./`, priksa folder kasebut ing host saiki: file sampeyan isih 100 persen utuh.

Ngapa docker system prune --volumes mbusak volume database nalika aku mung mandegake wadhah kanggo 2 menit kanggo nganyari variabel lingkungan?

Ngenteni kasebut kanthi teliti, kanggo mbedakake antarane `docker stop` lan `docker rm`: yen wadhah mung `Stopped / Exited` (katon ing `docker ps -a`), `docker system prune --volumes` nindakake ora mbusak volume amarga wadhah sing mandheg isih ngrujuk. Nanging, yen sampeyan mlayu `docker rm ` (kanggo nggawé ulang wadhah karo gendéra anyar) utawa yen sampeyan liwati `docker system prune -a --volumes` (sing pisanan mbusak kabeh wadhah sing mandheg lan banjur Mbusak kabeh volume sing ora ditempelake maneh ing wadhah apa wae), volume dadi "dangling" milidetik wadhah sing mandheg dicopot lan langsung dibusak.

Apa aku bisa mbalekake volume Docker sing wis dibusak ing Docker Desktop ing Mac (Apple Silicon / Intel) utawa Windows (WSL2)?

Ya, yen sampeyan langsung metu saka Docker Desktop supaya mesin virtual mandheg mlaku `fstrim` marang disk virtual. Ing macOS, Docker nyimpen kabeh volume Linux ing njero `-/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw`; ing Windows WSL2, nang `%LocalAppData%\Docker\wsl\disk\docker_data.vhdx`. Nyalin iku `.raw` utawa `.vhdx` langsung menyang SSD eksternal, pasang mung diwaca ing Linux (nggunakake `losetup -Pr` kanggo `.raw` utawa `qemu-nbd -r` kanggo `.vhdx`), lan mlaku `ext4magic` utawa `PhotoRec` marang virtual `ext4` partisi ing njero.

Apa yen aku mbusak Kubernetes StatefulSet karo kubectl delete statefulset postgres? Apa sing mbusak PVC uga?

Ora. Kanthi gawan ing Kubernetes (kajaba `persistentVolumeClaimRetentionPolicy.whenDeleted: Delete` dikonfigurasi sacara eksplisit ing Kubernetes 1.27+), mlaku `kubectl delete statefulset` mbusak pengontrol StatefulSet lan Pods, ninggalake kabeh `PersistentVolumeClaims` (`data-postgres-0`, `data-postgres-1`) lan data dhasare 100 persen utuh. Cukup aplikasi maneh sampeyan `StatefulSet` YAML manifest kanthi jeneng sing padha lan `volumeClaimTemplates`, lan Pod anyar bakal kanthi otomatis masang maneh menyang sing wis ana `data-postgres-0` PVCs.

Kepiye cara ngakses file ing Docker overlay2 direktori nalika wadhah ora gelem miwiti (CrashLoopBackOff / Exited (1))?

Aja mbusak wadhah tabrakan. Anggere wadhah mandheg isih ana ing `docker ps -a`, lapisan bisa ditulis lan volume mounts rampung utuh ing disk. Sampeyan bisa nyalin file langsung saka wadhah sing mandheg nggunakake `docker cp :/var/lib/postgresql/data /root/rescued_data`, utawa golek host sing tepat `overlay2` dalan kanthi mlaku `docker inspect --format '{{.GraphDriver.Data.UpperDir}}'` lan maca direktori sing langsung minangka `root`.

Apa yen ana wong sing mbusak Kubernetes Namespace (kubectl delete ns prod) lan kabeh PVC wis reclaimPolicy: Delete ing AWS EBS utawa GCP?

Yen driver CSI awan (`ebs.csi.aws.com` utawa `pd.csi.storage.gke.io`) wis kaleksanan `DeleteVolume` amarga `reclaimPolicy` wus `Delete`, langsung mriksa telung jaring safety tingkat awan: (1) mriksa ** AWS Recycle Bin kanggo EBS Volumes ** ing AWS EC2 console (yen aturan retensi aktif ing akun / tags, volume EBS dibusak lenggah ing Recycle Bin lan bisa dibalèkaké ing 1 klik); (2) priksa **AWS Backup / DLM / GCP Snapshot Schedules** kanggo gambar volume otomatis paling anyar; lan (3) priksa manawa database sampeyan lagi streaming WAL/binlogs (`WAL-G`, `pgBackRest`, `Litestream`) menyang ember S3/GCS.


Pandhuan sing gegandhengan


Sumber lan panuntun dhumateng resmi

  • Dokumentasi Resmi Kubernetes (kubernetes.io / Cloud Native Computing Foundation): Obyek Panyimpenan sing dienggo Perlindhungan (kubernetes.io/pvc-protection Finalizers), Reclaiming PersistentVolumes (Retain vs. Delete), lan Operasi etcd Kluster kanggo Kubernetes (etcdctl snapshot restore).
  • Docker Inc. (docs.docker.com): Atur Data ing Docker: Volume, Bind Mounts, overlay2 Arsitektur Driver Storage, lan Referensi Prune.
  • Organisasi Kernel Linux (kernel.org): Ing /proc Sistem berkas (procfs Bukak File Descriptor Recovery liwat /proc/<pid>/fd/) lan Spesifikasi Jurnal Ext4.
  • ext4magic Dokumentasi Proyek (ext4magic.sourceforge.net): Forensic Inode lan Jurnal Transaksi Recovery ing Ext3/Ext4 Filesystems.
  • Institut Standar lan Teknologi Nasional (NIST) (Amerika Serikat): Publikasi Khusus 800-190: Pandhuan Keamanan Kontainer Aplikasi & Pemulihan Insiden.