Volume Produksi Docker utawa Kubernetes Dibusak: Nylametake docker system prune --volumes, overlay2 Inodes, lan etcd Snapshots¶
Nomer darurat beda-beda miturut negara. Kaca iki nggunakake 911, nomer kanggo Amerika Serikat lan Kanada. Akeh negara liya nggunakake 112, 999, utawa nomer liyane. Temokake nomer kanggo negara sampeyan.
mlaku docker compose down -v, docker system prune -a --volumes, docker volume rm, utawa kubectl delete pvc ing tab terminal sing salah ing produksi minangka salah sawijining cara paling cepet kanggo ngilangke volume database PostgreSQL, MySQL, MariaDB, Redis, MongoDB, utawa SQLite langsung. Ora kaya gambar mesin virtual maya sing pindhah menyang tong sampah 7 dina, driver volume lokal Docker (/var/lib/docker/volumes/<name>/_data) lan Kubernetes local-path / hostPath Penyedia langsung ngetokake POSIX standar unlink() lan rmdir() nelpon langsung marang sistem file host Linux sing ndasari (ext4, XFS, utawa Btrfs). Saben tulisan sing kedadeyan ing host Linux kasebut sawise printah rampung bisa nimpa pamblokiran database sing dibebasake ing disk. Yen disk server sampeyan diisi nganti 100 persen (0 bytes free) sadurunge ana wong mlayu docker system prune --volumes ing gupuh, konsultasi disk server 100% lengkap, database WAL log bloat, lan nol-spasi Recovery; kanggo pambusakan file Linux bare-metal njaba wadhah, ndeleng sengaja rm -rf utawa Shift-Delete ing ext4, XFS, APFS, lan NTFS: remount mung diwaca lan ukiran inode lan mesin virtual VMDK, VHDX, lan qcow2 snapshot chain korupsi lan ngluwari hypervisor. Yen pemadaman produksi iki nyebabake perawatan pasien rumah sakit, kiriman darurat, utawa sistem keamanan sarana umum, hubungi nomer darurat lokal sampeyan (911 ing Amerika Serikat lan Kanada, 112 liwat Uni Eropa, 999 ing Inggris) utawa priksa direktori kita nomer darurat.
Apa iki saiki¶
- ** AJA mlayu
docker compose up -d,systemctl restart docker, utawa redeploy pod Kubernetes**: Nggawe maneh wadhah utawa miwiti maneh daemon Docker nulis megabyte anyaroverlay2metadata, log JSON wadah, lan file database kosong sing anyar diinisialisasi (initdb) langsung liwat persisext4utawaXFSpamblokiran ing ngendi volume sing wis dibusak disimpen sawetara detik kepungkur. - ** Priksa ing 5 detik yen proses wadhah (
postgres,mysqld,redis-server,node,python) isih mlaku ing memori (ps auxlansudo lsof +L1)**: Yen ana wong sing mbusak file ing volume sing dipasang utawa mbatalake volume anonim nalika wadhah isih mlaku, kernel Linux tetep 100 persen file database sing ora disambungake urip ing disk anggere ID Proses host wadah kasebut (PID) nduwe deskriptor file sing mbukak (FD). Sampeyan bisa nyalin file kasebut langsung metu saka/proc/<PID>/fd/utawa/proc/<PID>/root/karo mundhut data nul sadurunge wadhah mandheg. - Yen wadhah wis mandheg lan volume wis dibusak (
docker compose down -vutawadocker system prune --volumes), beku disk host langsung nulis: Mungkasi daemon Docker (sudo systemctl stop docker.socket docker.service containerd) supaya wadhah latar mburi lan healthchecks mandheg nulis log menyang/var/lib/docker, banjur remount/var/lib/docker(utawa root/partisi) mung diwaca (sudo mount -o remount,ro /) utawa njupuk potret volume awan tingkat blok langsung (AWS EBS Snapshot,GCP Persistent Disk Snapshot,DigitalOcean Volume Snapshot) sadurunge ndemek apa wae. - Yen sampeyan ana ing Kubernetes (
kubectl delete pvcutawakubectl delete namespace), priksa PersistentVolume (PV)ReclaimPolicy(kubectl get pv) langsung: - Yen ing
PVwispersistentVolumeReclaimPolicy: Retain, data sampeyan ora dibusak: ingPVmung dipindhah menyangReleasedstatus lan disk awan sampeyan (EBS,Azure Disk,Ceph RBD,NFS) 100 persen utuh. - Malah yen
kubectl delete pvcwis macet ingTerminating, Kubernetes Finalizers (kubernetes.io/pvc-protection) mblokir driver CSI saka mbusak panyimpenan ndasari anggere sembarang Pod isih referensi PVC sing. Aja mbusak Pod nganti sampeyan nyalin data metu saka Pod sing mlaku. - Gambar partisi blok sing ndasari (
dd/ddrescueliwat SSH) menyang disk eksternal utawa volume awan sing kapisah sadurunge mlakuext4magic,extundelete, utawaPhotoRec: Aja mbukak pemahat file sing nulis file output menyang partisi disk sing padha sing sampeyan pulih/var/lib/docker/volumes. - Kanggo kahanan kluster Kubernetes sing rusak utawa ora sengaja (
kubectl delete crd/namespace), mandegkube-apiserverlan mulihake paling anyaretcdsnapshot (etcdctl snapshot restore): Yen klaster negara ingetcdwis dibusak nalika volume panyimpenan maya ndasari isih ana, mulihakeetcdwaras kabehPV,PVC,Secret, lanConfigMapobyek bebarengan.
Danger
** Aja mbusak kubernetes.io/pvc-protection utawa kubernetes.io/pv-protection finalizer karo kubectl patch ... -p '{"metadata":{"finalizers":null}}' nalika PVC macet ing Terminating**: Finalizer iku mung siji-sijine sing njaga driver CSI awan sampeyan (aws-ebs-csi-driver, pd.csi.storage.gke.io, Longhorn, Rook-Ceph) saka nerbitake hardware DeleteVolume Telpon API kanggo numpes disk produksi. Nalika PVC macet ing Terminating, Pod isih mlaku lan volume isih dipasang: mbukak kubectl exec utawa kubectl cp (utawa pg_dump / mysqldump) langsung kanggo ngluwari 100 persen database sampeyan sadurunge Pod mungkasi.
Wit keputusan: volume Docker sing wis dibusak, overlay2 lapisan, utawa Kubernetes PVC triage¶
Gunakake diagram alur iki kanggo milih dalan pemulihan nol-rugi sing paling cepet adhedhasar apa proses wadhah isih urip, volume didhukung dening disk lokal utawa panyimpenan CSI awan, lan sistem file sing dadi host. /var/lib/docker:
flowchart TD
Start["Data Produksi Ilang Sawise Docker utawa Kubernetes Mbusak Command"] --> WhichEnv{"Platform & Komando Endi sing Nimbulake Kerugian?"}
WhichEnv -->|"Docker: Container Still Running (File Unlinked Inside Volume/Container)"| ProcRescue["Rescue Open Unlinked File Descriptors liwat sudo lsof +L1 lan /proc/PID/fd/N utawa /proc/PID/root/"]
WhichEnv -->|"Docker: Kontainer Dihentikan & Volume Dibusak (nyipta mudhun -v / prune --volume)"| StopDocker["Langsung Run: sudo systemctl stop docker.socket docker containerd"]
WhichEnv -->|"Kubernetes: kubectl mbusak pvc / pv / statefulset / namespace"| K8sCheck{"Apa Pod Isih Mlaku UTAWA PVC Macet ing Terminasi?"}
K8sCheck -->|"Ya: Pod Still Running (Finalizer Blocking CSI Delete)"| K8sExec["AJA Mbusak Pod -> Jalanake kubectl exec / pg_dump / tar liwat stdout menyang Disk Lokal Langsung"]
K8sCheck -->|"Ora: Pod Gone -> Priksa kubectl njaluk pv"| ReclaimCheck{"Apa Kebijakan Reclaim PV?"}
ReclaimCheck -->|"Retain (Status PV = Dirilis)"| ClearClaimRef["100% Data Utuh -> Sunting PV kanggo Mbusak spec.claimRef.uid -> Ikatan maneh menyang PVC Anyar"]
ReclaimCheck -->|"Busak (Cloud EBS / PD / Longhorn / Local-Path)"| CloudOrLocal{"Volume CSI Cloud utawa Disk Host Lokal?"}
CloudOrLocal -->|"Cloud CSI (AWS EBS / GCP / Azure / Ceph)"| CloudSnap["Priksa Keranjang Daur Ulang Cloud Console / Snapshot EBS Otomatis / Ceph RBD Trash (rbd trash ls)"]
CloudOrLocal -->|"Local Node Disk (local-path / hostPath)"| StopDocker
StopDocker --> RemountRO["Remount Host Partition Read-Only (mount -o remount,ro) UTAWA Gambar Blokir Stream liwat SSH"]
RemountRO --> HostFS{"Apa Filesystem Hosts /var/lib/docker?"}
HostFS -->|"ext4"| Ext4Rec["Gunakake debugfs / ext4magic Journal Inode Transaction Rollback (-cap wektu)"]
HostFS -->|"XFS / Btrfs"| CarveRec["Gunakake Btrfs Roots / xfs_irecover / Custom Signature Carving kanggo SQLite, Postgres Heap, utawa InnoDB .ibd"]
Fase 1: Kepiye Docker lan Kubernetes mbusak volume ing disk¶
Sadurunge nglakokake prentah pemulihan, ngerti ngendi data sampeyan urip ing host Linux lan apa sing didemek printah mbusak:
| Jinis Panyimpenan & Busak Command | Host Filesystem Path ing Linux Node | Apa sing Ditindakake dening Mbusak Command | Cara Recovery Langsung |
|---|---|---|---|
Named Docker Volume (docker volume rm utawa docker compose down -v) |
/var/lib/docker/volumes/<vol_name>/_data |
Mbusak link file ing njero _data/ lan mbusak entri direktori saka /var/lib/docker/volumes/metadata.db (boltdb). |
Mungkasi Docker langsung, remount /var/lib/docker mung diwaca, lan mbalekake inode sing ora disambung liwat ext4magic / debugfs utawa ukiran teken. |
Volume Docker Anonim (docker run -v /var/lib/mysql + docker rm -v utawa docker system prune --volumes) |
/var/lib/docker/volumes/<64-char-hex-id>/_data |
docker system prune --volumes mbusak saben volume anonim lan jenenge sing saiki ora dirujuk dening paling ora siji wadhah sing wis ana. |
Mungkasi Docker langsung; yen sampeyan duwe ** liyane ** mandegake wadhah sing volume ora * pruned, mriksa isih /var/lib/docker/volumes/*/_data folder pisanan sadurunge ngukir pamblokiran unallocated. |
** Lapisan Wadhah sing Bisa Ditulis (docker rm <container> nalika data ditulis ing njaba volume)** |
/var/lib/docker/overlay2/<layer-id>/diff/ |
Mbukak link ndhuwur wadhah diff/ direktori nang overlay2 driver panyimpenan. |
Mungkasi Docker langsung lan waras wit direktori sing ora disambungake saka file ext4 jurnal utawa XFS pamblokiran. |
** Docker Desktop ing macOS utawa Windows (docker system prune --volumes)** |
Ing siji gambar disk mesin virtual: -/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw (macOS) utawa %LocalAppData%\Docker\wsl\disk\docker_data.vhdx (Windows WSL2). |
Dikirim unlink() nang Linux VM, lan yen discard/TRIM aktif ing Docker.raw / docker_data.vhdx, punched bolongan jarang ing file inang. |
Mungkasi Docker Desktop sajrone 5 detik, salin Docker.raw utawa docker_data.vhdx menyang drive external, lan mbukak ext4magic / PhotoRec marang gambar disk virtual. |
** Kubernetes PVC karo Delete Kabijakan Reclaim ing Ceph RBD (Rook-Ceph) utawa Longhorn** |
Kolam Ceph OSD (rbd) utawa /var/lib/longhorn/replicas/. |
Yen csi-rbdplugin wis trash diaktifake (rbd trash), gambar RBD dipindhah menyang tong sampah Ceph; digunakake pamblokiran sing unmapped. |
Mlayu rbd trash ls --pool <pool> ing njero rook-ceph-tools pod lan mulihake karo rbd trash restore. |
Fase 2: Nylametake file database sing mbukak saka wadhah sing isih aktif (/proc/<pid>/fd/)¶
Enteni: wadhahe isih mlaku saiki?
Asring, insinyur nglakokake skrip migrasi aplikasi, tugas cron pembersihan, utawa manual rm -rf /var/lib/postgresql/data/* utawa rm /app/data/production.sqlite3 nang wadhah mlaku, utawa mbusak folder bind-mount host ndasari /opt/app/data nalika wadhah dhewe (postgres, mysqld, mariadbd, redis-server, mongod, gunicorn, node) isih munggah lan mlaku.
Ing Linux, file ora dibusak saka disk nalika rm utawa unlink() diarani. Blok disk lan inode tetep ** 100 persen dialokasikan lan bisa diwaca ** nganti proses pungkasan nyekel deskriptor file mbukak (FD) menyang inode kasebut nutup utawa metu.
Langkah 1: Temokake ID Proses host (PID) saka proses wadhah¶
Ing simpul host Linux (ora ana ing wadhah), goleki host PID saka database utawa proses aplikasi:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
mysqld 18492 999 14u REG 259,1 482344960 0 1849201 /var/lib/mysql/app_prod/users.ibd (deleted)
mysqld 18492 999 15u REG 259,1 125829120 0 1849202 /var/lib/mysql/app_prod/orders.ibd (deleted)
mysqld 18492 999 16u REG 259,1 79691776 0 1849190 /var/lib/mysql/ibdata1 (deleted)
python3 22104 1000 4u REG 259,1 89128960 0 3102944 /data/production.sqlite3 (deleted)
NLINK: kuwi 0, lan jeneng berkas dipungkasi nganggo (deleted). File wis rampung ilang saka ls /var/lib/docker/volumes/..., nanging ** saben bait (482 MB ing users.ibd, 89 MB ing production.sqlite3) isih urip ing njero /proc/<PID>/fd/<FD>**.
Langkah 2: Nyalin file database sing ora disambungake langsung /proc/<PID>/fd/¶
Aja ** aja ** mandheg utawa miwiti maneh wadhah. Nalika proses isih mlaku, nyalin deskriptor file mentah langsung menyang folder sing aman ing host sampeyan:
sudo mkdir -p /root/proc_emergency_rescue
# Example 1: Rescuing an unlinked SQLite3 database (PID 22104, FD 4)
sudo cp /proc/22104/fd/4 /root/proc_emergency_rescue/production.sqlite3
# Check if PID 22104 also holds the -wal file open (e.g. FD 5) and copy it too:
sudo cp /proc/22104/fd/5 /root/proc_emergency_rescue/production.sqlite3-wal
# Example 2: Automatically rescuing ALL open (deleted) MySQL/Postgres files for PID 18492
for fd in /proc/18492/fd/*; do
target=$(sudo readlink "$fd" 2>/dev/null)
if echo "$target" | grep -q "(deleted)"; then
fd_num=$(basename "$fd")
clean_name=$(echo "$target" | sed 's/ (deleted)//; s/\//_/g')
echo "Rescuing FD $fd_num -> $clean_name"
sudo cp "$fd" "/root/proc_emergency_rescue/${fd_num}${clean_name}"
fi
done
Langkah 3: Luwih apik: Jalanake dump logis langsung (pg_dump / mysqldump / .backup) sadurunge wadhah nutup FDs¶
Yen postgres utawa mysqld isih mlaku karo file data sawijining mbukak ing memori lan /proc/<PID>/fd/, coba mbukak dump logis langsung liwat soket unix utawa sambungan TCP sing ana dhisik:
# If the database engine can still read its open file descriptors:
docker exec <container_name> pg_dumpall -U postgres > /root/proc_emergency_rescue/live_dump.sql
# Or for MySQL:
docker exec <container_name> mysqldump -u root -p --all-databases --single-transaction > /root/proc_emergency_rescue/live_mysql.sql
.1, .2 file bagean antarane pangolahan buruh dinamis, supaya pg_dump bisa gagal ing tabel sing ora disimpen; nggabungake query SQL urip karo nyalin saben mbukak /proc/<pid>/fd/* ing kabeh postgres PID backend anak nyimpen data tumpukan tabel maksimal.)*
Fase 3: Mbalekake volume Docker sing wis dibusak ext4 utawa XFS sawise wadhah mandheg¶
Apa yen sampeyan mlayu docker compose down -v utawa docker system prune --volumes, dadi wadhah ** wis mandheg ** lan kabeh /proc/<pid>/fd/ deskriptor ditutup? Saiki sampeyan lagi lomba nglawan latar mburi disk nulis ing host Linux.
Langkah 1: Beku sistem file host Linux sajrone 10 detik¶
- Sampun Docker, containerd, kubelet, lan cron sanalika dadi ora ana wadah, healthcheck, utawa log rotator nulis byte liyane menyang
/var/lib/docker: - ** Priksa piranti pemblokiran lan sistem file sing ditahan
/var/lib/docker**: - ** Pasang maneh partisi Mung Waca (utawa stream gambar partisi mentah liwat SSH menyang mesin liyane)**:
- Yen
/var/lib/dockerana ing partisi khusus utawa volume LVM dhewe (contone/dev/nvme0n1p2utawa/dev/mapper/vg0-docker), pasang maneh mung bisa diwaca: - Yen
/var/lib/dockerana ing oyod/pemisahan (/), coba meksa partisi root mung diwaca nggunakake pemicu Magic SysRq kernel (echo u | sudo tee /proc/sysrq-trigger) utawasudo mount -o remount,ro /. - Yen sampeyan nganggo VPS awan (AWS EC2, GCP, DigitalOcean, Hetzner, Linode): Langsung menyang konsol web awan banjur klik Gawe Snapshot ing volume blok server (utawa stream piranti blok mentah liwat SSH menyang server liya:
sudo dd if=/dev/nvme0n1p1 bs=4M status=progress | gzip -1 | ssh user@backup-server "cat > /mnt/storage/docker_host_raw.img.gz").
Langkah 2: Priksa manawa data sampeyan isih ana ing volume LAIN sing ora dipotong utawa overlay2 lapisan¶
Sadurunge ngukir pamblokiran disk sing ora disedhiyakake, priksa manawa database sampeyan ora disimpen ing bind mount utawa volume sing beda-beda sing isih urip:
# Search the entire /var/lib/docker tree read-only for any surviving database files:
sudo find /var/lib/docker -name "PG_VERSION" -o -name "ibdata1" -o -name "*.sqlite*" -o -name "*.db" -o -name "dump.rdb" -o -name "WiredTiger" 2>/dev/null
docker volume rm app_db_data, mung kanggo nemokake karo find /var/lib/docker sing docker-compose.yml wis bener digawe projectname_app_db_data (kang isih 100 persen utuh ing disk) nalika app_db_data minangka volume lawas sing ora digunakake wiwit nem sasi kepungkur.
Langkah 3: Mbalekake direktori volume Docker sing ora disambungake ext4 nggunakake ing ext4 Jurnal (ext4magic lan debugfs)¶
kapan docker volume rm mbusak /var/lib/docker/volumes/<vol_name>/_data ing an ext4 sistem berkas, ext4 nol metu pointer blok langsung (i_block / oyod wit ombone) ing tabel inode urip. Nanging, ext4 jurnal (node <8>, biasane 128 MB kanggo 1024 MB buffer ring bunder) isih nyimpen salinan sajarah tabel inode lan blok direktori sing tepat saka sadurunge docker compose down -v dieksekusi.
+-------------------------------------------------------------------------+
| HOW ext4magic RECOVERS A DELETED DOCKER VOLUME FROM THE EXT4 JOURNAL |
| |
| Live Inode Table on Disk (After docker volume rm): |
| Inode #918402 (users.ibd): [Size = 0, Extent Pointers = ZEROED] |
| |
| Ext4 Journal Ring Buffer (Inode <8> on Disk): |
| Transaction #41902 (10 mins ago): |
| Inode #918402 (users.ibd): [Size = 482MB, Extents = Blocks 9000..9999]|
| |
| -> ext4magic reads the old Inode Copy from Journal Transaction #41902 |
| and extracts Blocks 9000..9999 directly to your rescue drive. |
+-------------------------------------------------------------------------+
Mlayu ext4magic marang partisi mung diwaca (utawa .img salinan snapshot ing server kapindho), nemtokake timestamp (-a) saka 5 menit sadurunge sampeyan mlayu docker compose down -v utawa docker system prune --volumes:
# Install ext4magic on your rescue machine (never on the affected partition)
sudo apt-get install -y ext4magic e2fsprogs
# Dump the ext4 journal from the frozen partition / image to a file in RAM (/dev/shm)
sudo debugfs -R "dump <8> /dev/shm/ext4_journal.copy" /dev/nvme0n1p1
# Recover all files deleted under var/lib/docker/volumes after (Current Time - 15 minutes)
sudo mkdir -p /mnt/external_rescue/docker_recovered
sudo ext4magic /dev/nvme0n1p1 \
-j /dev/shm/ext4_journal.copy \
-f var/lib/docker/volumes \
-a $(date -d "-15 minutes" +%s) \
-r -d /mnt/external_rescue/docker_recovered
-j /dev/shm/ext4_journal.copy: Waca snapshot jurnal sing disimpen ing RAM sanalika sawise mandheg Docker.
- -f var/lib/docker/volumes: Nargetake wit direktori volume Docker sing tepat (cathetan: ngilangi titik utama / ing ext4magic jalur relatif).
- -a $(date -d "-15 minutes" +%s): Ngandika ext4magic kanggo mbangun maneh wit direktori nggunakake salinan inode sing bener 15 menit kepungkur (tengen sadurunge mbusak ora sengaja).
- -r: Mbalekake direktori lengkap sing ora ana blok sing ditimpa; yen -r kantun file sebagian ditindhes, mbukak pass kapindho ngganti -r karo -m (ukir sihir + mode hibrida jurnal).
Langkah 4: Ukir file mentah SQLite, PostgreSQL, utawa MySQL InnoDB nalika jurnal wis dibungkus (XFS utawa ext4)¶
Warning
**Kenapa XFS ora ndhukung ext4magic lan mbutuhake langsung unmounting mung diwaca **: Boten kados ext4, sing nulis salinan inode 256-byte lengkap menyang jurnal, XFS nyathet delta maksud metadata lan kanthi agresif nggunakake maneh wit B inode sing dibebasake (agi). Yen /var/lib/docker lungguh ing an XFS partisi (standar ing RHEL, Rocky Linux, AlmaLinux, lan Amazon Linux), unmounting /var/lib/docker utawa remounting / mung diwaca ing sawetara detik minangka langkah sing paling penting sadurunge ngukir.
Yen host nggunakake XFS utawa yen ext4 jurnal sing wis dibungkus ing pointer inode, gunakake tanda tangan lan ukiran struktur ing gambar disk mentah (docker_host_raw.img):
- ** Kanggo Database SQLite3 (
.sqlite,.db- contone, Vaultwarden, Nextcloud, Home Assistant, Gitea, Grafana)**: Saben database SQLite3 diwiwiti kanthi header ASCII 16-byte sing ora jelas ing bait 0 saka kaca 4 kilobyte pisanan:53 51 4c 69 74 65 20 66 6f 72 6d 61 74 20 33 00("SQLite format 3\0"), lan nyimpen ukuran kaca database sing tepat (Bytes 16..17, biasane4096) lan ukuran database total ing kaca (Bytes 28..31, integer 32-bit big-endian) ing jero header sing padha. - Mlayu
PhotoReckaro[File Opt]->[x] sqliteaktif, utawa mindai partisi langsung karogrep -ab "SQLite format 3"kanggo nemokake offset byte pas file SQLite dibusak lan extract karodd:Sawise dijupuk, mbukak# Find the exact byte offsets of every "SQLite format 3" header on the partition: sudo grep -ab "SQLite format 3" /dev/nvme0n1p1sqlite3 carved.db ".recover" | sqlite3 repaired.dbkanggo mbangun maneh saben kaca godhong B-wit utuh.
Tip
** Carane ngetung persis dd ukuran byte file SQLite3 sing ora disambungake saka header**: Sawise grep -ab "SQLite format 3" menehi offset byte wiwitan OFFSET saka database SQLite dibusak, maca bita 16..17 (Uint16 Ukuran Halaman Big-endian, contone, 0x1000 = 4096 bita) lan bita 28..31 (Uint32 Big-endian Page Count, contone, 25000 kaca) nggunakake hexdump -C -s OFFSET -n 32. Multiplying Page Size x Page Count menehi dawa bita pas kanggo pass kanggo dd (iflag=skip_bytes,count_bytes) nalika file kasebut cedhak karo disk.
- ** Kanggo MySQL / MariaDB InnoDB (
.ibdfile tabel)**: Saben InnoDB.ibdTabel kasusun saka urutan16 KB(16,384byte) kaca sing ngemotFIL_PAGE_INDEX(0x45BF) header lan tabel kang unikspace_id. Gunakake Percona / TwinDBundrop-for-innodb(stream_parser+c_parser) toolkit kanggo mindhai/dev/nvme0n1p1kanggo kabeh16 KBKaca indeks InnoDB lan mbucal saben baris langsung menyang file TSV/SQL kanggo diimpor maneh. - ** Kanggo PostgreSQL Heap Files (
$PGDATA/base/<db_oid>/<relfilenode>)**: PostgreSQL nyimpen saben tabel minangka contiguous8 KB(8,192byte) kaca tumpukan ngemotPageHeaderData(pd_lower,pd_upper,pd_special). Gunakakepg_filedumputawapglazarus/pdukanggo mindai wilayah disk sing ora dialokasikan kanggo8 KBkaca sing cocog karo tata letak kolom tabel sampeyan (HeapTupleHeaderData) lan mbangun maneh baris.
Fase 4: Kubernetes PVC, PV, lan etcd Snapshot Recovery Darurat¶
Ing Kubernetes (EKS, GKE, AKS, k3s, RKE2, kubeadm, OpenShift), kanthi ora sengaja mbusak PersistentVolumeClaim (kubectl delete pvc), StatefulSet, utawa kabeh Namespace (kubectl delete ns prod) nderek arsitektur sing beda banget tinimbang Docker mandiri.
Skenario A: PVC nuduhake Terminating (Penyelamat Lifesaver)¶
mlaku:
Yen ingSTATUS kolom ngandika Terminating, ambegan lega: data sampeyan durung dibusak.
- Kenging punapa? Saben Kubernetes PVC digawe Wiwit Kubernetes 1.9 otomatis kalebu finalizers: [kubernetes.io/pvc-protection].
- Sanalika Pod isih dipasang ing PVC kasebut, pengontrol Kubernetes ora gelem ngrampungake pambusakan PVC lan ** ora tau nelpon driver CSI. DeleteVolume RPC**.
- Tindakan Langsung: Aja Aja mbusak utawa miwiti maneh Pod, lan aja NOT patch metu finalizer. Langsung nyalin kabeh data metu saka Pod sing mlaku menyang mesin lokal utawa ember S3:
# Stream a compressed tar backup or pg_dump out of the still-running pod immediately:
kubectl exec -n <namespace> <pod-name> -- pg_dumpall -U postgres > ./rescued_k8s_db.sql
# Or tar an entire mounted volume directory out over stdout:
kubectl exec -n <namespace> <pod-name> -- tar czf - /var/lib/postgresql/data > ./rescued_pvc_files.tar.gz
Skenario B: PVC dibusak, lan kubectl get pv nuduhake PersistentVolume ing Released negara (ReclaimPolicy: Retain)¶
mlaku:
Yen PersistentVolume sampeyan (pv-prod-postgres) isih kadhaptar karo STATUS: Released (amarga StorageClass digunakake reclaimPolicy: Retain), ** 100 persen data sampeyan isih ora disentuh ing disk dhasar **. Nanging, Kubernetes nolak supaya PVC anyar ikatan menyang a Released PV amarga PV isih ngandhut a spec.claimRef pointing menyang lawas dibusak PVC kang UID.
Carane nggawe a Released PV Available maneh ing 10 detik tanpa mundhut data:
1. Owahi obyek PersistentVolume lan mbusak spec.claimRef blok:
kubectl get pv maneh: status langsung ganti saka Releasedkanggo Available.
3. Gawe maneh sampeyan PersistentVolumeClaim (spesifikasi volumeName: <your-pv-name> lan cocog kapasitas / panyimpenanClassName). Bakal langsung nyambung menyang PV sing ana kanthi kabeh file utuh.
Skenario C: Mulihake negara kluster Kubernetes sing wis dibusak saka etcd snapshot (etcdctl snapshot restore)¶
Apa yen ana sing mlayu kubectl delete namespace prod utawa helm uninstall lan mbusak 200 manifes Kubernetes, Rahasia, ConfigMaps, lan definisi PV ing manajemen dhewe. kubeadm, k3s, utawa RKE2 pesawat kontrol?
- ** Temokake sing paling anyar
etcdsnapshot** (kanthi standar ing/var/lib/rancher/k3s/server/db/snapshots/,/var/lib/rancher/rke2/server/db/snapshots/, utawa jadwal sampeyan/var/backups/etcd/snapshot.db). - Verifikasi integritas snapshot sadurunge ndemek kluster urip:
- ** Mandheg
kube-apiserverlanetcding simpul bidang kontrol**: - ing
kubeadm(pods statis): Pindhah/etc/kubernetes/manifests/kube-apiserver.yamllan/etc/kubernetes/manifests/etcd.yamlsementara kanggo/root/manifests_hold/dadikubeletmandheg loro kontaner kanthi resik, banjur gawe serep/var/lib/etcdkanggo/var/lib/etcd.bak. - ing
k3s: Mlakusudo systemctl stop k3slansudo k3s server --cluster-reset --cluster-reset-restore-path=/path/to/snapshot. - ** Mulihake
etcddirektori data onkubeadm**: - Obah
etcd.yamllankube-apiserver.yamlbali menyang/etc/kubernetes/manifests/. Ing 30 detik,kube-apiservernyambung maneh menyang dibalèkakéetcdnyimpen lan saben obyek Namespace, Rahasia, ConfigMap, lan PersistentVolume sing wis dibusak.
Fase 5: Hardening Docker lan volume Kubernetes nglawan pambusakan sing ora disengaja¶
Sawise database produksi sampeyan dipulihake, gunakake papat pagar arsitektur iki supaya salah ketik CLI siji ora bisa mbusak volume langsung maneh:
- Ganti saben produksi Kubernetes
StorageClasssakareclaimPolicy: DeletekanggoreclaimPolicy: Retain: Kanthi gawan, Cloud StorageClasses built-in (gp2,gp3,standard-rwo) standar kanggoreclaimPolicy: Delete, tegese mbusak PVC ngrusak disk awan. Nggawe aStorageClasskaroreclaimPolicy: Retain(utawa patch PV sing ana karokubectl patch pv <pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}') dadi mbusak PVC utawa Namespace ora bakal mbusak volume blok sing ndasari. - Aktifake AWS EBS Recycle Bin / Jadwal Snapshot GCP / Velero karo CSI Snapshots: Ngaktifake AWS EBS Recycle Bin nahan volume EBS sing wis dibusak sajrone 7 nganti 30 dina sanajan driver CSI nelpon.
DeleteVolume. - ** Aja nggunakake
docker system prune --volumesing skrip pembersihan otomatis utawa pelari CI/CD**: Yen wadhah basis data produksi nabrak sedhela (Exited (1)) ing wektu sing tepat proyek cron mlakudocker system prune -a --volumes -f, Docker nganggep volume anonim "ora dienggo" lan mbusak kanthi permanen. Strip--volumessaka kabeh skrip ngresiki. - Gunakake mount bind host eksplisit (
/srv/prod_data/postgres:/var/lib/postgresql/data) utawa database sing dikelola eksternal kanggo beban kerja produksi stateful: Ora kaya/var/lib/docker/volumes/, host bind-mount direktori njaba/var/lib/docker/sing ** tau kena ** deningdocker compose down -v,docker volume rm, utawadocker system prune --volumes.
Nomer darurat regional lan protokol agensi¶
Nalika wadhah utawa pambusakan panyimpenan Kubernetes mengaruhi operasi klinis rumah sakit, kiriman keamanan umum, utawa data pribadhi sing diatur, langsung mundhak:
- Amerika Serikat lan Kanada: Nelpon
911yen gagal kluster wadhah mateni pemantauan pasien klinis, sistem EHR, utawa kiriman darurat. Laporake prasarana kontainer sing mbebayani utawa kedadeyan cyber menyang CISA (1-844-Say-CISA/cisa.gov/report) utawa Pusat Keamanan Siber Kanada (1-833-CYBER-88). - Inggris Raya: Nelpon
999kanggo ancaman kanggo urip utawa infrastruktur NHS kritis. Ing UK GDPR, karusakan permanen utawa mundhut kasedhiyan data pribadhi tanpa serep sing bisa dibalekake mbutuhake ngevaluasi kabar pelanggaran 72 jam menyang ICO (0303 123 1113). - Uni Eropa: Nelpon
112ing kabeh negara anggota EU. Entitas miturut NIS2 Directive kudu ngirim bebaya awal menyang CSIRT nasional sajrone 24 jam sawise layanan macet, lan mundhut data pribadi permanen nyebabake GDPR Artikel 33 nglaporake sajrone 72 jam. - Australia lan Selandia Baru: Nelpon
000ing Australia utawa111ing Selandia Anyar; laporake insiden infrastruktur kritis menyang ACSC (1300 CYBER1). - India lan Asia-Pasifik: Nelpon
112ing India kanggo nanggepi darurat; laporake kedadeyan keamanan awan utawa wadhah menyang CERT-In (incident@cert-in.org.in) ing jendela wajib 6 jam.
Pitakonan sing kerep ditakoni¶
Apa docker compose down -v mbusak host bind mounts (./data:/var/lib/mysql) utawa mung volume Docker Named lan Anonim?¶
`docker compose down -v` ONLY mbusak jeneng volume sing diumumake ing `volumes:` bagean saka `docker-compose.yml` lan volume Anonim ditempelake ing wadhah; iku ora tau mbusak file ing mount bind host (kayata `- /opt/mysql_data:/var/lib/mysql` utawa `- ./postgres_data:/var/lib/postgresql/data`). Yen file nulis sampeyan nggunakake path host sing diwiwiti karo `/` utawa `./`, priksa folder kasebut ing host saiki: file sampeyan isih 100 persen utuh.
Ngapa docker system prune --volumes mbusak volume database nalika aku mung mandegake wadhah kanggo 2 menit kanggo nganyari variabel lingkungan?¶
Ngenteni kasebut kanthi teliti, kanggo mbedakake antarane `docker stop` lan `docker rm`: yen wadhah mung `Stopped / Exited` (katon ing `docker ps -a`), `docker system prune --volumes` nindakake ora mbusak volume amarga wadhah sing mandheg isih ngrujuk. Nanging, yen sampeyan mlayu `docker rm
Apa aku bisa mbalekake volume Docker sing wis dibusak ing Docker Desktop ing Mac (Apple Silicon / Intel) utawa Windows (WSL2)?¶
Ya, yen sampeyan langsung metu saka Docker Desktop supaya mesin virtual mandheg mlaku `fstrim` marang disk virtual. Ing macOS, Docker nyimpen kabeh volume Linux ing njero `-/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw`; ing Windows WSL2, nang `%LocalAppData%\Docker\wsl\disk\docker_data.vhdx`. Nyalin iku `.raw` utawa `.vhdx` langsung menyang SSD eksternal, pasang mung diwaca ing Linux (nggunakake `losetup -Pr` kanggo `.raw` utawa `qemu-nbd -r` kanggo `.vhdx`), lan mlaku `ext4magic` utawa `PhotoRec` marang virtual `ext4` partisi ing njero.
Apa yen aku mbusak Kubernetes StatefulSet karo kubectl delete statefulset postgres? Apa sing mbusak PVC uga?¶
Ora. Kanthi gawan ing Kubernetes (kajaba `persistentVolumeClaimRetentionPolicy.whenDeleted: Delete` dikonfigurasi sacara eksplisit ing Kubernetes 1.27+), mlaku `kubectl delete statefulset` mbusak pengontrol StatefulSet lan Pods, ninggalake kabeh `PersistentVolumeClaims` (`data-postgres-0`, `data-postgres-1`) lan data dhasare 100 persen utuh. Cukup aplikasi maneh sampeyan `StatefulSet` YAML manifest kanthi jeneng sing padha lan `volumeClaimTemplates`, lan Pod anyar bakal kanthi otomatis masang maneh menyang sing wis ana `data-postgres-0` PVCs.
Kepiye cara ngakses file ing Docker overlay2 direktori nalika wadhah ora gelem miwiti (CrashLoopBackOff / Exited (1))?¶
Aja mbusak wadhah tabrakan. Anggere wadhah mandheg isih ana ing `docker ps -a`, lapisan bisa ditulis lan volume mounts rampung utuh ing disk. Sampeyan bisa nyalin file langsung saka wadhah sing mandheg nggunakake `docker cp
Apa yen ana wong sing mbusak Kubernetes Namespace (kubectl delete ns prod) lan kabeh PVC wis reclaimPolicy: Delete ing AWS EBS utawa GCP?¶
Yen driver CSI awan (`ebs.csi.aws.com` utawa `pd.csi.storage.gke.io`) wis kaleksanan `DeleteVolume` amarga `reclaimPolicy` wus `Delete`, langsung mriksa telung jaring safety tingkat awan: (1) mriksa ** AWS Recycle Bin kanggo EBS Volumes ** ing AWS EC2 console (yen aturan retensi aktif ing akun / tags, volume EBS dibusak lenggah ing Recycle Bin lan bisa dibalèkaké ing 1 klik); (2) priksa **AWS Backup / DLM / GCP Snapshot Schedules** kanggo gambar volume otomatis paling anyar; lan (3) priksa manawa database sampeyan lagi streaming WAL/binlogs (`WAL-G`, `pgBackRest`, `Litestream`) menyang ember S3/GCS.
Pandhuan sing gegandhengan¶
- Server disk 100% kebak (0 bita gratis): database WAL log bloat, B-tree rusak, lan nol-space Recovery
rm -rfutawaShift+Deletesing ora disengaja ing Linux, macOS, utawa Windows: remount mung maca lan ukiran inode langsung- Mesin virtual VMDK, VHDX, lan korupsi rantai snapshot qcow2 lan ngluwari hypervisor
- Komit Git kunci AWS, file
.env, utawa kunci SSH pribadi menyang repo GitHub umum - Krisis hosting web, migrasi, lan pemulihan serep
- Serep komputer, strategi 3-2-1, lan pemulihan bencana
- Nomer darurat miturut negara
Sumber lan panuntun dhumateng resmi¶
- Dokumentasi Resmi Kubernetes (
kubernetes.io/ Cloud Native Computing Foundation): Obyek Panyimpenan sing dienggo Perlindhungan (kubernetes.io/pvc-protectionFinalizers), Reclaiming PersistentVolumes (Retainvs.Delete), lan OperasietcdKluster kanggo Kubernetes (etcdctl snapshot restore). - Docker Inc. (
docs.docker.com): Atur Data ing Docker: Volume, Bind Mounts,overlay2Arsitektur Driver Storage, lan Referensi Prune. - Organisasi Kernel Linux (
kernel.org): Ing/procSistem berkas (procfsBukak File Descriptor Recovery liwat/proc/<pid>/fd/) lan Spesifikasi Jurnal Ext4. ext4magicDokumentasi Proyek (ext4magic.sourceforge.net): Forensic Inode lan Jurnal Transaksi Recovery ing Ext3/Ext4 Filesystems.- Institut Standar lan Teknologi Nasional (
NIST) (Amerika Serikat): Publikasi Khusus 800-190: Pandhuan Keamanan Kontainer Aplikasi & Pemulihan Insiden.