Docker kapena Kubernetes Production Volume Yachotsedwa: Kupulumutsa docker system prune --volumes, overlay2 Inodes, ndi etcd Zithunzi¶
Manambala angozi amasiyana malinga ndi dziko. Tsambali limagwiritsa ntchito 911, nambala ya United States ndi Canada. Mayiko ena ambiri amagwiritsa ntchito 112, 999, kapena nambala ina. Pezani nambala ya dziko lanu.
Kuthamanga docker compose down -v, docker system prune -a --volumes, docker volume rm, kapena kubectl delete pvc pa tabu yolakwika yopangira ndi imodzi mwa njira zofulumira kwambiri zofufutira zamoyo za PostgreSQL, MySQL, MariaDB, Redis, MongoDB, kapena SQLite database. Mosiyana ndi zithunzi zamakina amtambo omwe amapita ku bin yamasiku 7, woyendetsa voliyumu waku Docker (/var/lib/docker/volumes/<name>/_data) ndi Kubernetes local-path / hostPath operekera zakudya nthawi yomweyo amapereka POSIX yokhazikika unlink() ndi rmdir() imayimba mwachindunji motsutsana ndi maziko a Linux host filesystem (ext4, XFS, kapena Btrfs). Kulemba kulikonse komwe kumachitika pa Linux host host lamulo likatha limatha kutsitsa midadada yomasulidwa pa disk. Ngati seva yanu itadzaza mpaka 100 peresenti (0 bytes free) munthu asanathawe docker system prune --volumes ndi mantha, funsani seva disk 100% yodzaza, database ya WAL log bloat, ndi kuchira kwa zero; pakuchotsa mafayilo opanda zitsulo a Linux kunja kwa zotengera, onani mwangozi rm -rf kapena Shift-Delete pa ext4, XFS, APFS, ndi NTFS: kuwerengera kokha ndi kujambula kwa inode ndi makina pafupifupi VMDK, VHDX, ndi qcow2 chithunzithunzi unyolo ziphuphu ndi hypervisor kupulumutsa. Ngati kutha kwa kupangaku kumakhudza chisamaliro cha odwala kuchipatala, kutumiza mwadzidzidzi, kapena njira zotetezera anthu, imbani nambala yanu yadzidzidzi yakuderalo (911 ku United States ndi Canada, 112 ku European Union, 999 ku United Kingdom) kapena onani chikwatu chathu cha manambala angozi.
Chitani izi tsopano¶
- OSATI kuthamanga
docker compose up -d,systemctl restart docker, kapena tumizaninso Kubernetes pod: Kubwezeretsanso chidebe kapena kuyambitsanso daemon ya Docker kumalemba ma megabytes atsopano.overlay2metadata, zipika za JSON, ndi mafayilo opanda kanthu omwe angoyamba kumene (initdb) molunjika pa zenizeniext4kapenaXFSmidadada pomwe voliyumu yanu yochotsedwa idakhala masekondi apitawo. - Chongani masekondi 5 ngati chidebe ndondomeko (
postgres,mysqld,redis-server,node,python) ikugwirabe ntchito mu kukumbukira (ps auxndisudo lsof +L1): Ngati wina wachotsa mafayilo mkati mwa voliyumu yokwezedwa kapena kulumikiza voliyumu yosadziwika pomwe chidebecho chikugwirabe ntchito, Linux kernel imasunga 100 peresenti ya mafayilo osalumikizidwa pa disk malinga ngati ID ya process ya chotengerayo (PID) imakhala ndi chofotokozera chotseguka (FD). Mutha kukopera mafayilowo mwachindunji kuchokera/proc/<PID>/fd/kapena/proc/<PID>/root/ndi ziro deta kutayika chidebecho chisanayambe. - ** Ngati chidebecho chidayimitsidwa kale ndipo voliyumu idachotsedwa (
docker compose down -vkapenadocker system prune --volumes), amaundana disk host imalemba nthawi yomweyo**: Imitsa daemon ya Docker (sudo systemctl stop docker.socket docker.service containerd) kotero zotengera zakumbuyo ndi ma healthchecks amasiya kulemba zipika/var/lib/docker, ndiye mwina onjezerani/var/lib/docker(kapena mizu/kugawa) kuwerenga kokha (sudo mount -o remount,ro /) kapena tengani chithunzithunzi chambiri chamtambo (AWS EBS Snapshot,GCP Persistent Disk Snapshot,DigitalOcean Volume Snapshot) musanagwire kalikonse. - Ngati muli pa Kubernetes (
kubectl delete pvckapenakubectl delete namespace), onani PersistentVolume (PV)ReclaimPolicy(kubectl get pv) nthawi yomweyo: - Ngati ndi
PVanalipersistentVolumeReclaimPolicy: Retain, deta yanu sachotsedwa: thePVanangosamukira kuReleasedmawonekedwe ndi diski yanu yamtambo (EBS,Azure Disk,Ceph RBD,NFS) ndi 100 peresenti yokha. - Ngakhale
kubectl delete pvcali muTerminating, Kubernetes Omaliza (kubernetes.io/pvc-protection) letsani dalaivala wa CSI kuti achotse zosungira zomwe zili pansi bola ngati Pod iliyonse imatchulanso PVC. Osachotsa Pod mpaka mutakopera deta kuchokera pa Pod yomwe ikuyenda. - Fananizani gawo loyambira (
dd/ddrescuepa SSH) kupita ku disk yakunja kapena voliyumu yosiyana yamtambo musanayendetseext4magic,extundelete, kapenaPhotoRec: Osathamangitsa osema mafayilo omwe amalemba mafayilo otulutsa pagawo lomwelo la disk lomwe mukuchira/var/lib/docker/volumes. - Zowonongeka kapena kufufutidwa mwangozi gulu la gulu la Kubernetes (
kubectl delete crd/namespace), kusiyakube-apiserverndi kubwezeretsa zatsopanoetcdchithunzithunzi (etcdctl snapshot restore): Ngati masango ali mkatietcdidafufutidwa pomwe ma voliyumu osungira mitambo akadalipo, kubwezeretsedwaetcdamachira onsePV,PVC,Secret,ndiConfigMapzinthu nthawi imodzi.
Danger
Osachotsa kubernetes.io/pvc-protection kapena kubernetes.io/pv-protection finalizer ndi kubectl patch ... -p '{"metadata":{"finalizers":null}}' pamene PVC inakanidwa Terminating: Chomaliza ndicho chinthu chokhacho chomwe chikusunga dalaivala wanu wamtambo wa CSI (aws-ebs-csi-driver, pd.csi.storage.gke.io, Longhorn, Rook-Ceph) kuchokera popereka hardware DeleteVolume Kuyimba kwa API kuti muwononge disk yanu yopanga. Pomwe PVC imakhazikika Terminating, Pod ikugwirabe ntchito ndipo voliyumu idakalipo: thamanga kubectl exec kapena kubectl cp (kapena pg_dump / mysqldump) nthawi yomweyo kupulumutsa 100 peresenti ya database yanu Pod isanathe.
Mtengo wosankha: voliyumu ya Docker yachotsedwa, overlay2 wosanjikiza, kapena Kubernetes PVC triage¶
Gwiritsani ntchito tchatichi kuti musankhe njira yochira yotaya ziro yothamanga kwambiri potengera ngati chidebecho chikadalipobe, kaya voliyumuyo idathandizidwa ndi diski yakumaloko kapena kusungirako kwamtambo CSI, komanso mafayilo omwe amasunga. /var/lib/docker:
flowchart TD
Start["Zambiri Zopanga Zikusowa Pambuyo pa Docker kapena Kubernetes Chotsani Lamulo"] --> WhichEnv{"Ndi Platform & Command iti yomwe idayambitsa Kutayika?"}
WhichEnv -->|"Docker: Container Ikugwirabe Ntchito (Mafayilo Osalumikizidwa Mkati Mwa Volume / Chidebe)"| ProcRescue["Pulumutsani Mafotokozedwe A Fayilo Osalumikizidwa kudzera pa sudo lsof +L1 ndi /proc/PID/fd/N kapena /proc/PID/root/"]
WhichEnv -->|"Docker: Chidebe Choyimitsidwa & Voliyumu Yachotsedwa (lemba pansi -v / prune --volumes)"| StopDocker["Thamangani Nthawi yomweyo: sudo systemctl siyani docker.socket docker yosungidwa"]
WhichEnv -->|"Kubernetes: kubectl chotsani pvc / pv / statefulset / namespace"| K8sCheck{"Kodi Pod Ikugwirabe Ntchito Kapena PVC Imayima Pomaliza?"}
K8sCheck -->|"Inde: Pod Ikuyendabe (Finalizer Kutsekereza CSI Chotsani)"| K8sExec["OSAFUTA Pod -> Thamangani kubectl exec / pg_dump / tar pa stdout kupita ku Local Disk Nthawi yomweyo"]
K8sCheck -->|"Ayi: Pod Gone -> Onani kubectl kupeza pv"| ReclaimCheck{"Kodi PV ReclaimPolicy inali chiyani?"}
ReclaimCheck -->|"Sungani (Mkhalidwe wa PV = Watulutsidwa)"| ClearClaimRef["100% Data Intact -> Sinthani PV kuti Chotsani spec.claimRef.uid -> Kumanganso ku PVC Yatsopano"]
ReclaimCheck -->|"Chotsani (Mtambo EBS / PD / Longhorn / Local-Path)"| CloudOrLocal{"Cloud CSI Volume kapena Local Host Disk?"}
CloudOrLocal -->|"Cloud CSI (AWS EBS / GCP / Azure / Ceph)"| CloudSnap["Onani Cloud Console Recycle Bin / Automated EBS Snapshots / Ceph RBD Zinyalala (rbd zinyalala ls)"]
CloudOrLocal -->|"Local Node Disk (njira yakomweko / hostPath)"| StopDocker
StopDocker --> RemountRO["Remount Host Partition Read-Only (mount -o remount,ro) KAPENA Stream Block Image pa SSH"]
RemountRO --> HostFS{"Kodi Filesystem Hosts /var/lib/docker ndi chiyani?"}
HostFS -->|"ext4"| Ext4Rec["Gwiritsani ntchito debugfs / ext4magic Journal Inode Transaction Rollback (-chidindo chanthawi)"]
HostFS -->|"XFS / Btrfs"| CarveRec["Gwiritsani Ntchito Mizu ya Btrfs / xfs_irecover / Siginecha Yamakonda Kujambula kwa SQLite, Postgres Heap, kapena InnoDB .ibd"]
Gawo 1: Momwe Docker ndi Kubernetes amachotseratu ma voliyumu pa disk¶
Musanagwiritse ntchito lamulo lililonse lobwezeretsa, mvetsetsani komwe deta yanu imakhala pa Linux host host ndi zomwe lamulo lochotsa lidakhudza:
| Mtundu Wosungira & Chotsani Lamulo | Host Filesystem Path pa Linux Node | Zomwe Delete Command Idachitadi | Mwamsanga Kuchira Njira |
|---|---|---|---|
** Wotchedwa Docker Volume (docker volume rm kapena docker compose down -v)** |
/var/lib/docker/volumes/<vol_name>/_data |
Chotsani mafayilo mkati _data/ ndikuchotsa zolemba zachikwatu kuchokera /var/lib/docker/volumes/metadata.db (boltdb). |
Imani Docker nthawi yomweyo, onjezerani /var/lib/docker werengani-pokha, ndikubwezeretsanso ma innode osalumikizidwa kudzera ext4magic / debugfs kapena kujambula siginecha. |
Volume yosadziwika ya Docker (docker run -v /var/lib/mysql + docker rm -v kapena docker system prune --volumes) |
/var/lib/docker/volumes/<64-char-hex-id>/_data |
docker system prune --volumes imachotsa voliyumu iliyonse yosadziwika ndi dzina lomwe silinatchulidwe ndi chidebe chimodzi chomwe chilipo. |
Imitsani Docker nthawi yomweyo; mukadakhala ndi **zina ** zoyimitsa zotengera zomwe mavoliyumu ake *sanadulidwe, fufuzani zotsalira /var/lib/docker/volumes/*/_data zikwatu poyamba asanasema midadada yosagawika. |
** Chigawo Cholembedwa (docker rm <container> pamene deta inalembedwa kunja kwa voliyumu)** |
/var/lib/docker/overlay2/<layer-id>/diff/ |
Chotsani kumtunda kwa chidebecho diff/ directory mkati mwa overlay2 dalaivala yosungirako. |
Imitsani Docker nthawi yomweyo ndikubwezeretsanso mtengo wosalumikizidwa kuchokera ku fayilo ya ext4 magazini kapena XFS midadada. |
** Docker Desktop pa macOS kapena Windows (docker system prune --volumes)** |
Mkati mwa chithunzi cha disk cha makina amodzi: -/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw (macOS) kapena %LocalAppData%\Docker\wsl\disk\docker_data.vhdx (Windows WSL2). |
Watumizidwa unlink() mkati mwa Linux VM, ndipo ngati discard/TRIM ikugwira ntchito Docker.raw / docker_data.vhdx, anabowola mabowo ochepa mufayilo yosungira. |
Siyani Docker Desktop mkati mwa masekondi 5, koperani Docker.raw kapena docker_data.vhdx kupita kugalimoto yakunja, ndikuthamanga ext4magic / PhotoRec motsutsana ndi chithunzi cha disk. |
**Kubernetes PVC yokhala ndi Delete ReclaimPolicy pa Ceph RBD (Rook-Ceph) kapena Longhorn ** |
Ceph OSD dziwe (rbd) kapena /var/lib/longhorn/replicas/. |
Ngati csi-rbdplugin ali trash loledwa (rbd trash), chithunzi cha RBD chimasunthidwa ku bin ya zinyalala ya Ceph; apo ayi midadada ilibe mapu. |
Thamangani rbd trash ls --pool <pool> mkati mwa rook-ceph-tools pod ndi kubwezeretsa ndi rbd trash restore. |
Gawo 2: Kupulumutsa mafayilo otseguka a database kuchokera pachidebe chomwe chikuyendabe (/proc/<pid>/fd/)¶
Dikirani: kontena ikugwirabe ntchito pompano?
Nthawi zambiri, mainjiniya amayendetsa script yosamukira, ntchito yoyeretsa, kapena buku rm -rf /var/lib/postgresql/data/* kapena rm /app/data/production.sqlite3 mkati mwa chidebe chothamanga, kapena kuchotsa chikwatu chomwe chili pansi pa host bind-mount /opt/app/data pamene chotengeracho (postgres, mysqld, mariadbd, redis-server, mongod, gunicorn, node)idakalipobe.
Mu Linux, fayilo sichichotsedwa pa disk pamene rm kapena unlink() amatchedwa. Ma disk ake amatchinga ndi inode amakhalabe ** 100 peresenti yogawidwa ndi yowerengeka ** mpaka ndondomeko yomaliza yokhala ndi chofotokozera chotseguka (FD) ku inodeyo amatseka kapena kutuluka.
Gawo 1: Pezani ID process process (PID) ya ndondomeko ya chidebe¶
Pa Linux host node (osati mkati mwa chidebe), pezani wolandirayo PID za database yanu kapena njira yofunsira:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
mysqld 18492 999 14u REG 259,1 482344960 0 1849201 /var/lib/mysql/app_prod/users.ibd (deleted)
mysqld 18492 999 15u REG 259,1 125829120 0 1849202 /var/lib/mysql/app_prod/orders.ibd (deleted)
mysqld 18492 999 16u REG 259,1 79691776 0 1849190 /var/lib/mysql/ibdata1 (deleted)
python3 22104 1000 4u REG 259,1 89128960 0 3102944 /data/production.sqlite3 (deleted)
NLINK: akuti 0, ndipo dzina lafayilo limatha ndi (deleted). Fayilo yachoka kwathunthu ls /var/lib/docker/volumes/..., komabe bayiti iliyonse (482 MB mu users.ibd, 89 MB mu production.sqlite3) akadali ndi moyo mkati /proc/<PID>/fd/<FD>.
Khwerero 2: Koperani mafayilo a database osalumikizidwa mwachindunji /proc/<PID>/fd/¶
Osa** kuyimitsa kapena kuyambitsanso chidebecho. Pamene ndondomekoyi ikugwirabe ntchito, lembani zofotokozera za fayilo yaiwisi mwachindunji ku foda yotetezeka pa omwe akukulandirani:
sudo mkdir -p /root/proc_emergency_rescue
# Example 1: Rescuing an unlinked SQLite3 database (PID 22104, FD 4)
sudo cp /proc/22104/fd/4 /root/proc_emergency_rescue/production.sqlite3
# Check if PID 22104 also holds the -wal file open (e.g. FD 5) and copy it too:
sudo cp /proc/22104/fd/5 /root/proc_emergency_rescue/production.sqlite3-wal
# Example 2: Automatically rescuing ALL open (deleted) MySQL/Postgres files for PID 18492
for fd in /proc/18492/fd/*; do
target=$(sudo readlink "$fd" 2>/dev/null)
if echo "$target" | grep -q "(deleted)"; then
fd_num=$(basename "$fd")
clean_name=$(echo "$target" | sed 's/ (deleted)//; s/\//_/g')
echo "Rescuing FD $fd_num -> $clean_name"
sudo cp "$fd" "/root/proc_emergency_rescue/${fd_num}${clean_name}"
fi
done
Khwerero 3: Zabwinonso: Yambitsani dambo lomveka bwino (pg_dump / mysqldump / .backup) chidebecho chisanatseke ma FD¶
Ngati postgres kapena mysqld ikugwirabe ntchito ndi mafayilo ake a data otsegulidwa mu kukumbukira ndi /proc/<PID>/fd/, yesani kuthamangitsa kutayira komwe kulipo pa unix socket kapena kulumikizana kwa TCP kaye:
# If the database engine can still read its open file descriptors:
docker exec <container_name> pg_dumpall -U postgres > /root/proc_emergency_rescue/live_dump.sql
# Or for MySQL:
docker exec <container_name> mysqldump -u root -p --all-databases --single-transaction > /root/proc_emergency_rescue/live_mysql.sql
.1, .2 magawo amafayilo pamachitidwe a ogwira ntchito mwachangu, kotero pg_dump akhoza kulephera pa matebulo osasungidwa; kuphatikiza funso lamoyo la SQL ndikukopera kutsegulidwa kulikonse /proc/<pid>/fd/* kudutsa *onse ** postgres ma PID am'mbuyo a ana amasunga mulu wa data womwe ungatheke.)
Gawo 3: Kubwezeretsanso voliyumu ya Docker yochotsedwa ext4 kapena XFS chidebecho chitayima¶
Bwanji ngati mutathamanga docker compose down -v kapena docker system prune --volumes, kotero kuti chidebecho chayimitsidwa kale ** ndi zonse /proc/<pid>/fd/ zofotokozera zatsekedwa? Tsopano muli pa mpikisano wotsutsana ndi disk yakumbuyo ikulemba pa Linux host host.
Khwerero 1: Mangani ma fayilo a Linux mkati mwa masekondi 10¶
- **Imitsani Docker, yosungidwa, kubelet, ndi cron nthawi yomweyo ** kotero palibe chidebe, thanzi, kapena chipika chozungulira chomwe chimalemba china chake
/var/lib/docker: - ** Onani chida chotchinga ndi fayilo yomwe ili nayo
/var/lib/docker**: - Kwezaninso magawowa Kuwerenga-Only (kapena tsitsani chithunzi chogawa pa SSH kumakina ena):
- Ngati
/var/lib/dockerili pagawo lake lodzipatulira kapena voliyumu ya LVM (mwachitsanzo/dev/nvme0n1p2kapena/dev/mapper/vg0-docker), onjezerani kuti muwerenge-pokhapokha: - Ngati
/var/lib/dockerali pa muzu/kugawa (/), yesani kukakamiza magawo a mizu kuti awerenge-pokha pogwiritsa ntchito kernel Magic SysRq trigger (echo u | sudo tee /proc/sysrq-trigger) kapenasudo mount -o remount,ro /. - Ngati muli pamtambo VPS (AWS EC2, GCP, DigitalOcean, Hetzner, Linode): Nthawi yomweyo pitani kumtambo wanu wapaintaneti ndikudina Pangani Chithunzithunzi pa block voliyumu ya seva (kapena tsitsani chipangizo chopangira SSH kupita ku seva yachiwiri:
sudo dd if=/dev/nvme0n1p1 bs=4M status=progress | gzip -1 | ssh user@backup-server "cat > /mnt/storage/docker_host_raw.img.gz").
Khwerero 2: Onani ngati deta yanu ikadali mu voliyumu INA yosadulidwe kapena overlay2 wosanjikiza¶
Musanajambule midadada yosagawika, onetsetsani kuti nkhokwe yanu sinasungidwe pamtengo kapena voliyumu yodziwika bwino yomwe idapulumuka:
# Search the entire /var/lib/docker tree read-only for any surviving database files:
sudo find /var/lib/docker -name "PG_VERSION" -o -name "ibdata1" -o -name "*.sqlite*" -o -name "*.db" -o -name "dump.rdb" -o -name "WiredTiger" 2>/dev/null
docker volume rm app_db_data, kungozindikira ndi find /var/lib/docker kuti docker-compose.yml anali atalengadi projectname_app_db_data (yomwe ikadali 100 peresenti yokha pa disk) pamene app_db_data inali voliyumu yakale yosagwiritsidwa ntchito kuyambira miyezi isanu ndi umodzi yapitayo.
Khwerero 3: Kubwezeretsanso maulalo osalumikizidwa a Docker pa ext4 pogwiritsa ntchito ext4 Magazini (ext4magic ndi debugfs)¶
Pamene docker volume rm zochotsa /var/lib/docker/volumes/<vol_name>/_data pa ext4 file system, ext4 ziro pazilolezo za Direct block (i_block / muzu wamtengo) patebulo lokhalamo. Komabe, ext4 magazini (inode <8>, kawirikawiri 128 MB ku 1024 MB ring ring buffer) ikadali ndi zolemba zakale zamatebulo enieni a innode ndi zikwatu zachikwatu kuyambira kale. docker compose down -v anaphedwa.
+-------------------------------------------------------------------------+
| HOW ext4magic RECOVERS A DELETED DOCKER VOLUME FROM THE EXT4 JOURNAL |
| |
| Live Inode Table on Disk (After docker volume rm): |
| Inode #918402 (users.ibd): [Size = 0, Extent Pointers = ZEROED] |
| |
| Ext4 Journal Ring Buffer (Inode <8> on Disk): |
| Transaction #41902 (10 mins ago): |
| Inode #918402 (users.ibd): [Size = 482MB, Extents = Blocks 9000..9999]|
| |
| -> ext4magic reads the old Inode Copy from Journal Transaction #41902 |
| and extracts Blocks 9000..9999 directly to your rescue drive. |
+-------------------------------------------------------------------------+
Thamangani ext4magic motsutsana ndi magawo anu owerengera okha (kapena .img chithunzithunzi pa seva yachiwiri), kufotokoza nthawi (-a) kuchokera **5 mphindi musanathamangire ** mudathamanga docker compose down -v kapena docker system prune --volumes:
# Install ext4magic on your rescue machine (never on the affected partition)
sudo apt-get install -y ext4magic e2fsprogs
# Dump the ext4 journal from the frozen partition / image to a file in RAM (/dev/shm)
sudo debugfs -R "dump <8> /dev/shm/ext4_journal.copy" /dev/nvme0n1p1
# Recover all files deleted under var/lib/docker/volumes after (Current Time - 15 minutes)
sudo mkdir -p /mnt/external_rescue/docker_recovered
sudo ext4magic /dev/nvme0n1p1 \
-j /dev/shm/ext4_journal.copy \
-f var/lib/docker/volumes \
-a $(date -d "-15 minutes" +%s) \
-r -d /mnt/external_rescue/docker_recovered
-j /dev/shm/ext4_journal.copy: Amawerenga chithunzithunzi cha magazini omwe mudasunga mu RAM mutangoyimitsa Docker.
- -f var/lib/docker/volumes: Imatsata mtengo weniweni wa chikwatu cha Docker (chidziwitso: chotsani kutsogolera / mu ext4magic njira zosiyanasiyana).
- -a $(date -d "-15 minutes" +%s): Amanena ext4magic kuti amangenso chikwatu cholembera pogwiritsa ntchito makope a inode ovomerezeka mphindi 15 zapitazo (kungochotsa mwangozi).
- -r: Imabwezeretsanso maulalo athunthu pomwe palibe midadada idalembedwa; ngati -r amaphonya owona overwritten pang'ono, kuthamanga yachiwiri pass m'malo -r ndi -m (zojambula zamatsenga + hybrid mode).
Khwerero 4: Kujambula mafayilo a SQLite, PostgreSQL, kapena MySQL InnoDB yaiwisi pamene magazini yatsekedwa (XFS kapena ext4)¶
Warning
**Chifukwa XFS sichikuthandizira ext4magic ndipo imafuna kutsika pompopompo powerenga-pokha **: Mosiyana ext4, yomwe imalemba zolemba zonse za 256-byte inode mumagazini yake, XFS logs metadata intent deltas ndikugwiritsanso ntchito mwaukali mitengo ya B-inode yomasulidwa (agi). Ngati /var/lib/docker akukhala pa XFS magawo (osakhazikika pa RHEL, Rocky Linux, AlmaLinux, ndi Amazon Linux), akutsika /var/lib/docker kapena kukweranso / kuwerenga-pokha masekondi ndi sitepe imodzi yofunika kwambiri pamaso kusema.
Ngati wolandirayo akugwiritsa ntchito XFS kapena ngati ext4 magazini yomwe yakulungidwa kale pazolozera za inode, gwiritsani ntchito siginecha ndi mawonekedwe ojambulidwa motsutsana ndi chithunzi cha disk yaiwisi (docker_host_raw.img):
- Kwa SQLite3 Databases (
.sqlite,.db- mwachitsanzo, Vaultwarden, Nextcloud, Wothandizira Pakhomo, Gitea, Grafana): Dongosolo lililonse la SQLite3 limayamba ndi mutu wodziwika bwino wa 16-byte ASCII pa byte 0 patsamba lake loyamba la 4 kilobyte:53 51 4c 69 74 65 20 66 6f 72 6d 61 74 20 33 00("SQLite format 3\0"), ndikusunga kukula kwake kwamasamba (Bytes 16..17, kawirikawiri4096) ndi kukula kwa database yonse m'masamba (Bytes 28..31, big-endian 32-bit integer) mkati mwa mutu womwewo. - Thamangani
PhotoRecndi[File Opt]->[x] sqliteyambitsa, kapena jambulani gawolo mwachindunji ndigrep -ab "SQLite format 3"kuti mupeze zenizeni zenizeni za fayilo yanu ya SQLite yochotsedwa ndikuyichotsa nayodd:Akachotsa, thamangani# Find the exact byte offsets of every "SQLite format 3" header on the partition: sudo grep -ab "SQLite format 3" /dev/nvme0n1p1sqlite3 carved.db ".recover" | sqlite3 repaired.dbkuti amangenso tsamba lililonse latsamba la B-mtengo.
Tip
** Momwe mungawerengere zenizeni dd kukula kwa fayilo ya SQLite3 yosalumikizidwa kuchokera pamutu wake **: Kamodzi grep -ab "SQLite format 3" zimakupatsani mwayi woyambira OFFSET pa database yanu yochotsedwa ya SQLite, werengani ma byte 16..17 (uint16 big-endian Tsamba Kukula, mwachitsanzo, 0x1000 = 4096 mabayiti) ndi ma byte 28..31 (Uint32 big-endian Page Count, mwachitsanzo, 25000 masamba) pogwiritsa ntchito hexdump -C -s OFFSET -n 32. Kuchulukitsa Page Size x Page Count zimakupatsirani kutalika kwa baiti kuti mudutseko dd (iflag=skip_bytes,count_bytes) pamene fayiloyo ikugwirizana ndi disk.
- Kwa MySQL / MariaDB InnoDB (
.ibdtable owona): InnoDB iliyonse.ibdtable imakhala ndi sequential16 KB(16,384byte) masamba omwe ali ndiFIL_PAGE_INDEX(0x45BF) mutu ndi tebulo lapaderaspace_id. Gwiritsani ntchito Percona / TwinDBundrop-for-innodb(stream_parser+c_parser) zida kuti sikani/dev/nvme0n1p1kwa onse16 KBMasamba a index ya InnoDB ndikutaya mzere uliwonse mwachindunji mu mafayilo a TSV/SQL kuti mulowetsenso. - Kwa Mafayilo a PostgreSQL Mululu (
$PGDATA/base/<db_oid>/<relfilenode>): PostgreSQL imasunga tebulo lililonse ngati lolumikizana8 KB(8,192byte) mulu wa masamba omwe aliPageHeaderData(pd_lower,pd_upper,pd_special). Gwiritsanipg_filedumpkapenapglazarus/pdukusanthula madera osagawika a disk8 KBmasamba ofananira ndi magawo a tebulo lanu (HeapTupleHeaderData) ndikumanganso mizere.
Gawo 4: Kubernetes PVC, PV, ndi etcd Chithunzithunzi cha Emergency Recovery¶
Ku Kubernetes (EKS, GKE, AKS, k3s, RKE2, kubeadm, OpenShift), kuchotsa mwangozi PersistentVolumeClaim (kubectl delete pvc), StatefulSet, kapena Namespace yonse (kubectl delete ns prod) amatsata zomangamanga zosiyana kwambiri kuposa Docker yoyima.
Chitsanzo A: Mawonekedwe a PVC Terminating (The Finalizer Lifesaver)¶
Thamangani:
Ngati ndiSTATUS ndime imati Terminating, pumani m'malo: chidziwitso chanu SICHINAchotsedwe panobe.
- Chifukwa chiyani? Kubernetes PVC iliyonse yopangidwa Popeza Kubernetes 1.9 imaphatikizansopo finalizers: [kubernetes.io/pvc-protection].
- Malingana ngati Pod ikukwerabe PVC, wolamulira wa Kubernetes amakana kumaliza kuchotsedwa kwa PVC ndipo sayitana oyendetsa CSI. DeleteVolume RPC.
- Zochita Mwamsanga: Chitani OSATI kufufuta kapena kuyambitsanso Pod, ndipo chitani OSATI tulutsani chomaliza. Koperani nthawi yomweyo zonse zomwe zili mu Pod yomwe ikuyenda kumakina anu am'deralo kapena ndowa ya S3:
# Stream a compressed tar backup or pg_dump out of the still-running pod immediately:
kubectl exec -n <namespace> <pod-name> -- pg_dumpall -U postgres > ./rescued_k8s_db.sql
# Or tar an entire mounted volume directory out over stdout:
kubectl exec -n <namespace> <pod-name> -- tar czf - /var/lib/postgresql/data > ./rescued_pvc_files.tar.gz
Chitsanzo B: The PVC inachotsedwa, ndi kubectl get pv ikuwonetsa PersistentVolume mu Released dziko (ReclaimPolicy: Retain)¶
Thamangani:
Ngati PersistentVolume yanu (pv-prod-postgres) adalembedwabe ndi STATUS: Released (chifukwa StorageClass yake imagwiritsidwa ntchito reclaimPolicy: Retain), **100 peresenti ya deta yanu idakali yosakhudzidwa pa disk yomwe ili pansi **. Komabe, Kubernetes amakana kulola PVC yatsopano kumangiriza a Released PV chifukwa PV ikadali ndi a spec.claimRef kuloza ku PVC wakale zichotsedwa UID.
Momwe mungapangire a Released PV Available kachiwiri mumasekondi a 10 ndikutayika kwa ziro:
1. Sinthani chinthu cha PersistentVolume ndikuchotsa spec.claimRef chipika:
kubectl get pv kachiwiri: mawonekedwe amasintha nthawi yomweyo kuchokera ku Released ku Available.
3. Panganinso yanu PersistentVolumeClaim ( kufotokoza volumeName: <your-pv-name> ndi kufananiza mphamvu/storageClassName). Idzamangiriza ku PV yomwe ilipo nthawi yomweyo ndi mafayilo onse osasinthika.
Chitsanzo C: Kubwezeretsanso gulu lopukuta la Kubernetes kuchokera ku etcd chithunzithunzi (etcdctl snapshot restore)¶
Bwanji ngati wina athamanga kubectl delete namespace prod kapena helm uninstall ndikufafaniza 200 Kubernetes kuwonetsera, Zinsinsi, ConfigMaps, ndi matanthauzo a PV pa kudziyendetsa nokha. kubeadm, k3s, kapena RKE2 ndege yoyendetsa?
- **Pezani zomwe zaposachedwa
etcdchithunzi ** (mwachisawawa mu/var/lib/rancher/k3s/server/db/snapshots/,/var/lib/rancher/rke2/server/db/snapshots/, kapena zomwe mwapanga/var/backups/etcd/snapshot.db). - **Tsimikizirani kukhulupirika kwa chithunzithunzi musanagwire gulu lamoyo **:
- Imani
kube-apiserverndietcdpa node (ma) ndege: - Pa
kubeadm(static pods): Sunthani/etc/kubernetes/manifests/kube-apiserver.yamlndi/etc/kubernetes/manifests/etcd.yamlkwakanthawi kuti/root/manifests_hold/chonchokubeletimayimitsa zotengera zonse ziwiri mwaukhondo, kenako ndikubwezeretsanso/var/lib/etcdku/var/lib/etcd.bak. - Pa
k3s: Thamanganisudo systemctl stop k3sndisudo k3s server --cluster-reset --cluster-reset-restore-path=/path/to/snapshot. - ** Bwezerani
etcddata directory pakubeadm**: - Sunthani
etcd.yamlndikube-apiserver.yamlkubwerera ku/etc/kubernetes/manifests/. Mkati masekondi 30,kube-apiserverzimalumikizananso ndi zobwezeretsedwaetcdsitolo ndi chilichonse chochotsedwa Namespace, Chinsinsi, ConfigMap, ndi PersistentVolume chinthu zimawonekeranso.
Gawo 5: Kulimbitsa Docker ndi Kubernetes voliyumu motsutsana ndi kuchotsedwa mwangozi¶
Nawonso database yanu yopangira ikabwezeretsedwa, khazikitsani zida zinayi zomangira kuti CLI typo imodzi isachotsenso voliyumu yomwe ilipo:
- Sinthani Kubernetes iliyonse yopanga
StorageClasskuchokerareclaimPolicy: DeletekureclaimPolicy: Retain: Mwachikhazikitso, zomangidwa mumtambo StorageClasses (gp2,gp3,standard-rwo) kusakhazikika kureclaimPolicy: Delete, kutanthauza kuti kuchotsa PVC kumawononga diski yamtambo. Pangani aStorageClassndireclaimPolicy: Retain(kapena sungani ma PV omwe alipo ndikubectl patch pv <pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}') kotero kuchotsa PVC kapena Namespace sikumachotsa voliyumu yomwe ili pansi. - ** Yambitsani AWS EBS Recycle Bin / GCP Snapshot Schedules / Velero yokhala ndi CSI Snapshots **: Kuthandizira AWS EBS Recycle Bin kumasunga voliyumu iliyonse ya EBS yomwe yachotsedwa kwa masiku 7 mpaka 30 ngakhale woyendetsa CSI atayimba foni.
DeleteVolume. - Musagwiritse ntchito
docker system prune --volumesm'malemba oyeretsa okha kapena othamanga a CI/CD: Ngati chotengera chosungiramo zinthu zopangira chiwonongeka mwachidule (Exited (1)) panthawi yomwe ntchito ya cron ikugwira ntchitodocker system prune -a --volumes -f, Docker amawona mavoliyumu ake osadziwika "osagwiritsidwa ntchito" ndikuchotsa kwamuyaya. Kuvula--volumeskuchokera pazolemba zonse zoyeretsa. - **Gwiritsani ntchito zida zomangirira zowonekera (
/srv/prod_data/postgres:/var/lib/postgresql/data) kapena nkhokwe zakunja zoyendetsedwa ndi ntchito zambiri zopanga **: Mosiyana/var/lib/docker/volumes/, mayendedwe a bind-mount omwe ali kunja/var/lib/docker/sanakhudzidwe konse ndidocker compose down -v,docker volume rm, kapenadocker system prune --volumes.
Nambala zangozi zachigawo ndi ndondomeko za bungwe¶
Pamene chidebe kapena kuchotsedwa kwa Kubernetes kukhudza ntchito zachipatala, kutumiza chitetezo cha anthu, kapena zambiri zaumwini, zimakula nthawi yomweyo:
- United States ndi Canada: Imbani
911ngati kulephera kwa chidebe kumalepheretsa kuwunika kwa odwala, machitidwe a EHR, kapena kutumiza mwadzidzidzi. Nenani za kufufutidwa kwa zida zoyipa kapena zochitika za cyber ku CISA (1-844-Say-CISA/cisa.gov/report) kapena Canadian Center for Cyber Security (1-833-CYBER-88). - United Kingdom: Imbani
999zowopseza moyo kapena zofunikira za NHS. Pansi pa UK GDPR, chiwonongeko chamuyaya kapena kutayika kwa kupezeka kwa zidziwitso zanu popanda zosunga zobwezeretsera kumafuna kuwunika chidziwitso chakuphwanya kwa maola 72 ku ICO (0303 123 1113). - European Union: Imbani
112kudutsa mayiko onse a EU. Mabungwe omwe ali pansi pa NIS2 Directive akuyenera kupereka chenjezo ku CSIRT ya dziko lawo pasanathe maola 24 kuchokera pamene ntchito yazimitsidwa kwambiri, ndipo kutayika kwa data kwamuyaya kumayambitsa GDPR Article 33 lipoti mkati mwa maola 72. - Australia ndi New Zealand: Imbani
000ku Australia kapena111ku New Zealand; nenani zochitika zovuta za zomangamanga ku ACSC (1300 CYBER1). - India ndi Asia-Pacific: Imbani
112ku India kuti ayankhe mwadzidzidzi; nenani zachitetezo chamtambo kapena chotengera ku CERT-In (incident@cert-in.org.in) mkati mwa zenera lovomerezeka la maola 6.
Mafunso omwe amafunsidwa pafupipafupi¶
Amatero docker compose down -v Chotsani kukwera kwa host bind (./data:/var/lib/mysql) kapena ma volume a Named and Anonymous Docker okha?¶
`docker compose down -v` ZOKHA zochotsa Ma voliyumu otchulidwa omwe adalengezedwa mu `volumes:` gawo la `docker-compose.yml` ndi ma voliyumu Osadziwika omwe amalumikizidwa ku makontena; izo ayi imachotsa mafayilo mkati mwa khwekhwe bind mount (monga `- /opt/mysql_data:/var/lib/mysql` kapena `- ./postgres_data:/var/lib/postgresql/data`). Ngati fayilo yanu yolemba idagwiritsa ntchito njira yoyambira `/` kapena `./`, yang'anani fodayo pa wolandirayo pompano: mafayilo anu akadali 100 peresenti.
Chifukwa chiyani? docker system prune --volumes Chotsani voliyumu yanga ya database ndikangoyimitsa chidebecho kwa mphindi 2 kuti ndisinthe kusintha kwachilengedwe?¶
Dikirani mosamala kusiyana pakati `docker stop` ndi `docker rm`: ngati chidebe ndi chabe `Stopped / Exited` (zowoneka mu `docker ps -a`), `docker system prune --volumes` amachita ayi Chotsani ma voliyumu ake chifukwa chotengera choyimitsidwa chikuwalozerabe. Komabe, ngati munathamanga `docker rm
Kodi ndingabwezeretse voliyumu yochotsedwa ya Docker mkati mwa Docker Desktop pa Mac (Apple Silicon / Intel) kapena Windows (WSL2)?¶
Inde, ngati musiya Docker Desktop nthawi yomweyo kuti makinawo asiye kugwira ntchito `fstrim` motsutsana ndi disk virtual. Pa macOS, Docker imasunga ma voliyumu onse a Linux mkati `-/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw`; pa Windows WSL2, mkati `%LocalAppData%\Docker\wsl\disk\docker_data.vhdx`. Koperani izo `.raw` kapena `.vhdx` fayilo ku SSD yakunja nthawi yomweyo, igwirizanitse kuti muwerenge mu Linux (pogwiritsa ntchito `losetup -Pr` za `.raw` kapena `qemu-nbd -r` za `.vhdx`), ndi kuthamanga `ext4magic` kapena `PhotoRec` motsutsana ndi virtual `ext4` kugawa mkati.
Bwanji ngati nditachotsa Kubernetes StatefulSet ndi kubectl delete statefulset postgres? Kodi izi zinachotsanso ma PVC anga?¶
Ayi. Mwachikhazikitso ku Kubernetes (pokhapokha `persistentVolumeClaimRetentionPolicy.whenDeleted: Delete` idakonzedwa bwino pa Kubernetes 1.27+), ikuyenda `kubectl delete statefulset` imachotsa wolamulira wa StatefulSet ndi ma Pods ake, kusiya zonse `PersistentVolumeClaims` (`data-postgres-0`, `data-postgres-1`) ndi deta yawo yokhazikika 100 peresenti. Ingogwiritsaninso ntchito yanu `StatefulSet` YAML chiwonetsero ndi dzina lomwelo ndi `volumeClaimTemplates`, ndipo ma Pods atsopano adzalumikizanso ndi omwe alipo `data-postgres-0` Zithunzi za PVC.
Kodi ndimapeza bwanji mafayilo mkati mwa Docker overlay2 chikwatu pamene chidebe chikukana kuyamba (CrashLoopBackOff / Exited (1))?¶
Osachotsa chidebe chophwanyika. Malingana ngati chidebe choyimitsidwa chidakalipo `docker ps -a`, wosanjikiza wake wolembedwa ndi ma voliyumu okwera amakhala osasunthika pa disk. Inu mukhoza mwina kukopera owona mwachindunji kuchokera anasiya chidebe ntchito `docker cp
Bwanji ngati wina achotsa Kubernetes Namespace (kubectl delete ns prod) ndi ma PVC onse anali nawo reclaimPolicy: Delete pa AWS EBS kapena GCP?¶
Ngati cloud CSI driver (`ebs.csi.aws.com` kapena `pd.csi.storage.gke.io`) aphedwa kale `DeleteVolume` chifukwa `reclaimPolicy` anali `Delete`, yang'anani maukonde atatu otetezedwa amtambo nthawi yomweyo: (1) fufuzani ** AWS Recycle Bin for EBS Volumes ** mu AWS EC2 console (ngati lamulo losungirako linali logwira ntchito pa akaunti / ma tag, voliyumu ya EBS yochotsedwa ikukhala mu Recycle Bin ndipo ikhoza kubwezeretsedwa mu 1 pitani); (2) fufuzani ** AWS Backup / DLM / GCP Snapshot Schedule ** kuti mupeze chithunzithunzi chaposachedwa kwambiri cha voliyumu; ndi (3) onani ngati database yanu ikukhamukira WAL/binlogs (`WAL-G`, `pgBackRest`, `Litestream`) ku chidebe cha S3/GCS.
Maupangiri ogwirizana¶
- Seva disk 100% yodzaza (0 byte yaulere): database ya WAL log bloat, B-tree yowonongeka, ndi kuchira kwa zero
- Mwangozi
rm -rfkapena `Shift+Delete' pa Linux, macOS, kapena Windows: kukwera kowerengeka komweko ndikujambula kwa inode - Virtual makina VMDK, VHDX, ndi qcow2 chithunzithunzi unyolo ziphuphu ndi hypervisor kupulumutsa
- Kuchita Mwangozi Git kwa makiyi a AWS, fayilo ya
.env, kapena kiyi yachinsinsi ya SSH ku GitHub repo - Vuto la kuchititsa pa intaneti, kusamuka, ndi kubwezeretsanso zosunga zobwezeretsera
- Zosungira zamakompyuta, njira ya 3-2-1, ndi kubwezeretsa masoka
- Nambala zadzidzidzi malinga ndi dziko
Magwero ndi malangizo ovomerezeka¶
- Kubernetes Official Documentation (
kubernetes.io/ Cloud Native Computing Foundation): Chinthu Chosungira Mukugwiritsa Ntchito Chitetezo (kubernetes.io/pvc-protectionOmaliza), Reclaiming PersistentVolumes (Retainvs.Delete), ndi KuchitaetcdMagulu a Kubernetes (etcdctl snapshot restore). - Malingaliro a kampani Docker Inc.
docs.docker.com): * Sinthani Zambiri mu Docker: Volumes, Bind Mounts,overlay2Zomangamanga Zoyendetsa Dalaivala, ndi Prune Reference *. - Bungwe la Linux Kernel (
kernel.org): Ndi/procFilesystem (procfsTsegulani File Descriptor Recovery kudzera/proc/<pid>/fd/) ndi Ext4 Journal Specification. ext4magicZolemba za Project (ext4magic.sourceforge.net): Forensic Inode ndi Journal Transaction Recovery pa Ext3/Ext4 Filesystems.- National Institute of Standards and Technology (
NIST) (United States): Special Publication 800-190: Application Container Security Guide & Incident Recovery.