Docker or Kubernetes Production Volume Muku: Whakaorangia docker system prune --volumes, overlay2 Inodes, me etcd Nga whakaahua¶
He rereke nga nama ohorere i ia whenua. Ka whakamahia e tenei wharangi te 911, te nama mo te United States me Canada. He maha nga whenua e whakamahi ana i te 112, 999, tetahi atu nama ranei. Kimihia te nama mo to whenua.
Rere ana docker compose down -v, docker system prune -a --volumes, docker volume rm, ranei kubectl delete pvc i runga i te ripa kapeka he i roto i te hanga ko tetahi o nga huarahi tere ki te muru i te PostgreSQL ora, MySQL, MariaDB, Redis, MongoDB, SQLite ranei rōrahi pātengi raraunga. Kaore i rite ki nga whakaahua miihini mariko kapua ka neke ki te ipu para 7-ra, ko te taraiwa rōrahi rohe a Docker (/var/lib/docker/volumes/<name>/_data) me Kubernetes local-path / hostPath ka tukuna e nga kaiwhakarato te POSIX paerewa unlink() a rmdir() ka karanga tika ki te punaha konae kaihautu Linux (ext4, XFS, ranei Btrfs). Ko nga tuhi katoa ka puta i runga i taua kaihautu Linux i muri i te mutunga o te whakahau ka taea te tuhirua i nga poraka papaa raraunga kua wetekina ki runga kōpae. Mena kua whakakiia to kōpae tūmau ki te 100 ōrau (0 bytes free) i mua i te omatanga o tetahi docker system prune --volumes i roto i te panic, consult Ko te kōpae tūmau 100% ki tonu, te putunga rangitaki WAL pātengi raraunga, me te whakaora mokowā-kore; mo te whakakore i nga konae Linux i waho o nga ipu, tirohia ohorere rm -rf, Shift-Muku ranei i te ext4, XFS, APFS, me te NTFS: panui-anake te remount me te whakairo inode a Mīhini mariko VMDK, VHDX, me te qcow2 te pirau mekameka hopu whakaahua me te whakaoranga hypervisor. Mena ka pa te mate o tenei whakaputanga ki te tiaki i nga turoro o te hohipera, te tuku ohorere, me nga punaha haumaru taputapu a te iwi, waea atu ki to nama whawhati tata (911 i te United States me Canada, 112 puta noa i te Kotahitanga o Europi, 999 i roto i te United Kingdom) tirohia to maatau raarangi ingoa tau ohorere.
Mahia tenei inaianei¶
- ** KAUA e oma
docker compose up -d,systemctl restart docker, ka tukuna ano ranei te Pod Kubernetes**: Ma te hanga ano i te ipu, ki te whakaara ano ranei i te daemon Docker ka tuhia nga megabytes o nga mea hou.overlay2metadata, ipu JSON rangitaki, me nga konae putunga putunga hou i arawhitia (initdb) tika i runga i te tikaext4raneiXFSporaka i noho ai to rōrahi kua mukua i ngā hēkona ki muri. - Tirohia i roto i te 5 hēkona mena kei te mahi ipu (
postgres,mysqld,redis-server,node,python) kei te haere tonu i roto i te mahara (ps auxasudo lsof +L1): Ki te mukua e te tangata nga konae kei roto i te rōrahi kua whakairihia, ka wetewete ranei i tetahi rōrahi ingoamuna i te wa e rere tonu ana te ipu, ka noho tonu te kernel Linux 100 ōrau o ngā kōnae pātengi raraunga kore hono i runga i te kōpae i te mea ko te ID Tukanga o te ipu (PID) e pupuri ana i tetahi kaiwhakaahua konae tuwhera (FD). Ka taea e koe te kape tika i aua konae ki waho/proc/<PID>/fd/ranei/proc/<PID>/root/me te kore ngaronga raraunga i mua i te tu o te ipu. - Mēnā kua tū kē te ipu ka mukua te rōrahi (
docker compose down -vraneidocker system prune --volumes), whakatio te kopae kaihautu ka tuhi tonu: Kati te daemon Docker (sudo systemctl stop docker.socket docker.service containerd) na nga ipu papamuri me nga tirotiro hauora ka mutu te tuhi i nga raarangi ki roto/var/lib/docker, katahi ka whakaara ano/var/lib/docker(te pakiaka ranei/wehewehe) panui-anake (sudo mount -o remount,ro /) ka mau tonu ranei he hopukanga o te rōrahi kapua taumata paraka (AWS EBS Snapshot,GCP Persistent Disk Snapshot,DigitalOcean Volume Snapshot) i mua i te pa ki tetahi mea. - Mena kei runga koe i nga Kubernetes (
kubectl delete pvcraneikubectl delete namespace), tirohia te PersistentVolume (PV)ReclaimPolicy(kubectl get pv) tonu: - Mena ko te
PVipersistentVolumeReclaimPolicy: Retain, ko o raraunga kaore i mukua: tePVneke noa kiReleasedtūnga me tō kōpae kapua (EBS,Azure Disk,Ceph RBD,NFS) he 100 paiheneti tonu. - Ahakoa
kubectl delete pvckua mau ki rotoTerminating, Kubernetes Whakaoti (kubernetes.io/pvc-protection) aukatihia te atekōkiri CSI kia kore e mukua te rokiroki o raro i te mea kei te korero tonu tetahi Pod ki taua PVC. Kaua e mukua te Pod kia kape ra ano koe i nga raraunga mai i te Pod e rere ana. - Whakaahua te arai paraka o raro (
dd/ddrescuei runga i te SSH) ki te kōpae o waho, te rōrahi kapua motuhake ranei i mua i te rereext4magic,extundelete, raneiPhotoRec: Kaua rawa e whakahaere i nga kaiwhakairo konae e tuhi ana i nga konae whakaputa ki runga i te wehenga kōpae kotahi e ora ana koe/var/lib/docker/volumes. - Mo te ahua o te roopu Kubernetes kua kino, kua murua ohorere ranei (
kubectl delete crd/namespace), mutukube-apiserverme te whakahoki i to mea houetcdwhakaahua (etcdctl snapshot restore): Mēnā kei roto te kāhuietcdi murua i te wa e noho tonu ana nga pukapuka rokiroki kapua, e whakaora anaetcdwhakaora katoaPV,PVC,Secret, aConfigMaptaonga i te wa kotahi.
Danger
** Kaua rawa e tangohia te kubernetes.io/pvc-protection ranei kubernetes.io/pv-protection whakaotinga me kubectl patch ... -p '{"metadata":{"finalizers":null}}' ka mau te PVC ki roto Terminating**: Ko taua kaiwhakamutu anake te mea hei pupuri i to taraiwa CSI kapua (aws-ebs-csi-driver, pd.csi.storage.gke.io, Longhorn, Rook-Ceph) mai i te tuku taputapu DeleteVolume Karanga API ki te whakangaro i to kōpae whakaputa. I te wa e mau ana te PVC ki roto Terminating, kei te rere tonu te Pod, kei te mau tonu te rōrahi: rere kubectl exec ranei kubectl cp (ranei pg_dump / mysqldump) ki te whakaora i te 100 ōrau o tō pātengi raraunga i mua i te mutunga o te Pod.
Rakau whakatau: kua mukua te rōrahi Docker, overlay2 paparanga, kubernetes PVC triage ranei¶
Whakamahia tenei tūtohi rerenga ki te whiriwhiri i te huarahi whakaora kore-mate tere ake i runga i te mea kei te ora tonu te tukanga ipu, ahakoa i tautokohia te rōrahi e te kōpae paetata, te rokiroki kapua CSI ranei, a ko wai te punaha konae e manaaki ana. /var/lib/docker:
flowchart TD
Start["Kua ngaro nga Raraunga Whakaputa I muri i te Whakahau a Docker, Kubernetes ranei te Mukua"] --> WhichEnv{"Ko tehea te papa me te whakahau i mate ai?"}
WhichEnv -->|"Docker: E Rere tonu ana te Ipu (Ko nga konae kua kore e hono i roto i te rōrahi/ipu)"| ProcRescue["Whakaorangia Whakatuwheratia nga Kaitohu Kōnae Kore Hononga ma te sudo lsof +L1 me /proc/PID/fd/N ranei /proc/PID/root/"]
WhichEnv -->|"Docker: I Tuhia te Ipu & Kua Mukua Rōrahi (tito whakararo -v / puru --rōrahi)"| StopDocker["Whakahaere tonu: sudo systemctl mutu docker.socket docker containerd"]
WhichEnv -->|"Kubernetes: kubectl muku pvc / pv / statefulset / mokowāingoa"| K8sCheck{"Kei te Rere tonu te Pod, kei te mau ranei te PVC ki te whakamutu?"}
K8sCheck -->|"Āe: E Rere tonu ana te Pod (Kaiwhakamutunga te aukati CSI Muku)"| K8sExec["KAUA E Mukua Pod -> Whakahaerehia te kubectl exec / pg_dump / tar ki runga i te stdout ki te Kōpae Paetata Inamata"]
K8sCheck -->|"Kao: Pod Kua ngaro -> Tirohia kubectl tiki pv"| ReclaimCheck{"He aha te kaupapa here Reclaim PV?"}
ReclaimCheck -->|"Pupuri (Tūnga PV = Kua Tukua)"| ClearClaimRef["100% Raraunga Tonu -> Whakatika PV ki te Tango spec.claimRef.uid -> Herea ano ki te PVC Hou"]
ReclaimCheck -->|"Mukua (Cloud EBS / PD / Longhorn / Ara-a-rohe)"| CloudOrLocal{"Rōrahi Kapua CSI, Kōpae Kaihautū Rohe rānei?"}
CloudOrLocal -->|"Kapua CSI (AWS EBS / GCP / Azure / Ceph)"| CloudSnap["Tirohia te Ipu Para Papatohu Kapua / Nga Whakaahua EBS Aunoa / Ceph RBD Parapara (rbd paru ls)"]
CloudOrLocal -->|"Kōpae Kōpuku Paetata (ara-rohe / hostPath)"| StopDocker
StopDocker --> RemountRO["Whakanuia ano te Wehewehenga Kaihautu Panui-anake (maunga -o remount,ro) RĀNEI Aroma Paraka Atahanga ki runga i te SSH"]
RemountRO --> HostFS{"He aha nga Kaihautu Pūnaha Kōnae /var/lib/docker?"}
HostFS -->|"ext4"| Ext4Rec["Whakamahia debugfs / ext4magic Journal Inode Transaction Rollback (-he waitohuwā)"]
HostFS -->|"XFS / Btrfs"| CarveRec["Whakamahia Btrfs Roots / xfs_irecover / Whakairo Waitohu Ritenga mo SQLite, Postgres Heap, InnoDB .ibd ranei"]
Wāhanga 1: Me pehea te whakakore a Docker me Kubernetes i nga pukapuka kei runga kōpae¶
I mua i te whakahaere i tetahi whakahau whakaora, me mohio kei hea to raraunga i noho tinana i runga i te kaihautu Linux me nga mea i pa ki te whakahau muku:
| Momo Rokiroki & Muku Whakahau | Kaihautū Ara Pūnaha Kōnae i runga i te Linux Node | He aha te Whakahau Whakamuku I Mahi | Tikanga Whakaora Tere |
|---|---|---|---|
Whakaingoatia Rōrahi Docker (docker volume rm ranei docker compose down -v) |
/var/lib/docker/volumes/<vol_name>/_data |
I wetewetehia nga konae kei roto _data/ ka tangohia te urunga whaiaronga mai /var/lib/docker/volumes/metadata.db (boltdb). |
Kati tonu i a Docker, whakanekehia /var/lib/docker panui-anake, me te whakaora i nga inodes kore hono ma ext4magic / debugfs whakairo waitohu ranei. |
** Rōrahi Docker ingoakore (docker run -v /var/lib/mysql + docker rm -v ranei docker system prune --volumes)** |
/var/lib/docker/volumes/<64-char-hex-id>/_data |
docker system prune --volumes ka mukua ia rōrahi ingoamuna me te whakaingoatia karekau i te tohutoro i tenei wa e te ipu kotahi. |
Kati tonu a Docker; Mēnā he ētahi atu kua aukatihia e koe nga ipu kua kare i tapatapahia, tirohia e toe ana /var/lib/docker/volumes/*/_data kopaki i te tuatahi i mua i te whakairo i nga poraka kaore i tohatohahia. |
Ipu Pae Tuhituhi (docker rm <container> i te wa i tuhia ai te raraunga ki waho o te rōrahi) |
/var/lib/docker/overlay2/<layer-id>/diff/ |
I wetekina te hononga o runga o te ipu diff/ whaiaronga kei roto i te overlay2 taraiwa rokiroki. |
Kati tonu a Docker ka whakaora i te rakau whaiaronga kore hono mai i te ext4 hautaka ranei XFS poraka. |
** Papamahi Docker i runga i te macOS, i te Matapihi ranei (docker system prune --volumes)** |
I roto i te ahua kōpae miihini mariko kotahi: -/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw (macOS) ranei %LocalAppData%\Docker\wsl\disk\docker_data.vhdx (Windows WSL2). |
Tukuna unlink() i roto i te Linux VM, a mena discard/TRIM kei te kaha Docker.raw / docker_data.vhdx, ka werohia nga kohao iti ki te konae kaihautu. |
Whakamutua te Papamahi Docker i roto i te 5 hēkona, kape Docker.raw ranei docker_data.vhdx ki te puku o waho, ka oma ext4magic / PhotoRec ki te ahua kōpae mariko. |
Kubernetes PVC ki Delete Kaupapa Whakaora mo Ceph RBD (Rook-Ceph) ranei Longhorn |
puna Ceph OSD (rbd) ranei /var/lib/longhorn/replicas/. |
Mehemea csi-rbdplugin kua trash kua whakahohea (rbd trash), ka nekehia te ahua RBD ki te ipu paru Ceph; ki te kore e kore e mapi nga poraka. |
Rere rbd trash ls --pool <pool> roto te rook-ceph-tools pod me te whakaora ki rbd trash restore. |
Wāhanga 2: Whakaorangia ngā kōnae pātengi raraunga tuwhera mai i tētahi ipu e rere tonu ana (/proc/<pid>/fd/)¶
Tatari: kei te rere tonu te ipu inaianei?
I te nuinga o te wa, ka whakahaerehia e te miihini he tuhinga heke tono, he mahi cron horoi, he pukapuka pukapuka ranei rm -rf /var/lib/postgresql/data/* ranei rm /app/data/production.sqlite3 i roto i te ipu e rere ana, ka mukua ranei te kōpaki here-maunga o raro /opt/app/data ko te ipu ano (postgres, mysqld, mariadbd, redis-server, mongod, gunicorn, node) kei kei te haere tonu me te rere.
I roto i te Linux, karekau he konae e mukua mai i te kōpae ina rm ranei unlink() ka kiia. Ko ona paraka kōpae me ona inode kei te noho 100 paiheneti kua tohatohahia me te panui tae noa ki te tukanga whakamutunga e pupuri ana i tetahi kaiwhakaahua konae tuwhera (FD) ki taua inode ka kati, ka puta ranei.
Hipanga 1: Kimihia te ID Tukanga kaihautu (PID) o te tukanga ipu¶
I runga i te node ope Linux (kaore i roto i te ipu), kimihia te kaihautu PID o to papaunga raraunga, tukanga tono ranei:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
mysqld 18492 999 14u REG 259,1 482344960 0 1849201 /var/lib/mysql/app_prod/users.ibd (deleted)
mysqld 18492 999 15u REG 259,1 125829120 0 1849202 /var/lib/mysql/app_prod/orders.ibd (deleted)
mysqld 18492 999 16u REG 259,1 79691776 0 1849190 /var/lib/mysql/ibdata1 (deleted)
python3 22104 1000 4u REG 259,1 89128960 0 3102944 /data/production.sqlite3 (deleted)
NLINK: e kii ana 0, ka mutu te ingoa kōnae ki te (deleted). Kua ngaro katoa te konae ls /var/lib/docker/volumes/..., heoi ia paita kotahi (482 MB i roto users.ibd, 89 MB i roto production.sqlite3) kei te ora tonu i roto /proc/<PID>/fd/<FD>.
Hipanga 2: Tāruahia ngā kōnae pātengi raraunga kore hono tika ki waho /proc/<PID>/fd/¶
Kaua **kaore e mutu, ka whakaara ano ranei i te ipu. I te wa e haere tonu ana te mahi, kape tika nga kaiwhakaahua konae ki tetahi kōpaki haumaru i runga i to kaihautu:
sudo mkdir -p /root/proc_emergency_rescue
# Example 1: Rescuing an unlinked SQLite3 database (PID 22104, FD 4)
sudo cp /proc/22104/fd/4 /root/proc_emergency_rescue/production.sqlite3
# Check if PID 22104 also holds the -wal file open (e.g. FD 5) and copy it too:
sudo cp /proc/22104/fd/5 /root/proc_emergency_rescue/production.sqlite3-wal
# Example 2: Automatically rescuing ALL open (deleted) MySQL/Postgres files for PID 18492
for fd in /proc/18492/fd/*; do
target=$(sudo readlink "$fd" 2>/dev/null)
if echo "$target" | grep -q "(deleted)"; then
fd_num=$(basename "$fd")
clean_name=$(echo "$target" | sed 's/ (deleted)//; s/\//_/g')
echo "Rescuing FD $fd_num -> $clean_name"
sudo cp "$fd" "/root/proc_emergency_rescue/${fd_num}${clean_name}"
fi
done
Hipanga 3: He pai ake: Whakahaerehia he putunga arorau ora (pg_dump / mysqldump / .backup) i mua i te kati o te ipu i nga FD¶
Mehemea postgres ranei mysqld kei te rere tonu me ona konae raraunga e tuwhera ana ki te mahara me /proc/<PID>/fd/, ngana ki te whakahaere i tetahi putunga arorau ora i runga i te turanga unix, hononga TCP ranei i te tuatahi:
# If the database engine can still read its open file descriptors:
docker exec <container_name> pg_dumpall -U postgres > /root/proc_emergency_rescue/live_dump.sql
# Or for MySQL:
docker exec <container_name> mysqldump -u root -p --all-databases --single-transaction > /root/proc_emergency_rescue/live_mysql.sql
.1, .2 wehewehe i nga konae puta noa i nga mahi a nga kaimahi, na pg_dump ka taka pea i runga i nga tepu keteroki kore; te whakakotahi i te uiui SQL ora me te kape i nga tuwhera katoa /proc/<pid>/fd/* puta noa katoa postgres Ka tiakina e nga PID tuara tamaiti te nui rawa o nga raraunga puranga ripanga.)
Wāhanga 3: Te whakaora i te rōrahi Docker kua mukua ext4 ranei XFS ka mutu te ipu¶
He aha mena ka oma koe docker compose down -v ranei docker system prune --volumes, no reira kua kua tu te ipu me te katoa /proc/<pid>/fd/ kua katia nga whakaahuatanga? Inaianei kei te whakataetae koe ki te kopae papamuri e tuhi ana i runga i te kaihautu Linux.
Hipanga 1: Whakatiohia te punaha konae manaaki Linux i roto i te 10 hēkona¶
- Kati tonu te Docker, containerd, kubelet, me te cron na karekau he ipu, he tirotiro hauora, he kaitahuri rakau ranei e tuhi i tetahi atu paita ki
/var/lib/docker: - Tirohia ko tehea taputapu paraka me te punaha konae kei te pupuri
/var/lib/docker: - Whakahokia ano te wehewehenga Panui-anake (whakarerehia ranei he atahanga wehewehenga mata ki runga SSH ki tetahi atu miihini):
- Mehemea
/var/lib/dockerkei runga i tana ake wehewehenga i whakatapua, i te rōrahi LVM ranei (hei tauira/dev/nvme0n1p2ranei/dev/mapper/vg0-docker), whakairihia te panui-anake tonu: - Mehemea
/var/lib/dockerkei runga i te putake/wehewehe (/), ngana ki te akiaki i te arai pakiaka kia panui-anake ma te whakamahi i te kaaka Magic SysRq keu (echo u | sudo tee /proc/sysrq-trigger) raneisudo mount -o remount,ro /. - Mena kei runga koe i te VPS kapua (AWS EC2, GCP, DigitalOcean, Hetzner, Linode): Haere tonu ki to papatohu tukutuku kapua ka paato i te Waihanga Whakaahua ki te rōrahi poraka o te tūmau (whakarerehia ranei te taputapu paraka mata ki runga SSH ki te tūmau tuarua:
sudo dd if=/dev/nvme0n1p1 bs=4M status=progress | gzip -1 | ssh user@backup-server "cat > /mnt/storage/docker_host_raw.img.gz").
Hipanga 2: Tirohia mehemea kei te noho tonu to raraunga i roto i TĒTAHI rōrahi karekau ranei overlay2 paparanga¶
I mua i te whakairo i nga paraka kōpae kare i tohatohahia, manatokohia karekau to papaunga raraunga i penapena i roto i te maunga here, i te ingoa rereke ranei i ora:
# Search the entire /var/lib/docker tree read-only for any surviving database files:
sudo find /var/lib/docker -name "PG_VERSION" -o -name "ibdata1" -o -name "*.sqlite*" -o -name "*.db" -o -name "dump.rdb" -o -name "WiredTiger" 2>/dev/null
docker volume rm app_db_data, ka kitea anake me find /var/lib/docker tera docker-compose.yml i tino hanga projectname_app_db_data (kei te 100 ōrau tonu kei runga i te kōpae) i te wā app_db_data he pukapuka tawhito kaore i whakamahia mai i nga marama e ono ki muri.
Hipanga 3: Te whakaora i nga raarangi pukapuka Docker kua kore e hono ext4 te whakamahi i te ext4 Tuhituhi (ext4magic a debugfs)¶
Ina docker volume rm ka mukua /var/lib/docker/volumes/<vol_name>/_data i runga i te ext4 pūnaha kōnae, ext4 ka kore nga tohu paraka tika (i_block / extent tree root) i te ripanga inode ora. Heoi ano, ko te ext4 hautaka (inode <8>, te tikanga 128 MB ki 1024 MB porowhita whakakai porowhita) kei te pupuri tonu i nga kape tawhito o nga ripanga inode tika me nga poraka whaiaronga mai i mua tonu docker compose down -v i patua.
+-------------------------------------------------------------------------+
| HOW ext4magic RECOVERS A DELETED DOCKER VOLUME FROM THE EXT4 JOURNAL |
| |
| Live Inode Table on Disk (After docker volume rm): |
| Inode #918402 (users.ibd): [Size = 0, Extent Pointers = ZEROED] |
| |
| Ext4 Journal Ring Buffer (Inode <8> on Disk): |
| Transaction #41902 (10 mins ago): |
| Inode #918402 (users.ibd): [Size = 482MB, Extents = Blocks 9000..9999]|
| |
| -> ext4magic reads the old Inode Copy from Journal Transaction #41902 |
| and extracts Blocks 9000..9999 directly to your rescue drive. |
+-------------------------------------------------------------------------+
Rere ext4magic ki to wahanga panui-anake (te .img kape whakaahua i runga i te tūmau tuarua), e tohu ana i te tohuwā (-a) mai i **5 meneti i mua i a ** i oma koe docker compose down -v ranei docker system prune --volumes:
# Install ext4magic on your rescue machine (never on the affected partition)
sudo apt-get install -y ext4magic e2fsprogs
# Dump the ext4 journal from the frozen partition / image to a file in RAM (/dev/shm)
sudo debugfs -R "dump <8> /dev/shm/ext4_journal.copy" /dev/nvme0n1p1
# Recover all files deleted under var/lib/docker/volumes after (Current Time - 15 minutes)
sudo mkdir -p /mnt/external_rescue/docker_recovered
sudo ext4magic /dev/nvme0n1p1 \
-j /dev/shm/ext4_journal.copy \
-f var/lib/docker/volumes \
-a $(date -d "-15 minutes" +%s) \
-r -d /mnt/external_rescue/docker_recovered
-j /dev/shm/ext4_journal.copy: Ka panuihia te whakaahua hautaka i tiakina e koe ki te RAM i muri tonu i te aukati i a Docker.
- -f var/lib/docker/volumes: E whai ana i te rakau whaiaronga pukapuka Docker tika (tuhipoka: waihohia te arataki / i roto ext4magic ara whanaunga).
- -a $(date -d "-15 minutes" +%s): Ka korero ext4magic ki te hanga ano i te rakau whaiaronga ma te whakamahi i nga kape inode whaimana 15 meneti ki muri (i mua tonu i te mukunga ohorere).
- -r: Ka whakaora i nga whaiaronga katoa kaore he poraka i tuhiruatia; mehemea -r ka ngaro i nga konae kua tuhiruatia, whakahaere i te tuku tuarua hei whakakapi -r me -m (whakairo makutu + aratau ranu hautaka).
Hipanga 4: Te whakairo i nga konae SQLite mata, PostgreSQL, MySQL InnoDB ranei ina takai te hautaka (XFS ranei ext4)¶
Warning
He aha XFS e kore e tautoko ext4magic me te hiahia kia wetewete te panui-anake tonu: Kaore i rite ext4, e tuhi ana i nga kape inode 256-paita katoa ki roto i tana hautaka, XFS ka tuhi i nga tohu metadata intent me te kaha ki te whakamahi ano i nga inode B-rakau (agi). Mehemea /var/lib/docker noho ana i runga i te XFS wehewehenga (taunoa i runga i te RHEL, Rocky Linux, AlmaLinux, me Amazon Linux), wetewete /var/lib/docker te whakanohonoho ranei / Ko te panui-anake i roto i nga hēkona ko te taahiraa tino nui i mua i te whakairo.
Mena kei te whakamahi te kaihautu XFS mehemea ranei te ext4 hautaka kua takaia ki runga i nga tohu inode, whakamahia te waitohu me te whakairo hanganga ki te ahua kōpae mata (docker_host_raw.img):
- Mo nga Raraunga Raraunga SQLite3 (
.sqlite,.db- hei tauira, Vaultwarden, Nextcloud, Kaiāwhina Kāinga, Gitea, Grafana): Ka timata nga papaa raraunga SQLite3 me te pane ASCII 16-paita e kore e pohehe i te paita 0 o tana wharangi 4 kilopaita tuatahi:53 51 4c 69 74 65 20 66 6f 72 6d 61 74 20 33 00("SQLite format 3\0"), me te penapena i te rahinga wharangi papaarangi tika (Bytes 16..17, te tikanga4096) me te rahinga pātengi raraunga katoa i ngā whārangi (Bytes 28..31, nui-endian 32-bit integer) kei roto tonu i taua pane. - Rere
PhotoRecme[File Opt]->[x] sqlitewhakahohea, karapa tika ranei i te wehewehengagrep -ab "SQLite format 3"ki te kimi i te paita tika o to konae SQLite kua mukua me te tangodd:Kia tangohia, rere# Find the exact byte offsets of every "SQLite format 3" header on the partition: sudo grep -ab "SQLite format 3" /dev/nvme0n1p1sqlite3 carved.db ".recover" | sqlite3 repaired.dbki te hanga ano i ia wharangi rau rakau B mau tonu.
Tip
Me pehea te tatau i te tika dd paita te rahi o te konae SQLite3 kore hono mai i tana pane: Kotahi grep -ab "SQLite format 3" ka hoatu ki a koe te paita timata OFFSET o to raraunga SQLite kua mukua, panui paita 16..17 (uint16 Rahi Whārangi nui-endian, hei tauira, 0x1000 = 4096 paita) me nga paita 28..31 (Uint32 Kaute Whārangi nui-endian, hei tauira, 25000 wharangi) whakamahi hexdump -C -s OFFSET -n 32. Whakareatia Page Size x Page Count homai ki a koe te roa paita tika hei tuku atu dd (iflag=skip_bytes,count_bytes) i te wa e piri ana te konae ki runga kōpae.
- Mo MySQL / MariaDB InnoDB (
.ibdkōnae ripanga): Ia InnoDB.ibdhe raupapa raupapa16 KB(16,384paita) nga wharangi kei roto teFIL_PAGE_INDEX(0x45BF) pane me te ahurei o te ripangaspace_id. Whakamahia te Percona / TwinDBundrop-for-innodb(stream_parser+c_parser) kete taputapu hei matawai/dev/nvme0n1p1mo te katoa16 KBNga wharangi taurangi InnoDB ka maka tika ia rarangi ki nga konae TSV/SQL hei kawemai ano. - Mo nga konae PostgreSQL Heap (
$PGDATA/base/<db_oid>/<relfilenode>): Ka penapenahia e PostgreSQL ia tepu hei piri8 KB(8,192paita) heap page kei rotoPageHeaderData(pd_lower,pd_upper,pd_special). Whakamahiapg_filedumpraneipglazarus/pduki te matawai i nga rohe kōpae kaore i tohatohahia8 KBnga wharangi e rite ana ki te tahora tuple o to ripanga (HeapTupleHeaderData) me te hanga ano i nga rarangi.
Wāhanga 4: Kubernetes PVC, PV, me etcd Whakaahuatanga Whakaora ohorere¶
Kei Kubernetes (EKS, GKE, AKS, k3s, RKE2, kubeadm, OpenShift), te muku ohorere i tetahi PersistentVolumeClaim (kubectl delete pvc), he StatefulSet, he Mokowā Ingoa katoa ranei (kubectl delete ns prod) e whai ana i tetahi hoahoanga tino rereke atu i te Docker tuuturu.
Tauari A: Ko nga whakaaturanga PVC Terminating (Ko te Kaiwhakaora Whakaora)¶
Whakahaere:
Mena ko teSTATUS ta te pou Terminating, kia mapu: KORE ano to raraunga i mukua.
- He aha? Ko ia Kubernetes PVC i hangaia mai i te Kubernetes 1.9 kei roto aunoa finalizers: [kubernetes.io/pvc-protection].
- I te mea kei te mau tonu tetahi Pod i taua PVC, kaore te kaiwhakahaere Kubernetes e whakaae ki te whakaoti i te whakakorenga PVC me te kaore e karanga te taraiwa CSI. DeleteVolume RPC.
- Mahinga Inamata: Kaua KORE te muku, te whakaara ano ranei i te Pod, ka KORE taapihia te kaiwhakaoti. Tāruahia nga raraunga katoa mai i te Pod e rere ana ki to miihini rohe, ki te peere S3 ranei:
# Stream a compressed tar backup or pg_dump out of the still-running pod immediately:
kubectl exec -n <namespace> <pod-name> -- pg_dumpall -U postgres > ./rescued_k8s_db.sql
# Or tar an entire mounted volume directory out over stdout:
kubectl exec -n <namespace> <pod-name> -- tar czf - /var/lib/postgresql/data > ./rescued_pvc_files.tar.gz
Tauari B: I whakakorea te PVC, a kubectl get pv whakaatu i te PersistentVolume i roto i Released āhua (ReclaimPolicy: Retain)¶
Whakahaere:
Mena ko to Puka Tonu (pv-prod-postgres) kei te rārangi tonu me STATUS: Released (na te mea kua whakamahia tana StorageClass reclaimPolicy: Retain), 100 paihēneti o ō raraunga kei te noho tonu i runga i te kōpae kei raro. Heoi, kare a Kubernetes e whakaae kia herea tetahi PVC hou ki a Released PV no te mea kei roto tonu te PV a spec.claimRef e tohu ana ki nga PVC tawhito kua mukua UID.
Me pehea te hanga a Released PV Available ano i roto i te 10 hēkona kaore he ngaronga raraunga:
1. Whakatikaina te ahanoa PersistentVolume ka muku i te spec.claimRef poraka:
kubectl get pv ano: ka huri tonu te mana mai i Releasedki Available.
3. Hangaia ano to PersistentVolumeClaim (whakaahua volumeName: <your-pv-name> me te rite ki te kaha / rokirokiClassName). Ka herea ki te PV o naianei me nga konae katoa e mau tonu ana.
Tauari C: Te whakaora i te ahua kahui Kubernetes kua murua mai i te etcd whakaahua (etcdctl snapshot restore)¶
Ka pehea mena ka oma tetahi kubectl delete namespace prod ranei helm uninstall ka murua e 200 nga whakaaturanga Kubernetes, Nga Mea ngaro, Mahere Whirihora, me nga whakamaramatanga PV i runga i te whakahaere whaiaro kubeadm, k3s, ranei RKE2 rererangi whakahaere?
- ** Kimihia to mea hou
etcdwhakaahua** (ma te taunoa i roto/var/lib/rancher/k3s/server/db/snapshots/,/var/lib/rancher/rke2/server/db/snapshots/, ranei to whakaritenga/var/backups/etcd/snapshot.db). - Manatokohia te tika o te hopu whakaahua i mua i te pa ki te kahui ora:
- Kati
kube-apiserveraetcdi runga i te (ng) pona rererangi whakahaere: - Kei
kubeadm(panuku pateko): Nuku/etc/kubernetes/manifests/kube-apiserver.yamla/etc/kubernetes/manifests/etcd.yamlmo te wa poto/root/manifests_hold/nakubeletka mutu nga ipu e rua kia ma, ka hoki ki runga/var/lib/etcdki/var/lib/etcd.bak. - Kei
k3s: Reresudo systemctl stop k3sasudo k3s server --cluster-reset --cluster-reset-restore-path=/path/to/snapshot. - Whakahokia te
etcdwhaiaronga raraunga i rungakubeadm: - Nuku
etcd.yamlakube-apiserver.yamlhoki ki roto/etc/kubernetes/manifests/. I roto i te 30 hēkona,kube-apiserverka hono ano ki te mea kua whakahokia maietcdtoa me nga mokowāingoa katoa kua mukua, ngaro, ConfigMap, me PersistentVolume ahanoa ka puta ano.
Wāhanga 5: Whakapakeke i nga pukapuka a Docker me Kubernetes ki te whakakore ohorere¶
Kia whakahokia mai to papaunga raraunga whakaputa, whakatinanahia enei arai hoahoanga e wha kia kore ai e taea e tetahi momo CLI kotahi te whakakore i te reo ora:
- ** Hurihia ia hanga Kubernetes
StorageClassmaireclaimPolicy: DeletekireclaimPolicy: Retain**: Ma te taunoa, i roto i te kapua StorageClasses (gp2,gp3,standard-rwo) taunoa kireclaimPolicy: DeleteKo te tikanga ko te whakakore i te PVC ka whakangaro i te kōpae kapua. Waihanga aStorageClassmereclaimPolicy: Retain(Pakihia ranei nga PV o naianei mekubectl patch pv <pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}') no reira ko te whakakore i te PVC, i te Mokowāingoa rānei e kore e mukua te rōrahi paraka o raro. - Whakahohehia te AWS EBS Recycle Bin / GCP Snapshot Schedules / Velero me CSI Snapshots: Ko te whakahohe i te AWS EBS Recycle Bin ka pupuri i nga rōrahi EBS kua mukua mo te 7 ki te 30 ra ahakoa waea mai te taraiwa CSI
DeleteVolume. - ** Kaua rawa e whakamahi
docker system prune --volumesi roto i nga tuhinga horoi aunoa, i nga kaikawe CI/CD**: Ki te tukinga poto te ipu papaa raraunga whakaputa (Exited (1)) i te wa tika ka mahi te mahi crondocker system prune -a --volumes -f, Ka whakaaro a Docker ki ana pukapuka ingoamuna "kaore i whakamahia" ka mukua ake. Tihorea--volumesmai i nga tuhinga horoi katoa. - Whakamahia nga whakairinga here kaihautu marama (
/srv/prod_data/postgres:/var/lib/postgresql/data) ranei nga papaaarangi whakahaere-a-waho mo nga kawenga mahi whakaputa mana: Kaore i rite/var/lib/docker/volumes/, kaihautu here here-maunga whaiaronga ki waho/var/lib/docker/e kaore i pa edocker compose down -v,docker volume rm, raneidocker system prune --volumes.
Nga nama ohorere a rohe me nga tikanga umanga¶
Ina pa ana tetahi ipu, te mukunga rokiroki Kubernetes ranei ki nga mahi haumanu o te hohipera, te tuku haumaru mo te iwi, me nga raraunga whaiaro kua whakaritea, ka piki tonu:
- Te Hononga o Amerika me Kanata: Waea
911ki te whakakorehia e te korenga o te kohinga ipu te aro turuki i nga turoro haumanu, nga punaha EHR, te tuku ohorere ranei. Panuitia te whakakore i nga hanganga ipu kino me nga maiki ipurangi ki CISA (1-844-Say-CISA/cisa.gov/report) ranei te Canadian Center for Cyber Security (1-833-CYBER-88). - United Kingdom: Waea
999mo te whakatuma ki te ora, ki nga hanganga NHS nui ranei. I raro i te UK GDPR, ko te whakakorenga, te ngaro ranei o te waatea o nga raraunga whaiaro me te kore he taapiri ka taea te whakahoki mai me arotake i te panui pakaru 72-haora ki te ICO (0303 123 1113). - Uniana Pākehā: Waea
112puta noa i nga whenua mema EU katoa. Ko nga hinonga kei raro i te NIS2 Directive me tuku he whakatupato moata ki to ratou CSIRT motu i roto i te 24 haora i muri mai i te mate nui o te ratonga, me te mau tonu o te ngaronga o nga raraunga whaiaro GDPR Tuhinga 33 te ripoata i roto i te 72 haora. - Ahitereiria me Aotearoa: Waea
000i Ahitereiria ranei111kei Aotearoa; pūrongo i ngā maiki hanganga tino nui ki te ACSC (1300 CYBER1). - India me Ahia-Kiwa: Waea
112i Inia mo te urupare ohorere; pūrongohia ngā maiki haumarutanga kapua, ipuipu ranei ki CERT-In (incident@cert-in.org.in) i roto i te matapihi 6-haora whakahau.
Pātai auau¶
E docker compose down -v ūkui i ngā maunga herenga kaihautū (./data:/var/lib/mysql) ko nga pukapuka Whakaingoa me te ingoakore Docker anake ranei?¶
`docker compose down -v` Ka muku ANAKE nga pukapuka kua whakaingoatia i roto i te `volumes:` wāhanga o `docker-compose.yml` me nga pukapuka ingoamu e piri ana ki nga ipu; te reira kore rawa ka muku i nga konae kei roto i te maunga herea kaihautu (pēnei i `- /opt/mysql_data:/var/lib/mysql` ranei `- ./postgres_data:/var/lib/postgresql/data`). Mena i whakamahia e to konae tito he ara kaihautu timata mai i te `/` ranei `./`, tirohia taua kōpaki kei runga i te kaihautu inaianei: kei te 100 paiheneti tonu o koutou kōnae.
Na te aha docker system prune --volumes mukua taku rōrahi pātengi raraunga ina mutu noa taku ipu mo te 2 meneti hei whakahou i te taurangi taiao?¶
Tatari marie mo te wehewehenga `docker stop` a `docker rm`: mehemea he ipu noa `Stopped / Exited` (ka kitea i roto `docker ps -a`), `docker system prune --volumes` mahi kaore mukua ona pukapuka na te mea kei te tohutoro tonu te ipu kua mutu. Heoi, ki te oma koe `docker rm
Ka taea e au te whakahoki mai i tetahi rōrahi Docker kua mukua i roto i te Papamahi Docker i runga i te Mac (Apple Silicon / Intel) Windows ranei (WSL2)?¶
Ae, ki te whakamutua tonutia e koe te Papamahi Docker ka mutu te whakahaere o te miihini mariko `fstrim` ki te kōpae mariko. I runga i te macOS, ka tiakina e Docker nga pukapuka Linux katoa i roto `-/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw`; ki Windows WSL2, ki roto `%LocalAppData%\Docker\wsl\disk\docker_data.vhdx`. Tāruatia tērā `.raw` ranei `.vhdx` konae ki te SSD o waho tonu, whakapirihia te panui-anake i te Linux (ma te whakamahi `losetup -Pr` mo `.raw` ranei `qemu-nbd -r` mo `.vhdx`), ka oma `ext4magic` ranei `PhotoRec` ki te mariko `ext4` wehewehe i roto.
He aha mena ka mukua e ahau tetahi Kubernetes StatefulSet me kubectl delete statefulset postgres? I mukua ano aku PVC?¶
Kao. Ma te taunoa i Kubernetes (mehemea `persistentVolumeClaimRetentionPolicy.whenDeleted: Delete` i tino whirihorahia i runga i te Kubernetes 1.27+), e rere ana `kubectl delete statefulset` ka mukua te kaiwhakahaere StatefulSet me ona Pods, mahue katoa `PersistentVolumeClaims` (`data-postgres-0`, `data-postgres-1`) me o ratou raraunga kei raro 100 paiheneti mau tonu. Me tono ano i to `StatefulSet` Whakaatu YAML me te ingoa kotahi me te `volumeClaimTemplates`, a ko nga Pods hou ka hono aunoa ki te waa `data-postgres-0` Ko nga PVC.
Me pehea taku uru atu ki nga konae kei roto i te Docker overlay2 whaiaronga ka kore te ipu e pai ki te timata (CrashLoopBackOff / Exited (1))?¶
Kaua e mukua te ipu kua pakaru. I te mea kei te noho tonu te ipu kua mutu `docker ps -a`, kua mau tonu tana paparanga tuhi me nga kohinga rōrahi ki runga kōpae. Ka taea e koe te kape tika i nga konae mai i te ipu kua mutu ma te whakamahi `docker cp
He aha mena ka mukua e tetahi tetahi Kubernetes Namespace (kubectl delete ns prod) me nga PVC katoa reclaimPolicy: Delete i runga i te AWS EBS, i te GCP ranei?¶
Mena ko te taraiwa CSI kapua (`ebs.csi.aws.com` ranei `pd.csi.storage.gke.io`) kua oti kē `DeleteVolume` no te mea `reclaimPolicy` ko `Delete`, tirohia kia toru nga kupenga haumaru taumata-kapua tonu: (1) tirohia ** AWS Recycle Bin mo EBS Volumes** i roto i te AWS EC2 console (mehemea he kaha te ture pupuri i runga i nga kaute / tohu, ka noho te pukapuka EBS kua mukua ki roto i te Recycle Bin ka taea te whakahoki mai i te 1 panui); (2) tirohia **AWS Backup / DLM / GCP Snapshot Schedules** mo te hopukanga rōrahi aunoa hou; me (3) tirohia mena kei te rere to papaarangi raraunga WAL/binlogs (`WAL-G`, `pgBackRest`, `Litestream`) ki te peere S3/GCS.
Nga kaiarahi e pa ana¶
- Kopae tūmau 100% ki tonu (0 paita kore utu): te putunga rangitaki WAL pātengi raraunga, te rakau B kua kino, me te whakaora mokowā-kore
rm -rfraneiShift+Mukui runga i te Linux, macOS, Windows ranei: te panui-anake te remount me te whakairo inode- Mīhini mariko VMDK, VHDX, me te qcow2 te pirau mekameka hopu whakaahua me te whakaoranga hypervisor
- Ko te tuku ohorere a Git mo nga taviri AWS, te konae
.env, te kī SSH tūmataiti ranei ki te repo GitHub tūmatanui - Nga raruraru manaaki Tukutuku, te hekenga, me te whakaora whakaora
- Nga taapiri rorohiko, te rautaki 3-2-1, me te whakaora i nga aitua
- Ko nga nama ohorere ma te whenua
Nga puna me nga aratohu whai mana¶
- Kubernetes Tuhinga Whaimana (
kubernetes.io/ Cloud Native Computing Foundation): Te Ahanoa Rokiroki kei te Whakamahi Parenga (kubernetes.io/pvc-protectionWhakaoti), Reclaiming PersistentVolumes (Retainvs.Delete), me te MahietcdHuihuinga mo Kubernetes (etcdctl snapshot restore). - Docker Inc. (
docs.docker.com): Whakahaere Raraunga ki Docker: Rōrahi, Herea Maunga,overlay2Hangahanga Atekōkiri Rokiroki, me te Tohutoro Tapahi. - Te Whakahaere Kernel Linux (
kernel.org): *Ko te/procPūnaha Kōnae (procfsWhakatuwheratia te Whakaora Whakaahuatanga Kōnae ma/proc/<pid>/fd/) me Ext4 Tuhinga Tuhitaka *. ext4magicTuhinga Kaupapa (ext4magic.sourceforge.net): Forensic Inode and Journal Transaction Recovery on Ext3/Ext4 Filesystems.- National Institute of Standards and Technology (
NIST) (Te Hononga o Amerika): *Te Whakaputa Motuhake 800-190: Te Aratohu Haumaru Ipu Taupānga me te Whakaora Mai.