I-Server Disk 100% Egcwele (0 Amabhayithi Amahhala): Ithola Isizindalwazi Esiphahlazekile Nini rm futhi apt Yehlulekile Ngokushoda kwama-Inodes/Isikhala¶

Ukuthola ukuthi iseva ye-Linux yokukhiqiza isifikile 100% ukusetshenziswa kwediski (0 bytes free) noma 100% ukusetshenziswa kwe-inode (df -i) ngenkathi usebenzisa i-PostgreSQL, i-MySQL/InnoDB, i-MariaDB, noma isizindalwazi se-SQLite esebenzayo ingenye yezindlela zokuhluleka eziyingozi kakhulu ekuphathweni kwezinhlelo. Kuma-byte ayiziro noma ama-inode wamahhala, inkinga iqubuka ngokushesha: i-kernel yesistimu yokusebenza ngeke isakwazi ukudala amasokhethi esizinda se-Unix /tmp noma /var/run, bash yehluleka ukuqeda ithebhu ngoba ayikwazi ukubhala amapayipi egobolondo esikhashana, apt noma dnf ayikwazi ukugijima ukuhlanza amaphakheji ngoba abaphathi bephakheji badinga amafayela wokukhiya wesikhashana, futhi injini yakho yedathabhesi iphahlazeka phakathi Kwelogi Yokubhala Phambili (WAL) indawo yokuhlola noma yenqaba ukuvala ngokuhlanzekile ngoba ayikwazi ukunikeza ibhulokhi elilodwa lamakhilobhayithi angu-4 (0.004 megabyte) endaweni yayo. .pid noma vala ifayela. Okubi nakakhulu, abaphathi abathukile bavame ukugijima rm -rf /var/log/*.log kumafayela okungena e-multi-gigabyte futhi ubuke ngokuthuthumela njenge df -h isabika 100% Use% (0 Avail), noma bacisha ngendlela ebulalayo pg_wal (pg_xlog) noma ib_logfile0 ukukhulula isikhala, konakalise unomphela isizindalwazi ngale kokutholwa kokwenziwe. Uma usule ngephutha amafayela esizindalwazi abalulekile ngesikhathi somzamo wokuhlanza, yima ngokushesha bese ushintshela kokuthi ngengozi rm -rf noma Shift-Delete read-only remount kanye ne-inode carving; uma i-hardware yakho yesitoreji engaphansi iphonsa amaphutha e-I/O noma izikhiyele ngokumelene nokubhala, thintana I-SSD ne-NVMe imodi yokufunda kuphela kungazelelwe kanye ne-TRIM ekhubaza ukutakula noma IZwi elonakele, i-Excel, i-PDF, nokutholwa kwefayela lesizindalwazi. Uma lokhu kucima kuthinta amasistimu okuphepha kwempilo yasesibhedlela, ukuthunyelwa kwezimo eziphuthumayo, noma izinsiza zomphakathi ezibalulekile, shayela inombolo yakho yasendaweni yezimo eziphuthumayo (911 e-United States naseCanada, 112 kwi-European Union, 999 e-United Kingdom) noma uthintane nemibhalo yethu yomhlaba wonke ye izinombolo eziphuthumayo.

Yenza lokhu manje¶

1. UNGASUSI pg_wal/, pg_xlog/, pg_multixact/, ib_logfile*, undo_*, noma -wal / -shm amafayela okwenza isikhala: Ukususa Ilogi Yokubhala Phambili noma phinda/hlehlisa amafayela okungena e-InnoDB kuyilapho isizindalwazi siphahlazeka noma singahloliwe kucekela phansi irekhodi okuwukuphela kwalo lamakhasi angazibophezeli nangaphakathi endizeni. I-database enediski egcwele ingabuyiselwa ngamaphesenti angu-100 uma isikhala sesikhululiwe; isizindalwazi amafayela we-WAL asuswe ngaso mathupha rm uzobhekana ne-B-tree engenakuhlehliswa kanye nenqwaba yenkohlakalo.
2. Thola emizuzwaneni emi-5 ukuthi uphumile kumabhulokhi ediski (df -h), ngaphandle kwama-inode esistimu (df -i), noma ubambe amafayela angaxhumekile avuliwe (lsof +L1):
3. Gijima df -h ukuhlola umthamo wokuvimba futhi df -i ukuhlola umthamo we-inode. Iseva enamagigabhayithi angu-50 (amabhayithi ayizigidi eziyizinkulungwane ezingu-53.6) wesikhala samahhala sediski isazophonsa No space left on device (ENOSPC) uma df -i imibukiso IUse% = 100% (imvamisa okubangelwa izigidi zeseshini encane, inqolobane, noma amafayela we-spool we-imeyili).
4. Uma df -h imibukiso 100% esetshenzisiwe (0 Avail) ngisho nangemva kokuba usugijima rm kumafayili amakhulu welogi, sebenzisa sudo lsof +L1. Noma yiliphi ifayela elisuswe nge rm ngenkathi nginx, rsyslog, docker, postgres, noma mysqld isabambe isichazi sefayela elivuliwe siyaqhubeka nokusebenzisa amaphesenti angu-100 amabhulokhi ediski kuze kube yilapho inqubo iqala kabusha noma unciphisa isichazi ngokuqondile.
5. Amagigabhayithi amahhala esikhala ngomzuzwana ongu-1 ngaphandle kokuqalisa kabusha amademoni ngokunqamula amafayela avuliwe endaweni (: > /path/to/logfile noma : > /proc/<pid>/fd/<n>):
6. Esikhundleni sokuthi rm /var/log/syslog (okushiya amabhulokhi ekhiyiwe isigawuli esigijimayo), uziro ifayela endaweni kusetshenziswa igobolondo eliqondisa kabusha elingenalutho: sudo sh -c ': > /var/log/syslog' noma sudo truncate -s 0 /var/log/syslog.
7. Uma usuvele wagijima rm efayeleni elikhulu lokungena futhi df -h ayizange ilahle, thola i-ID yenqubo yayo (PID) kanye nesichazi sefayela (FD) phakathi sudo lsof +L1 futhi uyinciphise iqonde ku-kernel procfs interface: sudo sh -c ': > /proc/<PID>/fd/<FD>'. Isikhala sizobuyela kuso df -h khona manjalo.
8. Vula igumbi lokuphefumula eliphuthumayo elisheshayo ext4 / ext3 ngokwehlisa iphesenti le-root block block (tune2fs -m 1): Ngokuzenzakalelayo, ext4 igcina amaphesenti angu-5 ayo yonke ingxenye (25 gigabytes kwe a 500 gigabyte disk) ngokukhethekile UID 0 (root), okusho ukuthi izinsiza ezingezona izimpande njenge postgres (UID 109), mysql, futhi www-data shaya No space left on device ukusetshenziswa kwamaphesenti angama-95. Phinda uthole amaphesenti angu-80 alokho kugodla ku-inthanethi ngomzuzwana ongu-1 ngaphandle kwesikhathi sokuphumula esiziro ngokugijima sudo tune2fs -m 1 /dev/sda1 (noma idivayisi yakho ye-root block kusuka findmnt /).
9. Vacuum systemd-journald izingodo kanambambili nezilondolozi zephakheji ezindala ngokuphephile: Gijimani sudo journalctl --vacuum-size=50M (ezungezisa ngokuphephile futhi isuse kanambambili /var/log/journal izingobo zomlando kuze kufike kumamegabhayithi angama-50) futhi sudo apt-get clean (noma sudo dnf clean all), esusa okulandiwe .deb / .rpm ezinqolobaneni /var/cache/apt/archives/ ngaphandle kokudinga ukubhalela kusizindalwazi se-dpkg.
10. Uma okungenani igigabhayithi e-1 ukuya ku-5 gigabytes (amabhayithi ayizigidi eziyizinkulungwane ezingu-1.07 kuya ku-5.36) wesikhala sesikhululekile, letha isizindalwazi ngemodi yomsebenzisi oyedwa noma yokutakula ukuze uqedele indawo yayo yokuhlola ye-WAL ngaphambi kokuvumela ithrafikhi yeklayenti: Vumela i-PostgreSQL, MySQL/InnoDB, noma i-SQLite iphinde idlale ilogi yayo ye-Writ-Ahead (WAL), flushe amakhasi angcolile uwafake kumafayela edatha ayinhloko, bese uphinda usebenzise izingxenye ze-WAL eziqumbile ngaphambi kokuvula kabusha uhlelo lwakho lokusebenza lokuvikela noma iseva yewebhu.

Isihlahla sesinqumo: iseva ye-zero-space kanye ne-triage yesizindalwazi esaphukile¶

Sebenzisa lesi sihlahla sesinqumo ukuze ubone ukuthi kungani isistimu yokusebenza ibika ENOSPC (No space left on device) bese ukhetha indlela yokubuyisela engacekeli phansi:

flowchart TD
    Start["Ukuphonsa Iseva noma Isizindalwazi: Asikho isikhala esisele kudivayisi (ENOSPC)"] --> CheckDF["Gijimani: df -h KANYE df -i"]

    CheckDF --> DiagType{"Ibika ini i-df?"}

    DiagType -->|"I-df -h Ibonisa ukusetshenziswa okungu-100% (0 Iyatholakala) KANYE ne-df -i <95%"| BlockFull["Vimba Ukukhathala: Hlola ukuthi i-du -shx / Ifanisa i-df -h"]
    DiagType -->|"I-df -h Ibonisa Ama-GB Amahhala KODWA df -i Ibonisa 100% IUse"| InodeFull["Ukukhathala kwe-Inode: Izigidi Zamafayela Amancane 0-to-4KB"]
    DiagType -->|"Kokubili i-df -h kanye ne-df -i Bonisa < 90% Esetshenzisiwe"| RamOrQuota["Umkhawulo Ofihliwe: Hlola i-df -h /tmp (i-tmpfs RAM igcwele), i-Docker Overlay2, noma i-XFS/ext4 Disk Quotas"]

    BlockFull --> GhostCheck{"Ingabe Usuvele Usebenzisa I-rm Kumafayela Amakhulu Ngaphandle Kokulahla I-df?"}
    GhostCheck -->|"Yebo: du Incane Kakhulu Kune-df"| UnlinkedFD["Gijimani: sudo lsof +L1 -> Truncate Ghost File nge: > /proc/PID/fd/FD"]
    GhostCheck -->|"Cha: Amafayela Angempela Agcwalisa Idiski"| EmergencySpace["Dala Igumbi Lokuphefumula Ngokushesha (1 GB ukuya ku-5 GB)"]

    EmergencySpace --> Ext4Check{"Ingabe i-Filesystem ext4 / ext3?"}
    Ext4Check -->|"Yebo"| Tune2fs["Qalisa: sudo tune2fs -m 1 /dev/root_dev (Ikhulula Ngokushesha u-4% weDiski yomsebenzisi we-DB Ongeyena Umsuka)"]
    Ext4Check -->|"Cha (XFS / Btrfs / ZFS)"| TruncLogs["I-Truncate Logs In-Place (: > /var/log/syslog) futhi Run journalctl --vacuum-size=50M"]

    InodeFull --> InodeClean["Thola Uhlu Lwemibhalo Olunezigidi Zamafayela Usebenzisa thola / -xdev -printf '%h\n' | hlunga | uniq -c | hlunga -k 1 -r | ikhanda -20 -> Susa ngokuthi thola -delete"]

    UnlinkedFD --> DBRecover["Qhubekela ku-Database Engine WAL & B-Tree Recovery"]
    Tune2fs --> DBRecover
    TruncLogs --> DBRecover
    InodeClean --> DBRecover
    RamOrQuota --> DBRecover

    DBRecover --> WhichDB{"Iyiphi Injini Yesizindalwazi Ephahlazekile?"}
    WhichDB -->|"I-PostgreSQL"| PGFlow["Gcina i-pg_wal Iqinile -> Qala ama-Postgres ukuze udlale kabusha i-WAL -> Qalisa I-CHECKPOINT; -> Drop Stale Replication Slots"]
    WhichDB -->|"I-MySQL / MariaDB (InnoDB)"| MyFlow["Gcina ib_logfile* Iqinile -> Qala i-mysqld -> Hlanza Izingodo Ezinambambili nge-PURGE BINARY LOGS -> Lungiselela Amathebula"]
    WhichDB -->|"I-SQLite3"| SqFlow["Yenza isipele .db + .db-wal + .db-shm -> Qalisa i-PRAGMA wal_checkpoint(TRUNCATE) noma i-sqlite3 broken.db '.recover'"]

Isigaba 1: Kungani imiyalo evamile (rm, apt, tab-completion) yehluleka kumabhayithi angu-0 mahhala¶

Lapho ukwahlukanisa kwe-Linux kushaya amaphesenti angu-100 ukuvimba noma ukukhathala kwe-inode, imiyalo yokulungisa evamile ihluleka ngezindlela ezingalindelekile: - Iyacindezela Tab phakathi bash ukujikijela cannot create temp file for here-document: No space left on device: Yesimanje bash-completion imibhalo yakha amafayela esikhashana ku /tmp. Uma /tmp ihlala empandeni / ukwahlukanisa esikhundleni sika-a tmpfs Ukukhweza kwe-RAM, ukuqedwa kwegobolondo kanye heredocs (<<EOF) ehluleka. Lungisa lokhu ngokushesha ngokukhomba TMPDIR ukuze wabelane ngememori ye-RAM:

export TMPDIR=/dev/shm

sudo find /var/lib/php/sessions -type f -delete

Izimbangela ezi-5 ezihlukene ze No space left on device (ENOSPC)¶

Ngaphambi kokuthi uthinte noma yimaphi amafayela esizindalwazi, qhathanisa lezi zimo ezinhlanu ukuze uqinisekise ukuthi iyiphi insiza ephelile ngempela:

Isigaba sesi-2: Ukubuyisela isikhala sediski ngokuphephile ngaphansi kwamasekhondi angu-60¶

Landela lezi zinyathelo eziqondile ukuze uthole kabusha indawo ephakathi kwamamegabhayithi angu-500 namagigabhayithi angu-50 (amabhayithi ayizigidi eziyizinkulungwane ezingu-0.53 kuya kwezingu-53.6) ngaphandle kokonakalisa izinqubo ezisebenzayo:

Isinyathelo 1: Nciphisa vula amafayela asusiwe ("ghost") ngokusebenzisa /proc/<pid>/fd/¶

Uma wena noma umbhalo we-logrotate ozenzakalelayo usuwasusile amafayela amakhulu kodwa isikhala sediski asibuyiswanga, hlola ithebula lefayela elivuliwe elingaxhunywanga le-kernel:

sudo lsof +L1

COMMAND    PID     USER   FD   TYPE DEVICE    SIZE/OFF NLINK    NODE NAME
mysqld    1942    mysql    6u   REG  259,1 34896609280     0  918273 /tmp/ibXz91aB (deleted)
rsyslogd   612   syslog    7w   REG  259,1 18253611008     0  131092 /var/log/syslog.1 (deleted)

sudo sh -c ': > /proc/612/fd/7'

Isinyathelo sesi-2: Thola futhi ext4 amabhlogo agodliwe ku-inthanethi nge tune2fs¶

Kunoma yikuphi ext4 noma ext3 kusistimu yefayela, hlola inani lamanje lebhulokhi ebekelwe:

df -h /
sudo tune2fs -l /dev/nvme0n1p1 | grep -i "reserved block"

sudo tune2fs -m 1 /dev/nvme0n1p1

Isinyathelo sesi-3: Nciphisa amalogu ombhalo asebenzayo bese uvala journald¶

Thola amafayela amakhulu kunawo wonke ohlelweni lwamafayela lwempande (ukuhlala ohlelweni lwefayela olufanayo nge -xdev ngakho ungachithi isikhathi ukuskena /proc noma izikhwebu zangaphandle):

sudo find / -xdev -type f -size +100M -exec ls -lh {} + 2>/dev/null | sort -k 5 -hr | head -n 25

sudo truncate -s 0 /var/log/nginx/error.log
sudo truncate -s 0 /var/lib/docker/containers/*/*-json.log

sudo journalctl --vacuum-size=50M

Isigaba sesi-3: Ukubuyisela i-PostgreSQL ngemuva kokuphahlazeka okugcwele kwediski okungu-100% (pg_wal isisu)¶

Lapho iseva ye-PostgreSQL iphelelwa isikhala sediski, iyangena PANIC: could not write to file "pg_wal/xlogtemp.1234": No space left on device, ikhipha ngokushesha izinqubo zengane yomphathi weposi, futhi yenqaba ukuqala.

Kungani pg_wal (pg_xlog ku-PostgreSQL 9.6 nangaphezulu) igcwalisa yonke idiski¶

Ngokusebenza okujwayelekile, i-PostgreSQL ibhala lonke ushintsho kumafayela engxenye ye-16 megabyte (16.7 million byte) ngaphakathi. $PGDATA/pg_wal/ futhi iwasebenzise kabusha ngemva kokunye CHECKPOINT kanye okweqile max_wal_size. Nokho, i-PostgreSQL inqatshelwe ukususa noma ukugaywa kabusha endala pg_wal amafayela ngaphansi kwezimo ezintathu: 1. Isikhala esiyintandane noma esisalelayo Esibonakalayo/Esizwakalayo Sokuphindaphinda: Iseva eyikhophi (standby), isixhumi se-Debezium/Kafka, noma pg_receivewal iklayenti inqanyuliwe ezinsukwini ezedlule, kodwa i-slot yayo yokuphindaphinda (pg_replication_slots) kusenjalo active = false kokuyinhloko, iyala i-PostgreSQL ukuthi igcine yonke ingxenye eyodwa ye-WAL ekhiqizwe kusukela ekunqanyulweni. 2. Ukwehluleka archive_command: archive_mode = on ivuliwe postgresql.conf, kanye nombhalo oyisipele (pgBackRest, WAL-G, Barman, noma aws s3 cp) ibuyisela ikhodi yokuphuma engeyona uziro (isibonelo, ukhiye we-S3 ophelelwe yisikhathi noma umsingathi oyisipele ongafinyeleleki). I-PostgreSQL igcina yonke ingxenye ye-WAL engagciniwe ingaphakathi pg_wal/ kuze archive_command izimbuyiselo 0. 3. I-disk-full deadlock: pg_wal lakhula kwaze kwaba 0 bytes wahlala /var/lib/postgresql, futhi manje i-PostgreSQL ayikwazi ngisho nokuqala ukulahla i-slot yokuphindaphinda noma ukuqedela indawo yokuhlola ngoba ukuqalisa kabusha kudinga ama-megabytes amaningana esikhala samahhala sediski ukuze kubhalwe amafayela wesimo sesikhashana.

Isinyathelo ngesinyathelo sokuhlenga i-PostgreSQL zero-space (I-Zero Data Loss)¶

1. Dala igigabhayithi engu-1 ukuya ku-5 (amabhayithi ayizigidi eziyizinkulungwane ezingu-1.07 kuya ku-5.36) wesikhala sesikhashana ngaphandle pg_wal/:
2. Ukusetshenziswa kokuqala sudo tune2fs -m 0 /dev/<partition> futhi unciphise amalogi wombhalo wesistimu/PostgreSQL (sudo truncate -s 0 /var/lib/postgresql/*/main/log/*.log noma /var/log/postgresql/*.log).
3. Kuthiwani uma isizindalwazi sinokuhlukaniswa kwaso okuzinikele ngamalogi wombhalo onguziro kanye ne-XFS (so tune2fs ayitholakali)? Hambisa u-10 uye ku-20 kokuthi kudala .ready noma amafayela angasebenzi aphuma kuhla lwemibhalo okungeyona eye-WAL, noma hambisa kabusha pg_wal uhla lwemibhalo okwesikhashana kwenye ukwahlukanisa ngesikhala samahhala usebenzisa isixhumanisi esingokomfanekiso:

   # ONLY while PostgreSQL is completely stopped (verify with: ps aux | grep postgres)
   sudo mkdir -p /mnt/extra_space/pg_wal_rescue
   sudo chown -R postgres:postgres /mnt/extra_space/pg_wal_rescue
   sudo mv /var/lib/postgresql/16/main/pg_wal/* /mnt/extra_space/pg_wal_rescue/
   sudo rmdir /var/lib/postgresql/16/main/pg_wal
   sudo ln -s /mnt/extra_space/pg_wal_rescue /var/lib/postgresql/16/main/pg_wal
   sudo chown -h postgres:postgres /var/lib/postgresql/16/main/pg_wal
   

4. Uma archive_command iyahluleka futhi ivimba ukugaywa kabusha kwe-WAL, idlule okwesikhashana postgresql.conf: Uma i-S3/backup target yakho iphansi futhi pg_wal/archive_status/ iqukethe izinkulungwane .ready amafayela, hlela postgresql.conf (archive_command = '/bin/true') kuphela njengesinyathelo esiphuthumayo sokugcina sokumisa idiski ukuthi ingafi, wazi ukuthi kufanele uthathe okusha pg_basebackup ngokushesha ngemva kokululama ngoba uchungechunge lwengobo yomlando ye-WAL luzoba negebe.
5. **Qala i-PostgreSQL bese uhlola ngokushesha izikhala zokuphindaphinda **: Qala i-PostgreSQL (sudo systemctl start postgresql) bese uxhuma nge sudo -u postgres psql:

   SELECT slot_name, plugin, slot_type, active,
          pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal
   FROM pg_replication_slots;
   

6. Yehlisa noma yisiphi isikhala sokuphindaphinda esifile/esingasebenzi esigoqa amagigabhayithi e-WAL: Uma ubona indawo engasebenzi (active = f) ephethe 80 gigabytes of retained_wal:

   SELECT pg_drop_replication_slot('name_of_dead_slot');
   

7. Phoqa ngokushesha CHECKPOINT ukuvuselela okuqumbile pg_wal izingxenye:

   CHECKPOINT;
   

   Isikhathi CHECKPOINT; iqeda ngemuva kokulahla i-slot efile, i-PostgreSQL izosusa ukuxhumanisa ngokuzenzakalelayo noma isebenzise kabusha ama-gigabytes wamafayela engxenye ye-16 megabyte endala ku- pg_wal/, ibuyisela ukusetshenziswa kwediski yakho kokujwayelekile.
8. Vimbela ukufa kwe-slot disk yesikhathi esizayo: Setha max_slot_wal_keep_size phakathi postgresql.conf (etholakala ku-PostgreSQL 13+) kukepisi ephephile (njenge max_slot_wal_keep_size = 20GB) ngakho-ke i-PostgreSQL yenza i-slot ye-replica efile ingasebenzi esikhundleni sokuphahlazeka kusizindalwazi esiyinhloko ngamabhayithi angu-0 mahhala.

Isigaba 4: Ukubuyisela i-MySQL / MariaDB (InnoDB) ngamabhayithi angu-0 mahhala¶

Lapho i-MySQL noma i-MariaDB ishaya 100% ukusetshenziswa kwediski (/var/lib/mysql), i-InnoDB ithengisa konke okubhalayo noma ukuphahlazeka ngakho [ERROR] [MY-012640] [InnoDB] Error number 28 means 'No space left on device'.

Ukukhomba okugcwele /var/lib/mysql¶

Gijima sudo du -sh /var/lib/mysql/* | sort -hr | head -n 15 ukuze ubone ukuthi iyiphi kulezi zigebengu ezine ezidle idiski: 1. Amalogi kanambambili Anganqunyelwe (binlog.000001, mysql-bin.000042): Ngokuzenzakalelayo ku-MySQL 8.0+, ukugawulwa kwemithi kanambambili kunikwe amandla nokugcinwa kwezinsuku ezingama-30 (binlog_expire_logs_seconds = 2592000), engakhiqiza kalula amakhulukhulu amagigabhayithi emicimbi yerowu ngesikhathi sokungenisa ngobuningi. 2. Umbuzo obalekile ubhala amatafula esikhashana amakhulu (#sql* noma ibtmp1): Okukhulu GROUP BY noma ORDER BY umbuzo ugcwalisiwe ibtmp1 noma /tmp/#sql_* kuze kube yilapho idiski ishaya amabhayithi angu-0. 3. ** I-Monolithic ibdata1 i-system tablespace bloat**: Uma innodb_file_per_table ivaliwe noma okwenziwa isikhathi eside kuvimbele uhlu lomlando we-InnoDB (hlehlisa amalogi ngaphakathi ibdata1 noma undo_001) kusuka ekuhlanzweni. 4. Ilogi yombuzo ongasheshi we-MySQL noma ilogi yephutha (/var/log/mysql/mysql-slow.log).

Isinyathelo ngesinyathelo MySQL / MariaDB zero-space rescue¶

1. UNGALOKOTHI ususe ibdata1, ib_logfile0, ib_logfile1, #innodb_redo/*, noma undo_001 nge rm: Ukususa noma yimaphi kulawa mafayela kwenza wonke .ibd ithebula kusizindalwazi sakho alifundeki.
2. Isikhala sokuqala samahhala (amamegabhayithi angama-500 kuye kwangamagigabhayithi angu-2):
3. Thinta /var/log/mysql/error.log futhi mysql-slow.log nge sudo truncate -s 0 /var/log/mysql/*.log.
4. Vuliwe ext4, gijima sudo tune2fs -m 1 /dev/<partition>.
5. Uma mysqld okwamanje imisiwe futhi /var/lib/mysql/ibtmp1 (indawo yetafula yesikhashana engacindezelwe) inkulu, ibtmp1 idalwe kabusha ngokuzenzakalelayo ekuqaleni, noma ungahambisa amadala amabili noma amathathu amalogi kanambambili (binlog.000001, binlog.000002 - akukaze kube yi-binlog enenombolo ephezulu kakhulu esebenzayo) ukuze /dev/shm okwesikhashana kanjalo mysqld ingakwazi ukuqala.
6. Qala i-MySQL bese uhlanza izingodo kanambambili ngokuphepha usebenzisa i-SQL (PURGE BINARY LOGS): Kungani kufanele usebenzise i-SQL esikhundleni sokuthi rm /var/lib/mysql/binlog.*? Ngoba i-MySQL ilandelela lonke igama lefayela lokungena kanambambili binlog.index. Uma ususa izingodo kanambambili nge rm ngaphandle kokubuyekeza binlog.index, i-MySQL izonqaba ukuqala noma ukuphindaphinda kuzophuka. Kanye mysqld iqala, ngena (sudo mysql) bese ugijima:

   SHOW BINARY LOGS;
   PURGE BINARY LOGS BEFORE NOW() - INTERVAL 1 DAY;
   

   (Noma hlanza yonke into kuze kufike kufayela lokungena elisebenzayo eliboniswe ngezansi kokuthi SHOW BINARY LOGS; usebenzisa PURGE BINARY LOGS TO 'binlog.000085';)
7. Setha usayizi welogi kanambambili unomphela kanye nekepisi yokugcina phakathi /etc/mysql/mysql.conf.d/mysqld.cnf:

   [mysqld]
   binlog_expire_logs_seconds = 259200   # Keep only 3 days (259,200 seconds)
   max_binlog_size = 256M
   innodb_temp_data_file_path = ibtmp1:12M:autoextend:max:5G
   

   I-Capping ibtmp1 kwe max:5G iqinisekisa ukuthi isikhohlakali esingakhonjwanga JOIN noma ORDER BY umbuzo wehluleka nge-a table is full iphutha esikhundleni sokwandisa ibtmp1 kuze kube yilapho yonke iseva ye-Linux ifa ngamabhayithi angu-0 mahhala.

Isigaba sesi-5: Ukubuyisela isizindalwazi se-SQLite3 esonakele (database or disk is full / SQLITE_FULL / malformed)¶

Izinhlelo zokusebenza ezishumekiwe, iziqukathi ze-Docker (Nextcloud, Vaultwarden, Home Assistant, Plex, Grafana, Gitea), kanye nezinsizakalo zeselula/zedeskithophu zithembele ku-SQLite3. Lapho idiski ishaya amabhayithi angu-0 mahhala kuyilapho i-SQLite iphakathi kwephephabhuku elibuyisela emuva noma i-Write-Ahead Log (-wal) indawo yokuhlola, uzohlangabezana nezimo eziphuthumayo ezimbili: 1. Kukhulu .db-wal ifayela bloat (the .db-wal ifayela selikhule laba ngu-10 gigabytes ngenkathi i .db ifayela lingamamegabhayithi angu-200 kuphela, ngoba abafundi asebenesikhathi eside bavimbele indawo yokuhlola ye-WAL, futhi manje idiski isigcwele ngamaphesenti angu-100). 2. RuntimeError: database disk image is malformed (SQLITE_CORRUPT) okubangelwa i-hardware ephukile fsync(), isiqukathi esibulawe phakathi nokubuyiselwa emuva ngo-0 bytes mahhala, noma othile osusa ifayela -wal noma -journal ifayela ngesandla.

Isinyathelo 1: Ukunciphisa ukuqunjelwa .db-wal ifayela nokusula i-SQLite lapho isikhala sediski siseduze noziro¶

1. Misa isiqukathi sohlelo lokusebenza noma isevisi ebambe isizindalwazi se-SQLite sivuliwe (sudo systemctl stop app noma docker stop container_name).
2. Kopisha app.db, app.db-wal, futhi app.db-shm ku-partition noma idrayivu yangaphandle enesikhala esiningi samahhala (noma khulula 2x usayizi we app.db kudiski yendawo kuqala).
3. Vula i-database nge- sqlite3 CLI futhi uphoqelele indawo yokuhlola ye-WAL efinyeziwe, egoqa wonke amafreyimu azibophezele ukusuka kuwo app.db-wal phakathi app.db futhi iyashwabana app.db-wal ku-0 amabhayithi*:

   sqlite3 /path/to/app.db "PRAGMA wal_checkpoint(TRUNCATE);"
   

4. Qinisekisa i-B-tree kanye nohlu lwamahhala lobuqotho besakhiwo:

   sqlite3 /path/to/app.db "PRAGMA integrity_check;"
   

   Uma iphrinta ok, isizindalwazi sakho sinempilo engamaphesenti angu-100.
5. Kanjani VACUUM isizindalwazi esikhulu se-SQLite lapho idiski yendawo ingenaso isikhala esanele samahhala: Okujwayelekile VACUUM; idinga kufika ku-2x usayizi wesizindalwazi esikhaleni samahhala sediski endaweni efanayo futhi /var/tmp. Kunalokho, sebenzisa VACUUM INTO (yethulwe ku-SQLite 3.27.0) ukuze ubhale ikhophi esanda kuhlanganiswe, ehlukanisiwe ngokuqondile kudiski ekhweziwe ehlukile noma ukwabelana kwenethiwekhi:

   sqlite3 /path/to/app.db "VACUUM INTO '/mnt/external_usb/app_compacted.db';"
   

Isinyathelo sesi-2: Ukuhlenga isihlahla esingu-B esonakele (database disk image is malformed) nge .recover¶

Uma PRAGMA integrity_check; imibiko row 412 missing from index, btreeInitPage() returns error code 11, noma database disk image is malformed:

1. Qala uzame ukwakha kabusha izinkomba zesibili eziphukile (REINDEX): Ngokuvamile idatha yethebula langempela (table B-trees) iphelele ngamaphesenti angu-100 futhi inkomba yesibili kuphela yesihlahla esingu-B sonakala lapho idiski igcwala. Gijima:

   sqlite3 /mnt/external_usb/copy_of_app.db "REINDEX;"
   sqlite3 /mnt/external_usb/copy_of_app.db "PRAGMA integrity_check;"
   

   Uma PRAGMA integrity_check; manje uyabuya ok, uthole amaphesenti angu-100 esizindalwazi sakho ngemigqa elahlekile eyiziro.
2. Uma REINDEX yehluleka, sebenzisa isithwebuli se-forensic B-tree (.recover): Ngokungafani nabadala .dump umyalo (omisa ukuthekelisa ithebula lapho ishaya isikhombi sesihlahla sika-B esonakele), .recover (yakhiwe ngaphakathi sqlite3 inguqulo 3.29.0+) yeqa izikhombi zesihlahla se-B ezonakele ngokuphelele, iskena wonke amakhasi aluhlaza angamakhilobhayithi angu-4 efayela ngokulandelana ukusuka ku-byte 0 ukuya ku-EOF, yakha kabusha yonke imigqa efundekayo, futhi ibeka noma imaphi amarekhodi ayizintandane endaweni lost_and_found ithebula:

   sqlite3 /mnt/external_usb/copy_of_app.db ".recover" | sqlite3 /mnt/external_usb/repaired_app.db
   

3. Qinisekisa imininingwane egciniwe esanda kwakhiwa kabusha:

   sqlite3 /mnt/external_usb/repaired_app.db "PRAGMA integrity_check;"
   

   Hlola ukuthi a lost_and_found ithebula lakhiwe (sqlite3 /mnt/external_usb/repaired_app.db ".tables") ukuhlola noma yimiphi imigqa eyizintandane etholwe emakhasini angaxhunywanga. Ukuze uthole imininingwane yolwazi ebethelwe ye-SQLCipher (efana nezipele ze-Signal noma ze-WhatsApp), bheka umhlahlandlela wethu ozinikele kokuthi i-SQLite ebethelwe, iSignal, nenkohlakalo eyisipele yengxoxo ye-WhatsApp kanye nokukhipha ukhiye wendawo.

Isigaba sesi-6: Isimo esiphuthumayo "Ifayela le-Tripwire Ballast" kanye nohlu lokuhlola lokuvikela lokuqapha¶

Uma iseva yakho nesizindalwazi sekubuyile ku-inthanethi, sebenzisa lezi zivikelo ezine ngokushesha ukuze ungaphinde ubanjwe ngamabhayithi angu-0 mahhala futhi:

• Dala igigabhayithi engu-1 (1,073,741,824 byte) Ifayela Le-Emergency Ballast kuwo wonke ama-partition wesizindalwazi: Qalisa sudo fallocate -l 1G /var/lib/EMERGENCY_DELETE_ME.ballast. Uma iseva ike yashaya 100% idiski igcwele, iyasebenza sudo rm /var/lib/EMERGENCY_DELETE_ME.ballast ikhulula igigabhayithi e-1 yamabhulokhi ahlanganayo nge-millisecond engu-1.
• Ikhanda systemd-journald kanye namalogi esiqukathi se-Docker unomphela: Setha SystemMaxUse=200M phakathi /etc/systemd/journald.conf futhi ulungiselele /etc/docker/daemon.json nge "log-opts": {"max-size": "20m", "max-file": "3"}.
• ** Zihlukanise /var/log, /tmp, futhi /var/lib/postgresql emiqulwini ehlukene ye-LVM**: Ukugcina /var/log futhi /tmp ehlukene namavolumu esizindalwazi kuqinisekisa ukuthi irekhodi lewebhu elibalekayo alilokothi liphahlaze isizindalwazi noma likhiyele ngaphandle ukungena kwe-SSH.
• Isexwayiso kokuthi KOKUBILI df -h (> 80%) KANYE df -i (> 80%) plus pg_replication_slots lag: Lungiselela ukuqapha (node_filesystem_avail_bytes futhi node_filesystem_files_free) ukuxwayisa ngaphambi kokuthi ukusetshenziswa kweqe amaphesenti angama-85.

Izinombolo zezimo eziphuthumayo zesifunda kanye nezinqubomgomo ze-ejensi¶

Ngenkathi idiski yeseva egcwele ngokuvamile iyisimo esiphuthumayo sangaphakathi se-IT, ukucisha okunomthelela ekunakekelweni kwezempilo kwe-Electronic Health Record (EHR) isizindalwazi, ukuthunyelwa kwezimo eziphuthumayo (CAD), i-telemetry yamanzi/amandla kamasipala, noma ukutholakala kwedatha yomuntu siqu esemthethweni kungase kudinge ukukhuphuka okusemthethweni okuphuthumayo:

• I-United States ne-Canada: Dayela 911 uma ukuphahlazeka kweseva kukhubaza ukuqapha komtholampilo noma ukuthunyelwa, futhi kusebenze izinqubo zesikhathi sokuphumula. Bika ukuhlaselwa kwengqalasizinda ebalulekile noma ukuhlaselwa kwe-ransomware ku-CISA (1-844-Say-CISA / cisa.gov/report)* noma Canadian Center for Cyber Security (1-833-CYBER-88).
• United Kingdom: Dayela 999 ngezinsongo zempilo noma izinhlelo ezibucayi ze-NHS. Ngaphansi kwe-UK GDPR kanye ne-NIS Regulations, hlola ukuthi ukulahlekelwa ukutholakala kwedatha egciniwe unomphela kudinga isaziso ku-ICO (0303 123 1113) emahoreni angama-72.
• I-European Union: Dayela 112 kuwo wonke amazwe angamalungu e-EU. Amabhizinisi abalulekile ngaphansi kwe-NIS2 Directive kufanele azise i-CSIRT kazwelonke phakathi kwamahora angu-24 okunqamuka okukhulu, kanye nokulahlekelwa kwedatha yomuntu siqu unomphela ngaphandle kokuthola isipele kubangela ukubika kwe-GDPR I-Athikili 33 phakathi namahora angu-72.
• I-Australia ne-New Zealand: Dayela 000 e-Australia noma 111 eNew Zealand; xhumana ne-ACSC (1300 CYBER1)* ezehlakalweni zengqalasizinda.
• I-India ne-Asia-Pacific: Dayela 112 impendulo ephuthumayo; bika ukucisha okubangelwa i-cyber ku-CERT-In (incident@cert-in.org.in)* ngaphakathi kwewindi eliyisibopho lamahora angu-6.

Imibuzo evame ukubuzwa¶

Kungani? df -h hlala ku-100% (0 Avail) ngisho nangemva kokuba ngisuse ifayela lokungena elingu-50 GB nge rm?¶

Kuzinhlelo zefayela ze-Linux, rm izingcingo kuphela susa ukuxhumanisa(), ukususa igama lefayela kumkhombandlela ngenkathi i-kernel igcina amabhulokhi ediski abelwe kuze kube yilapho yonke inqubo ebambe lelo fayela ivuliwe ivala isichazi sefayela layo (i_isibalo == 0). Gijima sudo lsof +L1 ukuthola i-ID Yenqubo (I-PID) kanye Nesichazi Sefayela (FD) kwe (kususiwe) ifayela, bese ugijima sudo sh -c ':> /proc/PID/fd/FD' ukuvimbela amabhulokhi endaweni ngokushesha ngaphandle kokuqalisa kabusha isevisi.

Kuthiwani uma ngingakwazi ngisho ne-SSH kuseva ngoba /tmp noma /var/run igcwele 100% futhi sshd uyenqaba ukwaba iseshini?¶

Okokuqala zama ukwenza umyalo oqondile ongewona we-PTY phezu kwe-SSH ngaphandle kokwaba i-pseudo-terminal esebenzayo: ssh -T impande @ iseva "sh -c ': > /var/log/syslog'". Uma impande ukungena ngemvume kukhutshaziwe, sebenzisa i-serial console yomhlinzeki wakho wamafu (AWS EC2 Serial Console, DigitalOcean Console, Linode Lish, noma i-Proxmox/VMware VNC) ukuze ungene ngemvume ngokuqondile. othi S0 noma qalisa kumodi yokutakula yomsebenzisi oyedwa.

Othile usewasusile amafayela ngaphakathi kwe-PostgreSQL's pg_wal lwemibhalo ngaphambi kokufunda lo mhlahlandlela, futhi manje i-PostgreSQL yenqaba ukuqala ngayo PANIC: could not locate a valid checkpoint record. Ingabe yonke idatha ilahlekile?¶

Cha, imigqa eminingi yethebula isengaphakathi $PGDATA/isisekelo/, nakuba imisebenzi yangaphakathi endizeni ngaphakathi kwamafayela e-WAL asusiwe ilahleka futhi amanye amakhasi angase adabuke. Qala wenze ikhophi ephelele yakho konke $PGDATA uhla lwemibhalo (cp -a /var/lib/postgresql/16/main /mnt/backup/pgdata_raw). Bese ugijima sudo -u postgres /usr/lib/postgresql/16/bin/pg_resetwal -f /var/lib/postgresql/16/main, qala i-PostgreSQL ngemodi yomsebenzisi oyedwa, bese ugijima ngokushesha pg_dumpall endaweni entsha (indb) iqoqo; ungalokothi uqhubeke nokusebenzisa ithrafikhi yokukhiqiza kuqoqo ngemuva pg_resetwal.

Ngizisusa kanjani izigidi zamafayela amancane uma rm -rf ilenga noma yehluleka ku-inode-iphelile (df -i 100%) iseva?¶

Lapho uhla lwemibhalo luqukethe izigidi zeseshini encane noma amafayela enqolobane, inkomba yawo yohla lwemibhalo (htree) ikhula ibe yinkulu, ibangele ls futhi rm -rf ukumisa ukuhlunga amagama wefayela ku-RAM. Kunalokho, gijima sudo thola /path/to/million_files -mindepth 1 -delete, noma dala uhla lwemibhalo lwe-RAM olungenalutho (mkdir /dev/shm/empty) bese ugijima sudo rsync -a --delete /dev/shm/empty//path/to/million_files/ ukususa ama-inode ngokushesha.

Kungani i-Docker igcwalise yonke idiski yami yeseva lapho iziqukathi zami zincane, futhi ngihlanza kanjani /var/lib/docker ngokuphepha?¶

I-Docker iqongelela isikhala sediski kumalogi angashintshiwe e-JSON stdout (/var/lib/docker/containers/*/*-json.log), ukulenga imbondela2 izendlalelo zesithombe, kanye nenqolobane ye-buildkit. Ungalokothi ngesandla rm -rf amafolda ngaphakathi /var/lib/docker/overlay2/. Qala unciphise amalogi e-JSON (sudo truncate -s 0 /var/lib/docker/containers/*/*-json.log) ngakho-ke i-daemon inegumbi lokuphefumula, bese ugijima i-sudo docker system prune -a -f (eshiya --amavolumu ngakho-ke amavolumu e-database ahlala engathintwa).

Ngingakwazi ukukhulisa i-EBS yefu / ivolumu yediski eqhubekayo bukhoma ngenkathi iseva iku-100% ukusetshenziswa kwediski?¶

Yebo, ukunwetshwa kwevolumu eku-inthanethi kusebenza ngaphandle kokuqalisa kabusha, ngokubamba okukodwa: iskripthi somsizi growpart /dev/nvme0n1 1 izama ukudala ifayela lesikhashana /tmp usebenzisa mktemp futhi yehluleka kumabhayithi angu-0 mahhala. Dlula lokho kwehluleka ngokugijima sudo TMPDIR=/dev/shm growpart/dev/nvme0n1 1 (usebenzisa i-RAM-backed /dev/shm mount) kulandele sudo resize2fs /dev/nvme0n1p1 (ext4) noma sudo xfs_growfs / (XFS).

Iziqondiso ezihlobene¶

• Ngengozi rm -rf noma Shift+Delete ku-Linux, macOS, noma iWindows: funda kuphela ukukhweza kabusha
• I-SQLite Ebethelwe, Isiginali, nenkohlakalo eyisipele yengxoxo ye-WhatsApp kanye nokukhipha ukhiye wendawo
• I-Word Elonakele, i-Excel, i-PDF, nefayela lesizindalwazi kanye nokutholwa kwedokhumenti engalondoloziwe
• I-SSD ne-NVMe imodi yokufunda kuphela kungazelelwe, amabhulokhi ayisipele angu-0%, kanye ne-TRIM evimbela ukuhlenga
• Inkinga yokuba usokhaya wewebhu, ukufuduka, nokubuyisela ikhophi yasenqolobaneni
• Izinombolo zesimo esiphuthumayo ngezwe

Imithombo neziqondiso ezisemthethweni¶

• I-PostgreSQL Global Development Group: Imibhalo Esemthethweni Ye-PostgreSQL 16 - Isahluko 30: Ukwethembeka kanye Nelogi Yokubhala Phambili (Ukucushwa kwe-WAL, Izikhala Zokuphindaphinda, kanye Nokutholwa Kokufeyila Okugcwele Kwediski).
• I-SQLite Consortium (sqlite.org): Indlela Yokonakalisa Ifayela Ledathabheye Ye-SQLite (Ukukhiya Ifayela, Ijenali Engaxhumekile/Amafayela e-WAL, kanye .recover Isandiso se-CLI).
• I-Oracle Corporation / i-MySQL Documentation kanye ne-MariaDB Foundation: I-InnoDB Crash Recovery, Izinqubo Zokuhlanzwa Kwelogi kanambambili (PURGE BINARY LOGS), kanye ne-Temporary Tablespace (ibtmp1) Ukuphatha.
• Inhlangano ye-Linux Kernel (kernel.org): Ext4 Filesystem Administration (tune2fs Amabhulokhi Agodliwe, Amathebula e-Inode, kanye /proc/<pid>/fd/ Ukunqanyulwa Kwesichazi Sefayela).
• Isikhungo Sikazwelonke Samazinga Nobuchwepheshe (NIST) (United States): I-Special Publication 800-123: Umhlahlandlela Wokuvikeleka Kweseva Okujwayelekile Nokuphathwa Kokutholakala.