ʻO ka hoʻopau ʻana i ka hana ʻino, nā kī API i hoʻoheheʻe ʻia, a i ʻole nā ​​​​mea hana ao i holoi ʻia¶

ʻO kēlā me kēia ʻenekini lako polokalamu, luna hoʻoponopono ʻōnaehana, a me nā mea hana mamao ʻenehana ke kū nei i ka manawa hoʻohaʻahaʻa i ka wā i manaʻo ʻia ai kahi kauoha localhost a i ʻole staging hoʻokō kūʻē production: he mea kuhikuhi ole DELETE me ka ole a WHERE pauku, a DROP TABLE a i ʻole terraform destroy, a git push --force ka mea e hoʻopau i ka lālā nui, a i ʻole kahi hana lehulehu i loaʻa nā kī huna AWS a i ʻole Stripe. ʻO ka mea e hoʻokaʻawale ai i kahi hiccup liʻiliʻi he 15 mau minuke mai kahi pōʻino ʻikepili hoʻopau ʻana i ka ʻoihana e hoʻōki koke i kou mau lima, e hoʻokuʻu i nā kaʻa palapala ma mua o ke kākau ʻia ʻana o nā poloka mālama, a me ka hoʻokō ʻana i ka protocol recovery kūpono.

E hana i kēia i kēia manawa¶

1. **E hoʻokuʻu i kou mau lima a ʻAʻole e holo wikiwiki i nā kauoha "hoʻoponopono", e hoʻomaka hou i ka kikowaena waihona, a i ʻole e pani i kāu pukaaniani hoʻopau. .git/logs reflog i nā kaha manawa kūʻai pololei, commit SHA hashes, a me nā ID waiwai e pono ai e hoʻohuli i ka pōʻino. E kope mua i kāu scrollback hope a pau i kahi waihona kikokikona palekana.
2. E haʻi aku i kahi hanana Severity-1 (SEV-1) ma kāu hui ʻo Slack/Teams channel i loko o 2 mau minuke. Mai hoʻolilo malū i 45 mau minuke e hoʻāʻo e hoʻoponopono malū i kahi waihona hana haki wale nō i ka wā e paʻi ana nā mea kūʻai aku i nā hewa he 500 a ke hoʻopōʻino nei nā hana hope automated i nā papa i koe. E ʻaoʻao i ke kelepona ʻo Database Administrator (DBA) a i ʻole Site Reliability Engineer (SRE) koke.
3. **Inā ʻoe i hoʻokahe i kahi kī API, .env faila, a i ʻole ka huna huna i kahi repo GitHub lehulehu, E HOʻEKE MUA i ke kī i ka dashboard hāʻawi (ma mua o ka hoʻāʻo ʻana e kākau hou i ka mōʻaukala Git). git push. ʻO ka holoi ʻana i ka faila ma ka lua a i ʻole ka hoʻoikaika ikaika ʻana ʻaʻole * hoʻōki i ka mea hoʻouka i hoʻopaʻa mua i ke kī.
4. Inā ʻoe i holo pōʻino DELETE, UPDATE, a i ʻole DROP ma kahi waihona hoʻopukapuka, ʻoki i ka palapala noi Kākau i ka hele koke ʻana. E hoʻokomo i ka polokalamu pūnaewele i loko o ke ʻano Maintenance a i ʻole e hoʻopau i ka ʻae palapala a ka mea hoʻohana no laila ʻaʻole e kahakaha nā hana hou hou i nā tuple make ʻole i hoʻopaʻa ʻia a paʻakikī paha i ka Point-In-Time Recovery (PITR).
5. Mai holo VACUUM, OPTIMIZE TABLE, a i ʻole git gc --prune=now ma hope o ka holoi ʻia ʻana. Mālama nā ʻikepili pili ʻelua (PostgreSQL MVCC, MySQL InnoDB) a me Git i nā lālani holoi ʻia a me nā keiki makua ʻole i hoʻopaʻa paʻa ʻia ma ka disk a hiki i ka loaʻa ʻana o ka ʻōpala/holo ʻōpala i nā ʻaoʻao.
6. ** Ina a terraform destroy a i ʻole e holo mau ana ka palapala hoʻopau kapua ma kāu kikowaena, kaomi Ctrl + C ONCE (ʻaʻole ʻelua) a i ʻole e pepehi i ka hōʻailona kau IAM.** Ke kaomi ʻana Ctrl + C hoʻokahi manawa e hoʻokuʻu ai i kahi kūlana maikaʻi no laila e mālama ʻo Terraform i ka hapa .tfstate o na mea e noho nei; kaomi ana Ctrl + C ʻelua hoʻoikaika i ka hoʻohemo hewa ʻole i hiki ke hōʻino a nalowale paha kāu faila mokuʻāina.

ʻO ke kaʻina hana hoʻāʻo pilikia: hāʻule waihona waihona, git force-push, a i ʻole leak huna¶

E hahai i kēia lāʻau hoʻoholo e holo i kāu mau minuke 5 mua o ka hoʻopaʻa ʻenehana.

flowchart TD
    A["Hana Hana ʻEnekinia Emergency"] --> B{"ʻO wai ka hāʻule i kū mai nei?"}
    B -->|"ʻO SQL DROP / DELETE / UPDATE i ka Prod"| C["Hoʻopau i ka palapala kākau / ʻano mālama; Mai Holo i ka VACUUM"]
    B -->|"Kiʻi API / .env / Kiʻi Kūʻokoʻa ma Git"| D["Hoʻopau a hoʻololi i ka mea huna ma Cloud Dashboard i loko o 60 kekona"]
    B -->|"Puke git pōʻino --force / Lala Holoi"| E["Holo i ka git reflog ma ka ʻāina; Kope hope Good Commit SHA"]
    B -->|"Hoʻopau ʻia ka terraform pōʻino / Holoi iā S3 a i ʻole VM"| F["Pepehi IAM Session a i ʻole kaomi Ctrl+C i hoʻokahi manawa; Mālama .tfstate"]
    C --> G["E hoʻopaʻa pololei i ka leka manawa UTC; Hoʻomaka i ka Cloud PITR Clone a i ʻole Binlog Flashback"]
    D --> H["Hoʻoponopono CloudTrail / Loaʻa i nā Logs no nā Spin-Up ʻAʻole ʻae ʻia; Holoi Repo me git-filter-repo"]
    E --> I["Nīnau i nā hanana GitHub / Reflog & Restore Branch me git push --force-with-lease"]
    F --> J["E nānā i ka S3 Versioning Delete Markers, EBS Snapshots, a i ʻole Cloud Recycle Bin"]

Māhele 1: ʻIkepili hana ulia pōpilikia DROP, TRUNCATE, a i ʻole DELETE me ka ʻole WHERE¶

I ka wā e loaʻa ai kahi hulina SQL luku i kahi ʻikepili hana ola, pili ka mīkini hoʻihoʻi pololei i kāu mīkini waihona (PostgreSQL, MySQL/MariaDB, SQL Server, MongoDB, a i ʻole DynamoDB) a inā he DML ka ʻōlelo (DELETE / UPDATE) aiʻole DDL (DROP TABLE / TRUNCATE) hana.

KaʻAnuʻu Hana 1: E hoʻopaʻa i ka manawa UTC pololei a hiki i ka lua¶

Ma mua o ka hoʻopā ʻana i kekahi console backup, e kākau iho i ka manawa UTC o kēia manawa (date -u ma Linux/macOS terminal, a i ʻole SELECT NOW(); i loko o ka SQL prompt) a koho i ka lua pololei * ma mua o ka hoʻomaka ʻana o kāu nīnau luku. Hana ʻia ka Point-In-Time Recovery (PITR) ma ka hoʻokani hou ʻana i ka ʻikepili kākau-Ahead Log (WAL ma PostgreSQL) a i ʻole Binary Log (binlog i MySQL) a hiki i T_minus_1_second ma mua o ka hiki ʻana mai o kāu nīnau ʻino.

KaʻAnuʻu Hana 2: Hoʻopau i nā hana hope a me nā palapala noi (Paʻa i nā tuple MVCC)¶

Ma PostgreSQL, e hana ana DELETE FROM users; a i ʻole UPDATE orders SET status = 'cancelled'; ʻaʻole ** holoi maoli i hoʻokahi byte o ka ʻikepili mai ka faila papa ma ka disk i kēia manawa. No ka mea, hoʻohana ʻo PostgreSQL i ka Multi-Version Concurrency Control (MVCC), a DELETE kau wale i ka xmax poʻomanaʻo ID transaction ma kēlā me kēia lālani tuple e hōʻailona iā ia he ʻike ʻole ʻia e nā nīnau e hiki mai ana, e waiho ana i ka ʻikepili lālani piha e noho paʻa ana i loko o ka base/ waihona a hiki i ka hiki ʻana mai o autovacuum e hōʻailona i ka wahi hiki ke hoʻohana hou ʻia.

• Inā ʻoe ma ka hoʻokipa ponoʻī a i ʻole VM PostgreSQL a ʻaʻohe kākoʻo hou: Hoʻopili koke e like me superuser a hoʻopau i ka autovacuum ma kēlā pākaukau (ALTER TABLE table_name SET (autovacuum_enabled = false);), a i ʻole e hoʻōki i ka lawelawe PostgreSQL a lawe i kahi kiʻi kiʻi filesystem pae kiʻekiʻe (LVM / EBS snapshot) o ka /var/lib/postgresql/data papa kuhikuhi. Nā hoʻonui forensic PostgreSQL kūikawā (pg_dirtyread) hiki ke heluhelu a hoʻihoʻi hou i ka 100 pakeneka o nā lālani i holoi ʻia a i ʻole ka hoʻoponopono mua ʻana i waho o ka papaʻaina puʻu ʻole i loko o nā minuke.

ʻIkepili hoʻihoʻi waihona ma ka ʻenekini a me ka paepae ao¶

Mahele 2: Leka nā kī API, .env nā mea huna, a i ʻole nā ​​kī pilikino i hāʻawi ʻia iā Git¶

Inā ua hana hewa ʻoe i kahi .env waihona, he id_rsa Ki SSH, kahi waihona JSON moʻokāki Firebase/GCP, kahi kī huna Stripe (sk_live_...), a i ʻole AWS AKIA... kī komo iā Git, e hahai i kēia kaʻina hana ʻelua: Phase A (Pepehi i ke kī ma ke ao) mua, a me Phase B (Scrub the Git repository) lua.

Māhele A: Hoʻopaʻa ʻia ka hōʻoia ʻana o ke ao he 60 kekona (E hana mua i kēia)¶

1. AWS (AKIA... ID Key Access + Key Access Key)¶

Ke ʻaihue nā bots i kahi kī AWS mai GitHub, paʻi kā lākou palapala automated i kāu moʻokāki i loko o 10 kekona e wili i nā manawa GPU EC2 kiʻekiʻe loa (p4d.24xlarge / g5.48xlarge) ma waena o 30+ mau wahi AWS honua (me nā ʻāpana koho e like me Jakarta, Osaka, a me Middle East) i kaʻu cryptocurrency, a i ʻole hana i kahi mea hoʻohana IAM backdoor me AdministratorAccess ma mua o kou holoi ʻana i ke kī kumu.

1. E hoʻopau a Holoi i ke Kiʻi Loaʻa Leka koke:
   • E wehe i ka **AWS Console -> IAM -> Nā mea hoʻohana -> [Hoʻohana i hoʻopaʻa ʻia] -> Nā hōʻoia palekana -> Nā kī komo ** -> Kaomi iā Deactivate, a laila Delete (a i ʻole ma CLI: aws iam update-access-key --access-key-id AKIAIOSFODNN7EXAMPLE --status Inactive --user-name MyUser).
2. E hoʻopili i kahi kikoʻī DenyAll kulekele e hoʻokaʻawale i ka manawa aws sts nā hōʻailona kau:
   • Mahele koʻikoʻi: ʻOiai ma hope o kou holoi ʻana i kahi AKIA... Kiʻi komo IAM, inā ua kāhea mua ka mea hoʻouka sts:GetSessionToken a i ʻole sts:AssumeRole i loko o nā kekona 30 mua, ko lākou manawa ASIA... paʻa ka hōʻailona session a hiki i ka 1 a 12 mau hola ke ʻole ʻoe e hoʻopili i kahi kulekele inline Deny (e like me AWSCompromisedKeyQuarantineV2 a i ʻole he kulekele hōʻole-a pau no nā hōʻailona i hoʻopuka ʻia ma mua o ka hōʻailona manawa o kēia manawa) i ka mea hoʻohana a kuleana paha.
3. Hoʻoponopono i ka AWS CloudTrail no ka hoʻomau ʻana i ka puka hope:
   • E wehe i ka AWS CloudTrail -> Moʻolelo hanana a kānana e ka Access Key ID a i ʻole User name i nā hola 2 i hala.
   • E nānā pono i kēia mau kelepona hoʻomau 5: CreateUser, CreateAccessKey, AttachUserPolicy, AssumeRole, a RunInstances. E nānā i ka AWS Cost Explorer a me EC2 Global View ma waena o nā wahi a pau e hoʻopau ai i nā hihia rogue, a wehe i kahi AWS Support Case (Account and Billing -> Compromised Account). Hoʻokuʻu mau ʻo AWS i nā bila helu hoʻopunipuni ke hōʻike ʻoe i kahi leak kī a hoʻopau koke i nā manawa.

2. Kaha (sk_live_...), OpenAI (sk-...), SendGrid, Twilio, a me Slack Token¶

• Stripe: E hele i Developers -> Nā kī API -> Nā kī maʻamau / Nā kī i kaupalena ʻia -> Kaomi i nā kiko ʻekolu (...) -> Ke kī ʻōwili. E koho i ka "Hoʻopau i kēia manawa" (a i ʻole e hoʻonoho i ka puka makani aloha 1-hola wale nō inā ʻoe e hoʻolalelale ana i ke kī hou i loko o kāu mau mea hoʻololi kaiapuni hana i kēia manawa).
• OpenAI / Anthropic: E hele i ka papa kuhikuhi API Keys a kaomi koke i ka Revoke / Delete no laila ʻaʻole hiki i nā mea hoʻoheheʻe koho automated e hoʻokahe i nā kaukani kālā o nā hōʻaiʻē LLM i nā minuke.
• GitHub Personal Access Tokens (PATs), Fine-Grained Tokens, and Deploy Keys: E hele i GitHub Settings -> Developer settings -> Personal access tokens and click Delete / Revoke on the leaked token, then check your organization's Audit Log (Settings -> Audit log) no ka repo.clone, repo.access, a i ʻole ka hana ʻana i ka webhook ʻae ʻole i ka wā o ka puka aniani.
• Nā Hoʻohui Pūnaewele (DATABASE_URL / postgres:// / mongodb+srv://) a me Redis URLs: E hoʻololi i ka ʻōlelo huna a ka mea hoʻohana waihona i loko o ka ʻenekini waihona (ALTER USER app_user WITH PASSWORD 'new_random_secret';), a laila hoʻopau i nā pilina TCP e wehe nei i kēia manawa no nā hōʻoia kahiko (SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE usename = 'app_user' AND pid <> pg_backend_pid(); ma PostgreSQL), no ka hoʻololi ʻana i ka ʻōlelo huna ʻikepili ma PostgreSQL a i ʻole MySQL wale nō e poloka i nā inoa inoa * hou * a waiho i ka pilina o ka mea hoʻouka kaua e ola a hiki i ke kiki ʻana.

Māhele B: Holoi i ka mea huna mai ka mōʻaukala Git (git filter-repo Hoʻokuʻu ʻia kahi huna huna a me GitHub)¶

Ke make a hoʻololi ʻia ke kī leaked i ka hana ʻana, e hoʻomaʻemaʻe i ke kaula huna mai kāu moʻolelo Git commit no laila e pau ka makaʻala ʻana o nā mea nānā palekana:

1. E kākau hou i ka mōʻaukala waihona me ka hoʻohana ʻana git filter-repo (a i ʻole BFG Repo-Cleaner):
   • E lawe i kahi clone aniani hou o kāu waihona a wehe i ka faila leaked (no ka laʻana, .env) mai kēlā me kēia hana mōʻaukala ma nā lālā a me nā hōʻailona:

     # 1. Remove a specific file (.env) from all commits in history:
     git filter-repo --invert-paths --path .env
     
     # OR 2. Replace a specific secret string with ***REMOVED*** everywhere:
     echo "sk_live_YOUR_SECRET_STRING==>***REMOVED***" > replacements.txt
     git filter-repo --replace-text replacements.txt
     

2. E hoʻoikaika i ka mōʻaukala i hoʻomaʻemaʻe ʻia a hoʻomaʻemaʻe i ka cache GitHub Pull Request / Commit view:
   • E hoʻihoʻi i ka mōʻaukala i hoʻomaʻemaʻe ʻia i ke kumu (git push origin --force --all a git push origin --force --tags).
   • Ka hana koʻikoʻi GitHub cached-commit: ʻOiai ma hope o kou hoʻoikaika ʻana i kahi mōʻaukala i hoʻomaʻemaʻe ʻia, ʻo ka mea ʻike i ka 40-character kahiko e hana i ka SHA URL (github.com/org/repo/commit/abc1234...) hiki ke nānā i ka hana kahiko haumia ma ka UI pūnaewele o GitHub a hiki i ka holo ʻana o GitHub i ka ʻōpala ʻaoʻao server. E hoʻokaʻaʻike iā ** Kākoʻo GitHub ** (a i ʻole e hoʻohana i ka ʻōlelo aʻoaʻo a me ke kākoʻo o ka waihona waihona) me ka hana kahiko SHA a noi iā lākou e kipaku i ka hana i hoʻopaʻa ʻia mai ka waihona pūnaewele a holo. git gc ma ka waihona mamao.

Mahele 3: Ka wehe ʻana i kahi pōʻino git push --force, git reset --hard, a i ʻole lālā i holoi ʻia¶

Inā 'oe i holo pohihihi git push --force a holoi i nā hana a kou mau hoa hui main, a holo git reset --hard HEAD-5 a nalowale ka lā piha o ka hana kūloko, mai hopohopo, no ka mea, ʻaneʻane ʻaʻole holoi koke ʻo Git i nā mea i hana ʻia.

No ke aha e noho maluhia ana kāu mau hana "nalo" i loko .git/logs (The Reflog)¶

I nā manawa a pau e hana ai ʻoe i kahi hana, hoʻololi i nā lālā, rebase, a hoʻihoʻi hou paha, hoʻopaʻa ʻo Git i ka mea ma mua HEAD a me ka ʻaoʻao o ka lālā SHA-1/SHA-256 mau kuhikuhi i loko o kahi papa helu kūloko i kapa ʻia ʻo Reference Log (git reflog), a mālama i nā mea hana keiki makua ʻole ma ka diski no ma ka liʻiliʻi loa he 14 a 90 mau lā ma mua o ka wehe ʻana i ka ʻōpala.

ʻO nā ʻōkuhi hoʻolaʻa hoʻōla ulia pōpilikia ʻo Git¶

Nānā 1: Holo ʻoe git push --force a kau i luna origin/main¶

1. ** E ʻimi i ka hana pololei SHA kēlā origin/main kuhikuhi ʻia ma mua pono o kāu paʻi ikaika:**
   • Method A (E nānā i kāu reflog tracking remote local): E holo i kēia kauoha ma ka pahu kahi āu i pana ai:

     git reflog show origin/main
     

     E ʻike ʻoe i ka hopena e like me:

     a1b2c3d (HEAD -> main, origin/main) refs/remotes/origin/main@{0}: update by push
     9f8e7d6 refs/remotes/origin/main@{1}: fetch: fast-forward
     

     Eia, 9f8e7d6 (origin/main@{1}) ka hana pololei origin/main aia ma ka ʻākau ma mua o kou hoʻopaʻa ikaika ʻana iā ia.
   • Method B (E nīnau i kekahi hoa hui i holo ʻole git pull aka): Kekahi hoa hana nana i kii main ma mua o kēia lā aia nō ka mōʻaukala hana paʻa e noho ana ma ko lākou wahi origin/main lālā.
   • Method C (Nīnau i ka GitHub REST API Events log): Inā nele kāu reflog kūloko i ka mokuʻāina mamao, e nīnau i ka API hana pana a GitHub e ʻike i ka "before" SHA hash o ka hanana paʻi ikaika:

     gh api /repos/OWNER/REPO/events --jq '.[] | select(.type=="PushEvent") | {actor: .actor.login, before: .payload.before, head: .payload.head, time: .created_at}'
     

2. Hoihoi origin/main i ka hana mua SHA me ka palekana:
   • E hōʻoia i ka pololei o ka hana (git log -n 5 9f8e7d6), a laila e hoʻihoʻi i kēlā SHA pololei i ka mamao main lālā hoʻohana --force-with-lease:

     git push --force-with-lease origin 9f8e7d6:refs/heads/main
     

Nānā 2: Holo ʻoe git reset --hard a i ʻole ka holoi ʻana i kahi lālā kūloko ma mua o ka pana ʻana¶

1. Holo git reflog i kāu waihona kūloko:

   git reflog --date=iso
   

2. ** E huli i ka commit SHA ma mua pono o ka reset: komo** (e.g., HEAD@{2}: commit: feat: add payment webhook), a hana i kahi lālā hoʻopakele hou e kuhikuhi pololei ana i kēlā SHA:

   git switch -c rescue-branch HEAD@{2}
   

3. Pehea inā ua nalowale ʻoe i kahi faila uncommitted āu git added (staged) a laila holoi ʻia me git reset --hard?
   • ʻO ka manawa āu e holo ai git add myfile.ts, Hoʻopili ʻo Git a kākau i nā ʻike piha o kēlā faila i loko .git/objects/ ma ke ʻano he puʻu lele, ʻoiai inā ʻaʻole ʻoe i holo git commit. E hoʻihoʻi iā ia ma ka helu ʻana i nā pōpō lele a pau i hoʻokaʻawale ʻia e ka manawa hoʻololi:

     git fsck --lost-found
     # Inspect the recovered blobs written into .git/lost-found/other/
     

Mahele 4: Pohō terraform destroy a i ʻole i holoi ʻia ka mālama ʻana i ke ao (S3, EBS, VM)¶

Inā hoʻomaka kahi pipeline Infrastructure-as-Code (IaC) a i ʻole kauoha CLI manual e wāwahi i nā kumuwaiwai ao hana:

1. Hoʻopau i ka hoʻokō me ka palekana:
   • Inā holo ma ka ʻāina i kāu kikowaena, kaomi Ctrl + C ONCE no laila hoʻopau ʻo Terraform i kāna kelepona API o kēia manawa a kākau i ka mea hou .tfstate waihona maʻemaʻe. Inā holo i loko o GitHub Actions, GitLab CI, a i ʻole Terraform Cloud, kaomi iā Cancel Run a hoʻopau koke i ka hana AWS/GCP IAM a ka mea holo CI ma ka ʻenehana kapua no laila ʻaʻole pau ka holoi ʻana o ka API i ka lele. AccessDenied.
2. E hoʻihoʻi i ka manawa .tfstate faila koke:
   • Hoʻoiho i kahi kope kūloko o kāu faila mokuʻāina mamao (terraform state pull > emergency-backup.tfstate) a e nānā i ka hoʻololi ʻana o ka bakeke S3 ma kāu bakeke mokuʻāina ʻo Terraform e hopu i ka mana mokuʻāina mua mai 10 mau minuke aku nei.
3. E nānā i ka mea hāʻawi kapuaʻi "Soft-Delete" a me nā ʻupena palekana Recycle Bin:
   • Amazon S3 (Me ka Bucket Versioning enabled): Ke holoi nei i nā faila i loko o ka bakeke S3 versioned (aws s3 rm --recursive) ʻaʻole ** holoi i kahi mea hoʻokahi; hoʻokomo wale ia i kahi zero-byte Delete Marker ma luna o kēlā me kēia mea. Wehe i ka Delete Markers (ma o S3 Batch Operations a i ʻole 5-line Python boto3 script) e hoʻihoʻi koke i ka 100 pakeneka o kāu mau faila.
   • AWS EBS Volumes & RDS Snapshots (Recycle Bin): E nānā i ka AWS Recycle Bin i loko o ka console inā hoʻonohonoho ʻia, a e nānā i ka AWS Backup vaults.
   • Google Cloud Platform (GCP) & Microsoft Azure: Holoi ʻia nā papahana GCP a me nā hui kumu waiwai Azure / Key Vaults / waihona waihona me Soft Delete hiki ke hoʻihoʻi hou ʻia no 7 a 30 mau lā ma kahi kaomi hoʻihoʻi hoʻokahi i ka console.

Pehea e hoʻihoʻi ai i kahi Terraform i haki .tfstate waihona mokuʻāina a i ʻole e hoʻokomo hou i nā ʻōnaehana makua ʻole¶

I ka manawa a terraform apply a i ʻole pōʻino terraform destroy ke hoʻopau ʻia i waena o ka lele ʻana, a i ʻole i ka wā e hoʻopau wale ai kekahi i ka faila mokuʻāina mamao ma S3/GCS, komo kāu ʻōnaehana kapuaʻi i kahi mokuʻāina hoʻokaʻawale weliweli kahi i loaʻa ai kekahi mau kumuwaiwai ma AWS/GCP aʻo nā mea ʻē aʻe e nalowale ana. .tfstate:

1. Ke hoʻihoʻi ʻana i kahi mamao i kākau ʻia a holoi ʻia paha .tfstate waihona mai S3 / GCS Versioning:
   • Mai hoʻoponopono lima i kahi JSON .tfstate faila ma ka lima i loko o ka mea hoʻoponopono kikokikona ke ʻole i hāʻule nā ​​​​koho ʻē aʻe, no ka mea, ua like ʻole kekahi "serial" a i ʻole "lineage" Na ke kaula UUID e ālai iā Terraform mai ke noi hou ʻana.
   • E wehe i kāu bakeke S3 (a i ʻole Google Cloud Storage bucket / Terraform Cloud Workspace States tab) kahi e noho ai ka hope hope, kaomi i ka Show versions toggle ma ka piko o ka console S3, e ʻimi i ka terraform.tfstate ʻO ka mana i hoʻopaʻa ʻia ma mua pono o ka holo ʻino, hoʻoiho iā ia, a hoʻihoʻi i hope me ka palekana e like me ke kūlana hou o kēia manawa:

     # Push a recovered historical state file back to the remote backend safely
     terraform state push recovered-pre-incident.tfstate
     

2. Wāhi i ka laka kūlana paʻa (Error acquiring the state lock:
   • Ke hooki oe i ka ino terraform destroy me Ctrl + C a pepehi paha i ka mea holo CI i kau ʻia, paʻa ka papaʻaina DynamoDB mamao a i ʻole ka faila laka GCS e ka ID kaʻina make.
   • E hōʻoia mua i kāu CI dashboard a me kāu mau hoa hui ʻaʻole e holo mau ana nā kaʻina Terraform ʻē aʻe i kēlā wahi hana, a laila hoʻokuʻu i ka laka makua ʻole me ka hoʻohana ʻana i ka UUID Lock ID i paʻi ʻia ma ka memo hewa:

     terraform force-unlock <LOCK_ID_UUID>
     

3. ** Ke kūkulu hou ʻana i kahi wahi i hoʻopau ʻia me ka hana ʻole ʻana i nā ʻikepili ola (terraform import:**
   • Ina a terraform destroy holoi ʻia kāu mau mea hoʻoili hoʻopaʻa noi mokuʻāina ʻole (aws_lb) a me nā node limahana ECS/EKS ma mua o kou hooki ʻana, maʻalahi e hoʻoponopono: holo wale terraform apply A e hana hou ʻo Terraform i nā ʻupena mokuʻāina āpau a helu i nā papa mai ka wā ʻōpala i 5 mau minuke i ka waiho ʻana i kāu waihona RDS e ola nei me ka ʻole.
   • Inā nalowale kāu waihona mokuʻāina i kahi waihona waihona mokuʻāina e ola nei a i ʻole S3 bākeke, e hoʻohana i kahi import {} poloka hoʻonohonoho (Terraform 1.5+) a i ʻole terraform import aws_db_instance.prod_db my-rds-instance-id e hoʻopaʻa i ka waihona ao i kēia manawa i loko o kāu faila mokuʻāina me ka ʻole o ka luku ʻana, a holo terraform plan e hooiaio "0 to destroy" ma mua o ka noi ʻana.

Mahele 5: Pehea e kamaʻilio ai i kahi outage i kāu luna a alakaʻi i kahi post-mortem hala ʻole¶

Ke hoʻokumu ʻoe i kahi hanana hana ʻoiai e hana mamao ana, ua wehewehe ʻia kou inoa ʻoihana ** ʻaʻole ** e ka hewa ponoʻī, akā ma kou wikiwiki o ka ʻike, kamaʻilio mālie, a me ka hoʻoponopono ʻōnaehana.

ʻO nā lula gula 3 o ke kamaʻilio hanana¶

1. **E hōʻike i ka hewa i loko o 2 mau minuke o ka ʻike ʻana.
2. E hoʻokaʻawale i ka "Mitigation" (Hoʻopau i ke koko) mai ka "Root-Cause Fix": I kahi pilikia, ʻo kāu pahuhopu wale nō ka hoʻihoʻi ʻana i ka lawelawe (hoʻihoʻi ʻana i ka hoʻolaha ʻino, hāʻule ʻole i ka clone database PITR, a i ʻole ka hoʻihoʻi ʻana i ka hae hiʻona). E mālama i ka hoʻoponopono hou ʻana i nā code hohonu no ka lā ʻapōpō.
3. E hoʻouna i nā mea hou o ka puʻuwai he 15 mau minuke ma ke kahawai hanana i ʻike nā luna hoʻokō a me nā kākoʻo mea kūʻai aku aia kekahi i ka mana.

Kopi-paʻi i ka Slack / Nā Pūʻulu Hōʻike Hōʻikeʻike¶

[SEV-1 INCIDENT] Hana ʻana [Laumana / Inoa Pūnaewele] Ka hopena - He aha ka mea i hanaia: Ma [HH:MM UTC], oiai e hooko ana [brief technical action, e.g., a data migration script], he mea i manao ole ia [DELETE / config change / force-push] ua hoʻohana ʻia i ka hana ʻana [database / service]. - Ka hopena o nā mea kūʻai aku i kēia manawa: [e.g., Checkout API is returning 500 errors / User profile updates are paused]. - Hoʻomaka koke ka hoʻēmi ʻana: 1. Kākau i ke kalaiwa i [table/service] ua hoʻomaha ʻia no ka mālama pono ʻana i ka ʻikepili. 2. Hoʻihoʻi i ka manawa (PITR) clone i [HH:MM:SS UTC] (10 kekona ma mua o ka hanana) ke wili nei i kēia manawa [AWS RDS / GCP]. - Ka Luna Hoʻokele: @[Your Name] (kōkua ʻia e @[DBA/On-Call SRE]). - Mea Hou Hou: I loko o 15 minuke ([HH:MM UTC]) a i ʻole i ka wā e hele mai ai ka clone hoʻōla ma ka pūnaewele.

Ka papa inoa o ka mālama ʻana i ka ʻenehana ma hope o ka hana ʻana (No kāu Post-Mortem Blameless)¶

ʻO kēlā me kēia hewa kanaka i ka hana ʻana he ** ʻano hoʻolālā ʻōnaehana ** (inā hiki i kahi kauoha terminal kope kope hoʻokahi ke hoʻokuʻu i ka hana me ka ʻole o ka pale, ke kali nei ka ʻōnaehana e hāʻule). E noi a e hoʻokō i kēia mau pale ʻenehana paʻa i kāu palapala post-mortem i ka lā aʻe:

• Ka kala kala a me ka prefix production prompts (kube-ps1, psql .psqlrc): E hoʻonohonoho i kāu pūpū a me ka mea kūʻai aku SQL i pili i kekahi PRODUCTION hōʻike i kahi ʻulaʻula ʻulaʻula [PROD-DB] wikiwiki a hoʻomaka i kēlā me kēia hui PostgreSQL i loko a BEGIN; poloka kalepa ma ka paʻamau (a i ʻole sets SET default_transaction_read_only = on;).
• E ho'ā safe-updates (--i-am-a-dummy) ma MySQL/MariaDB nā mea kūʻai: Hoʻohui safe-updates i [mysql] i loko -/.my.cnf no laila, hōʻole loa ʻo MySQL e hoʻokō i kekahi UPDATE a i ʻole DELETE ʻōlelo i nele a WHERE pauku kī a LIMIT.
• Hōʻā DeletionProtection = true ma nā waihona ʻikepili RDS hana a pau, nā bākeke S3, a me nā kumu waiwai kūlana Terraform (lifecycle { prevent_destroy = true }).
• E hoʻokō i nā lula hoʻomalu lālā ʻo Git ma main / production: Hoʻopau Allow force pushes a Allow deletions ma nā lālā i mālama ʻia ma GitHub/GitLab no laila ʻaʻole hiki i nā luna waihona git push --force i main.
• E hoʻokomo i nā makau nānā huna (gitleaks a i ʻole trufflehog) a me GitHub Push Protection: Block commits loaʻa AKIA, sk_live_, a i ʻole .env nā hiʻohiʻona ma mua git commit hiki ke lanakila.

Nīnau pinepine¶

E hoʻokuʻu ʻia paha au no ka hoʻolei ʻana i ka papa ʻikepili hana a i ʻole ka hoʻopau ʻana i kahi pūnaewele?¶

I loko o nā hui ʻenekinia olakino e hoʻomaʻamaʻa ana i kahi moʻomeheu palekana hala ʻole (maʻamau ma waena o nā ʻenehana hou, DevOps, a me nā hui SRE), ʻaʻole i puhi ʻia nā ʻenekinia no nā hewa kanaka pono i ka hana ʻana, inā lākou e hōʻike koke i ka hanana a kōkua i ka hoʻihoʻi ʻana i ka lawelawe. Hoʻomaopopo nā alakaʻi ʻenekinia inā loaʻa i ka mea hoʻomohala ʻōpio a i ʻole nā ​​​​mea ʻoi loa ka ʻae a me ka nele o nā pale e hoʻolei i kahi papa hana me kahi kauoha ʻike ʻole ʻia, ʻo ke kumu kumu he hemahema i ka mana komo a me ka hoʻohana ʻana i nā mea hana, ʻaʻole ka hewa. I ka hoʻomaʻamaʻa ʻana, ʻo ka ʻenekinia i ʻike mua i ka hoʻopau ʻana i ka lima i kēia manawa ke kanaka palekana loa ma ka hui e pili ana i kēlā ʻōnaehana. ʻO ka hana wale nō e alakaʻi mau ai i ka hoʻopau ʻana ʻo ka hoʻāʻo ʻana e uhi i kahi hanana nalowale ʻikepili, holoi ʻana i nā loiloi loiloi, a i ʻole ka wahaheʻe i ka wā o ka hoʻokolokolo ʻana.

No ke aha ʻaʻole pono iaʻu e hoʻololi koke i kaʻu huakaʻi hana ola i ka laʻana ʻikepili Point-In-Time Recovery (PITR) i hoʻihoʻi hou ʻia?¶

Inā ʻoe i holoi wale i nā lālani mai kahi papa hoʻokahi (no ka laʻana, ka `user_preferences` papaʻaina) ma 14:00 UTC, a he 20 mau minuke e wili ai i kahi clone PITR i hoʻihoʻi ʻia i 13:59:50 UTC, ua hoʻomau paha kāu ʻikepili hana ola i ka hana ʻana i nā tausani o nā kauoha a nā mea kūʻai aku kūpono a me nā uku ma ka `orders` papaaina ma waena o 14:00 a me 14:20 UTC. Inā ʻoe e hoʻolele makapō i kāu kikowaena DNS hana holoʻokoʻa i ka clone 13:59:50 PITR, e holoi ʻoe i kēlā mau minuke 20 o nā kauoha mea kūʻai aku hou. Akā, e mālama i ka ʻikepili nui ma ka pūnaewele (me nā kākau i hoʻomaha wale ʻia ma ka hiʻohiʻona haʻihaʻi) a hoʻokuʻu wale i nā lālani i nalowale mai ka 13:59:50 PITR clone hoʻi i ka papa hana ola.

Inā wau e hana i kahi waihona GitHub kūʻokoʻa pilikino ʻelua mau minuke ma hope o ka pana ʻana i kahi .env file, pono anei au e hoʻololi i nā kī API a pau?¶

ʻAe, pono e hoʻopau a hoʻololi koke ʻia ka 100 pakeneka o ia mau kī. Hoʻolaha ʻia kēlā me kēia hana i kahi waihona GitHub lehulehu i ka manawa maoli ma luna o ka lehulehu GitHub Events API ("ka Firehose") a kuhikuhi ʻia i loko o nā kekona e nā lawelawe waihona mālama pono kūpono (e like me GHArchive) a me nā bots hōʻiliʻili hōʻoiaʻiʻo hewa e holo ana i 24/7 ma waena o nā pūnaewele kapuaʻi. ʻO ka hoʻohuli ʻana i ka ʻike waihona mai Public a Private ʻelua mau minuke ma hope, ʻaʻole ia e holoi i ke kope i hoʻoiho ʻia i nā scrapers automated.

Pehea git push --force-with-lease pale i nā outage i hoʻohālikelike ʻia me git push --force?¶

Ke holo ʻoe maʻamau `git push --force`, Hoʻopau ʻole ʻo Git i ka lālā mamao me kāu lālā kūloko, me ka hoʻopau ʻole ʻana i nā hana a kekahi hoa hui i kiola ʻia i kēlā lālā i kēia manawa. Ke hoʻohana ʻoe `git push --force-with-lease` (a ʻoi aku ka maikaʻi, `--force-if-includes`), E nānā mua ʻo Git i ke kikowaena mamao e hōʻoia i ka pili ʻana o ka ʻaoʻao mamao o ka lālā i ka hana pololei SHA i kiʻi ʻia i kāu mīkini kūloko. Inā hoʻokomo kekahi hoa hana a i ʻole CI pipeline i kahi hana hou i ka lālā mamao mai kāu kiʻi hope ʻana, `--force-with-lease` hoʻopau palekana me ka ʻōlelo aʻoaʻo "stale info" ma mua o ka holoi ʻana i kā lākou hana.

Nānā ʻia nā kumu¶

• National Institute of Standards and Technology (NIST, United States): NIST SP 800-61 Rev. 2 / Rev. 3 Computer Security Incident Handling Guide and Credential Compromise Containment.
• Amazon Web Services (AWS), Google Cloud Platform (GCP), a me Microsoft Azure Documentation (United States): RDS Automated Point-In-Time Recovery (PITR) Architecture, AWS IAM Compromised Key Quarantine Policies (AWSCompromisedKeyQuarantineV2), a me S3 Versioning Delete-Marker Recovery.
• PostgreSQL Global Development Group a me MySQL / Oracle Documentation (International): PostgreSQL Write-Ahead Logging (WAL), MVCC Tuple Visibility, a me MySQL Row-Based Binary Log Flashback Recovery.
• Git SCM Official Documentation a me GitHub Security Lab (International): Git Reflog (git-reflog), Hoʻihoʻi hou i nā mea e lele ana (git-fsck), Kākau Hou Moʻolelo (git-filter-repo), and Secret Scanning Push Protection.
• USENIX Association / SREcon and Google Site Reliability Engineering (SRE) Books (International): Managing Incidents, Emergency Rollback Triage, and Blameless Postmortem Culture.

Nā alakaʻi pili¶

• Emergency remote work security and data privacy
• Computer backups 3-2-1 strategy and disaster recovery
• Accidental file deletion and formatted drive recovery
• Web hosting crisis migration and backup restoration
• Phishing and compromised accounts
• Helu pilikia