Nuwɔwɔ ƒe Nuwuwu le Vodada me, API Safui siwo do go, alo Alilikpo me Dɔwɔƒe si Wotutu¶

Kɔmpiutadziɖoɖowo dzikpɔla, dɔwɔɖoɖowo dzikpɔla, kple mɔ̃ɖaŋudɔwɔla ɖesiaɖe si le adzɔge ʋĩi la dzea ŋgɔ ɣeyiɣi si me dɔgbo nagbã esime woɖo sedede aɖe na localhost alo staging wua ame ɖe eŋu production: nusi womede dzesii o DELETE manɔmee a WHERE nyagbe, a DROP TABLE alo terraform destroy, a git push --force si ŋlɔa alɔdze gã la ɖe edzi, alo dutoƒo commit si me AWS alo Stripe adzame safui siwo le agbe le. Nusi ma aɖabaƒoƒo 15 ƒe hiccup sue aɖe ɖa tso nyatakakawo ƒe afɔku si wua dɔa nu gbɔe nye be nàtɔ te wò asiwo enumake, nàtɔ te nuŋɔŋlɔ ƒe ʋuwo hafi woaŋlɔ nudzraɖoƒe ƒe mɔxenuwo ɖe edzi, eye nàwɔ ɖoɖo nyuitɔ si dzi nàto agbugbɔ axɔ.

Wɔ esia fifia¶

1. Freeze wò asiwo eye MÈGAwɔ "ɖɔɖɔɖo" sededewo kabakaba, gbugbɔ dze database server la gɔme, alo tu wò terminal fesre. Wò terminal scrollback buffer si li fifia, shell ŋutinya, kple local .git/logs reflog me la, asitsatsa ƒe ɣeyiɣi ƒe dzesi siwo sɔ pɛpɛpɛ, commit SHA hashes, kple resource ID siwo hiã be woatsɔ atrɔ nusiwo gblẽ la le. Kɔpi wò terminal scrollback bliboa ɖe text file si le dedie me gbã.
2. Gblɔ Severity-1 (SEV-1) nudzɔdzɔ aɖe le wò ƒuƒoƒoa ƒe nudzɔdzɔ Slack/Teams channel me le miniti 2 me. Mègazã miniti 45 gbeɖe atsɔ adze agbagba aɖɔ nuwɔwɔ ƒe nyatakakadzraɖoƒe si gblẽ ɖo le adzame esime asisiwo le vodada 500 ƒom eye megbedɔwo le wo ɖokui si le kplɔ̃ susɔeawo gblẽm o. Axa Nyatakakadzraɖoƒe Dzikpɔla (DBA) alo Site Reliability Engineer (SRE) si le kaƒoƒoa dzi enumake.
3. Ne èɖe API safui aɖe do go la, . .env file, alo cloud secret to a public GitHub repo, REVOKE the key in the provider dashboard FIRST (hafi nàdze agbagba agbugbɔ Git ŋutinya aŋlɔ). Botnet siwo wowɔ le wo ɖokui si la ƒoa dutoƒo GitHub Firehose API eye wowɔa AWS, OpenAI, Stripe, kple GCP safui siwo wofi le ** sɛkɛnd 3 va ɖo 15** me tso wò git push. Fail la tutu le commit alo force-pushing evelia me metɔ te amedzidzela si wɔ clone na safuia xoxo o.
4. Ne èƒu du le vo me DELETE, . UPDATE, alo DROP le nuwɔwɔ ƒe nyatakakadzraɖoƒe dzi la, tso dɔwɔɖoɖoa ŊLƆ ʋuwo ƒe zɔzɔ dzi enumake. De web dɔwɔɖoɖoa ɖe Dzadzraɖo ƒe Nɔnɔme me alo te fli ɖe dɔwɔnua zãla ƒe nuŋɔŋlɔ ƒe mɔɖeɖewo me ale be asitsatsa yeye siwo va la nagaŋlɔ tuple kuku siwo womeɖe vacuum o ɖe edzi alo ana Point-In-Time Recovery (PITR) nasesẽ o.
5. Mègaƒu du gbeɖe o VACUUM, . OPTIMIZE TABLE, alo git gc --prune=now le tutu le vo me megbe. Ƒomedodo ƒe nyatakakadzraɖoƒe (PostgreSQL MVCC, MySQL InnoDB) kple Git siaa léa fli siwo wotutu kple tsyɔ̃evi ƒe dɔwɔwɔwo ɖe disk dzi vaseɖe esime gbeɖuɖɔ-nuƒoƒoƒu/vacuum pass aɖe agbugbɔ axaawo axɔ ŋutɔŋutɔ.
6. Ne a terraform destroy alo cloud deletion script gakpɔtɔ le dɔ wɔm le wò terminal me, zi edzi Ctrl + C ZI ÐEKA (menye zi eve o) alo wu IAM ƒe ɣeyiɣi ƒe dzesi. Zi edzi Ctrl + C zi ɖeka la, eʋãa amenuveve nɔnɔme flush ale Terraform xɔ akpa aɖe .tfstate nusiwo gakpɔtɔ li ŋu; tete ɖe ame dzi Ctrl + C zi eve zi fuɖeɖe si me amenuveve mele o si ate ŋu agblẽ alo abu wò nutoa me dukɔa ƒe faɛl.

Emergency triage dɔwɔwɔ: nyatakakadzraɖoƒe ƒe ɖiɖi, git force-push, alo adzame leak¶

Dze nyametsotsoti sia yome nàtsɔ aɖo wò miniti 5 gbãtɔ siwo nàtsɔ axe mɔ ɖe mɔ̃ɖaŋununya nu.

flowchart TD
    A["Nuwɔwɔ ƒe Mɔ̃ɖaŋununya ƒe Kpata"] --> B{"Dodokpɔ kae dzɔ teti koe nye ema?"}
    B -->|"SQL DROP / DELETE / UPDATE le vo me le Prod"| C["Dzudzɔ App Ŋɔŋlɔ / Beléle na Mɔnu; MÈGAƑU VACUUM O"]
    B -->|"API Safui si do go / .env / Ame ŋutɔ ƒe Safui le Git me"| D["Te fli ɖe eme & Trɔ Nya Ɣaɣla le Alilikpo Dashboard me Le Sɛkɛnd 60 Me"]
    B -->|"Vodada git push --force / Alɔdze si Wotutu ɖa"| E["Ƒu du git reflog Le Nutoa me; Kɔpi Mamlɛtɔ Nyui Ðokuitsɔtsɔna SHA"]
    B -->|"Afɔku terraform tsɔtsrɔ̃ / Deleted S3 alo VM"| F["Wu IAM Session alo Zi Ctrl+C dzi Zi ɖeka; Dzra .tfstate ɖo"]
    C --> G["Ŋlɔ UTC Ɣeyiɣi ƒe Dzesi si Sɔ pɛpɛpɛ; Ʋu Alilikpo PITR Clone alo Binlog Flashback"]
    D --> H["Dzro CloudTrail / Access Logs me hena Spin-Ups siwo ŋu womeɖe mɔ ɖo o; Klɔ Repo kple git-filter-repo"]
    E --> I["Bia GitHub Nudzɔdzɔwo / Reflog & Gbugbɔgaɖo Alɔdze kple git push --force-with-lease"]
    F --> J["Kpɔ S3 Versioning Delete Markers, EBS Snapshots, alo Alilikpo me Gbugbɔgazã ƒe Bin ɖa"]

Akpa 1: Nusiwo wowɔ le vo me ƒe nyatakakadzraɖoƒe DROP, . TRUNCATE, alo DELETE manɔmee WHERE¶

Ne SQL biabia si gblẽa nu la dze dɔwɔwɔ ƒe nyatakakadzraɖoƒe si le agbe dzi la, gbugbɔgaxɔ ƒe mɔnu tututu nɔ te ɖe wò nyatakakadzraɖoƒea ƒe mɔ̃ dzi (PostgreSQL, MySQL/MariaDB, SQL Server, MongoDB, alo DynamoDB) kple nenye be nyagbɔgblɔa nye DML (DELETE / . UPDATE) alo DDL (DROP TABLE / . TRUNCATE) dɔwɔwɔ.

Afɔɖeɖe 1: Ŋlɔ UTC ƒe ɣeyiɣi ƒe dzesi si sɔ pɛpɛpɛ ɖi vaseɖe evelia dzi¶

Hafi nàka asi backup console ɖesiaɖe ŋu la, ŋlɔ UTC ɣeyiɣi si tututu nèzãna fifia (date -u le Linux/macOS ƒe nuwuƒe, alo SELECT NOW(); le SQL ƒe nyabiasea me) eye nàbu sɛkɛnd si tututu hafi wò biabia si gblẽa nu la nadze dɔwɔwɔ gɔme ko. Point-In-Time Recovery (PITR) wɔa dɔ to nyatakakadzraɖoƒea ƒe Write-Ahead Log (WAL le PostgreSQL) alo Binary Log ( .binlog le MySQL me) vaseɖe T_minus_1_second do ŋgɔ teti na wò biabia gbegblẽa va ɖo.

Afɔɖeɖe 2: Dzudzɔ megbedɔwo kple dɔwɔwɔ ƒe nuŋɔŋlɔwo (Ta MVCC tuples ta) .¶

Le PostgreSQL me la, wole dɔ wɔm DELETE FROM users; alo UPDATE orders SET status = 'cancelled'; me tutu nyatakaka byte ɖeka tso kplɔ̃ ƒe faɛl si le disk dzi la me enumake o. Esi PostgreSQL zãa Multi-Version Concurrency Control (MVCC) ta la, a DELETE ɖeko wòɖoa xmax transaction ID header le row tuple ɖesiaɖe dzi be woade dzesii be womate ŋu akpɔe na etsɔme biabiawo o, agblẽ row data bliboa ɖi wòanɔ anyi ɖaa le eme base/ heap file vaseɖe esime autovacuum nava be wòade dzesi teƒe si woateŋu azã ake.

• Ne èle self-hosted alo VM PostgreSQL dzi eye mèkpɔ backup nyitsɔ laa o: Enumake do ka abe superuser eye nàtsi autovacuum nu le kplɔ̃ ma dzi (ALTER TABLE table_name SET (autovacuum_enabled = false);), alo nàtɔ te PostgreSQL subɔsubɔdɔa eye nàɖe raw block-level filesystem snapshot (LVM / . EBS snapshot) ƒe nɔnɔmetata /var/lib/postgresql/data nyatakakadzraɖoƒe. PostgreSQL ƒe ʋɔnudrɔ̃ƒe ƒe kekeɖenudɔ tɔxɛwo (pg_dirtyread) ate ŋu axlẽ fli siwo wotutu alo esiwo wogbugbɔ trɔ asi le do ŋgɔ ƒe alafa memama 100 tẽ tso heap kplɔ̃ si wometsɔ vacuum o la dzi le aɖabaƒoƒo ʋɛ aɖewo ko me.

Database gbugbɔgaxɔ matrix to mɔ̃ kple alilikpo ƒe mɔ̃ dzi¶

--- .

Akpa 2: API safui siwo do go, . .env nya ɣaɣlawo, alo ame ŋutɔ ƒe safui siwo wotsɔ na Git¶

Ne èwɔ nu aɖe le vo me .env file, an id_rsa SSH safui, Firebase/GCP subɔsubɔ akɔnta JSON faɛl, Stripe adzame safui (sk_live_...), alo AWS aɖe AKIA... access key to Git, zɔ ɖe phase eve ƒe ɖoɖo sia tututu dzi: Akpa A (Wu safuia le alilikpo me) gbã, kple Akpa B (Scrub the Git repository) evelia.

Akpa A: Alilikpo me ɖaseɖigbalẽwo ƒe sɛkɛnd 60 ƒe mɔxexe enumake (Wɔ esia GBÃTƆ) .¶

1. AWS (AKIA... Access Key ID + Adzame Mɔɖeɖe ƒe Safui) .¶

Ne bots fi AWS safui tso GitHub la, woƒe ŋɔŋlɔdzesi si wowɔna le wo ɖokui si la ƒoa wò akɔnta le sɛkɛnd 10 me be wòatrɔ GPU EC2 ƒe kpɔɖeŋu siwo lolo wu (p4d.24xlarge / . g5.48xlarge) le xexeame katã ƒe AWS nuto 30+ katã me (si me tiatia-ɖe nutowo abe Jakarta, Osaka, kple Titina Ɣedzeƒe ene hã le) be woaku cryptocurrency, alo awɔ megbeʋɔtru IAM zãla kple AdministratorAccess hafi nàtutu safui gbãtɔa ɖa.

1. Tɔ asi le Access Key si do go la ŋu eye nàtutu enumake:
   • Ʋu AWS Console -> IAM -> Zãlawo -> [Zãla si Wogblẽ] -> Dedienɔnɔ ƒe ɖaseɖigbalẽwo -> Mɔɖeɖe ƒe safuiwo -> Zi Deactivate dzi, emegbe Delete (alo to CLI dzi: aws iam update-access-key --access-key-id AKIAIOSFODNN7EXAMPLE --status Inactive --user-name MyUser).
2. Tsɔ nya aɖe si me kɔ la kpe ɖe eŋu DenyAll ɖoɖo si wowɔ be woaɖe akpa aɖeke dzi akpɔtɔ hena ɣeyiɣi kpui aɖe aws sts ɣeyiɣi ƒe dzesiwo:
   • Critical trap: Ne ètutu an AKIA... IAM access key, ne amedzidzela la yɔe xoxo sts:GetSessionToken alo sts:AssumeRole le sɛkɛnd 30 gbãtɔawo me la, woƒe ɣeyiɣi kpui aɖe ƒe ASIA... session token nɔa dɔ wɔm vaseɖe gaƒoƒo 1 va ɖo 12 negbe ɖe nètsɔ inline Deny ɖoɖo si me kɔ (abe AWSCompromisedKeyQuarantineV2 alo Gbegbe-nuwo katã ƒe ɖoɖo na dzesi siwo woɖe ɖe go do ŋgɔ na ɣeyiɣi ƒe dzesi si li fifia) na zãla alo akpa si wòwɔ.
3. Dzro AWS CloudTrail me hena megbeʋɔtru ƒe kutrikuku:
   • Ʋu AWS CloudTrail -> Nudzɔdzɔ ŋutinya eye nàɖe Access Key ID alo Zãla ƒe ŋkɔ si wogblẽ le gaƒoƒo 2 siwo va yi me.
   • Kpɔe ɖa be kutrikuku ƒe kaƒoƒo 5 siawo koŋ: CreateUser, . CreateAccessKey, . AttachUserPolicy, . AssumeRole, kple RunInstances. Kpɔ AWS Cost Explorer kple EC2 Global View le wo katã nutowo me be nàtso aʋatso nudzɔdzɔ ɖesiaɖe nu, eye nàʋu AWS Kpekpeɖeŋunya (Akɔnta kple Fexexe -> Akɔntabubu si Wogblẽ) enumake. AWS ɖea asi le akɔntabubu ƒe fe siwo wowɔ le ametafatafa me ŋu edziedzi ne ègblɔ nya vevi aɖe si do go eye nètso nudzɔdzɔwo nu enumake.

2. Fli ( .sk_live_...), OpenAI ( .sk-...), SendGrid, Twilio, kple Slack Dzesiwo¶

• Stripe: Yi Developers -> API keys -> Standard keys / Seɖoƒe safuiwo -> Zi dometsotso etɔ̃awo dzi (...) -> Mia safuia. Tia "Ewu enu fifia" (alo ɖo gaƒoƒo 1 ƒe amenuveve fesre ne èle safui yeyea zãm vevie ɖe wò nuwɔwɔ ƒe nɔnɔme ƒe tɔtrɔwo me fifia ko).
• OpenAI / Anthropic: Yi API Keys dashboard eye nàzi Revoke / Delete dzi enumake ale be automated proxy scrapers mateŋu aɖe dɔlar akpe geɖe ƒe LLM token credits le miniti ʋɛ aɖewo me o.
• GitHub Personal Access Tokens (PATs), Fine-Grained Tokens, kple Deploy Keys: Yi GitHub Settings -> Developer settings -> Personal access tokens eye nàzi Delete / Revoke dzi le token si do go la dzi, emegbe nàlé ŋku ɖe wò habɔbɔa ƒe Audit Log (Settings -> Audit log) na repo.clone, . repo.access, alo webhook wɔwɔ si ŋu womeɖe mɔ ɖo o le nukpɔkpɔ ƒe fesrea me.
• Nyatakakadzraɖoƒe ƒe Kadodo ƒe Kawo (DATABASE_URL / . postgres:// / . mongodb+srv://) kple Redis URLs: Trɔ nyatakakadzraɖoƒea zãla ƒe nyagbe ɣaɣla le nyatakakadzraɖoƒea ƒe mɔ̃a me (ALTER USER app_user WITH PASSWORD 'new_random_secret';), eye emegbe wu TCP kadodo siwo katã le dɔ wɔm siwo le ʋuʋu ɖi fifia siwo nye ɖaseɖigbalẽ xoxoawo tɔ nu (SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE usename = 'app_user' AND pid <> pg_backend_pid(); le PostgreSQL me), elabena nyatakakadzraɖoƒe ƒe nyagbe ɣaɣla tɔtrɔ le PostgreSQL alo MySQL me xea mɔ na yeye ƒe gege ɖe eme ko eye wògblẽa amedzidzela ƒe kadodo si ʋu xoxo la ɖi wònɔa agbe vaseɖe esime woaƒoe.

Akpa B: Nya ɣaɣla la ɖeɖeɖa tso Git ƒe ŋutinya me (git filter-repo & GitHub ƒe cache ɖeɖeɖa) .¶

Ne safui si do go la ku eye wotrɔe le ewɔwɔ me ko la, kɔ ka ɣaɣla la ɖa le wò Git commit ŋutinya me ale be dedienɔnɔ ƒe skannerwo nadzudzɔ nuxlɔ̃ame:

1. Gbugbɔ ŋlɔ nudzraɖoƒe ŋutinya to zazã me git filter-repo (alo BFG Repo-Dzadzɛnyenye):
   • Tsɔ wò nudzraɖoƒea ƒe ahuhɔ̃e ƒuƒlu ƒe nɔnɔmetata yeye aɖe eye nàɖe faɛl si do go la ɖa (le kpɔɖeŋu me, . .env) tso ŋutinya me commit ɖesiaɖe me le alɔdzewo kple tags katã me:

     # 1. Remove a specific file (.env) from all commits in history:
     git filter-repo --invert-paths --path .env
     
     # OR 2. Replace a specific secret string with ***REMOVED*** everywhere:
     echo "sk_live_YOUR_SECRET_STRING==>***REMOVED***" > replacements.txt
     git filter-repo --replace-text replacements.txt
     

2. Zi-zi-zi-tutu ŋutinya si ŋu wokɔ la eye nàkɔ GitHub He Biabia / Ðetsɔtsɔ ƒe nukpɔkpɔ siwo wodzra ɖo ɖe cached me:
   • Tu ŋutinya si ŋu wokɔ la ɖe megbe yi afisi wòtso (git push origin --force --all kple git push origin --force --tags).
   • GitHub cached-commit afɔɖeɖe vevi: Ne èzi ŋutinya si ŋu wokɔ dzi vɔ hã la, amesiame si nya commit SHA URL xoxo si me ŋɔŋlɔdzesi 40 le (github.com/org/repo/commit/abc1234...) ateŋu akpɔ commit xoxo si ŋu womekɔ o la kokoko le GitHub ƒe web UI dzi vaseɖe esime GitHub nawɔ dɔ le server-side gbeɖuɖɔ nuƒoƒoƒu. Te ɖe GitHub Kpekpeɖeŋu (alo zã nudzraɖoƒea ƒe Dedienɔnɔ Ŋuti Aɖaŋuɖoɖo / Kpekpeɖeŋu ƒe mɔnu) kple commit SHA xoxoa tututu eye nàbia wo be ɖe dereferenced/dangling commit ɖa le web cache la me eye woawɔe git gc le adzɔge nudzraɖoƒea.

--- .

Akpa 3: Nusiwo dzɔ le vo me la ɖeɖeɖa git push --force, . git reset --hard, alo alɔdzedɔwɔƒe si woɖe ɖa¶

Ne èƒu du le vo me git push --force eye nètutu wò ƒuƒoƒoa me tɔwo ƒe adzɔgbeɖeɖewo ɖa le main, alo ƒu du git reset --hard HEAD-5 eye wòbu ŋkeke bliboa ƒe nutoa me dɔwɔwɔ, mègatsi dzodzodzoe o, elabena Git metutua nusiwo wotsɔ de asi na enumake gbeɖe o kloe.

Nusita wò "bu" commits gakpɔtɔ bɔbɔ nɔ eme dedie .git/logs (The Reflog) ƒe agbalẽa .¶

Ɣesiaɣi si nèwɔ commit, trɔ alɔdzewo, rebase, alo reset la, Git ŋlɔa esi do ŋgɔ HEAD kple alɔdze ƒe nugbɔ SHA-1/SHA-256 ƒe nufiamɔ̃wo le nutoa me ƒe agbalẽ gã aɖe si woyɔna be Numekugbalẽ (git reflog), eye wònaa commit nu ŋutɔŋutɔ siwo nye tsyɔ̃eviwo nɔa disk dzi hena ŋkeke 14 va ɖo 90 ya teti hafi gbeɖuɖɔwo nuƒoƒoƒu ɖea wo ɖa.

Afɔɖeɖe ɖesiaɖe Git ƒe hayahaya kpata ƒe nuɖaɖawo¶

Nɔnɔme 1: ȃu du git push --force eye woŋlɔ nu ɖe edzi origin/main¶

1. Di commit SHA si tututu origin/main fia asi do ŋgɔ tututu do ŋgɔ na wò ŋusẽ-tutu:
   • Mɔnu A (Kpɔ wò nutoa me remote-tracking reflog): Wɔ sedede sia le terminal si nètu:

     git reflog show origin/main
     

     Àkpɔ emetsonuwo abe:

     a1b2c3d (HEAD -> main, origin/main) refs/remotes/origin/main@{0}: update by push
     9f8e7d6 refs/remotes/origin/main@{1}: fetch: fast-forward
     

     Le afisia la, 9f8e7d6 (origin/main@{1}) nye commit si tututu wowɔ origin/main nɔ edzi hafi wò ŋusẽ-tutua ŋlɔe ɖe edzi tututu.
   • Mɔnu B (Bia ƒuƒoƒo me tɔ ɖesiaɖe si meƒu du o git pull ke hã): Dɔwɔhati ɖesiaɖe si tsɔe vɛ main do ŋgɔ egbea gakpɔtɔ le commit ŋutinya si me gblẽ o la bɔbɔ nɔ woƒe nutoa me origin/main alɔdzedɔwɔƒea.
   • Mɔnu C (Bia GitHub REST API Nudzɔdzɔwo ƒe nuŋlɔɖi): Ne wò teƒea ƒe reflog mekpɔ adzɔge nɔnɔme o la, bia GitHub ƒe push events API be nàkpɔ... "before" SHA hash si nye ŋusẽ-tutu nudzɔdzɔa:

     gh api /repos/OWNER/REPO/events --jq '.[] | select(.type=="PushEvent") | {actor: .actor.login, before: .payload.before, head: .payload.head, time: .created_at}'
     

2. Gbugbɔgaɖoe origin/main na commit SHA si do ŋgɔ la dedie:
   • Kpɔ egbɔ be commit la dze nyuie (git log -n 5 9f8e7d6), emegbe nàtutu SHA ma tututu agbugbɔ ayi adzɔgemɔ̃a dzi main alɔdzedɔwɔƒea zazã --force-with-lease:

     git push --force-with-lease origin 9f8e7d6:refs/heads/main
     

Nɔnɔme 2: ȃu du git reset --hard alo tutu alɔdzedɔwɔƒe aɖe si le mia gbɔ hafi ƒoe¶

1. Duƒuƒu git reflog le miaƒe nutoa me nudzraɖoƒe:

   git reflog --date=iso
   

2. Di commit SHA la do ŋgɔ teti na reset: nya si woŋlɔ (e.g., . HEAD@{2}: commit: feat: add payment webhook), eye nàwɔ xɔname ƒe alɔdze yeye aɖe si fia asi SHA ma tẽ:

   git switch -c rescue-branch HEAD@{2}
   

3. Ke ne èbu uncommitted file si nè... git added (staged) eye emegbe wotutui ɖa kple git reset --hard? ?**
   • Ɣeyiɣi si me nèƒu du git add myfile.ts, Git ƒoa faɛl ma me nyawo katã nu ƒu heŋlɔa wo ɖe eme .git/objects/ abe dangling blob ene, ne mèƒu du kpɔ o gɔ̃ hã git commit. Gbugbɔe to blob siwo katã le tsitrenu siwo woɖo ɖe ɖoɖo nu le tɔtrɔɣi nu la ŋɔŋlɔ me:

     git fsck --lost-found
     # Inspect the recovered blobs written into .git/lost-found/other/
     

--- .

Akpa 4: Nusi dzɔ le vo me terraform destroy alo alilikpo me nudzraɖoƒe si wotutu (S3, EBS, VMs) .¶

Ne Infrastructure-as-Code (IaC) pipeline alo manual CLI sedede dze dɔwɔwɔ ƒe alilikpo me nunɔamesiwo gbãgbã gɔme:

1. Dzudzɔ amewuwua dedie:
   • Ne èle du dzi le nutoa me le wò terminal la, zi ** dzi .Ctrl + C ONCE** ale Terraform wu eƒe API yɔyɔ si li fifia nu eye wòŋlɔa yeyea .tfstate faɛl la dzadzɛ. Ne èle du dzi le GitHub Actions, GitLab CI, alo Terraform Cloud me la, zi Cancel Run dzi eye nàtsi CI duƒula ƒe AWS/GCP IAM akpa dzi enumake le cloud console me ale be le yameʋu me API tutuwo nado kpo nu AccessDenied.
2. Back up fifia ƒe ɣeyiɣia .tfstate faɛl enumake:
   • Wɔ wò adzɔge nɔnɔme ƒe faɛl ƒe kɔpi si le mia gbɔ (terraform state pull > emergency-backup.tfstate) eye nàlé ŋku ɖe S3 bucket versioning ŋu le wò Terraform state bucket dzi be nàxɔ state version si nɔ anyi va yi tso miniti 10 siwo va yi me.
3. Kpɔ alilikpo me dɔwɔƒe "Soft-Delete" kple Recycle Bin dedienɔnɔ ƒe mɔ̃wo ɖa:
   • Amazon S3 (Kple Bucket Versioning si wowɔ): Faɛlwo tutu le S3 bucket si wowɔ tɔtrɔ me (aws s3 rm --recursive) metutua nu ɖeka aɖeke o; ɖeko wòtsɔa zero-byte Delete Marker dea nu ɖesiaɖe tame. Ðe Delete Markers ɖa (to S3 Batch Operations alo 5-line Python dzi boto3 script) be nàgbugbɔ wò faɛlwo ƒe alafa memama 100 aɖo te enumake.
   • AWS EBS Volumes & RDS Snapshots (Recycle Bin): Kpɔ AWS Recycle Bin le console la me ne woɖoe, eye nàlé ŋku ɖe AWS Backup vaults ŋu.
   • Google Cloud Platform (GCP) & Microsoft Azure: GCP Dɔwɔna siwo wotutu kple Azure Dɔwɔnu Ƒuƒoƒo / Safui Vaults / Nudzraɖoƒe ƒe Akɔntabubu siwo Soft Delete wɔ dɔ la anɔ anyi woate ŋu agbugbɔ axɔ hena ŋkeke 7 va ɖo 30 to gbugbɔgaɖoanyi ƒe asiƒoƒo ɖeka dzi le console la me.

Alesi woawɔ axɔ Terraform si gbã .tfstate dukɔa ƒe faɛl alo gbugbɔ tsɔ xɔtuɖaŋu siwo nye tsyɔ̃eviwo tso duta vɛ¶

Ne a terraform apply alo le vo me terraform destroy wotɔ te le yameʋua ƒe titina, alo ne ame aɖe tutu adzɔge nɔnɔme ƒe faɛl le S3/GCS me le vo me la, wò alilikpo ƒe xɔtuɖoɖo gena ɖe ahɔhɔ̃ ƒe mama ƒe nɔnɔme si me afɔku le me afisi nunɔamesi aɖewo gakpɔtɔ le AWS/GCP me esime bubuwo bu le .tfstate:

1. Adzɔgemɔ̃ si woŋlɔ ɖe edzi alo esi wotutu ɖa la gbugbɔgaxɔ .tfstate faɛl tso S3 / GCS ƒe tɔtrɔ:
   • Mègatrɔ asi le JSON ŋu kple asi gbeɖe o .tfstate faɛl kple asi le nuŋɔŋlɔ ɖɔlila me negbe ɖe tiatia bubu ɖesiaɖe do kpo nu, elabena ɖeka mesɔ o "serial" alo "lineage" UUID ka axe mɔ na Terraform be wòagawɔ dɔ ake o.
   • Ʋu wò S3 bucket (alo Google Cloud Storage bucket / Terraform Cloud Workspace States tab) afisi adzɔge megbe megbenyawo le, zi Fia tɔtrɔwo ƒe tɔtrɔ si le S3 console la tame, di... terraform.tfstate version timestamped right before the bad run, wɔ eƒe kɔpi, eye nàtue ɖe megbe dedie abe nɔnɔme yeye si li fifia ene:

     # Push a recovered historical state file back to the remote backend safely
     terraform state push recovered-pre-incident.tfstate
     

2. Dziɖuɖu ƒe gaƒoɖokui si tsi anyi la gbagbã (Error acquiring the state lock):
   • Ne èɖe fu na vɔ̃ɖi aɖe terraform destroy kple Ctrl + C alo wu CI duƒula si wotsɔ he ɖe ati ŋu la, DynamoDB kplɔ̃ si le didiƒe alo GCS ƒe ʋuƒo ƒe faɛl la gakpɔtɔ nye esi wotu to dɔwɔwɔ ƒe ID kukua dzi.
   • Gbã la, kpɔe ɖa le wò CI dashboard me kple wò ƒuƒoƒoa me tɔwo be le nyateƒe me la, Terraform ƒe dɔwɔwɔ bubuwo zero gakpɔtɔ le du dzi ɖe dɔwɔƒe ma ŋu, emegbe nàɖe asi le tsyɔ̃evi ƒe gaƒoɖokui ŋu to Lock ID UUID si wota ɖe vodada ƒe gbedasi me zazã me:

     terraform force-unlock <LOCK_ID_UUID>
     

3. Nuto si ƒe akpa aɖe gblẽ la gbugbɔgatu evɔ womagbugbɔ nyatakakadzraɖoƒe siwo tsi agbe la awɔ o (terraform import):
   • Ne a terraform destroy tutu wò Application Load Balancers siwo me dukɔ mele o (aws_lb) kple ECS/EKS dɔwɔla nodes hafi nètɔ te, ema ɖɔɖɔɖo le bɔbɔe: ƒu du ko terraform apply eye Terraform agbugbɔ awɔ stateless networking kple compute layers katã tso gɔmedzedzea me le miniti 5 me esime wòagblẽ wò RDS database si tsi agbe la ɖi womawɔe o.
   • Ne wò state file bu stateful database alo S3 bucket si tsi agbe la, zã ** .import {} ɖoɖowɔɖi ƒe mɔxenu** (Terraform 1.5+) alo terraform import aws_db_instance.prod_db my-rds-instance-id be nàbla alilikpo me nunɔamesi si li xoxo la ake ɖe wò nɔnɔme ƒe faɛl me evɔ màgblẽe o, eye nàƒu du terraform plan be woaɖo kpe edzi "0 to destroy" hafi nàbiae.

--- .

Akpa 5: Alesi nàgblɔ elektrikŋusẽ ƒe nutsotso aɖe na wò dɔdzikpɔla eye nàkplɔ amekukuɖaka si ŋu fɔɖiɖi aɖeke mele o¶

Ne èhe nuwɔwɔ ƒe nudzɔdzɔ aɖe vɛ esime nèle dɔ wɔm le didiƒe la, wò dɔnyala ƒe ŋkɔ dzena menye vodadaa ŋutɔ me o, ke boŋ to wò nuwɔwɔ le gaglãgbe kabakaba, kadodo si me dziɖeɖi le, kple nuwo ɖɔɖɔɖo ɖe ɖoɖo nu me.

Sikase 3 Siwo Ku Ðe Nudzɔdzɔwo Ŋuti Kadodo Ŋu¶

1. Gblɔ nyateƒenyawo, ŋusẽkpɔɖeamedzi, kple ETA evɔ màɣla, ahe ɖe megbe, alo abu fɔe o: Gblɔ vodadaa le miniti 2 me tso esime nèke ɖe eŋu.
2. Ma "Mitigation" (Stopping the bleeding) tso "Root-Cause Fix" gbɔ: Le nɔnɔme kpata aɖe me la, wò taɖodzinu ɖeka koe nye be nàgbugbɔ subɔsubɔdɔa aɖo anyi (agbugbɔ dɔwɔwɔ gbegblẽa aɖo anyi, ado kpo nu ayi PITR nyatakakadzraɖoƒe ƒe nɔnɔmetata la gbɔ, alo atrɔ nɔnɔme ƒe aflaga la ɖa). Dzra deep code refactoring ɖo na etsɔ.
3. Tsɔ aɖabaƒoƒo 15 ƒe dzi ƒe tsotso ŋuti nyatakaka yeyewo ɖe nudzɔdzɔa ƒe mɔ̃ dzi edziedzi ale be dɔdzikpɔlawo kple asisiwo ƒe kpekpeɖeŋunalawo nanya be ame aɖee le ŋusẽ kpɔm ɖe edzi.

Copy-paste Slack / Teams Nudzɔdzɔwo Ŋuti Nyatakaka Script¶

[SEV-1 INCIDENT] Nuwɔwɔ [Subɔsubɔdɔ / Nyatakakadzraɖoƒe ƒe Ŋkɔ] Ŋusẽkpɔɖeamedzi - Nusi dzɔ: Le [HH:MM UTC], esime wole ewum [brief technical action, e.g., a data migration script], si nye nusi womeɖo be woawɔ o [DELETE / config change / force-push] wozãe le nuwɔwɔ me [database / service]. - Asitsalawo ƒe Ŋusẽkpɔɖeamedzi Fifia: [e.g., Checkout API is returning 500 errors / User profile updates are paused]. - Ele Enumake Nusiwo Woaɖe Ðe Edzi: 1. Ŋlɔ ʋuwo ƒe zɔzɔ ɖe [table/service] wotɔ vie be woatsɔ akpɔ nyatakakawo ƒe blibonyenye ta. 2. Point-In-Time Recovery (PITR) ƒe nɔnɔmetata na [HH:MM:SS UTC] (sɛkɛnd 10 do ŋgɔ na wɔnaa) le tsatsam fifia le [AWS RDS / GCP]. - Nudzɔdzɔwo ƒe Aʋafia: @[Your Name] (si kpe ɖe eŋu @[DBA/On-Call SRE]). - Nya yeye si kplɔe ɖo: Le miniti 15 me ([HH:MM UTC]) alo ne hayahaya ƒe nɔnɔmetata la va le Internet dzi ko.

Mɔ̃ɖaŋununya ƒe ametakpɔnu siwo le nudzɔdzɔa megbe ƒe ɖaseɖigbalẽ (Na wò Amekuku megbe Fɔmamanɔsitɔ) .¶

Amegbetɔ ƒe vodada ɖesiaɖe le nuwɔwɔ me nye ɖoɖowo ƒe ɖoɖowɔwɔ ƒe dometsotso ŋutɔŋutɔ (ne sedede ɖeka si wotsɔ kɔpi kpe ɖe nuwuƒe ƒe sedede ateŋu adzudzɔ ewɔwɔ guardrail manɔmee la, ɖoɖoa nɔ lalam be yeado kpo nu). Do mɔ̃ɖaŋununya ƒe mɔxenu siawo siwo wotsɔ kɔkrit wɔe ƒe susu eye nàwɔ wo ŋudɔ le wò amekuku ŋuti nuŋlɔɖi me le ŋufɔke:

• Amadede-kɔda kple ŋgɔdonya wɔwɔ ƒe nyabiasewo (kube-ps1, . psql .psqlrc): Trɔ asi le wò shell kple SQL client ale be kadodo ɖesiaɖe na PRODUCTION ɖea dzodzro si klẽna fiana [PROD-DB] bia eye wòdzea PostgreSQL ƒe ɣeyiɣi ɖesiaɖe si me wowɔa nu aduadu le gɔme le a BEGIN; asitsatsa ƒe mɔxexe le gɔmedzedzea me (alo ɖoɖowo SET default_transaction_read_only = on;).
• Na wòawɔ dɔ safe-updates (--i-am-a-dummy) le MySQL/MariaDB asitsalawo me: Tsɔ kpe ɖe eŋu safe-updates yi [mysql] le -/.my.cnf eyata MySQL gbe tẽ be yemawɔ ɖeke o UPDATE alo DELETE nyagbɔgblɔ si me a WHERE nyagbe vevi alo a LIMIT.
• Trɔ asi le eŋu DeletionProtection = true le nuwɔwɔ ƒe RDS nyatakakadzraɖoƒewo katã, S3 buckets, kple Terraform stateful resources (lifecycle { prevent_destroy = true }).
• Wɔ Git Alɔdzedɔwɔƒea Takpɔkpɔ Sewo dzi le main / . production: Tsɔe wɔ dɔe Allow force pushes kple Allow deletions le alɔdze siwo wokpɔ ta na le GitHub/GitLab me ale be nudzraɖoƒedzikpɔlawo gɔ̃ hã mateŋui o git push --force yi main.
• De pre-commit adzame scanning hooks (gitleaks alo trufflehog) kple GitHub Push Protection: Block commits siwo me AKIA, . sk_live_, alo .env kpɔɖeŋuwo le nutoa me do ŋgɔ git commit ekpɔa dzidzedze gɔ̃ hã.

--- .

Nya siwo wobiaa amewo enuenu¶

Ðe woaɖe asi le ŋunye le esi meda nusiwo wowɔ ƒe nyatakakadzraɖoƒe ƒe kplɔ̃ le vo me alo na nyatakakadzraɖoƒea tsi anyia?¶

Le mɔ̃ɖaŋudɔwɔƒe siwo le lãmesẽ me siwo wɔa dedienɔnɔ ƒe dekɔnu si ŋu fɔɖiɖi mele o (si sɔ le egbegbe mɔ̃ɖaŋunuwo, DevOps, kple SRE ƒuƒoƒowo me) me la, womeɖea mɔ̃ɖaŋudɔwɔlawo le dɔ me le amegbetɔ ƒe vodada anukwaretɔe le nuwɔwɔ me ta o, nenye be wogblɔ nudzɔdzɔa enumake eye wokpe asi ɖe subɔsubɔdɔa gbugbɔgaɖoanyi ŋu ko. Mɔ̃ɖaŋudɔwɔlawo ƒe ŋgɔnɔlawo kpɔe dze sii be ne mɔɖeɖe xoxowo kple dzɔlawo ƒe anyimanɔmanɔ nɔ dɔwɔla sue aɖe si be wòatsɔ sedede ɖeka si ŋu womelé ŋku ɖo o aƒu gbe ɖe nuwɔwɔ ƒe kplɔ̃ dzi la, ke nusi gbɔ wòtsoe nye vodada si le mɔɖeɖe ɖe nu ŋu kple dɔwɔnu siwo wotsɔ dea dɔwɔwɔ me ŋu, ke menye vɔ̃ɖinyenye o. Le nyateƒe me la, mɔ̃ɖaŋudɔwɔla si kpɔ elektrikŋusẽ ƒe nutsotso ma teƒe teti koe nye ema lae nye ame ɖeka kolia si le dedie wu le ƒuƒoƒoa me ku ɖe ɖoɖo ma ŋu fifia. Nuwɔna ɖeka kolia si hea nuwuwu vɛ ɣesiaɣie nye agbagbadzedze be woatsyɔ nu nyatakaka siwo bu ƒe nudzɔdzɔ aɖe dzi, atutu agbalẽdzikpɔkpɔ ŋuti nuŋlɔɖiwo ɖa, alo alakpadada le nudzɔdzɔa me dzodzro me.

Nukatae mele be matrɔ nye live production traffic enumake ayi Point-In-Time Recovery (PITR) database instance yeye si wogbugbɔ ɖo te o?¶

Ne ɖeko nètutu fliwo ɖa le kplɔ̃ ɖeka dzi le vo me (le kpɔɖeŋu me,... `user_preferences` table) le 14:00 UTC, eye wòxɔa miniti 20 hafi woate ŋu atrɔ PITR clone si wogbugbɔ ɖo 13:59:50 UTC, wò live production database ate ŋu ayi edzi awɔ dɔ tso asisiwo ƒe nudɔdɔ kple fexexe yeye akpe geɖe siwo le se nu ŋu le `orders` kplɔ̃ le ga 14:00 kple 14:20 UTC dome. Ne èʋuʋu wò nuwɔwɔ ƒe DNS ƒe nuwuƒe bliboa ŋkumaʋumaʋui yi 13:59:50 PITR ƒe nɔnɔmetata la dzi la, àtutu aɖabaƒoƒo 20 mawo siwo nye asisiwo ƒe nudɔdɔ yeyewo ɖa. Ke boŋ, na nyatakakadzraɖoƒe vevitɔa nanɔ internet dzi (kple nuŋɔŋlɔ siwo woatɔ ɖe nɔnɔme si gblẽ la dzi ko) eye nàɖo fli siwo bu la ko tso 13:59:50 PITR ƒe nɔnɔmetata la me ɖe agbe ƒe wɔwɔme kplɔ̃a dzi.

Ne mewɔ dutoƒo GitHub nudzraɖoƒe Private le miniti eve megbe le vo me tutu an .env file, ɖe wògahiã kokoko be matrɔ API safuiawo katãa?¶

Ẽ, ele kokoko be woaɖe asi le safui mawo ƒe alafa memama 100 ŋu ahatrɔ wo enumake. Woɖea adzɔgbeɖeɖe ɖesiaɖe si wotu ɖe dutoƒo GitHub nudzraɖoƒe aɖe ɖe go le ɣeyiɣi ŋutɔŋutɔ me to dutoƒo GitHub Nudzɔdzɔwo API ("Firehose") dzi eye wowɔa index le sɛkɛnd ʋɛ aɖewo me to dedienɔnɔ ƒe nudzraɖoƒe dɔwɔƒe siwo le se nu (abe GHArchive ene) kple ɖaseɖigbalẽwo ŋeŋe ƒe bots vɔ̃ɖi siwo wɔa dɔ le wo ɖokui si siwo le dɔ wɔm 24/7 le alilikpo me kadodowo dzi siaa dzi. Ne ètrɔ nudzraɖoƒea ƒe dzedzeme tso Dutoƒo yi Ame ŋutɔ gbɔ le aɖabaƒoƒo eve megbe la, metutu kɔpi si automated scrapers ɖe xoxo la ɖa o.

Aleke wòwɔnae git push --force-with-lease xe mɔ ɖe elektrikŋusẽ ƒe nutsotso nu ne wotsɔe sɔ kple git push --force?¶

Ne èƒu du standard `git push --force`, Git ŋlɔa alɔdzedɔwɔƒe si le didiƒe la ɖe alɔdzedɔwɔƒe si le mia gbɔ la dzi nɔnɔme aɖeke manɔmee, eye wògblẽa adzɔgbeɖeɖe ɖesiaɖe si ƒuƒoƒo me tɔ aɖe tu ɖe alɔdzedɔwɔƒe ma le ɣeyiɣi ma me la dome le ɖoɖoezizi me. Ne èzãe `git push --force-with-lease` (eye wònyo wu gɔ̃ hã, . `--force-if-includes`), Git léa ŋku ɖe adzɔge ʋĩ ƒe dɔwɔƒea ŋu gbã be yeakpɔe ɖa be adzɔge alɔdzedɔwɔƒea ƒe aɖaŋuɖoɖo gakpɔtɔ sɔ kple commit SHA si tututu wò nutoa me mɔ̃a xɔ zi mamlɛtɔ hã. Ne dɔwɔhati alo CI pɔmpi aɖe ƒo commit yeye ɖe alɔdze si le didiƒe tso esime nèxɔ mamlɛtɔ la, . `--force-with-lease` eɖea fu dedie kple "stale info" nuxlɔ̃ame tsɔ wu be wòatutu woƒe dɔa ɖa.

--- .

Wolé ŋku ɖe dzɔtsoƒewo ŋu¶

• Dukɔa ƒe Dzidzenuwo Kple Mɔ̃ɖaŋununya Dɔwɔƒe (NIST, United States): NIST SP 800-61 Rev. 2 / Rev. 3 Kɔmpiuta Dedienɔnɔ ƒe Nudzɔdzɔwo Gbɔkpɔkpɔ ƒe Mɔfiame kple Ðaseɖigbalẽwo ƒe Nugblẽfexexe.
• Amazon Nyatakakadzraɖoƒe Dɔwɔƒe (AWS), Google Cloud Platform (GCP), kple Microsoft Azure Documentation (United States): RDS ƒe Ðoɖowɔɖi si Wowɔna le Ɣeyiɣi Ðeka Me (PITR) ƒe Xɔtuɖaŋu, AWS IAM ƒe Ðoɖo Vevi Siwo Woɖe Ðe Vovo (AWSCompromisedKeyQuarantineV2), kple S3 Versioning Delete-Dzesi Gbugbɔgaxɔ.
• PostgreSQL Xexeame Katã ƒe Ŋgɔyiyi Ƒuƒoƒo kple MySQL / Oracle Nuŋlɔɖiwo (Dukɔwo Dome): *PostgreSQL Ŋɔŋlɔ-Do ŋgɔ Nuŋlɔɖi (WAL), MVCC Tuple Nukpɔkpɔ, kple MySQL Row-Based Binary Log Flashback Gbugbɔgaxɔ *.
• Git SCM Dziɖuɖua ƒe Nuŋlɔɖiwo kple GitHub Dedienɔnɔ Dɔwɔƒe (Dukɔwo Dome): Git Reflog (git-reflog), Nu Siwo Le Ʋuʋudedi Gbugbɔgaxɔ (git-fsck), Ŋutinya Gbugbɔgaŋlɔ (git-filter-repo), kple Adzame Scanning Push Takpɔkpɔ.
• USENIX Habɔbɔ / SREcon kple Google Site Reliability Engineering (SRE) Agbalẽwo (Dukɔwo Dome): Nudzɔdzɔwo Dzikpɔkpɔ, Kpata ƒe Rollback Triage, kple Amekuku megbe Dekɔnu si ŋu fɔbubu aɖeke mele o.

Mɔfiame siwo do ƒome kplii¶

• Adzɔge dɔwɔƒe ƒe dedienɔnɔ kple nyatakakawo ƒe adzamenyawo gbɔ kpɔkpɔ kpata
• Kɔmpiuta dzi backups 3-2-1 aɖaŋu kple afɔkuwo gbugbɔgaxɔ
• Fail tutu le vo me kple ʋu si woɖo ɖe ɖoɖo nu gbugbɔgaxɔ
• Web hosting xaxa ƒe ʋuʋu kple backup gbugbɔgaɖoanyi
• Fishing kple akɔntabubu siwo ŋu woda asi ɖo
• Xexlẽdzesi siwo woatsɔ awɔ nudzɔdzɔ kpatawoe