Dipanga dia mudimu dia mpukapuka, dipatuka dia mafungu a API, anyi dijimija dia bintu bia mu matutu¶

Muena mudimu yonso wa programe ya ordinatere, mulombodi wa malu a ordinatere, ne muena mudimu wa ku bule yonso udi upeta tshikondo tshia dibungama padi dîyi difila bua localhost anyi staging udi wenza malu mabi production: unindexed DELETE kakuyi WHERE tshikoso, a DROP TABLE anyi terraform destroy, a git push --force udi ufunda tshitupa tshinene, anyi tshipungidi tshia patoke tshidi ne AWS anyi Stripe secret keys. Tshidi tshitapulula tshinyangu tshikese tshia minite 15 ne tshipupu tshia ndekelu tshia mudimu ntshia kuimanyika bianza biebe diakamue, kuimanyika mudimu wa difunda kumpala kua bipese bia kulama bifunda, ne kuenza mudimu muimpe wa dipingaja dia bintu.

Enza nunku mpindieu¶

1. Kukanga bianza biebe ne KUENZEDI mêyi a "kulongolola" lukasa, enza mudimu ne tshilaminu tshia malu, anyi kanga dididishi diebe dia terminal. Buffer wa terminal webe wa mpindieu, malu a kale a shell, ne muaba uudi musombele .git/logs reflog udi na bikondo bia mudimu, commit SHA hashes, ne resource IDs idi ikengedibua bua kupingaja dinyanguka. Kumpala, funda tshiamu tshieba tshijima tshia terminal scrollback mu tshikandakanda tshia bukubi.
2. Manyisha bualu bua Severity-1 (SEV-1) mu tshisanji tshia Slack/Teams tshia tshisumbu tshiebe mu minite 2. Kupitshishi minite 45 nansha kakese bua kuteta bua kulongolola mu musokoko tshilaminu tshia bintu tshidi tshinyanguke nkayebe padi bantu badi basumba bintu benza bilema 500 ne midimu ya panyima idi inyanga mesa adi mashale. Udi mua kubikila muena mudimu wa tshibutshilu tshia malu (DBA) anyi muena mudimu wa malu a kueyemena (SRE) diakamue.
3. Biwikala mupatule tshifungu tshia API, .env tshikandakanda, anyi tshisokomenu tshia mu tshilaminu tshia GitHub tshia bantu bonso, KUANGA tshifungu tshidi mu tshiamu tshia mudimu tshia mufidi (kumpala kua kuteta kufunda kabidi malu a kale a Git). Botnets idi yenza mudimu nkayayi idi ikosolola API wa GitHub Firehose wa bantu bonso ne yenza mudimu ne AWS, OpenAI, Stripe, ne GCP miiba mu 3 too ne ku 15 sekonde ya wewe git push. Kujimija tshikandakanda mu tshikondo tshibidi anyi kuenzeja ku bukole kakuena kupangisha muena tshinyangu uvua mumane kuenza clone wa tshifungu to.
4. Wewe muenze bualu bua mpukapuka DELETE, UPDATE, anyi DROP pa tshilaminu tshia bintu, upuekeshe difunda dia programe diakamue. Teka programe wa ku Enternete mu mushindu wa dilama anyi umbusha bukenji bua kufunda bua muena mudimu wa programe bua malu mapiamapia adi alua kaayi afunda pa mutu pa tuple mifue idi kayiyi milengeja anyi akolesha Point-In-Time Recovery (PITR).
5. Kunyema nansha kakese VACUUM, OPTIMIZE TABLE, anyi git gc --prune=now kunyima kua dijimija dia mpukapuka. Bionso bibidi bia malanda (PostgreSQL MVCC, MySQL InnoDB) ne Git bidi bilama milongo mijimija ne bintu bia tshianana bidi kabiyi binyanguke pa diske too ne padi disangisha dia bintu bia bukoya/dia tshianana dipingaja mabeji.
6. Bikala terraform destroy anyi script wa dijimija dia matutu utshidi wenza mudimu mu terminal webe, fina Ctrl + C MUSANGU UMUE (ki misangu ibidi to) anyi kushipa tshimanyinu tshia tshisangilu tshia IAM. Kufina Ctrl + C musangu umua udi ubanga dilengeja dia nsombelu muimpe nunku Terraform udi ulama tshitupa .tfstate tshia tshidi tshitshidiku; kufina Ctrl + C misangu ibidi udi wenza bua dijimija dia difu didi mua kunyangakaja anyi kujimija mukanda webe wa ditunga.

Mudimu wa tshimpitshimpi wa disungula: dijimija dia tshilaminu tshia malu, dienzeja dia git, anyi dipatuka dia malu masokoka¶

Londa mutshi wa dipangadika edi bua kufila minite 5 ya kumpala ya dikuba dia malu a tekiniki.

flowchart TD
    A["Tshilumbu tshia tshimpitshimpi tshia malu a dienza dia bintu"] --> B{"Ndipanga kayi didi dienzeke mpindieu?"}
    B -->|"SQL DROP / DELETE / UPDATE in Prod"| C["Lekela App Writes / Maintenance Mode; KUENJI VACUUM"]
    B -->|"API Key / .env / Private Key mu Git"| D["Kumbusha ne kunyungulula tshisokomenu mu Cloud Dashboard mu tusunsa 60"]
    B -->|"Git push --force / Deleted Branch"| E["Enza git reflog Locally; Copy Last Good Commit SHA"]
    B -->|"Kubutula terraform mu mpukapuka / Kujimija S3 anyi VM"| F["Shipa tshisangilu tshia IAM anyi Fina Ctrl+C musangu umua; Lama .tfstate"]
    C --> G["Funda tshimanyishilu tshia dîba dia UTC; Trigger Cloud PITR Clone anyi Binlog Flashback"]
    D --> H["Tangila CloudTrail / Access Logs bua Spin-Ups idi kayiyi mianyishibue; Scrub Repo na git-filter-repo"]
    E --> I["Kulomba GitHub Events / Reflog & Restore Branch na git push --force-with-lease"]
    F --> J["Tangila S3 Versioning Delete Markers, EBS Snapshots, anyi Cloud Recycle Bin"]

Tshitupa 1: Tshilaminu tshia bintu tshia mpukapuka DROP, TRUNCATE, anyi DELETE kupanga WHERE¶

Padi lukonku lua SQL lunyanguka lubuela mu tshilaminu tshia bintu, mushindu mulelela wa kutshipingaja udi ulonda tshiamu tshieba tshia tshilaminu tshia bintu (PostgreSQL, MySQL/MariaDB, SQL Server, MongoDB, anyi DynamoDB) ne ni meyi avua DML (DELETE / UPDATE) anyi DDL (DROP TABLE / TRUNCATE) bualu.

Tshitupa 1: Funda tshimanyishilu tshia dîba tshia UTC too ne ku tshibidi¶

Kumpala kua kulenga tshintu tshionso tshia kulama, funda dîba dia UTC dia mpindieu (date -u mu Linux/macOS, anyi SELECT NOW(); munda mua tshikebelu tshia SQL) ne kuela meji bua sekonde menemene kumpala kua lukonko luebe lua dibutula kutuadijalu kuenza mudimu. Point-In-Time Recovery (PITR) udi wenza mudimu pa kupingaja tshikandakanda tshia Write-Ahead Log (WAL mu PostgreSQL) anyi Binary Log (binlog mu MySQL) too ne T_minus_1_second kumpala kua lukonku lueba lubi kufikalu.

Tshitupa 2: Lekela midimu ya panyima ne difunda dia programe (Lama tuples ya MVCC)¶

Mu PostgreSQL, kuenza DELETE FROM users; anyi UPDATE orders SET status = 'cancelled'; ki udi mua kujimija nansha byte umua wa malu mu tshikandakanda tshia mesa tshidi pa diske diakamue to. Bualu PostgreSQL udi wenza mudimu ne Multi-Version Concurrency Control (MVCC), a DELETE udi anu uteka xmax mutu wa ID wa tshienzedi pa mulongo wonso bua kumuleja bu udi kayi mua kumueneka ku nkonko ya matuku atshilualua, kushiya mulongo mujima wa malu munda mua base/ tshibutshilu tshia mukanda too ne padi autovacuum ulua bua kuleja muaba udibu mua kuenza nende mudimu kabidi.

• Biwikala pa self-hosted nansha VM PostgreSQL kadi kuyi ne tshilaminu tshia matuku mashale aa: Diakamue udisangisha bu superuser ne uleke autovacuum pa mesa au (ALTER TABLE table_name SET (autovacuum_enabled = false);), anyi wimanyika mudimu wa PostgreSQL ne wangate tshimfuanyi tshia tshiamu tshia mudimu (LVM / EBS tshimfuanyi) tshia /var/lib/postgresql/data tshilaminu. Biamu bia mudimu bia pa buabi bia PostgreSQL (pg_dirtyread) udi mua kubala ne kupingaja 100 pa lukama ya milongo mijimija anyi milongolola kumpala kua dîba buludiludi mu mesa a tshibungubungu adi kaayi malengeja mu minite mikese.

Matrice wa dipingaja dia tshilaminu tshia bintu ku diambuluisha dia motere ne tshiamu tshia matutu¶

Tshitupa 2: Nshinga ya API mipatuke, .env malu masokoka, anyi nshinga ya pa nkaya mifila kudi Git¶

Shi uvua muenze bualu kampanda ku mpukapuka .env tshikandakanda, an id_rsa SSH key, Firebase/GCP service account JSON file, Stripe secret key (sk_live_...), anyi AWS AKIA... bua kubuela mu Git, londa ndondo wa bitupa bibidi ebi: Tshitupa A (Shipa tshifungu mu ditu) tshia kumpala, ne Tshitupa B (Scrub the Git repository) tshibidi.

Tshitupa A: Diakamue dia 60-sekonde dia dilama dia malu a dianyisha dia mu matutu (Enza bualu ebu KUMPALA)¶

1. AWS (AKIA... ID wa tshifungu tshia kubuela + Tshifungu tshia kubuela mu musokoko)¶

Padi bot wiba tshifungu tshia AWS ku GitHub, script wabu udi wenza mudimu nkayende udi ubuela mu konte webe mu masekonde 10 bua kupatula GPU EC2 ya bunene bua bungi (p4d.24xlarge / g5.48xlarge) mu bitupa bionso bia AWS bia pa buloba bujima 30+ (kusangisha ne bitupa bia opt-in bu mudi Jakarta, Osaka, ne Moyen Orient) bua kupeta makuta a tshiamu tshia arjan, anyi kuenza muena mudimu wa IAM wa kunyima ne AdministratorAccess kumpala kua wewe kujimija tshifungu tshia ntuadijilu.

1. Kujika ne kujimija tshiamu tshia Access Key tshidi tshipatuke diakamue:
   • Fungula AWS Console -> IAM -> Users -> [Compromised User] -> Security credentials -> Access keys -> Fina pa Deactivate, pashishe Delete (anyi ku diambuluisha dia CLI: aws iam update-access-key --access-key-id AKIAIOSFODNN7EXAMPLE --status Inactive --user-name MyUser).
2. Lamakaja bualu budi bumueneka patoke DenyAll mikenji ya kujikija tshitupa tshîpi aws sts bimanyinu bia tshisangilu:
   • Buteyi bua mushinga mukole: Nansha wewe mumane kujimija AKIA... IAM access key, pikala muena tshibawu ukadi mubikile sts:GetSessionToken anyi sts:AssumeRole mu masekonde 30 a kumpala, tshikondo tshiabu tshia tshitupa tshîpi ASIA... tshimanyishilu tshia tshisangilu tshidi tshishala ne mushinga too ne ku meba 1 too ne ku 12 pa kumbusha wewe muele mukenji wa Kubenga (bu mudi AWSCompromisedKeyQuarantineV2 anyi mukenji wa kubenga bionso bua bimanyinu bifila kumpala kua tshikondo tshia mpindieu) kudi muena mudimu anyi mudimu.
3. Tangila AWS CloudTrail bua kulama tshiibi tshia panyima:
   • Fungula AWS CloudTrail -> Event history ne sungula ku Access Key ID anyi User name mu meba 2 adi mapita.
   • Tangila patoke bua mabikila 5 aa: CreateUser, CreateAccessKey, AttachUserPolicy, AssumeRole, ne RunInstances. Tangila AWS Cost Explorer ne EC2 Global View mu bitupa bionso bua kujikija bilumbu bionso bia dishima, ne unzulula diakamue AWS Support Case (Account and Billing -> Compromised Account). AWS utu ulekela bitadi bia dishima bia dishima paudi ufila luapolo lua dinyanguka dia tshifungu ne ujikija malu ne lukasa.

2. Mulonda (sk_live_...), OpenAI (sk-...), SendGrid, Twilio, ne Slack Tokens¶

• Stripe: Enda ku Developers -> API keys -> Standard keys / Restricted keys -> Click the three dots (...) -> Roll key. Sungula "Expire now" (anyi tela dîba 1 anu pikala udi wenza mudimu ne tshifungu tshipiatshipia mu bintu biebe bia mudimu mpindieu).
• OpenAI / Anthropic: Enda ku tshibuelelu tshia API Keys ne tuaku pa Revoke / Delete diakamue bua biamu bia proxy bidi bienza mudimu nkayabi kabiyi mua kujimija binunu bia dolare ya LLM token credits mu minite mikese.
• GitHub Personal Access Tokens (PATs), Fine-Grained Tokens, and Deploy Keys: Enda ku GitHub Settings -> Developer settings -> Personal access tokens ne tuaku pa Delete / Revoke pa tshimanyinu tshidi tshipatuke, pashishe tangila Audit Log wa bulongolodi buebe (Settings -> Audit log) bua repo.clone, repo.access, anyi dienza dia webhook didi kadiyi dianyishibue mu tshikondo tshia dimanyisha.
• Mifundu ya disangisha dia tshilaminu tshia bintu (DATABASE_URL / postgres:// / mongodb+srv://) ne Redis URLs: Shintulula meyi a tshisokomenu a muena mudimu wa tshilaminu tshia bintu munda mua tshiamu tshia tshilaminu tshia bintu (ALTER USER app_user WITH PASSWORD 'new_random_secret';), ne pashishe kujikija malanda onso a TCP adi menzeke mpindieu adi atangila mikanda ya kale (SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE usename = 'app_user' AND pid <> pg_backend_pid(); mu PostgreSQL), bualu bua kushintulula mot de passe wa tshilaminu tshia bintu mu PostgreSQL anyi MySQL kudi anu kupangisha bipiabipia bia kubuelamu ne kushiya dipetangana dia muena tshinyangu didi dikale diunzulula too ne padidi dibutula.

Tshitupa B: Kumbusha bualu busokoka bua mu malu a kale a Git (git filter-repo & GitHub cache eviction)¶

Padi tshifungu tshidi tshipatuka tshifua ne tshienda tshinyunguluka mu mudimu, umbusha meyi masokoka mu histoire webe wa Git commit bua biamu bia bukubi bileke kudimuija:

1. Funda kabidi malu a kale a tshilaminu tshia bintu ku diambuluisha dia git filter-repo (anyi BFG Repo-Cleaner):
   • Angata tshimfuanyi tshipiatshipia tshia tshilaminu tshieba ne umbusha mukanda udi mupatuke (tshilejilu, .env) ku tshipungidi tshionso tshia kale mu matamba onso ne bimanyinu:

     # 1. Remove a specific file (.env) from all commits in history:
     git filter-repo --invert-paths --path .env
     
     # OR 2. Replace a specific secret string with ***REMOVED*** everywhere:
     echo "sk_live_YOUR_SECRET_STRING==>***REMOVED***" > replacements.txt
     git filter-repo --replace-text replacements.txt
     

2. Force-push the cleaned history ne purge cached GitHub Pull Request / Commit views:
   • Pingaja malu a kale adibu balengeje ku ntuadijilu (git push origin --force --all ne git push origin --force --tags).
   • Tshitupa tshia mushinga tshia GitHub cached-commit: Nansha wewe mumane kuenzeja malu adibu basukule, muntu yonso udi mumanye URL wa SHA wa kale wa maleta 40 (github.com/org/repo/commit/abc1234...) udi mua kumona tshipungidi tshia kale tshidi katshiyi tshilengejibue pa web UI wa GitHub too ne padi GitHub wenza mudimu wa disangisha dia bintu bia bukoya ku luseke lua serveur. Udi mua kubikila GitHub Support (anyi wenze mudimu ne tshibuelelu tshia Security Advisory / Support tshia tshilaminu) ne SHA wa kale menemene ne ubalombe bua kumbusha dereferenced/dangling commit mu cache wa web ne kuenza mudimu git gc pa tshilaminu tshia kule.

Tshitupa 3: Kupingaja bualu bua mpukapuka git push --force, git reset --hard, anyi filiale mufute¶

Wewe munyeme ku mpukapuka git push --force ne mushipe malu a bena tshisumbu nebe main, anyi kunyema git reset --hard HEAD-5 ne mujimije dituku dijima dia mudimu wa muaba, kutshinyi to, bualu Git katu ujimija bintu bia mudimu diakamue to.

Bua tshinyi bintu biebe "bijimija" bitshidi anu bisombe bimpe munda .git/logs (The Reflog)¶

Dîba dionso diudi wenza commit, ushintulula matamba, rebase, anyi reset, Git udi ufunda malu a kale HEAD ne bimanyinu bia SHA-1/SHA-256 munda mua mukanda wa muaba udibu babikila ne: Reference Log (git reflog), ne udi ulama bintu bia mudimu bidi kabiyi ne baledi pa diske munkatshi mua matuku 14 too ne ku 90 kumpala kua disangisha dia bintu bia bukoya kubimbusha.

Mishindu ya kulongolola malu mu tshimpitshimpi ya Git¶

Tshilejilu 1: Uvua munyeme git push --force ne mufunda origin/main¶

1. Keba tshijadiki tshijalame tshia SHA tshidi origin/main udi uleja ku dia balume kumpala kua dikuma diebe dia bukole:
   • Mushindu A (Tangila reflog wa dilonda dia ku bule): Enza mudimu ne dîyi edi mu terminal muudi muele:

     git reflog show origin/main
     

     Neumone tshipeta bu mudi:

     a1b2c3d (HEAD -> main, origin/main) refs/remotes/origin/main@{0}: update by push
     9f8e7d6 refs/remotes/origin/main@{1}: fetch: fast-forward
     

     Apa, 9f8e7d6 (origin/main@{1}) i tshijadiki tshilelela origin/main uvua pa mutu kumpala kua force-push webe kumufunda.
   • Mushindu B (Ebeja muena tshisumbu yonso udi kayi muanji kunyema git pull kadi): Muena mudimu netu yonso uvua muangate main Ku ntuadijilu kua lelu utshidi anu ne malu a kale adi kaayi manyanguke adi mu tshimenga tshiabu origin/main ditamba.
   • Mushindu C (Ebeja mukanda wa malu a GitHub REST API): Shi reflog webe kena ne tshitupa tshia ku bule, lomba API wa push events wa GitHub bua kumona "before" SHA hash wa tshienzedi tshia force-push:

     gh api /repos/OWNER/REPO/events --jq '.[] | select(.type=="PushEvent") | {actor: .actor.login, before: .payload.before, head: .payload.head, time: .created_at}'
     

2. Kupingaja origin/main ku tshitupa tshia kumpala tshia SHA bimpe:
   • Tangila mudi tshipungidi tshimueneka tshimpe (git log -n 5 9f8e7d6), pashishe upingaje SHA au ku tshiamu tshia kule main filiale wenza mudimu ne --force-with-lease:

     git push --force-with-lease origin 9f8e7d6:refs/heads/main
     

Tshilejilu 2: Uvua munyeme git reset --hard anyi mufute filiale wa muaba udibu kumpala kua kuela¶

1. Nyema git reflog mu tshilaminu tshieba:

   git reflog --date=iso
   

2. Keba tshijadiki tshia SHA kumpala kua reset: dibuela (e.g., HEAD@{2}: commit: feat: add payment webhook), ne kuenza tshitupa tshipiatshipia tshia dipandisha tshidi tshileja buludiludi ku SHA au:

   git switch -c rescue-branch HEAD@{2}
   

3. Ntshinyi tshiudi mua kuenza wewe mujimije mukanda udi kauyi mujadikibue uudi git added (staged) ne pashishe kujimija ne git reset --hard?
   • Dîba diudi unyema git add myfile.ts, Git udi ufila ne ufunda malu onso adi mu mukanda au mu .git/objects/ bu tshintu tshidi tshilenga, nansha wewe kuyi muanji kunyema git commit. Upingaje pa kufunda bipese bionso bidi bilenga bilondeshela dîba dia dishintulula:

     git fsck --lost-found
     # Inspect the recovered blobs written into .git/lost-found/other/
     

Tshitupa 4: Tshipupu terraform destroy anyi tshilaminu tshia mu matutu tshijimija (S3, EBS, VMs)¶

Shi Infrastructure-as-Code (IaC) pipeline nansha manual CLI command ubanga kubutula bintu bia mu matutu a mudimu:

1. Lekela dishipangana bimpe:
   • Shi udi wenza mudimu muaba unudi mu tshiamu tshieba, fina Ctrl + C MUSANGU UMUE nunku Terraform udi ujikija dibikila diende dia API dia mpindieu ne ufunda malu adibu bashintulule .tfstate file mukezuke. Shi udi wenza mudimu munda mua GitHub Actions, GitLab CI, anyi Terraform Cloud, tuaku pa Cancel Run ne ujimije diakamue mudimu wa CI runner wa AWS/GCP IAM mu cloud console bua in-flight API deletes kupanga ne AccessDenied.
2. Kulama tshidi mpindieu .tfstate funda diakamue:
   • Teleja kopi wa muaba uudi musombele wa tshikandakanda tshiebe tshia ku bule (terraform state pull > emergency-backup.tfstate) ne tangila S3 bucket versioning pa Terraform state bucket webe bua kuangata version wa state wa kumpala wa minite 10 mimane kupita.
3. Tangila tshiamu tshia mudimu tshia "Soft-Delete" ne Recycle Bin:
   • Amazon S3 (Ne Bucket Versioning mulongolola): Kujimija mikanda mu S3 bucket (aws s3 rm --recursive) kena ujimija tshintu nansha tshimue; udi anu ubueja Delete Marker wa zero-byte pa mutu pa tshintu tshionso. Umbusha bimanyinu bia dibutula (ku diambuluisha dia S3 Batch Operations anyi 5-line Python boto3 script) bua kupingaja 100 pa lukama ya mikanda yebe diakamue.
   • AWS EBS Volumes & RDS Snapshots (Recycle Bin): Tangila AWS Recycle Bin mu tshiamu tshia mudimu pikalaye mulongolola, ne tangila AWS Backup vaults.
   • Google Cloud Platform (GCP) & Microsoft Azure: Deleted GCP Projects and Azure Resource Groups / Key Vaults / Storage Accounts with Soft Delete enabled remain recoverable for 7 to 30 days by one restore click in console.

Mushindu kayi wa kupeta Terraform munyanguke .tfstate tshikandakanda tshia ditunga anyi kubueja kabidi bintu bia mudimu bidi kabiyi ne baledi¶

Padi terraform apply anyi ku mpukapuka terraform destroy udi unyanguka munkatshi mua luendu, anyi padi muntu ujimija tshikandakanda tshia kule mu S3/GCS ku mpukapuka, tshiamu tshieba tshia mu matutu tshidi tshibuela mu nsombelu wa njiwu wa ditapuluka dia buongo mudi bintu bikuabu bitshidiku mu AWS/GCP eku bikuabu kabiyi mu .tfstate:

1. Kupingaja tshiamu tshia ku bule tshidibu bafunde anyi bafute .tfstate tshikandakanda tshia S3 / GCS Versioning:
   • Kuena ne bua kushintulula JSON ku bianza nansha kakese .tfstate tshikandakanda ku bianza mu tshiamu tshia kufunda natshi mikanda pa kumbusha pikala mishindu mikuabu yonso mipangile, bualu bualu bumue budi kabuyi bupetangana "serial" anyi "lineage" Mulongo wa UUID udi mua kupangisha Terraform bua kuenza mudimu kabidi.
   • Kangula tshibombi tshiebe tshia S3 (anyi tshibombi tshia Google Cloud Storage / Terraform Cloud Workspace States tab) muaba udi tshiamu tshia kunyima tshia kule, tuaku pa Leja mishindu tshidi ku mutu kua tshiamu tshia S3, keba terraform.tfstate tshitupa tshia tshikondo tshidi tshimanyishibua kumpala kua tshienzedi tshibi, wangate, ne upingaje bimpe bu muditshi tshipiatshipia:

     # Push a recovered historical state file back to the remote backend safely
     terraform state push recovered-pre-incident.tfstate
     

2. Kutshibula tshiibi tshia tshiibi (Error acquiring the state lock):
   • Paudi umbusha difu dibi terraform destroy ne Ctrl + C anyi kushipa CI runner, mesa a DynamoDB anyi GCS lock file udi ushala mukanga kudi ID wa mudimu mufue.
   • Tuanji kujadika mu CI dashboard webe ne kudi bena mudimu nebe ne: kakuena malu makuabu a Terraform adi enzeka muaba au to, pashishe uleke tshikunji tshidi katshiyi tshienza mudimu ne Lock ID UUID mufunda mu mukenji wa tshilema:

     terraform force-unlock <LOCK_ID_UUID>
     

3. Kuibakulula muaba uvua munyanguke ndambu kuyi wenza kabidi nzubu ya malu idi mishale (terraform import):
   • Shi terraform destroy uvua mubutule biamu biebe bia mudimu bidi kabiyi ne ditunga (aws_lb) ne ECS/EKS worker nodes kumpala kua wewe kuimanyika, tshidi tshipepele bua kulongolola: enza anu mudimu terraform apply ne Terraform neenze kabidi mishindu yonso ya dienza dia mudimu ne ya dibala dia bintu ku mbangilu mu minite 5 eku ushiya tshilaminu tshieba tshia RDS tshidi tshishale katshiyi tshilengejibue.
   • Shi tshikandakanda tshieba tshia ditunga tshijimije tshilaminu tshia tshilaminu tshia malu tshidi tshishale anyi tshibombi tshia S3, enza mudimu ne import {} tshitupa tshia dilongolola (Terraform 1.5+) anyi terraform import aws_db_instance.prod_db my-rds-instance-id bua kupingaja bintu bia mu matutu bidiku mu tshikandakanda tshieba kuyi ubibutula, ne wenze terraform plan bua kujadika "0 to destroy" kumpala kua kulomba.

Tshitupa 5: Mushindu kayi wa kumanyisha mfumuebe wa mudimu bualu bua dipanga dia nzembu ne kulombola tshikebelu tshia lufu tshidi katshiyi ne bualu¶

Paudi wenza bualu kampanda paudi wenza mudimu ku bule, lumu luebe lua mudimu ludi lumvuija ki ku tshilema nkayatshi to, kadi kudi lubilu luebe lua dienza malu patoke, diyukila dimpe ne dilongolola dia malu mu mushindu mulongolola.

Mikenji isatu ya mushinga mukole ya dimanyishangana dia malu adi menzeke¶

1. Amba malu, tshinyangu, ne ETA kuyi usokoka, kuyi ulengulula, anyi kuyi ubanda: Uleje tshilema mu minite 2 ya dipeta.
2. Tapulula "Mitigation" (Stopping the bleeding) ne "Root-Cause Fix": Mu tshilumbu tshia njiwu, tshipatshila tshiebe tshimuepele ntshia kupingaja mudimu (kupingaja mudimu mubi, kupanga kufika ku tshilaminu tshia tshilaminu tshia PITR, anyi kumbusha dibendele dia tshintu). Lama deep code refactoring bua malaba.
3. Tuma ngumu ya pa tshibidilu ya minite 15 ya dikuma dia mutshima mu tshisanji tshia malu adi menzeke bua balombodi ne badi bambuluisha bantu badi basumba bintu bamanye ne: muntu kampanda udi ne bukokeshi.

Copy-paste Slack / Teams Incident Declaration Script¶

[SEV-1 INCIDENT] Production [Service / Database Name] Impact - Tshivua tshienzeke: Ku [HH:MM UTC], patudi tuenza [brief technical action, e.g., a data migration script], tshidi katshiyi tshilongolola [DELETE / config change / force-push] uvua muenza mudimu mu dienza dia bintu [database / service]. - Tshilumbu tshia bantu badi basumba bintu: [e.g., Checkout API is returning 500 errors / User profile updates are paused]. - Dikepesha dia lukasa didi dienzeka: 1. Funda njila ku [table/service] mmuimanyika bua kulama bujalame bua malu. 2. Point-In-Time Recovery (PITR) clone to [HH:MM:SS UTC] (10 secondes avant the event) udi wenda unyunguluka mu [AWS RDS / GCP]. - Mulombodi wa tshilumbu: @[Your Name] (muambuluisha kudi @[DBA/On-Call SRE]). - Dishintuluka didi dilonda: Mu minite 15 ([HH:MM UTC]) anyi diakamue padi tshimfuanyi tshia dipingaja tshilua ku Enternete.

Mulongo wa malu a bukubi a kunyima kua bualu (bua tshilema tshiebe tshia panyima pa lufu)¶

Tshilema tshionso tshia muntu mu dienza dia bintu tshidi mu bulelela dipanga dia dienza dia bintu (bikala dîyi dimue dia tshiamu tshia mudimu didi mua kulekela dienza dia bintu kakuyi tshibuikidiji, tshiamu etshi tshivua tshiindila bua kunyanguka). Ela meji ne enza mudimu ne bimanu ebi bia beton mu mukanda webe wa post-mortem dituku didi dilonda:

• Kode wa mekala ne dilomba dia dienza dia mekala (kube-ps1, psql .psqlrc): Longolola shell webe ne SQL client bua dipetangana dionso dikale ne PRODUCTION udi uleja tshitoke tshitoke [PROD-DB] udi ulomba ne utuadija tshisangilu tshionso tshia PostgreSQL munda mua BEGIN; tshipungidi tshia mudimu ku tshibangidilu (anyi tshisumbu SET default_transaction_read_only = on;).
• Kulekela safe-updates (--i-am-a-dummy) mu MySQL/MariaDB clients: Add safe-updates bua [mysql] mu -/.my.cnf nunku MySQL udi ubenga bua kuenza bualu nansha bumue UPDATE anyi DELETE meyi adi kaayi ne WHERE tshikoso tshia mushinga anyi a LIMIT.
• Teka DeletionProtection = true pa bintu bionso bia RDS, S3 buckets, ne Terraform stateful resources (lifecycle { prevent_destroy = true }).
• Kuenzeja mikenji ya bukubi bua filiale wa Git pa main / production: Kupangisha Allow force pushes ne Allow deletions pa matamba adibu balame mu GitHub/GitLab nunku nansha balombodi ba nzubu ya dilamina bintu kabena mua git push --force bua main.
• Teka bintu bia tshisokomenu bia pre-commit (gitleaks anyi trufflehog) ne GitHub Push Protection: Block commits containing AKIA, sk_live_, anyi .env bilejilu bia muaba kumpala git commit mene udi ukumbaja.

Nkonko itubu bela misangu ya bungi¶

Nebanipate ku mudimu bualu mvua mupue mesa a tshilaminu tshia bintu bia mudimu mu mpukapuka anyi bualu mvua muenze bua site ikale kayi wenza mudimu anyi?¶

Mu malongolodi a ba-enjenyere adi ne makanda a mubidi adi ne tshilele tshia bukubi tshidi katshiyi ne bualu (tshidi tshienzeka mu mishindu yonso ya malu a tekinoloji a matuku etu aa, a DevOps, ne a SRE), ba-enjenyere kabatu bipata mudimu bua bilema bia bantu bia bushuwa mu dienza dia bintu to, anu bobu bamanyisha bualu abu diakamue ne bambuluisha bua kupingaja mudimu. Balombodi ba malu a enzenjiere badi bitaba ne: bikala muenji wa mudimu mukese anyi mukulumpe muikale ne bukenji bua kuenza mudimu ne dipanga dia bimanu bia kulama nabi mesa a dienza dia bintu ne dîyi dimue didibu kabayi banji kukontonona, tshidi tshikebesha bualu ebu ntshia dipanga dia mushindu wa kuenza mudimu ne biamu bia mudimu, ki nkubi to. Mu kuenza, muenji wa mudimu uvua mumone dinyanguka edi nkayende udi mpindieu muntu umuepele udi ne bukubi mu tshisumbu pa bidi bitangila tshiamu atshi. Tshikadilu tshimuepele tshidi tshifikisha ku dijikija dia mudimu ntshia kuteta bua kusokoka bualu budi bujimina, kujimija mikanda ya dikonkonona dia malu, anyi kushima padibu bakonkonona bualu kampanda.

Bua tshinyi tshiena ne tshia kushintulula diakamue mudimu wanyi wa dienza dia bintu ku tshilaminu tshipiatshipia tshia Point-In-Time Recovery (PITR)?¶

Shi uvua anu mufute milongo ya mesa umue ku mpukapuka (tshilejilu, `user_preferences` mesa) pa dîba 14:00 UTC, ne bidi bilomba minite 20 bua kuenza tshimfuanyi tshia PITR tshipingajibua ku 13:59:50 UTC, tshilaminu tshiebe tshia bintu tshidi mua kuikala tshitungunuka ne kuenza mudimu ne binunu bia malomba ne makuta a bantu bapiabapia badi bakumbane mu `orders` mesa pankatshi pa 14:00 ne 14:20 UTC. Wewe mushintulule tshitupa tshieba tshia DNS tshia 13:59:50 PITR clone, neujimije minite 20 ya dilomba dia bakiliya bapiabapia. Pamutu pa nanku, ikala ne tshilaminu tshinene tshia malu pa buloba (ne difunda dijika anu pa tshitupa tshidi tshinyanguke) ne utume anu milongo idi mijimine ya mu tshimfuanyi tshia 13:59:50 PITR mu mesa a dienza dia bintu.

Meme muenze tshilaminu tshia GitHub tshia patoke minite ibidi kunyima kua meme mumane kuela .env tshikandakanda, ndi anu dijinga ne kushintulula mishindu yonso ya API anyi?¶

Eyowa, 100 pa lukama ya tushinyi atu tudi anu ne tshia kujimijibua ne kunyunguluka diakamue. Bualu buonso budibu bafila mu tshilaminu tshia GitHub budi bumanyishibua mu tshikondo tshiakane ku diambuluisha dia GitHub Events API ("the Firehose") ne budi bumanyishibua mu tusunsa tukese kudi midimu ya dilama dia malu a bukubi (bu mudi GHArchive) ne kudi biamu bia mudimu bibi bidi bienza mudimu 24/7 mu matutu. Kushintulula tshimuenekelu tshia tshilaminu tshia bintu kumbukila ku Public too ne ku Private minite ibidi pashishe kakuena kujimija kopi uvua biamu bia mudimu bia pa nkayabi bimane kuangata to.

Mmunyi mudi git push --force-with-lease kupangisha dijika dia nzembu pa kumona git push --force?¶

Paudi wenza mudimu ne tshibidilu `git push --force`, Git udi ufunda filiale wa ku bule ne filiale wa muaba unudi, ubutula mu musokoko bionso bidi muena mudimu nebe mutume ku filiale au mu tshikondo atshi. Paudi wenza mudimu ne `git push --force-with-lease` (ne bitambe buimpe, `--force-if-includes`), Git udi utangila kumpala tshiamu tshia mudimu tshia ku bule bua kujadika ni tshitupa tshia ku bule tshidi anu tshipetangana ne SHA wa commit uvua tshiamu tshiebe tshia muaba au tshipete. Shi muena mudimu nebe anyi CI pipeline muele commit mupiamupia ku filiale wa kule katshia ku dipeta diebe dia ndekelu, `--force-with-lease` udi ulekela mudimu bimpe ne didimuija dia "stale info" pamutu pa kubutula mudimu wabu.

Miaba mifumina¶

• Tshilongelu tshia ditunga tshia mikenji ne bia mudimu (NIST, États-Unis): NIST SP 800-61 Rev. 2 / Rev. 3 Computer Security Incident Handling Guide and Credential Compromise Containment.
• Amazon Web Services (AWS), Google Cloud Platform (GCP), ne Microsoft Azure Documentation (United States): RDS Automated Point-In-Time Recovery (PITR) Architecture, AWS IAM Compromised Key Quarantine Policies (AWSCompromisedKeyQuarantineV2), ne S3 Versioning Delete-Marker Recovery.
• PostgreSQL Global Development Group ne MySQL / Oracle Documentation (International): PostgreSQL Write-Ahead Logging (WAL), MVCC Tuple Visibility, ne MySQL Row-Based Binary Log Flashback Recovery.
• Git SCM Official Documentation and GitHub Security Lab (International): Git Reflog (git-reflog), Dangling Object Recovery (git-fsck), Difunda dia malu a kale (git-filter-repo), ne bukubi bua tshisokomenu tshia tshikebelu.
• USENIX Association / SREcon ne Google Site Reliability Engineering (SRE) Books (International): Management Incidents, Emergency Rollback Triage, and Blameless Postmortem Culture.

Malu adi apetangana¶

• Mudimu wa tshimpitshimpi wa ku bule ne bukubi bua malu masokoka
• Dilama dia bintu ku ordinatere 3-2-1 mayele ne dipingaja dia bintu kunyima kua bipupu
• Dijimija dia mukanda ku mpukapuka ne dipingaja dia tshiamu tshia mashinyi
• Tshilumbu tshia dilama dia bintu bia ku Enternete ne dipingaja dia bintu bia kulama
• Dishima ne konte minyanguke
• Nomba ya tshimpitshimpi