Taƴgol peewnugol ngol e aksidaa, kuutorɗe API ɓuuɓɗe, walla kuutorɗe ruulde mumtuɗe¶

Kala injenieer lowre, gardiiɗo systems, e golloowo karallaagal to woɗɗi, ina jokki e jokkondirde e sahaa mo reedu mum ɓuuɓnata nde yamiroore fotnoo localhost walla staging executes dow production: ko alaa indeks DELETE tawa alaa a WHERE kuulal, a DROP TABLE walla terraform destroy, gooto git push --force ko winnditoo e caltal mawngal, walla commit public jogiiɗo kuutorɗe sirlu AWS walla Stripe nguurndam. Ko seerndi hiccude tokoose 15 minit e musiiba domen joofɗo golle ko dartinde juuɗe maa ɗoon e ɗoon, ɓuuɓnude jolngo binndol hade bolokaaji mooftirɗi ɗii winndeede, e huutoraade porotokol moƴƴitingol.

Waɗde ɗum jooni¶

1. Fooɗtu juuɗe maa e WOTO dognu yamiroore "fix" yaawnde, fuɗɗoo serveer domen, walla uddu windo terminal maa. Buffer scrollback terminal maa jooni, daartol shell, e nokku .git/logs reflog ina waɗi waktuuji jokkondire laaɓtuɗe, hashes SHA commit, e IDeji doole potɗi huutoreede ngam waylude bonneede. Koppir doggol maa terminal ngol fof e nder fiilngo binndol kisal ko adii.
2. Bayyina kewu Severity-1 (SEV-1) e nder kewu ekipaaji maa Slack/Teams channel e nder hojomaaji 2. Hoto meeɗ waɗde hojomaaji 45 ngam etaade feewnude e nder sirlu lowre peewnugol ɓuuɓnde tan tawa soodooɓe ina piytoo 500 boowal e golle caggal otomaatik ina mbonna taƴe heddiiɗe ɗee. Page on-call njuɓɓudi lowre (DBA) walla Injenieer koolkisagol lowre (SRE) ɗoon e ɗoon.
3. So a yaltinii coktirgal API, .env fiilngo, walla sirlu ruulde to repo GitHub renndo, WOƊƊU coktirgal ngal e nder tablo dokkoowo ADI (ko adii nde eto-ɗaa winndude daartol Git). Botnets otomaatik ina njuppa API GitHub Firehose renndo ngoo e huutoraade coktirgal AWS, OpenAI, Stripe, e GCP wujjaaɗo e nder **5s maa secondaire git push. Mumtude fiilde ndee e commit ɗiɗaɓo walla doole-pushing haɗataa jokkorgal gonngal e tafde jokkorgal ngal.
4. So a dogi aksidaa DELETE, UPDATE, walla DROP e dow lowre peewnugol, taƴu jolngo WRITE kuutorgal ɗoon e ɗoon. Waɗde kuutorgal web ngal e nder Mode Maintenance walla ittu yamiroore binndol kuutortooɗo app oo mbele jokkondire kese naatɗe ɗee ina mbaasa winndude tupleeji maayɗi ɗi mbaylaaka walla caɗtu Point-In-Time Recovery (PITR).
5. Woto meeɗ dogde VACUUM, OPTIMIZE TABLE, walla git gc --prune=now caggal momtugol ngol e aksidaa. Baseeji jokkondire ɗiɗi ɗii fof (PostgreSQL MVCC, MySQL InnoDB) e Git ina njogii rowrowre mumtunde e golle yaɓɓuɗe tawa ko e disko haa nde paaspoor mooftugol/vacuum ƴetti kelle ɗee e ɓanndu.
6. So a terraform destroy walla binndol momtugol ruulde ina doga haa jooni e nder terminal maa, toɓɓu Ctrl + C WOƊƊI (wonaa laabi ɗiɗi) walla warde toɓɓere jonnde IAM. Toɓɓude Ctrl + C nde wootere ina addana ɓuuɓri dowla mbelndi noon Terraform hisna feccere .tfstate e ko woodi haa jooni; toɓɓude Ctrl + C laabi ɗiɗi doole ruttaade mo moƴƴaani baawɗo bonnude walla majjude fiilde dowla maa nokkuure.

Golle triage e nder sahaa ɓuuɓɗo: doggol domen, git force-push, walla ɓuuɓgol sirlu¶

Rewru e ndee lekki pellital ngam ruttaade hojomaaji 5 gadani maa e jokkondire karallaagal.

flowchart TD
    A["Emergency Innjiniyaaruuji peewnugol"] --> B{"Hol ŋakkeende nde waɗi jooni ndee?"}
    B -->|"Aksidaa SQL ƊAƊƊI / MUUMTUƊO / KESO e nder Prod"| C["Stop App Winndi / Mode Toppitagol; WOTO Doggo VACUUM"]
    B -->|"Kuutorgal API ɓuuɓngal / .env / Kuutorgal keeringal e nder Git"| D["Sosde & Rottude Sirlu e nder Dashboard Cloud E nder Leƴƴanɗe 60"]
    B -->|"Git push --force / Cakkital momtaangal"| E["Doggo git reflog E nokku; Koppi Caggal Moƴƴere Komit SHA"]
    B -->|"Terraform aksidaa halka / Mumtu S3 walla VM"| F["War Session IAM walla Toɓɓu Ctrl+C laawol gootol; Hisnude .tfdowla"]
    C --> G["Winndude waktuuji UTC laaɓtuɗi; PITR ruulde jolngo walla ƴoogirde Binlog"]
    D --> H["Ƴeewndo CloudTrail / Jokkondire jokkondire ngam Spin-Ups ɗi ngalaa yamiroore; Ƴeewndo Repo e git-filtir-repo"]
    E --> I["Ƴeewto Kewuuji GitHub / Reflog & Wartirde Catal e git push --doole-e-lease"]
    F --> J["Ƴeewto S3 Versioning Mumtu Maandeeji, Nate EBS, walla Bin Recycle Cloud"]

Feccere 1: Dataaji peewnugol aksidaaji DROP, TRUNCATE, walla DELETE tawa alaa WHERE¶

So naamnal SQL bonngal yanii e lowre peewnugol nguurndam, feere ƴellitgol laaɓtunde fawii ko e mooftirgel lowre maa (PostgreSQL, MySQL/MariaDB, Serwer SQL, MongoDB, walla DynamoDB) e so tawii konngol ngol ko DML (DELETE / UPDATE) walla DDL (DROP TABLE / TRUNCATE) gollal.

Ɗatal 1: Winndu waktuuji UTC tigi-rigi haa les ɗiɗaɓol ngol¶

Ko adii nde memotaako kala konsol restoraaji, winndu waktu UTC jooni oo tigi-rigi (date -u e nder jokkorgal Linux/macOS, walla SELECT NOW(); nder SQL prompt) e hiisaade ɗiɗaɓo tigi-rigi tan ko adii naamnal maa halkoowal fuɗɗaade huutoraade. Toɓɓere-e-waktu (PITR) ina gollira e ƴellitde doggol binndol yeeso (WAL e nder PostgreSQL) walla Binndol (binlog e nder MySQL) haa T_minus_1_second ko adii nde naamnal maa bonngal ina ara.

Ɗatal 2: Jokku golle caggal e binndi kuutorgal (Reende tuple MVCC)¶

E nder PostgreSQL, ko huutoraade DELETE FROM users; walla UPDATE orders SET status = 'cancelled'; not momtataa tigi rigi bayt gooto e dokke e nder fiilngo tabalde e disko ɗoon e ɗoon. Sabu PostgreSQL huutortoo ko Koolkisagol mbaydiiji keewɗi (MVCC), a DELETE tan setti ko xmax hoore ID transaction e dow kala tuple row ngam maande ɗum no yiyataa e naamne garooje, accude domen row timmuɗo jooɗiiɗo e nder base/ fiilngo heap haa autovacuum ari ngam maande nokku kuutorteeɗo kadi.

• So aɗa woni e self-hosted walla VM PostgreSQL tawa a alaa backup mo ɓooyaani: Ɗoon e ɗoon jokkondir e superuser e muumtude autovacuum e dow oon taabal (ALTER TABLE table_name SET (autovacuum_enabled = false);), walla dartinde sarwiis PostgreSQL e ƴettude nate njuɓɓudi fiilngo to bannge bolol raw (LVM / EBS foto) mo... /var/lib/postgresql/data regginorde. Yaajde keertiiɗe PostgreSQL (pg_dirtyread) ina waawi janngude e heɓtude 100 e nder teemedere rowrowre momtaande walla ko adii hesɗitinde to bannge tabalde mooftirgel ngel alaa vacuum e nder hojomaaji seeɗa.

Matriks keɓgol domenuuji e mooftirgel e laylaytol ruulde¶

Feccere 2: Kuutorɗe API ɓuuɓɗe, .env sirluuji, walla kuutorɗe keertiiɗe baɗaaɗe e Git¶

So a waɗii e aksidaa an... .env fiilde, an id_rsa Kuutorgal SSH, fiilde JSON konte sarwiis Firebase/GCP, kuutorgal sirlu Stripe (sk_live_...), walla AWS gooto AKIA... access key to Git, rewa e ngool doggol fasaaji ɗiɗi laaɓtuɗi: Fas A (Warde key e nder duule) gadano, e Fas B (Scrub repository Git) ɗiɗaɓo.

Faandaare A: Ko ɗoon e ɗoon 60-sekonnde jokkere enɗam ruulde (Waɗ ɗum ADDI)¶

1. AWS (AKIA... ID coktirgal + coktirgal sirlu)¶

So botuuji ngujjii coktirgal AWS e GitHub, binndol maɓɓe otomaatik ina fiyra konte maa e nder leƴƴanɗe 10 ngam ƴellitde inɗe GPU EC2 ɓurɗe mawnude (p4d.24xlarge / g5.48xlarge) e nder denndaangal diiwanuuji AWS winndereeji 30+ (ina heen diiwanuuji opt-in hono Jakarta, Osaka, e Fuɗnaange hakkundeejo) ngam ƴellitde kaalis kripto, walla sosde kuutortooɗo IAM backdoor e AdministratorAccess hade maa momtude coktirgal asliwal.

1. Softinde e Mumtude Ɓoggi Access Key ɓuuɓɗi ɗii ɗoon e ɗoon:
   • Uddit Konsol AWS -> IAM -> Huutortooɓe -> [Kuutortooɗo ƴaañaaɗo] -> Kaɓirɗe kisal -> Kuutorɗe jokkondire -> Ɓoɗɗu Mumtu, caggal ɗuum Mumtu (walla rewrude e CLI: aws iam update-access-key --access-key-id AKIAIOSFODNN7EXAMPLE --status Inactive --user-name MyUser).
2. Jokkondira e laaɓɗo DenyAll politik ngam neutraliser wakkati aws sts toɓɓe jonnde:
   • Trap critique: Hay caggal nde a momti an AKIA... IAM access key, so tawii jokkorgal ngal noddii tawo sts:GetSessionToken walla sts:AssumeRole e nder leƴƴanɗe 30 gadane ɗee, ko waktuuji maɓɓe ASIA... token session ina heddii e golle haa waktuuji 1 haa 12 so wonaa tawa aɗa jokkondiri e politik Deny laaɓtuɗo (hono AWSCompromisedKeyQuarantineV2 walla politik Deny-all ngam toɓɓe yaltinaaɗe ko adii limre waktu jooni ndee) to kuutortooɗo walla darnde.
3. Ƴeewto AWS CloudTrail ngam tiiɗnaare damal caggal:
   • Uddit AWS CloudTrail -> Daartol kewu e filtirde e ID kuutorgal keɓe walla Innde kuutortooɗo gonɗo e nder waktuuji 2 ɓennuɗi ɗii.
   • Ƴeewto teeŋti e ɗeeɗoo noddaali 5 tiiɗnaare: CreateUser, CreateAccessKey, AttachUserPolicy, AssumeRole, be RunInstances. Ƴeewto AWS Cost Explorer e EC2 Global View e nder diiwanuuji denndaangal ngam joofnude kala inɗe fenaande, e udditde ɗoon e ɗoon Case Ballal AWS (Konte e Faggudu -> Konte Ɓooyɗe). AWS ina heewi woppude njoɓdi hiisawal fenaande so a habrii ɓuuɓri key e joofnude instances ko yaawi.

2. Ɗaɗi (sk_live_...), Uddit AI (sk-...), NelduGrid, Twilio, e Toɓɓe Slack¶

• Stripe: Yah to Developers -> Kuutorɗe API -> Kuutorɗe kuutorteeɗe / Kuutorɗe keertiiɗe -> Ɓoɗɗu toɓɓe tati ɗee (...) -> Rogere coktirgal. Suɓo "Expire now" (walla waɗ windoore moƴƴere waktu 1 tan so tawii aɗa huutoroo jokkorgal kesal e nder waylooji nokkuure peewnugol maa jooni jooni).
• OpenAI / Anthropic: Yah to dashboard API Keys e ɓuɗɗu Woɗtu / Mumtu ɗoon e ɗoon ngam ƴoƴre otomaatik waawa ustude ujunnaaje ujunnaaje dolaar njoɓdi token LLM e nder hojomaaji seeɗa.
• Tokenuuji keɓgol neɗɗo GitHub (PATs), Tokenuuji ɓutti, e kuutorɗe njuɓɓudi: Yah to Ceŋorɗe GitHub -> Cuɓe ƴellitooɓe -> Toɓɓe keɓgol neɗɗo e ɓuɗɗu Mumtu / Woppu e dow token ɓuuɓɗo oo, caggal ɗuum ƴeewto njuɓɓudi maa **gt'sSettings -> Audit log) ngam repo.clone, repo.access, walla sosde webhook mo yamiraaka e nder windo peeñgol.
• Jokkondire jokkorgal (DATABASE_URL / postgres:// / mongodb+srv://) e URLs Redis: Waylu konngol kuutortooɗo domen nder mooftirgel domen (ALTER USER app_user WITH PASSWORD 'new_random_secret';), caggal ɗuum joofnu denndaangal jokkorɗe TCP kuutorteeɗe udditaaɗe jooni ɗee jeyaaɗe e seedanteeje ɓooyɗe (SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE usename = 'app_user' AND pid <> pg_backend_pid(); e nder PostgreSQL), sabu waylude konngol kuutorgal domen e nder PostgreSQL walla MySQL ina haɗa tan naatgol kesol e accude jokkorde jokkorgal udditiingal jooni ngal ina wuura haa nde jolnaa.

Faandaare B: Ɓoornude sirlu ummoraade e daartol Git (git filter-repo & Ittugol kaɓirgol GitHub)¶

So tawii coktirgal ɓuuɓngal ngal maayii, wirniima e peewnugol, laɓɓin doggol sirlu ngol e nder daartol maa Git commit ngam skanndeeji kisal ɗii ngoppa jeertinde:

1. Winndu daartol defterdu huutoraade git filter-repo (walla BFG Repo-Laɓɓinoowo):
   • Ƴettu mbaydi kesiri ndi alaa ko woni e mum e nder defterdu maa, ittu fiilde ɓuuɓnde ndee (yeru, .env) ummoraade e kala gollal daartol e nder denndaangal calɗi e tagɗe:

     # 1. Remove a specific file (.env) from all commits in history:
     git filter-repo --invert-paths --path .env
     
     # OR 2. Replace a specific secret string with ***REMOVED*** everywhere:
     echo "sk_live_YOUR_SECRET_STRING==>***REMOVED***" > replacements.txt
     git filter-repo --replace-text replacements.txt
     

2. Doole-push daartol laaɓtungol ngol e purge GitHub Pull Request / Commit yiyngooji:
   • Push daartol laaɓtungol ngol artira e iwdi (git push origin --force --all e git push origin --force --tags).
   • Ɗatal GitHub cached-commit teeŋtungal: Hay caggal nde doole-push daartol laaɓtungol, kala gannduɗo URL SHA commit ɓooyɗo mo alkule 40 (github.com/org/repo/commit/abc1234...) ina waawi yiytude haa jooni commit ɓooyɗo mo laaɓaani e UI web GitHub haa GitHub dogna mooftugol coodguuli bannge serwer. Jokkondir e Ballal GitHub (walla huutoro portal Wasiyaaji / Ballal Kisal defterdu nduu) e SHA commit ɓooyɗo tigi rigi e naamnaade ɓe riiwtude commit dereferenced/dangling e nder cache web e dogde git gc dow defterdu woɗɗundu.

Feccere 3: Uddude aksidaa git push --force, git reset --hard, walla caltal momtaangal¶

So a dogii e aksidaa git push --force e momtude commits wondiiɓe maa main, walla dogi git reset --hard HEAD-5 e majjude ñalawma timmuɗo e golle nokkuure, woto hul, sabu Git fotde meeɗaa momtude geɗe baɗaaɗe ɗoon e ɗoon.

Ko waɗi commits maa "lost" ina njooɗii haa jooni e nder jam .git/logs (Reflog oo)¶

Kala nde sos-ɗaa commit, mbayli-ɗaa branches, rebase, walla reset, Git ina winnda ko adii HEAD e toɓɓe SHA-1/SHA-256 toɓɓere nder deftere nokkuure wiyeteende Deftere tuugnorgal (git reflog), e jogaade geɗe commit orphaned tigi-rigi e disko fotde ko famɗi fof balɗe 14 haa 90 hade mooftude mbuuɗu ittude ɗe.

Ɗatal-ɗatal Git restoraaji cellal emergency¶

Sifaa 1: A dogi git push --force e winndude dow origin/main¶

1. Yiytu commit SHA tigi-rigi oo origin/main toɗɗii ko ñaamo hade doole-push maa:
   • Laawol A (Ƴeewto reflog remote-tracking nokkuure maa): Doggo ndee yamiroore e nder terminal ɗo pus-ɗaa ɗoo:

     git reflog show origin/main
     

     Maa a yiy yaltinnde hono:

     a1b2c3d (HEAD -> main, origin/main) refs/remotes/origin/main@{0}: update by push
     9f8e7d6 refs/remotes/origin/main@{1}: fetch: fast-forward
     

     Ɗoo, 9f8e7d6 (origin/main@{1}) woni commit tigi-rigi origin/main wonnoo ko e dow ñaamo hade doole maa-push winndude ɗum.
   • Laawol B (Naamndo kala sehil mo dogaani git pull yet): Kala gollodiiɗo mo fetti main ko adii hannde haa jooni ina jogii daartol commit intact jooɗiiɗo e nokku maɓɓe origin/main caltal.
   • Laawol C (Naamndo doggol kewuuji GitHub REST API): So tawii reflog nokku maa ina ŋakki ngonka to woɗɗi, naamndo API kewuuji push GitHub ngam yiyde... "before" SHA hash kewu doole-push:

     gh api /repos/OWNER/REPO/events --jq '.[] | select(.type=="PushEvent") | {actor: .actor.login, before: .payload.before, head: .payload.head, time: .created_at}'
     

2. Wartirde origin/main to ko adii waɗde SHA e kisal:
   • Hoolkiso wonde gollal ngal ina wayi no moƴƴi (git log -n 5 9f8e7d6), caggal ɗuum pus oon SHA tigi-rigi ngartaa e remote main calɗi kuutortooɗi --force-with-lease:

     git push --force-with-lease origin 9f8e7d6:refs/heads/main
     

Sifaa 2: A dogi git reset --hard walla momtude caltal nokkuwal hade mum pusde¶

1. Doggo git reflog e nder defterdu maa:

   git reflog --date=iso
   

2. Yiytu commit SHA ko adii... reset: naatgol (yeru, HEAD@{2}: commit: feat: add payment webhook), e sosde caltal hisnude kesal toɗɗiingal to bannge oon SHA:

   git switch -c rescue-branch HEAD@{2}
   

3. Hol ko woni so a majjii fiilde nde alaa ko woni e mum nde a... git added (staged) caggal ɗuum wippitaa e git reset --hard?
   • Wakkati mo doggata git add myfile.ts, Git ina ɓuuɓna, ina winnda ko woni e oon fiilngo fof e nder .git/objects/ ko blob dangling, hay so a meeɗaa dogaade git commit. Heɓtu ɗum e limtude denndaangal blobs dangling renndinaaɗi e sahaa waylude:

     git fsck --lost-found
     # Inspect the recovered blobs written into .git/lost-found/other/
     

Feccere 4: Aksidaa terraform destroy walla mooftirgel duule momtaa (S3, EBS, VMs)¶

So tawii laylaytol Infrastructure-as-Code (IaC) walla yamiroore CLI huutorteende fuɗɗiima ustude doole ruulde peewnugol:

1. Woppu warngo ngoo e dow jam:
   • So ina doga e nokku e nder terminal maa, toɓɓu Ctrl + C ONCE ko noon Terraform timminirta noddaango mum API jooni ngoo, winnda ko hesɗitinaa .tfstate fiilde laaɓnde. So tawii ko dogdu nder Golle GitHub, GitLab CI, walla Terraform Cloud, ɓuɗɗu Woppu Doggol e ɗoon e ɗoon softin darnde AWS/GCP IAM dognoowo CI e nder konsol ruulde ngam momtude API nder dingiral waasa AccessDenied.
2. Back up ko jooni .tfstate fiilde ɗoon e ɗoon:
   • Sosde koppi nokkuure fiilde dowla maa to woɗɗi (terraform state pull > emergency-backup.tfstate) e ƴeewtaade mbayliigu bucket S3 e bucket dowla maa Terraform ngam nanngude mbayliigu dowla ɓennuɗo oo gila e hojomaaji 10 ɓennuɗi ɗii.
3. Ƴeewto dokkoowo duule "Soft-Delete" e resooji kisal Bin:
   • Amazon S3 (E nder mbaydi Bucket ina softi): Mumtude piille e nder mbaydi S3 (aws s3 rm --recursive) momtataa huunde wootere; ina waɗa tan Mumtu Marker bayte zero e dow kala huunde. Ittu Maandeeji Mumtugol (rewrude e Golle Batch S3 walla Python 5-line boto3 script) ngam artirde 100 e nder teemedere fiilde maa ɗoon e ɗoon.
   • Toɓɓe AWS EBS & Nate RDS (Binnde : Ƴeewto Binnde AWS e nder konsol so tawii ko nde njuɓɓudi, ƴeewto AWS Backup vaults.
   • Google Cloud Platform (GCP) & Microsoft Azure: Eɓɓooje GCP momtaaɗe e Pelle Ngalu Azure / Kuutorgal Kuuɓtodinngal / Konte Desorɗe ɗe Mumtugol Soft softi ina keddii ina mbaawi heɓtineede fotde balɗe 7 haa 30 e dow doggol gootol ngam artirde e nder konsol.

No Terraform ɓuuɓɗo heɓtirtee .tfstate fayndaare dowla walla re-import infrastructures yaayaaɓe¶

Nde a terraform apply walla ko aksidaa terraform destroy ina taƴaa e hakkunde diwgol, walla so neɗɗo momtii e aksidaa fiilde ngonka woɗɗuka e nder S3/GCS, njuɓɓudi ruulde maa naata e ngonka peccitagol ngaandi kulɓiniindi ɗo won e doole ina ngoodi haa jooni e AWS/GCP tawi woɗɓe ina ŋakki heen .tfstate:

1. Rewrude e remote binndaaɗe walla momtaaɗe .tfstate fiilde ummoraade e S3 / GCS mbaydi:
   • Hoto meeɗ waylude JSON e junngo .tfstate fiilngo e junngo e nder binndol so wonaa tawa kala cuɓal goɗngal ina ŋakki, sabu gootol ngol yahdaani "serial" walla "lineage" Doggol UUID maa haɗ Terraform meeɗde huutoraade kadi.
   • Uddit lowre maa S3 (walla lowre Google Cloud Storage / lowre Terraform Cloud Workspace Dowlaaji) ɗo backend to woɗɗi oo hoɗi ɗoo, ɓuɗɗu e Hollit mbaydiiji toɓɓere to dow konsol S3, yiytu... terraform.tfstate version timestamped ko adii dogdu bonndu nduu, ƴettu ɗum, e pusde ɗum caggal e dow jam no ngonka keso jooni ka nii:

     # Push a recovered historical state file back to the remote backend safely
     terraform state push recovered-pre-incident.tfstate
     

2. Fuɗɗude coktirgal dowla cokaaɗo (Error acquiring the state lock):
   • Nde a aborti bonɗo terraform destroy e Ctrl + C walla warde dognoowo CI cokaaɗo, tabalde DynamoDB to woɗɗi walla fiilde coktirgal GCS ina heddii e coktirgal ID golle maayɗe.
   • Ko adii fof hoolkiso e nder dashboard maa CI e wondiiɓe maa wonde zero golle Terraform goɗɗe ina njokki e dogde e oon nokku gollorɗo, caggal ɗuum yaltin coktirgal ngal alaa ko huutortoo Lock ID UUID tappirde e mesaas majjere:

     terraform force-unlock <LOCK_ID_UUID>
     

3. Mahngo nokku mo feccere mum bonni tawa sosde lowre domen (terraform import):
   • So tawii a terraform destroy momtii Balancers Load Application maa mo alaa leydi (aws_lb) e nodule gollotooɗe ECS/EKS hade maa dartinde ɗum, ɗum ina weeɓi feewnude: doga tan terraform apply e Terraform maa sos denndaangal resooji ɗi ngalaa leydi e hiisaade laabi gila e fuɗɗoode e nder hojomaaji 5 tawa aɗa acca lowre maa RDS heddiinde nde memotaako.
   • So tawii fiilde dowla maa majjii e lowre dowla walla bucket S3, huutoro import {} bolol konfiguraasiyoŋ (Terraform 1.5+) walla terraform import aws_db_instance.prod_db my-rds-instance-id ngam haɓɓude ngalu ruulde gonngu nguu e nder fiilde dowla maa tawa a bonnaani ɗum, e dogde terraform plan ngam tabitinde "0 to destroy" hade mum waɗde ɗaɓɓaande.

Feccere 5: No jokkondirta e gardiiɗo maa e outage e ardaade post-mortem mo alaa ko bonnata¶

So a waɗii kewu peewnugol tawa aɗa gollira e woɗɗude, innde maa to bannge karallaagal ina siforee wonaa e majjere ndee e hoore mum, kono ko e jaawgol maa laaɓal, jokkondiral calminaangal, e peewnugol njuɓɓudi.

Kuule 3 kaŋŋe jokkondiral keɓe¶

1. Hollu geɗe, batte, e ETA tawa suuɗaaki, leeltinaani, walla tuumaani: Haaldu boowal e nder hojomaaji 2 caggal yiytugol ngol.
2. Seerndu "Mitigation" (Dartaade ƴiiƴam) e "Root-Cause Fix": E nder caɗeele, faandaare maa tan ko artirde sarwiis (rowtude deploy bonɗo oo, waasde yahde e clone lowre PITR, walla ustude bannge sifaa oo). Hisnude refaktoring kode luggiɗnde ngam janngo.
3. Post 15-minit heartbeat updates e nder laylaytol kewu ngam ardiiɓe e ballal soodooɓe ngannda won ko woni e dow mum.

Koppir-paste Slack / Teams Skript bayyinaango kewu¶

[SEV-1 KEƁAL] Peewnugol [Sarwiis / Innde lowre] Batte - Ko waɗi: At [HH:MM UTC], nde o waɗata [brief technical action, e.g., a data migration script], ko anniyaaka [DELETE / config change / force-push] huutoraa ko e peewnugol [database / service]. - Batte soodooɓe hannde: [e.g., Checkout API is returning 500 errors / User profile updates are paused]. - Mitigation ko yaawi e nder golle: 1. Winndu trafik to [table/service] dartinaama ngam hisnude laaɓal dokke. 2. Toɓɓere-e-waktu (PITR) klon to [HH:MM:SS UTC] (leƴƴanɗe 10 ko adii kewu nguu) ina wirna jooni e nder... [AWS RDS / GCP]. - Komandaajo keɓe: @[Your Name] (ina wallittee e... @[DBA/On-Call SRE]). - Kesɗitingol garoowo: E nder hojomaaji 15 ([HH:MM UTC]) walla so tawii clone recovery oo arii e internet.

Doggol ƴeewndorɗe kisal injiniyaagal caggal kewkewe (Ngam Post-Mortem maa mo alaa aybe)¶

Kala majjere aadee e peewnugol ko goonga ko gap design systems (so yamiroore terminal gootol koppi-paste ina waawi woppude peewnugol tawa alaa gardo, system oo ina ɗamininoo waasde). Eɓɓu e huutoraade ɗeeɗoo gardorɗe karallaagal konkreet e nder winndannde maa post-mortem ñalnde wonnde :

• Kod-color e prefix peewnugol ɗaɓɓaande (kube-ps1, psql .psqlrc): Konfigu shell maa e SQL client ngam kala jokkondiral e PRODUCTION hollirta ko boɗewol jalbowol [PROD-DB] prompt e fuɗɗoraade kala joɗnde PostgreSQL jokkondirnde e nder a BEGIN; bolol njulaagu e dow mbaydi (walla setuuji SET default_transaction_read_only = on;).
• Softina safe-updates (--i-am-a-dummy) e nder kiliyaneeɓe MySQL/MariaDB: Ɓeydu safe-updates to [mysql] nder -/.my.cnf ko noon MySQL salii huutoraade hay gooto UPDATE walla DELETE haala ka ŋakki a WHERE konngol keyngol walla a LIMIT.
• Uddit DeletionProtection = true e dow denndaangal lowre RDS peewnugol, buckets S3, e kaɓirɗe Terraform stateful (lifecycle { prevent_destroy = true }).
• Doole kuule ndeenka calɗi Git on main / production: Softinde Allow force pushes e Allow deletions e dow calɗi reentorɗi e nder GitHub/GitLab ko noon hay njuɓɓudi defterdu waawaa git push --force to main.
• Sosde jokkere enɗam sirlu (gitleaks walla trufflehog) e GitHub Push Reende: Blok ina waɗi AKIA, sk_live_, walla .env patterns nokkuuji ko adii git commit hay ina daña.

Naamne keewɗe naamneede¶

Mbele mi riiwaama sabu mi woppii tabalde domen peewnugol e dow majjere walla mi waɗii lowre ndee ustaade?¶

E nder juɓɓule injiniyankooɓe celluɗe kuutortooɗe pinal kisal ngal alaa ko bonnata (standard across modern tech, DevOps, e SRE teams), injiniyankooɓe ndartinaaka ngam majjere aadee nuunɗunde e nder peewnugol, so tawii ɓe kabrii ko waɗi koo ɗoon e ɗoon e wallitde artirde sarwiis. Ardiiɓe innjiniyaaruuji ina nganndi wonde so tawii ƴellitoowo tokooso walla mawɗo ina joginoo yamiroore raw e ŋakkeende gardiiɗo ngam jippinde taƴre peewnugol e yamiroore wootere nde ƴeewtaaka, sabaabu mum ko ŋakkeende e jokkondire e kuutorɗe ƴellitgol, wonaa bonannde. E nder golle, injenieer mo meeɗaa yiyde oon jolngo e junngo mum, ko hannde woni neɗɗo gooto ɓurɗo kisal e nder kippu oo ko fayti e oon sifaa. Giɗli tan jogorɗi arde e jokkondire ko etaade suɗaade kewu nguu, momtude binndanɗe ƴeewndo, walla fenaande e nder wiɗto kewu nguu.

Hol ko waɗi mi fotaani waylude ɗoon e ɗoon jolngo am peewnugol nguurndam feewde e innde lowre domen Point-In-Time Recovery (PITR) keso artiraande?¶

So a momtii tan ko rowrowre e nder tabalde wootere (yeru, the... `user_preferences` table) e waktu 14:00 UTC, e ɗum ƴetta hojomaaji 20 ngam wirnude PITR clone artiraa e 13:59:50 UTC, lowre maa peewnugol nguurndam ina waawi jokkude e ƴellitde ujunnaaje ujunnaaje yamiroore soodooɓe kesi laawɗuɗi e yoɓde e nder `orders` taabal hakkunde 14w00 e 14w20h UTC. So a ɓuuɓtii denndaangal toɓɓere DNS peewnugol maa e dow 13:59:50 PITR clone, aɗa momta ɗeen hojomaaji 20 jamirooje soodooɓe kese. E nder ɗuum, jogo lowre mawnde ndee e internet (e binndi dartinaaɗi tan e mbaadi taƴndi ndii) e yaltinde tan ko rowrowre ŋakkunde e 13:59:50 PITR clone ngartiraa e tabalde peewnugol nguurndam.

So mi waɗii defterdu GitHub renndo Private hojomaaji ɗiɗi caggal nde pus-mi an .env file, mbele miɗo foti waylude denndaangal kuutorɗe API ɗee haa jooni?¶

Eey, 100 e nder teemedere e ɗeen coktirgal ina poti haa jooni woppeede e wirneede ɗoon e ɗoon. Kala gollal pusngal e defterdu GitHub renndo ina yaltinee e sahaa tigi-rigi e dow API GitHub Events renndo ("the Firehose") e indeks e nder leƴƴanɗe seeɗa e sarwisaaji arsiif kisal laawɗuɗi (hono GHArchive) e botuuji bonɗi otomaatik mooftirɗi seedanteeje cloudcross/ lowre 24. So a ƴettii yiyngo defterdu nduu gila e Public haa e Private hojomaaji ɗiɗi caggal ɗuum, momtataa koppi mo ƴoogirɗe otomaatik ƴettunoo gila jooni.

Hol no waɗirta git push --force-with-lease haɗde ustaare so yerondiraama e git push --force?¶

Nde a dogata standart `git push --force`, Git ina winnda caltal to woɗɗi ngal e caltal maa nokkuure, tawa ina halka kala ko sehil maa pusnoo e oon caltal e nder ɗuum. Nde kuutorto-ɗaa `git push --force-with-lease` (e ko ɓuri ɗum moƴƴude, `--force-if-includes`), Git ƴeewtotoo serveer to woɗɗi oo ko adii ngam hoolkisaade so tawii tip branch to woɗɗi oo ina yahdi e commit SHA mo masiŋ maa nokku oo sakkitii ƴettude. So tawii gollodiiɗo maa walla pipeline CI pusii alkule kese to branch remote gila e fetch maa cakkitiiɗo, `--force-with-lease` aborts e kisal e jeertinaango "stale info" e nokku momtugol golle maɓɓe.

Fuɗɗoode ƴeewtaama¶

• Duɗal ngenndiwal toppitiingal tolnooji e karallaagal (NIST, Amerik): NIST SP 800-61 Rev. 2 / Rev. 3 Ardorde jokkondire e keɓe kisal ordinateer e jokkondire hakkunde jokkere enɗam.
• Sarwisaaji Web Amazon (AWS), Google Cloud Platform (GCP), e Binndanɗe Microsoft Azure (Dental Dowlaaji Amerik): Arsitektuur RDS ngam ƴellitde toɓɓere-e-waktu (PITR), AWS IAM kuule kuutorteeɗe ngam haɓaade mboros oo (AWSCompromisedKeyQuarantineV2), e S3 Version Mumtugol-Maande Restoraasiyoŋ.
• Fedde ƴellitaare winndereere PostgreSQL e binndanɗe MySQL / Oracle (hakkunde leyɗeele): Binnditagol PostgreSQL (WAL), Yiyngo Tuple MVCC, e Ƴellitaare Binnditagol Binnditagol Binnditagol MySQL tuugingol e Rowre.
• Git SCM Binndanɗe laawɗuɗe e Lab Kisal GitHub (Hakkunde Leyɗeele): Git Reflog (git-reflog), Ƴeewndo Geɗe Dangling (git-fsck), Binndol daartol (git-filter-repo), e Reende Push Skannde Sirlu.
• Defte Fedde USENIX / SREcon e Innjiniyaaruuji Koolkisagol Lowre Google (SRE) (Hakkunde Leyɗeele): Njuɓɓudi Kewuuji, Triage Rollback Emergency, e Pinal Postmortem mo alaa aybe.

Ardiiɓe jokkondirɓe¶

• Kisal golle to woɗɗi e sirlu dokke
• Backups ordinateer 3-2-1 peeje e reentaade musiiba
• Mumtugol fiilngo e aksidaa e ƴellitgol diisnondirɗo
• Eggugol kiriis jaɓɓugol web e moƴƴitingol backup
• Phishing e konte mbonɗe
• Limooje emergency