Okuvaako kw'okufulumya mu butanwa, Ebisumuluzo bya API Ebikulukuta, oba Ebikozesebwa mu Kire Ebisaziddwamu¶

Buli yinginiya wa pulogulaamu, omuddukanya enkola, n’omukozi ow’ekikugu ow’ewala okukkakkana ng’ayolekedde akaseera akagwa olubuto ng’ekiragiro kigendereddwamu localhost oba staging atta nga avuganya production: ekintu ekitaliiko index DELETE awatali a WHERE akawaayiro, a DROP TABLE oba terraform destroy, a git push --force that overwrites the main branch, or a public commit containing live AWS or Stripe secret keys. What separates a 15-minute minor hiccup from a career-ending data catastrophe is stopping your hands immediately, freezing write traffic before storage blocks are overwritten, and executing the right recovery protocol.

Kino kikole kati¶

1. Freeze your hands and do NOT run hasty "fix" commands, restart the database server, or close your terminal window. Your current terminal scrollback buffer, shell history, and local .git/logs reflog contain the exact transaction timestamps, commit SHA hashes, and resource IDs needed to reverse the damage. Copy your entire terminal scrollback into a safe text file first.
2. Declare a Severity-1 (SEV-1) incident in your team's incident Slack/Teams channel within 2 minutes. Never spend 45 minutes trying to secretly fix a broken production database alone while customers are hitting 500 errors and automated background jobs are corrupting the remaining tables. Page the on-call Database Administrator (DBA) or Site Reliability Engineer (SRE) immediately.
3. If you leaked an API key, .env file, or cloud secret to a public GitHub repo, REVOKE the key in the provider dashboard FIRST (before trying to rewrite Git history). Automated botnets scrape the public GitHub Firehose API and execute stolen AWS, OpenAI, Stripe, and GCP keys within 3 to 15 seconds of your git push. Deleting the file in a second commit or force-pushing does not stop an attacker who already cloned the key.
4. If you ran an accidental DELETE, . UPDATE, oba DROP on a production database, cut application WRITE traffic immediately. Put the web application into Maintenance Mode or revoke the app user's write permissions so new incoming transactions do not overwrite unvacuumed dead tuples or complicate Point-In-Time Recovery (PITR).
5. Never run VACUUM, . OPTIMIZE TABLE, oba git gc --prune=now after an accidental deletion. Both relational databases (PostgreSQL MVCC, MySQL InnoDB) and Git keep deleted rows and orphaned commits intact on disk until a garbage-collection/vacuum pass physically reclaims the pages.
6. If a terraform destroy or cloud deletion script is still running in your terminal, press Ctrl + C ONCE (not twice) or kill the IAM session token. Pressing Ctrl + C once triggers a graceful state flush so Terraform saves the partial .tfstate of what still exists; pressing Ctrl + C twice forces an ungraceful abort that can corrupt or lose your local state file.

Emergency triage workflow: database drop, git force-push, or secret leak¶

Follow this decision tree to route your first 5 minutes of technical containment.

flowchart TD
    A["Production Engineering Emergency"] --> B{"Which failure just occurred?"}
    B -->|"Accidental SQL DROP / DELETE / UPDATE in Prod"| C["Stop App Writes / Maintenance Mode; Do NOT Run VACUUM"]
    B -->|"Leaked API Key / .env / Private Key in Git"| D["Revoke & Rotate Secret in Cloud Dashboard Within 60 Seconds"]
    B -->|"Accidental git push --force / Deleted Branch"| E["Run git reflog Locally; Copy Last Good Commit SHA"]
    B -->|"Accidental terraform destroy / Deleted S3 or VM"| F["Kill IAM Session or Press Ctrl+C Once; Preserve .tfstate"]
    C --> G["Record Exact UTC Timestamp; Trigger Cloud PITR Clone or Binlog Flashback"]
    D --> H["Audit CloudTrail / Access Logs for Unauthorized Spin-Ups; Scrub Repo with git-filter-repo"]
    E --> I["Query GitHub Events / Reflog & Restore Branch with git push --force-with-lease"]
    F --> J["Check S3 Versioning Delete Markers, EBS Snapshots, or Cloud Recycle Bin"]

Part 1: Accidental production database DROP, . TRUNCATE, oba DELETE without WHERE¶

When a destructive SQL query hits a live production database, the exact recovery mechanism depends on your database engine (PostgreSQL, MySQL/MariaDB, SQL Server, MongoDB, or DynamoDB) and whether the statement was a DML (DELETE / . UPDATE) oba DDL (DROP TABLE / . TRUNCATE) okulongoosa.

Omutendera 1: Wandiika timestamp entuufu eya UTC okutuuka ku eyookubiri¶

Nga tonnakwata ku backup console yonna, wandiika obudde obutuufu obwa UTC obuliwo (date -u mu Linux/macOS terminal, oba SELECT NOW(); inside the SQL prompt) and estimate the exact second just before your destructive query started executing. Point-In-Time Recovery (PITR) works by replaying the database's Write-Ahead Log (WAL in PostgreSQL) or Binary Log (binlog in MySQL) up to T_minus_1_second right before your bad query arrived.

Step 2: Stop background jobs and application writes (Protect MVCC tuples)¶

In PostgreSQL, executing DELETE FROM users; oba UPDATE orders SET status = 'cancelled'; does not actually erase a single byte of data from the table file on disk right away. Because PostgreSQL uses Multi-Version Concurrency Control (MVCC), a DELETE simply sets the xmax transaction ID header on each row tuple to mark it as invisible to future queries, leaving the complete row data sitting intact inside the base/ heap file until autovacuum comes along to mark the space reusable.

• If you are on self-hosted or VM PostgreSQL and have no recent backup: Immediately connect as superuser and disable autovacuum on that table (ALTER TABLE table_name SET (autovacuum_enabled = false);), or stop the PostgreSQL service and take a raw block-level filesystem snapshot (LVM / . EBS snapshot) of the /var/lib/postgresql/data directory. Specialized PostgreSQL forensic extensions (pg_dirtyread) can read and recover 100 percent of deleted or pre-update rows directly out of the unvacuumed heap table in minutes.

Database recovery matrix by engine and cloud platform¶

Ekitundu 2: Ebisumuluzo bya API ebivuddemu, . .env ebyama, oba ebisumuluzo eby'ekyama ebiweereddwayo eri Git¶

Singa mu butanwa wakola an .env fayiro, an id_rsa Ekisumuluzo kya SSH, fayiro ya JSON ya akawunti y'empeereza ya Firebase/GCP, ekisumuluzo eky'ekyama ekya Stripe (sk_live_...), oba AWS AKIA... ekisumuluzo ky'okuyingira ku Git, goberera omutendera guno gwennyini ogw'emitendera ebiri: Omutendera A (Tta ekisumuluzo mu kire) okusooka, ne Omutendera B (Scrub the Git repository) ogwokubiri.

Omutendera A: Okuziyiza amangu ebiwandiiko by’ekire mu ssekindi 60 (Kino okikole OKUSOOKA) .¶

1. AWS (AKIA... ID y'ekisumuluzo ky'okuyingira + Ekisumuluzo ky'okuyingira eky'ekyama)¶

Bots bwe babba ekisumuluzo kya AWS okuva mu GitHub, script yaabwe ey’obwengula ekuba akawunti yo mu sikonda 10 okukyusakyusa ebifaananyi bya GPU EC2 eby’obunene obusinga obunene (p4d.24xlarge / . g5.48xlarge) mu bitundu byonna 30+ eby’ensi yonna ebya AWS (nga mw’otwalidde n’ebitundu eby’okulonda nga Jakarta, Osaka, ne Middle East) okusima cryptocurrency, oba okukola backdoor IAM user ne AdministratorAccess nga tonnasazaamu kisumuluzo ekyasooka.

1. Deactivate era Delete Access Key eyafulumye amangu ddala:
   • Ggulawo AWS Console -> IAM -> Abakozesa -> [Omukozesa alina obuzibu] -> Ebiwandiiko by'obukuumi -> Ebisumuluzo by'okuyingira -> Nywa ku Deactivate, olwo Delete (oba ng'oyita mu CLI: aws iam update-access-key --access-key-id AKIAIOSFODNN7EXAMPLE --status Inactive --user-name MyUser).
2. Teekako ekiwandiiko ekitegeerekeka DenyAll enkola ey’okumalawo eby’ekiseera aws sts obubonero bw'olutuula:
   • Critical trap: Ne bw'omala okusazaamu an AKIA... IAM access key, singa omulumbaganyi yayita dda sts:GetSessionToken oba sts:AssumeRole mu sikonda 30 ezisooka, zaabwe ez’ekiseera ASIA... session token esigala nga ntuufu okumala essaawa 1 okutuuka ku 12 okuggyako nga ogattiddeko enkola etegeerekeka mu layini Deny (nga AWSCompromisedKeyQuarantineV2 oba enkola ya Deny-all ku tokens ezifulumiziddwa nga timestamp eriwo kati tennabaawo) eri omukozesa oba omulimu.
3. Okubalirira AWS CloudTrail okusobola okugumiikiriza mu mulyango ogw'emabega:
   • Ggulawo AWS CloudTrail -> Ebyafaayo by'ebibaddewo era osengejje okusinziira ku Access Key ID oba Erinnya ly'omukozesa erikoseddwa mu ssaawa 2 eziyise.
   • Kebera mu ngeri ey’enjawulo ku ssimu zino 5 ez’okugumiikiriza: CreateUser, . CreateAccessKey, . AttachUserPolicy, . AssumeRole, era RunInstances. Kebera AWS Cost Explorer ne EC2 Global View mu bitundu byonna okukomya ebikolwa byonna ebikyamu, era oggulewo AWS Support Case (Account and Billing -> Compromised Account) ey'amangu. AWS bulijjo ereka ssente z'okubalirira ez'obufere ng'oloopa okukulukuta kw'ekisumuluzo n'okukomyawo emirundi mu bwangu.

2. Omusono (sk_live_...), OpenAI (sk-...), Obubonero bwa SendGrid, Twilio, ne Slack¶

• Stripe: Genda ku Abakola -> Ebisumuluzo bya API -> Ebisumuluzo ebya bulijjo / Ebisumuluzo ebikugirwa -> Nywa ku bubonero busatu (...) -> Okuyiringisiza ekisumuluzo. Londa "Expire now" (oba teeka eddirisa ly'ekisa ery'essaawa 1 yokka singa oba okola nnyo okuteeka ekisumuluzo ekipya mu nkyukakyuka zo ez'embeera y'okufulumya mu kiseera kino).
• OpenAI / Anthropic: Genda ku dashboard ya API Keys onyige Revoke / Delete amangu ago kale automated proxy scrapers tezisobola kuggyawo nkumi na nkumi za doola za LLM token credits mu ddakiika ntono.
• GitHub Personal Access Tokens (PATs), Fine-Grained Tokens, ne Deploy Keys: Genda ku GitHub Settings -> Developer settings -> Personal access tokens onyige Delete / Revoke ku kabonero akafulumye, olwo okebere Audit Log y'ekitongole kyo (Settings -> Audit log) a repo.clone, . repo.access, oba okutonda webhook okutakkirizibwa mu kiseera ky'eddirisa ly'okubikkula.
• Emiguwa gy'okuyunga database (DATABASE_URL / . postgres:// / . mongodb+srv://) ne Redis URLs: Kyusa ekigambo ky'omukozesa wa database munda mu yingini ya database (ALTER USER app_user WITH PASSWORD 'new_random_secret';), n'oluvannyuma okukomya enkolagana zonna eza TCP ezikola eziggule mu kiseera kino ez'ebiwandiiko ebikadde (SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE usename = 'app_user' AND pid <> pg_backend_pid(); mu PostgreSQL), kubanga okukyusa password ya database mu PostgreSQL oba MySQL kiziyiza okuyingira okupya kwokka era kireka omukutu gw'omulumbaganyi ogwaggulwawo edda nga gulamu okutuusa nga gukubiddwa.

Omutendera B: Okusenya ekyama okuva mu byafaayo bya Git (git filter-repo & Okugoba cache ya GitHub)¶

Ekisumuluzo ekivuddemu bwe kimala okufa era nga kikyusiddwa mu kukola, yoza olunyiriri olw'ekyama okuva mu byafaayo byo ebya Git commit olwo sikaani z'ebyokwerinda zikomye okulabula:

1. Ddamu okuwandiika ebyafaayo by'etterekero ng'okozesa git filter-repo (oba BFG Repo-Okuyonja):
   • Ddira ekifaananyi ekipya eky’endabirwamu etaliiko kintu kyonna eky’etterekero lyo era oggyewo fayiro evuddemu (okugeza, .env) okuva mu buli commit ey'ebyafaayo okubuna amatabi gonna ne tags:

     # 1. Remove a specific file (.env) from all commits in history:
     git filter-repo --invert-paths --path .env
     
     # OR 2. Replace a specific secret string with ***REMOVED*** everywhere:
     echo "sk_live_YOUR_SECRET_STRING==>***REMOVED***" > replacements.txt
     git filter-repo --replace-text replacements.txt
     

2. Force-push ebyafaayo ebiyonjo era olongoose cached GitHub Pull Request / Commit views:
   • Sindika ebyafaayo ebiyonjo okudda ku nsibuko (git push origin --force --all era git push origin --force --tags).
   • Ekikulu GitHub cached-commit step: Ne bw'omala okukaka-okusika ebyafaayo ebiyonjo, omuntu yenna amanyi enkadde 40-ennukuta commit SHA URL (github.com/org/repo/commit/abc1234...) ekyasobola okulaba commit enkadde etalongooseddwa ku web UI ya GitHub okutuusa nga GitHub ekola okukung'aanya kasasiro ku ludda lwa seva. Tuukirira GitHub Support (oba kozesa omukutu gw'okubuulirira / Obuwagizi ogw'etterekero) ne commit SHA enkadde entuufu era obasabe okugoba dereferenced/dangling commit okuva mu web cache ne badduka git gc ku tterekero ery'ewala.

Ekitundu 3: Okuzzaawo ekintu ekibaddewo mu butanwa git push --force, . git reset --hard, oba ettabi eryasaziddwamu¶

Singa wadduka mu butanwa git push --force era n'osangulawo commits za banno ku main, oba yadduka git reset --hard HEAD-5 era n'afiirwa olunaku olujjuvu olw'emirimu gy'omu kitundu, totya, kubanga Git kumpi tesazaamu bintu biweereddwayo mu bwangu.

Lwaki "lost" commits zo zikyatudde bulungi munda .git/logs (Ekitongole kya Reflog)¶

Buli lw'okola commit, okukyusa amatabi, rebase, oba reset, Git ewandiika eby'emabega HEAD ne branch tip SHA-1/SHA-256 pointers munda mu ledger y'ekitundu eyitibwa Reference Log (git reflog), era ekuuma ebintu byennyini eby'okussa mulekwa ku disiki okumala waakiri ennaku 14 ku 90 nga okukung'aanya kasasiro tekunnaba kuziggyawo.

Omutendera ku mutendera Git enkola z'okudda engulu mu bwangu¶

Ensonga 1: Wadduka git push --force era n’awandiika waggulu origin/main¶

1. Funa commit entuufu SHA nti origin/main yasonga ku right before your force-push:
   • Enkola A (Kebera remote-tracking reflog yo ey'omu kitundu): Kozesa ekiragiro kino mu terminal gye wasika:

     git reflog show origin/main
     

     Ojja kulaba ebifulumizibwa nga:

     a1b2c3d (HEAD -> main, origin/main) refs/remotes/origin/main@{0}: update by push
     9f8e7d6 refs/remotes/origin/main@{1}: fetch: fast-forward
     

     Wano, 9f8e7d6 (origin/main@{1}) ye commit yennyini origin/main yali ku right nga force-push yo tennagiwandiika.
   • Enkola B (Buuza munno yenna atadduse git pull yet): Omukozi munne yenna eyaleeta main emabegako leero akyalina ebyafaayo bya commit ebitaliiko kamogo nga bitudde mu kitundu kyabwe origin/main olusaga.
   • Enkola C (Buuza ekiwandiiko kya GitHub REST API Events): Singa reflog yo ey'omu kitundu ebulwa embeera ey'ewala, buuza GitHub's push events API okulaba... "before" SHA hash y'omukolo gw'okusika amaanyi:

     gh api /repos/OWNER/REPO/events --jq '.[] | select(.type=="PushEvent") | {actor: .actor.login, before: .payload.before, head: .payload.head, time: .created_at}'
     

2. Okuwona origin/main ku commit SHA eyasooka mu ngeri ey'obukuumi:
   • Kakasa nti commit erabika bulungi (git log -n 5 9f8e7d6), olwo onyige SHA eyo yennyini okudda ku remote main ettabi nga likozesa --force-with-lease:

     git push --force-with-lease origin 9f8e7d6:refs/heads/main
     

Ensonga 2: Wadduka git reset --hard oba yasazaamu ettabi ly’omu kitundu nga tannasika¶

1. Okudduka git reflog mu tterekero lyo ery'omu kitundu:

   git reflog --date=iso
   

2. Funa commit SHA nga... reset: okuyingira (e.g., HEAD@{2}: commit: feat: add payment webhook), era okole ettabi eppya ery’okununula nga lisonga butereevu ku SHA eyo:

   git switch -c rescue-branch HEAD@{2}
   

3. Ate singa ofiirwa fayiro uncommitted nti ggwe git added (staged) n’oluvannyuma n’asangulwamu ne git reset --hard? ?**
   • Akaseera k’odduka git add myfile.ts, Git enyigiriza era n'ewandiika ebirimu mu fayiro eyo mu bujjuvu mu .git/objects/ nga dangling blob, ne bw'oba toddukangako git commit. Kizuule ng'owandiika ebiwujjo byonna ebilengejja nga bisunsuddwa okusinziira ku budde bw'okukyusa:

     git fsck --lost-found
     # Inspect the recovered blobs written into .git/lost-found/other/
     

Ekitundu 4: Obubenje terraform destroy oba okutereka mu kire okusaziddwamu (S3, EBS, VMs) .¶

Singa payipu ya Infrastructure-as-Code (IaC) oba ekiragiro kya CLI eky’omu ngalo kitandika okumenya eby’obugagga by’ekire ky’okufulumya:

1. Komya okutta mu ngeri ey'obukuumi:
   • Bw’oba ​​okola mu kitundu mu terminal yo, nyweza ** .Ctrl + C ONCE** kale Terraform emaliriza okuyita kwayo kwa API okuliwo kati n'ewandiika ebirongooseddwa .tfstate fayiro muyonjo. Bw'oba oddukira munda mu GitHub Actions, GitLab CI, oba Terraform Cloud, nyweza Cancel Run era amangu ago olemese omulimu gw'omuddusi wa CI AWS/GCP IAM mu cloud console kale okusazaamu API mu nnyonyi kulemererwa AccessDenied.
2. Back up eby'omulembe .tfstate fayiro amangu ago:
   • Wano wefunire kkopi ya wano eya fayiro yo ey’embeera ey’ewala (terraform state pull > emergency-backup.tfstate) era kebera S3 bucket versioning ku Terraform state bucket yo okukwata state version eyasooka okuva mu ddakiika 10 eziyise.
3. Kebera obutimba bw'obukuumi bw'omugabi w'ebire "Soft-Delete" ne Recycle Bin:
   • Amazon S3 (Nga Bucket Versioning ekoleddwa): Okusazaamu fayiro mu baketi ya S3 eriko enkyusa (aws s3 rm --recursive) te kusazaamu kintu kimu; kimala kuyingiza zero-byte Delete Marker waggulu wa buli kintu. Ggyawo Delete Markers (nga oyita mu S3 Batch Operations oba Python eya layini 5 boto3 script) okuzzaawo fayiro zo ebitundu 100 ku 100 mu bwangu.
   • AWS EBS Volumes & RDS Snapshots (Recycle Bin): Kebera AWS Recycle Bin mu console bwe kiba nga kitegekeddwa, era kebera AWS Backup vaults.
   • Google Cloud Platform (GCP) & Microsoft Azure: Pulojekiti za GCP ezisaziddwamu ne Azure Resource Groups / Key Vaults / Storage Accounts nga Soft Delete zisobozeseddwa zisigala nga ziddamu okumala ennaku 7 okutuuka ku 30 nga ziyita mu kunyiga omulundi gumu ogw'okuzzaawo mu console.

Engeri y'okuzzaawo Terraform eyamenyese .tfstate fayiro ya gavumenti oba okuddamu okuyingiza ebikozesebwa ebitaliimu bamulekwa¶

Bwe a terraform apply oba mu butanwa terraform destroy esaliddwako wakati mu nnyonyi, oba omuntu bw’asazaamu mu butanwa fayiro y’embeera ey’ewala mu S3/GCS, enkola yo ey’ekire eyingira mu mbeera ey’akabi ey’obwongo obw’enjawulo ng’eby’obugagga ebimu bikyaliwo mu AWS/GCP ate ebirala nga bibulamu .tfstate:

1. Okuzzaawo remote ewandiikiddwa oba esaziddwamu .tfstate fayiro okuva mu S3 / GCS Enkyusa:
   • Tolongoosa JSON mu ngalo .tfstate fayiro n'omukono mu mulongoosa w'ebiwandiiko okuggyako nga buli nkola endala eremereddwa, kubanga emu tekwatagana "serial" oba "lineage" UUID string ejja kuziyiza Terraform okuddamu okusaba.
   • Ggulawo baketi yo eya S3 (oba Google Cloud Storage bucket / Terraform Cloud Workspace States tab) awali backend ewala, nyweza Show versions toggle waggulu ku S3 console, funa... terraform.tfstate version timestamped right before the bad run, giwanule, era gizze emabega mu ngeri ey'obukuumi ng'embeera empya eriwo kati:

     # Push a recovered historical state file back to the remote backend safely
     terraform state push recovered-pre-incident.tfstate
     

2. Okumenya kkufulu ya state ekwatiddwa (Error acquiring the state lock):
   • Bw’oggyamu olubuto olubi terraform destroy ne Ctrl + C oba okutta omuddusi wa CI awaniriddwa, emmeeza ya DynamoDB ey'ewala oba fayiro y'okusiba GCS esigala nga esibiddwa ID y'enkola enfu.
   • Sooka okakasa mu dashboard yo eya CI era ne banno nti zero enkola endala eza Terraform mu butuufu zikyadduka ku kifo ekyo eky'okukoleramu, olwo osumulule kkufulu ya mulekwa ng'okozesa Lock ID UUID ewandiikiddwa mu bubaka bw'ensobi:

     terraform force-unlock <LOCK_ID_UUID>
     

3. Okuddamu okuzimba embeera eyasaanawo ekitundu awatali kuddamu kutondawo databases ezisigaddewo (terraform import):
   • Singa a terraform destroy yasangulawo Application Load Balancers zo ezitaliiko mbeera (aws_lb) ne ECS/EKS worker nodes nga tonnagiyimiriza, ekyo kyangu okutereeza: simply run terraform apply ne Terraform ejja kuddamu okukola emikutu gyonna egy’embeera etaliiko n’okubalirira layers okuva ku ntandikwa mu ddakiika 5 ate nga ereka database yo eya RDS esigaddewo nga tekwatibwako.
   • Singa fayiro yo ey'essaza yabula okulondoola database y'embeera oba ekibbo kya S3 ekisigaddewo, kozesa ** .import {} okusengeka block** (Terraform 1.5+) oba terraform import aws_db_instance.prod_db my-rds-instance-id okusiba eky'obugagga ky'ekire ekiriwo okudda mu fayiro yo ey'embeera nga tokisaanyizzaawo, n'okudduka terraform plan okukakasa "0 to destroy" nga tonnaba kusaba.

Ekitundu 5: Engeri y’okumanyisaamu maneja wo ku kuvuddeko amasannyalaze n’okukulembera okukebera omulambo nga tewali musango¶

When you cause a production incident while working remotely, your professional reputation is defined not by the mistake itself, but by your speed of transparency, calm communication, and systematic remediation.

The 3 Golden Rules of Incident Communication¶

1. Yogera ensonga, okukosebwa, ne ETA awatali kwekweka, kulwawo, oba okunenya: Loopa ensobi mu ddakiika 2 okuva lwe ozudde.
2. Separate "Mitigation" (Stopping the bleeding) from "Root-Cause Fix": In an emergency, your only goal is restoring service (rolling back the bad deploy, failing over to the PITR database clone, or flipping the feature flag off). Save deep code refactoring for tomorrow.
3. Post regular 15-minute heartbeat updates in the incident channel so executives and customer support know someone is in control.

Copy-paste Slack / Teams Ekiwandiiko ky'okulangirira ebibaddewo¶

[SEV-1 INCIDENT] Okufulumya [Erinnya ly'Empeereza / Database] Ebikosa - Ekyabaddewo: Ku [HH:MM UTC], nga bw’akola [brief technical action, e.g., a data migration script], ekintu ekitali kigenderere [DELETE / config change / force-push] yakozesebwa mu kukola ebintu [database / service]. - Ekikosa bakasitoma mu kiseera kino: [e.g., Checkout API is returning 500 errors / User profile updates are paused]. - Okukendeeza ku buzibu mu bwangu kugenda mu maaso: 1. Wandiika traffic ku [table/service] eyimiriziddwa okukuuma obulungi bwa data. 2. Okuzzaawo ensonga mu kiseera (PITR) clone to [HH:MM:SS UTC] (nga wabulayo sekondi 10 omukolo tegunnabaawo) mu kiseera kino ewuuta mu [AWS RDS / GCP]. - Omuduumizi w'ebibaddewo: @[Your Name] (nga bayambibwako... @[DBA/On-Call SRE]). - Ekiddako: Mu ddakiika 15 ([HH:MM UTC]) oba amangu ddala nga clone y’okudda engulu ezze ku mutimbagano.

Olukalala lw’okukebera obukuumi bwa yinginiya oluvannyuma lw’akabenje (For your Blameless Post-Mortem)¶

Buli nsobi y'omuntu mu kukola mu butuufu eba ekituli mu nteekateeka y'enkola (singa ekiragiro kya terminal kimu ekya copy-pasted kisobola okusuula okufulumya awatali guardrail, enkola yali erinze okulemererwa). Teesa era oteeke mu nkola ebikuuma bino eby’ekikugu ebya seminti mu kiwandiiko kyo eky’okukebera omulambo enkeera:

• Ebiragiro by'okufulumya langi-code n'entandikwa (kube-ps1, . psql .psqlrc): Tegeka shell yo ne SQL client bwekityo okuyungibwa kwonna ku PRODUCTION eraga emmyuufu eyakaayakana [PROD-DB] okubuuza era n’atandika buli lutuula lwa PostgreSQL olukwatagana munda mu a BEGIN; transaction block nga bwekiba (oba sets SET default_transaction_read_only = on;).
• Okuyinzisa safe-updates (--i-am-a-dummy) mu bakasitoma ba MySQL/MariaDB: Yongera safe-updates ku [mysql] mu -/.my.cnf kale MySQL outright egaana okukola kyonna UPDATE oba DELETE ekigambo ekibulamu a WHERE akawaayiro akakulu oba a LIMIT.
• Ggya ku mulamwa DeletionProtection = true ku database zonna eza RDS ez'okufulumya, baketi za S3, n'ebikozesebwa mu mbeera ya Terraform (lifecycle { prevent_destroy = true }).
• Okussa mu nkola amateeka g'okukuuma ettabi lya Git ku... main / . production: Okulemala Allow force pushes era Allow deletions ku matabi agakuumibwa mu GitHub/GitLab kale n'abaddukanya etterekero tebasobola git push --force ku main.
• Teeka ebikoola bya sikaani eby'ekyama nga tonnaba kukola (gitleaks oba trufflehog) ne GitHub Push Protection: Block ekola nga erimu AKIA, . sk_live_, oba .env patterns mu kitundu nga tezinnabaawo git commit n’okutuuka ku buwanguzi.

Ebibuuzo ebitera okubuuzibwa¶

Nnagobwa olw'okusuula emmeeza ya database y'okufulumya mu butanwa oba okuvaako omukutu okuvaako?¶

Mu bibiina bya yinginiya ebiramu ebikola obuwangwa bw’obukuumi obutaliimu musango (omutindo mu ttiimu zonna eza tekinologiya ow’omulembe, DevOps, ne SRE), bayinginiya tebagobwa olw’ensobi z’abantu ez’amazima mu kukola, kasita baba nga baloopa ekibaddewo amangu ddala era ne bayamba okuzzaawo empeereza. Abakulembeze ba yinginiya bakimanyi nti singa omukugu omuto oba omukulu yalina olukusa olusookerwako n’obutaba na bikondo okusuula emmeeza y’okufulumya n’ekiragiro kimu ekitannaba kwekenneenyezebwa, ekikolo ekivaako kibeera kizibu mu kufuga okuyingira n’ebikozesebwa mu kuteeka mu nkola, so si bubi. Mu nkola, yinginiya eyaakalaba okuvaako okwo ku lulwe kati ye muntu omu asinga obukuumi ku ttiimu ku bikwata ku nkola eyo. Enneeyisa yokka eviirako bulijjo okukomyawo kwe kugezaako okubikka ku kintu ekibaddewo okufiirwa data, okusazaamu ebiwandiiko by’okubala ebitabo, oba okulimba mu kiseera ky’okunoonyereza ku bibaddewo.

Lwaki sisaanidde kukyusa mangu ntambula yange ey'okufulumya obutereevu okudda ku nkola ya database ya Point-In-Time Recovery (PITR) eyaakazzibwawo?¶

Singa wasazaamu ennyiriri mu butanwa zokka okuva ku mmeeza emu (okugeza,... `user_preferences` emmeeza) ku ssaawa 14:00 UTC, era kitwala eddakiika 20 okuwuuta PITR clone ekomezeddwawo ku 13:59:50 UTC, database yo ey’okufulumya obutereevu eyinza okuba nga yagenda mu maaso n’okukola enkumi n’enkumi z’ebiragiro bya bakasitoma abapya ebituufu n’okusasula mu `orders` emmeeza wakati w’essaawa 14:00 ne 14:20 UTC. Singa muzibe w’amaaso owuuba DNS endpoint yo yonna ey’okufulumya okutuuka ku 13:59:50 PITR clone, ojja kusangulawo eddakiika ezo 20 ez’ebiragiro bya bakasitoma ebipya ddala. Mu kifo ky'ekyo, kuuma database enkulu ku mutimbagano (nga ebiwandiiko biyimiridde ku kifaananyi ekimenyese kyokka) era fulumya ennyiriri zokka ezibula okuva mu 13:59:50 PITR clone okudda mu mmeeza y'okufulumya obutereevu.

Singa nkola etterekero lya GitHub ery'olukale Private eddakiika bbiri oluvannyuma lw'okusika mu butanwa an .env fayiro, nkyalina okukyusakyusa ebisumuluzo byonna ebya API?¶

Yee, ebisumuluzo ebyo ebitundu 100 ku 100 bikyalina okusazibwamu n’okukyusibwa amangu ddala. Buli kwewaayo okunyigirizibwa mu tterekero lya GitHub ery'olukale kuweebwa ku mpewo mu kiseera ekituufu ku GitHub Events API ey'olukale ("the Firehose") era n'ewandiikibwa mu sikonda ntono okuva mu mpeereza zombi ez'okutereka eby'okwerinda mu mateeka (nga GHArchive) ne bots ez'okukungula ebiwandiiko eby'obwengula ezikola 24/7 mu mikutu gy'ekire. Okukyusa okulabika kw'etterekero okuva ku Public okudda ku Private oluvannyuma lw'eddakiika bbiri tekisangula kkopi automated scrapers gye yawanula edda.

Akola atya git push --force-with-lease okutangira okuvaako kw’amasannyalaze bw’ogeraageranya ne git push --force?¶

Bw’odduka standard `git push --force`, Git awatali bukwakkulizo ewandiika ku ttabi ery'ewala n'ettabi lyo ery'omu kitundu, mu kasirise n'esaanyaawo commits zonna munne ze yasika ku ttabi eryo mu kiseera kino. Bw’oba ​​okozesa `git push --force-with-lease` (era n’okusingawo, `--force-if-includes`), Git esooka kukebera seva ey’ewala okukakasa nti ensonga y’ettabi ery’ewala ekyakwatagana n’okussaawo SHA entuufu ekyuma kyo eky’omu kitundu kye kyasembayo okuleeta. Singa mukozi munno oba payipu ya CI yasika commit empya ku ttabi ery'ewala okuva lwe wasembayo okuleeta, . `--force-with-lease` aborts safely with a "stale info" warning mu kifo ky'okusangula emirimu gyabwe.

Ensonda zeekebeddwa¶

• National Institute of Standards and Technology (NIST, United States): NIST SP 800-61 Rev. 2 / Rev. 3 Ekitabo ky’okukwata ebizibu by’obukuumi bwa kompyuta n’okuziyiza okukkaanya ku biwandiiko.
• Amazon Web Services (AWS), Google Cloud Platform (GCP), ne Microsoft Azure Documentation (Amerika): Enzimba ya RDS ey’okuzzaawo ebifo mu kiseera (PITR), Enkola za Kalantiini ezikulu eza AWS IAM ezikoseddwa (AWSCompromisedKeyQuarantineV2), ne S3 Versioning Okuzzaawo-Obubonero.
• Ekibiina ky'enkulaakulana y'ensi yonna ekya PostgreSQL ne MySQL / Oracle Documentation (Ensi Yonna): Okuwandiika mu maaso PostgreSQL (WAL), MVCC Tuple Visibility, ne MySQL Row-Based Binary Log Flashback Recovery.
• Git SCM Ebiwandiiko ebitongole ne GitHub Security Lab (Ensi Yonna): Git Reflog (git-reflog), Okuzzaawo Ebintu Ebiwaniridde (git-fsck), Okuddamu Okuwandiika Ebyafaayo (git-filter-repo), and Secret Scanning Push Protection.
• USENIX Association / SREcon and Google Site Reliability Engineering (SRE) Books (International): Managing Incidents, Emergency Rollback Triage, and Blameless Postmortem Culture.

Ebiragiro ebikwatagana nabyo¶

• Emergency remote work security and data privacy
• Computer backups 3-2-1 enkola n'okuzzaawo akatyabaga
• Accidental file deletion and formatted drive recovery
• Web hosting crisis migration and backup restoration
• Phishing and compromised accounts
• Ennamba z’amangu