Accidental Production Outage, Leaked API Keys, or Deleted Cloud Infrastructure¶
Every software engineer, systems administrator, and technical remote worker eventually faces the stomach-dropping moment when a command meant for localhost eða staging executes against production: an unindexed DELETE án a WHERE clause, a DROP TABLE eða terraform destroy, a git push --force sem skrifar yfir aðalgreinina, eða opinbera skuldbindingu sem inniheldur lifandi AWS eða Stripe leynilykla. Það sem skilur 15 mínútna minniháttar hiksta frá gagnaslysi sem lýkur starfsferli er að stöðva hendurnar strax, frysta skrifumferð áður en geymslukubbum er skrifað yfir og framkvæma rétta batareglur.
Gerðu þetta núna¶
- Frystu hendurnar og EKKI keyra skyndilegar „fix“ skipanir, endurræstu gagnagrunnsþjóninn eða lokaðu flugstöðvarglugganum þínum. Núverandi biðminni fyrir flugstöðina þína, skeljasögu og staðbundinn
.git/logsreflog contain the exact transaction timestamps, commit SHA hashes, and resource IDs needed to reverse the damage. Copy your entire terminal scrollback into a safe text file first. - Declare a Severity-1 (SEV-1) incident in your team's incident Slack/Teams channel within 2 minutes. Never spend 45 minutes trying to secretly fix a broken production database alone while customers are hitting 500 errors and automated background jobs are corrupting the remaining tables. Page the on-call Database Administrator (DBA) or Site Reliability Engineer (SRE) immediately.
- If you leaked an API key,
.envfile, or cloud secret to a public GitHub repo, REVOKE the key in the provider dashboard FIRST (before trying to rewrite Git history). Automated botnets scrape the public GitHub Firehose API and execute stolen AWS, OpenAI, Stripe, and GCP keys within 3 to 15 seconds of yourgit push. Deleting the file in a second commit or force-pushing does not stop an attacker who already cloned the key. - If you ran an accidental
DELETE,UPDATE, eðaDROPon a production database, cut application WRITE traffic immediately. Put the web application into Maintenance Mode or revoke the app user's write permissions so new incoming transactions do not overwrite unvacuumed dead tuples or complicate Point-In-Time Recovery (PITR). - Never run
VACUUM,OPTIMIZE TABLE, eðagit gc --prune=nowafter an accidental deletion. Both relational databases (PostgreSQL MVCC, MySQL InnoDB) and Git keep deleted rows and orphaned commits intact on disk until a garbage-collection/vacuum pass physically reclaims the pages. - If a
terraform destroyor cloud deletion script is still running in your terminal, pressCtrl + CONCE (not twice) or kill the IAM session token. PressingCtrl + Conce triggers a graceful state flush so Terraform saves the partial.tfstateof what still exists; pressingCtrl + Ctwice forces an ungraceful abort that can corrupt or lose your local state file.
Danger
The #1 Fatal Mistake After Leaking a Secret in Git: Never think you are safe because you ran git rm .env && git commit -m "remove secret" && git push. Every historical commit in Git remains publicly readable forever in the commit graph, and even if you overwrite the branch with git push --force, the orphaned commit SHA remains accessible via the GitHub API and public archive crawlers (GHArchive). Any secret that touches a remote repository for even 1 second must be treated as 100 percent compromised and revoked/rotated at the issuing provider immediately.
Emergency triage workflow: database drop, git force-push, or secret leak¶
Follow this decision tree to route your first 5 minutes of technical containment.
flowchart TD
A["Production Engineering Emergency"] --> B{"Which failure just occurred?"}
B -->|"Accidental SQL DROP / DELETE / UPDATE in Prod"| C["Stop App Writes / Maintenance Mode; Do NOT Run VACUUM"]
B -->|"Leaked API Key / .env / Private Key in Git"| D["Revoke & Rotate Secret in Cloud Dashboard Within 60 Seconds"]
B -->|"Accidental git push --force / Deleted Branch"| E["Run git reflog Locally; Copy Last Good Commit SHA"]
B -->|"Accidental terraform destroy / Deleted S3 or VM"| F["Kill IAM Session or Press Ctrl+C Once; Preserve .tfstate"]
C --> G["Record Exact UTC Timestamp; Trigger Cloud PITR Clone or Binlog Flashback"]
D --> H["Audit CloudTrail / Access Logs for Unauthorized Spin-Ups; Scrub Repo with git-filter-repo"]
E --> I["Query GitHub Events / Reflog & Restore Branch with git push --force-with-lease"]
F --> J["Check S3 Versioning Delete Markers, EBS Snapshots, or Cloud Recycle Bin"]
Part 1: Accidental production database DROP, TRUNCATE, eða DELETE without WHERE¶
When a destructive SQL query hits a live production database, the exact recovery mechanism depends on your database engine (PostgreSQL, MySQL/MariaDB, SQL Server, MongoDB, or DynamoDB) and whether the statement was a DML (DELETE / UPDATE) or DDL (DROP TABLE / TRUNCATE) operation.
Step 1: Record the exact UTC timestamp down to the second¶
Before touching any backup console, write down the exact current UTC time (date -u in Linux/macOS terminal, or SELECT NOW(); inside the SQL prompt) and estimate the exact second just before your destructive query started executing. Point-In-Time Recovery (PITR) works by replaying the database's Write-Ahead Log (WAL in PostgreSQL) or Binary Log (binlog in MySQL) up to T_minus_1_second right before your bad query arrived.
Step 2: Stop background jobs and application writes (Protect MVCC tuples)¶
In PostgreSQL, executing DELETE FROM users; eða UPDATE orders SET status = 'cancelled'; does not actually erase a single byte of data from the table file on disk right away. Because PostgreSQL uses Multi-Version Concurrency Control (MVCC), a DELETE simply sets the xmax transaction ID header on each row tuple to mark it as invisible to future queries, leaving the complete row data sitting intact inside the base/ heap file until autovacuum comes along to mark the space reusable.
- If you are on self-hosted or VM PostgreSQL and have no recent backup: Immediately connect as
superuserand disable autovacuum on that table (ALTER TABLE table_name SET (autovacuum_enabled = false);), or stop the PostgreSQL service and take a raw block-level filesystem snapshot (LVM/EBSsnapshot) of the/var/lib/postgresql/datadirectory. Specialized PostgreSQL forensic extensions (pg_dirtyread) can read and recover 100 percent of deleted or pre-update rows directly out of the unvacuumed heap table in minutes.
Database recovery matrix by engine and cloud platform¶
| Database Platform | Fastest Recovery Method for Accidental DROP / DELETE |
Step-by-Step Technical Execution |
|---|---|---|
| AWS RDS / Aurora (PostgreSQL / MySQL) | Automated Point-In-Time Recovery (PITR) to a New Instance (Restores to any second within the 1-to-35-day retention window) | 1. Open AWS RDS Console -> Automated backups -> Select your Prod DB -> Restore to point in time. 2. Choose Custom date and time and enter the exact UTC timestamp 10 to 30 seconds BEFORE your bad query ran. 3. RDS spins up a parallel clone ( prod-db-recovery) in 10 to 25 minutes. Dump the intact table (pg_dump -t my_table eða mysqldump) from the recovery clone and import it back into the live Prod DB (or swap the DNS CNAME/endpoint to the recovered clone). |
| Google Cloud SQL / Azure Database | Point-In-Time Clone | Í GCP stjórnborðinu -> Veldu Cloud SQL tilvik -> Klón -> Veldu Klón frá fyrri tímapunkti (UTC) -> Dragðu út vantar línur úr klóninu í gegnum pg_dump / mysqldump eða erlend gagnaumbúðir (postgres_fdw). |
| Supabase / Neon / PlanetScale | Instant Branch / Time-Travel Restore | Neon: Búðu til samstundis afrita-í-skrifa útibú frá sögulegum tímastimpli (Timestamp eða LSN 1 mínútu síðan) á 2 sekúndum og afritaðu töfluna til baka.Supabase: Mælaborð -> Gagnagrunnur -> Öryggisafrit -> Tímabundin endurheimt. PlanetScale: Notaðu hnappinn fyrir skema til baka / spóla til baka í notendaviðmóti dreifingarbeiðnarinnar. |
Self-Hosted MySQL / MariaDB (Með binlog_format = ROW) |
Binlog Inversion (binlog2sql / mysqlbinlog --flashback) |
Þegar MySQL keyrir með binlog_format = ROW (sjálfgefið síðan MySQL 5.7.7), hver DELETE eða UPDATE skrifar heildarmyndina fyrir og eftir hverja línu sem verður fyrir áhrifum inn í tvíundarskrána (mysql-bin.0000XX). Hlaupa binlog2sql --flashback (eða MariaDB mysqlbinlog --flashback) á þeirri nákvæmu færslustöðu til að búa til sjálfkrafa nákvæmlega andstæðan INSERT eða UPDATE SQL forskrift og endurspilaðu það á 60 sekúndum án þess að endurheimta fullt öryggisafrit. |
| Snjókorn / BigQuery (skýjagagnavöruhús) | Innbyggt SQL tímaferð (AT / BEFORE) |
Núll afrita endurheimt þörf. Spurðu töfluna eins og hún var til fyrir 5 mínútum: Snjókorn: CREATE OR REPLACE TABLE my_table AS SELECT * FROM my_table AT(OFFSET => -60*5); (eða UNDROP TABLE my_table;).BigQuery: SELECT * FROM dataset.my_table FOR SYSTEM_TIME AS OF TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 10 MINUTE); |
| MongoDB Atlas / eftirlíkingarsett | Oplog PITR endurheimt eða seinkað aukameðlimur | Í MongoDB Atlas: Afritun -> Endurheimta -> Tímapunktur (endurspilar local.oplog.rs söfnun allt að nákvæmri sekúndu fyrir fall). |
Hluti 2: API lyklar sem lekið hafa, .env leyndarmál, eða einkalyklar framdir til Git¶
Ef þú framdir óvart .env skrá, an id_rsa SSH lykill, Firebase/GCP þjónustureikning JSON skrá, Stripe leynilykill (sk_live_...), eða AWS AKIA... aðgangslykill að Git, fylgdu nákvæmlega þessari tveggja fasa röð: Fasi A (dreptu lykilinn í skýinu) fyrst og Phase B (skrúbbaðu Git geymsluna) í öðru lagi.
Áfangi A: Strax 60 sekúndna innilokun skýjaskilríkja (Gerðu þetta FYRST)¶
1. AWS (AKIA... Auðkenni aðgangslykils + leynilegur aðgangslykill)¶
Þegar vélmenni stela AWS lykli frá GitHub lendir sjálfvirka handritið þeirra á reikninginn þinn innan 10 sekúndna til að snúa upp hámarksstærð GPU EC2 tilvika (p4d.24xlarge / g5.48xlarge) á öllum 30+ alþjóðlegum AWS svæðum (þar á meðal svæðum sem eru opt-in eins og Jakarta, Osaka og Miðausturlönd) til að grafa dulritunargjaldmiðil eða búa til bakdyra IAM notanda með AdministratorAccess áður en þú eyðir upprunalega lyklinum.
- Slökktu strax á og eyddu aðgangslyklinum sem lekið var:
- Opnaðu AWS Console -> IAM -> Notendur -> [Notandi í hættu] -> Öryggisskilríki -> Aðgangslyklar -> Smelltu á Slökkva, síðan á Eyða (eða í gegnum CLI:
aws iam update-access-key --access-key-id AKIAIOSFODNN7EXAMPLE --status Inactive --user-name MyUser).
- Opnaðu AWS Console -> IAM -> Notendur -> [Notandi í hættu] -> Öryggisskilríki -> Aðgangslyklar -> Smelltu á Slökkva, síðan á Eyða (eða í gegnum CLI:
- Hengdu skýringu við
DenyAllpolicy to neutralize temporaryaws stssession tokens:- Critical trap: Even after you delete an
AKIA...IAM access key, if the attacker already calledsts:GetSessionTokeneðasts:AssumeRoleduring the first 30 seconds, their temporaryASIA...session token remains valid for up to 1 to 12 hours unless you attach an explicit inline Deny policy (such asAWSCompromisedKeyQuarantineV2or a Deny-all policy for tokens issued before the current timestamp) to the user or role.
- Critical trap: Even after you delete an
- Audit AWS CloudTrail for backdoor persistence:
- Open AWS CloudTrail -> Event history and filter by the compromised Access Key ID or User name over the past 2 hours.
- Check specifically for these 5 persistence calls:
CreateUser,CreateAccessKey,AttachUserPolicy,AssumeRole, ogRunInstances. Check AWS Cost Explorer and EC2 Global View across all regions to terminate any rogue instances, and open an immediate AWS Support Case (Account and Billing -> Compromised Account). AWS routinely waives fraudulent compute bills when you report a key leak and terminate instances promptly.
2. Stripe (sk_live_...), OpenAI (sk-...), SendGrid, Twilio, and Slack Tokens¶
- Stripe: Go to Developers -> API keys -> Standard keys / Restricted keys -> Click the three dots (
...) -> Roll key. Choose "Expire now" (or set a 1-hour grace window only if you are actively deploying the new key into your production environment variables right now). - OpenAI / Anthropic: Go to the API Keys dashboard and click Revoke / Delete immediately so automated proxy scrapers cannot drain thousands of dollars of LLM token credits in minutes.
- GitHub Personal Access Tokens (PATs), Fine-Grained Tokens, and Deploy Keys: Go to GitHub Settings -> Developer settings -> Personal access tokens and click Delete / Revoke on the leaked token, then check your organization's Audit Log (
Settings -> Audit log) forrepo.clone,repo.access, or unauthorized webhook creation during the exposure window. - Database Connection Strings (
DATABASE_URL/postgres:///mongodb+srv://) and Redis URLs: Change the database user's password inside the database engine (ALTER USER app_user WITH PASSWORD 'new_random_secret';), and then terminate all currently open active TCP connections belonging to the old credentials (SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE usename = 'app_user' AND pid <> pg_backend_pid();in PostgreSQL), because changing a database password in PostgreSQL or MySQL only blocks new logins and leaves an attacker's already-open connection live until kicked.
Phase B: Scrubbing the secret from Git history (git filter-repo & GitHub cache eviction)¶
Once the leaked key is dead and rotated in production, clean the secret string out of your Git commit history so security scanners stop alerting:
- Rewrite repository history using
git filter-repo(or BFG Repo-Cleaner):- Take a fresh bare mirror clone of your repository and remove the leaked file (for example,
.env) from every historical commit across all branches and tags:
- Take a fresh bare mirror clone of your repository and remove the leaked file (for example,
- Force-push the cleaned history and purge cached GitHub Pull Request / Commit views:
- Push the cleaned history back to origin (
git push origin --force --alloggit push origin --force --tags). - Important GitHub cached-commit step: Even after you force-push a cleaned history, anyone who knows the old 40-character commit SHA URL (
github.com/org/repo/commit/abc1234...) can still view the old uncleaned commit on GitHub's web UI until GitHub runs server-side garbage collection. Contact GitHub Support (or use the repository's Security Advisory / Support portal) with the exact old commit SHA and ask them to evict the dereferenced/dangling commit from the web cache and rungit gcon the remote repository.
- Push the cleaned history back to origin (
Part 3: Undoing an accidental git push --force, git reset --hard, or deleted branch¶
If you accidentally ran git push --force and wiped out your teammates' commits on main, or ran git reset --hard HEAD-5 and lost a full day of local work, do not panic, because Git almost never deletes committed objects immediately.
Why your "lost" commits are still sitting safely inside .git/logs (The Reflog)¶
Alltaf þegar þú býrð til commit, skiptir um grein, endurbasar eða endurstillir, skráir Git það fyrra HEAD og greinarábendingar SHA-1/SHA-256 ábendingar inni í staðbundinni höfuðbók sem kallast Reference Log (git reflog), og geymir raunverulega munaðarlausa commit hluti á disknum í að minnsta kosti 14 til 90 daga áður en sorphirða fjarlægir þá.
Skref-fyrir-skref uppskriftir fyrir Git neyðarbata¶
Atburðarás 1: Þú hljópst git push --force og skrifaði yfir origin/main¶
- ** Finndu nákvæma skuldbindingu SHA sem
origin/mainbenti á rétt fyrir þvingunarýtinguna þína:**- Aðferð A (Athugaðu staðbundið fjarmælingarafritið þitt): Keyrðu þessa skipun í flugstöðinni þar sem þú ýtir:
Þú munt sjá úttak eins og:
Here,
a1b2c3d (HEAD -> main, origin/main) refs/remotes/origin/main@{0}: update by push 9f8e7d6 refs/remotes/origin/main@{1}: fetch: fast-forward9f8e7d6(origin/main@{1}) is the exact commitorigin/mainwas on right before your force-push overwrote it. - Aðferð B (Spyrðu hvaða liðsfélaga sem hefur ekki hlaupið
git pullenn): Allir vinnufélagar sem sóttumainearlier today still has the intact commit history sitting in their localorigin/mainútibú. - Method C (Query the GitHub REST API Events log): If your local reflog lacks the remote state, query GitHub's push events API to see the
"before"SHA hash of the force-push event:
- Aðferð A (Athugaðu staðbundið fjarmælingarafritið þitt): Keyrðu þessa skipun í flugstöðinni þar sem þú ýtir:
Þú munt sjá úttak eins og:
- Restore
origin/mainto the previous commit SHA safely:- Verify the commit looks right (
git log -n 5 9f8e7d6), then push that exact SHA back to the remotemainbranch using--force-with-lease:
- Verify the commit looks right (
Scenario 2: You ran git reset --hard or deleted a local branch before pushing¶
- Run
git reflogin your local repository: - Locate the commit SHA right before the
reset:entry (e.g.,HEAD@{2}: commit: feat: add payment webhook), and create a new rescue branch pointing directly at that SHA: - What if you lost an uncommitted file that you
git added (staged) and then wiped out withgit reset --hard?- The moment you run
git add myfile.ts, Git compresses and writes the full contents of that file into.git/objects/as a dangling blob, even if you never rangit commit. Recover it by listing all dangling blobs sorted by modification time:
- The moment you run
Part 4: Accidental terraform destroy or deleted cloud storage (S3, EBS, VMs)¶
If an Infrastructure-as-Code (IaC) pipeline or manual CLI command starts tearing down production cloud resources:
- Stöðva framkvæmdina á öruggan hátt:
- If running locally in your terminal, press
Ctrl + CONCE so Terraform finishes its current API call and writes the updated.tfstatefile cleanly. If running inside GitHub Actions, GitLab CI, or Terraform Cloud, click Cancel Run and immediately disable the CI runner's AWS/GCP IAM role in the cloud console so in-flight API deletes fail withAccessDenied.
- If running locally in your terminal, press
- Back up the current
.tfstatefile immediately:- Download a local copy of your remote state file (
terraform state pull > emergency-backup.tfstate) and check S3 bucket versioning on your Terraform state bucket to grab the previous state version from 10 minutes ago.
- Download a local copy of your remote state file (
- Check cloud provider "Soft-Delete" and Recycle Bin safety nets:
- Amazon S3 (With Bucket Versioning enabled): Deleting files in a versioned S3 bucket (
aws s3 rm --recursive) does not delete a single object; it merely inserts a zero-byte Delete Marker on top of each object. Remove the Delete Markers (via S3 Batch Operations or a 5-line Pythonboto3script) to restore 100 percent of your files instantly. - AWS EBS Volumes & RDS Snapshots (Recycle Bin): Check AWS Recycle Bin in the console if configured, and check AWS Backup vaults.
- Google Cloud Platform (GCP) & Microsoft Azure: Deleted GCP Projects and Azure Resource Groups / Key Vaults / Storage Accounts with Soft Delete enabled remain recoverable for 7 to 30 days via a single restore click in the console.
- Amazon S3 (With Bucket Versioning enabled): Deleting files in a versioned S3 bucket (
How to recover a broken Terraform .tfstate state file or re-import orphaned infrastructure¶
Þegar a terraform apply or accidental terraform destroy is interrupted mid-flight, or when someone accidentally deletes the remote state file in S3/GCS, your cloud infrastructure enters a dangerous split-brain state where some resources still exist in AWS/GCP while others are missing from .tfstate:
- Recovering an overwritten or deleted remote
.tfstatefile from S3 / GCS Versioning:- Never manually edit a JSON
.tfstatefile by hand in a text editor unless every other option has failed, because a single mismatched"serial"eða"lineage"UUID string will block Terraform from ever applying again. - Open your S3 bucket (or Google Cloud Storage bucket / Terraform Cloud Workspace States tab) where the remote backend lives, click the Show versions toggle at the top of the S3 console, find the
terraform.tfstateversion timestamped right before the bad run, download it, and push it back safely as the new current state:
- Never manually edit a JSON
- Breaking a stuck state lock (
Error acquiring the state lock):- When you abort a bad
terraform destroymeðCtrl + Cor kill a hung CI runner, the remote DynamoDB table or GCS lock file remains locked by the dead process ID. - First verify in your CI dashboard and with your teammates that zero other Terraform processes are actually still running against that workspace, then release the orphaned lock using the Lock ID UUID printed in the error message:
- When you abort a bad
- Rebuilding a partially destroyed environment without recreating surviving databases (
terraform import):- If a
terraform destroywiped out your stateless Application Load Balancers (aws_lb) and ECS/EKS worker nodes before you stopped it, that is easy to fix: simply runterraform applyand Terraform will recreate all stateless networking and compute layers from scratch in 5 minutes while leaving your surviving RDS database untouched. - If your state file lost track of a surviving stateful database or S3 bucket, use an
import {}configuration block (Terraform 1.5+) orterraform import aws_db_instance.prod_db my-rds-instance-idto bind the existing cloud resource back into your state file without destroying it, and runterraform planto confirm"0 to destroy"before applying.
- If a
Hluti 5: Hvernig á að koma á framfæri bilun til yfirmanns þíns og leiða saklausa skurðaðgerð¶
Þegar þú veldur framleiðsluatviki á meðan þú ert að vinna í fjarvinnu er faglegt orðspor þitt skilgreint ekki af mistökunum sjálfum, heldur af hraða þínum á gagnsæi, rólegum samskiptum og kerfisbundnum úrbótum.
Gullnu reglurnar 3 um samskipti atvika¶
- Tilgreinið staðreyndir, áhrif og ETA án þess að fela, tefja eða kenna um: Tilkynntu villuna innan 2 mínútna frá uppgötvun.
- Aðskiljið „Mitigation“ (Stoppið blæðinguna) frá „Root-Cause Fix“: Í neyðartilvikum er eina markmiðið þitt að endurheimta þjónustuna (afturkalla slæma dreifingu, mistakast yfir í PITR gagnagrunnsklóninn eða fletta eiginleikafánanum af). Vistaðu djúpa kóða endurvinnslu fyrir morgundaginn.
- Settu reglulega 15 mínútna hjartsláttaruppfærslur í atviksrásinni svo stjórnendur og þjónustuver viti að einhver er við stjórnvölinn.
Copy-paste Slack / Teams Incident Declaration Script¶
[SEV-1 TILVAL] Framleiðsla [Þjónusta / Nafn gagnagrunns] Áhrif - Hvað gerðist: Kl
[HH:MM UTC], meðan verið er að framkvæma[brief technical action, e.g., a data migration script], óviljandi[DELETE / config change / force-push]var beitt til framleiðslu[database / service]. - Núverandi áhrif viðskiptavina:[e.g., Checkout API is returning 500 errors / User profile updates are paused]. - Tafar í vinnslu: 1. Skrifaðu umferð til[table/service]hefur verið gert hlé til að varðveita gagnaheilleika. 2. Point-In-Time Recovery (PITR) klóna til[HH:MM:SS UTC](10 sekúndum fyrir viðburðinn) er núna að snúast upp[AWS RDS / GCP]. - Aðvikastjóri:@[Your Name](aðstoð af@[DBA/On-Call SRE]). - Næsta uppfærsla: Eftir 15 mínútur ([HH:MM UTC]) eða um leið og bata klóninn kemur á netið.
Gátlisti fyrir verkfræðilegar öryggisráðstafanir eftir atvik (Fyrir saklausa eftirárás)¶
Sérhver mannleg mistök í framleiðslu eru í raun og veru kerfishönnunarbil (ef ein afritaða skipun í flugstöðinni getur sleppt framleiðslu án varnar, þá beið kerfið eftir að mistakast). Leggðu til og útfærðu þessar steinsteyptu tæknilegu varnarliði í skurðarskjalinu þínu daginn eftir:
- Tilkynningar um litakóða og forskeyti framleiðslu (
kube-ps1,psql.psqlrc): Stilltu skel og SQL biðlara þannig að allar tengingar viðPRODUCTIONsýnir skærrauður[PROD-DB]hvetja og byrjar hverja gagnvirka PostgreSQL lotu í aBEGIN;viðskiptablokk sjálfgefið (eða seturSET default_transaction_read_only = on;). - ** Virkja
safe-updates(--i-am-a-dummy) í MySQL/MariaDB viðskiptavinum:** Bæta viðsafe-updatestil[mysql]inn-/.my.cnfsvo MySQL neitar beinlínis að framkvæma neittUPDATEeðaDELETEyfirlýsingu sem skortir aWHERElykilákvæði eða aLIMIT. - Kveiktu á
DeletionProtection = trueá öllum framleiðslu RDS gagnagrunnum, S3 fötum og Terraform staðbundnum tilföngum (lifecycle { prevent_destroy = true }). - ** Framfylgja verndarreglum Git útibúa á
main/production:** SlökkvaAllow force pushesogAllow deletionsá vernduðum útibúum í GitHub/GitLab svo jafnvel geymslustjórar geta það ekkigit push --forcetilmain. - Install pre-commit secret scanning hooks (
gitleakseðatrufflehog) and GitHub Push Protection: Block commits containingAKIA,sk_live_, eða.envpatterns locally beforegit commiteven succeeds.
Algengar spurningar¶
Will I be fired for accidentally dropping a production database table or causing a site outage?¶
In healthy engineering organizations that practice a blameless safety culture (standard across modern tech, DevOps, and SRE teams), engineers are not fired for honest human mistakes in production, provided they report the incident immediately and help restore service. Engineering leaders recognize that if a junior or senior developer had the raw permissions and lack of guardrails to drop a production table with a single unreviewed command, the root cause is a flaw in access control and deployment tooling, not malice. In practice, the engineer who just experienced that outage first-hand is now the single safest person on the team regarding that system. The only behavior that consistently leads to termination is attempting to cover up a data loss event, deleting audit logs, or lying during the incident investigation.
Why shouldn't I immediately switch my live production traffic over to the newly restored Point-In-Time Recovery (PITR) database instance?¶
If you only accidentally deleted rows from a single table (for example, the `user_preferences` table) at 14:00 UTC, and it takes 20 minutes to spin up a PITR clone restored to 13:59:50 UTC, your live production database may have continued processing thousands of legitimate new customer orders and payments in the `orders` table between 14:00 and 14:20 UTC. If you blindly swing your whole production DNS endpoint over to the 13:59:50 PITR clone, you will wipe out those 20 minutes of brand-new customer orders. Instead, keep the main database online (with writes paused only on the broken feature) and export just the missing rows from the 13:59:50 PITR clone back into the live production table.
Ef ég geri opinbera GitHub geymslu að einkaaðila tveimur mínútum eftir að ég ýtti óvart á .env skrá, þarf ég samt að snúa öllum API lyklum?¶
Já, enn verður að afturkalla 100 prósent af þessum lyklum og snúa þeim strax. Sérhver skuldbinding sem ýtt er á opinbera GitHub geymslu er send út í rauntíma í gegnum opinbera GitHub Events API („Eldslangan“) og skráð innan nokkurra sekúndna af bæði lögmætum öryggisskjalaþjónustum (eins og GHArchive) og illgjarnum sjálfvirkum skilríkjauppskeru vélmennum sem keyra 24/7 á skýjanetum. Að fletta sýnileika geymslunnar úr Public í Private tveimur mínútum síðar eyðir ekki afritinu sem sjálfvirkar sköfur hafa þegar hlaðið niður.
Hvernig virkar git push --force-with-lease koma í veg fyrir truflanir miðað við git push --force?¶
Þegar þú keyrir standard `git push --force`, Git skrifar skilyrðislaust yfir ytri útibúið með staðbundnu útibúinu þínu og eyðileggur hljóðlaust allar skuldbindingar sem liðsfélagi hefur ýtt í greinina á meðan. Þegar þú notar `git push --force-with-lease` (og jafnvel betra, `--force-if-includes`), Git athugar fyrst ytra netþjóninn til að ganga úr skugga um að ytri útibúsábendingin passi enn nákvæmlega við commit SHA staðbundna vélin þín sem síðast var sótt. Ef samstarfsmaður eða CI leiðsla ýtti nýrri skuldbindingu til ytri útibúsins síðan þú sóttir síðast, `--force-with-lease` hættir á öruggan hátt með viðvörun um „gamal upplýsingar“ í stað þess að þurrka út vinnu þeirra.
Heimildir athugaðar¶
- National Institute of Standards and Technology (NIST, Bandaríkin): NIST SP 800-61 Rev. 2 / Rev. 3 Leiðbeiningar um meðhöndlun tölvuöryggisatvika og innilokun skilríkja.
- Amazon Web Services (AWS), Google Cloud Platform (GCP) og Microsoft Azure Documentation (Bandaríkin): RDS Automated Point-In-Time Recovery (PITR) Architecture, AWS IAM Compromised Key Quarantine Policys (
AWSCompromisedKeyQuarantineV2), og S3 Versioning Delete-Marker Recovery. - PostgreSQL Global Development Group og MySQL / Oracle Documentation (alþjóðleg): PostgreSQL Write-Ahead Logging (WAL), MVCC Tuple Visibility og MySQL Row-Based Binary Log Flashback Recovery.
- Git SCM Official Documentation and GitHub Security Lab (International): Git Reflog (
git-reflog), Endurheimt hangandi hluta (git-fsck), Endurskrifa sögu (git-filter-repo), og Secret Scanning Push Protection. - USENIX Association / SREcon og Google Site Reliability Engineering (SRE) bækur (alþjóðlegar): Stjórnun atvika, neyðartilvikum, og saklausri skurðaðgerð.