Impanuka y'Umuhinguzi, Imfunguruzo za API zasohotse, canke Ibikorwa remezo vy'Igicu Vyasubiwemwo¶

Injeniyeri wese wa porogarama, umuyobozi wa sisitemu be n’umukozi wese akora kure, amaherezo arashikirwa n’igihe co gutera inda igihe itegeko ryari rigenewe . localhost canke staging kwica ku production: ikintu kitari mu rutonde DELETE ata WHERE ingingo, a DROP TABLE canke terraform destroy, a git push --force iyo ihindura ishami ry'ingenzi, canke isezerano rya bose ririmwo imfunguruzo z'ibanga za AWS canke Stripe. Igitandukanya iminota 15 y’agahinda gatoyi n’ivyago vy’amakuru bishobora guhera akazi ni uguhagarika ibiganza vyawe ubwo nyene, guhagarika uruja n’uruza rw’abanditsi imbere y’uko amabuye y’ububiko yandikwako, no gushitsa uburyo bwo gusubirana bubereye.

Ivyo ubikore ubu¶

1. Funga ibiganza vyawe kandi NTUkoreshe amabwirizwa yihuta "gukosora", wongere utangure umukozi w'urutonde rw'amakuru, canke ufunge idirisha ryawe ry'inyuma. Ububiko bwawe bwo gusubira inyuma, amateka y'igikoko, n'aho hantu .git/logs reflog irimwo ibimenyetso vy'igihe nyavyo vy'ibikorwa, gukora SHA hashes, n'Indangamuntu z'ibikoresho bikenewe kugira ngo usubize inyuma ivyononekaye. Kopa igice cawe cose c'iherezo muri dosiye y'inyandiko itekanye mbere.
2. Menya ivyabaye vy’uburemere-1 (SEV-1) mu muhora w’ivy’ivyabaye w’umugwi wawe mu minota 2. Ntukigere umara iminota 45 ugerageza gukosora mu mpisho urutonde rw’amakuru y’ibikorwa rwavunitse wenyene mu gihe abakiriya bariko baratera amakosa 500 kandi ibikorwa vy’inyuma vy’ubuhinga bwa none biriko birahonya ameza asigaye. Page umuyobozi w’urutonde rw’amakuru (DBA) canke Injeniyeri w’ukwizigirwa kw’urubuga (SRE) ubwo nyene.
3. Niba wavuze urufunguzo rwa API, .env dosiye, canke ibanga ry'igicu ku bubiko bwa GitHub bwa bose, SUBIRAMWO urufunguzo mu dashboard y'umutanga MBERE (mbere yo kugerageza kwandika amateka ya Git). Botnets zikora zicapura GitHub Firehose API ya bose maze zigakora AWS, OpenAI, Stripe, na GCP ziyibwe mu **5s zawe mu segonda 53 git push. Gukuraho dosiye mu bikorwa vya kabiri canke gusunika ku nguvu ntibihagarika umuterabwoba yari amaze gukora clone y'urufunguzo.
4. Iyo wirutse mu mpanuka DELETE, UPDATE, canke DROP ku rutonde rw'amakuru rw'ibikorwa, uce uruja n'uruza rw'ibikorwa WRITE ubwo nyene. Shira iporogarama y'urubuga mu buryo bwo gucungera canke ukureho uburenganzira bwo kwandika bw'ukoresha porogaramu kugira ngo amafaranga mashasha yinjira ntiyandike hejuru y'ibipfuye bitagira ubusa canke ngo bigorane Gusubirana kw'Igihe (PITR).
5. Ntukigere wiruka VACUUM, OPTIMIZE TABLE, canke git gc --prune=now inyuma yo gukuraho mu mpanuka. Ivyo bitabo vyose bibiri (PostgreSQL MVCC, MySQL InnoDB) na Git bigumya imirongo yasubiwemwo n'ibikorwa vy'impfuvyi bitagiramwo ikintu kuri disiki gushika igihe co gukuraho imyanda/ubusa gisubira gusaba amapaji.
6. Nimba a terraform destroy canke inyandiko yo gukuraho igicu iracariko irakora muri terminal yawe, kanda Ctrl + C RImwe (si kabiri) canke wice ikimenyetso c'ikiganiro ca IAM. Gukanda Ctrl + C rimwe rivyura ubuntu bw'igihugu rero Terraform ikiza igice .tfstate y’ivyo bikiriho; gukanda Ctrl + C kabiri bihatira gukura inda ata buntu bishobora kwonona canke gutakaza dosiye yawe ya leta.

Ivyihutirwa gutoranya ibikorwa: gutonyanga urutonde rw'amakuru, gusunika inguvu za git, canke gusohoka kw'ibanga¶

Kurikiza iki giti c’ingingo kugira ngo ukoreshe iminota 5 ya mbere y’ugukingira ubuhinga.

flowchart TD
    A["Ivyihutirwa vy'ubuhinga bwo guhingura"] --> B{"Ni ukunanirwa ukuhe kwabaye?"}
    B -->|"Impanuka SQL GUTERA / GUSUBIZA / GUSUBIZA MU GIHUGU muri Prod"| C["Hagarika App Yandika / Uburyo bwo Gucungera; NTUKIRUKE UBUSA"]
    B -->|"Urufunguzo rwa API rwasohotse / .env / Urufunguzo rwihariye muri Git"| D["Gukuraho no guhindura ibanga mu gicu Dashboard mu masegonda 60"]
    B -->|"Impanuka git gusunika --inkomezi / Ishami ryasubiwemwo"| E["Gukoresha git reflog Mu karere; Kopa Ivyiza vya nyuma Ukora SHA"]
    B -->|"Impanuka terraform gusenyura / Yasubiwemwo S3 canke VM"| F["Kwica Igihe ca IAM canke Kanda Ctrl+C Rimwe; Kuzigama .tfstate"]
    C --> G["Kwandika Igihe nyaco ca UTC; Igicu co gutera PITR Clone canke Binlog gusubira inyuma"]
    D --> H["Igenzura CloudTrail / Gushika ku bitabo vy’Ibikorwa vy’Ivyiyumviro Bitaremewe; Gusukura Repo na git-akayunguruzo-repo"]
    E --> I["Ibaza GitHub Ivyabaye / Reflog & Gusubizaho Ishami na git push --gukoresha-n'ugukodesha"]
    F --> J["Suzuma S3 Versioning Gukuraho Ibimenyetso, EBS Amafoto, canke Igicu Gusubiramwo Igikoresho"]

Igice ca 1: Ububiko bw'amakuru y'impanuka DROP, TRUNCATE, canke DELETE ata WHERE¶

Iyo ikibazo ca SQL gisenya giteye urutonde rw'amakuru y'ibikorwa, uburyo nyabwo bwo kugarura buvana na moteri yawe y'urutonde rw'amakuru (PostgreSQL, MySQL/MariaDB, Server ya SQL, MongoDB, canke DynamoDB) be n'uko iyo mvugo yari DML (DELETE / UPDATE) canke DDL (DROP TABLE / TRUNCATE) igikogwa.

Intambwe ya 1: Andika ikidodo nyaco c'isaha ya UTC gushika ku ya kabiri¶

Imbere yo gukora kuri console iyo ari yo yose yo gusubiza inyuma, wandike isaha nyayo ya UTC (date -u mu nzira ya Linux/macOS, canke SELECT NOW(); imbere mu SQL prompt) maze ugereranye isegonda nyayo mbere y'uko ikibazo cawe co gusenyura gitangura gukora. Igihe co gusubirana (PITR) gikora mu gusubiramwo urutonde rw'amakuru rwo kwandika imbere (WAL muri PostgreSQL) canke Igitabo ca kabiri (binlog muri MySQL) gushika kuri T_minus_1_second imbere gato y’uko ikibazo cawe kibi gishika.

Intambwe ya 2: Hagarika ibikorwa vy'inyuma n'inyandiko z'ibisabwa (Kingira ama tuples ya MVCC)¶

Mu Inyuma ya SQL, gushitsa DELETE FROM users; canke UPDATE orders SET status = 'cancelled'; nti mu vy'ukuri gukuraho byte imwe y'amakuru muri dosiye y'imeza iri kuri disiki ubwo nyene. Kubera ko PostgreSQL ikoresha Igenzura ry'Imirongo myinshi (MVCC), a DELETE gusa gushinga xmax umutwe w'indangamuntu y'ubucuruzi ku murongo wose kugira ngo uyishireko ikimenyetso nk'iyitaboneka ku bibazo vyo muri kazoza, usige amakuru yose y'umurongo yicaye imbere muri base/ dosiye y'ikirundo gushika autovacuum iza kugira ngo ishire ikimenyetso ku kibanza gishobora gusubira gukoreshwa.

• Niba uri kuri self-hosted canke VM PostgreSQL kandi udafise ububiko bushasha: Ubwo nyene uhuze nk' superuser kandi uzimye autovacuum kuri iyo meza (ALTER TABLE table_name SET (autovacuum_enabled = false);), canke uhagarike igikorwa ca PostgreSQL maze ufate ifoto y'urugero rwa dosiye (LVM / EBS ifoto) y' /var/lib/postgresql/data ububiko. Ivyiyumviro vyihariye vya PostgreSQL (pg_dirtyread) ishobora gusoma no gusubiza 100 kw’ijana vy’imirongo yasubiwemwo canke imbere y’uko ivugururwa ataco ihinduye mu meza y’ikirundo itagira ubusa mu minota mikeyi.

Ububiko bw'amakuru bwo gusubizaho amakuru biciye kuri moteri n'urubuga rw'igicu¶

Igice ca 2: Imfunguruzo za API zasohotse, .env amabanga, canke imfunguruzo z'ibanga zatanzwe kuri Git¶

Iyo wakoze ikintu mu mpanuka .env dosiye, id_rsa Urufunguzo rwa SSH, dosiye JSON ya konti ya serivisi ya Firebase/GCP, urufunguzo rw'ibanga rwa Stripe (sk_live_...), canke AWS . AKIA... urufunguzo rwo gushika kuri Git, ukurikize iyi nzira nyayo y'inzira zibiri: Intambwe A (Kwica urufunguzo mu gicu) imbere, na Intambwe B (Scrub ububiko bwa Git) ubwa kabiri.

Icigwa ca A: Ubwo nyene guhagarika ivyemezo vy’igicu mu masegonda 60 (Ivyo ubikore MBERE)¶

1. AWS (AKIA... Urufunguzo rwo kwinjira ID + Urufunguzo rwo kwinjira rw'ibanga)¶

Iyo bots yivye urufunguzo rwa AWS kuri GitHub, inyandiko yabo yikora itera kuri konti yawe mu masegonda 10 kugira ngo ihindure instances za GPU EC2 zifise ubunini burengeye (p4d.24xlarge / g5.48xlarge) mu turere twose 30+ tw’isi yose twa AWS (harimwo n’uturere two guhitamwo nka Jakarta, Osaka, n’Uburasirazuba bwo Hagati) kugira ngo ucukure amafaranga y’ibanga, canke ureme umukoresha wa IAM w’inyuma afise AdministratorAccess imbere yo gukuraho urufunguzo rw'umwimerere.

1. Uzimye kandi usibe urufunguzo rwo kwinjira rwasohotse ubwo nyene:
   • Gufungura AWS Console -> IAM -> Abakoresha -> [Ukoresha yashizwe mu kaga] -> Ivyemezo vy'umutekano -> Imfunguruzo zo gushikako -> Fyonda Guhagarika, hanyuma Gukuraho (canke biciye kuri CLI: aws iam update-access-key --access-key-id AKIAIOSFODNN7EXAMPLE --status Inactive --user-name MyUser).
2. Shirako ibitomoye DenyAll politike yo gukuraho ivy'igihe gito aws sts Ibimenyetso vy'ikiganiro:
   • Umutego w'agaciro: N'inyuma yo gufuta AKIA... IAM urufunguzo rwo kwinjira, nimba uwutera amaze guhamagara sts:GetSessionToken canke sts:AssumeRole mu masegonda 30 ya mbere, ivy’agateganyo vyabo . ASIA... ikimenyetso c'igihe kiguma gikora gushika ku masaha 1 gushika kuri 12 kiretse ushizeko umurongo utomoye Kwanka (nk' AWSCompromisedKeyQuarantineV2 canke politike yo Kwanka vyose ku bimenyetso vyasohowe imbere y'ikidodo c'igihe kiriho) ku mukoresha canke uruhara.
3. Igenzura rya AWS CloudTrail ku bijanye n'ugushikama kw'inyuma:
   • Gufungura AWS CloudTrail -> Amateka y'ivyabaye maze ucungurure Indangamuntu y'urufunguzo rwo gushikako canke Izina ry'ukoresha mu masaha 2 aheze.
   • Suzuma cane cane aya majambo 5 y’ugushikama: CreateUser, CreateAccessKey, AttachUserPolicy, AssumeRole, na RunInstances. Suzuma AWS Cost Explorer na EC2 Global View mu turere twose kugira ngo ushireho ingero z'ubuhendanyi, maze ufungure ubwo nyene Ikibazo c'Infashanyo ya AWS (Konti n'Ibiciro -> Konti Yahungabanijwe). AWS ihora irekura amafaranga y’ubuhendanyi iyo umenyesheje ukuvuza kw’urufunguzo maze ugahagarika instances vuba.

2. Umurongo (sk_live_...), GufunguraAI (sk-...), WoherezaUruzitiro, Twilio, n'Ibimenyetso vy'Ubugoyagoye¶

• Stripe: Genda kuri Abahinguzi -> Imfunguruzo za API -> Imfunguruzo zisanzwe / Imfunguruzo zibujijwe -> Fyonda ku tudodo dutatu (...) -> Urufunguzo rw'uruzitiro. Hitamwo "Expire now" (canke ushireho idirisha ry'ubuntu ry'isaha 1 gusa iyo uriko urakoresha urufunguzo rushasha mu bihinduka vy'ibidukikije vyawe ubu).
• OpenAI / Anthropic: Genda ku rubuga rwa API Keys maze ukande Gukuraho / Gukuraho ubwo nyene kugira ngo ibikoresho vy’ubuhinga bwa none ntibishobore gukuraho ibihumbi vy’amadolari y’amahera y’ibimenyetso vya LLM mu minota mikeyi.
• Ibimenyetso vy'Ukwinjira kwa GitHub (PATs), Ibimenyetso vy'Ubuntu Bwiza, n'Imfunguruzo zo Gukoresha: Genda kuri Imitunganyirize ya GitHub -> Imitunganyirize y'abahinguzi -> Ibimenyetso vy'Ukwinjira kwa GitHub hanyuma ukande Gukuraho / Gukuraho ku kimenyetso casohotse, hanyuma usuzume Lo**udi y'ishirahamwe ryawe (**gSettings -> Audit log) ku repo.clone, repo.access, canke gukora webhook itaremewe mu gihe c'idirisha ry'ugushirwa ahabona.
• Imirongo y'ihuriro ry'urutonde rw'amakuru (DATABASE_URL / postgres:// / mongodb+srv://) na Redis URLs: Guhindura ijambobanga ry'ukoresha urutonde rw'amakuru imbere muri moteri y'urutonde rw'amakuru (ALTER USER app_user WITH PASSWORD 'new_random_secret';), hanyuma uheze amahuzu yose ya TCP akora ubu yuguruye ari ay'ivyemezo vya kera (SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE usename = 'app_user' AND pid <> pg_backend_pid(); mu PostgreSQL), kuko guhindura ijambobanga ry'urutonde rw'amakuru muri PostgreSQL canke MySQL bibuza gusa bishasha kwinjira kandi bikasiga uruja n'uruza rw'umuterabwoba rwari rwaramaze gufungurwa ruriho gushika ruteye.

Icigwa ca B: Gukuraho ibanga ryo muri kahise ka Git (git filter-repo & Gukuraho ububiko bwa GitHub)¶

Igihe urufunguzo rwavutse rwapfuye kandi rwahinduwe mu gukora, usukure urudodo rw'ibanga mu mateka yawe ya Git kugira ngo abapima umutekano bahagarike kugabisha:

1. Subira kwandika amateka y'ububiko ukoresheje git filter-repo (canke BFG Repo-Isuku):
   • Fata indorerwamo nshasha y'ububiko bwawe maze ukureho dosiye yasohotse (nk'akarorero, .env) kuva ku mateka yose y'amashami yose n'ibimenyetso:

     # 1. Remove a specific file (.env) from all commits in history:
     git filter-repo --invert-paths --path .env
     
     # OR 2. Replace a specific secret string with ***REMOVED*** everywhere:
     echo "sk_live_YOUR_SECRET_STRING==>***REMOVED***" > replacements.txt
     git filter-repo --replace-text replacements.txt
     

2. Sunika ku nguvu amateka yasukuwe no gusukura ububiko bwa GitHub Gukuraho Gusaba / Gukora:
   • Sunika amateka yasukuwe asubire aho yakomotse (git push origin --force --all na git push origin --force --tags).
   • Intambwe ihambaye ya GitHub cached-commit: Naho umaze gusunika ku nguvu amateka yasukuwe, umuntu wese azi SHA URL ya kera y'inyuguti 40 (github.com/org/repo/commit/abc1234...) irashobora kubona ivyagezwe vya kera bitasukuwe ku rubuga rwa GitHub gushika GitHub ikoresheje gukusanya imyanda ku ruhande rwa server. Baza GitHub Support (canke ukoreshe urubuga rw'ububiko bw'impanuro / ubufasha) n'ivyo SHA vya kera nyavyo maze ubasabe kwirukana ivyo vyavuye mu bubiko bw'urubuga maze ukoreshe git gc ku bubiko bwa kure.

Igice ca 3: Gukuraho impanuka git push --force, git reset --hard, canke ishami ryafuswe¶

Iyo wirutse mu mpanuka git push --force kandi uzimanganya ivyo abo mufatanije ikipe bakoze ku main, canke yirutse git reset --hard HEAD-5 kandi yatakaje umusi wose w'ibikorwa vyo mu karere, ntimuhagarike umutima, kuko Git hafi ntiyigera ifuta ibintu vyakozwe ubwo nyene.

Kubera iki ama commits yawe "yazimiye" aguma yicaye mu mutekano imbere .git/logs (Ivyasubiwemwo)¶

Igihe cose uremye umugambi, uhinduye amashami, ugasubiramwo, canke ugasubiramwo, Git yandika ivya kera HEAD n’isonga ry’ishami SHA-1/SHA-256 ibimenyetso biri mu gitabu c’ivya kera citwa Igitabo c’Ivyabona (git reflog), kandi igumya ibintu vy'impfuvyi vy'ukuri kuri disiki nibura imisi 14 gushika kuri 90 imbere y'uko ugukusanya imyanda kubikuraho.

Intambwe ku yindi Git gukira vyihutirwa¶

Ivyabaye 1: Warirutse git push --force kandi yanditse hejuru origin/main¶

1. Rondera SHA nyayo iyo origin/main yerekanwa iburyo imbere y'inguvu-gusunika kwawe:
   • Uburyo A (Suzuma reflog yawe yo gukurikirana kure): Gukoresha iri tegeko mu terminal aho wasunikiye:

     git reflog show origin/main
     

     Uzobona igisohoka nka:

     a1b2c3d (HEAD -> main, origin/main) refs/remotes/origin/main@{0}: update by push
     9f8e7d6 refs/remotes/origin/main@{1}: fetch: fast-forward
     

     Aha, 9f8e7d6 (origin/main@{1}) ni co kintu nyaco origin/main yari ku ruhande rw’imbere y’uko inguvu zawe ziyisunika.
   • Uburyo B (Baza uwo mufatanije wese atarirutse git pull yet): Uwo mukorana wese yazanye main mbere y'uyu musi aracari n'amateka y'ibikorwa vyiwe vyicaye mu karere kabo origin/main ishami.
   • Uburyo C (Baza GitHub REST API Ivyabaye log): Niba reflog yawe yo mu karere idafise igihugu ca kure, ubaza GitHub push ivyabaye API kugira ngo ubone "before" SHA hash y'inguvu-gusunika ikintu:

     gh api /repos/OWNER/REPO/events --jq '.[] | select(.type=="PushEvent") | {actor: .actor.login, before: .payload.before, head: .payload.head, time: .created_at}'
     

2. Gusubiramwo origin/main ku vya kera gukora SHA ata nkomanzi:
   • Genzura ko ivyo wiyemeje bisa n'ibibereye (git log -n 5 9f8e7d6), hanyuma usunikire iyo SHA nyayo usubire ku kure . main ishami rikoresheje --force-with-lease:

     git push --force-with-lease origin 9f8e7d6:refs/heads/main
     

Ivyabaye 2: Warirutse git reset --hard canke yasivye ishami ryo mu karere imbere yo gusunika¶

1. Kwiruka git reflog mu bubiko bwawe:

   git reflog --date=iso
   

2. Tora SHA imbere y' reset: kwinjira (nk'akarorero, HEAD@{2}: commit: feat: add payment webhook), maze ureme ishami rishasha ry’ugutabara ryerekeza kuri iyo SHA:

   git switch -c rescue-branch HEAD@{2}
   

3. None iyo utakaza dosiye itashizweho iyo git added (staged) hanyuma igakurwaho n’ git reset --hard?
   • Igihe wiruka git add myfile.ts, Git ifatanya kandi ikandika ibirimwo vyose muri iyo dosiye muri .git/objects/ nk'igikoko kizunguruka, naho woba utigeze wiruka git commit. Kugarura mu gutanga urutonde rw'ibice vyose bizunguruka vyatondejwe hakurikijwe igihe co guhindura:

     git fsck --lost-found
     # Inspect the recovered blobs written into .git/lost-found/other/
     

Igice ca 4: Impanuka terraform destroy canke ububiko bw'igicu bwasubiwemwo (S3, EBS, VMs)¶

Iyo umurongo w'ibikorwa remezo nk'itegeko (IaC) canke itegeko rya CLI ry'amaboko ritangura gusenya ibikoresho vy'igicu:

1. Hagarika kwicwa ata nkomanzi:
   • Niba uriko urakoresha mu karere kawe, kanda **Ctrl + C RImwe ** rero Terraform iraheza guhamagara API yayo ubu maze ikandika ivyavuguruwe .tfstate dosiye isukuye. Niba uriko urakoresha imbere mu bikorwa vya GitHub, GitLab CI, canke Terraform Cloud, kanda Gukuraho Gukoresha maze ubwo nyene uzimye uruhara rwa AWS/GCP IAM rw'umunyarugamba wa CI mu gicu kugira ngo API yo mu ndege isubire kunanirwa na AccessDenied.
2. Gusubiza inyuma ivya none .tfstate dosiye ubwo nyene:
   • Gukuraho kopi y'aho hantu y'idosiye yawe ya kure (terraform state pull > emergency-backup.tfstate) maze usuzume verisiyo y'indobo ya S3 ku ndobo yawe ya leta ya Terraform kugira ngo ufate verisiyo ya leta ya kera kuva mu minota 10 iheze.
3. Suzuma umutanga igicu "Soft-Delete" n'insenga z'umutekano:
   • Amazon S3 (Ifise ubuhinga bwo guhindura amadosiye mu ndobo): Gukuraho amadosiye mu ndobo ya S3 yahinduwe (aws s3 rm --recursive) nti ikuraho ikintu na kimwe; gusa yinjiza zero-byte Gukuraho Ikimenyetso hejuru y'ikintu cose. Kuraho ibimenyetso vyo gukuraho (biciye mu bikorwa vya S3 canke Python y'imirongo 5 boto3 script) kugira ngo usubiremwo 100 kw’ijana vy’amadosiye yawe ubwo nyene.
   • Ibitabo vya AWS EBS n'Ibishushanyo vya RDS (Igikoresho co Gusubiramwo): Suzuma Igikoresho co Gusubiramwo AWS muri konsole nimba vyatunganijwe, hanyuma usuzume Ibikoresho vyo Gusubiramwo vya AWS.
   • Igicu ca Google (GCP) & Microsoft Azure: Imigambi ya GCP yasubiwemwo n'Imigwi y'Ibikoresho vya Azure / Amakonti y'Ububiko afise Gukuraho Biyoroshe ashoboye kuguma ashobora gusubirwamwo imisi 7 gushika kuri 30 biciye mu gukanda rimwe gusa mu gikoresho.

Uko wosubirana Terraform yamenetse .tfstate dosiye ya Leta canke gusubira kwinjiza mu gihugu ibikorwa remezo vy'impfuvyi¶

Igihe a terraform apply canke impanuka terraform destroy ni ugucika hagati mu rugendo, canke iyo umuntu asubije mu mpanuka dosiye y’intara iri kure muri S3/GCS, ibikorwa remezo vyawe vy’igicu vyinjira mu ntara y’ubwonko buteye akaga aho ibikoresho bimwebimwe bikiriho muri AWS/GCP mu gihe ibindi bibura muri .tfstate:

1. Gusubizaho remote yanditswe canke yasubiwemwo .tfstate dosiye kuva kuri S3 / GCS Guhindura:
   • Ntukigere uhindura JSON n'amaboko .tfstate dosiye n'ukuboko mu muhinduzi w'inyandiko kiretse iyo ubundi buryo bwose bwananiwe, kubera ko uburyo bumwe budahuye "serial" canke "lineage" UUID urudodo ruzobuza Terraform gusubira gukoresha.
   • Ugurure indobo yawe ya S3 (canke indobo y'ububiko bw'igicu ca Google / Ikibanza c'akazi c'igicu ca Terraform Ibihugu) aho umurongo w'inyuma wo kure uba, kanda Erekana verisiyo guhindura hejuru ya console ya S3, urondere terraform.tfstate verisiyo yashizweko ikidodo c'igihe imbere y'uko igenda nabi, uyikureho, hanyuma uyisubize inyuma ata nkomanzi nk'uko biri ubu:

     # Push a recovered historical state file back to the remote backend safely
     terraform state push recovered-pre-incident.tfstate
     

2. Guca urufunguzo rwa Leta rwafashe (Error acquiring the state lock):
   • Iyo ukuye inda mbi . terraform destroy na Ctrl + C canke kwica umunyarugamba wa CI amanitswe, imeza ya DynamoDB iri kure canke dosiye y'ugufunga ya GCS iguma ipfungiwe n'Indangamuntu y'igikorwa yapfuye.
   • Mbere suzuma mu CI dashboard yawe n'abo mufatanije ko zero izindi nzira za Terraform mu vy'ukuri zikiriko zirakora ku kibanza c'akazi, hanyuma urekure urufunguzo rw'impfuvyi ukoresheje Lock ID UUID icapwe mu butumwa bw'ikosa:

     terraform force-unlock <LOCK_ID_UUID>
     

3. Gusubira kwubaka ibidukikije vyasambutse bimwe bimwe ata gusubira kurema amakuru yarokotse (terraform import):
   • Nimba a terraform destroy yazimanganye ibikoresho vyawe bitagira igihugu (aws_lb) na ECS/EKS umukozi nodes imbere y'uko ubihagarika, ivyo biroroshe gukosora: gusa run terraform apply na Terraform izosubira kurema urubuga rwose rudafise igihugu no guharura ibice kuva mu ntango mu minota 5 mu gihe izosiga urutonde rwawe rwa RDS rwarokotse rudakozweko.
   • Niba dosiye yawe ya leta yatakaje urutonde rw'urutonde rw'amakuru rwa leta canke indobo ya S3, koresha import {} ububiko bw'imiterere (Igishushanyo 1.5+) canke terraform import aws_db_instance.prod_db my-rds-instance-id kugira ngo ubohe umutungo w'igicu uriho usubire muri dosiye yawe ya leta utayisenyuye, hanyuma ukoreshe terraform plan kwemeza "0 to destroy" imbere yo gusaba.

Igice ca 5: Uko womenyesha umuyobozi wawe ko umuriro uhagaze maze ukarongora igikorwa co gupima urupfu ataco umugayo¶

Iyo uteje ingorane mu gihe ukora kure, izina ryawe ry’umwuga risobanurwa nti n’ikosa ubwaryo, ahubwo risobanurwa n’umuvuduko wawe wo guserukira abantu mu mucyo, guhanahana amakuru utuje, no gutorera umuti ibibazo mu buryo butunganye.

Amategeko 3 y'Inzahabu y'Imenyekanisha ry'Ivyabaye¶

1. Vuga ibintu, ingaruka, na ETA ata kwihisha, gucerezwa, canke gutera iteka: Kumenyesha ikosa mu minota 2 y’ukuvumbura.
2. Gutandukanya "Mitigation" (Guhagarika amaraso) na "Root-Cause Fix": Mu gihe c'ivyihutirwa, intumbero yawe yonyene ni ugusubiza inyuma igikorwa (gusubiza inyuma gukoresha nabi, kunanirwa gushika ku rutonde rw'amakuru rwa PITR, canke guhindura ibendera ry'ibintu). Bika kode yimbitse y'ugusubiramwo ejo.
3. Shira amakuru y’iminota 15 y’ugutera kw’umutima mu muhora w’ivyabaye kugira abarongozi n’abafasha abakiriya bamenye ko hari uwuriko arabigenzura.

Kopa-gushiramwo Slack / Inyandiko y'Itangazo ry'Ivyabaye vy'Imigwi¶

[IBIBAZO VYA SEV-1] Umusaruro [Ibikorwa / Izina ry'Urutonde rw'amakuru] Ingaruka - Ivyabaye: Ku isaha ya 10.00. [HH:MM UTC], mu gihe bariko barakora [brief technical action, e.g., a data migration script], ikintu kitari gitegekanijwe [DELETE / config change / force-push] vyakoreshejwe mu guhingura [database / service]. - Ingaruka z'abaguzi ubu: [e.g., Checkout API is returning 500 errors / User profile updates are paused]. - Ivyo gutorera umuti ububabare biriko birakorwa: 1. Wandike uruja n'uruza kuri [table/service] yarahagarikiwe kugira ngo amakuru azigame ubutungane. 2. Gusubira mu gihe (PITR) clone ku [HH:MM:SS UTC] (amasegonda 10 imbere y’uko ico kintu kiba) ubu iriko irazunguruka mu 2019. [AWS RDS / GCP]. - Umukuru w'Ivyabaye: @[Your Name] (afashwa na @[DBA/On-Call SRE]). - Igikurikira: Mu minota 15 ([HH:MM UTC]) canke igihe nyene iyo clone yo gukira iza kuri Internet.

Urutonde rw'ibintu vyo kwirinda inyuma y'ivyabaye (Ku bw'Inyuma y'Urupfu rwawe Ata Caha)¶

Ikosa ryose ry’umuntu mu gukora ni systems design gap (niba itegeko rimwe ry’iherezo ry’ugukopa rishobora gutuma ibikorwa bivaho ata n’imwe irindi, iyo system yari irindiriye kunanirwa). Tegura kandi ushire mu ngiro izo nzira z’ubuhinga zikomeye mu nyandiko yawe y’inyuma y’urupfu umusi ukurikira:

• Ibara-kode n'intango y'ibikorwa (kube-ps1, psql .psqlrc): Gutunganya igikoko cawe n'umukiriya wa SQL kugira ngo ubufatanye bwose bube kuri PRODUCTION yerekana umutuku ukeye [PROD-DB] gusaba no gutangura buri gihe c'imigenderanire ya PostgreSQL imbere muri BEGIN; ububiko bw'ubucuruzi ku mburabuzi (canke gushinga SET default_transaction_read_only = on;).
• Gushoboza safe-updates (--i-am-a-dummy) mu bakiriya ba MySQL/MariaDB: Wongereko safe-updates ku [mysql] muri -/.my.cnf rero MySQL iranka gukora ikintu cose UPDATE canke DELETE imvugo idafise WHERE urufunguzo rw'ingenzi canke a LIMIT.
• Hingira DeletionProtection = true ku makuru yose y'ibikorwa vya RDS, indobo za S3, n'ibikoresho vya Terraform (lifecycle { prevent_destroy = true }).
• Gushitsa Amategeko y'Ukurinda Ishami rya Git ku main / production: Guhagarika Allow force pushes na Allow deletions ku mashami akinzwe muri GitHub/GitLab rero n'abarongozi b'ububiko ntibashobora git push --force ku main.
• Shiraho imbere y'uko ukora ibanga ry'ugupima (gitleaks canke trufflehog) na GitHub Push Uburinzi: Guhagarika ibikorwa birimwo AKIA, sk_live_, canke .env ibigereranyo vyo mu karere imbere git commit mbere araroranirwa.

Ibibazo bikunda kubazwa¶

Noba nzokwirukanwa kubera gutera mu mpanuka imeza y’urutonde rw’amakuru y’ibikorwa canke gutuma urubuga ruhagarara?¶

Mu mashirahamwe y’abahinga afise amagara meza akora umuco wo gukingira ataco ukora (uwusanzwe mu mirwi yose y’ubuhinga bwa none, DevOps, na SRE), abahinga ntibirukanwa kubera amakosa y’abantu b’ukuri mu guhingura, igihe cose bamenyesheje ivyo bintu ubwo nyene kandi bagafasha gusubizaho igikorwa. Abarongozi b’ubuhinga baratahura ko iyo umuhinga muto canke mukuru afise uruhusha rudasanzwe be n’ukubura ibikingi vyo gutera imeza y’ubuhinga afise itegeko rimwe ritasubiwemwo, imvo nyamukuru ni akaga mu kugenzura ukuntu umuntu ashobora gushika no gukoresha ibikoresho, atari ububisha. Mu vy’ukuri, uwo muhinga aherutse kwibonera iyo nzira y’ugucika intege ubu ni we muntu wenyene afise umutekano kuruta abandi bose mu mugwi ku bijanye n’iyo nzira. Inyifato yonyene iguma ijana mu guhagarika ni ukugerageza gupfuka ikintu cabaye co gutakaza amakuru, gukuraho ibitabo vy’igenzura, canke kubesha mu gihe c’itohoza ry’ivyo bintu.

Ni kuki ntashobora guca mpindura uruja n’uruza rwanje rw’ibikorwa vy’ubuzima ku rutonde rw’amakuru rwa Point-In-Time Recovery (PITR) rwagarutse?¶

Niba gusa wasivye imirongo mu mbonerahamwe imwe (nk'akarorero, i `user_preferences` table) ku isaha ya 14:00 UTC, kandi bifata iminota 20 kugira ngo uhindure igishushanyo ca PITR kigaruwe ku isaha ya 13:59:50 UTC, urutonde rwawe rw’ibintu bishobora kuba rwabandanije gukora ibihumbi vy’amategeko mashasha y’abaguzi yemewe n’amahera yishuye mu `orders` imeza hagati ya 14h00 na 14h20 UTC. Niwahindura mu buryo bw’impumyi iherezo ryawe ryose ry’ibikorwa vya DNS ku 13:59:50 PITR clone, uzokuraho iyo minota 20 y’amategeko mashasha y’abaguzi. Ahubwo, ushire urutonde rw’amakuru nyamukuru kuri interineti (n’ivyanditswe bihagaze gusa ku kintu cavunitse) maze urungike hanze imirongo gusa yabuze kuva ku 13:59:50 PITR clone usubire mu meza y’ibikorwa vy’ubuzima.

Iyo nkoze ububiko bwa GitHub bwa bose bwite iminota ibiri inyuma yo gusunika mu mpanuka .env dosiye, mbega ndacari nkeneye guhindura imfunguruzo zose za API?¶

Egome, 100 kw’ijana vy’izo mfunguruzo biracari bitegerezwa gukurwaho no guhindurwa ubwo nyene. Ivyo vyose bishikirizwa mu bubiko bwa GitHub bwa bose bimenyeshwa mu gihe nyaco ku rubuga rwa GitHub Events API ("the Firehose") kandi bikaba bishirwa mu masegonda makeyi n'ibikorwa vy'ububiko bw'umutekano vyemewe (nka GHArchive) n'ubuhinga bubi bwo kwimbura ivyemezo7 bukoresha urubuga rwa 24. Guhindura ukuntu ububiko buboneka kuva kuri bose gushika ku bwihariye haciye iminota ibiri ntibikuraho kopi y’ibikoresho vy’ubuhinga bwa none vyari vyaramaze gukurwako.

Bigenda gute git push --force-with-lease gukingira umuriro ugereranyije n’ git push --force?¶

Iyo ukoresha ivyagezwe `git push --force`, Git yandika ataco isaba ku ishami ryo kure n'ishami ryawe ryo mu karere, igasenyura mu gacerere ibikorwa vyose umugenzi wiwe yasunikiye kuri iryo shami muri ico gihe. Iyo ukoresheje `git push --force-with-lease` (kandi mbere n’ivyiza kuruta, `--force-if-includes`), Git ibanza gusuzuma umukozi wa kure kugira ngo isuzume ko umutwe w'ishami ryo kure ugihuye n'ivyo SHA imashini yawe yo mu karere yaheruka gutora. Niba umukozi mukorana canke umuyoboro wa CI yasunikiye isezerano rishasha kw'ishami ryo kure kuva igihe waheruka gutora, `--force-with-lease` aborts ata nkomanzi n'imburi "stale info" aho gukuraho ibikorwa vyabo.

Amasoko yasuzumwe¶

• Ikigo c’Igihugu c’Ivy’Imirongo Ngenderwako n’Ikoranabuhanga (NIST, Leta Zunze Ubumwe za Amerika): NIST SP 800-61 Rev. 2 / Rev. 3 Igitabu co gutorera umuti ingorane zijanye n’umutekano wa mudasobwa n’ugukingira ivy’ugusenyuka kw’ivyemezo.
• Ibikorwa vy'urubuga rwa Amazon (AWS), Urubuga rw'Igicu rwa Google (GCP), n'Inyandiko za Microsoft Azure (Leta Zunze Ubumwe za Amerika): Ubwubatsi bw'Ivyiyumviro vy'Ivyiyumviro vy'Ivyiyumviro (PITR), AWS IAM Amategeko y'Ikigo c'Ikigobe (AWSCompromisedKeyQuarantineV2), na S3 Guhindura Verisiyo Gukuraho-Ikimenyetso Gusubirana.
• Ishirahamwe ry'Iterambere ry'Isi yose rya PostgreSQL n'Inyandiko za MySQL / Oracle (Impuzamakungu): Ivyanditswe imbere vya PostgreSQL (WAL), Ukuboneka kwa MVCC Tuple, n'Ivyanditswe vy'Ivyandikano vy'Ivyandikano vy'Ivyandikano vy'Ivyandikano vy'Ivyandikano vy'Ivyandikano vya MySQL.
• Git SCM Inyandiko Yemewe na GitHub Umutekano Lab (Impuzamakungu): Git Reflog (git-reflog), Gusubiza Ibintu Bimanitse (git-fsck), Gusubira kwandika amateka (git-filter-repo), n’Ibanga ry’Ugukingira Gusunika.
• Ishirahamwe rya USENIX / SREcon na Google Site Engineering (SRE) Ibitabo (Impuzamakungu): Gucungera Ivyabaye, Gutoranya Ivyihutirwa, n’Umuco w’Inyuma y’Urupfu Ata Mugayo.

Inyobora zifitaniye isano¶

• Ivyihutirwa vy'akazi kure umutekano n'ubuzima bwite bw'amakuru
• Ububiko bwa mudasobwa 3-2-1 ingamba no gusubizaho ivyago
• Gukuraho dosiye ku mpanuka no gusubizaho umuvuduko
• Ukwimuka kw'ibibazo vy'ukwakira urubuga no kugarura ububiko
• Ubuhendanyi n'amakonti yashizweho
• Inomero z'ivyihutirwa