Give your AI-built app a small recovery drill

“We can roll it back” sounds great until the first time somebody asks who has the command, which image was good, and whether the database still understands that old version. I would rather discover those questions during a small rehearsal. AI-generated code deserves the same operational homework as everything else we ship.

This exercise uses an isolated staging application and synthetic records. It is not an invitation to break a live service. The Google SRE Workbook chapter on incident response provides useful background on coordination. Here is my practical starter drill for a small team.

Choose one failure you can reverse

Pick a read-only dependency timeout. Replace the staging dependency with a stub that delays its response beyond the client's configured deadline. Write down the expected user experience: an understandable error, a bounded wait, and a working retry button. Define a stop condition before injecting the fault, such as unexpected writes or traffic reaching production.

Assign an operator and an observer. The operator follows the runbook; the observer records events and checks the stop condition. For a solo project, record a sanitized timeline and repeat the exercise later using only the written instructions. Familiarity with the application can hide missing steps.

Start the clock at the symptom

Record when the fault starts, when monitoring detects it, when someone identifies the affected dependency, and when the user-facing check recovers. These are different moments. A green process-health endpoint is not recovery if the user's request still hangs.

Capture the release identifier, relevant trace IDs, and the configuration switch used for the drill. Do not paste complete prompts, customer payloads, or secrets into the incident note. The question is whether the evidence helps distinguish a model error, a tool timeout, and a broken service boundary.

Recover, then verify from the outside

Remove the injected delay or restore the known-good staging configuration. Use a fresh client request to verify the actual workflow. Inspect queued work and background retries; the visible page can recover while a queue keeps replaying failed jobs. If rollback is your chosen recovery, verify the image and schema compatibility beforehand rather than guessing during the drill.

Your recovery note should include the exact environment, command or UI action, expected response, validation request, and stop condition. “Restart it” is not a runbook. “Restart staging worker X, expect ready within Y seconds, then submit this synthetic request and inspect its recorded result” is something another person can execute.

Keep the lesson small enough to use

Finish with three observations: what detected the failure, what delayed diagnosis, and which instruction was missing. Give each correction an owner and a verification step. Rerun the affected part after the fix; writing a ticket alone does not prove recovery improved.

Avoid turning one rehearsal into a reliability score for the whole app. You tested one failure mode under particular conditions. Next time, choose a different boundary, such as an unavailable model endpoint or a full queue. In Japanese, 復旧(ふっきゅう)を確認(かくにん)しました means “I verified recovery.” Save that sentence for the point when the user's workflow actually works again.