AI incident response: a practical playbook for LLMs

AI incidents are not limited to compromised infrastructure. A system may disclose data through prompt injection, execute an unauthorised tool action, retrieve a poisoned RAG document, generate harmful content or change behaviour after a model update.

The established incident-response lifecycle still applies, but teams need new evidence, owners and ways to reconstruct model-assisted decisions.

Define an AI incident

Set thresholds for confidentiality, integrity, availability, user safety and compliance. One hallucination may be a quality defect, while repeated output that can cause harm should trigger the response process.

Useful classes include:

• data or system-prompt disclosure;
• action without valid authorisation;
• dataset, index or agent-memory poisoning;
• guardrail bypass and model abuse;
• supplier, API key or plugin compromise;
• dangerous quality or behaviour drift.

Each class needs severity criteria, an owner and escalation path.

Evidence for investigation

Capture request identifiers, model version, prompt-template version, guardrail versions, RAG sources, tool calls, policy decisions and filter outcomes. Apply pseudonymisation, access control and retention; observability must not create an unrestricted copy of user data.

For an external model, retain the region, endpoint version and provider response identifier. Without these, later reconstruction may be impossible.

Containment

Prepare switches that disable one tool, RAG source, memory store, automatic execution or model. Reducing autonomy often preserves service more safely than shutting everything down.

Rotate compromised keys, quarantine documents, freeze model versions and restrict service-account roles. Preserve evidence before rebuilding an index or deleting memory.

Root-cause analysis

Separate model failure from architecture failure. Prompt injection creates high impact when the system lets untrusted content invoke a privileged tool without independent authorisation.

Analyse the complete path: data source, retrieval, prompt, model, parser, tool policy and target-system effect. Use an AI application threat model to find similar paths.

Recovery and regression

Verify a fix against the captured case and variants using another language, encoding, file type and tool sequence. Add the scenario to the evaluation suite. Monitor refusal rates and false positives because a broad block may destroy the product’s utility.

Afterwards, update data classification, permissions, supplier runbooks and release criteria. Connect the playbook to organisation-wide breach response, legal communication and privacy processes.

Minimal playbook card

1. Incident class and impact level.
2. Evidence sources and preservation requirements.
3. Technical, business, legal and supplier owners.
4. Safe switches for reducing autonomy.
5. Data-protection and secret-rotation procedure.
6. Reproduction, remediation and regression tests.
7. Criteria for returning to normal operation.

AI incident readiness is not the retention of every prompt. It is the ability to limit impact, preserve the right evidence and prove that the same path has been closed.

---

Artefacts needed for reconstruction

Retain the model version, system prompt, tool configuration, retrieved-document IDs, policy decisions, output and actions taken in external systems. Records need a common clock and request ID. Log enough for security, while redacting secrets and enforcing retention.

Containment depends on the layer: disable a tool, revoke the agent token, freeze a model version, disconnect a RAG source or route traffic to a safe mode. Do not immediately delete the index or conversations if they are evidence. Reproduce the case on a copy, add it to evaluation and verify that the control still permits legitimate work.

Sources: NIST GenAI Profile, NIST SP 800-61 Rev. 3, OWASP LLM Top 10.