AI Incident Response: NIST SP 800-61 Framework, EU AI Act Serious Incident Reporting, and AI-Specific Playbooks for Enterprise Systems
AI Incident Response Reference
AI-Specific Incident Categories: Beyond Traditional IR Playbooks
Traditional incident response frameworks (NIST SP 800-61, SANS IR methodology) categorize incidents as: denial of service, malicious code, unauthorized access, inappropriate usage, and scans/probes. AI systems introduce new incident categories that existing IR playbooks don't address: prompt injection attacks, model failures causing harmful outputs, AI-assisted data exfiltration, adversarial attacks on AI models, AI supply chain compromises, and regulatory non-compliance from AI decisions.
AI incident taxonomy (NAIR — National AI Incident Repository categories, based on AIID):
Category 1: AI System Failures — The AI system produces outputs outside its acceptable performance range. Subcategories: hallucination leading to harmful advice (medical, legal, financial), model drift causing degraded accuracy over time, safety guideline bypass, and AI system unavailability. Key metrics: hallucination rate spike, accuracy degradation beyond SLA threshold, safety classifier failure rate.
Category 2: AI Security Incidents — Attacks targeting or exploiting AI systems. Subcategories: prompt injection (direct and indirect), model inversion/data extraction, adversarial evasion, AI supply chain compromise, and unauthorized model access. These incidents may require evidence preservation for forensic analysis.
Category 3: AI Compliance Violations — AI system actions that violate regulatory requirements. Subcategories: GDPR Article 22 violation (automated decision without required human review), CCPA consumer rights failure, FINRA suitability violation from AI recommendation, discriminatory AI output triggering civil rights exposure. These incidents may require regulatory self-disclosure.
Category 4: AI Data Incidents — Data breaches or exposures through AI systems. Subcategories: RAG retrieval exposing unauthorized documents, AI outputs containing PII from training data memorization, cross-tenant data contamination, and AI-assisted insider threat (using AI to exfiltrate data).
Prompt Injection Attack (P0)
Confirmed prompt injection causing unauthorized agent action. Response: immediately suspend affected agent, preserve conversation logs as evidence, assess scope of unauthorized actions, notify security team, initiate IR plan. Potential regulatory notification if PII exposed.
Model Drift / Accuracy Degradation (P1)
AI performance metrics drop below SLA threshold (e.g., accuracy drops from 85% to 65%). Response: roll back to previous model version, notify affected customers, conduct root cause analysis, implement additional monitoring. Regulatory notification if compliance-sensitive decisions affected.
Cross-Tenant Data Exposure (P0)
AI RAG retrieval returns documents belonging to another tenant. Response: immediately suspend retrieval for all tenants, audit all recent retrievals for cross-contamination, notify affected tenants, assess GDPR/CCPA breach notification obligations, conduct forensic analysis of access logs.
NIST SP 800-61 Four Phases Applied to AI Incidents
Phase 1: Preparation. NIST 800-61 preparation includes establishing IR capability before incidents occur. For AI systems: create AI-specific IR playbooks for each incident category; define detection thresholds and alert rules for AI behavior anomalies; establish communication trees for AI incidents (security team, legal/compliance, customer success, C-suite); pre-negotiate AI incident investigation retainer with a forensics firm that has AI expertise; and conduct tabletop exercises for AI incident scenarios quarterly.
Phase 2: Detection and Analysis. AI incidents require AI-specific detection methods. Automated detection signals for AI incidents: sudden increase in AI error rate or hallucination rate (model failure); prompt pattern matching alerts (prompt injection); unusual agent tool call patterns (compromised agent); cross-tenant access pattern anomalies (data isolation failure); model performance drift beyond control limits (model drift). Manual detection: user complaints about AI behavior, compliance team flagging AI decisions, customer reports of seeing other customers' data.
Phase 3: Containment, Eradication, and Recovery. AI incident containment may require: immediately disabling the affected AI agent or workflow; rolling back to a previous, known-good model version; suspending RAG retrieval for affected knowledge bases; revoking compromised API keys or agent credentials; and applying emergency prompt guardrails. Eradication investigates root cause: was the injection pattern preventable? Did RAG permissions fail? Was the model version the issue? Recovery involves implementing fixes, testing in staging, controlled re-enablement with enhanced monitoring, and post-incident review within 5 business days.
Phase 4: Post-Incident Activity. AI post-incident requirements: complete lessons learned document within 5 business days; update AI threat model with new attack patterns; update monitoring rules to detect similar future incidents; if regulatory notification was required, document the notification and regulator response; update AI IR playbooks based on gaps identified; and if the incident was material, brief the board/audit committee.
Regulatory Notification Requirements for AI Incidents
GDPR (Article 33/34): Any personal data breach — including exposure through AI systems — must be reported to the supervisory data protection authority within 72 hours of becoming aware (Article 33). If the breach is likely to result in high risk to individuals, notification to affected individuals is also required without undue delay (Article 34). For AI incidents involving RAG retrieval of unauthorized personal data, cross-tenant contamination, or AI output containing training data PII, GDPR breach notification must be assessed immediately.
EU AI Act (Article 62): For high-risk AI systems, providers must report serious incidents to national market surveillance authorities within 15 working days of becoming aware. A "serious incident" under Article 62 is one that results in (or could reasonably lead to): death or serious injury, disruption to critical infrastructure, infringement of fundamental rights, or serious property damage. The market surveillance authority can request additional information and may conduct its own investigation.
SEC Cybersecurity Rule (December 2023): Public companies must disclose material cybersecurity incidents on Form 8-K within 4 business days of determining materiality. AI incidents — including a material breach through an AI system — fall under this disclosure requirement. The SEC specifically noted that AI system compromises that result in unauthorized access to material information are covered.
Financial services sector (FINRA, OCC, FDIC): Financial institutions must notify their primary regulator "as soon as possible" (with a 36-hour deadline for banking organizations under the November 2021 Computer-Security Incident Notification Rule) for significant AI incidents affecting customer data or operational continuity.
AI Incident Response Readiness Checklist
- Create AI-specific IR playbooksDevelop playbooks for each AI incident category: prompt injection, model failure, cross-tenant exposure, compliance violation, AI data breach
- Define AI incident detection thresholdsConfigure automated alerts: hallucination rate spike, error rate increase, anomalous tool call patterns, cross-tenant access patterns, model performance drift
- Establish AI incident communication treeDefine escalation path for AI incidents: security team (immediate), legal/compliance (within 1 hour), C-suite (P0 incidents), customer notification procedures
- Map regulatory notification requirementsDocument notification deadlines: GDPR (72 hours), EU AI Act Article 62 (15 working days), SEC 8-K (4 business days), financial regulator (36 hours for banks)
- Preserve AI incident evidenceImplement log retention for prompt/completion logs, agent action logs, and access logs; confirm forensic readiness with immutable log storage
- Conduct AI tabletop exercisesRun quarterly tabletop exercises for AI incident scenarios; include prompt injection, cross-tenant exposure, and model failure scenarios
- Document model rollback proceduresMaintain versioned model deployment history; test rollback procedure to previous model version; target 15-minute rollback capability
- Pre-engage AI forensics retainerEngage incident response firm with AI forensics expertise on retainer; define engagement triggers and response time commitments
- EU AI Act Article 62 proceduresFor high-risk AI systems: implement serious incident detection procedure; assign regulatory notification owner; document 15-working-day reporting process
- Post-incident review processRequire written lessons learned document within 5 business days of incident close; track remediation actions to completion; report to audit committee quarterly
Frequently Asked Questions
What qualifies as a 'serious incident' under EU AI Act Article 62?
Article 62 defines a serious incident as one that directly or indirectly results in: death or serious injury to a person; disruption of the management and operation of critical infrastructure; infringement of obligations under Union law intended to protect fundamental rights; serious damage to property or the environment. In practice, an AI system making a medical recommendation that leads to patient harm, or an AI system in financial services making incorrect decisions affecting clients materially, would likely qualify. The EU AI Act does not set a financial harm threshold for 'serious damage to property.'
Does GDPR breach notification apply to AI incidents?
Yes, whenever an AI incident involves a breach of personal data security — including unauthorized disclosure (RAG retrieval exposing another user's data), unauthorized access (prompt injection enabling access to personal information), or accidental loss of personal data. The 72-hour notification clock runs from when the organization 'becomes aware' of the breach, which for AI incidents means when the security or compliance team confirms the incident, not when it is reported by a user.
How should we classify AI incident severity (P0, P1, P2)?
AI incident severity should be classified based on: scope (number of users/tenants affected), data sensitivity (PII, PHI, financial data involved), regulatory impact (does the incident trigger mandatory reporting?), operational impact (is the AI system down?), and reputational risk. A recommended taxonomy: P0 — confirmed data breach, prompt injection causing unauthorized action, multi-tenant contamination, EU AI Act Article 62 serious incident; P1 — model failure causing compliance-sensitive decisions, accuracy below SLA threshold, single-user unauthorized access; P2 — degraded AI performance, single-user complaint, recoverable model error.
What evidence should be preserved after an AI incident?
AI incident evidence includes: all conversation logs (prompts and completions) from the affected time window; agent action logs showing tool calls, parameters, and results; authentication and access logs; model version and configuration at time of incident; RAG query logs and retrieval results; monitoring alert triggers and timestamps; and any attacker-controlled inputs (injected prompts, malicious documents). Evidence should be immediately exported to immutable storage to prevent modification. Chain of custody must be maintained if law enforcement involvement is anticipated.
How does Claire support AI incident response for customers?
Claire provides: real-time alerting for anomalous AI behavior (unusual tool calls, injection pattern detection, cross-tenant access anomalies); immutable audit logs accessible via API for IR investigation; 24/7 security incident response SLA with 15-minute acknowledgment for P0 incidents; dedicated security incident contact channel separate from general support; post-incident review calls with Claire's security team; and IR playbook templates for Claire-specific incident scenarios. Our SOC 2 Type II Incident Response procedures are available for customer review.
How Claire Addresses AI Incident Response
Claire's incident response architecture provides real-time AI behavior alerting, immutable audit logs for forensic investigation, 24/7 P0 incident response, and compliance notification support for GDPR 72-hour and EU AI Act 15-day reporting requirements. Request our IR procedures documentation and playbook templates as part of a security briefing.