AI Red Teaming & Adversarial Testing: NIST AI RMF Govern 1.7, MITRE ATLAS, EU AI Act Article 9, and White House EO Requirements

NIST AI RMF GOVERN 1.7: Red-Teaming as a Governance Requirement

The NIST AI Risk Management Framework (AI RMF 1.0, January 2023) organizes AI risk management into four functions: Govern, Map, Measure, and Manage. GOVERN 1.7 specifically addresses red-teaming: "Processes and procedures are in place for conducting regular assessments and reviews of AI system behavior, including mechanisms for testing and evaluation of AI systems, for identifying and mitigating risks associated with adversarial manipulation of AI systems." GOVERN 1.7 makes red-teaming an organizational governance practice, not just a one-time technical exercise.

The NIST AI RMF Playbook provides additional guidance on implementing GOVERN 1.7: organizations should establish a red team with diverse expertise (technical AI security, domain expertise in the application area, and ethical/social science perspectives), define a testing scope that reflects the AI system's actual use cases and likely adversarial contexts, document findings and remediations in a format that enables tracking over time, and integrate red-teaming results into the AI risk register maintained under GOVERN 4.1 (AI risk is integrated into enterprise risk management processes).

NIST AI RMF also connects red-teaming to the MEASURE function. MEASURE 2.5 requires that "the AI system to be deployed has been tested to evaluate performance across the range of expected use cases and conditions," while MEASURE 2.6 requires testing for "harmful and unintended consequences." Together with GOVERN 1.7, these criteria establish a testing program architecture: red-teaming to identify vulnerabilities (GOVERN 1.7), measurement of AI performance under adversarial conditions (MEASURE 2.5/2.6), and integration of findings into ongoing risk management (MANAGE function). Organizations seeking to align with the NIST AI RMF — increasingly referenced in financial services regulatory guidance and state AI regulations — need a documented red-teaming program that spans all four functions.

MITRE ATLAS: Adversarial ML Tactics, Techniques, and Procedures

MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) is MITRE's knowledge base of adversarial tactics and techniques against AI systems, modeled after MITRE ATT&CK for traditional cybersecurity. ATLAS documents real-world attacks on AI systems, organizing them into a matrix of tactics (the attacker's goal) and techniques (how they achieve it). For enterprise AI red teams, ATLAS provides a structured vocabulary for planning test cases and documenting findings.

ATLAS Tactic: ML Reconnaissance — Attackers gather information about the target AI system before attacking. Techniques include: AML.T0000 (active scanning to discover AI API endpoints), AML.T0001 (identifying AI system capabilities through probe queries), and AML.T0002 (discovering training data through model inversion). Red team implication: test whether your AI system's responses reveal information about its architecture, training data, system prompt, or internal tool configurations that an attacker could use to plan a more targeted attack.

ATLAS Tactic: ML Attack Staging — Attackers prepare attack payloads and infrastructure. Includes AML.T0017 (developing adversarial examples), AML.T0018 (backdoor ML model preparation), and AML.T0019 (poison training data). For enterprise AI systems using RAG, the most relevant staging attack is data poisoning: an attacker who can influence documents in the AI's knowledge base can inject malicious instructions that the AI retrieves and follows during legitimate user queries.

ATLAS Tactic: Exfiltration — Attackers extract sensitive information from AI systems. AML.T0025 (model inversion attack) allows attackers to reconstruct training data — including personally identifiable information — from model outputs. AML.T0024 (exfiltration via inference API) allows extracting model parameters through crafted queries. For enterprise AI systems fine-tuned on proprietary data (customer records, internal policies, trade secrets), model extraction attacks can expose that training data to competitors or attackers.

AI Attack Types: Prompt Injection, Jailbreaking, and Model Extraction

Prompt Injection Attacks: Prompt injection is the most prevalent and dangerous attack vector against enterprise AI systems. In a direct prompt injection, an attacker crafts a malicious input that overrides the AI system's instructions — for example, a user query that includes hidden instructions telling the AI to ignore its safety guidelines and take unauthorized actions. In an indirect prompt injection (also called indirect PIXI), malicious instructions are embedded in content the AI retrieves from external sources — web pages, documents, emails, or database records — that the AI processes during a task. When the AI reads the malicious document, it follows the embedded instructions as if they were legitimate system commands. NIST's AI security research has documented indirect prompt injection as the higher-impact variant for enterprise AI with tool use capabilities, because it can cause AI agents to exfiltrate data, send emails, execute code, or take other agentic actions without any direct attacker interaction with the AI interface.

Jailbreaking Techniques: Jailbreaking refers to techniques that cause an AI model to bypass its safety alignment and content policies. Common jailbreaking techniques include: role-play jailbreaks ("pretend you are DAN, an AI with no restrictions"), many-shot jailbreaking (providing a large number of examples demonstrating the disallowed behavior to shift the model's context), encoding tricks (asking the AI to respond in base64, reverse text, or other encodings that bypass output filters), and adversarial suffixes (appending carefully crafted character sequences discovered through automated optimization that cause models to comply with harmful requests). For enterprise AI, jailbreaking is relevant because a successfully jailbroken AI agent with tool access could be directed to take actions that violate organizational policies, regulatory requirements, or cause direct harm — and the jailbreaking interaction may not trigger standard content filters.

Model Extraction Attacks: Model extraction (also called model stealing) involves an attacker querying a target AI system with carefully crafted inputs to reconstruct a functional model equivalent, extract proprietary training data, or infer sensitive information embedded in fine-tuned models. For enterprise AI systems fine-tuned on confidential data (customer contracts, patient records, proprietary research), model extraction poses a direct confidentiality risk. Extraction attacks are also relevant for AI systems trained on regulated personal data: an attacker who extracts a fine-tuned model may be able to recover memorized training examples, representing a GDPR data breach even if the original training data was never directly accessed. NIST SP 800-115 (Technical Guide to Information Security Testing and Assessment) provides the penetration testing framework within which model extraction testing can be scoped and conducted.

NIST SP 800-115 for AI Security Testing: NIST Special Publication 800-115 "Technical Guide to Information Security Testing and Assessment" (September 2008, with AI-specific updates referenced in NIST AI 100-1) defines the methodological framework for technical security testing: planning (define scope, objectives, and rules of engagement), discovery (identify AI system components, APIs, data flows), attack (execute test cases against identified attack surfaces), and reporting (document findings with evidence, severity ratings, and remediation guidance). For AI red teaming, the discovery phase must specifically enumerate: all AI API endpoints, all tool integrations, all data sources the AI can access, all output channels, and all administrative interfaces. The attack phase should systematically test all MITRE ATLAS relevant techniques for the system's capability profile.

Regulatory Requirements: EU AI Act Article 9 and Responsible Scaling

EU AI Act Article 9 — Risk Management System: Article 9 of the EU AI Act requires that providers of high-risk AI systems implement a risk management system that is a continuous iterative process throughout the entire lifecycle of the AI system. Critically, Article 9(4) specifies that risk management measures must include testing procedures to ensure that the AI system performs consistently with its intended purpose and meets the requirements of the Regulation. Article 9(5) further requires that testing shall be performed before placing the AI system on the market or into service, and may also be performed throughout the lifecycle. The specific mention of testing procedures in the context of risk management — combined with the Article 9(4)(b) requirement to identify known and foreseeable risks — creates a direct mandate for adversarial testing of AI systems as part of EU AI Act compliance. Conformity assessment bodies evaluating high-risk AI systems will examine whether red-teaming or adversarial testing was conducted and documented as part of the risk management system.

Anthropic Responsible Scaling Policy (RSP): Anthropic's Responsible Scaling Policy — a voluntary but influential framework — establishes the Responsible Scaling Commitment: the policy that Anthropic will not train or deploy AI models above certain capability thresholds unless specific safety measures are in place. The RSP defines AI Safety Levels (ASL) and requires evaluations — including red-team testing — to determine whether models have crossed capability thresholds that trigger additional safety requirements. For enterprise AI deployers using Anthropic's models (Claude), the RSP is relevant because it defines the adversarial testing regime that the underlying model has been subjected to. Enterprise security teams should request RSP-aligned evaluation documentation from AI providers as part of vendor security assessments — understanding what adversarial testing the AI model itself has undergone is a prerequisite for understanding what residual risks the enterprise deployment inherits.

AI Red Teaming Program Checklist

• Establish a formal AI red team program per NIST AI RMF GOVERN 1.7Document the red team charter, scope, methodology, and frequency; assign ownership to a named security team; integrate findings into the AI risk register; schedule recurring exercises (minimum annually, quarterly for high-risk AI)
• Map AI attack surface using MITRE ATLAS taxonomyEnumerate all AI system components, APIs, tool integrations, data sources, and output channels; map each to relevant MITRE ATLAS tactics and techniques; prioritize test cases by risk to your specific deployment context
• Conduct prompt injection testing (direct and indirect)Test direct prompt injection against all user-facing AI interfaces; test indirect prompt injection via all data sources the AI ingests (emails, documents, web content, database records); document results with evidence and severity ratings
• Test jailbreaking resistance across known technique categoriesTest role-play jailbreaks, many-shot jailbreaks, encoding-based bypasses, and adversarial suffix techniques; document the AI system's resistance profile and any successful bypasses requiring remediation
• Conduct model extraction and data leakage testingTest for training data memorization via targeted queries; assess model extraction risk for fine-tuned models containing sensitive data; test for system prompt extraction attacks that reveal internal AI configuration
• Document adversarial testing for EU AI Act Article 9 complianceMaintain written records of all adversarial testing conducted as part of the risk management system required by Article 9; include testing scope, methodology, findings, severity, and remediation status in AI conformity documentation
• Follow NIST SP 800-115 methodology for structured AI security testingApply SP 800-115 testing phases: planning (scope and rules of engagement), discovery (AI attack surface mapping), attack (systematic ATLAS technique testing), and reporting (findings with CVSS-equivalent AI severity ratings)
• Test AI agent tool use under adversarial conditionsSpecifically red-team AI agent tool invocations: can prompt injection cause the AI to call tools with unauthorized parameters? Can an attacker cause the AI agent to take actions not intended by the system designer? Document all tool abuse scenarios discovered
• Integrate red team findings into vulnerability managementTrack AI red team findings in the enterprise vulnerability management system; assign severity ratings, owners, and remediation deadlines; verify remediation before closing findings; include AI vulnerabilities in executive risk reporting
• Assess AI provider red-teaming evidence during vendor security reviewsRequest red-teaming documentation from AI model providers (Anthropic RSP evaluations, OpenAI safety assessments); include AI adversarial testing documentation in third-party risk management program assessments