AI governance glossary
Red teaming (AI)
What is AI red teaming?
AI red teaming is structured adversarial testing in which testers try to make an AI system fail, by producing harmful output, leaking data, bypassing safeguards or taking unintended actions, so that weaknesses are found and fixed before attackers or users find them.
Why it matters
- The EU AI Act requires providers of general-purpose AI models with systemic risk to carry out and document adversarial testing (Article 55).
- It is how prompt injection, data leakage and unsafe behaviour are found before release, and the OWASP Top 10 for LLM Applications gives it a checklist.
- Findings only count if they are fixed and re-tested, so red teaming belongs in the release process, not as a one-off exercise.
In detail
Structured adversarial testing in which testers try to make an AI system fail: produce harmful output, leak data, ignore its instructions or be misused. It is one of the evaluations expected of the most capable general-purpose models and a common control for generative AI.
Source: EU AI Act (Regulation (EU) 2024/1689). A plain-language explanation for orientation, not the legal text; follow the source for the binding wording.
Red-team testing control →AI red team test plan →
Laws and policies on record that use it
- Illinois AI Safety Measures Act (SB 315)Illinois (United States) · Adopted
- Colorado ADMT law (SB 26-189)Colorado (United States) · Adopted
- Law on Artificial Intelligence (2025)Kazakhstan · Adopted
- Law No. 132/2025 on artificial intelligenceItaly · In force
- Texas Responsible AI Governance Act (TRAIGA)Texas (United States) · In force
- EU AI ActEuropean Union · Partially applicable
- Framework Convention on AI (CETS 225)Council of Europe · Adopted
- Colorado AI ActColorado (United States) · Repealed
Duties that mention it
- Establish accountability processes and a risk-management process (guardrails 1 and 2)Australian Voluntary AI Safety Standard · Australia
- Test and monitor systems, enable human control, and be transparent with users (guardrails 4 to 6)Australian Voluntary AI Safety Standard · Australia
- Provide contestability, supply-chain transparency and records (guardrails 7 to 9)Australian Voluntary AI Safety Standard · Australia
- Do not deploy or provide AI for prohibited practicesEU AI Act · European Union
- Ensure AI literacy of staff operating AI systemsEU AI Act · European Union
- Establish a risk management system for high-risk AIEU AI Act · European Union
- Apply data governance and quality criteria to training, validation and testing dataEU AI Act · European Union
- Design high-risk systems to log events automaticallyEU AI Act · European Union
Related concepts
- Prompt injectionPrompt injection is an attack in which crafted input, typed by a user or hidden in content the model reads, ov...
- OWASP Top 10 for LLM ApplicationsThe OWASP Top 10 for LLM Applications is the Open Worldwide Application Security Project's list of the ten mos...
- Agentic AIAgentic AI is an AI system that pursues a goal by planning and taking actions with some autonomy, such as call...
- AI systemA machine-based system that operates with some autonomy, may adapt after it is deployed, and infers from the i...
- AI GRC (governance, risk and compliance)AI GRC (governance, risk and compliance) is the integrated discipline of assigning accountability and policy f...
- AI riskAI risk is the combination of how likely an AI system is to cause harm and how severe that harm would be, to p...
All glossary terms · Think a definition is off? Tell us; corrections are logged on the corrections page.
Frequently asked questions
What is AI red teaming?
AI red teaming is structured adversarial testing in which testers try to make an AI system fail, by producing harmful output, leaking data, bypassing safeguards or taking unintended actions, so that weaknesses are found and fixed before attackers or users find them.
Where does the term red teaming come from?
This explanation follows EU AI Act (Regulation (EU) 2024/1689). It is a plain-language paraphrase for orientation; the source has the binding wording.
Which laws and policies use the term red teaming?
Among the records on this site: Illinois AI Safety Measures Act (SB 315) (Illinois); Colorado ADMT law (SB 26-189) (Colorado); Law on Artificial Intelligence (2025) (Kazakhstan); Law No. 132/2025 on artificial intelligence (Italy) and Texas Responsible AI Governance Act (TRAIGA) (Texas).