Artificial Intelligence Policy Tracker

AI governance glossary

Red teaming (AI)

What is AI red teaming?

AI red teaming is structured adversarial testing in which testers try to make an AI system fail, by producing harmful output, leaking data, bypassing safeguards or taking unintended actions, so that weaknesses are found and fixed before attackers or users find them.

Why it matters

  • The EU AI Act requires providers of general-purpose AI models with systemic risk to carry out and document adversarial testing (Article 55).
  • It is how prompt injection, data leakage and unsafe behaviour are found before release, and the OWASP Top 10 for LLM Applications gives it a checklist.
  • Findings only count if they are fixed and re-tested, so red teaming belongs in the release process, not as a one-off exercise.

In detail

Structured adversarial testing in which testers try to make an AI system fail: produce harmful output, leak data, ignore its instructions or be misused. It is one of the evaluations expected of the most capable general-purpose models and a common control for generative AI.

Source: EU AI Act (Regulation (EU) 2024/1689). A plain-language explanation for orientation, not the legal text; follow the source for the binding wording.

Red-team testing control →AI red team test plan →

Laws and policies on record that use it

Duties that mention it

  • Prompt injectionPrompt injection is an attack in which crafted input, typed by a user or hidden in content the model reads, ov...
  • OWASP Top 10 for LLM ApplicationsThe OWASP Top 10 for LLM Applications is the Open Worldwide Application Security Project's list of the ten mos...
  • Agentic AIAgentic AI is an AI system that pursues a goal by planning and taking actions with some autonomy, such as call...
  • AI systemA machine-based system that operates with some autonomy, may adapt after it is deployed, and infers from the i...
  • AI GRC (governance, risk and compliance)AI GRC (governance, risk and compliance) is the integrated discipline of assigning accountability and policy f...
  • AI riskAI risk is the combination of how likely an AI system is to cause harm and how severe that harm would be, to p...

All glossary terms · Think a definition is off? Tell us; corrections are logged on the corrections page.

Frequently asked questions

What is AI red teaming?

AI red teaming is structured adversarial testing in which testers try to make an AI system fail, by producing harmful output, leaking data, bypassing safeguards or taking unintended actions, so that weaknesses are found and fixed before attackers or users find them.

Where does the term red teaming come from?

This explanation follows EU AI Act (Regulation (EU) 2024/1689). It is a plain-language paraphrase for orientation; the source has the binding wording.

Which laws and policies use the term red teaming?

Among the records on this site: Illinois AI Safety Measures Act (SB 315) (Illinois); Colorado ADMT law (SB 26-189) (Colorado); Law on Artificial Intelligence (2025) (Kazakhstan); Law No. 132/2025 on artificial intelligence (Italy) and Texas Responsible AI Governance Act (TRAIGA) (Texas).