MIT AI Risk Repository · Risk Sub-Category · 47.01.02

Technical vulnerabilities (Robustness - vulnerability to jailbreaking

Category: Technical and operational risks

Open the MIT Risk Navigator All entries from G'sell2024

Description

"Individuals can manipulate models into performing actions that violate the model’s usage restrictions—a phenomenon known as “jailbreaking.” These manipulations may result in causing the model to perform tasks that the developers have explicitly prohibited (see section 3.2.1.). For instance, users may ask the model to provide information on how to conduct illegal activities— asking for detailed instructions on how to build a bomb or create highly toxic drugs."

From Regulating under Uncertainty: Governance Options for Generative AI (G'sell2024), as extracted by the MIT AI Risk Repository (CC BY 4.0).

Classification

Domain: 2. Privacy & Security
Subdomain: 2.2 AI system security vulnerabilities and attacks
Causal entity: Human
Intent: Intentional
Timing: Post-deployment

Subdomain definition: Vulnerabilities in AI systems, software development toolchains, and hardware that can be exploited, resulting in unauthorized access, data and privacy breaches, or system manipulation causing unsafe outputs or behavior.

Real-world incidents in this subdomain

Browse all incidents in this subdomain

How other frameworks describe this risk

Privacy loss
A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms (Abercrombie2024)
Exploiting Limited Generalization of Safety Finetuning
Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)
Jailbreaks and Prompt Injections Threaten Security of LLMs
Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)
“Model Psychology” Attacks
Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)
Attacking LLMs via Additional Modalities a
Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)
Vulnerability to Poisoning and Backdoors
Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)
Model Attacks
Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)
Hardware Vulnerabilities
Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)

Other entries from G'sell2024

Technical and operational risks Risk Category · 7.3
Technical vulnerabilities (Robustness - unexpected behaviour) Risk Sub-Category · 7.3
Technical vulnerabilities (Robustness - unexpected behaviour) Additional evidence
Technical vulnerabilities (Robustness - vulnerability to jailbreaking Additional evidence
Technical vulnerabilities (The risk of misalignment) Risk Sub-Category · 7.1
Technical vulnerabilities (The risk of misalignment) Additional evidence
Factually incorrect content (inaccuracies and fabricated sources) Risk Sub-Category · 3.1
Factually incorrect content (inaccuracies and fabricated sources) Additional evidence
Opacity (the black box problem) Risk Sub-Category · 7.4
Opacity (industry opacity) Risk Sub-Category · 6.4
Opacity (industry opacity) Additional evidence
Ethical and social risks Risk Category