MIT AI Risk Repository · Risk Sub-Category · 64.04.01

Prompt injection

Category: Misuse tactics to compromise GenAI systems (Model integrity)

Open the MIT Risk Navigator All entries from Marchal2024

Description

"Prompt Injections are a form of Adversarial Input that involve manipulating the text instructions given to a GenAI system (Liu et al., 2023). Prompt Injections exploit loopholes in a model’s architec- tures that have no separation between system instructions and user data to produce a harmful output (Perez and Ribeiro, 2022). While researchers may use similar techniques to test the robustness of GenAI models, malicious actors can also leverage them. For example, they might flood a model with manipulative prompts to cause denial-of-service attacks or to bypass an AI detection software."

From Generative AI Misuse: A Taxonomy of Tactics and Insights from Real-World Data (Marchal2024), as extracted by the MIT AI Risk Repository (CC BY 4.0).

Classification

Domain: 2. Privacy & Security
Subdomain: 2.2 AI system security vulnerabilities and attacks
Causal entity: Human
Intent: Intentional
Timing: Post-deployment

Subdomain definition: Vulnerabilities in AI systems, software development toolchains, and hardware that can be exploited, resulting in unauthorized access, data and privacy breaches, or system manipulation causing unsafe outputs or behavior.

Real-world incidents in this subdomain

Browse all incidents in this subdomain

How other frameworks describe this risk

Privacy loss
A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms (Abercrombie2024)
Exploiting Limited Generalization of Safety Finetuning
Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)
Jailbreaks and Prompt Injections Threaten Security of LLMs
Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)
“Model Psychology” Attacks
Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)
Attacking LLMs via Additional Modalities a
Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)
Vulnerability to Poisoning and Backdoors
Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)
Model Attacks
Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)
Hardware Vulnerabilities
Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)

Other entries from Marchal2024

Misuse tactics that exploit GenAI capabilities (Realistic depiction of human likeness) Risk Category · 4.3
Impersonation Risk Sub-Category · 4.3
Impersonation Additional evidence
Appropriated Likeness Risk Sub-Category · 4.3
Appropriated Likeness Additional evidence
Sockpuppeting Risk Sub-Category · 4.1
Sockpuppeting Additional evidence
Non-consensual intimate imagery (NCII) Risk Sub-Category · 4.3
Non-consensual intimate imagery (NCII) Additional evidence
Child sexual abuse material (CSAM) Risk Sub-Category · 4.3
Child sexual abuse material (CSAM) Additional evidence
Misuse tactics that exploit GenAI capabilities (Realistic depictions of non-humans) Risk Category · 4.3