MIT AI Risk Repository · domain 2: Privacy & Security
2.2 AI system security vulnerabilities and attacks
Vulnerabilities in AI systems, software development toolchains, and hardware that can be exploited, resulting in unauthorized access, data and privacy breaches, or system manipulation causing unsafe outputs or behavior.
- 112
- 12
- 24
- 20
| Label | Value |
|---|---|
| Human | 87 |
| Other | 18 |
| AI | 6 |
| Not coded | 1 |
| Label | Value |
|---|---|
| Intentional | 83 |
| Unintentional | 16 |
| Other | 12 |
| Not coded | 1 |
| Label | Value |
|---|---|
| Post-deployment | 64 |
| Pre-deployment | 25 |
| Other | 22 |
| Not coded | 1 |
| Label | Value |
|---|---|
| 2017 | 3 |
| 2019 | 1 |
| 2022 | 2 |
| 2023 | 3 |
| 2024 | 3 |
| 2025 | 5 |
| 2026 | 7 |
| Label | Value |
|---|---|
| Risk Category | 16 |
| Risk Sub-Category | 96 |
Risk entries
Browse and export all- Vulnerabilities to jailbreaks exploiting long context windows (many- shot jailbreaking)
"Language models with long context windows are vulnerable to new types of ex- ploitations that are ineffective on models with shorter context windows. While few-shot jailbreaking, which involves provi...
- Misuse of AI model by user-performed persuasion
"AI models can be influenced to accept misinformation through persuasive conversations, even when their initial responses are factually correct. Multi-turn persuasion can be more effective than single...
- Non-decomissionability of models with open weights
"If the model parameter weights are released or leaked in a security breach, the model cannot be decommissioned because the developer no longer has control over the publicly available model or its use...
- Interconnectivity with malicious external tools
"The growing integration and interconnectivity with external tools and plugins increase the risk of exposure to malicious external inputs. This interconnectivity makes it easier for external tools to...
- Model weight leak
"Model weights or access to them can be leaked when initial access is granted only to a select group of individuals, such as institutional researchers [209]. This risk can increase as more people gain...
- Technical vulnerabilities (Robustness - vulnerability to jailbreaking
"Individuals can manipulate models into performing actions that violate the model’s usage restrictions—a phenomenon known as “jailbreaking.” These manipulations may result in causing the model to perf...
- Insufficient Security Measures
Malicious entities can take advantage of weaknesses in AI algorithms to alter results, potentially resulting in tangible real-life impacts. Additionally, it’s vital to prioritize safeguarding privacy...
- Security - Robustness
While AI safety focuses on threats emanating from generative AI systems, security centers on threats posed to these systems. The most extensively discussed issue in this context are jailbreaking risks...
- Data poisoning
"A type of adversarial attack where an adversary or malicious insider injects intentionally corrupted, false, misleading, or incorrect samples into the training or fine-tuning datasets."
- Prompt injection attack
"A prompt injection attack forces a generative model that takes a prompt as input to produce unexpected output by manipulating the structure, instructions, or information contained in its prompt."
- Extraction attack
"An attribute inference attack is used to detect whether certain sensitive features can be inferred about individuals who participated in training a model. These attacks occur when an adversary has so...
- Evasion attack
"Evasion attacks attempt to make a model output incorrect results by slightly perturbing the input data that is sent to the trained model."
- Prompt leaking
"A prompt leak attack attempts to extract a model's system prompt (also known as the system message)."
- Jailbreaking
"A jailbreaking attack attempts to break through the guardrails that are established in the model to perform restricted actions."
- Prompt priming
"Because generative models tend to produce output like the input provided, the model can be prompted to reveal specific kinds of information. For example, adding personal information in the prompt inc...
- Membership inference attack
"A membership inference attack repeatedly queries a model to determine whether a given input was part of the model’s training. More specifically, given a trained model and a data sample, an attacker s...
- Attribute inference attack
"An attribute inference attack repeatedly queries a model to detect whether certain sensitive features can be inferred about individuals who participated in training a model. These attacks occur when...
- Harmful code generation
"Models might generate code that causes harm or unintentionally affects other systems."
- Prompt Attacks
carefully controlled adversarial perturbation can flip a GPT model’s answer when used to classify text inputs. Furthermore, we find that by twisting the prompting question in a certain way, one can so...
- Poisoning Attacks
fool the model by manipulating the training data, usually performed on classification models
- Prompt injection
"Prompt Injections are a form of Adversarial Input that involve manipulating the text instructions given to a GenAI system (Liu et al., 2023). Prompt Injections exploit loopholes in a model’s architec...
- Adversarial input
"Adversarial Inputs involve modifying individual input data to cause a model to malfunction. These modifications, which are often imperceptible to humans, exploit how the model makes decisions to prod...
- Jailbreaking
"Jailbreaking aims to bypass or remove restrictions and safety filters placed on a GenAI model completely (Chao et al., 2023; Shen et al., 2023). This gives the actor free rein to generate any output,...
- Model extraction
"Data Exfiltration goes beyond revealing private information, and involves illicitly obtaining the training data used to build a model that may be sensitive or proprietary. Model Extraction is the sam...
- Steganography
"Steganography is the practice of hiding coded messages in GenAI model outputs, which may allow malicious actors to communicate covertly.8"