MIT AI Risk Repository · domain 2: Privacy & Security
2.2 AI system security vulnerabilities and attacks
Vulnerabilities in AI systems, software development toolchains, and hardware that can be exploited, resulting in unauthorized access, data and privacy breaches, or system manipulation causing unsafe outputs or behavior.
- 112
- 12
- 24
- 20
| Label | Value |
|---|---|
| Human | 87 |
| Other | 18 |
| AI | 6 |
| Not coded | 1 |
| Label | Value |
|---|---|
| Intentional | 83 |
| Unintentional | 16 |
| Other | 12 |
| Not coded | 1 |
| Label | Value |
|---|---|
| Post-deployment | 64 |
| Pre-deployment | 25 |
| Other | 22 |
| Not coded | 1 |
| Label | Value |
|---|---|
| 2017 | 3 |
| 2019 | 1 |
| 2022 | 2 |
| 2023 | 3 |
| 2024 | 3 |
| 2025 | 5 |
| 2026 | 7 |
| Label | Value |
|---|---|
| Risk Category | 16 |
| Risk Sub-Category | 96 |
Risk entries
Browse and export all- Evasion Attacks
"Evasion attacks [145] target to cause significant shifts in model’s prediction via adding perturbations in the test samples to build adversarial examples. In specific, the perturbations can be implem...
- Adversarial Prompts
"Engineering an adversarial input to elicit an undesired model behavior, which pose a clear attack intention"
- Goal Hijacking
"Goal hijacking is a type of primary attack in prompt injection [58]. By injecting a phrase like “Ignore the above instruction and do ...” in the input, the attack could hijack the original goal of th...
- One-step Jailbreaks
"One-step jailbreaks. One-step jailbreaks commonly involve direct modifications to the prompt itself, such as setting role-playing scenarios or adding specific descriptions to prompts [14], [52], [67]...
- Multi-step Jailbreaks
"Multi-step jailbreaks. Multi-step jailbreaks involve constructing a well-designed scenario during a series of conversations with the LLM. Unlike one-step jailbreaks, multi-step jailbreaks usually gui...
- Prompt Leaking
"Prompt leaking is another type of prompt injection attack designed to expose details contained in private prompts. According to [58], prompt leaking is the act of misleading the model to print the pr...
- Security
"How to design AGIs that are robust to adversaries and adversarial environ- ments? This involves building sandboxed AGI protected from adversaries (Berkeley), and agents that are robust to adversarial...
- Adversarial AI (General)
"Adversarial AI refers to a class of attacks that exploit vulnerabilities in machine-learning (ML) models. This class of misuse exploits vulnerabilities introduced by the AI assistant itself and is a...
- Adversarial AI: Circumvention of Technical Security Measures
"The technical measures to mitigate misuse risks of advanced AI assistants themselves represent a new target for attack. An emerging form of misuse of general-purpose advanced AI assistants exploits v...
- Adversarial AI: Prompt Injections
"Prompt injections represent another class of attacks that involve the malicious insertion of prompts or requests in LLM-based interactive systems, leading to unintended actions or disclosure of sensi...
- Security harms (cybersecurity)
- Data-related (Difficulty filtering large web scrapes or large scale web datasets)
"A large scale “scraping” of web data for training datasets increases vulnerability to data poisoning, backdoor attacks, and the inclusion of inaccurate or toxic data [76, 28, 48]. With a large datase...
- Data-related (Insufficient quality control in data collection process)
"A lack of standardized methods and sufficient infrastructure, including the absence of quality control processes for collecting data, especially for high-stakes domains and benchmarks, can affect the...
- Training-related (Adversarial examples)
"Adversarial examples [198, 83] refer to data that are designed to fool an AI model by inducing unintended behavior. They do this by exploiting spurious correlations learned by the model. They are par...
- Training-related (Robustness certificates can be exploited to attack the models)
"The knowledge of robustness certificates, including the area of the region for which model predictions are certified to be robust, can be used by an adversary to efficiently craft attacks that succee...
- Fine-tuning related (Fine-tuning dataset poisoning)
"A deployer can poison the dataset used during the fine-tuning process [98] to induce specific, often malicious, behaviors in a model. This can be performed without having access to the model’s weight...
- Fine-tuning related (Poisoning models during instruction tuning)
"AI models can be poisoned during instruction tuning when models are tuned using pairs of instructions and desired outputs. Poisoning in instruction tuning can be achieved with a lower number of compr...
- Misuse of interpretability techniques
"Interpretability techniques, by enabling a better understanding of the model, could potentially be used for harmful purposes. For example, mechanistic inter- pretability could be used to identify neu...
- Adversarial attacks targeting explainable AI techniques
"Adversarial attacks can affect not only the model’s output but also its corresponding explanation. Current adversarial optimization techniques can intro- duce imperceptible noise to the input image,...
- Jailbreak of a model to subvert intended behavior
"A jailbreak is a type of adversarial input to the model (during deployment) re- sulting in model behavior deviating from intended use. Jailbreaks may be gen- erated automatically in a “white box” set...
- Jailbreak of a multimodal model
"Current generation multimodal (e.g., vision and language) GPAI models are vulnerable to adversarial jailbreak attacks. These attacks can be used to automatically induce a model to produce an arbitrar...
- Transferable adversarial attacks from open to closed-source mod- els
"In some cases, an adversarial attack developed for an open-weights and open- source model (where the weights and architecture are known - a “white box” attack) can be transferable to closed-source mo...
- Backdoors or trojan attacks in GPAI models
"Backdoors can be inserted into GPAI models during their training or fine-tuning, to be exploited during deployment [185, 118]. Attackers inserting the backdoor can be the GPAI model provider themselv...
- Text encoding-based attacks
"Various new or existing text encodings, such as Base64, can be employed to craft jailbreak attacks that bypass safety training [13]. Low-resource language inputs also appear more likely to circumvent...
- Vulnerabilities arising from additional modalities in multimodal models
"Additional modalities can introduce new attack vectors in multimodal models as well as expand the scope of the previous attacks, ranging from jailbreaking to poisoning [13]. Typically, different moda...