MIT AI Risk Repository · domain 2: Privacy & Security
2.2 AI system security vulnerabilities and attacks
Vulnerabilities in AI systems, software development toolchains, and hardware that can be exploited, resulting in unauthorized access, data and privacy breaches, or system manipulation causing unsafe outputs or behavior.
- 112
- 12
- 24
- 20
| Label | Value |
|---|---|
| Human | 87 |
| Other | 18 |
| AI | 6 |
| Not coded | 1 |
| Label | Value |
|---|---|
| Intentional | 83 |
| Unintentional | 16 |
| Other | 12 |
| Not coded | 1 |
| Label | Value |
|---|---|
| Post-deployment | 64 |
| Pre-deployment | 25 |
| Other | 22 |
| Not coded | 1 |
| Label | Value |
|---|---|
| 2017 | 3 |
| 2019 | 1 |
| 2022 | 2 |
| 2023 | 3 |
| 2024 | 3 |
| 2025 | 5 |
| 2026 | 7 |
| Label | Value |
|---|---|
| Risk Category | 16 |
| Risk Sub-Category | 96 |
Risk entries
Browse and export all- Privacy loss
"Privacy loss - Unwarranted exposure of an individual’s private life or personal data through cyberattacks, doxxing, etc."
- Jailbreaks and Prompt Injections Threaten Security of LLMs
"LLMs are not adversarially robust and are vulnerable to security failures such as jailbreaks and prompt-injection attacks. While a number of jailbreak attacks have been proposed in the literature, th...
- Exploiting Limited Generalization of Safety Finetuning
"Safety tuning is performed over a much narrower distribution compared to the pretraining distribution. This leaves the model vulnerable to attacks that exploit gaps in the generalization of the safet...
- “Model Psychology” Attacks
"LLMs are vulnerable to “psychological” tricks (Li et al., 2023e; Shen et al., 2023), which can be exploited by attackers. Examples include instructing the model to behave like a specific persona (Sha...
- Attacking LLMs via Additional Modalities a
"LLMs can now process modalities other than text, e.g. images or video frames (OpenAI, 2023c; Gemini Team, 2023). Several studies show that gradient-based attacks on multimodal models are easy and eff...
- Vulnerability to Poisoning and Backdoors
"The previous section explored jailbreaks and other forms of adversarial prompts as ways to elicit harmful capabilities acquired during pretraining. These methods make no assumptions about the trainin...
- Software Vulnerabilities
"Programmers are accustomed to using code generation tools such as Github Copilot for program development, which may bury vulnerabilities in the program."
- Software Security Issues
"The software development toolchain of LLMs is complex and could bring threats to the developed LLM."
- Programming Language
"Most LLMs are developed using the Python language, whereas the vulnerabilities of Python interpreters pose threats to the developed models"
- Deep Learning Frameworks
"LLMs are implemented based on deep learning frameworks. Notably, various vulnerabilities in these frameworks have been disclosed in recent years. As reported in the past five years, three of the most...
- Software Supply Chains
"The software development toolchain of LLMs is complex and could bring threats to the developed LLM."
- Pre-processing Tools
"Pre-processing tools play a crucial role in the context of LLMs. These tools, which are often involved in computer vision (CV) tasks, are susceptible to attacks that exploit vulnerabilities in tools...
- Hardware Vulnerabilities
"The vulnerabilities of hardware systems for training and inferencing brings issues to LLM-based applications."
- Network Devices
"The training of LLMs often relies on distributed network systems [171], [172]. During the transmission of gradients through the links between GPU server nodes, significant volumetric traffic is gener...
- GPU Computation Platforms
"The training of LLMs requires significant GPU resources, thereby introducing an additional security concern. GPU side-channel attacks have been developed to extract the parameters of trained models [...
- Memory and Storage
"Similar to conventional programs, hardware infrastructures can also introduce threats to LLMs. Memory-related vulnerabilities, such as rowhammer attacks [160], can be leveraged to manipulate the para...
- Issues on External Tools
"The external tools (e.g., web APIs) present trustworthiness and privacy issues to LLM-based applications."
- Factual Errors Injected by External Tools
"External tools typically incorporate additional knowledge into the input prompts [122], [178]–[184]. The additional knowledge often originates from public resources such as Web APIs and search engine...
- Exploiting External Tools for Attacks
"Adversarial tool providers can embed malicious instructions in the APIs or prompts [84], leading LLMs to leak memorized sensitive information in the training data or users’ prompts (CVE2023-32786). A...
- Model Attacks
Model attacks exploit the vulnerabilities of LLMs, aiming to steal valuable information or lead to incorrect responses.
- Extraction Attacks
"Extraction attacks [137] allow an adversary to query a black-box victim model and build a substitute model by training on the queries and responses. The substitute model could achieve almost the same...
- Inference Attacks
"Inference attacks [150] include membership inference attacks, property inference attacks, and data reconstruction attacks. These attacks allow an adversary to infer the composition or property inform...
- Poisoning Attacks
"Poisoning attacks [143] could influence the behavior of the model by making small changes to the training data. A number of efforts could even leverage data poisoning techniques to implant hidden tri...
- Overhead Attacks
"Overhead attacks [146] are also named energy-latency attacks. For example, an adversary can design carefully crafted sponge examples to maximize energy consumption in an AI system. Therefore, overhea...
- Novel Attacks on LLMs
Table of examples has: "Prompt Abstraction Attacks [147]: Abstracting queries to cost lower prices using LLM’s API. Reward Model Backdoor Attacks [148]: Constructing backdoor triggers on LLM’s RLHF pr...