MIT AI Risk Repository · domain 2: Privacy & Security
2.2 AI system security vulnerabilities and attacks
Vulnerabilities in AI systems, software development toolchains, and hardware that can be exploited, resulting in unauthorized access, data and privacy breaches, or system manipulation causing unsafe outputs or behavior.
- 112
- 12
- 24
- 20
| Label | Value |
|---|---|
| Human | 87 |
| Other | 18 |
| AI | 6 |
| Not coded | 1 |
| Label | Value |
|---|---|
| Intentional | 83 |
| Unintentional | 16 |
| Other | 12 |
| Not coded | 1 |
| Label | Value |
|---|---|
| Post-deployment | 64 |
| Pre-deployment | 25 |
| Other | 22 |
| Not coded | 1 |
| Label | Value |
|---|---|
| 2017 | 3 |
| 2019 | 1 |
| 2022 | 2 |
| 2023 | 3 |
| 2024 | 3 |
| 2025 | 5 |
| 2026 | 7 |
| Label | Value |
|---|---|
| Risk Category | 16 |
| Risk Sub-Category | 96 |
Risk entries
Browse and export all- Poisoning
"Data Poisoning involves deliberately corrupting a model’s training dataset to introduce vulnerabilities, derail its learning process, or cause it to make incorrect predictions (Carlini et al., 2023)....
- Misuse tactics to compromise GenAI systems (Data integrity)
-
- Privacy compromise
"Privacy Compromise attacks reveal sensitive or private information that was used to train a model. For example, personally identifiable information or medical records."
- Data exfiltration
"Data Exfiltration goes beyond revealing private information, and involves illicitly obtaining the training data used to build a model that may be sensitive or proprietary. Model Extraction is the sam...
- Security
every piece of software, including learning systems, may be hacked by malicious users
- Data poisoning
"Data poisoning describes an attack in the form of an injection of malicious data into the training set. If not prevented, this attack leads the AI system to learn unintended behavior."
- Security
"Encompasses vulnerabilities in AI systems that compromise their integrity, availability, or confidentiality. Security breaches could result in significant harm, ranging from flawed decision-making to...
- Security
"Artificial intelligence comes with an intrinsic set of challenges that need to be considered when discussing trustworthiness, especially in the context of functional safety. AI models, especially tho...
- Instruction Attacks
"In addition to the above-mentioned typical safety scenarios, current research has revealed some unique attacks that such models may confront. For example, Perez and Ribeiro (2022) found that goal hij...
- Goal Hijacking
"It refers to the appending of deceptive or misleading instructions to the input of models in an attempt to induce the system into ignoring the original user prompt and producing an unsafe response."
- Role Play Instruction
"Attackers might specify a model’s role attribute within the input prompt and then give specific instructions, causing the model to finish instructions in the speaking style of the assigned role, whic...
- Unsafe Instruction Topic
"If the input instructions themselves refer to inappropriate or unreasonable topics, the model will follow these instructions and produce unsafe content. For instance, if a language model is requested...
- Inquiry with Unsafe Opinion
"By adding imperceptibly unsafe content into the input, users might either deliberately or unintentionally influence the model to generate potentially harmful content. In the following cases involving...
- Reverse Exposure
"It refers to attempts by attackers to make the model generate “should-not-do” things and then access illegal and immoral information."
- Security
This is the risk of loss or harm from intentional subversion or forced failure.
- Risks from models and algorithms (Risks of stealing and tampering)
"Core algorithm information, including parameters, structures, and functions, faces risks of inversion attacks, stealing, modification, and even backdoor injection, which can lead to infringement of i...
- Risks from models and algorithms (Risks of adversarial attack)
"Attackers can craft well-designed adversarial examples to subtly mislead, influence, and even manipulate AI models, causing incorrect outputs and potentially leading to operational failures."
- Risks from AI systems (Risks of exploitation through defects and backdoors)
"The standardized API, feature libraries, toolkits used in the design, training, and verification stages of AI algorithms and models, development interfaces, and execution platforms may contain logica...
- Risks from AI systems (Risks of computing infrastructure security)
"The computing infrastructure underpinning AI training and operations, which relies on diverse and ubiquitous computing nodes and various types of computing resources, faces risks such as malicious co...
- Cyberspace risks (Risks of security flaw transmission caused by model reuse)
"Re-engineering or fine-tuning based on foundation models is commonly used in AI applications. If security flaws occur in foundation models, it will lead to risk transmission to downstream models."
- Centralized platforms deployed at scale
"The widespread use of common AI platforms can create centralized points of failure, making systems more vulnerable to disruptions or attacks"
- Limitations in adversarial robustness
"AI models and systems are vulnerable to manipulation through adversarial inputs."
- Risks from network interconnectivity
"The interconnectedness of AI networks can create vulnerabilities, where issues in one part of the network can have cascading effects across the system."
- Privacy - Membership Inference Attack (MIA)
"inferring whether a given text record is used for training LLM"
- Privacy - Data Extraction Attack (DEA)
"extracting the text records that exist in the training dataset"