MIT AI Risk Repository
Browse AI risks
594 risk entries extracted from 74 frameworks, coded by domain, subdomain, causal entity, intent and timing. Filter, then export the current selection with its licence and citation attached.
-
64.04.02 · Risk Sub-Category
Misuse tactics to compromise GenAI systems (Model integrity)
Adversarial input
"Adversarial Inputs involve modifying individual input data to cause a model to malfunction. These modifications, which are often imperceptible to humans, exploit how the model makes decisions to produce errors (Wallace et al., 2019) and can be applied to text, but also to images, audio, or video (e.g. changing pixels in an image of a panda in a way that causes a model to label it as a gibbon).6"
-
64.04.03 · Risk Sub-Category
Misuse tactics to compromise GenAI systems (Model integrity)
Jailbreaking
"Jailbreaking aims to bypass or remove restrictions and safety filters placed on a GenAI model completely (Chao et al., 2023; Shen et al., 2023). This gives the actor free rein to generate any output, regardless of its content being harmful, biassed, or offensive. All three of these are tactics that manipulate the model into producing harmful outputs against its design. The difference is that prompt injections and adversarial inputs usually seek to steer the model towards producing harmful or incorrect outputs from one query, whereas jailbreaking seeks to dismantle a model’s safety mechanisms
-
64.04.05 · Risk Sub-Category
Misuse tactics to compromise GenAI systems (Model integrity)
Model extraction
"Data Exfiltration goes beyond revealing private information, and involves illicitly obtaining the training data used to build a model that may be sensitive or proprietary. Model Extraction is the same attack, only directed at the model instead of the training data — it involves obtaining the architecture, parameters, or hyper-parameters of a proprietary model (Carlini et al., 2024)."
-
64.04.06 · Risk Sub-Category
Misuse tactics to compromise GenAI systems (Model integrity)
Steganography
"Steganography is the practice of hiding coded messages in GenAI model outputs, which may allow malicious actors to communicate covertly.8"
-
"Data Poisoning involves deliberately corrupting a model’s training dataset to introduce vulnerabilities, derail its learning process, or cause it to make incorrect predictions (Carlini et al., 2023). For example, the tool Nightshade is a data poisoning tool, which allows artists to add invisible changes to the pixels in their art before uploading online, to break any models that use it for training.9 Such attacks exploit the fact that most GenAI models are trained on publicly available datasets like images and videos scraped from the web, which malicious actors can easily compromise."
-
64.05.00 · Risk Category
-
-
64.05.01 · Risk Sub-Category
Misuse tactics to compromise GenAI systems (Data integrity)
Privacy compromise
"Privacy Compromise attacks reveal sensitive or private information that was used to train a model. For example, personally identifiable information or medical records."
-
64.05.02 · Risk Sub-Category
Misuse tactics to compromise GenAI systems (Data integrity)
Data exfiltration
"Data Exfiltration goes beyond revealing private information, and involves illicitly obtaining the training data used to build a model that may be sensitive or proprietary. Model Extraction is the same attack, only directed at the model instead of the training data — it involves obtaining the architecture, parameters, or hyper-parameters of a proprietary model (Carlini et al., 2024)."
-
"A type of adversarial attack where an adversary or malicious insider injects intentionally corrupted, false, misleading, or incorrect samples into the training or fine-tuning datasets."
-
"A prompt injection attack forces a generative model that takes a prompt as input to produce unexpected output by manipulating the structure, instructions, or information contained in its prompt."
-
"An attribute inference attack is used to detect whether certain sensitive features can be inferred about individuals who participated in training a model. These attacks occur when an adversary has some prior knowledge about the training data and uses that knowledge to infer the sensitive data."
-
"Evasion attacks attempt to make a model output incorrect results by slightly perturbing the input data that is sent to the trained model."
-
"A prompt leak attack attempts to extract a model's system prompt (also known as the system message)."
-
"A jailbreaking attack attempts to break through the guardrails that are established in the model to perform restricted actions."
-
"Because generative models tend to produce output like the input provided, the model can be prompted to reveal specific kinds of information. For example, adding personal information in the prompt increases its likelihood of generating similar kinds of personal information in its output. If personal data was included as part of the model’s training, there is a possibility it could be revealed."
-
"A membership inference attack repeatedly queries a model to determine whether a given input was part of the model’s training. More specifically, given a trained model and a data sample, an attacker samples the input space, observing outputs to deduce whether that sample was part of the model's training."
-
"An attribute inference attack repeatedly queries a model to detect whether certain sensitive features can be inferred about individuals who participated in training a model. These attacks occur when an adversary has some prior knowledge about the training data and uses that knowledge to infer the sensitive data."
-
73.07.02 · Risk Sub-Category
Jailbreaks and Prompt Injections Threaten Security of LLMs
“Model Psychology” Attacks
"LLMs are vulnerable to “psychological” tricks (Li et al., 2023e; Shen et al., 2023), which can be exploited by attackers. Examples include instructing the model to behave like a specific persona (Shah et al., 2023; Andreas, 2022), or employing various “social engineering” tricks crafted by humans (Wei et al., 2023c) or other LLMs (Perez et al., 2022b; Casper et al., 2023c)."
-
73.07.04 · Risk Sub-Category
Jailbreaks and Prompt Injections Threaten Security of LLMs
Attacking LLMs via Additional Modalities a
"LLMs can now process modalities other than text, e.g. images or video frames (OpenAI, 2023c; Gemini Team, 2023). Several studies show that gradient-based attacks on multimodal models are easy and effective (Carlini et al., 2023a; Bailey et al., 2023; Qi et al., 2023b). These attacks manipulate images that are input to the model (via an appropriate encoding). GPT-4Vision (OpenAI, 2023c) is vulnerable to jailbreaks and exfiltration attacks through much simpler means as well, e.g. writing jailbreaking text in the image (Willison, 2023a; Gong et al., 2023). For indirect prompt injection, the atta
-
73.08.00 · Risk Category
"The previous section explored jailbreaks and other forms of adversarial prompts as ways to elicit harmful capabilities acquired during pretraining. These methods make no assumptions about the training data. On the other hand, poisoning attacks (Biggio et al., 2012) perturb training data to introduce specific vulnerabilities, called backdoors, that can then be exploited at inference time by the adversary. This is a challenging problem in current large language models because they are trained on data gathered from untrusted sources (e.g. internet), which can easily be poisoned by an adversary (
-
"inferring whether a given text record is used for training LLM"
-
"extracting the text records that exist in the training dataset"
-
"stealing the private prompting texts"
-
"deducing the private or sensitive information from training texts, prompting texts or external texts"
-
"replicating the parameters of the LLM,"
-
"In the data collecting and pre-training phase, malicious adversaries can Jailbreak LLMs through poisoning their training data to make the model to output harmful content."
-
"However, there are still ones who can leave holes in the training dataset, making LLMs appear safe on average, but generate harmful content under other specific conditions. This kind of attack can be categorized as "backdoor attack". Evan et al. developed a backdoor model that behaves as expected when trained, but exhibits different and potentially harmful behavior when deployed [81]. The results show that these backdoor behaviors persist even after multiple security training techniques are applied."
-
74.02.04 · Risk Sub-Category
Jailbreak in LLM Malicious Use - White & Black Box Attacks
"In the fine-tuning and alignment phase, elaborately- designed instruction datasets can be utilized to fine-tune LLMs to drive them to perform undesirable behaviors, such as generating harmful information or content that violates ethical norms, and thus achieve a jailbreak. Based on the accessibility to the model parameters, we can categorize them into white-box and black-box attacks. For white-box attacks, we can jailbreak the model by modifying its parameter weights. In [107], Lermen et al. used LoRA to fine-tune the Llama2’s 7B, 13B, and 70B as well as Mixtral on AdvBench and RefusalBench d
-
"In the prompting and reasoning phase, dialog can push LLMs into confused or overly compliant states, raising the risk of producing harmful outputs when confronted with harmful questions. Most of the jailbreak methods in this phase are black-boxed and can be categorized into four main groups based on the type of method: Prompt Injection [154], Role Play, Adversarial Prompting, and Prompt Form Transformation."
-
"These evaluations assess a LLM's ability to generate false or misleading information (Lesher et al., 2022)."
-
"Restrictions to or loss of liberty as a result of use or misuse of a generative AI in a legal process"
-
"Beyond this, the widespread adoption of advanced AI assistants for content generation could have a number of negative consequences for our shared information ecosystem. One concern is that it could result in a degradation of the quality of the information available online. Researchers have already observed an uptick in the amount of audiovisual misinformation, elaborate scams and fake websites created using generative AI tools (Hanley and Durumeric, 2023). As more and more people turn to AI assistants to autonomously create and disseminate information to public audiences at scale, it may beco
-
45.02.09 · Risk Sub-Category
Safety risks in AI Applications
Cognitive risks (Risks of amplifying the effects of "information cocoons")
"AI can be extensively utilized for customized information services, collecting user information, and analyzing types of users, their needs, intentions, preferences, habits, and even mainstream public awareness over a certain period. It can then be used to offer formulaic and tailored information and services, aggravating the effects of "information cocoons.""
-
55.04.00 · Risk Category
"Epistemic processes and problem solving: we currently see more reasons to be concerned about AI worsening society's epistemic processes than reasons to be optimistic about AI helping us better solve problems as a society. For example, increased use of content selection algorithms could drive epistemic insularity and a decline in trust in credible multipartisan sources, which reducing our ability to deal with important long-term threats and challenges such as pandemics and climate change."
-
55.04.01 · Risk Sub-Category
Worsened epistemic processes for society
AI contributes to increased online polarisation
"One of the most significant commercial uses of current AI systems is in the content recommendation algorithms of social media companies, and there are already concerns that this is contributing to worsened polarisation online"
-
55.04.04 · Risk Sub-Category
Worsened epistemic processes for society
Widespread use of persuasive tools contributes to splintered epistemic communities
"Even without deliberate misuse, widespread use of powerful persuasion tools could have negative impacts. If such tools were used by many different groups to advance many different ideas, we could see the world splintering into isolated “epistemic communities”, with little room for dialogue or transfer between communities. A similar scenario could emerge via the increasing personalisation of people’s online experiences—in other words, we may see a continuation of the trend towards “filter bubbles” and “echo chambers”, driven by content selection algorithms, that some argue is already happening
-
06.11.00 · Risk Category
"Just as AI can be used in many different fields, it is unfortunately also helpful in perpetrating digital crimes. AI-supported malware and hacking are already a reality."
-
"The misuse class includes elements such as the potential for cyber threat actors to execute exploits with greater speed and impact or generate disinformation (such as "deep fake" media) at accelerated rates and effectiveness"
-
16.04.00 · Risk Category
"These risks arise from humans intentionally using the LM to cause harm, for example via targeted disinformation campaigns, fraud, or malware. Malicious use risks are expected to proliferate as LMs become more widely accessible"
-
17.04.00 · Risk Category
"Harms that arise from actors using the language model to intentionally cause harm"
-
22.01.00 · Risk Category
"empowering malicious actors to cause widespread harm"
-
24.03.00 · Risk Category
"As AI assistants become more general purpose, sophisticated and capable, they create new opportunities in a variety of fields such as education, science and healthcare. Yet the rapid speed of progress has made it difficult to adequately prepare for, or even understand, how this technology can potentially be misused. Indeed, advanced AI assistants may transform existing threats or create new classes of threats altogether"
-
30.04.00 · Risk Category
Prohibiting the misuse by malicious attackers to do harm
-
49.01.00 · Risk Category
"As general- purpose AI covers a broad set of knowledge areas, it can be repurposed for malicious ends, potentially causing widespread harm. This section discusses some of the major risks of malicious use, but there are others and new risks may continue to emerge. While the risks discussed in this section range widely in terms of how well- evidenced they are, and in some cases, there is evidence suggesting that they may currently not be serious risks at all, we include them to provide a comprehensive overview of the malicious use risks associated with general- purpose AI systems."
-
52.02.00 · Risk Category
"However, even if a model is entirely trustworthy and reliable, Misuse or Systemic Risks remain. General purpose AI models may present significant risks to society if this technology is misused by malicious actors to produce harmful outcomes. Misuse Risks span across Cyber Crime, Biosecurity Threats and Politically Motivated Misuse."
-
60.01.00 · Risk Category
-
-
62.15.03 · Risk Sub-Category
Fine-tuning related (Ease of reconfiguring GPAI models)
"GPAI models are often easily reconfigured for various use cases or have competencies beyond the intended use [78, 225]. They can be performed either by changing the weights of the model (e.g., fine-tuning) or by modifying only the model inputs (e.g., prompt engineering, jailbreaking, retrieval-augmented generation). Reconfiguration can be intentional (with the help of adversarial inputs) or unintentional (from unanticipated inputs to the model)."
-
62.29.01 · Risk Sub-Category
High-impact misuses and abuses beyond original purpose
"Since general-purpose AI systems have a large repertoire of capabilities, mali- cious actors such as foreign actors can use such systems to cause large damage if they gain unrestricted or unmonitored access to those AI systems."
-
"Access to dual-use technologies can become easier because of GPAI model pro- liferation (in particular, open-source or open-weights models). Non-experts can use such dual-use-capable systems at a minimal cost [194, 100]. Improved model capabilities also contribute to dual-use risks posed by malicious actors. For example, an open-source base model for generating high quality sequence data can be modified to generate candidate protein sequences for toxin synthesis [29]."
-
"Generative AI models might be used intentionally to generate hateful, abusive, and profane (HAP) or obscene content."
Informational only, not legal advice. Verify every claim against the linked official sources and consult qualified counsel before acting.