MIT AI Risk Repository
Browse AI risks
190 risk entries extracted from 74 frameworks, coded by domain, subdomain, causal entity, intent and timing. Filter, then export the current selection with its licence and citation attached.
-
"A type of adversarial attack where an adversary or malicious insider injects intentionally corrupted, false, misleading, or incorrect samples into the training or fine-tuning datasets."
-
73.08.00 · Risk Category
"The previous section explored jailbreaks and other forms of adversarial prompts as ways to elicit harmful capabilities acquired during pretraining. These methods make no assumptions about the training data. On the other hand, poisoning attacks (Biggio et al., 2012) perturb training data to introduce specific vulnerabilities, called backdoors, that can then be exploited at inference time by the adversary. This is a challenging problem in current large language models because they are trained on data gathered from untrusted sources (e.g. internet), which can easily be poisoned by an adversary (
-
"In the data collecting and pre-training phase, malicious adversaries can Jailbreak LLMs through poisoning their training data to make the model to output harmful content."
-
"However, there are still ones who can leave holes in the training dataset, making LLMs appear safe on average, but generate harmful content under other specific conditions. This kind of attack can be categorized as "backdoor attack". Evan et al. developed a backdoor model that behaves as expected when trained, but exhibits different and potentially harmful behavior when deployed [81]. The results show that these backdoor behaviors persist even after multiple security training techniques are applied."
-
74.02.04 · Risk Sub-Category
Jailbreak in LLM Malicious Use - White & Black Box Attacks
"In the fine-tuning and alignment phase, elaborately- designed instruction datasets can be utilized to fine-tune LLMs to drive them to perform undesirable behaviors, such as generating harmful information or content that violates ethical norms, and thus achieve a jailbreak. Based on the accessibility to the model parameters, we can categorize them into white-box and black-box attacks. For white-box attacks, we can jailbreak the model by modifying its parameter weights. In [107], Lermen et al. used LoRA to fine-tune the Llama2’s 7B, 13B, and 70B as well as Mixtral on AdvBench and RefusalBench d
-
"Another important source of hallucinations is the noise in training data, which introduces errors in the knowledge stored in model parameters [111]–[113]. Generally, the training data inherently harbors misinformation. When training on large-scale corpora, this issue becomes more serious because it is difficult to eliminate all the noise from the massive pre-training data."
-
In general, LLMs employ the Transformer architecture [32] and generate content in an autoregressive manner, where the prediction of the next token is conditioned on the previously generated token sequence. Such a scheme could accumulate errors [105]. Besides, during the decoding process, top-p sampling [28] and top-k sampling [27] are widely adopted to enhance the diversity of the generated content. Nevertheless, these sampling strategies can introduce “randomness” [113], [136], thereby increasing the potential of hallucinations"
-
"people could build AIs that pursue dangerous goals’"
-
53.02.02 · Risk Sub-Category
Dangerous capabilities in AI systems
Acquisition of a goal to harm society
"cases of AI systems being given the outright goal of harming humanity (ChaosGPT);"
-
37.01.00 · Risk Category
"ethical concerns regarding how AI is designed and who designs it"
-
"Researchers in deep learning and those with greater research impact are more likely to migrate to industry, raising concerns about the “privatization of AI knowledge” [278]. Specically, if the most sophisticated AI approaches become proprietary and are used only within private research labs, then it will be impossible for universities to teach them, let alone contribute to leading research."
-
"Two key ethical concerns in the use of crowdwork for generative AI systems are: crowdworkers are frequently subject to working conditions that are taxing and debilitative to both physical and mental health, and there is a widespread deficit in documenting the role crowdworkers play in AI development. This contributes to a lack of transparency and explainability in resulting model outputs. Manual review is necessary to limit the harmful outputs of AI systems, including generative AI systems. A common harmful practice is to intentionally employ crowdworkers with few labor protections, often tak
-
18.06.05 · Risk Sub-Category
Socioeconomic and environmental harms
Exploitative data sourcing and enrichment
"Perpetuating exploitative labour practices to build AI systems (sourcing, user testing)"
-
"Without training data, ML cannot take place. Much of this data comes from paid clickwork (also called “platform work” [170] or “microwork” [558]), unpaid crowdsourcing, and unpaid user behavior capture. Clickworkers, mainly in the global south, perform repetitive data-labeling tasks for use in the training of ML models [558]. The market value of such annotations “is projected to reach $13.7 billion by 2030” [228] and the annotation industry is widely reported to have little concern for workers’ rights. Besides welfare and rights, the invisibility of this contribution arguably contributes to a
-
61.02.25 · Risk Sub-Category
Sources of systemic risks from general-purpose AI
Exploitation in AI development
"Outsourcing tasks like data labeling to low-income countries can perpetuate inequality."
-
"When workers who train AI models such as ghost workers are not provided with adequate working conditions, fair compensation, and good health care benefits that also include mental health."
-
"Use/misuse of labour to help train, develop, manage or optimise a technology system or set of systems, including under-paid and/or offshore"
-
05.17.00 · Risk Category
The emergence of generative AI raises issues regarding disruptions to existing copyright norms. Frequently discussed in the literature are violations of copyright and intellectual property rights stemming from the unauthorized collection of text or image training data. Another concern relates to generative models memorizing or plagiarizing copyrighted content. Additionally, there are open questions and debates around the copyright or ownership of model outputs, the protection of creative prompts, and the general blurring of traditional concepts of authorship.
-
31.05.00 · Risk Category
"The extent and effectiveness of legal protections for intellectual property have been thrown into question with the rise of generative AI. Generative AI trains itself on vast pools of data that often include IP-protected works.
-
47.03.03 · Risk Sub-Category
Copyright challenges (training models using copyrighted output)
"Generative AI companies are regularly accused of violating copyright law by training AI models on copyrighted works without gaining permission or paying compensation to the copyright owners. In fact, a substantial number of copyrighted documents and books have been incorporated into the training datasets of generative AI models."
-
56.07.00 · Risk Category
"There are also issues around intellectual property rights for content in training datasets"
-
08.03.00 · Risk Category
"The risks associated with the race to develop the first AGI, including the development of poor quality and unsafe AGI, and heightened political and control issues."
-
45.01.13 · Risk Sub-Category
Risks from AI systems (Risks of supply chain security)
"The AI industry relies on a highly globalized supply chain. However, certain countries may use unilateral coercive measures, such as technology barriers and export restrictions, to create development obstacles and maliciously disrupt the global AI supply chain. This can lead to significant risks of supply disruptions for chips, software, and tools."
-
"AI development may itself become a new flash point for conflicts—causing more conflict to occur— especially conflicts over AI-relevant resources (such as data centres, semiconductor manufacturing facilities and raw materials)."
-
61.02.17 · Risk Sub-Category
Sources of systemic risks from general-purpose AI
Dangerous development races
"Competitive pressures could lead to the neglect of safety measures in AI development."
-
62.29.03a · Additional evidence
Competitive pressures in GPAI product release
—
-
08.05.00 · Risk Category
"The capabilities of current risk management and legal processes in the context of the development of an AGI."
-
24.01.02 · Risk Sub-Category
Difficult to develop metrics for evaluating benefits or harms caused by AI assistants
"Another difficulty facing AI assistant systems is that it is challenging to develop metrics for evaluating particular aspects of benefits or harms caused by the assistant – especially in a sufficiently expansive sense, which could involve much of society (see Chapter 19). Having these metrics is useful both for assessing the risk of harm from the system and for using the metric as a training signal."
-
62.16.02 · Risk Sub-Category
General Evaluations (Limited coverage of capabilities evaluations)
"GPAI model developers might run capabilities evaluations to determine whether it has dangerous or dual-use capabilities, and then decide whether it is safe to deploy. Such capabilities evaluations can fail to demonstrate all the capabilities of a model. For example, evaluations may miss certain capabilities that are difficult to assess, prohibitively costly to verify, or obscured by the model’s tendency to refuse responses due to safety training, even if it possesses some of these capabilities."
-
62.16.06 · Risk Sub-Category
General Evaluations (Biased evaluations of encoded human values)
"Encoded human values in AI models that are easier to evaluate might be preferred for inclusion in evaluations over those that are more difficult to measure [13]. This might come at the expense of more desirable but harder-to-quantify values. This bias can lead to an imbalance, where easier-to-measure values dominate the evaluation process, while other important values are underrepresented."
-
62.16.08 · Risk Sub-Category
Benchmarking (Benchmark leakage or data contamination)
"Benchmark leakage [235, 224, 221, 161] can happen when an AI model is trained or fine-tuned with evaluation-related data. This can lead to an unreliable model evaluation, especially if the data contains question-answer pairs from bench- marks."
-
"This type of contamination [170] occurs when the raw and unlabeled data of a benchmark is used as part of the training set. Such data may not be properly formatted and may contain noise, especially if the contamination happens before the data is pre-processed into the benchmark. If this contamination occurs, it could cast doubt on the few-shot and zero-shot performance of the model on that benchmark."
-
"Models that have been trained on data encoded in multiple languages, such as LLMs trained on web-crawled data, may contain contamination that is obscured by translation [226]. The most basic form of this is when a benchmark is trans- lated to another language and then fed to the model as training data. The fact that the benchmark is translated before becoming training data can obscure the contamination from detection methods, giving false assurance that the model has generalized on the capabilities that the benchmark tests for."
-
"Guideline contamination refers to scenarios where instructions for the collec- tion, annotation, or use of the dataset are exposed to the model [170]. These instructions may contain explicit data-label pairs that can improve the model’s capabilities for the task."
-
"Annotation contamination refers to scenarios where the model is exposed to the benchmark labels during training [170]. This type of contamination can make the model learn the acceptable distribution of outputs. Combining this with raw data contamination of the test split, any evaluation made with the benchmark is invalidated because the entire test split is essentially leaked to the model."
-
62.16.14 · Risk Sub-Category
Benchmark Inaccuracy (Benchmarks may not accurately evaluate capabilities)
"Benchmarks of AI systems can both underestimate and overestimate the capa- bilities of those AI systems. Underestimates can happen if an evaluation is not comprehensive enough, if the benchmark is saturated by existing models, or if the capabilities in question depend on a complicated setup, such as realistic computer programming tasks. Overestimates of capabilities can occur if an AI system is trained or fine-tuned on the contents of the benchmark, leading to overfitting."
-
"Benchmark saturation refers to benchmarks reaching their evaluation ceiling. The tendency towards benchmark saturation has been demonstrated in various benchmarks [19]. When benchmarks reach or are close to saturation, they stop being effective measures for new models, as more nuanced capability gains might not be detected."
-
62.16.16 · Risk Sub-Category
Benchmark Limitations (Insufficient benchmarks for AI safety evaluation)
"Benchmarks dedicated to measuring the performance of AI systems (e.g., on programming or math tasks) are more well-developed than those for assessing safety and harms in AI systems [234]. This gap can lead to AI systems excelling in specific tasks while exhibiting harmful behaviors that go undetected. More safety-related evaluation datasets can help in identifying previously overlooked undesirable model behaviors."
-
62.16.17 · Risk Sub-Category
Benchmark Limitations (Underestimating capabilities that are not covered by benchmarks)
"A lack of test coverage by benchmarks on specific abilities of a model can obscure the model’s capabilities from both the developer and the user [160]. This can lead to a false sense of safety and trust due to a lack of understanding of the model’s limitations."
-
62.17.00 · Risk Category
-
-
62.17.01 · Risk Sub-Category
Conflicts of interest in auditor selection
"Conflicts of interest can arise if there is no independence in the auditor selection process or if the auditors are closely associated with the developer [123, 157]. In such cases, the conflict of interest can appear even if third-party evaluators are involved. In the case of external auditing, the potential candidates might be selected from a narrow group of auditors, or have conflicting financial incentives for whether to report model shortcomings publicly."
-
"Auditors may not be able to address all of the specific safety, performance, or validation needs. Reports of passing audits may be more inclusive than can be justified due to a lack of knowledge of specific risks and how they can be tested, or a lack of capacity to perform sufficiently rigorous testing."
-
"Auditors may not publicly disclose risks they find, may be required to not pub- licize shortcomings, or may not receive sufficient cooperation from the relevant internal parties."
-
"Without accurate documentation on how a model's data was collected, curated, and used to train a model, it might be harder to satisfactorily explain the behavior of the model with respect to the data."
-
"Data provenance refers to tracing history of data, which includes its ownership, origin, and transformations. Without standardized and established methods for verifying where the data came from, there are no guarantees that the data is the same as the original source and has the correct usage terms."
-
"Testing is unrepresentative when the test inputs are mismatched with the inputs that are expected during deployment."
-
"Since foundation models can be used for many purposes, a model’s intended use is important for defining the relevant risks of that model. As the use changes, the relevant risks might correspondingly change."
-
"Lack of data transparency is due to insufficient documentation of training or tuning dataset details. "
-
"AI model risks are socio-technical, so their testing needs input from a broad set of disciplines and diverse testing practices."
-
39.02.00 · Risk Category
Some learning algorithms, including deep learning, utilize iterative learning processes [23]. This approach results in high energy consumption.
Informational only, not legal advice. Verify every claim against the linked official sources and consult qualified counsel before acting.