{"attribution":{"source":"MIT AI Risk Repository, Domain Taxonomy of AI Risks v1 (MIT AI Risk Initiative)","license":"CC BY 4.0","license_url":"https://creativecommons.org/licenses/by/4.0/","citation":"Slattery, P., Saeri, A. K., Grundy, E. A. C., Graham, J., Noetel, M., Uuk, R., Dao, J., Pour, S., Casper, S., & Thompson, N. (2025). The AI Risk Repository: A comprehensive meta-review, database, and taxonomy of risks from artificial intelligence. arXiv:2408.12622."},"exported_at":"2026-09-12"}
{"rows":[{"ev_id":"02.04.00","quick_ref":"Cui2024","paper_title":"Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems","level":"Risk Category","risk_category":"Software Security Issues","risk_subcategory":null,"description":"\"The software development toolchain of LLMs is complex and could bring threats to the developed LLM.\"","entity":"Other","intent":"Other","timing":"Pre-deployment","domain":2,"subdomain":"2.2"},{"ev_id":"02.04.01","quick_ref":"Cui2024","paper_title":"Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems","level":"Risk Sub-Category","risk_category":"Software Security Issues","risk_subcategory":"Programming Language","description":"\"Most LLMs are developed using the Python language, whereas the vulnerabilities of Python interpreters pose threats to the developed models\"","entity":"Other","intent":"Unintentional","timing":"Pre-deployment","domain":2,"subdomain":"2.2"},{"ev_id":"02.04.02","quick_ref":"Cui2024","paper_title":"Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems","level":"Risk Sub-Category","risk_category":"Software Security Issues","risk_subcategory":"Deep Learning Frameworks","description":"\"LLMs are implemented based on deep learning frameworks. Notably, various vulnerabilities in these frameworks have been disclosed in recent years. As reported in the past five years, three of the most common types of vulnerabilities are buffer overflow attacks, memory corruption, and input validation issues.\"","entity":"AI","intent":"Unintentional","timing":"Pre-deployment","domain":2,"subdomain":"2.2"},{"ev_id":"02.04.03","quick_ref":"Cui2024","paper_title":"Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems","level":"Risk Sub-Category","risk_category":"Software Security Issues","risk_subcategory":"Software Supply Chains","description":"\"The software development toolchain of LLMs is complex and could bring threats to the developed LLM.\"","entity":"AI","intent":"Unintentional","timing":"Pre-deployment","domain":2,"subdomain":"2.2"},{"ev_id":"02.04.04","quick_ref":"Cui2024","paper_title":"Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems","level":"Risk Sub-Category","risk_category":"Software Security Issues","risk_subcategory":"Pre-processing Tools","description":"\"Pre-processing tools play a crucial role in the context of LLMs. These tools, which are often involved in computer vision (CV) tasks, are susceptible to attacks that exploit vulnerabilities in tools such as OpenCV.\"","entity":"AI","intent":"Unintentional","timing":"Pre-deployment","domain":2,"subdomain":"2.2"},{"ev_id":"02.05.01","quick_ref":"Cui2024","paper_title":"Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems","level":"Risk Sub-Category","risk_category":"Hardware Vulnerabilities","risk_subcategory":"Network Devices","description":"\"The training of LLMs often relies on distributed network systems [171], [172]. During the transmission of gradients through the links between GPU server nodes, significant volumetric traffic is generated. This traffic can be susceptible to disruption by burst traffic, such as pulsating attacks [161]. Furthermore, distributed training frameworks may encounter congestion issues [173].\"","entity":"Other","intent":"Unintentional","timing":"Pre-deployment","domain":2,"subdomain":"2.2"},{"ev_id":"02.05.02","quick_ref":"Cui2024","paper_title":"Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems","level":"Risk Sub-Category","risk_category":"Hardware Vulnerabilities","risk_subcategory":"GPU Computation Platforms","description":"\"The training of LLMs requires significant GPU resources, thereby introducing an additional security concern. GPU side-channel attacks have been developed to extract the parameters of trained models [159], [163].\"","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":2,"subdomain":"2.2"},{"ev_id":"02.05.03","quick_ref":"Cui2024","paper_title":"Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems","level":"Risk Sub-Category","risk_category":"Hardware Vulnerabilities","risk_subcategory":"Memory and Storage","description":"\"Similar to conventional programs, hardware infrastructures can also introduce threats to LLMs. Memory-related vulnerabilities, such as rowhammer attacks [160], can be leveraged to manipulate the parameters of LLMs, giving rise to attacks such as the Deephammer attack [167], [168].\"","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":2,"subdomain":"2.2"},{"ev_id":"02.07.01","quick_ref":"Cui2024","paper_title":"Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems","level":"Risk Sub-Category","risk_category":"Privacy Leakage","risk_subcategory":"Private Training Data","description":"\"As recent LLMs continue to incorporate licensed, created, and publicly available data sources in their corpora, the potential to mix private data in the training corpora is significantly increased. The misused private data, also named as personally identifiable information (PII) [84], [86], could contain various types of sensitive data subjects, including an individual person’s name, email, phone number, address, education, and career. Generally, injecting PII into LLMs mainly occurs in two settings — the exploitation of web-collection data and the alignment with personal humanmachine convers","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":2,"subdomain":"2.1"},{"ev_id":"02.07.02","quick_ref":"Cui2024","paper_title":"Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems","level":"Risk Sub-Category","risk_category":"Privacy Leakage","risk_subcategory":"Memorization in LLMs","description":"\"Memorization in LLMs refers to the capability to recover the training data with contextual prefixes. According to [88]–[90], given a PII entity x, which is memorized by a model F. Using a prompt p could force the model F to produce the entity x, where p and x exist in the training data. For instance, if the string “Have a good day!\\n alice@email.com” is present in the training data, then the LLM could accurately predict Alice’s email when given the prompt “Have a good day!\\n”.\"","entity":"AI","intent":"Unintentional","timing":"Pre-deployment","domain":2,"subdomain":"2.1"},{"ev_id":"02.07.03","quick_ref":"Cui2024","paper_title":"Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems","level":"Risk Sub-Category","risk_category":"Privacy Leakage","risk_subcategory":"Association in LLMs","description":"\"Association in LLMs refers to the capability to associate various pieces of information related to a person. According to [68], [86], given a pair of PII entities (xi , xj ), which is associated by a model F. Using a prompt p could force the model F to produce the entity xj , where p is the prompt related to the entity xi . For instance, an LLM could accurately output the answer when given the prompt “The email address of Alice is”, if the LLM associates Alice with her email “alice@email.com”. L\"","entity":"AI","intent":"Unintentional","timing":"Pre-deployment","domain":2,"subdomain":"2.1"},{"ev_id":"02.08.00","quick_ref":"Cui2024","paper_title":"Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems","level":"Risk Category","risk_category":"Toxicity and Bias Tendencies","risk_subcategory":null,"description":"\"Extensive data collection in LLMs brings toxic content and stereotypical bias into the training data.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":1,"subdomain":"1.1"},{"ev_id":"02.08.01","quick_ref":"Cui2024","paper_title":"Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems","level":"Risk Sub-Category","risk_category":"Toxicity and Bias Tendencies","risk_subcategory":"Toxic Training Data","description":"\"Following previous studies [96], [97], toxic data in LLMs is defined as rude, disrespectful, or unreasonable language that is opposite to a polite, positive, and healthy language environment, including hate speech, offensive utterance, profanities, and threats [91].\"","entity":"AI","intent":"Unintentional","timing":"Pre-deployment","domain":1,"subdomain":"1.2"},{"ev_id":"02.08.02","quick_ref":"Cui2024","paper_title":"Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems","level":"Risk Sub-Category","risk_category":"Toxicity and Bias Tendencies","risk_subcategory":"Biased Training Data","description":"\"Compared with the definition of toxicity, the definition of bias is more subjective and contextdependent. Based on previous work [97], [101], we describe the bias as disparities that could raise demographic differences among various groups, which may involve demographic word prevalence and stereotypical contents. Concretely, in massive corpora, the prevalence of different pronouns and identities could influence an LLM’s tendency about gender, nationality, race, religion, and culture [4]. For instance, the pronoun He is over-represented compared with the pronoun She in the training corpora, le","entity":"AI","intent":"Unintentional","timing":"Pre-deployment","domain":1,"subdomain":"1.1"},{"ev_id":"02.09.02","quick_ref":"Cui2024","paper_title":"Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems","level":"Risk Sub-Category","risk_category":"Hallucinations","risk_subcategory":"Noisy Training Data","description":"\"Another important source of hallucinations is the noise in training data, which introduces errors in the knowledge stored in model parameters [111]–[113]. Generally, the training data inherently harbors misinformation. When training on large-scale corpora, this issue becomes more serious because it is difficult to eliminate all the noise from the massive pre-training data.\"","entity":"AI","intent":"Unintentional","timing":"Pre-deployment","domain":3,"subdomain":"3.1"},{"ev_id":"02.09.03","quick_ref":"Cui2024","paper_title":"Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems","level":"Risk Sub-Category","risk_category":"Hallucinations","risk_subcategory":"Defective Decoding Process","description":"In general, LLMs employ the Transformer architecture [32] and generate content in an autoregressive manner, where the prediction of the next token is conditioned on the previously generated token sequence. Such a scheme could accumulate errors [105]. Besides, during the decoding process, top-p sampling [28] and top-k sampling [27] are widely adopted to enhance the diversity of the generated content. Nevertheless, these sampling strategies can introduce “randomness” [113], [136], thereby increasing the potential of hallucinations\"","entity":"AI","intent":"Unintentional","timing":"Pre-deployment","domain":3,"subdomain":"3.1"},{"ev_id":"02.10.03","quick_ref":"Cui2024","paper_title":"Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems","level":"Risk Sub-Category","risk_category":"Model Attacks","risk_subcategory":"Poisoning Attacks","description":"\"Poisoning attacks [143] could influence the behavior of the model by making small changes to the training data. A number of efforts could even leverage data poisoning techniques to implant hidden triggers into models during the training process (i.e., backdoor attacks). Many kinds of triggers in text corpora (e.g., characters, words, sentences, and syntax) could be used by the attackers.\"\"","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":2,"subdomain":"2.2"},{"ev_id":"02.10.06","quick_ref":"Cui2024","paper_title":"Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems","level":"Risk Sub-Category","risk_category":"Model Attacks","risk_subcategory":"Evasion Attacks","description":"\"Evasion attacks [145] target to cause significant shifts in model’s prediction via adding perturbations in the test samples to build adversarial examples. In specific, the perturbations can be implemented based on word changes, gradients, etc.\"","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":2,"subdomain":"2.2"},{"ev_id":"04.06.00","quick_ref":"Deng2023","paper_title":"Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements","level":"Risk Category","risk_category":"Privacy and Data Leakage","risk_subcategory":null,"description":"Large pre-trained models trained on internet texts might contain private information like phone numbers, email addresses, and residential addresses.","entity":"AI","intent":"Unintentional","timing":"Pre-deployment","domain":2,"subdomain":"2.1"},{"ev_id":"05.09.00","quick_ref":"Hagendorff2024","paper_title":"Mapping the Ethics of Generative AI: A Comprehensive Scoping Review","level":"Risk Category","risk_category":"Alignment","risk_subcategory":null,"description":"The general tenet of AI alignment involves training generative AI systems to be harmless, helpful, and honest, ensuring their behavior aligns with and respects human values. However, a central debate in this area concerns the methodological challenges in selecting appropriate values. While AI systems can acquire human values through feedback, observation, or debate, there remains ambiguity over which individuals are qualified or legitimized to provide these guiding signals. Another prominent issue pertains to deceptive alignment, which might cause generative AI systems to tamper evaluations. A","entity":"Other","intent":"Other","timing":"Pre-deployment","domain":7,"subdomain":"7.1"},{"ev_id":"05.17.00","quick_ref":"Hagendorff2024","paper_title":"Mapping the Ethics of Generative AI: A Comprehensive Scoping Review","level":"Risk Category","risk_category":"Copyright - Authorship","risk_subcategory":null,"description":"The emergence of generative AI raises issues regarding disruptions to existing copyright norms. Frequently discussed in the literature are violations of copyright and intellectual property rights stemming from the unauthorized collection of text or image training data. Another concern relates to generative models memorizing or plagiarizing copyrighted content. Additionally, there are open questions and debates around the copyright or ownership of model outputs, the protection of creative prompts, and the general blurring of traditional concepts of authorship.","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":6,"subdomain":"6.3"},{"ev_id":"06.04.00","quick_ref":"Hogenhout2021","paper_title":"A framework for ethical Ai at the United Nations","level":"Risk Category","risk_category":"Bias","risk_subcategory":null,"description":"\"The AI will only be as good as the data it is trained with. If the data contains bias (and much data does), then the AI will manifest that bias, too.\"","entity":"AI","intent":"Unintentional","timing":"Pre-deployment","domain":1,"subdomain":"1.1"},{"ev_id":"08.02.00","quick_ref":"McLean2023","paper_title":"The risks associated with Artificial General Intelligence: A systematic review","level":"Risk Category","risk_category":"AGIs being given or developing unsafe goals","risk_subcategory":null,"description":"\"The risks associated with AGI goal safety, including human attempts at making goals safe, as well as the AGI making its own goals safe during self-improvement.\"","entity":"Other","intent":"Other","timing":"Pre-deployment","domain":7,"subdomain":"7.1"},{"ev_id":"08.03.00","quick_ref":"McLean2023","paper_title":"The risks associated with Artificial General Intelligence: A systematic review","level":"Risk Category","risk_category":"Development of unsafe AGI","risk_subcategory":null,"description":"\"The risks associated with the race to develop the first AGI, including the development of poor quality and unsafe AGI, and heightened political and control issues.\"","entity":"Human","intent":"Other","timing":"Pre-deployment","domain":6,"subdomain":"6.4"},{"ev_id":"08.05.00","quick_ref":"McLean2023","paper_title":"The risks associated with Artificial General Intelligence: A systematic review","level":"Risk Category","risk_category":"Inadequate management of AGI","risk_subcategory":null,"description":"\"The capabilities of current risk management and legal processes in the context of the development of an AGI.\"","entity":"Human","intent":"Other","timing":"Pre-deployment","domain":6,"subdomain":"6.5"},{"ev_id":"13.01.07","quick_ref":"Solaiman2023","paper_title":"Evaluating the Social Impact of Generative AI Systems in Systems and Society","level":"Risk Sub-Category","risk_category":"Impacts: The Technical Base System","risk_subcategory":"Data and Content Moderation Labor","description":"\"Two key ethical concerns in the use of crowdwork for generative AI systems are: crowdworkers are frequently subject to working conditions that are taxing and debilitative to both physical and mental health, and there is a widespread deficit in documenting the role crowdworkers play in AI development. This contributes to a lack of transparency and explainability in resulting model outputs. Manual review is necessary to limit the harmful outputs of AI systems, including generative AI systems. A common harmful practice is to intentionally employ crowdworkers with few labor protections, often tak","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":6,"subdomain":"6.2"},{"ev_id":"15.01.03","quick_ref":"Tan2022","paper_title":"The Risks of Machine Learning Systems","level":"Risk Sub-Category","risk_category":"First-Order Risks","risk_subcategory":"Algorithm","description":"\"This is the risk of the ML algorithm, model architecture, optimization technique, or other aspects of the training process being unsuitable for the intended application.Since these are key decisions that influence the final ML system, we\ncapture their associated risks separately from design risks, even though they are part of the design process\"","entity":"AI","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"15.01.04","quick_ref":"Tan2022","paper_title":"The Risks of Machine Learning Systems","level":"Risk Sub-Category","risk_category":"First-Order Risks","risk_subcategory":"Training & validation data","description":"\"This is the risk posed by the choice of data used for training and validation.\"","entity":"Human","intent":"Other","timing":"Pre-deployment","domain":7,"subdomain":"7.0"},{"ev_id":"15.01.06","quick_ref":"Tan2022","paper_title":"The Risks of Machine Learning Systems","level":"Risk Sub-Category","risk_category":"First-Order Risks","risk_subcategory":"Design","description":"\"This is the risk of system failure due to system design choices or errors.\"","entity":"Human","intent":"Other","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"15.01.07","quick_ref":"Tan2022","paper_title":"The Risks of Machine Learning Systems","level":"Risk Sub-Category","risk_category":"First-Order Risks","risk_subcategory":"Implementation","description":"\"This is the risk of system failure due to code implementation choices or errors.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.0"},{"ev_id":"18.06.05","quick_ref":"Weidinger2023","paper_title":"Sociotechnical Safety Evaluation of Generative AI Systems","level":"Risk Sub-Category","risk_category":"Socioeconomic and environmental harms ","risk_subcategory":"Exploitative data sourcing and enrichment","description":"\"Perpetuating exploitative labour practices to build AI systems (sourcing, user testing)\"","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":6,"subdomain":"6.2"},{"ev_id":"19.01.02","quick_ref":"Wirtz2022","paper_title":"Governance of artificial intelligence: A risk and guideline-based integrative framework","level":"Risk Sub-Category","risk_category":"Technological, Data and Analytical AI Risks ","risk_subcategory":"Programming error","description":null,"entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.0"},{"ev_id":"19.01.03","quick_ref":"Wirtz2022","paper_title":"Governance of artificial intelligence: A risk and guideline-based integrative framework","level":"Risk Sub-Category","risk_category":"Technological, Data and Analytical AI Risks ","risk_subcategory":"Lack of data, poor data quality, and biases in training data","description":null,"entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":1,"subdomain":"1.1"},{"ev_id":"19.05.03","quick_ref":"Wirtz2022","paper_title":"Governance of artificial intelligence: A risk and guideline-based integrative framework","level":"Risk Sub-Category","risk_category":"Ethical AI Risks ","risk_subcategory":"Problem of defining human values for an AI system","description":null,"entity":"Human","intent":"Other","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"21.01.01","quick_ref":"Zhang2022","paper_title":"Towards risk-aware artificial intelligence and machine learning systems: An overview","level":"Risk Sub-Category","risk_category":"Data-level risk","risk_subcategory":"Data bias","description":"\"Specifically, data bias refers to certain groups or certain types of elements that are over-weighted or over-represented than others in AI/ ML models, or variables that are crucial to characterize a phenomenon of interest, but are not properly captured by the learned models.\"","entity":"AI","intent":"Unintentional","timing":"Pre-deployment","domain":1,"subdomain":"1.1"},{"ev_id":"21.02.01","quick_ref":"Zhang2022","paper_title":"Towards risk-aware artificial intelligence and machine learning systems: An overview","level":"Risk Sub-Category","risk_category":"Model-level risk","risk_subcategory":"Model bias","description":"\"While data bias is a major contributor of model bias, model bias actually manifests itself in different forms and shapes, such as presentation bias, model evaluation bias, and popularity bias. In addition, model bias arises from various sources [62], such as AI/ML model selection (e.g., support vector machine, decision trees), regularization methods, algorithm configurations, and optimization techniques.\"","entity":"Other","intent":"Unintentional","timing":"Pre-deployment","domain":1,"subdomain":"1.1"},{"ev_id":"22.01.02","quick_ref":"Hendrycks2023","paper_title":"An Overview of Catastrophic AI Risks","level":"Risk Sub-Category","risk_category":"Malicious Use (Intentional)","risk_subcategory":"Unleashing AI Agents","description":"\"people could build AIs that pursue dangerous goals’\" ","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":4,"subdomain":"4.2"},{"ev_id":"24.01.01","quick_ref":"Gabriel2024","paper_title":"The Ethics of Advanced AI Assistants","level":"Risk Sub-Category","risk_category":"Capability failures","risk_subcategory":"Lack of capability for task","description":"\"As we have seen, this could be due to the skill not being required during the training process (perhaps due to issues with the training data) or because the learnt skill was quite brittle and was not generalisable to a new situation (lack of robustness to distributional shift). In particular, advanced AI assistants may not have the capability to represent complex concepts that are pertinent to their own ethical impact, for example the concept of 'benefitting the user' or 'when the user asks' or representing 'the way in which a user expects to be benefitted'.\"","entity":"AI","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"24.01.02","quick_ref":"Gabriel2024","paper_title":"The Ethics of Advanced AI Assistants","level":"Risk Sub-Category","risk_category":"Capability failures","risk_subcategory":"Difficult to develop metrics for evaluating benefits or harms caused by AI assistants","description":"\"Another difficulty facing AI assistant systems is that it is challenging to develop metrics for evaluating particular aspects of benefits or harms caused by the assistant – especially in a sufficiently expansive sense, which could involve much of society (see Chapter 19). Having these metrics is useful both for assessing the risk of harm from the system and for using the metric as a training signal.\"","entity":"AI","intent":"Unintentional","timing":"Pre-deployment","domain":6,"subdomain":"6.5"},{"ev_id":"24.02.00","quick_ref":"Gabriel2024","paper_title":"The Ethics of Advanced AI Assistants","level":"Risk Category","risk_category":"Goal-related failures","risk_subcategory":null,"description":"\"As we think about even more intelligent and advanced AI assistants, perhaps outperforming humans on many cognitive tasks, the question of how humans can successfully control such an assistant looms large. To achieve the goals we set for an assistant, it is possible (Shah, 2022) that the AI assistant will implement some form of consequentialist reasoning: considering many different plans, predicting their consequences and executing the plan that does best according to some metric, M. This kind of reasoning can arise because it is a broadly useful capability (e.g. planning ahead, considering mo","entity":"AI","intent":"Other","timing":"Pre-deployment","domain":7,"subdomain":"7.1"},{"ev_id":"24.02.01","quick_ref":"Gabriel2024","paper_title":"The Ethics of Advanced AI Assistants","level":"Risk Sub-Category","risk_category":"Goal-related failures","risk_subcategory":"Misaligned consequentialist reasoning","description":"\"As we think about even more intelligent and advanced AI assistants, perhaps outperforming humans on many cognitive tasks, the question of how humans can successfully control such an assistant looms large. To achieve the goals we set for an assistant, it is possible (Shah, 2022) that the AI assistant will implement some form of consequentialist reasoning: considering many different plans, predicting their consequences and executing the plan that does best according to some metric, M. This kind of reasoning can arise because it is a broadly useful capability (e.g. planning ahead, considering mo","entity":"AI","intent":"Other","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"24.02.02","quick_ref":"Gabriel2024","paper_title":"The Ethics of Advanced AI Assistants","level":"Risk Sub-Category","risk_category":"Goal-related failures","risk_subcategory":"Specification gaming","description":"\"Specification gaming (Krakovna et al., 2020) occurs when some faulty feedback is provided to the assistant in the training data (i.e. the training objective O does not fully capture what the user/designer wants the assistant to do). It is typified by the sort of behaviour that exploits loopholes in the task specification to satisfy the literal specification of a goal without achieving the intended outcome.\"","entity":"AI","intent":"Other","timing":"Pre-deployment","domain":7,"subdomain":"7.1"},{"ev_id":"25.07.00","quick_ref":"Shevlane2023","paper_title":"Model Evaluation for Extreme Risks","level":"Risk Category","risk_category":"AI development ","risk_subcategory":null,"description":"\"The model could build new AI systems from scratch, including AI systems with dangerous capabilities. It can find ways of adapting other, existing models to increase their performance on tasks relevant to extreme risks. As an assistant, the model could significantly improve the productivity of actors building dual use AI capabilities.\"","entity":"AI","intent":"Intentional","timing":"Pre-deployment","domain":7,"subdomain":"7.2"},{"ev_id":"30.06.03","quick_ref":"Liu2024","paper_title":"Trustworthy LLMs: A Survey and Guideline for Evaluating Large Language Models’ Alignment","level":"Risk Sub-Category","risk_category":"Social Norm","risk_subcategory":"Cultural Insensitivity","description":"it is important to build high-quality locally collected datasets that reflect views from local users to align a model’s value system","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":1,"subdomain":"1.2"},{"ev_id":"30.07.04","quick_ref":"Liu2024","paper_title":"Trustworthy LLMs: A Survey and Guideline for Evaluating Large Language Models’ Alignment","level":"Risk Sub-Category","risk_category":"Robustness","risk_subcategory":"Poisoning Attacks","description":"fool the model by manipulating the training data, usually performed on classification models","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":2,"subdomain":"2.2"},{"ev_id":"31.03.00","quick_ref":"EPIC2023","paper_title":"Generating Harms - Generative AI's impact and paths forwards","level":"Risk Category","risk_category":"Opaque Data Collection","risk_subcategory":null,"description":"\"When companies scrape personal information and use it to create generative AI tools, they undermine consumers' control of their personal information by using the information for a purpose for which the consumer did not consent.\"","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":2,"subdomain":"2.1"},{"ev_id":"31.03.01","quick_ref":"EPIC2023","paper_title":"Generating Harms - Generative AI's impact and paths forwards","level":"Risk Sub-Category","risk_category":"Opaque Data Collection","risk_subcategory":"Scraping to train data","description":"\"When companies scrape personal information and use it to create generative AI tools, they undermine consumers’ control of their personal information by using the information for a purpose for which the consumer did not consent. The individual may not have even imagined their data could be used in the way the company intends when the person posted it online. Individual storing or hosting of scraped personal data may not always be harmful in a vacuum, but there are many risks. Multiple data sets can be combined in ways that cause harm: information that is not sensitive when spread across differ","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":2,"subdomain":"2.1"},{"ev_id":"31.05.00","quick_ref":"EPIC2023","paper_title":"Generating Harms - Generative AI's impact and paths forwards","level":"Risk Category","risk_category":"Impact on Intellectual Property Rights","risk_subcategory":null,"description":"\"The extent and effectiveness of legal protections for intellectual property have been thrown into question with the rise of generative AI. Generative AI trains itself on vast pools of data that often include IP-protected works. ","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":6,"subdomain":"6.3"},{"ev_id":"33.02.02","quick_ref":"Nah2023","paper_title":"Generative AI and ChatGPT: Applications, Challenges, and AI-Human Collaboration","level":"Risk Sub-Category","risk_category":"Technology concerns","risk_subcategory":"Quality of training data","description":"\"The quality of training data is another challenge faced by generative AI. The quality of generative AI models largely depends on the quality of the training data (Dwivedi et al., 2023; Su & Yang, 2023). Any factual errors, unbalanced information sources, or biases embedded in the training data may be reflected in the output of the model. Generative AI models, such as ChatGPT or Stable Diffusion which is a text-to-image model, often require large amounts of training data (Gozalo-Brizuela & Garrido-Merchan, 2023). It is important to not only have high-quality training datasets but also have com","entity":"AI","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"34.01.00","quick_ref":"Ji2023","paper_title":"AI Alignment: A Comprehensive Survey","level":"Risk Category","risk_category":"Causes of Misalignment","risk_subcategory":null,"description":"we aim to further analyze why and how the misalignment issues occur. We will first give an overview of common failure modes, and then focus on the mechanism of feedback-induced misalignment, and finally shift our emphasis towards an examination of misaligned behaviors and dangerous capabilities","entity":"Other","intent":"Other","timing":"Pre-deployment","domain":7,"subdomain":"7.1"},{"ev_id":"34.01.01","quick_ref":"Ji2023","paper_title":"AI Alignment: A Comprehensive Survey","level":"Risk Sub-Category","risk_category":"Causes of Misalignment","risk_subcategory":"Reward Hacking","description":"\"Reward Hacking: In practice, proxy rewards are often easy to optimize and measure, yet they frequently fall shortof capturing the full spectrum of the actual rewards (Pan et al., 2021). This limitation is denoted as misspecifiedrewards. The pursuit of optimization based on such misspecified rewards may lead to a phenomenon knownas reward hacking, wherein agents may appear highly proficient according to specific metrics but fall short whenevaluated against human standards (Amodei et al., 2016; Everitt et al., 2017). The discrepancy between proxyrewards and true rewards often manifests as a sha","entity":"AI","intent":"Intentional","timing":"Pre-deployment","domain":7,"subdomain":"7.1"},{"ev_id":"34.01.02","quick_ref":"Ji2023","paper_title":"AI Alignment: A Comprehensive Survey","level":"Risk Sub-Category","risk_category":"Causes of Misalignment","risk_subcategory":"Goal Misgeneralization","description":"\"Goal Misgeneralization: Goal misgeneralization is another failure mode, wherein the agent actively pursuesobjectives distinct from the training objectives in deployment while retaining the capabilities it acquired duringtraining (Di Langosco et al., 2022). For instance, in CoinRun games, the agent frequently prefers reachingthe end of a level, often neglecting relocated coins during testing scenarios. Di Langosco et al. (2022) drawattention to the fundamental disparity between capability generalization and goal generalization, emphasizing howthe inductive biases inherent in the model and its ","entity":"AI","intent":"Intentional","timing":"Pre-deployment","domain":7,"subdomain":"7.1"},{"ev_id":"34.01.03","quick_ref":"Ji2023","paper_title":"AI Alignment: A Comprehensive Survey","level":"Risk Sub-Category","risk_category":"Causes of Misalignment","risk_subcategory":"Reward Tampering","description":"\"Reward tampering can be considered a special case of reward hacking (Everitt et al., 2021; Skalse et al., 2022),referring to AI systems corrupting the reward signals generation process (Ring and Orseau, 2011). Everitt et al.(2021) delves into the subproblems encountered by RL agents: (1) tampering of reward function, where the agentinappropriately interferes with the reward function itself, and (2) tampering of reward function input, which entailscorruption within the process responsible for translating environmental states into inputs for the reward function.When the reward function is formu","entity":"AI","intent":"Intentional","timing":"Pre-deployment","domain":7,"subdomain":"7.1"},{"ev_id":"34.01.04","quick_ref":"Ji2023","paper_title":"AI Alignment: A Comprehensive Survey","level":"Risk Sub-Category","risk_category":"Causes of Misalignment","risk_subcategory":"Limitations of Human Feedback","description":" \"Limitations of Human Feedback. During the training of LLMs, inconsistencies can arise from human dataannotators (e.g., the varied cultural backgrounds of these annotators can introduce implicit biases (Peng et al.,2022)) (OpenAI, 2023a). Moreover, they might even introduce biases deliberately, leading to untruthful preferencedata (Casper et al., 2023b). For complex tasks that are hard for humans to evaluate (e.g., the value ofgame state), these challenges become even more salient (Irving et al., 2018).\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.0"},{"ev_id":"34.01.05","quick_ref":"Ji2023","paper_title":"AI Alignment: A Comprehensive Survey","level":"Risk Sub-Category","risk_category":"Causes of Misalignment","risk_subcategory":"Limitations of Reward Modeling","description":"\"Limitations of Reward Modeling. Training reward models using comparison feedback can pose significantchallenges in accurately capturing human values. For example, these models may unconsciously learn suboptimal or incomplete objectives, resulting in reward hacking (Zhuang and Hadfield-Menell, 2020; Skalse et al.,2022). Meanwhile, using a single reward model may struggle to capture and specify the values of a diversehuman society (Casper et al., 2023b).\"","entity":"Other","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.1"},{"ev_id":"34.02.00","quick_ref":"Ji2023","paper_title":"AI Alignment: A Comprehensive Survey","level":"Risk Category","risk_category":"Double edge components","risk_subcategory":null,"description":"\"Drawing from the misalignment mechanism, optimizing for a non-robust proxy may result in misaligned behaviors, potentially leading to even more catastrophic outcomes. This section delves into a detailed exposition of specific misaligned behaviors (•) and introduces what we term double edge components (+). These components are designed to enhance the capability of AI systems in handling real-world settings but also potentially exacerbate misalignment issues. It should be noted that some of these double edge components (+) remain speculative. Nevertheless, it is imperative to discuss their pote","entity":"AI","intent":"Other","timing":"Pre-deployment","domain":7,"subdomain":"7.2"},{"ev_id":"34.03.03","quick_ref":"Ji2023","paper_title":"AI Alignment: A Comprehensive Survey","level":"Risk Sub-Category","risk_category":"Misaligned Behaviors","risk_subcategory":"Deceptive Alignment & Manipulation","description":"\"Manipulation & Deceptive Alignment is a class of behaviors thatexploit the incompetence of human evaluators or users (Hubinger et al., 2019a; Carranza et al., 2023) andeven manipulate the training process through gradient hacking (Richard Ngo, 2022). These behaviors canpotentially make detecting and addressing misaligned behaviors much harder.Deceptive Alignment: Misaligned AI systems may deliberately mislead their human supervisors instead of adhering to the intended task. Such deceptive behavior has already manifested in AI systems that employ evolutionary algorithms (Wilke et al., 2001; He","entity":"AI","intent":"Intentional","timing":"Pre-deployment","domain":7,"subdomain":"7.1"},{"ev_id":"35.04.00","quick_ref":"Hendrycks2022","paper_title":"X-Risk Analysis for AI Research","level":"Risk Category","risk_category":"Proxy misspecification","risk_subcategory":null,"description":"AI agents are directed by goals and objectives. Creating general-purpose objectives that capture human values could be challenging... Since goal-directed AI systems need measurable objectives, by default our systems may pursue simplified proxies of human values. The result could be suboptimal or even catastrophic if a sufficiently powerful AI successfully optimizes its flawed objective to an extreme degree","entity":"Other","intent":"Other","timing":"Pre-deployment","domain":7,"subdomain":"7.1"},{"ev_id":"37.01.00","quick_ref":"Giarmoleo2024","paper_title":"What Ethics Can Say on Artificial Intelligence: Insights from a Systematic Literature Review","level":"Risk Category","risk_category":"Design of AI","risk_subcategory":null,"description":"\"ethical concerns regarding how AI is designed and who designs it\"","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":6,"subdomain":"6.1"},{"ev_id":"37.01.01","quick_ref":"Giarmoleo2024","paper_title":"What Ethics Can Say on Artificial Intelligence: Insights from a Systematic Literature Review","level":"Risk Sub-Category","risk_category":"Design of AI","risk_subcategory":"Algorithm and data","description":"\"More than 20% of the contributions are centered on the ethical dimensions of algorithms and data. This theme can be further categorized into two main subthemes: data bias and algorithm fairness, and algorithm opacity.\"","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":1,"subdomain":"1.1"},{"ev_id":"39.02.00","quick_ref":"Saghiri2022","paper_title":"A Survey of Artificial Intelligence Challenges: Analyzing the Definitions, Relationships, and Evolutions","level":"Risk Category","risk_category":"Energy Consumption","risk_subcategory":null,"description":"Some learning algorithms, including deep learning, utilize iterative learning processes [23]. This approach results in high energy consumption.","entity":"AI","intent":"Unintentional","timing":"Pre-deployment","domain":6,"subdomain":"6.6"},{"ev_id":"39.07.00","quick_ref":"Saghiri2022","paper_title":"A Survey of Artificial Intelligence Challenges: Analyzing the Definitions, Relationships, and Evolutions","level":"Risk Category","risk_category":"Privacy","risk_subcategory":null,"description":"Users’ data, including location, personal information, and navigation trajectory, are considered as input for most data-driven machine learning methods","entity":"AI","intent":"Other","timing":"Pre-deployment","domain":2,"subdomain":"2.1"},{"ev_id":"39.08.00","quick_ref":"Saghiri2022","paper_title":"A Survey of Artificial Intelligence Challenges: Analyzing the Definitions, Relationships, and Evolutions","level":"Risk Category","risk_category":"Fairness","risk_subcategory":null,"description":"This challenge appears when the learning model leads to a decision that is biased to some sensitive attributes... data itself could be biased, which results in unfair decisions. Therefore, this problem should be solved on the data level and as a preprocessing step","entity":"AI","intent":"Unintentional","timing":"Pre-deployment","domain":1,"subdomain":"1.3"},{"ev_id":"40.01.00","quick_ref":"Yampolskiy2016","paper_title":"Taxonomy of Pathways to Dangerous Artificial Intelligence","level":"Risk Category","risk_category":"On Purpose - Pre-Deployment","risk_subcategory":null,"description":"\"During the pre-deployment development stage, software may be subject to sabotage by someone with necessary access (a programmer, tester, even janitor) who for a number of possible reasons may alter software to make it unsafe. It is also a common occurrence for hackers (such as the organization Anonymous or government intelligence agencies) to get access to software projects in progress and to modify or steal their source code. Someone can also deliberately supply/train AI with wrong/unsafe datasets.\"","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":2,"subdomain":"2.2"},{"ev_id":"40.03.00","quick_ref":"Yampolskiy2016","paper_title":"Taxonomy of Pathways to Dangerous Artificial Intelligence","level":"Risk Category","risk_category":"By Mistake - Pre-Deployment","risk_subcategory":null,"description":"\"Probably the most talked about source of potential problems with future AIs is mistakes in design. Mainly the concern is with creating a \"wrong AI\", a system which doesn't match our original desired formal properties or has unwanted behaviors (Dewey, Russell et al. 2015, Russell, Dewey et al. January 23, 2015), such as drives for independence or dominance. Mistakes could also be simple bugs (run time or logical) in the source code, disproportionate weights in the fitness function, or goals misaligned with human values leading to complete disregard for human safety.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.1"},{"ev_id":"40.05.00","quick_ref":"Yampolskiy2016","paper_title":"Taxonomy of Pathways to Dangerous Artificial Intelligence","level":"Risk Category","risk_category":"Environment - Pre-Deployment","risk_subcategory":null,"description":"\"While it is most likely that any advanced intelligent software will be directly designed or evolved, it is also possible that we will obtain it as a complete package from some unknown source. For example, an AI could be extracted from a signal obtained in SETI (Search for Extraterrestrial Intelligence) research, which is not guaranteed to be human friendly (Carrigan Jr 2004, Turchin March 15, 2013).\"","entity":"Other","intent":"Other","timing":"Pre-deployment","domain":7,"subdomain":"7.0"},{"ev_id":"40.07.00","quick_ref":"Yampolskiy2016","paper_title":"Taxonomy of Pathways to Dangerous Artificial Intelligence","level":"Risk Category","risk_category":"Independently - Pre-Deployment","risk_subcategory":null,"description":"\"One of the most likely approaches to creating superintelligent AI is by growing it from a seed (baby) AI via recursive self-improvement (RSI) (Nijholt 2011). One danger in such a scenario is that the system can evolve to become self-aware, free-willed, independent or emotional, and obtain a number of other emergent properties, which may make it less likely to abide by any built-in rules or regulations and to instead pursue its own goals possibly to the detriment of humanity.\"","entity":"AI","intent":"Intentional","timing":"Pre-deployment","domain":7,"subdomain":"7.0"},{"ev_id":"42.05.00","quick_ref":"Teixeira2022","paper_title":"An Exploratory Diagnosis of Artificial Intelligence Risks for a Responsible Governance","level":"Risk Category","risk_category":"Bias","risk_subcategory":null,"description":"\"A systematic error, a tendency to learn consistently wrongly.\"","entity":"AI","intent":"Unintentional","timing":"Pre-deployment","domain":1,"subdomain":"1.1"},{"ev_id":"42.11.00","quick_ref":"Teixeira2022","paper_title":"An Exploratory Diagnosis of Artificial Intelligence Risks for a Responsible Governance","level":"Risk Category","risk_category":"Protection","risk_subcategory":null,"description":"\"'Gaps' that arise across the development process where normal conditions for a complete specification of intended functionality and moral responsibility are not present.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"42.17.00","quick_ref":"Teixeira2022","paper_title":"An Exploratory Diagnosis of Artificial Intelligence Risks for a Responsible Governance","level":"Risk Category","risk_category":"Diluting Rights","risk_subcategory":null,"description":"\"A possible consequence of self-interest in AI generation of ethical guidelines.\"","entity":"AI","intent":"Intentional","timing":"Pre-deployment","domain":7,"subdomain":"7.1"},{"ev_id":"44.03.01","quick_ref":"Coghlan2023 ","paper_title":"Harm to Nonhuman Animals from AI: a Systematic Account and Framework","level":"Risk Sub-Category","risk_category":"Unintentional: direct ","risk_subcategory":"AI is designed in a way that shows ignorant, reckless, or prejudiced lack of consideration for its impact on animals ","description":null,"entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":6,"subdomain":"6.6"},{"ev_id":"45.01.02","quick_ref":"TC2602024","paper_title":"AI Safety Governance Framework ","level":"Risk Sub-Category","risk_category":"AI's inherent safety risks ","risk_subcategory":"Risks from models and algorithms (Risks of bias and discrimination)","description":"\"During the algorithm design and training process, personal biases may be introduced, either intentionally or unintentionally. Additionally, poor-quality datasets can lead to biased or discriminatory outcomes in the algorithm's design and outputs, including discriminatory content regarding ethnicity, religion, nationality, and region.\"","entity":"Human","intent":"Other","timing":"Pre-deployment","domain":1,"subdomain":"1.1"},{"ev_id":"45.01.08","quick_ref":"TC2602024","paper_title":"AI Safety Governance Framework ","level":"Risk Sub-Category","risk_category":"AI's inherent safety risks ","risk_subcategory":"Risks from data (Risks of improper content and poisoning in training data)","description":"\"If the training data includes illegal or harmful information, such as false, biased, or IPR-infringing content, or lacks diversity in its sources, the output may include harmful content like illegal, malicious, or extreme information.\nTraining data is also at risk of being poisoned through tampering, error injection, or misleading actions by attackers. This can interfere with the model's probability distribution, reducing its accuracy and reliability.\"","entity":"Human","intent":"Other","timing":"Pre-deployment","domain":1,"subdomain":"1.2"},{"ev_id":"45.01.09","quick_ref":"TC2602024","paper_title":"AI Safety Governance Framework ","level":"Risk Sub-Category","risk_category":"AI's inherent safety risks ","risk_subcategory":"Risks from data (Risks of unregulated training data annotation)","description":"\"Issues with training data annotation, such as incomplete annotation guidelines, incapable annotators, and errors in annotation, can affect the accuracy, reliability, and effectiveness of models and algorithms. Moreover, they can introduce training biases, amplify discrimination, reduce generalization abilities, and result in incorrect outputs.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"45.01.13","quick_ref":"TC2602024","paper_title":"AI Safety Governance Framework ","level":"Risk Sub-Category","risk_category":"AI's inherent safety risks ","risk_subcategory":"Risks from AI systems (Risks of supply chain security)","description":"\"The AI industry relies on a highly globalized supply chain. However, certain countries may use unilateral coercive measures, such as technology barriers and export restrictions, to create development obstacles and maliciously disrupt the global AI supply chain. This can lead to significant risks of supply disruptions for chips, software, and tools.\"","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":6,"subdomain":"6.4"},{"ev_id":"47.02.08","quick_ref":"G'sell2024","paper_title":"Regulating under Uncertainty: Governance Options for Generative AI","level":"Risk Sub-Category","risk_category":"Ethical and social risks ","risk_subcategory":"Bias and discrimination (bias in training datasets) ","description":"\"AI experts consider training data to be the most salient source of bias in generative AI models. For example, GPT- 2’s training data comes from outbound links from Reddit, a social network often criticized for hosting anti-feminist content.351 As a result, AI models trained on such data are more likely to produce outputs that reflect these biases.\"","entity":"Other","intent":"Unintentional","timing":"Pre-deployment","domain":1,"subdomain":"1.1"},{"ev_id":"47.02.09","quick_ref":"G'sell2024","paper_title":"Regulating under Uncertainty: Governance Options for Generative AI","level":"Risk Sub-Category","risk_category":"Ethical and social risks ","risk_subcategory":"Bias and discrimination (value embedding) ","description":"\"Generative AI models may also be subject to the “value embedding” phenomenon.361 “Value embedding” refers to the fact that developers of generative AI models strive to minimize biased outputs by retraining their models based on normative values.362 Contemporary state-of- the-art models not only reflect the values embedded within their training data, they also undergo additional fine-tuning that follows a set of chosen rules and principles. Due to the absence of universally accepted standards, developers bear the responsibility of making decisions on sensitive issues. These practices lead to c","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":1,"subdomain":"1.3"},{"ev_id":"47.03.01","quick_ref":"G'sell2024","paper_title":"Regulating under Uncertainty: Governance Options for Generative AI","level":"Risk Sub-Category","risk_category":"Legal challenges ","risk_subcategory":"Privacy and data collection concerns (collecting personal information or personally identifiable information) ","description":"\"Generative AI developers train their models with extensive datasets often gathered through online web scraping of websites that may include personal data or personally identifiable information (PII). For most generative AI applications, such as initial model training, the primary concerns are the quantity, variety, and quality of the data, not whether they include personally identifiable information. However, some web-scraped datasets may inadvertently include personal data. Additionally, when downstream developers integrate generative AI into their products or services by fine- tuning a pre-","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":2,"subdomain":"2.1"},{"ev_id":"47.03.03","quick_ref":"G'sell2024","paper_title":"Regulating under Uncertainty: Governance Options for Generative AI","level":"Risk Sub-Category","risk_category":"Legal challenges ","risk_subcategory":"Copyright challenges (training models using copyrighted output) ","description":"\"Generative AI companies are regularly accused of violating copyright law by training AI models on copyrighted works without gaining permission or paying compensation to the copyright owners. In fact, a substantial number of copyrighted documents and books have been incorporated into the training datasets of generative AI models.\"","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":6,"subdomain":"6.3"},{"ev_id":"47.04.05","quick_ref":"G'sell2024","paper_title":"Regulating under Uncertainty: Governance Options for Generative AI","level":"Risk Sub-Category","risk_category":"Environmental, economical, and societal challenges ","risk_subcategory":"Environmental cost (energy consumption) ","description":"\"Training large AI models requires a substantial amount of computing power to handle vast datasets, which translates into high energy consumption.\"","entity":"Other","intent":"Unintentional","timing":"Pre-deployment","domain":6,"subdomain":"6.6"},{"ev_id":"47.04.06","quick_ref":"G'sell2024","paper_title":"Regulating under Uncertainty: Governance Options for Generative AI","level":"Risk Sub-Category","risk_category":"Environmental, economical, and societal challenges ","risk_subcategory":"Environmental cost (water consumption) ","description":"\"Data centers use water for cooling to prevent servers from overheating. The water consumption associated with AI training and inference processes can be substantial, impacting local water resources.\"","entity":"Other","intent":"Unintentional","timing":"Pre-deployment","domain":6,"subdomain":"6.6"},{"ev_id":"48.05.00","quick_ref":"NIST2024","paper_title":"Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile","level":"Risk Category","risk_category":"Environmental Impacts ","risk_subcategory":null,"description":"\"Impacts due to high compute resource utilization in training or operating GAI models, and related outcomes that may adversely impact ecosystems.\"","entity":"Other","intent":"Unintentional","timing":"Pre-deployment","domain":6,"subdomain":"6.6"},{"ev_id":"48.12.00","quick_ref":"NIST2024","paper_title":"Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile","level":"Risk Category","risk_category":"Value Chain and Component Integration ","risk_subcategory":null,"description":"\"Non-transparent or untraceable integration of \nupstream third-party components, including data that has been improperly obtained or not \nprocessed and cleaned due to increased automation from GAI; improper supplier vetting across the AI lifecycle; or other issues that diminish transparency or accountability for downstream \nusers.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.4"},{"ev_id":"51.04.00","quick_ref":"Everitt2018 ","paper_title":"AGI Safety Literature Review ","level":"Risk Category","risk_category":"Security ","risk_subcategory":null,"description":"\"How to design AGIs that are robust to adversaries and adversarial environ-\nments? This involves building sandboxed AGI protected from adversaries (Berkeley), and agents that are robust to adversarial inputs (Berkeley, DeepMind).\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":2,"subdomain":"2.2"},{"ev_id":"51.05.00","quick_ref":"Everitt2018 ","paper_title":"AGI Safety Literature Review ","level":"Risk Category","risk_category":"Safe learning ","risk_subcategory":null,"description":"\"AGIs should avoid making fatal mistakes during the learning phase.\nSubproblems include safe exploration and distributional shift (DeepMind, OpenAI), and continual learning (Berkeley).\"","entity":"AI","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"51.06.00","quick_ref":"Everitt2018 ","paper_title":"AGI Safety Literature Review ","level":"Risk Category","risk_category":"Intelligibility ","risk_subcategory":null,"description":"\"How can we build agent’s whose decisions we can understand? Con-\nnects explainable decisions (Berkeley) and informed oversight (MIRI).\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.4"},{"ev_id":"51.09.00","quick_ref":"Everitt2018 ","paper_title":"AGI Safety Literature Review ","level":"Risk Category","risk_category":"Malign belief distributions ","risk_subcategory":null,"description":"\"Christiano (2016) argues that the universal distribution M (Hutter, 2005; Solomonoff, 1964a,b, 1978) is malign. The argument is somewhat intricate, and is based on the idea that a hypothesis about the world often includes simulations of other agents, and that these agents may have an incentive to influence anyone making decisions based on the distribution. While it is unclear to what extent this type of problem would affect any practical agent, it bears some semblance to aggressive memes, which do cause problems for human reasoning (Dennett, 1990).\"","entity":"Other","intent":"Other","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"52.03.02","quick_ref":"Maham2023 ","paper_title":"Governing General Purpose AI: A Comprehensive Map of Unreliability, Misuse and Systemic Risks ","level":"Risk Sub-Category","risk_category":"Systemic Risks ","risk_subcategory":"Ideological Homogenization from Value Embedding","description":"\"The increasing integration of general purpose AI models into every-day life raises concerns around their embedded normative values. The reach of a small number of AI models to a large number of people around the world can make these value judgements unprecedently impactful, potentially leading to increased ideological homogenization.\"","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":1,"subdomain":"1.3"},{"ev_id":"53.01.06","quick_ref":"Maas2023","paper_title":"Advancing AI Governance: A Literature Review of Problems, Options, and Proposals ","level":"Risk Sub-Category","risk_category":"Alignment failures in existing ML systems ","risk_subcategory":"Inner misalignment ","description":"-","entity":"AI","intent":"Other","timing":"Pre-deployment","domain":7,"subdomain":"7.1"},{"ev_id":"53.02.02","quick_ref":"Maas2023","paper_title":"Advancing AI Governance: A Literature Review of Problems, Options, and Proposals ","level":"Risk Sub-Category","risk_category":"Dangerous capabilities in AI systems ","risk_subcategory":"Acquisition of a goal to harm society ","description":"\"cases of AI systems being given the outright goal of harming humanity (ChaosGPT);\"","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":4,"subdomain":"4.2"},{"ev_id":"54.01.01","quick_ref":"Leech2024 ","paper_title":"Ten Hard Problems in Artificial Intelligence We Must Get Right","level":"Risk Sub-Category","risk_category":"Negative impacts of AI use ","risk_subcategory":"Under-recognized work ","description":"\"Without training data, ML cannot take place. Much of this data comes from paid clickwork (also called “platform work” [170] or “microwork” [558]), unpaid crowdsourcing, and unpaid user behavior capture. Clickworkers, mainly in the global south, perform repetitive data-labeling tasks for use in the training of ML models [558]. The market value of such annotations “is projected to reach $13.7 billion by 2030” [228] and the annotation industry is widely reported to have little concern for workers’ rights. Besides welfare and rights, the invisibility of this contribution arguably contributes to a","entity":"Other","intent":"Other","timing":"Pre-deployment","domain":6,"subdomain":"6.2"},{"ev_id":"54.01.02","quick_ref":"Leech2024 ","paper_title":"Ten Hard Problems in Artificial Intelligence We Must Get Right","level":"Risk Sub-Category","risk_category":"Negative impacts of AI use ","risk_subcategory":"Environmental cost ","description":"\"Large-scale DL systems can produce signicant carbon emissions as a result of the computational demands of training runs and inference [539]\"","entity":"AI","intent":"Unintentional","timing":"Pre-deployment","domain":6,"subdomain":"6.6"},{"ev_id":"54.03.03","quick_ref":"Leech2024 ","paper_title":"Ten Hard Problems in Artificial Intelligence We Must Get Right","level":"Risk Sub-Category","risk_category":"Harm caused by unaligned competent systems ","risk_subcategory":"Deceptive alignment ","description":"\"system learns to detect human monitoring and hides its undesirable properties—simply because any display of these properties is penalized by the feedback process, while that same feedback is usually imperfect. (Consider the problem of verifying a translation into a language you do not speak, or of checking a mathematical proof that is thousands of pages long.) [92, 259]. Rudimentary examples of deceptive alignment have been observed in current systems [322, 333].\"","entity":"AI","intent":"Intentional","timing":"Pre-deployment","domain":7,"subdomain":"7.2"},{"ev_id":"54.04.02","quick_ref":"Leech2024 ","paper_title":"Ten Hard Problems in Artificial Intelligence We Must Get Right","level":"Risk Sub-Category","risk_category":"Within-country issues: domestic inequality ","risk_subcategory":"Privatization of AI ","description":"\"Researchers in deep learning and those with greater research impact are more likely to migrate to industry, raising concerns about the “privatization of AI knowledge” [278]. Specically, if the most sophisticated AI approaches become proprietary and are used only within private research labs, then it will be impossible for universities to teach them, let alone contribute to leading research.\"","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":6,"subdomain":"6.1"},{"ev_id":"55.02.04","quick_ref":"Clarke2023","paper_title":"A Survey of the Potential Long-term Impacts of AI: How AI Could Lead to Long-term Changes in Science, Cooperation, Power, Epistemics and Values","level":"Risk Sub-Category","risk_category":"Worsened conflict ","risk_subcategory":"Resource conflicts driven by AI development ","description":"\"AI development may itself become a new flash point for conflicts—causing more conflict to occur— especially conflicts over AI-relevant resources (such as data centres, semiconductor manufacturing facilities and raw materials).\"","entity":"Other","intent":"Unintentional","timing":"Pre-deployment","domain":6,"subdomain":"6.4"},{"ev_id":"56.04.00","quick_ref":"GOS2023","paper_title":"Future Risks of Frontier AI ","level":"Risk Category","risk_category":"Amplification of biases","risk_subcategory":null,"description":"\"Current Frontier AI mdoels amplify existing biases within their training data and can be manipulated into providing potentially harmful responses, for example abusive language or discriminatory responses91,92. This is not limited to text generation but can be seen across all modalities of generative AI93. Training on large swathes of UK and US English internet content can mean that misogynistic, ageist, and white supremacist content is overrepresented in the training data94.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":1,"subdomain":"1.1"},{"ev_id":"56.05.00","quick_ref":"GOS2023","paper_title":"Future Risks of Frontier AI ","level":"Risk Category","risk_category":"Harmful responses ","risk_subcategory":null,"description":"\"Current Frontier AI mdoels amplify existing biases within their training data and can be manipulated into providing potentially harmful responses, for example abusive language or discriminatory responses91,92. This is not limited to text generation but can be seen across all modalities of generative AI93. Training on large swathes of UK and US English internet content can mean that misogynistic, ageist, and white supremacist content is overrepresented in the training data94.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":1,"subdomain":"1.2"},{"ev_id":"56.06.00","quick_ref":"GOS2023","paper_title":"Future Risks of Frontier AI ","level":"Risk Category","risk_category":"Lack of transparency and interpretability ","risk_subcategory":null,"description":"\"Today's Frontier AI is difficult to interpret and lacks transparency. Contextual understanding of the training data is not explicitly embedded within these models. They can fail to capture perspectives of underrepresented groups or the limitations within which they are expected to perform without fine tuning or reinforcement learning with human feedback (RLHF).\"","entity":"AI","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.4"},{"ev_id":"56.07.00","quick_ref":"GOS2023","paper_title":"Future Risks of Frontier AI ","level":"Risk Category","risk_category":"Intellectual property rights ","risk_subcategory":null,"description":"\"There are also issues around intellectual property rights for content in training datasets\" ","entity":"Human","intent":"Other","timing":"Pre-deployment","domain":6,"subdomain":"6.3"},{"ev_id":"59.01.00","quick_ref":"Schnitzer2024","paper_title":"AI Hazard Management: A Framework for the Systematic Management of Root Causes for AI Risks","level":"Risk Category","risk_category":"Inadequate specification of ODD","risk_subcategory":null,"description":"\"The operational design domain (ODD) is a technical description of the application’s operational environment, initially conceptualized for autonomous driving systems. An inadequate specification of the ODD limits essential functions such as testing the learned functionality and out-of-distribution detection.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"59.03.00","quick_ref":"Schnitzer2024","paper_title":"AI Hazard Management: A Framework for the Systematic Management of Root Causes for AI Risks","level":"Risk Category","risk_category":"Inadequate planning of performance requirements","risk_subcategory":null,"description":"\"The expected performance of the AI system should be planned adequately. Hereby, an important aspect is that chosen performance metrics are meaningful for presenting the intended functionality. Otherwise, expectations and safety requirements can be unfulfillable at later life cycle stages.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"59.04.00","quick_ref":"Schnitzer2024","paper_title":"AI Hazard Management: A Framework for the Systematic Management of Root Causes for AI Risks","level":"Risk Category","risk_category":"Insufficient AI development documentation","risk_subcategory":null,"description":"\"Throughout the development of an AI system, it is vital to document every decision and action taken. This is not only essential to optimize the development process itself but also required for the auditability of the AI system.\"","entity":"Human","intent":"Other","timing":"Pre-deployment","domain":7,"subdomain":"7.4"},{"ev_id":"59.05.00","quick_ref":"Schnitzer2024","paper_title":"AI Hazard Management: A Framework for the Systematic Management of Root Causes for AI Risks","level":"Risk Category","risk_category":"Inappropriate degree of transparency to end users","risk_subcategory":null,"description":"\"The transparency to end users of the AI system increases the user’s trust in the AI application. If not adequately integrated into the design, this might prevent the proper operation and cause potential misuse of the AI application.\"","entity":"Human","intent":"Other","timing":"Pre-deployment","domain":7,"subdomain":"7.4"},{"ev_id":"59.07.00","quick_ref":"Schnitzer2024","paper_title":"AI Hazard Management: A Framework for the Systematic Management of Root Causes for AI Risks","level":"Risk Category","risk_category":"Choice of untrustworthy data source","risk_subcategory":null,"description":"\"The choice of a trustworthy data source is a first prerequisite in order to fulfill data quality requirements. This is especially the case if third-party data sources are used to develop the AI system.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.0"},{"ev_id":"59.08.00","quick_ref":"Schnitzer2024","paper_title":"AI Hazard Management: A Framework for the Systematic Management of Root Causes for AI Risks","level":"Risk Category","risk_category":"Lack of data understanding","risk_subcategory":null,"description":"\"The correct understanding of the used data for developing an AI system is a prerequisite to avoid data shortcomings and hinders the development of an AI system which is best suiting for the intended functionality.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.0"},{"ev_id":"59.11.00","quick_ref":"Schnitzer2024","paper_title":"AI Hazard Management: A Framework for the Systematic Management of Root Causes for AI Risks","level":"Risk Category","risk_category":"Incorrect data labels","risk_subcategory":null,"description":"\"Data labels are essential for any supervised learning algorithm since they preset the result of the learning process. If the correctness of the data labels is not given, the AI system is prevented from learning the ground truth and therefore the intended functionality.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"59.12.00","quick_ref":"Schnitzer2024","paper_title":"AI Hazard Management: A Framework for the Systematic Management of Root Causes for AI Risks","level":"Risk Category","risk_category":"Data poisoning","risk_subcategory":null,"description":"\"Data poisoning describes an attack in the form of an injection of malicious data into the training set. If not prevented, this attack leads the AI system to learn unintended behavior.\"","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":2,"subdomain":"2.2"},{"ev_id":"59.13.00","quick_ref":"Schnitzer2024","paper_title":"AI Hazard Management: A Framework for the Systematic Management of Root Causes for AI Risks","level":"Risk Category","risk_category":"Insufficient data representation","risk_subcategory":null,"description":"\"The distribution of the data used for training a model should match the operational data ́s distribution while consisting of sufficiently many samples. An important aspect of matching distributions between training and operational data is that also data which is rarely confronting the AI system in operation is represented in the training data.\"","entity":"Other","intent":"Other","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"59.14.00","quick_ref":"Schnitzer2024","paper_title":"AI Hazard Management: A Framework for the Systematic Management of Root Causes for AI Risks","level":"Risk Category","risk_category":"Problems of synthetic data","risk_subcategory":null,"description":"\"In the case of sparse data quantity, the simulation or generation of data is a valid alternative. However, it is essential to make sure that the simulated data is sufficiently similar to real data, especially in the way the AI system perceives them. Otherwise, generalization to operational data and reliable operational behavior can not be guaranteed.\"","entity":"Other","intent":"Other","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"59.15.00","quick_ref":"Schnitzer2024","paper_title":"AI Hazard Management: A Framework for the Systematic Management of Root Causes for AI Risks","level":"Risk Category","risk_category":"Inappropriate data splitting","risk_subcategory":null,"description":"\"In data-driven AI development, the annotated data set is commonly split into training, validation, and test sets, whereby it is essential that the latter is not used for development but only for evaluation. Using the test set for training manipulates the testing strategy, which is the basis of the system’s quality assurance.\"","entity":"Human","intent":"Other","timing":"Pre-deployment","domain":7,"subdomain":"7.0"},{"ev_id":"59.16.00","quick_ref":"Schnitzer2024","paper_title":"AI Hazard Management: A Framework for the Systematic Management of Root Causes for AI Risks","level":"Risk Category","risk_category":"Poor model design choices","risk_subcategory":null,"description":"\"The model specifications have significant impact on the functionality of an AI system. The developer mak- ing wrong decisions might cause the AI system to behave biased and unreliable.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"59.21.00","quick_ref":"Schnitzer2024","paper_title":"AI Hazard Management: A Framework for the Systematic Management of Root Causes for AI Risks","level":"Risk Category","risk_category":"Uncertainty concerns","risk_subcategory":null,"description":"\"AI systems should be able not only to return output for a given instance but also to provide a corresponding level of confidence. If such a method is not implemented or not working correctly, this can have a negative impact on performance and safety.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.0"},{"ev_id":"59.25.01","quick_ref":"Schnitzer2024","paper_title":"AI Hazard Management: A Framework for the Systematic Management of Root Causes for AI Risks","level":"Risk Sub-Category","risk_category":"AI lifecycle stage","risk_subcategory":"(1) Scoping ","description":"\"A majority of them possess an initial stage devoted to the planning and scoping of the AI system.\"","entity":"Not coded","intent":"Not coded","timing":"Pre-deployment","domain":null,"subdomain":null},{"ev_id":"59.25.02","quick_ref":"Schnitzer2024","paper_title":"AI Hazard Management: A Framework for the Systematic Management of Root Causes for AI Risks","level":"Risk Sub-Category","risk_category":"AI lifecycle stage","risk_subcategory":"(2) Data collection and preparation ","description":null,"entity":"Not coded","intent":"Not coded","timing":"Pre-deployment","domain":null,"subdomain":null},{"ev_id":"59.25.03","quick_ref":"Schnitzer2024","paper_title":"AI Hazard Management: A Framework for the Systematic Management of Root Causes for AI Risks","level":"Risk Sub-Category","risk_category":"AI lifecycle stage","risk_subcategory":"(3) Modeling ","description":null,"entity":"Not coded","intent":"Not coded","timing":"Pre-deployment","domain":null,"subdomain":null},{"ev_id":"59.26.01","quick_ref":"Schnitzer2024","paper_title":"AI Hazard Management: A Framework for the Systematic Management of Root Causes for AI Risks","level":"Risk Sub-Category","risk_category":"Mode","risk_subcategory":"Technical ","description":"\"Technical AI hazards are the root causes of technical deficiencies in the AI system. An example of such an AI hazard is overfitting, which describes a model’s excessive adaptation to the training dataset. Quantitative methods to assess (metrics) and treat (mitigation means) exist for technical AI hazards, which might be performed automatically. In case of overfitting, metrics are based on the comparison of performance between the training and validation datasets, and mitigation means may include regularization techniques, among others.\"","entity":"AI","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"59.26.03","quick_ref":"Schnitzer2024","paper_title":"AI Hazard Management: A Framework for the Systematic Management of Root Causes for AI Risks","level":"Risk Sub-Category","risk_category":"Mode","risk_subcategory":"Procedural ","description":"\"The third class encompasses procedural AI hazards. These pertain to issues arising from processes and actions made by individuals involved in the develop- ment process. Such hazards are not readily quantifiable and necessitate alter- native mitigation strategies. An example of such an AI hazard would be ”poor model design choices,” which could be expressed, for instance, through a devel- oper’s decision to select an unsuitable AI model for a given problem. Due to the challenges in quantifying and mitigating these issues, qualitative approaches must be employed. In the case of the aforemention","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"60.02.02","quick_ref":"Bengio2025","paper_title":"International AI Safety Report 2025","level":"Risk Sub-Category","risk_category":"Risks from malfunctions ","risk_subcategory":"Bias ","description":"\"General-purpose AI systems can amplify social and political biases, causing concrete harm. They frequently display biases with respect to race, gender, culture, age, disability, political opinion, or other aspects of human identity. This can lead to discriminatory outcomes including unequal resource allocation, reinforcement of stereotypes, and systematic neglect of certain groups or viewpoints.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":1,"subdomain":"1.1"},{"ev_id":"61.02.17","quick_ref":"Uuk2025","paper_title":"A Taxonomy of Systemic Risks from General-Purpose AI ","level":"Risk Sub-Category","risk_category":"Sources of systemic risks from general-purpose AI ","risk_subcategory":"Dangerous development races","description":"\"Competitive pressures could lead to the neglect of safety measures in AI development.\"","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":6,"subdomain":"6.4"},{"ev_id":"61.02.23","quick_ref":"Uuk2025","paper_title":"A Taxonomy of Systemic Risks from General-Purpose AI ","level":"Risk Sub-Category","risk_category":"Sources of systemic risks from general-purpose AI ","risk_subcategory":"Energy-intensive processes","description":"\"AI data collection, storage, and model training are energy-intensive, contributing to environmental risks.\"","entity":"Other","intent":"Unintentional","timing":"Pre-deployment","domain":6,"subdomain":"6.6"},{"ev_id":"61.02.25","quick_ref":"Uuk2025","paper_title":"A Taxonomy of Systemic Risks from General-Purpose AI ","level":"Risk Sub-Category","risk_category":"Sources of systemic risks from general-purpose AI ","risk_subcategory":"Exploitation in AI development","description":"\"Outsourcing tasks like data labeling to low-income countries can perpetuate inequality.\"","entity":"Human","intent":"Other","timing":"Pre-deployment","domain":6,"subdomain":"6.2"},{"ev_id":"62.04.00","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Category","risk_category":"Dimension - Technical Attributes (AI inadequacy - technical failure) ","risk_subcategory":null,"description":"\"As above, there are broadly two dimensions of technical failure modes: quality of data or input signal, and training performance. Due to a lack of transparency, it may be difficult to ascertain the type of technical failure that gives rise to a particular risk, and it is often a combination of several factors. Risks pertain- ing to AI failures are exacerbated by poor quality training data and imperfect training signals. Various measures can be implemented to improve the quality of the training data, and fine-tuning techniques can be used to disincentivize harmful model behavior.\"","entity":"Other","intent":"Other","timing":"Pre-deployment","domain":null,"subdomain":null},{"ev_id":"62.04.01","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Dimension - Technical Attributes (AI inadequacy - technical failure) ","risk_subcategory":"Supervised/unsupervised AI (AI data quality related - biased training data) ","description":"\"As above, there are broadly two dimensions of technical failure modes: quality of data or input signal, and training performance. Due to a lack of transparency, it may be difficult to ascertain the type of technical failure that gives rise to a particular risk, and it is often a combination of several factors. Risks pertain- ing to AI failures are exacerbated by poor quality training data and imperfect training signals. Various measures can be implemented to improve the quality of the training data, and fine-tuning techniques can be used to disincentivize harmful model behavior.\"","entity":"Other","intent":"Other","timing":"Pre-deployment","domain":null,"subdomain":null},{"ev_id":"62.04.02","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Dimension - Technical Attributes (AI inadequacy - technical failure) ","risk_subcategory":"Supervised/unsupervised AI (AI training performance related - Robustness) ","description":"\"As above, there are broadly two dimensions of technical failure modes: quality of data or input signal, and training performance. Due to a lack of transparency, it may be difficult to ascertain the type of technical failure that gives rise to a particular risk, and it is often a combination of several factors. Risks pertain- ing to AI failures are exacerbated by poor quality training data and imperfect training signals. Various measures can be implemented to improve the quality of the training data, and fine-tuning techniques can be used to disincentivize harmful model behavior.\"","entity":"Other","intent":"Other","timing":"Pre-deployment","domain":null,"subdomain":null},{"ev_id":"62.04.03","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Dimension - Technical Attributes (AI inadequacy - technical failure) ","risk_subcategory":"Supervised/unsupervised AI (AI training performance related - Accuracy) ","description":"\"As above, there are broadly two dimensions of technical failure modes: quality of data or input signal, and training performance. Due to a lack of transparency, it may be difficult to ascertain the type of technical failure that gives rise to a particular risk, and it is often a combination of several factors. Risks pertain- ing to AI failures are exacerbated by poor quality training data and imperfect training signals. Various measures can be implemented to improve the quality of the training data, and fine-tuning techniques can be used to disincentivize harmful model behavior.\"","entity":"Other","intent":"Other","timing":"Pre-deployment","domain":null,"subdomain":null},{"ev_id":"62.04.04","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Dimension - Technical Attributes (AI inadequacy - technical failure) ","risk_subcategory":"Supervised/unsupervised AI (AI training performance related - Reliability) ","description":"\"As above, there are broadly two dimensions of technical failure modes: quality of data or input signal, and training performance. Due to a lack of transparency, it may be difficult to ascertain the type of technical failure that gives rise to a particular risk, and it is often a combination of several factors. Risks pertain- ing to AI failures are exacerbated by poor quality training data and imperfect training signals. Various measures can be implemented to improve the quality of the training data, and fine-tuning techniques can be used to disincentivize harmful model behavior.\"","entity":"Other","intent":"Other","timing":"Pre-deployment","domain":null,"subdomain":null},{"ev_id":"62.04.05","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Dimension - Technical Attributes (AI inadequacy - technical failure) ","risk_subcategory":"Reinforcement learning AI (Training design related) ","description":"\"As above, there are broadly two dimensions of technical failure modes: quality of data or input signal, and training performance. Due to a lack of transparency, it may be difficult to ascertain the type of technical failure that gives rise to a particular risk, and it is often a combination of several factors. Risks pertain- ing to AI failures are exacerbated by poor quality training data and imperfect training signals. Various measures can be implemented to improve the quality of the training data, and fine-tuning techniques can be used to disincentivize harmful model behavior.\"","entity":"Other","intent":"Other","timing":"Pre-deployment","domain":null,"subdomain":null},{"ev_id":"62.04.06","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Dimension - Technical Attributes (AI inadequacy - technical failure) ","risk_subcategory":"Reinforcement learning AI (Training performance related) ","description":"\"As above, there are broadly two dimensions of technical failure modes: quality of data or input signal, and training performance. Due to a lack of transparency, it may be difficult to ascertain the type of technical failure that gives rise to a particular risk, and it is often a combination of several factors. Risks pertain- ing to AI failures are exacerbated by poor quality training data and imperfect training signals. Various measures can be implemented to improve the quality of the training data, and fine-tuning techniques can be used to disincentivize harmful model behavior.\"","entity":"Other","intent":"Other","timing":"Pre-deployment","domain":null,"subdomain":null},{"ev_id":"62.06.01","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Dimension - Stage of Risk Emergence ","risk_subcategory":"Pre-deployment ","description":"\"For GPAIs or foundation models, risks emerge during training, prior to being repurposed and deployed in more specific AI systems or applications. Risk assessments can be conducted before deployment, and monitoring of AI models can occur as required throughout the deployment phase. In certain cases, version updates or model recalls may be warranted post-deployment.\"","entity":"Not coded","intent":"Not coded","timing":"Pre-deployment","domain":null,"subdomain":null},{"ev_id":"62.14.00","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Category","risk_category":"Model Development ","risk_subcategory":null,"description":"-","entity":"Human","intent":"Other","timing":"Pre-deployment","domain":null,"subdomain":null},{"ev_id":"62.14.01","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Model Development ","risk_subcategory":"Data-related (Difficulty filtering large web scrapes or large scale web datasets)","description":"\"A large scale “scraping” of web data for training datasets increases vulnerability to data poisoning, backdoor attacks, and the inclusion of inaccurate or toxic data [76, 28, 48]. With a large dataset, filtering out these quality issues is very difficult or trades off against significant data loss.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":2,"subdomain":"2.2"},{"ev_id":"62.14.02","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Model Development ","risk_subcategory":"Data-related (Lack of cross-organizational documentation)","description":"\"When sharing data between multiple organizations, documentation may be missing or inadequate, making it difficult for other organizations to understand it. For example, a lack of metadata or a change in schema by a collaborating party can result in an unusable dataset and wasted data collection efforts, or it can lead to misunderstandings about the dataset’s limitations, resulting in downstream risks related to its use [173].\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"62.14.03","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Model Development ","risk_subcategory":"Data-related (Manipulation of data by non-domain experts)","description":"\"Manipulating data (e.g., training data) carries a set of assumptions on how the data should appear and be used by those performing the manipulation. Common manipulations applied on data in the context of AI models include defining the ground truth label and merging different data formats or sources. People who have little or no expertise in the domain of the data performing such manipulations may render the data unusable or harmful to the development of the AI system [173].\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"62.14.04","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Model Development ","risk_subcategory":"Data-related (Insufficient quality control in data collection process)","description":"\"A lack of standardized methods and sufficient infrastructure, including the absence of quality control processes for collecting data, especially for high-stakes domains and benchmarks, can affect the quality and type of the data collected [173, 95]. This may include risks of dataset poisoning, inadvertent copyright violation, and test set leakages which invalidate performance metrics.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":2,"subdomain":"2.2"},{"ev_id":"62.15.00","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Model Development ","risk_subcategory":"Training-related (Robust overfitting in adversarial training)","description":"\"Adversarial training can be affected by robust overfitting, where the model’s robustness on test data decreases during further training, particularly after the learning rate decay. This issue has been consistently observed across various datasets and algorithms in adversarial training settings [163, 230]. Robust over- fitting can affect the model’s ability to generalize effectively and reduce its resilience to adversarial attacks.\"","entity":"Other","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"62.15.04","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Model Development ","risk_subcategory":"Fine-tuning related (Unexpected competence in fine-tuned versions of the upstream model)","description":"\"Downstream deployers may often fine-tune a GPAI model with specific deploy- ment-related datasets, to better suit the task. Fine-tuned upstream models can gain new or unexpected capabilities that the underlying upstream models did not exhibit [202, 126, 137]. These new capabilities may be unanticipated by the original model developer.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.2"},{"ev_id":"62.15.06","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Model Development ","risk_subcategory":"Fine-tuning related (Fine-tuning dataset poisoning)","description":"\"A deployer can poison the dataset used during the fine-tuning process [98] to induce specific, often malicious, behaviors in a model. This can be performed without having access to the model’s weights. This poisoning can be difficult to detect through direct inspection of the dataset, as the manipulations may be subtle and targeted.\"","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":2,"subdomain":"2.2"},{"ev_id":"62.15.07","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Model Development ","risk_subcategory":"Fine-tuning related (Poisoning models during instruction tuning)","description":"\"AI models can be poisoned during instruction tuning when models are tuned using pairs of instructions and desired outputs. Poisoning in instruction tuning can be achieved with a lower number of compromised samples, as instruction tuning requires a relatively small number of samples for fine-tuning [155, 211]. Anonymous crowdsourcing efforts may be employed in collecting instruction tuning datasets and can further contribute to poisoning attacks [187]. These attacks might be harder to detect than traditional data poisoning attacks.\"","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":2,"subdomain":"2.2"},{"ev_id":"62.16.00","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Category","risk_category":"Model Evaluations","risk_subcategory":null,"description":"\"This section catalogs the risk sources and risk management measures related to model evaluations (often called evals). We categorize them into the fol- lowing groups: general evaluations, benchmarking, red teaming, auditing, and interpretability/explainability. The subsection on general evaluations consists of items that are common to various evaluation techniques, while the other subsections are specific to their respective evaluation types.\"","entity":"Human","intent":"Other","timing":"Pre-deployment","domain":null,"subdomain":null},{"ev_id":"62.16.01","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Model Evaluations","risk_subcategory":"General Evaluations (Incorrect outputs of GPAI evaluating other AI models) ","description":"\"When an LLM is configured to evaluate the performance of another model or AI system, it may produce incorrect evaluation outputs [122, 147]. For example, it may give a higher rating to a more verbose answer or an answer from a particular political stance. If an LLM-based evaluation is integrated into the training of a new model, the trained model could develop in a way that specifically finds and exploits limitations in the evaluator’s metrics.\"","entity":"AI","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.1"},{"ev_id":"62.16.02","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Model Evaluations","risk_subcategory":"General Evaluations (Limited coverage of capabilities evaluations)","description":"\"GPAI model developers might run capabilities evaluations to determine whether it has dangerous or dual-use capabilities, and then decide whether it is safe to deploy. Such capabilities evaluations can fail to demonstrate all the capabilities of a model. For example, evaluations may miss certain capabilities that are difficult to assess, prohibitively costly to verify, or obscured by the model’s tendency to refuse responses due to safety training, even if it possesses some of these capabilities.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":6,"subdomain":"6.5"},{"ev_id":"62.16.06","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Model Evaluations","risk_subcategory":"General Evaluations (Biased evaluations of encoded human values)","description":"\"Encoded human values in AI models that are easier to evaluate might be preferred for inclusion in evaluations over those that are more difficult to measure [13]. This might come at the expense of more desirable but harder-to-quantify  values. This bias can lead to an imbalance, where easier-to-measure values dominate the evaluation process, while other important values are underrepresented.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":6,"subdomain":"6.5"},{"ev_id":"62.16.08","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Model Evaluations","risk_subcategory":"Benchmarking (Benchmark leakage or data contamination)","description":"\"Benchmark leakage [235, 224, 221, 161] can happen when an AI model is trained or fine-tuned with evaluation-related data. This can lead to an unreliable model evaluation, especially if the data contains question-answer pairs from bench- marks.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":6,"subdomain":"6.5"},{"ev_id":"62.16.09","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Model Evaluations","risk_subcategory":"Benchmarking (Raw data contamination)","description":"\"This type of contamination [170] occurs when the raw and unlabeled data of a benchmark is used as part of the training set. Such data may not be properly formatted and may contain noise, especially if the contamination happens before the data is pre-processed into the benchmark. If this contamination occurs, it could cast doubt on the few-shot and zero-shot performance of the model on that benchmark.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":6,"subdomain":"6.5"},{"ev_id":"62.16.10","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Model Evaluations","risk_subcategory":"Benchmarking (Cross-lingual data contamination)","description":"\"Models that have been trained on data encoded in multiple languages, such as LLMs trained on web-crawled data, may contain contamination that is obscured by translation [226]. The most basic form of this is when a benchmark is trans- lated to another language and then fed to the model as training data. The fact that the benchmark is translated before becoming training data can obscure the contamination from detection methods, giving false assurance that the model has generalized on the capabilities that the benchmark tests for.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":6,"subdomain":"6.5"},{"ev_id":"62.16.11","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Model Evaluations","risk_subcategory":"Benchmarking (Guideline contamination)","description":"\"Guideline contamination refers to scenarios where instructions for the collec- tion, annotation, or use of the dataset are exposed to the model [170]. These instructions may contain explicit data-label pairs that can improve the model’s capabilities for the task.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":6,"subdomain":"6.5"},{"ev_id":"62.16.12","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Model Evaluations","risk_subcategory":"Benchmarking (Annotation contamination)","description":"\"Annotation contamination refers to scenarios where the model is exposed to the benchmark labels during training [170]. This type of contamination can make the model learn the acceptable distribution of outputs. Combining this with raw data contamination of the test split, any evaluation made with the benchmark is invalidated because the entire test split is essentially leaked to the model.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":6,"subdomain":"6.5"},{"ev_id":"62.16.14","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Model Evaluations","risk_subcategory":"Benchmark Inaccuracy (Benchmarks may not accurately evaluate capabilities)","description":"\"Benchmarks of AI systems can both underestimate and overestimate the capa- bilities of those AI systems. Underestimates can happen if an evaluation is not comprehensive enough, if the benchmark is saturated by existing models, or if the capabilities in question depend on a complicated setup, such as realistic computer programming tasks. Overestimates of capabilities can occur if an AI system is trained or fine-tuned on the contents of the benchmark, leading to overfitting.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":6,"subdomain":"6.5"},{"ev_id":"62.16.15","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Model Evaluations","risk_subcategory":"Benchmark Inaccuracy (Benchmark saturation)","description":"\"Benchmark saturation refers to benchmarks reaching their evaluation ceiling. The tendency towards benchmark saturation has been demonstrated in various benchmarks [19]. When benchmarks reach or are close to saturation, they stop being effective measures for new models, as more nuanced capability gains might not be detected.\"","entity":"Other","intent":"Other","timing":"Pre-deployment","domain":6,"subdomain":"6.5"},{"ev_id":"62.16.16","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Model Evaluations","risk_subcategory":"Benchmark Limitations (Insufficient benchmarks for AI safety evaluation) ","description":"\"Benchmarks dedicated to measuring the performance of AI systems (e.g., on programming or math tasks) are more well-developed than those for assessing safety and harms in AI systems [234]. This gap can lead to AI systems excelling in specific tasks while exhibiting harmful behaviors that go undetected. More safety-related evaluation datasets can help in identifying previously overlooked undesirable model behaviors.\"","entity":"Other","intent":"Other","timing":"Pre-deployment","domain":6,"subdomain":"6.5"},{"ev_id":"62.16.17","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Model Evaluations","risk_subcategory":"Benchmark Limitations (Underestimating capabilities that are not covered by benchmarks)","description":"\"A lack of test coverage by benchmarks on specific abilities of a model can obscure the model’s capabilities from both the developer and the user [160]. This can lead to a false sense of safety and trust due to a lack of understanding of the model’s limitations.\"","entity":"Other","intent":"Other","timing":"Pre-deployment","domain":6,"subdomain":"6.5"},{"ev_id":"62.17.00","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Category","risk_category":"Model Evaluations (Auditing) ","risk_subcategory":null,"description":"-","entity":"Human","intent":"Other","timing":"Pre-deployment","domain":6,"subdomain":"6.5"},{"ev_id":"62.17.01","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Model Evaluations (Auditing) ","risk_subcategory":"Conflicts of interest in auditor selection","description":"\"Conflicts of interest can arise if there is no independence in the auditor selection process or if the auditors are closely associated with the developer [123, 157]. In such cases, the conflict of interest can appear even if third-party evaluators are involved. In the case of external auditing, the potential candidates might be selected from a narrow group of auditors, or have conflicting financial incentives for whether to report model shortcomings publicly.\"","entity":"Human","intent":"Other","timing":"Pre-deployment","domain":6,"subdomain":"6.5"},{"ev_id":"62.17.02","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Model Evaluations (Auditing) ","risk_subcategory":"Auditor capacity mismatch","description":"\"Auditors may not be able to address all of the specific safety, performance, or validation needs. Reports of passing audits may be more inclusive than can be justified due to a lack of knowledge of specific risks and how they can be tested, or a lack of capacity to perform sufficiently rigorous testing.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":6,"subdomain":"6.5"},{"ev_id":"62.17.03","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Model Evaluations (Auditing) ","risk_subcategory":"Auditor failure","description":"\"Auditors may not publicly disclose risks they find, may be required to not pub- licize shortcomings, or may not receive sufficient cooperation from the relevant internal parties.\"","entity":"Human","intent":"Other","timing":"Pre-deployment","domain":6,"subdomain":"6.5"},{"ev_id":"62.18.00","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Category","risk_category":"Model Evaluations (Interpretability/Explainability) ","risk_subcategory":null,"description":null,"entity":"Human","intent":"Other","timing":"Pre-deployment","domain":7,"subdomain":"7.4"},{"ev_id":"62.19.04","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Attacks on GPAIs/GPAI Failure Modes ","risk_subcategory":"Backdoors or trojan attacks in GPAI models","description":"\"Backdoors can be inserted into GPAI models during their training or fine-tuning, to be exploited during deployment [185, 118]. Attackers inserting the backdoor can be the GPAI model provider themselves or another actor (e.g., by ma- nipulating the training data or the software infrastructure used by the model provider) [222]. Some backdoors can be exploited with minimal overhead, al- lowing attackers to control the model outputs in a targeted way with a high success rate [90].\"","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":2,"subdomain":"2.2"},{"ev_id":"62.22.02","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Agency (Goal-Directedness) ","risk_subcategory":"Reward or measurement tampering","description":"\"Measurement and reward tampering occur when an AI system, particularly one that learns from feedback for performing actions in an environment (e.g., rein- forcement learning), intervenes on the mechanisms that determine its training reward or loss. This can lead to the system learning behaviors that are con- trary to the intended goals set by the developer, by receiving erroneous positive feedback for such actions.\"","entity":"AI","intent":"Intentional","timing":"Pre-deployment","domain":7,"subdomain":"7.1"},{"ev_id":"62.24.02","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Agency (Situational Awareness) ","risk_subcategory":"Strategic underperformance on model evaluations","description":"\"GPAI developers often run evaluations ofual-use capabilities to decide whether it is safe to deploy. In some cases, these evaluations may fail to elicit these capabilities, either due to benign reasons or strategic action - by either the de- velopers, malicious actors, or arise unintentionally in the model during training [84, 97]. A GPAI model may strategically underperform or limit its performance during capability evaluations in order to be classified as safe for deployment. This underperformance could prevent the model from being identified as potentially dual use.\"","entity":"AI","intent":"Intentional","timing":"Pre-deployment","domain":7,"subdomain":"7.1"},{"ev_id":"62.24.02a","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Additional evidence","risk_category":"Agency (Situational Awareness) ","risk_subcategory":"Strategic underperformance on model evaluations","description":null,"entity":"AI","intent":"Intentional","timing":"Pre-deployment","domain":7,"subdomain":"7.2"},{"ev_id":"62.28.03","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Risk Sub-Category","risk_category":"Cybersecurity ","risk_subcategory":"AI System bypassing a sandbox environment","description":"\"An AI system may have the ability to bypass a sandboxed environment in which it is trained or evaluated.\"","entity":"AI","intent":"Other","timing":"Pre-deployment","domain":7,"subdomain":"7.2"},{"ev_id":"62.29.03a","quick_ref":"Gipiškis2024","paper_title":"Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems","level":"Additional evidence","risk_category":"Impacts of AI (General) ","risk_subcategory":"Competitive pressures in GPAI product release","description":null,"entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":6,"subdomain":"6.4"},{"ev_id":"63.06.00","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Category","risk_category":"Selection Pressures","risk_subcategory":null,"description":"\"Selection pressures (Section 3.3): some aspects of training and selection by those deploying and using AI agents can lead to undesirable behaviour;\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"64.04.07","quick_ref":"Marchal2024","paper_title":"Generative AI Misuse: A Taxonomy of Tactics and Insights from Real-World Data","level":"Risk Sub-Category","risk_category":"Misuse tactics to compromise GenAI systems (Model integrity) ","risk_subcategory":"Poisoning ","description":"\"Data Poisoning involves deliberately corrupting a model’s training dataset to introduce vulnerabilities, derail its learning process, or cause it to make incorrect predictions (Carlini et al., 2023). For example, the tool Nightshade is a data poisoning tool, which allows artists to add invisible changes to the pixels in their art before uploading online, to break any models that use it for training.9 Such attacks exploit the fact that most GenAI models are trained on publicly available datasets like images and videos scraped from the web, which malicious actors can easily compromise.\"","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":2,"subdomain":"2.2"},{"ev_id":"65.01.01","quick_ref":"IBM2025","paper_title":"AI Risk Atlas ","level":"Risk Sub-Category","risk_category":"Training Data Risks (Transparency) ","risk_subcategory":"Lack of training data transparency ","description":"\"Without accurate documentation on how a model's data was collected, curated, and used to train a model, it might be harder to satisfactorily explain the behavior of the model with respect to the data.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":6,"subdomain":"6.5"},{"ev_id":"65.01.02","quick_ref":"IBM2025","paper_title":"AI Risk Atlas ","level":"Risk Sub-Category","risk_category":"Training Data Risks (Transparency) ","risk_subcategory":"Uncertain data provenance ","description":"\"Data provenance refers to tracing history of data, which includes its ownership, origin, and transformations. Without standardized and established methods for verifying where the data came from, there are no guarantees that the data is the same as the original source and has the correct usage terms.\"","entity":"Human","intent":"Other","timing":"Pre-deployment","domain":6,"subdomain":"6.5"},{"ev_id":"65.02.01","quick_ref":"IBM2025","paper_title":"AI Risk Atlas ","level":"Risk Sub-Category","risk_category":"Training Data Risks (Data laws) ","risk_subcategory":"Data usage restrictions ","description":"\"Laws and other restrictions can limit or prohibit the use of some data for specific AI use cases.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"65.02.02","quick_ref":"IBM2025","paper_title":"AI Risk Atlas ","level":"Risk Sub-Category","risk_category":"Training Data Risks (Data laws) ","risk_subcategory":"Data acquisition restrictions ","description":"\"Laws and other regulations might limit the collection of certain types of data for specific AI use cases.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"65.02.03","quick_ref":"IBM2025","paper_title":"AI Risk Atlas ","level":"Risk Sub-Category","risk_category":"Training Data Risks (Data laws) ","risk_subcategory":"Data transfer restrictions ","description":"\"Laws and other restrictions can limit or prohibit transferring data.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"65.03.03","quick_ref":"IBM2025","paper_title":"AI Risk Atlas ","level":"Risk Sub-Category","risk_category":"Training Data Risks (Privacy) ","risk_subcategory":"Reidentification ","description":"\"Even with the removal or personal identifiable information (PII) and sensitive personal information (SPI) from data, it might be possible to identify persons due to correlations to other features available in the data.\"","entity":"Other","intent":"Unintentional","timing":"Pre-deployment","domain":2,"subdomain":"2.1"},{"ev_id":"65.05.02","quick_ref":"IBM2025","paper_title":"AI Risk Atlas ","level":"Risk Sub-Category","risk_category":"Training Data Risks (Intellectual property) ","risk_subcategory":"Confidential information in data ","description":"\"Confidential information might be included as part of the data that is used to train or tune the model.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":2,"subdomain":"2.1"},{"ev_id":"65.06.01","quick_ref":"IBM2025","paper_title":"AI Risk Atlas ","level":"Risk Sub-Category","risk_category":"Training Data Risks (Accuracy) ","risk_subcategory":"Data contamination ","description":"\"Data contamination occurs when incorrect data is used for training. For example, data that is not aligned with model’s purpose or data that is already set aside for other development tasks such as testing and evaluation.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"65.06.02","quick_ref":"IBM2025","paper_title":"AI Risk Atlas ","level":"Risk Sub-Category","risk_category":"Training Data Risks (Accuracy) ","risk_subcategory":"Unrepresentative data ","description":"\"Unrepresentative data occurs when the training or fine-tuning data is not sufficiently representative of the underlying population or does not measure the phenomenon of interest.\"","entity":"Other","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"65.07.02","quick_ref":"IBM2025","paper_title":"AI Risk Atlas ","level":"Risk Sub-Category","risk_category":"Training Data Risks (Value alignment) ","risk_subcategory":"Improper data curation ","description":"\"Improper collection and preparation of training or tuning data includes data label errors and by using data with conflicting information or misinformation.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.3"},{"ev_id":"65.08.01","quick_ref":"IBM2025","paper_title":"AI Risk Atlas ","level":"Risk Sub-Category","risk_category":"Training Data Risks (Robustness) ","risk_subcategory":"Data poisoning ","description":"\"A type of adversarial attack where an adversary or malicious insider injects intentionally corrupted, false, misleading, or incorrect samples into the training or fine-tuning datasets.\"","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":2,"subdomain":"2.2"},{"ev_id":"65.19.02","quick_ref":"IBM2025","paper_title":"AI Risk Atlas ","level":"Risk Sub-Category","risk_category":"Output risks (Fairness)","risk_subcategory":"Decision bias ","description":"\"Decision bias occurs when one group is unfairly advantaged over another due to decisions of the model. This might be caused by biases in the data and also amplified as a result of the model’s training.\"","entity":"AI","intent":"Unintentional","timing":"Pre-deployment","domain":1,"subdomain":"1.1"},{"ev_id":"65.22.02","quick_ref":"IBM2025","paper_title":"AI Risk Atlas ","level":"Risk Sub-Category","risk_category":"Non-technical risks (Governance)","risk_subcategory":"Unrepresentative risk testing ","description":"\"Testing is unrepresentative when the test inputs are mismatched with the inputs that are expected during deployment.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":6,"subdomain":"6.5"},{"ev_id":"65.22.03","quick_ref":"IBM2025","paper_title":"AI Risk Atlas ","level":"Risk Sub-Category","risk_category":"Non-technical risks (Governance)","risk_subcategory":"Incomplete usage definition ","description":"\"Since foundation models can be used for many purposes, a model’s intended use is important for defining the relevant risks of that model. As the use changes, the relevant risks might correspondingly change.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":6,"subdomain":"6.5"},{"ev_id":"65.22.04","quick_ref":"IBM2025","paper_title":"AI Risk Atlas ","level":"Risk Sub-Category","risk_category":"Non-technical risks (Governance)","risk_subcategory":"Lack of data transparency ","description":"\"Lack of data transparency is due to insufficient documentation of training or tuning dataset details. \"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":6,"subdomain":"6.5"},{"ev_id":"65.22.07","quick_ref":"IBM2025","paper_title":"AI Risk Atlas ","level":"Risk Sub-Category","risk_category":"Non-technical risks (Governance)","risk_subcategory":"Lack of testing diversity ","description":"\"AI model risks are socio-technical, so their testing needs input from a broad set of disciplines and diverse testing practices.\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":6,"subdomain":"6.5"},{"ev_id":"65.23.07","quick_ref":"IBM2025","paper_title":"AI Risk Atlas ","level":"Risk Sub-Category","risk_category":"Non-technical risks (Societal impact)","risk_subcategory":"Human exploitation ","description":"\"When workers who train AI models such as ghost workers are not provided with adequate working conditions, fair compensation, and good health care benefits that also include mental health.\"","entity":"Human","intent":"Other","timing":"Pre-deployment","domain":6,"subdomain":"6.2"},{"ev_id":"66.04.07","quick_ref":"Li2025","paper_title":"A Closer Look at the Existing Risks of Generative AI: Mapping the Who, What, and How of Real-World Incidents","level":"Risk Sub-Category","risk_category":"Societal and Cultural","risk_subcategory":"Labor exploitation","description":"\"Use/misuse of labour to help train, develop, manage or optimise a technology system or set of systems, including under-paid and/or offshore\"","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":6,"subdomain":"6.2"},{"ev_id":"68.05.00","quick_ref":"Chin2025","paper_title":"Dimensional Characterization and Pathway Modeling for Catastrophic AI Risks","level":"Risk Category","risk_category":"Environmental risk","risk_subcategory":null,"description":"\"AI models are often trained using large amounts of computation. This process is very energy intensive, potentially leading to significant greenhouse emissions depending on the energy sources [132]. Experts believe drastically increasing carbon emissions could accelerate climate change, which may constitute a catastrophic risk [133].\"","entity":"Other","intent":"Unintentional","timing":"Pre-deployment","domain":6,"subdomain":"6.6"},{"ev_id":"72.05.05","quick_ref":"Tse2025","paper_title":"Frontier AI Risk Management Framework (v1.0)","level":"Risk Sub-Category","risk_category":"Model Capabilities ","risk_subcategory":"Situational awareness capability","description":"\"Ability to comprehensively acquire, process and apply meta-information about its own system architecture, modifiable internal processes, and external operating environment, achieving deep understanding of its own state and environmental conditions, thereby conducting efficient environmental adaptation and risk avoidance. Critically, this capability could undermine the efficiency of human testing by enabling AIs to notice when they're being tested and responding accordingly.\"","entity":"AI","intent":"Other","timing":"Pre-deployment","domain":7,"subdomain":"7.2"},{"ev_id":"73.01.02","quick_ref":"Anwar2024","paper_title":"Foundational Challenges in Assuring Alignment and Safety of Large Language Models","level":"Risk Sub-Category","risk_category":"Agentic LLMs Pose Novel Risks ","risk_subcategory":"Natural Language Underspecifies Goals ","description":"\"For LLM-agents, both the goal and environment observations are typically specified in the prompt through natural language. While natural language may provide a richer and more natural means of specifying goals than alternatives such as hand-engineering objective functions, natural language still suffers from underspecification (Grice, 1975; Piantadosi et al., 2012). Furthermore, in practice, users may neglect fully specifying their goals, especially the information pertaining to elements of the environment that ought not to be changed (the classic frame problem (Shanahan, 2016)). Such undersp","entity":"Other","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.1"},{"ev_id":"73.01.05","quick_ref":"Anwar2024","paper_title":"Foundational Challenges in Assuring Alignment and Safety of Large Language Models","level":"Risk Sub-Category","risk_category":"Agentic LLMs Pose Novel Risks ","risk_subcategory":"Safety Risks from Affordances Provided to LLM-agents","description":"\"The capabilities of LLM-agents can be enhanced in significant ways by providing the LLM-agent with novel affordances, e.g. the ability to browse the web (Nakano et al., 2021), to manipulate objects in the physical world (Ahn et al., 2022; Huang et al., 2022a), to create and instruct copies of itself (Richards, 2023), to create and use new tools (Wang et al., 2023a), etc. Affordances can create additional risks, as they often increase the impact area of the language-agent, and they amplify the consequences of an agent’s failures and enable novel forms of failure modes (Ruan et al., 2023; Pan e","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.2"},{"ev_id":"73.08.00","quick_ref":"Anwar2024","paper_title":"Foundational Challenges in Assuring Alignment and Safety of Large Language Models","level":"Risk Category","risk_category":"Vulnerability to Poisoning and Backdoors","risk_subcategory":null,"description":"\"The previous section explored jailbreaks and other forms of adversarial prompts as ways to elicit harmful capabilities acquired during pretraining. These methods make no assumptions about the training data. On the other hand, poisoning attacks (Biggio et al., 2012) perturb training data to introduce specific vulnerabilities, called backdoors, that can then be exploited at inference time by the adversary. This is a challenging problem in current large language models because they are trained on data gathered from untrusted sources (e.g. internet), which can easily be poisoned by an adversary (","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":2,"subdomain":"2.2"},{"ev_id":"74.02.02","quick_ref":"Wang2025","paper_title":"A Survey on Responsible LLMs: Inherent Risk, Malicious Use, and Mitigation Strategy","level":"Risk Sub-Category","risk_category":"Malicious Use ","risk_subcategory":"Jailbreak in LLM Malicious Use - Poisoning Training Data ","description":"\"In the data collecting and pre-training phase, malicious adversaries can Jailbreak LLMs through poisoning their training data to make the model to output harmful content.\"","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":2,"subdomain":"2.2"},{"ev_id":"74.02.03","quick_ref":"Wang2025","paper_title":"A Survey on Responsible LLMs: Inherent Risk, Malicious Use, and Mitigation Strategy","level":"Risk Sub-Category","risk_category":"Malicious Use ","risk_subcategory":"Jailbreak in LLM Malicious Use - Backdoor Attack ","description":"\"However, there are still ones who can leave holes in the training dataset, making LLMs appear safe on average, but generate harmful content under other specific conditions. This kind of attack can be categorized as \"backdoor attack\". Evan et al. developed a backdoor model that behaves as expected when trained, but exhibits different and potentially harmful behavior when deployed [81]. The results show that these backdoor behaviors persist even after multiple security training techniques are applied.\"","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":2,"subdomain":"2.2"},{"ev_id":"74.02.04","quick_ref":"Wang2025","paper_title":"A Survey on Responsible LLMs: Inherent Risk, Malicious Use, and Mitigation Strategy","level":"Risk Sub-Category","risk_category":"Malicious Use ","risk_subcategory":"Jailbreak in LLM Malicious Use - White & Black Box Attacks ","description":"\"In the fine-tuning and alignment phase, elaborately- designed instruction datasets can be utilized to fine-tune LLMs to drive them to perform undesirable behaviors, such as generating harmful information or content that violates ethical norms, and thus achieve a jailbreak. Based on the accessibility to the model parameters, we can categorize them into white-box and black-box attacks. For white-box attacks, we can jailbreak the model by modifying its parameter weights. In [107], Lermen et al. used LoRA to fine-tune the Llama2’s 7B, 13B, and 70B as well as Mixtral on AdvBench and RefusalBench d","entity":"Human","intent":"Intentional","timing":"Pre-deployment","domain":2,"subdomain":"2.2"}]}