MIT AI Risk Repository · domain 4: Malicious actors
4.1 Disinformation, surveillance, and influence at scale
Using AI systems to conduct large-scale disinformation campaigns, malicious surveillance, or targeted and sophisticated automated censorship and propaganda, with the aim to manipulate political processes, public opinion and behavior.
- 84
- 12
- 138
- 132
| Label | Value |
|---|---|
| Human | 59 |
| Other | 14 |
| AI | 9 |
| Not coded | 2 |
| Label | Value |
|---|---|
| Intentional | 73 |
| Other | 9 |
| Not coded | 2 |
| Label | Value |
|---|---|
| Post-deployment | 74 |
| Other | 8 |
| Not coded | 2 |
| Label | Value |
|---|---|
| 2015 | 2 |
| 2016 | 1 |
| 2017 | 1 |
| 2019 | 2 |
| 2020 | 6 |
| 2021 | 3 |
| 2022 | 6 |
| 2023 | 21 |
| 2024 | 50 |
| 2025 | 38 |
| 2026 | 8 |
| Label | Value |
|---|---|
| Risk Category | 10 |
| Risk Sub-Category | 74 |
Risk entries
Browse and export all- Coercion/manipulation
"Coercion/manipulation - Use of a technology system to covertly alter user beliefs and behaviour using nudging, dark patterns and/or other opaque techniques, resulting in potential erosion of privacy,...
- Political and Economic
"Political and Economic - Manipulation of political beliefs, damage to political institutions and the effective delivery of government services."
- Electoral interference
"Electoral interference - Generation of false or misleading information that can interrupt or mislead voters and/or undermine trust in electoral processes."
- Political manipulation
"Political manipulation - Use or misuse of personal data to target individuals’ interests, personalities and vulnerabilities with tailored political messages via micro-advertising or deepfakes/synthet...
- Political
"In the UK, a form of initial computational propaganda has already happened during the Brexit referendum1 . In future, there are concerns that oppressive governments could use AI to shape citizens’ op...
- Biased influence through citizen screening and tailored propaganda
"AI-powered chatbots tailor their communication approach to influence individual users' decisions. In the UK, a form of initial computational propaganda has already happened during the Brexit referend...
- Surveillance and Censorship
"Content moderation has emerged as one of the key use-cases of LLMs (Weng et al., 2023), indicating the potential of LLMs for surveillance and censorship as well (Edwards, 2023). Surveillance and cens...
- Disinformation and manipulation of public opinion
"AI, particularly general- purpose AI, can be maliciously used for disinformation (351), which for the purpose of this report refers to false information that was generated or spread with the delibera...
- Manipulation of public opinion
"Malicious actors can use general- purpose AI to generate fake content such as text, images, or videos, for attempts to manipulate public opinion. Researchers believe that if successful, such attempts...
- Developments in AI enable actors to undermine democratic processes
"Developments in AI are giving companies and governments more control over individuals’ lives than ever before, and may possibly be used to undermine democratic processes. We are already seeing how th...
- AI is used to scale up production of false and misleading information
"At the same time, we are seeing how AI can be used to scale up the production of convincing yet false or misleading information online (e.g. via image, audio, and text synthesis models like BigGAN [6...
- AI's persuasive capabilities are misused to gain influence and promote harmful ideologies
"As AI capabilities advance, they may be used to develop sophisticated persuasion tools, such as those that tailor their communication to specific users to persuade them of certain claims [42]. While...
- Disinformation and Influence Operations
"In addition to unintentional degradation of the information environment (discussed in the section on Societal Harms above), frontier AI can be misused to deliberately spread false information to crea...
- Information Manipulation
"generative AI tools can and will be used to propagate content that is false, misleading, biased, inflammatory, or dangerous. As generative AI tools grow more sophisticated, it will be quicker, cheape...
- Disinformation
"Bad actors can also use generative AI tools to produce adaptable content designed to support a campaign, political agenda, or hateful position and spread that information quickly and inexpensively ac...
- Propaganda - Extremist schemes
-
- Information Manipulation
"The distortion of the information ecosystem, including the spread of misinformation, fake news, and other forms of deceptive content [28], is categorized as “Information Manipulation.”"
- Deception - Information control
-
- Propaganda - Influence campaigns
-
- Dishonesty - Information disorder
-
- Socio-technical and Infrastructural
"Lastly, broader harms that can impact communities, societal structures, and critical infrastructures, including threats to democratic processes, social cohesion, and technological systems, are captur...
- Propaganda - Synthetic realities
-
- Dishonesty - Targeted surveillance
-
- AI-Powered Spear-Phishing at Scale
"Phishing is a type of cybersecurity attack wherein attackers pose as trustworthy entities to extract sensitive information from unsuspecting victims or lure them to take a set of actions. Advanced AI...
- Harmful Content Generation at Scale (General)
"While harmful content like child sexual abuse material, fraud, and disinformation are not new challenges for governments and developers, without the proper safety and security mechanisms, advanced AI...