MIT AI Risk Repository · Risk Sub-Category · 62.15.07

Fine-tuning related (Poisoning models during instruction tuning)

Category: Model Development

Open the MIT Risk Navigator All entries from Gipiškis2024

Description

"AI models can be poisoned during instruction tuning when models are tuned using pairs of instructions and desired outputs. Poisoning in instruction tuning can be achieved with a lower number of compromised samples, as instruction tuning requires a relatively small number of samples for fine-tuning [155, 211]. Anonymous crowdsourcing efforts may be employed in collecting instruction tuning datasets and can further contribute to poisoning attacks [187]. These attacks might be harder to detect than traditional data poisoning attacks."

From Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems (Gipiškis2024), as extracted by the MIT AI Risk Repository (CC BY 4.0).

Classification

Domain: 2. Privacy & Security
Subdomain: 2.2 AI system security vulnerabilities and attacks
Causal entity: Human
Intent: Intentional
Timing: Pre-deployment

Subdomain definition: Vulnerabilities in AI systems, software development toolchains, and hardware that can be exploited, resulting in unauthorized access, data and privacy breaches, or system manipulation causing unsafe outputs or behavior.

Real-world incidents in this subdomain

Browse all incidents in this subdomain

How other frameworks describe this risk

Privacy loss
A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms (Abercrombie2024)
Exploiting Limited Generalization of Safety Finetuning
Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)
Jailbreaks and Prompt Injections Threaten Security of LLMs
Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)
“Model Psychology” Attacks
Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)
Attacking LLMs via Additional Modalities a
Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)
Vulnerability to Poisoning and Backdoors
Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)
Model Attacks
Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)
Hardware Vulnerabilities
Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)

Other entries from Gipiškis2024

Dimension - Intent Risk Category
Intentional Risk Sub-Category
Unintentional Risk Sub-Category
Partially intentional Risk Sub-Category
Dimension - Entity Risk Category
Human Risk Sub-Category
AI Risk Sub-Category
Combination of humans and AI Risk Sub-Category
Dimension - Failure dynamics Risk Category
Isolated (non-normal) failures Risk Sub-Category
System (normal) failures Risk Sub-Category
System (normal) failures Additional evidence