Model Poisoning: Teaching an AI the Wrong Lessons on Purpose

The 30-second explanation

If you teach a student from a book with a few pages secretly rewritten, they will confidently repeat the wrong things. Poisoning does that to AI.

How it works

1. 1

   1. Find a source

   Identify data the model relies on.

2. 2

   2. Tamper

   Insert misleading or malicious content.

3. 3

   3. Learn or retrieve

   The system uses the tampered data.

4. 4

   4. Behave

   It produces the attacker's desired output.

Models learn from large amounts of data, and increasingly retrieve extra information at answer time. An attacker who can influence those sources, for example by editing public pages or contributing malicious examples, can shift behaviour. The tampering may be subtle and only show up for a specific input. Defenders track where data comes from, check what goes in, test models for unexpected behaviour, and restrict who can change datasets.

Real-world example

How to spot it

• Unexplained accuracy changes

  A sudden shift in behaviour after a data update.

• Unknown data sources

  Content with no clear origin or owner.

• Odd behaviour on specific inputs

  A trigger phrase that changes results.

• Uncontrolled edit access

  Many people able to change the data the AI trusts.

What to do

1. 1Record where every dataset comes from, and restrict who can change it.
2. 2Test models against a fixed set of checks after each data or model update.
3. 3Treat externally sourced content as untrusted until it has been reviewed.