Experiment: AI vs Human Alert Triage

The 30-second explanation

Take a batch of old alerts where you know the real outcome. Have the assistant triage them, hide the answers, then compare its suggestions with what actually happened.

How it works

1. 1

   1. Build the set

   Closed alerts with known outcomes, redacted.

2. 2

   2. Run

   Same prompt and output format for each.

3. 3

   3. Compare

   Suggestions against known outcomes.

4. 4

   4. Review

   Study misses and weak reasoning first.

Choose a small set of closed alerts with known outcomes, including harmless ones and real incidents, and remove sensitive data. Give each to the assistant with the same structured prompt. Record its suggested priority and reasoning, and compare it with the analyst's final verdict. Pay most attention to real incidents the assistant downgraded, because those cost the most. Also note whether its reasoning was sound even when the verdict was right. This describes a method for you to run: it does not report results.

Real-world example

How to spot it

• Only measuring accuracy

  Ignoring which errors were the costly kind.

• Test set too easy

  Mostly obvious alerts.

• Leaked answers

  The outcome visible in the alert text.

• Small, unrepresentative samples

  Conclusions drawn from a handful of cases.

What to do

1. 1Include hard and real incidents in the set, not only routine alerts.
2. 2Track missed real threats separately from false alarms.
3. 3Re-run the test when the model, prompt or alert types change.