"Is the bot working?" is hard to answer without numbers. But the numbers have to be the right ones, and you have to be able to trust them.
Objectives and measurement in ISO
ISO has two relevant pieces:
- Quality objectives: measurable goals, with owners, deadlines and a way to check them.
- Monitoring and measurement: decide what to measure, how, when, and how to analyze it.
And one more: monitoring and measuring resources, which includes calibration. Gauges are checked against a standard so measurements can be trusted.
Success criteria before the run
The most useful bot measure is the simplest: write down what success looks like before the bot starts.
- "Every invoice over $500 is flagged. Zero missed."
- "Each draft reply is under 150 words and answers the customer's question."
- "Every number in the report has a source and date."
If you can't write it down beforehand, you can't check it afterward. And the bot can't aim for it.
KPIs worth tracking
Few standard KPIs exist for bots yet. These are practical ones to consider, not industry benchmarks:
| Measure | What it tells you |
|---|---|
| Corrections per week | How often you fix the bot's work |
| Repeat corrections | Whether fixes are making it into skills |
| Items held at the gate | How often output isn't ready to ship |
| Test pass rate | How the bot does on its eval set |
| Unsourced claims found | How well it follows "don't make it up" |
| Cost per task | Whether it's efficient |
Repeat corrections matter most. A correction that keeps coming back means the cause wasn't fixed. See Fix the cause, not the answer.
Evals: your measurement system
An eval is a fixed set of test cases with known good answers. You run the bot on it and score the results. See Verification and evals.
Rerun it:
- after changing a skill or principle
- after a model update
- on a schedule, for bots you depend on
Keep the results with the version of the bot that produced them. Our AN Creation tester keeps one line per run in an index: date, script, bot, passed, failed, not reached.
Where bots differ: calibration
This is the weakest parallel in the whole Tools for bots series, and it's worth saying plainly.
A gauge is stable. You calibrate it against a standard, and you can trust it until the next calibration date.
A model isn't like that. Give it the same input twice and it can give different outputs. Change its version and it can behave differently on the same test. So you can't "calibrate" a bot once and trust it for a year.
Evals are closer to sampling inspection: you test a sample, regularly, and watch for drift. The trust is ongoing, not certified.
Measuring the measurer
If a bot checks another bot's work, who checks the checker? Now and then a person should review a sample of the tester's verdicts. A check that passes everything is a broken gauge. That's checking the gauge: compare the checker's verdicts against a person's on the same sample.
What this means for you
- Write success criteria before each important task.
- Track a few simple measures, especially repeat corrections.
- Keep an eval set, and rerun it after changes.
- Treat evals as sampling, not calibration.
- Check the checker now and then.
Good measurement doesn't make a bot perfect. It tells you, honestly, how close it is.