You can't calibrate a bot like a micrometer. That's true. But the bot isn't the gauge. The bot is the process. Quality has two well-proven tools for exactly that situation.
Part 1: SPC, watching the process
Statistical process control was developed by Walter Shewhart at Bell Labs in the 1920s. Its main tool is the control chart: a running plot of a measure over time, with a center line and control limits calculated from the process's own normal variation.
The chart answers one question: has something really changed, or is this normal noise?
- Points bouncing inside the limits: normal variation. Don't react to every wobble.
- A point outside the limits, or a run of points on one side of the center: something changed. Go find it.
For bots, the classic signal is a silent model update. Nothing in your setup changed, but the provider updated the model. The correction rate creeps up. A control chart catches that.
Good bot measures for a chart:
- Percent of drafts needing edits, per week.
- Items held at the gate, per week.
- Eval pass rate, per run.
These are counts and percentages, so the right chart is usually a p-chart (proportion defective). It works at modest volumes, such as tens of items a week.
Part 2: MSA, checking the checker
Measurement system analysis asks: can we trust the gauge? A gauge that disagrees with itself, or with other gauges, makes every measurement suspect.
The standard study is Gauge R&R:
- Repeatability. Same gauge, same part, measured several times. Does it agree with itself?
- Reproducibility. Different operators, same parts. Do they agree with each other?
For pass/fail checks, the equivalent is attribute agreement analysis: do checkers agree with each other, and with a known right answer?
For bots, the gauge is whatever checks the work: a tester bot, a bot grading another bot's output, a rubric, a person.
| MSA question | Bot version |
|---|---|
| Repeatability | Run the checker on the same 20 items 3 times. Same verdicts? |
| Reproducibility | Have two different checkers (another model, or a person) grade the same 20 items. Do they agree? |
| Bias | Compare the checker against 20 items with known right answers. |
A tester bot that passes everything, or flips its verdict on a rerun, is a broken gauge.
Why people skipped it
Both tools need data collection and some statistics. Small offices rarely have someone who does either.
How a bot does it
A bot can compute control limits, plot the chart and flag signals. It can also set up and analyze an agreement study. People supply the known right answers and decide what to do about a signal.
The bot brief
SPC: Using [weekly counts: items checked, items failing] from [file]:
build a p-chart. Use the first [n] weeks to set the center line and limits.
Flag any point outside the limits, and any run of 7 or more on one side of the center.
For each signal, list what changed around that date in [change log, provider release notes].
MSA: Using [checker verdicts file] (each item checked [3] times by [checker A],
once by [checker B], plus a KNOWN ANSWER column from a person):
report repeatability (% self-agreement), reproducibility (% A-B agreement),
and accuracy vs known answers. Show the items where checkers disagree.
Don't conclude a gauge is fine from fewer than 20 items; say so if that's all we have.Starting point only. Test it on your own data and model, and adjust as needed before you rely on it.
An example
A tester bot rechecks 20 drafts three times. It gives the same verdict on 17 of 20 (85%). Compared with a person's answers, it agrees on 16 of 20 (80%). Three of the four misses are drafts with a wrong unit price that it marked as passing.
So the gauge is weak exactly where it matters most. The fix: add a plain-code check that compares unit prices to the price file. Code doesn't guess. See Verification and evals.
Where bots differ
On a factory floor, the gauge and the process are separate things. With bots, it's tempting to have the same model do the work and grade it. That's like a machine inspecting its own parts with its own sensor. A checker on a different model, or a person, gives you real reproducibility.
What this means for you
- Chart one or two bot measures weekly. React to signals, not wobbles.
- Treat model updates as a reason to watch the chart closely.
- Run an agreement study on your checker before you trust it.
- If the checker is weak, add a code check or a person where it matters.