Ask a bot to do something and it'll often finish with "Done!" That's a claim, not proof. Managing bots means knowing the difference.
Two kinds of checking
| Verification | Evals | |
|---|---|---|
| Checks | One result | A bot's behavior across many cases |
| Question | Did this job come out right? | Does this bot do this kind of job right? |
| When | Before the result is used | Before you trust the bot, and after changes |
| Shop-floor parallel | Inspecting a part | A first-article and sampling program |
Verification: check the one job
Verification is simple. Before you use the result, check it against what was asked.
Good verification checks the result, not the activity:
- Not "the email command ran." But "the draft is in the drafts folder, addressed to Jim, with the right date."
- Not "the file was saved." But "the file opens, and it's the new version."
- Not "the test passed." But "the test checks the thing we care about."
That last one matters. A check can pass while checking the wrong thing. Always ask what the check is really checking.
Ask for proof with the status
A good bot doesn't just say "done." It shows you: a link, a screenshot, a file path, a short readback of what it wrote. In our review of 90 marketplace templates, the project-manager-type bots were the ones that asked for proof with every status update (good-practice.md). It's a habit worth copying everywhere.
Our AN Creation save skill goes a step further. After saving a record, it reads the file back and checks a fingerprint of its contents, so "saved" means saved.
Evals: check the bot
One good result doesn't prove the next one will be good. Models don't always do the same thing twice. So for anything you'll rely on, you test the bot on a set of cases.
That's an eval. It can be as simple as:
- Write ten realistic requests, including two tricky ones.
- Write down what a good answer looks like for each.
- Run the bot on all ten.
- Score them. Pass, fail, or "needs a person."
Run it again whenever you change the bot's instructions or its model.
Who does the checking?
- A person for anything high-stakes: money, customers, public posts.
- A tester bot for routine runs against a script. See Your tester bot is your internal auditor.
- Plain code for exact checks: does the file exist, does the number add up, is the date in the right format. Code doesn't guess.
The bot that did the work shouldn't be its only checker. In ISO terms, inspection is independent of production.
Where this differs from a factory
A gauge on a factory floor can be calibrated against a standard and trusted until the next calibration. A bot can't. Its output varies, so evals work more like sampling inspection than calibration. You keep checking. See Measure what matters.
What this means for you
- Replace "done?" with "show me."
- Check the result, not the activity.
- Keep a small test set for every bot you rely on.
- Rerun it after every change.
Proof is cheap. Rework is not.