Most people test by changing one thing, then another, then another. It feels careful. It's slow, and it misses the cases where two things only matter together.
What it is
Design of experiments (DOE) was developed by Ronald Fisher in the 1920s and 1930s, first for agricultural field trials in England. It spread into industry, and Genichi Taguchi later popularized a version in Japanese manufacturing.
The core idea: decide in advance which factors you'll change, at which levels, and run a planned set of combinations. Then the analysis can tell you:
- Main effects. How much each factor matters on its own.
- Interactions. When the effect of one factor depends on another.
A simple example is a 2-level full factorial. Three factors, each at two levels (low and high), gives 2 × 2 × 2 = 8 runs. Eight runs tell you about all three factors and how they interact. One-at-a-time testing can't show interactions at all.
Why people skipped it
Planning a DOE takes statistical knowledge. Analyzing it takes more. Most shops don't have someone who does it regularly, so they test one factor at a time, or not at all.
How a bot does it
A bot can plan the runs, randomize the order, and do the analysis. People run the actual tests and supply the results.
- Define the response. What you're measuring, and how.
- List the factors and levels. People choose these; they know what's practical.
- Have the bot design the run table, in random order, with replicates if you can afford them.
- People run the tests and record the results.
- Have the bot analyze: main effects, interactions, and how sure we can be.
- People confirm the best setting with a few extra runs.
It works on bots, too
Bot setups have factors: model, reasoning setting, instruction wording, whether examples are included. A small designed experiment tells you which of them actually change the result. Because bot output varies from run to run, replicates matter here.
The bot brief
Plan a designed experiment.
Response to measure: [what, how, units]. Factors and levels: [list, from us].
Runs we can afford: [n].
1. Propose a design (full or fractional factorial) that fits the budget.
Explain what it can and can't tell us (which effects are confounded).
2. Give the run table in randomized order, with run numbers.
Later, given results [file]:
3. Compute main effects and two-way interactions. Show the calculation.
4. Say which effects are clearly bigger than run-to-run noise, and which aren't.
5. Recommend a confirmation run. Don't overstate small samples.Starting point only. Test it on your own data and model, and adjust as needed before you rely on it.
An example
Testing a customer-reply bot. Response: percent of drafts a person sends without edits, scored on the same 20 test emails.
| Factor | Low | High |
|---|---|---|
| A: Model | Smaller | Larger |
| B: Examples in the skill | None | Three |
| C: Reasoning setting | Low | High |
Eight runs, each scored on all 20 emails. Suppose the analysis shows that adding examples (B) matters most, the larger model (A) helps a little, and reasoning (C) does nothing for this task. Then the cheap fix is examples, not a bigger, slower model. (Illustrative outcome; your results will be your own.)
Where bots differ
The statistics are easy for a bot. Two cautions:
- The bot shouldn't choose the factors alone. People know what's practical and what's already been ruled out.
- Small samples. A bot can compute an effect to three decimals from eight runs. That doesn't make it real. Ask how big the effect is compared to the noise.
What this means for you
- When several things might matter, test them together, by plan.
- Let the bot design and analyze. People pick the factors and run the tests.
- Use replicates when results vary run to run, as bot results do.
- Confirm the winner before rolling it out.