ASQ describes Six Sigma as a disciplined, data-driven method for improving processes by reducing variation and defects. The famous number is 3.4 defects per million opportunities. Lean Six Sigma combines it with Lean's focus on waste.
For bots, the number matters less than the method.
DMAIC
Six Sigma projects follow five steps. The Lean Enterprise Institute lays them out like this:
- Define. What's the problem, and what's the goal?
- Measure. Collect data. What's the baseline?
- Analyze. Use the data to find root causes.
- Improve. Develop and test fixes for those causes.
- Control. Keep the fix in place and keep monitoring.
Why it helps a bot
Asked to "fix" something, a bot tends to jump straight to Improve. It proposes a fix in the first reply. DMAIC holds it back: define first, measure, then analyze, then fix.
An example
Define. Customer replies drafted by a bot get sent back for edits too often. Goal: fewer than one in ten need edits.
Measure. Over two weeks, 18 of 50 drafts needed edits. Baseline: 36%.
Analyze. Sort the 18 by reason. Twelve had the wrong tone for repeat customers. Four had a wrong product name. Two were other. A Pareto view says tone is the big one.
Improve. Add a step to the skill: check the customer history first. Test on the 50 old cases. Edits drop to 6 of 50, or 12%. That's better, but it misses the goal, and those are the same cases used to find the cause. Run it on fresh drafts before calling it fixed.
Control. Add a line to the control plan: track the edit rate weekly, and react if it goes over 15%. Note the change in the skill's revision history.
Control is the step people skip
Teams fix the problem, celebrate and move on. Six months later it's back. Control is what stops that. For bots, it's a check that keeps running and a record of what changed.
Where bots differ
Six Sigma leans on statistics: capability, control charts, designed experiments. Most bot jobs don't have the volume for that. Fifty drafts is a small sample. Be honest about it. Use the structure, and don't dress up small numbers as statistics.
Also, a bot's output varies from run to run even with the same input. That is a kind of variation Six Sigma was built to study, but measuring it well takes more runs than most people do.
Back to Engineering tools for bots.