Most problem meetings start with memories: "it happens all the time", "it's always the night shift". A check sheet replaces them with a count. You decide in advance what you're counting, how each category is defined, where and for how long. Then you tally.
A check sheet records occurrences. It isn't a checklist of questions. The two get mixed up, and only one of them gives you data.
Where it comes from
The check sheet is one of the seven basic quality tools first highlighted by Kaoru Ishikawa, along with the fishbone, Pareto chart, histogram, scatter diagram, control chart and stratification. ASQ notes that more current lists swap stratification for the flowchart. The check sheet's partner is the data collection plan: the written decision about what to count, from which population, over what period, and who counts it.
How a bot helps
Customers say invoices are wrong. Before anyone blames sales or billing, have a bot draft the plan: every credit memo for the last 8 weeks, tallied by reason (wrong price, wrong quantity, wrong customer, duplicate, other), by department that issued the original, and by week. Each reason gets a one-line definition and an example, so two people would put the same memo in the same box.
The bot can then do the tally from the records, quickly and in full, which people rarely have time for. A person spot-checks a sample of its classifications.
On the bots' own work
The same tool works on a bot team: tally every correction Mike makes to a bot's output for two weeks, by type. "It keeps getting things wrong" becomes "14 of 31 corrections were stale data".
When to use it
Whenever a problem is being argued from impressions. It's a light tool. The definitions take an hour; the counting takes as long as the period you choose. Feed the result to stratification and the Pareto chart.
Where bots differ
A bot can only count what's in the records. Work that never makes it into a system, such as a phone call that fixed an order or a part reworked at the bench, is invisible to it. Say so on the sheet, and have a person observe for a few shifts to see what the records miss.
And a bot classifying free text will be confidently consistent, including when it's consistently wrong. Check a sample by hand before trusting the counts.
Sources: ASQ: seven basic quality tools; ASQ: check sheet.
Back to Engineering tools for bots.
Short outline
Check sheet: short outline
What it's for
A check sheet tallies occurrences of a problem by category, using written definitions agreed before counting starts. It turns impressions into counts that stratification, Pareto and run charts can use.
Columns
| Column | What goes in it |
|---|---|
| Question | What the count should answer, in one line |
| Population and period | Which records or events, from when to when |
| Category | One row per category of occurrence |
| Definition | What counts in this category, with one example and one near-miss that does not |
| Source | Where each count comes from: a system report, a log, or observation by a named person |
| Count by period | The tally, one column per day or week |
| Total | The row total |
| Not captured | What this source cannot see, and who will observe it |
| Checked by | Who hand-checked a sample of the classifications, and how many agreed |
Steps, each with its "done when"
- Write the question and the population. Done when one question, one population and one period are written down before any counting.
- Define every category. Done when each category has a definition, an example and a near-miss, and there is an "other" row with a rule for when it gets too big.
- Say where each count comes from and what it misses. Done when every category names its source, and the "not captured" column is filled or says "none found".
- Count. Done when every period in the plan has a count in every row, and blanks say "none" or "not collected".
- Check a sample of the classifications. Done when a person has re-classified at least 20 items, or all of them if fewer, and the agreement rate is recorded.
Common mistakes
- Confusing a check sheet with a checklist of questions.
- Starting to count before the categories are defined, then redefining them halfway.
- An "other" row that ends up holding a third of the count.
- Counting only what the system records and calling it the whole picture.
- Trusting a bot's classification of free text without a hand-checked sample.