AI Systems
Human-in-the-loop AI: review is a state, not a pause
One step needs judgment; the rest doesn't
Many processes could run on AI except for one step: approving a quote, checking an extracted figure, answering a sensitive message. Human-in-the-loop design lets everything around that step run on its own while the judgment stays with a person, at a defined point, with a record of what was decided.
The pattern is short. Rules or a model prepare a draft; it's validated and saved; the case is marked awaiting review; a reviewer approves, edits or rejects; the decision is saved; the next step runs. Two things make it reliable: the review is stored state rather than a paused script, and the reviewer gets a screen built for deciding.
Review is not an error screen
Review · Quote Q-212
J-442 · H.R., Northside · drafted 17:38 from T3's notes
- Water heater, 50 gal, installed$1,840
- Labour · 2 h$260
- Haul away old unit$95
- Total$2,195
- T3 job notes, 16:55
- 3 photos
- Price list v8
Labour estimated from notes: “tight closet, old unit seized”. Check hours.
Approving emails Q-212 to H.R. and sets a follow-up for Thursday.
Approve, Edit, Reject
What's proposed
The draft itself, laid out the way the customer will see it.
Where it came from
The notes, photos and price list it was drafted from, one click away.
What it wasn't sure about
Anything inferred rather than read is flagged, so attention goes where it's needed.
What approving will do
The consequence, stated before the click.
Three ways out
Approve, edit, or reject with a reason. None of them is a dead end.
Three decisions, three records
Q-212 · approved
- status
- approved
- reviewer
- operations manager
- at
- 08:15
- changes
- none
- next
- quote emailed to H.R., follow-up set
- feedback
- counted as an accepted draft
Q-212 · edited and approved
- status
- approved with edits
- reviewer
- operations manager
- at
- 08:12–08:15
- changes
- labour 2 h → 3 h, total $2,195 → $2,325
- next
- edited quote emailed to H.R.
- feedback
- the edit is saved as a test case for the drafting prompt
Q-212 · rejected
- status
- returned for redraft
- reviewer
- operations manager
- at
- 08:14
- reason
- customer asked about a tankless option
- next
- nothing sent; T3 asked to confirm the space for tankless
- feedback
- reason recorded against the draft
How much review is enough
01
Spot-check a sample
When: mistakes are cheap and easy to undo
Example: internal job summaries
02
Review what it's unsure about
When: most cases are routine and a few aren't
Example: extracted invoice totals that don't match the purchase order
03
Review every one
When: it reaches a customer or moves money
Example: quotes, refunds, apology emails
04
Keep it with a person
When: the judgment is the job
Example: disputes, safety calls
What human-in-the-loop AI hands off
In
Out
Worth building where a mistake is costly
Good fit
- A mistake would be costly, visible to customers, or hard to reverse.
- There's enough volume that preparing the work automatically saves real time.
- Someone owns the decision and can be reached.
Not the right fit
- Every step needs review. That's a manual process with extra software.
- Reviewers approve everything without looking. The rules need fixing instead.
Questions about human-in-the-loop AI
What if the reviewer doesn't respond?
The case stays in awaiting review with its draft and context saved. Reminders and escalation to a second reviewer are part of the design, on times you choose.
Can reviewers change the draft, or only approve it?
They can edit. The original and the edited version are both kept, so you can see what the AI proposed and what went out.
Doesn't review cancel out the time saved?
Checking a prepared draft is much quicker than writing one, and everything around the decision runs on its own. If reviewers start approving without looking, the review level is wrong for that step.
Where does the review happen?
Where the reviewer already works: an internal screen, a notification that opens the case, or a queue in a tool you already use. What matters is that it shows the context and what approving will do.