Skip to content

AI Systems

AI classification: every label needs a “not sure”

A mixed inbox of bookings, reschedules, billing questions and the occasional emergency has to be sorted before anyone can act. Classification labels each message from a list you wrote, with a confidence. Below the threshold it says “not sure” and a person decides, and the labels that matter most are set by rules, not by the model.
Discuss your systemSee where to draw the line ↓
BookRescheduleBillingEmergencyrule: gas, smoke, floodingReviewa person decidesRescheduletext · 555-0142Can you move Thursday to Friday?reschedule .94Billingemail · L.P.INV-2207 was charged twice??billing .91Bookform · 14 Elm StDripping tap in the kitchenbook .88Emergencyrule: gas, smoke, floodingtext · 555-0163I smell gas near the heaterbook .55rule: emergencyReviewa person decidesemail · S.T.Worth repairing or replacing?not sure .48

A label is a decision someone acts on

Sorting the inbox looks like admin, but every label sends a message somewhere: to the scheduler, to finance, to an on-call technician. A wrong label means a customer waits in the wrong queue.

AI classification reads each message and picks a label from a fixed list, with a confidence. Its output is checked against the list, rules for the cases that must never be wrong come after it, and anything it isn't sure about goes to a person with its best guess shown.

Write the labels down first

Each label has a meaning and a line it doesn't cross. The model chooses from this list, and only this list.
BookA new visit for a new problemA change to a visit already booked (Reschedule)
RescheduleMove or cancel a visit already bookedA new problem at the same address (Book)
BillingA question or dispute about an invoice or paymentA price question before any work (Book)
EmergencyGas, smoke, flooding, or no heat in freezing weather“Urgent” without a safety risk (Book, high priority)
Not sureAnything the model can't place with confidenceA guess at the nearest label

Where to draw the line

Every confidence threshold trades review work against mistakes that slip through. We set it per label, from a week of real messages a person has checked. Here is one week, at three thresholds.
Labelled automatically
182
Sent to review
18
Wrong labels that got through
9

Little review work, but nine customers sat in the wrong queue that week.

What keeps a label honest

  1. Only labels from the list

    A fixed set of answers

    The output is checked against the list. Anything else is rejected and retried, never passed on.

  2. Rules outrank the model

    Some calls aren't the model's

    Safety words go to Emergency whatever the model says. Rules come after the model and win.

  3. “Not sure” is an answer

    Below the line, a person

    The message goes to review with the model's best guess and its confidence shown.

  4. Corrections become tests

    Every relabel counts

    Each label a person changes is saved as a test case for the next version.

What classification hands off

In

Message
email, text, form, transcript
Label list
with definitions
Rules
safety words, key customers
Classification
label + confidence → rules

Out

Label
for routing
Review queue
with the best guess
Corrections
saved as test cases

Worth building when the inbox needs sorting before anyone can act

Good fit

  • Messages arrive in volume and need sorting before anyone acts.
  • The labels can be written down and don't change every week.
  • The labels that would be costly to get wrong can be covered by rules.

Not the right fit

  • The categories are fuzzy or disputed. Agree them first.
  • Volume is low enough that whoever reads the inbox can sort it as they go.

Where it's built in

Next to classification

Questions about AI classification

How is this different from keyword rules?

Keyword rules miss anything phrased differently. The model reads what the message means. Rules stay for the cases where you want certainty, like safety words, and they run after the model.

Can a message have more than one label?

If the list allows it. A reschedule that also disputes an invoice can carry both, and each goes where it belongs. That's decided when the labels are written.

How do you choose the threshold?

From a week or two of your real messages, labelled by a person. That shows the trade-off between review work and slips, and you choose it per label.

What happens to the “not sure” messages?

They wait in a review queue with the model's best guess and its confidence. Confirming or changing the label takes one click, and that choice is saved.