Skip to content

AI Systems

Multimodal AI: the photo, the call and the text, read together

Customers describe a problem in whatever way is easiest: a photo of the nameplate, a rushed voicemail, a text with the address. A multimodal system reads all three into one record. Each fact keeps a note of where it came from, what can be checked is checked, and when the inputs disagree, a person decides.
Discuss your systemSee a nameplate read ↓
Photo · nameplate.jpgVoicemail · 0:38Text · 14:12NORTHFIELDMODEL NF-50GSERIAL NFG 1212 A0441750 GAL · NATURAL GASINPUT ··· BTU/HR“…leaking from the bottom,it's maybe five years old…”9 Alder Court. Call after 4 please.Request R-1462Address9 Alder CourttextProblemleak at the basevoicemailUnitNorthfield NF-50GphotoFuelnatural gasphotoMadeDecember 2012photoCallbackafter 4 pmtextcaller: ~5 yrs · check on site

What each input is good for

People already send photos and voicemails because typing is harder. Each kind of input is reliable for some facts and poor for others, and the system should know which.

  1. 01

    Photos

    What's printed or visible

    Nameplates, labels, meter readings, visible damage. Printed text is usually the most reliable fact in the whole request.

    glare, angles and photos of the wrong unit.

  2. 02

    Voicemails and calls

    What happened, and how urgent

    The story in the customer's words, and whether they sound like it can wait. Transcribed only where callers are told.

    names, numbers and addresses, which transcription often mishears.

  3. 03

    Scans and handwriting

    Paper that already exists

    Signed job sheets, inspection forms, handwritten notes from the van.

    handwriting that should go to a person instead of a guess.

  4. 04

    Drawings and plans

    Layout and measurements

    A floor plan for an install quote, a sketch of where the unit sits. Useful for counting and locating.

    scale. Measurements read from a drawing need checking before anyone orders parts.

Reading a nameplate

Every field is read, then checked against something other than the model. One field can't be read, so it stays empty.

Nameplate photo, as sent by the customer

  1. 1BrandNorthfieldmanufacturer list
  2. 2ModelNF-50Gthe model's published format
  3. 3SerialNFG 1212 A04417the serial format
  4. 4MadeDecember 2012decoded from the serial
  5. 5Capacity50 gallonsmatches the model
  6. 6Fuelnatural gasmatches the model
  7. 7Input rating(empty)unreadable: glareAnother photo requested by text

Six fields filled and checked, one left empty with a request for a better photo. Nothing is filled from what the model expects a 50-gallon heater to say.

When the inputs disagree

Caller says “about five years old”; the nameplate says 2012The nameplate date, with the caller's estimate notedPrinted at the factory; memory isn't
The photo shows a different unit from the one on fileNeither. Flagged for the technicianIt may be a replacement nobody recorded
The voicemail transcript hears “Elder Court”; the text says “Alder Court”The typed address, confirmed in the replyTyped beats transcribed for names and numbers
The serial is too blurred to readNothing, and a request for another photoAn empty field is better than a guess

What a multimodal step hands off

In

Photos
nameplates, damage, displays
Audio
calls, voicemails
Text
messages, forms
Model + checks
read → match → reconcile

Out

Record fields
each with its source
Conflicts
for a person to settle
Requests
another photo, a confirmation

Worth building when people already send photos and voicemails

Good fit

  • Customers or staff send photos and voicemails, and someone retypes them.
  • The facts you need are printed somewhere: nameplates, labels, meters.
  • What's read can be checked against your records or a known format.

Not the right fit

  • A form with the right fields would collect the information more reliably.
  • The decision needs a trained eye on site, like diagnosing a fault from a photo.

Where it reads more than text

Next to multimodal AI

Questions about multimodal AI

Can it diagnose a fault from a photo?

We don't build it to. It reads what's printed and visible, like a model number or water at the base, and leaves diagnosis to a technician.

What happens to the photos?

They're stored with the job, under the same access rules as your other records, and only the fields you need are extracted.

Are voicemails transcribed?

Where callers are told calls may be recorded and transcribed. The transcript is kept with the audio so anyone can check what was said.

How reliable is it?

It depends on your photos and your cases, so we measure it on a set of your real examples before relying on it. Anything it can't read stays empty.