Skip to content

Automation & Orchestration

Workflow monitoring and recovery: no automation fails quietly

An automation that fails silently is worse than none, because people stop checking. Monitoring makes every run visible. Failures, retries and stuck cases are caught by the system rather than reported by a customer, and a failed run is repaired from the step that failed instead of started again.
Discuss your systemRepair one failed run ↓
Runs · todayr-2301Close-out J-420runningsucceededr-2302Close-out J-421runningretrying 1/3succeededr-2303Payment sync P-77runningstuck · no progress 2 hr-2304Close-out J-422runningfailed at Create invoicerepaired · resumedsucceededAlertsr-2304failed at Create invoiceoffice lead toldr-2303no progress for 2 hoffice lead told

People stop checking

The first few weeks after an automation goes live, someone checks it every morning. Then it works for a month, and they stop. That's exactly when it fails: an expired password, a renamed field, a timeout. The first sign is a customer asking where their invoice is.

Monitoring replaces the morning check. Every run has a record and a status; anything failed or stuck reaches a named person, in a channel they actually watch, with enough context to fix it.

Succeeded
46
Running
3
Retrying
1
Stuck
1
Failed
0

Repair one failed run

Run r-2304 failed while creating the invoice. The office lead sees why, fixes the cause and resumes it, without anything running twice.

Step 1 of 4: Alert

Alertto Office lead

Close-out run r-2304 (job J-422) failed at Create invoice: accounting rejected tax code HVAC-SVC. Three earlier steps are done. Fix the code mapping and resume.

Six parts of a run you can trust

  1. 01

    Run records

    What happened, step by step

    Each run has an ID, its inputs, the outcome of each step, and a final status.

  2. 02

    Idempotency

    Safe to run twice

    Steps that might run twice are designed to be safe to repeat.

  3. 03

    Retry policy

    Try again, within limits

    Temporary failures retry within limits. Steps with external side effects, such as sending a message or charging a card, follow explicit rules about whether they may retry.

  4. 04

    Stuck-state detection

    Silence is a signal

    A case that hasn't progressed within its expected time raises an alert.

  5. 05

    Alerting

    To someone who'll see it

    Failures reach a named owner, in a channel they actually watch.

  6. 06

    Repair paths

    Fix and resume

    A failed run can be inspected, corrected, and resumed from the step that failed.

What may retry

Whether a step can be retried depends on what it does to the world, not on how it failed.
Read the job from schedulingYesRetry up to five times, then alert
Create the invoice in accountingYes, keyed to the jobRetry three times, then hold and alert
Email the invoice to the customerOnly if it wasn't sentCheck the send record first
Charge a saved cardNoNever retried automatically; a person decides

What monitoring hands off

In

Runs
every execution
Step results
done, retrying, failed
Timing
how long each stage should take
Monitoring
watch · alert · repair

Out

Alerts
to a named owner
Run history
searchable, per case
Repair queue
failed and stuck cases

Worth it for any automation you depend on

Good fit

  • Any automated workflow the business depends on.
  • Especially ones that touch customers, money, or other systems.
  • Automations nobody has looked at since they were set up.

Not the right fit

  • A one-off script run by hand while someone watches it.

Where monitoring does its work

Next to monitoring

Questions about workflow monitoring and recovery

Can you add monitoring to automations we already have?

Usually, yes. We inventory what's running, add run records and alerts where the tool allows, and rebuild the fragile ones around saved state so they can be resumed.

Who gets the alerts?

A named owner per workflow, agreed up front, in the channel they actually use: email, Slack, Teams or a text for urgent ones. Shared inboxes don't count.

What's a stuck case?

One that hasn't failed but hasn't moved: an approval nobody has answered, a step waiting on a system that never replied. Each stage has an expected time, and silence past it raises an alert.

Why not just retry everything?

Because some steps change the world. Retrying a read is harmless; retrying a payment or an email can do the thing twice. Each step gets its own rule.