What “monitored” should mean for an automation

The most expensive automation failures are the quiet ones. A webhook stops arriving, a login expires, an API changes a field name, and the workflow keeps reporting success while doing nothing. Someone finds out weeks later, usually from a customer.

“Monitored” gets used loosely. For us it means four specific things, and nothing we build goes live without them.

1. Every run leaves a record

Each run of a system writes down what triggered it, what it did at each step and how it ended. Not just errors: successful runs too, because “it ran and did nothing” is a failure mode of its own.

With a run log, the question “did the invoice go out?” has an answer in seconds instead of a search through three tools.

2. Safe retries happen on their own

Plenty of failures are temporary: a service is slow, a rate limit is hit, a network call drops. The system should retry those, with a growing pause between attempts.

The catch is that a retry must be safe. Sending the same email twice or creating the same invoice twice is worse than not sending it at all. So steps are designed to be repeatable, checking whether the work was already done before doing it again. When a step can’t be made safe to repeat, it doesn’t retry. It stops and asks.

3. Failures reach a person

When a run fails for good, someone is told, in the place they already look, with enough context to act: which run, which step, what went wrong and what it was working on.

Alerts need care. A system that raises an alert for every hiccup trains people to ignore it. We alert on what needs a person and summarize the rest.

4. Someone watches the quiet signals

Some failures never throw an error. The number of orders processed drops to zero on a weekday. A daily report arrives empty. A queue grows instead of draining.

So we also watch for the absence of normal activity, and for volumes that move far outside their usual range. These checks are simple, and they catch the problems that error handling never sees.

What you get at handover

When a system is handed over, the monitoring goes with it: the run log, the alert rules, who receives them and a short guide to what each alert means and what to do about it.

The goal isn’t a system that never fails. Everything that depends on other people’s services will fail sometimes. The goal is a system that tells you when it does.