Operational resilience: the system is up and has not been working for weeks
The dashboard is green. The system responds. Users log in, screens open, response time is within the agreement. And the operation has been stopped for weeks.
It is not a contradiction — it is the difference between two questions almost everyone treats as one. Availability measures whether the system responds. Operational resilience measures whether the work happens. They are different things, and only the first usually has an alarm.
Why the monitor does not see it
All monitoring is built on events. Something happens, something is recorded, a rule evaluates the record and decides whether to raise an alert. The model works well for what breaks.
It is blind to what simply does not happen. A process that should have run and did not generates no event at all. There is no line in the log, no return code, nothing for a rule to evaluate. Silence is indistinguishable from a quiet day.
And silence comes in three forms, worth separating because they are investigated differently:
What should have started and did not. The queue exists, messages arrive, and there is nobody consuming on the other side. Everything piles up, in order, with no error.
What started and never finished. The document sits in an intermediate state — prepared, released, waiting — and never advances. Each record looks normal on its own; the problem only appears in the total.
What finished wrong and nobody compared. The result came out, it is plausible, and it is incorrect. This is the hardest one, because it requires an expectation to compare against.
The number that gives it away: zero in error
In an environment we support, invoices stopped going out. No error on screen, no log entry, no alert.
The investigation turned when somebody counted documents by state instead of looking for a failure:
- 250 documents stuck in preparation
- 7 released
- zero completed
- zero in error
That last number is the finding. Nothing had failed. Nothing had even tried. The component that dispatches work pointed at an address that did not exist, and more than a hundred thousand units were queued — some of them for over a month. The system passed any availability check.
The whole case, including the wrong hypothesis that cost two days, is in output that never prints: no spool, no error, no log.
The measurement that is missing: counting the expected
The fix is not more monitoring. It is a different kind of monitoring, which nobody installs by default because it requires something the tool does not have: what should happen.
The question changes from “what failed?” to “what should have happened and did not?“. And it is answerable with simple counting, per time window:
- documents by state — how many came in, how many completed, how many stayed in the middle;
- queue by consumer — how many units waiting, and how old the oldest one is;
- interface by period — how many messages in the same window yesterday, last week, the same day last month;
- routine by calendar — did the thing that runs every Monday run this Monday?
None of these counts is sophisticated. What they require is the part that is usually missing: somebody has to have written down what is expected. With no declared expectation there is no detectable absence — only a number with no reference.
How to build it on top of what already exists
1. List the processes that produce a countable result. Issuing, integration, closing, loading, shipping. If the result cannot be counted, its absence cannot be detected.
2. Write down the expectation per window. It does not have to be precise. “Between 200 and 400 per business day” is already infinitely better than nothing, because zero becomes visible immediately.
3. Alert on absence, not on error. The rule that matters is “N were expected and 0 arrived” — and it is different from every rule you already have.
4. An owner per process. An alert with no recipient becomes noise in two weeks, and switched-off noise is worse than no alert at all, because it creates a feeling of coverage.
What goes wrong, by name
Using availability as an operations indicator. A system can spend months at 99.9% availability without processing part of the work. Both numbers are true at the same time, and only one of them reaches the business.
Monitoring the component, not the result. Service up, process stopped. Watching the part is necessary and not sufficient — what matters is the document that comes out the other side.
Treating “nobody complained” as evidence. In the case above, the document behind the ticket was less than 0.1% of what was queued. Nobody had raised a ticket about the rest — and the rest was almost everything.
Measuring averages instead of counts. Average processing time stays excellent when nothing is being processed. Averages hide absence; counts do not.
Trusting the alarm that never rang. An alarm that never fired has two possible explanations, and most operations only consider one of them.
In a regulated environment this stopped being optional
Operational resilience has entered the vocabulary of financial-sector rules, and the text asks for more than a written policy: it asks for demonstration. Declaring that a control exists and demonstrating that the control works are different exercises — the second requires exactly the evidence that the work happened.
It is the same logic as an audit in any regulated sector: it is not enough to state that the process runs; you have to be able to show how many times it ran, when, and what came out. We cover this in IT for financial institutions.
The honest limit
Counting the expected requires knowing what to expect — and a good share of operations do not. Finding that out is business work, not infrastructure work, and it is the part that usually stalls. When nobody can say how many documents a normal day produces, the problem comes before monitoring.
This also does not replace what already exists. Failure monitoring is still necessary — what is proposed here is a layer nobody installed, not a swap of one for the other.
And there is a limit of scope: detecting the absence shortens the time to discover, not the time to fix. In the case above, detection would have saved most of eleven days. The fix itself would still have required investigation — and coordination, because restarting a work dispatcher with a hundred thousand queued units is an event in itself.
If the starting point is finding what is already stuck with nobody knowing, that is exactly what the IT diagnostic looks for. To sustain it day to day, the path is observability. And if the discussion is putting automated processes into that environment, it is worth first understanding what autonomy inherits.