From the audit · Alerting

Silence has no symptom

Alerting fails in one direction only. A noisy alert is reported within the hour by the person it woke; a suppressed one is reported by nobody, ever, because a suppressed problem is not a quiet problem — it is not a problem at all. Nothing is logged that a human reads, no dashboard tile turns amber, and the next incident review has no artefact to point at.

So configuration drifts towards silence in a way it never drifts towards noise. That asymmetry, not carelessness, is what produces a four-figure count of open maintenance windows.

How you get to 1,617

One window per release, per service, opened by a deployment pipeline that has no step for closing it. One opened by hand during an incident at 02:00, scoped generously because nobody wanted to argue about scope at 02:00. One opened for a migration with an end date that has since passed — which, depending on how it was configured, either expired cleanly or kept suppressing.

Each of those was a correct decision at the time. None of them was ever a decision to stop alerting on that service indefinitely, which is nevertheless what the tenant now does.

The second mechanism: entity-level overrides

Maintenance windows are at least visible in one list. Anomaly-detection overrides are worse: they live on the entity, they are invisible from the global configuration screens, and they turn detection off rather than routing it elsewhere. On the same tenant there were 106 of them.

The pattern behind them is familiar — a host or service was noisy during an incident, somebody disabled detection on it to get through the night, and the re-enable never happened because it was never a ticket.

The shape of the query

The window inventory with its schedule, which is enough to separate "ends on Friday" from "never ends":

What to do, in what order

Do not re-enable everything. A tenant that has been quiet for two years will produce a wall of problems on the first day, most of them true, and the team will turn it all off again by Friday — a worse outcome than where you started.

Order by blast radius instead: windows scoped to production services that page someone, then overrides on entities that carry customer traffic, then the long tail. Re-enable in batches small enough that the resulting problems can be triaged rather than skimmed, and fix the underlying noise as you go. On the audited tenant this was the first remediation, because it changes whether an incident reaches a human and costs nothing to reverse.

The finding behind this piece

The takeaway

Nobody decided to stop alerting. 1,617 people decided something reasonable, once each, and nothing in the product asked whether the total still made sense.

More from the series

The $91,576 question: Full-Stack hosts with nothing monitored on them From the audit · FinOps The Grail bucket nobody chose: 462 days of retention From the audit · Grail Personal data in logs: 50 sources, about 30.7 million records a day From the audit · Pipelines What actually lands on a DPS invoice From the audit · FinOps

An audit takes one run. The findings take one meeting. What you do with them is up to you.

Certified Dynatrace experts

Send

Obzervability

Privacy Terms of use Accessibility