The dead-man's switch for telemetry
Most monitoring systems alert you when something bad happens. Very few notify you when your monitoring stops altogether.
Think about how alerting usually works: a detector triggers, you get paged, you investigate. The whole model depends on activity being present. But what happens when that activity suddenly ceases?
The attacker who goes quiet
A skilled attacker doesn’t always trip alarms. They can go quiet — disabling an agent, blocking the SDK’s outbound calls, or dropping telemetry before it ever reaches the collector. The application keeps serving requests. Dashboards stay calm. Everyone assumes that calm equals safe.
That assumption is misleading.
The same thing happens without an attacker at all: a crashed collector, a broken deploy, an agent someone disabled “temporarily.” The result is identical — you’ve gone blind, and no alert tells you.
Why this failure mode is so dangerous
In API environments this is especially perilous, because so many security decisions depend on telemetry arriving as expected. When the data stops, the decisions silently stop being trustworthy.
What makes it unsettling is that it’s intentionally invisible:
- There is no alert for the alert that never fired.
- There is no log entry for the log that was never written.
- The gap itself is the signal.
Most tools concentrate on what has happened. Almost none focus on what should have happened but didn’t.
Silence as a signal
The fix is straightforward: implement a heartbeat. If a service is expected to check in every N seconds and suddenly goes silent, that silence becomes the event. No complex models, no anomaly-detection black box — just a clear, deterministic rule:
This service hasn’t reported in longer than it normally does.
What APIGuard does with it
APIGuard is a runtime API-security layer. Each protected service runs an SDK that reports to the APIGuard collector on a predictable cadence — a heartbeat every N seconds over a secure channel.
This is the dead-man’s switch, built in. When the cadence breaks:
- Beat on time → no action. The system keeps monitoring.
- Cadence broken → APIGuard raises one alert: what went quiet, and when — so you investigate before the blind spot becomes an incident.
It’s a deterministic rule, not a model. That makes it three things every on-call engineer actually wants at 3 a.m.:
- Inspectable — you can see exactly why it fired.
- Tunable — you set the cadence and the tolerance.
- Explainable — no “the model thinks” hand-waving.
Why it matters
False confidence is worse than a noisy alert. A pager storm is annoying; a silent blind spot is how incidents run for hours before anyone notices.
Detecting suspicious activity is crucial. But recognizing the suspicious absence of activity is what stops attackers — and quiet failures — from hiding in plain sight.
This is one of the detectors built into APIGuard. More notes soon.
Have you ever hit a silent failure in monitoring or telemetry and only found out days later? How did you catch it?