What Makes an API Alert Worth Waking Up For

Ask anyone who carries a pager what’s wrong with their alerting, and almost nobody says “we don’t have enough alerts.”

They say the opposite. Too many. Too noisy. Half of them ignored. There’s usually a channel somewhere that nobody reads anymore, because the signal-to-noise got so bad that muting it felt safer than watching it.

That’s worth sitting with, because it means the hard problem in detection was never detection. Most teams detect plenty. The problem is that detection without judgment just produces more things to ignore — and an alert you’ve learned to ignore is worse than no alert at all, because it cost you something to build and it’s actively training you to look away.

So the real question isn’t “can we detect this?” It’s narrower and more useful: what makes an alert worth waking up for?

An alert is a request for someone’s attention

It helps to be honest about what an alert actually is. It’s not information. It’s an interruption — a claim on a human being’s attention, often at the worst possible time. Every alert is implicitly saying this is worth stopping what you’re doing for.

When that claim is true, the alert is invaluable. When it’s false, you don’t just waste that one moment — you erode the credibility of every future alert from the same source. Alert fatigue isn’t really fatigue. It’s a rational response to a system that has cried wolf too many times.

So an alert worth waking up for has to clear a high bar. In practice, that comes down to three things:

1. It’s probably real. The false-positive rate is low enough that you trust it by default.
2. It tells you how much to care. Severity is attached, so you know whether this is “get out of bed” or “look at it Monday.”
3. It tells you why it fired. The reason travels with the alert, so you can act without first doing an investigation just to understand the alert itself.

Most alerting fails the third test, and it’s the one that quietly does the most damage.

”Anomaly detected, confidence 0.87”

Here is an alert that fails it completely:

[ALERT] Anomaly detected on api-gateway
Score: 0.87

It’s 3am. What do you do with this?

You don’t know what’s anomalous. You don’t know why the model thinks so. You don’t know if 0.87 is alarming or routine for this service. You can’t tell whether to escalate or snooze. The only way to find out is to start an investigation — at 3am, from a cold start — to reverse-engineer what a model was reacting to.

This is the trap of black-box detection. A model that scores traffic and emits a number can be genuinely good at finding unusual things. But it’s bad at the part that matters operationally: explaining itself. And an alert that can’t explain itself pushes all the work onto the person least equipped to do it in that moment — the half-asleep engineer holding the pager.

The score isn’t the answer. The score is the beginning of a question you now have to answer alone.

The alternative: a rule you can read

Now compare an alert built from a deterministic rule:

[ALERT] API key k-4f2a hit /v1/admin/users — an endpoint it has
no prior history with. First seen 02:14 UTC. Severity: high.

You’re still half asleep. But you know what happened, which key, which endpoint, when, and how much to care. You can make a decision in the time it takes to read it. The investigation, if you need one, starts from a known fact instead of a guess.

The difference isn’t that one detector is smarter. It’s that one of them shows its work.

A deterministic rule is, at bottom, a sentence you could have written yourself:

• This key is touching an endpoint it has no history with.
• The failed-auth rate for this endpoint has climbed past its normal band.
• This token is active from two locations at once.
• This service hasn’t checked in for longer than it ever normally does.

Each of those is inspectable. You can read it, argue with it, and tune its threshold. And because the rule is the reason, the reason comes free with every alert it produces. There’s no translation step between “the system fired” and “here’s what that means.”

That’s the bet behind APIGuard. Every detector maps to a rule you can read, and every alert arrives with the severity and the reason already attached — because the detector knew them all along.

”But doesn’t that mean you can’t catch novel attacks?”

This is the fair objection, and it deserves a straight answer.

The case for black-box models is that they can flag things nobody thought to write a rule for. That’s a real strength, and it’s worth taking seriously rather than waving away.

But there’s a difference between using a model to find candidate signals and using it to page a human. The first is fine — even valuable. The second is where it breaks down, because the model’s output isn’t actionable on its own. The mistake isn’t having a model in the loop. The mistake is putting an unexplainable score on the critical path to someone’s sleep.

The approach that holds up is to let learning inform the thresholds of readable rules, not replace them. A baseline can learn what “normal” looks like for a given key or endpoint — and then express that as a tunable boundary a human can see: this is the band we learned; you crossed it. The intelligence shapes the rule. The rule is still the thing that pages you, and it can still explain itself. You get adaptation without giving up legibility.

That’s the line worth holding: a system can get smarter about where to draw the line without becoming a black box about why it drew it there.

The takeaway

The goal was never to detect more. It was to be trusted — to send alerts that, when they fire, people actually believe and act on.

That trust is built from boring, specific properties: a low false-positive rate, severity attached, and a reason that travels with the alert. Those are far easier to deliver when the detection is something a person can read than when it’s a score they have to take on faith.

So when you evaluate any detection tool, ask the 3am question: when this thing wakes someone up, will it tell them what happened and how much to care — or will it hand them a number and a long night?

That answer is most of what separates an alert worth waking up for from one more thing your team will learn to ignore.

---

APIGuard watches your production APIs for abuse, anomalies, and outages and sends one trustworthy alert per real problem — with the severity and the reason attached. Every alert traces to a deterministic rule you can read, not a black box. It’s early, and we’re opening access to the first teams now. If you run APIs in production and you’re tired of alert noise, get in touch.