The three failure modes that never trigger an alert

Alert rules are built around a threshold someone configured in advance. That works great for the failures you already anticipated — error rate above 5%, latency above 2 seconds. It works badly for the failures that creep in slowly enough that no single measurement ever crosses the line.

We see three patterns repeatedly in AI workloads. First, cost drift: token usage per session inching upward 2-3% a week as a prompt template grows, never triggering a spike alert because there's no single bad moment, just a slow accumulation. Second, latency creep: P50 latency drifting from 200ms to 450ms over three weeks as a downstream dependency degrades, staying comfortably under any P99 alert threshold the whole time. Third, quiet hallucination clusters: a model confidently returning a plausible-but-wrong answer to a narrow slice of requests, with no error thrown and no obvious signal in aggregate error rate.

Nudges exist specifically for this gap. Instead of comparing a metric against a fixed threshold, it compares current behavior against the system's own recent history, and scores how unusual the delta is. That's how a 20% drift over three weeks — invisible to any alert rule — still surfaces as a ranked, investigable nudge days before it would have become an incident.