How I Stopped 100 Telegram Alerts an Hour With a Circuit Breaker
My phone buzzed 100 times in an hour, all from one watchdog. The fix was sending fewer alerts.
When Watching the Watchdog Becomes the Job
If your on-call rotation spends more time triaging notification storms than fixing the underlying service, you don't have an infrastructure problem—you have an alert design problem. At Tacavar we spend a lot of time in this exact failure mode. A single watchdog that is too chatty can turn a restful night into a fire drill. Even worse, it trains your best engineers to mute channels and disconnect from the very signals they need. That's alert fatigue, and it's a direct tax on operational velocity.
The phrase "watchdog monitoring" implies a calm guard dog that barks for real threats. In practice, most watchdogs are hyperactive puppies that yap at every shadow. When everything is urgent, nothing is. Operators begin to ignore systems that cry wolf, and the harm is larger than missed incidents: you lose trust in your monitoring stack entirely.
The Crash Loop That Couldn't Stop Talking
Early versions of Tacavar's watchdog were textbook hyperactive. We had eight tg_alert() calls in the code with no throttle at the boundary. On a bad morning, a service entering a crash loop would fire over forty alerts in a single hour. My phone buzzed 100 times in an hour—and none of them were actionable. It wasn't a monitoring problem; it was a crash loop that couldn't stop talking.
The tell wasn't the crash itself—crashes happen and are expected. The problem was monitoring designed for the happy path. The watchdog assumed the human could restart something or intervene, so it fired every time state changed. There was no circuit breaker to stop the loop and no telegram alert throttling to consolidate the noise. Every failure mode became an emergency. That is the opposite of self-healing infrastructure.
When we finally looked at the logs, most alerts were generated in the first few minutes and just kept repeating. The system was actively harming time-to-recovery because we couldn't see the latest state through the mess of redundant messages. The fix started with a simple question: What action can a human take at each tier of alert?
Three Tiers: Self-Heal, Escalate, Bother a Human
Every good alerting system should have an escalation policy, not just a failure policy. At Tacavar we redesigned with three tiers.
Tier one is silent self-healing. The watchdog detects the anomaly, and the system attempts to remediate without a human in the loop. Restart a container, clear a stale lock, roll back a bad config. If it recovers, nobody's phone buzzes.
Tier two is AI escalation. If the fix doesn't work or can't be performed automatically, an agent correlates the failure with recent deployments and logs, and then prepares context. But it doesn't bother a human yet. At this tier, the system is still attempting to understand the blast radius and choose the appropriate playbook.
Tier three is the user alert—and this is the last resort, not the first reaction. A human is only notified when there is something actionable only a human can do. That might be a debt resolution, an architecture decision, or a manual step that requires judgment outside automated context.
The insight from Tacavar's watchdog v3 was not that humans should be removed from the loop. The insight is that the routing decision before the alert matters more than the alert itself. Asking "can a human do anything useful at this tier?" forces the monitoring system to behave like an operator, not an amplifier.
The Circuit Breaker That Ended 40-Alert Hours
We applied the circuit breaker pattern to our watchdog behavior. If a service or watchdog fails five times in one hour, the breaker opens and stops restarting the instance. It no longer tries "just one more time" in a tight loop. That pattern stops the restart storm cold.
The same pattern applies to alerting. After the breaker opens, we run a suppression window with telegram alert throttling: at most one user-facing alert per hour per incident. That one alert contains the full context, the remediation attempted, and the current state. No more stream of duplicates. No more forty-message hours. The result: the same failure that used to ping forty times now produces a single message—even if the underlying fault lasts for hours.
Circuit breakers are not new in distributed systems, but we rarely apply them to operational alerting itself. The chance of a continually failing process generating useful repeated notifications is close to zero. A breaker turns a pathological notification storm into a measured escalation.
Alerting as Last Resort, Not First Response
The goal of alerting isn't to report every state transition; it's to preserve value and organ health. When your default is silent self-healing, alerts become high-signal and rare. When an alert does land, your on-call knows it passed a real threshold and wasn't a heartbeat hiccup.
This is what self-healing infrastructure looks like in practice: the system recognizes failure, attempts recovery, curates context, and only then reaches out to a human. It's the difference between being woken up because a process restarted and being woken up because the automated recovery strategy exhausted its options.
For founders and operators, this is a philosophy shift. Your first response to an incident should never be an alert. It should be an action that makes the alert unnecessary.
What Your On-Call Can Learn From a Watchdog
Tacavar has learned a few hard-won lessons from building these systems.
First, count the tg_alert() calls. Every alert call in code is a promise that a human can do something at that moment. If the promise is false, delete the call. Second, if you repeatedly see crash loops generate dozens of identical alerts, do not tune the threshold—add a circuit breaker pattern to the watchdog itself. Third, build your monitoring around behavior tiers: self-heal first, generate context second, notify only when a human can act.
The outcome is a healthier on-call rotation and fewer distractions from the work you actually control. A watchdog that quietly heals most issues is doing more than one that pings you a thousand times. Tacavar's agent ops are designed around that insight.
We don't need more people watching the watchers. We need watchers that can be trusted to stay quiet until their bark matters.
Let Tacavar's self-healing agent ops take your alerts from storm to silence. See how at tacavar.com/agent-ops.