Circuit Breakers for AI Agents: Graceful Degradation in Production

Every production AI agent eventually faces the same failure modes: the LLM provider returns a 429 rate limit, the model hallucinates a response that breaks downstream parsing, or the inference endpoint goes down entirely. Teams that have not prepared for these scenarios experience cascading failures, corrupted data, and silent quality degradation that users discover through broken behavior rather than an honest error message.

The solution is not better prompts or a more reliable model. It is circuit breaker infrastructure that detects failure patterns, isolates failing dependencies, and routes around them. This guide covers the specific patterns we use at Tacavar to keep agent pipelines running when individual model calls fail, drawing on principles from our causal containment security baseline and our experience operating multi-model agent fleets.

Why Traditional Retry Logic Fails for LLM Calls

The default approach to handling LLM call failures is exponential backoff retry. It is also the approach that causes the most production incidents. Here is why.

When an LLM provider starts rate-limiting, it is usually doing so at the account level, not the request level. Retrying with backoff sends additional requests into an already-saturated queue, which extends the rate-limit window and can trigger more aggressive throttling. We have observed NIM endpoints that return 429s for 15 to 30 minutes after a burst of retries, even when the original request volume was modest.

Retry logic also masks the underlying problem. If your agent is failing because the model is producing malformed output (missing JSON fields, wrong schema, truncated responses), retrying the same prompt against the same model produces the same failure. The retry burns tokens, consumes rate limit budget, and delivers the same broken result. Worse, if the agent is operating in a multi-turn loop, retries introduce latency that compounds across turns until the agent times out or the user gives up.

The pattern is the same one we described in our analysis of mid-tier LLM routing and hallucination risk: the system keeps calling the model and hoping for a better outcome instead of recognizing that the model itself is the problem in that moment.

The Circuit Breaker Pattern for AI Agents

A circuit breaker is a state machine that sits between your agent logic and the external dependency (in this case, an LLM API). It has three states:

• Closed: Requests flow normally. The breaker counts failures but does not intervene.
• Open: The failure threshold has been exceeded. All requests are immediately rejected without calling the dependency. The agent receives a controlled error and can execute its fallback strategy.
• Half-Open: After a cooldown period, the breaker allows a single probe request through. If it succeeds, the breaker closes and normal traffic resumes. If it fails, the breaker reopens and the cooldown restarts.

For AI agents specifically, the circuit breaker should track multiple failure types independently:

• Transport failures: HTTP 429, 500, 502, 503, timeout, connection refused. These indicate infrastructure problems.
• Quality failures: Valid HTTP 200 but malformed JSON, schema validation failure, empty content, or response flagged by a critic agent. These indicate model capability problems.
• Latency failures: Response received but exceeded the SLA threshold (e.g., p99 above 30 seconds). These indicate provider congestion.

Tracking these separately matters because the recovery strategy differs. Transport failures resolve when the provider recovers. Quality failures may require switching models or simplifying the prompt. Latency failures may resolve on their own or may indicate a need to route to a faster model tier.

Implementing Model Fallback Chains

When a circuit breaker opens for your primary model, the agent needs a fallback target. A naive fallback chain simply lists alternative models and tries them in order. A well-designed fallback chain considers the trade-offs between models explicitly.

At Tacavar, our fallback chains are structured by model tier rather than model name. This approach, which we detail in why agent routing matters more than prompting, means that when a high-capability model (e.g., a frontier reasoning model) fails, we fall back to another high-capability model first, then to a mid-tier model, and only as a last resort to a lightweight model that may not handle the task adequately.

Key design principles for fallback chains:

• Never fall back to the same provider for the same failure type. If OpenAI is rate-limiting your account, falling back from GPT-4o to GPT-4o-mini does not help. The rate limit is account-scoped. Fall back to a different provider entirely.
• Cap the number of fallback hops. Each hop adds latency and cost. We enforce a maximum of two fallback hops for any single agent turn. After the second fallback fails, the agent enters degraded mode rather than continuing to try models.
• Make degraded mode explicit to the agent. When the agent is operating on a fallback model, it should know. The system prompt should include context about the current model's capabilities and limitations so the agent can adjust its behavior. A mid-tier model should not attempt complex multi-step reasoning that it cannot handle reliably.
• Log every fallback with the reason. When a circuit breaker opens, the event should be logged with the failure type, the model that triggered it, and the fallback model selected. This data is essential for tuning thresholds and identifying provider reliability trends over time.

Degraded Mode: When No Model Is Available

The most under-discussed pattern in agent infrastructure is what happens when every model in the fallback chain is unavailable. Most systems either crash, hang indefinitely, or silently return a garbage response. The correct behavior is degraded mode.

Degraded mode means the agent stops attempting LLM calls and falls back to deterministic logic, cached results, or a pre-computed default. The specific degraded behavior depends on the agent's role:

• Research agents: Return the most recent cached analysis with a timestamp indicating it may be stale. Do not attempt to generate new analysis.
• Classification agents: Fall back to keyword-based heuristics. They are less accurate than LLM classification but they are deterministic and never fail.
• Content generation agents: Queue the task for retry when models recover. Do not publish partially generated or empty content.
• Orchestration agents: Execute a simplified decision tree that does not require LLM reasoning. If the task genuinely requires model intelligence, surface an error to the operator rather than guessing.

Degraded mode should have a time-to-live. After a defined period (we use 10 minutes), the circuit breaker enters half-open state and probes the primary model again. If the probe succeeds, normal operation resumes. If it fails, degraded mode continues. This prevents a scenario where a provider recovers but your agents stay in degraded mode indefinitely.

Monitoring Circuit Breaker Health

A circuit breaker that opens silently is better than a crash, but it is still a signal that something is wrong. Every breaker state transition should emit a telemetry event. The metrics that matter:

• Breaker open rate: How often each model's breaker opens. A model that opens its breaker multiple times per week is unreliable for production use and should be demoted in the fallback chain.
• Time in open state: How long breakers stay open before recovery. Long open durations indicate provider outages rather than transient blips.
• Fallback trigger rate: How often the agent actually uses a fallback model versus the primary. Even if the breaker does not open, frequent single-call fallbacks indicate borderline reliability.
• Degraded mode frequency: How often the entire fallback chain is exhausted. This should be near zero in a healthy system. If it happens regularly, you need more provider diversity.

We integrate these metrics into our broader observability stack, which we describe in our agent telemetry guide. The key insight: beautiful dashboards do not prevent failures. Threshold-based alerts on breaker-open events do. If your monitoring shows a circuit breaker opening every hour for the same model, you have a provider problem that no amount of retry logic will fix.

Cost Implications of Circuit Breaker Design

Circuit breakers have a cost dimension that is easy to overlook. Every failed call to a primary model before the breaker opens is a chargeable API call. Every fallback call adds additional cost. If your breaker threshold is too high (too many failures before opening), you are paying for calls that were always going to fail. If your threshold is too low (opens after a single failure), you are prematurely routing to more expensive fallback models.

We tune breaker thresholds per model based on historical reliability data. For models with a known 429 pattern (certain NIM endpoints, for example), we use a threshold of 3 consecutive transport failures before opening. For models that rarely fail but when they do it is catastrophic (data corruption, safety filter false positives), we use a threshold of 1. The cost of a single corrupted agent turn is higher than the cost of a premature fallback.

This is the same cost-quality trade-off we explored in how nerfing agents doubled our quality: spending marginally more on infrastructure to avoid expensive failure modes is almost always the right call at production scale.

A Reference Implementation

Here is the core circuit breaker logic we use, simplified for clarity:

from enum import Enum
from time import time
from dataclasses import dataclass, field

class BreakerState(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"

@dataclass
class CircuitBreaker:
    failure_threshold: int = 5
    cooldown_seconds: int = 60
    _state: BreakerState = BreakerState.CLOSED
    _failures: int = 0
    _opened_at: float = 0.0

    @property
    def state(self) -> BreakerState:
        if self._state == BreakerState.OPEN:
            if time() - self._opened_at >= self.cooldown_seconds:
                self._state = BreakerState.HALF_OPEN
        return self._state

    def can_call(self) -> bool:
        return self.state in (BreakerState.CLOSED, BreakerState.HALF_OPEN)

    def record_success(self):
        self._failures = 0
        self._state = BreakerState.CLOSED

    def record_failure(self):
        self._failures += 1
        if self._state == BreakerState.HALF_OPEN or self._failures >= self.failure_threshold:
            self._state = BreakerState.OPEN
            self._opened_at = time()

The can_call() method is checked before every model invocation. If it returns False, the agent skips to its fallback chain immediately, without burning a single token on a call that would fail. The record_success() method resets the failure counter, so transient errors do not accumulate indefinitely. The half-open transition happens lazily on the next can_call() check, which avoids the need for a background timer.

In production, we wrap this with per-failure-type tracking, Prometheus metrics emission, and integration with our model router so the fallback chain is driven by breaker state rather than static configuration.

Common Pitfalls

Several mistakes recur in teams implementing circuit breakers for AI agents for the first time:

• Single breaker for all failure types. Using one failure counter for both transport errors and quality errors means a model that returns malformed JSON three times trips the breaker even though the API is healthy. Track failure types independently and use separate breakers.
• No cooldown differentiation. A 429 rate limit may need a 5-minute cooldown. A transient 502 may resolve in 30 seconds. Using the same cooldown for all failure types wastes time waiting for problems that resolve quickly.
• Forgetting to reset on deploy. If you restart your agent process, in-memory breaker state is lost. If a provider was down before the restart and is still down, the new process will burn through its failure threshold again before the breaker opens. Consider persisting breaker state for critical models.
• Silent fallback without notification. If the agent falls back to a cheaper model and produces lower-quality output, the user or operator should know. Silent quality degradation is the most insidious failure mode in production AI.

The Larger Pattern: Designing for Dependency Failure

Circuit breakers are one instance of a broader principle: production AI systems must be designed around the assumption that every external dependency will fail. LLM providers will rate-limit, hallucinate, and go offline. Vector databases will return stale results. Web search APIs will time out. The agent's job is not to prevent these failures but to degrade gracefully when they occur.

The teams that succeed with production AI are the ones that treat failure handling as core infrastructure, not an afterthought. A well-implemented circuit breaker costs a few hundred lines of code and prevents the majority of cascading failure incidents. It is, dollar for dollar, the highest-ROI infrastructure investment for any team running agents in production.

For more on our broader agent infrastructure philosophy, see our AI agent infrastructure stack overview and our approach to building durable agents that survive infrastructure failures without losing state.

Related Reading

• Causal Containment: A Security Baseline for Agent Systems
• Why Agent Routing Matters More Than Prompting
• Agent Telemetry and Observability in 2026
• How Nerfing Agents Doubled Our Quality
• Building Durable AI Agents