A sound implementation does not ask users to refresh until it works. Each phase has an input, an output, a timeout, and a terminal state. Breaker statistics must cover homogeneous calls. Open fails fast; half-open admits a small jittered probe set instead of synchronized traffic from every instance.
Distributed correctness comes from idempotency keys, leases, monotonic versions, and reconcilable facts—not an assumption of one delivery. A timeout means unknown outcome, not confirmed failure.
Define the system contract first
This capability crosses clients, networks, and servers, so a local optimization can create a system failure. Decisions must constrain both endpoints, persisted truth, and operating budgets together.
- Bulkheads isolate resources, deadlines cap one call, and a rolling window opens the breaker; fallback responses explicitly state which features remain safe.
- Define success, degraded, cancelled, and failed terminal states before UI, storage, and metrics consume the same state.
- Treat cleanup as protocol behavior: timers, handles, queues, and temporary data must be safely releasable in every terminal state.
The delivery standard for Circuit Breakers: Timeouts, Isolation, and Half-Open Probes is a usable normal path, convergent failures, bounded resources, and a state users can understand. The result is a production capability that can be explained, degraded safely, and rolled back—not a demo that works once.
What can break that contract
Production failures often appear when two individually valid actions overlap. Inspect stale messages, duplicate effects, exhausted resources, and mixed versions instead of patching only the current stack frame.
- Counting user 4xx responses as dependency failure opens the breaker incorrectly, while one global breaker lets a regional fault disable the world.
- A stale response arriving after a new task can overwrite healthy state or restart cancelled work without version fencing.
- An untested fallback receives all traffic during a primary failure and becomes the slower, more expensive bottleneck.
How to test the contract line by line
Do not stop verification when the final action succeeds. Count side effects, measure wait time, inspect privacy, and prove the next run begins from a clean baseline.
- Inject slow responses, connection refusal, partial 5xx, and business 4xx; observe concurrency, state transitions, half-open probes, and post-recovery tail latency.
- Run one hundred start, fail, retry, and cancel cycles; handles, listeners, queues, and temporary data must return to baseline.
- Cover direct, relayed, weak-network, background-tab, and mobile paths; do not rely on averages or one successful screenshot.
A capability becomes maintainable when it degrades safely, repetition adds no side effects, and its signals reveal a fault before user reports do.