A sound implementation does not ask users to refresh until it works. Each phase has an input, an output, a timeout, and a terminal state. A timeout means the caller does not know the result, not that the server did nothing. The idempotency record belongs in the same atomic boundary as the business write.
Distributed correctness comes from idempotency keys, leases, monotonic versions, and reconcilable facts—not an assumption of one delivery. A timeout means unknown outcome, not confirmed failure.
Define the system contract first
This capability crosses clients, networks, and servers, so a local optimization can create a system failure. Decisions must constrain both endpoints, persisted truth, and operating budgets together.
- The first request claims a key as pending and persists status plus response after commit; reuse with a different payload returns conflict.
- Give state one owner, a version, and terminal states; callbacks may mutate only the version that created them.
- Use explicit capability negotiation so older clients receive an explained fallback instead of a half-working state.
The delivery standard for Retries Are a Contract: Idempotency Across Writes and Side Effects is a usable normal path, convergent failures, bounded resources, and a state users can understand. The result is a production capability that can be explained, degraded safely, and rolled back—not a demo that works once.
What can break that contract
Failure and success must share one state model. An error toast that neither releases resources nor propagates a terminal state leaves dirty work for the next recovery attempt.
- Writing idempotency after the business commit leaves a duplicate window, while caching only 200 responses repeats deterministic validation side effects.
- Fixing only the UI leaves queues, locks, or expired credentials for the next operation to inherit and fail again.
- An untested fallback receives all traffic during a primary failure and becomes the slower, more expensive bottleneck.
How to test the contract line by line
Do not stop verification when the final action succeeds. Count side effects, measure wait time, inspect privacy, and prove the next run begins from a clean baseline.
- Disconnect before commit, after commit before response, and before side-effect delivery; retry the same key and verify one row, notification, and quota charge.
- Run one hundred start, fail, retry, and cancel cycles; handles, listeners, queues, and temporary data must return to baseline.
- Before release, record success rate, p50/p95/p99 latency, error classes, and resource high-water marks with explicit rollback thresholds.
Completion is not one passing path. Every terminal state reconciles, automation stays below user intent, and every operational cost has an explicit ceiling.