Before shipping it, separate protocol facts, product promises, and operating cost. Mixing those layers produces confident but incorrect decisions. Both partitions can look locally healthy. Safety comes from external quorum or arbitration, not self-reporting; non-mergeable writes such as quota and revocation need one authority.
Distributed correctness comes from idempotency keys, leases, monotonic versions, and reconcilable facts—not an assumption of one delivery. A timeout means unknown outcome, not confirmed failure.
The parts that make the design practical
List non-negotiable invariants before selecting performance knobs. Tuning can roll out gradually; identity, permission, and terminal-state rules cannot drift at runtime.
- Classify operations: append logs may merge with versions, while uniqueness and quota require quorum; edges without authority become read-only or queued.
- Separate protocol facts, user intent, and automatic recovery; automation may restore facts but never overturn an explicit choice.
- Treat cleanup as protocol behavior: timers, handles, queues, and temporary data must be safely releasable in every terminal state.
The delivery standard for Split-Brain Detection, Fencing, and Recovery by Layer is a usable normal path, convergent failures, bounded resources, and a state users can understand. The result is a production capability that can be explained, degraded safely, and rolled back—not a demo that works once.
Keep false assumptions out of production
Prioritize faults that silently preserve false facts: the interface looks recovered while a queue, permission, or counter has diverged. The defect often appears only on the next action.
- Allowing both regions to issue session ownership makes cancel and permission changes ambiguous after healing, creating a security rollback.
- Refresh and network change start two recovery paths, and duplicate side effects look like two genuine user actions.
- User or task IDs in metric labels create high-cardinality cost and leak unnecessary identity into diagnostics.
What the release gate should inspect
A release gate combines deterministic regression, randomized timing, and real browser pairs. Preserve the seed and state trace from every failure as a permanent replay case.
- Partition regions and attempt connect, trust revocation, quota charge, and logging; verify the allow matrix, then audit merge ordering and conflicts after healing.
- Run one hundred start, fail, retry, and cancel cycles; handles, listeners, queues, and temporary data must return to baseline.
- Before release, record success rate, p50/p95/p99 latency, error classes, and resource high-water marks with explicit rollback thresholds.
The result must be correct, recoverable, and explainable. If any part depends on refreshing the page or an engineer guessing, the protocol loop remains incomplete.