Before shipping it, separate protocol facts, product promises, and operating cost. Mixing those layers produces confident but incorrect decisions. Fast requests hide tails in averages, and omitted timeouts make reports look faster. Record start, terminal state, limit, and censoring for each phase.
Performance work starts with the full interval users wait, then separates CPU, queues, network, and persistence. Compare p95 and p99 with resource cost because averages hide the users most likely to leave.
The parts that make the design practical
Write the following choices as reviewable rules instead of scattering them across callbacks and UI conditions. Explicit rules make scaling, compatibility, and diagnosis less dependent on guesswork.
- Use mergeable histograms instead of client-reported percentiles, slice by low-cardinality network, relay, browser major, and result code, and report timeout rate.
- Separate protocol facts, user intent, and automatic recovery; automation may restore facts but never overturn an explicit choice.
- Treat cleanup as protocol behavior: timers, handles, queues, and temporary data must be safely releasable in every terminal state.
The delivery standard for Latency Percentiles: When a 100 ms Average Hides Ten Seconds is a usable normal path, convergent failures, bounded resources, and a state users can understand. The result is a production capability that can be explained, degraded safely, and rolled back—not a demo that works once.
Keep false assumptions out of production
An abnormal path is more than an error banner. It decides how in-flight work stops, how the peer learns the outcome, what residue remains, and whether the next operation inherits it.
- Averaging instance p95 values is invalid, while excluding failed requests rewards fast failure rather than better experience.
- A boolean failure cannot distinguish retryable, user-action, and permanent refusal, producing an endless loop.
- Ideal-size tests miss large files, long sessions, and concurrency that cross hidden limits and cause cascading failure.
What the release gate should inspect
Build golden cases from known inputs and controlled faults, then align production metrics with those results. Verification extends to production only when signals detect the same degradation early.
- Generate traffic from a known distribution with five percent timeouts and verify merged percentiles, counts, censoring, and cohorts reproduce the input.
- Drive the state machine with reordered, duplicate, and delayed messages, proving stale versions are ignored and explicit stop survives recovery.
- Cover direct, relayed, weak-network, background-tab, and mobile paths; do not rely on averages or one successful screenshot.
The release bar is clear: users understand the current state, failures stop or recover, resources stay bounded, and operators can identify the phase from minimum necessary evidence.