Before shipping it, separate protocol facts, product promises, and operating cost. Mixing those layers produces confident but incorrect decisions. Differential privacy bounds one user’s impact on released statistics but cannot fix overcollection. Minimize and aggregate before adding formal protection.
Data and models enter decisions only with a defined purpose, minimum inputs, and repeatable evaluation. Deterministic policy owns permissions and side effects; models provide evidence, confidence, and safe abstention.
The parts that make the design practical
Start from facts the data and protocol can guarantee, then decide what the interface may promise. Each rule below needs an owner, a bound, and a compatibility policy rather than an oral convention from one review.
- Clip contributions per user/day, allocate epsilon across declared queries, add Laplace or Gaussian noise, track cumulative budget, and suppress tiny cohorts.
- Bound every input by size, count, and time, returning a stable actionable error code when a budget is exceeded.
- Treat cleanup as protocol behavior: timers, handles, queues, and temporary data must be safely releasable in every terminal state.
The delivery standard for Differential Privacy for Product Metrics and When Not to Use It is a usable normal path, convergent failures, bounded resources, and a state users can understand. The result is a production capability that can be explained, degraded safely, and rolled back—not a demo that works once.
Keep false assumptions out of production
Boundaries turn hidden assumptions into incidents. Weak networks, refresh, concurrency, and capacity need combined coverage because retries can hide each one in isolation.
- Repeated daily queries on one segment compose privacy loss, while noising operational error codes can hide incidents without benefit.
- A stale response arriving after a new task can overwrite healthy state or restart cancelled work without version fencing.
- Ideal-size tests miss large files, long sessions, and concurrency that cross hidden limits and cause cascading failure.
What the release gate should inspect
A release gate combines deterministic regression, randomized timing, and real browser pairs. Preserve the seed and state trace from every failure as a permanent replay case.
- Simulate epsilon, clipping, and sample sizes on known distributions; verify error, composition, small-cohort protection, and decision stability.
- Disconnect, change networks, and recover mid-operation; reconcile endpoint state, persistence, and resource counts.
- Before release, record success rate, p50/p95/p99 latency, error classes, and resource high-water marks with explicit rollback thresholds.
The result must be correct, recoverable, and explainable. If any part depends on refreshing the page or an engineer guessing, the protocol loop remains incomplete.