Before shipping it, separate protocol facts, product promises, and operating cost. Mixing those layers produces confident but incorrect decisions. Deleting an open log does not free space until the handle closes. copytruncate can lose or duplicate lines; prefer reopen signals or journald.
A release must be reproducible, observable, and reversible, including configuration and data compatibility. A health check proves process life; critical routes, dependencies, and background work need separate gates.
The parts that make the design practical
Start from facts the data and protocol can guarantee, then decide what the interface may promise. Each rule below needs an owner, a bound, and a compatibility policy rather than an oral convention from one review.
- Rate-limit structured levels, reduce debug at a soft disk threshold, reserve audit and error at a hard threshold, and cap compressed retention bytes.
- Bound every input by size, count, and time, returning a stable actionable error code when a budget is exceeded.
- Retries need an idempotency key, backoff, and deadline; after the deadline create a new task instead of reviving old callbacks.
The delivery standard for Log Rotation and Disk Safety for Services That Must Stay Online is a usable normal path, convergent failures, bounded resources, and a state users can understand. The result is a production capability that can be explained, degraded safely, and rolled back—not a demo that works once.
Keep false assumptions out of production
Prioritize faults that silently preserve false facts: the interface looks recovered while a queue, permission, or counter has diverged. The defect often appears only on the next action.
- A stack trace every millisecond fills disk and slows service, while an unbounded compression backlog steals CPU and I/O after incidents.
- Fixing only the UI leaves queues, locks, or expired credentials for the next operation to inherit and fail again.
- Without backpressure or quota, a slow consumer raises memory, queue depth, and tail latency until unrelated users are affected.
What the release gate should inspect
Observe both endpoints, persisted records, and operational signals during verification. One button state or one successful response cannot prove the complete loop.
- Generate one hundred thousand errors per second while rotating and filling disk to 95 percent; service stays live, alerts fire, critical logs remain, and space returns.
- Race refresh, cancel, timeout, and remote completion in one scheduling window; assert one terminal state and one side effect.
- Before release, record success rate, p50/p95/p99 latency, error classes, and resource high-water marks with explicit rollback thresholds.
The result must be correct, recoverable, and explainable. If any part depends on refreshing the page or an engineer guessing, the protocol loop remains incomplete.