Before shipping it, separate protocol facts, product promises, and operating cost. Mixing those layers produces confident but incorrect decisions. A byte inserted near the beginning shifts every fixed boundary and destroys reuse. Content-defined chunking anchors boundaries to data patterns, limiting change to nearby chunks at the cost of rolling hashes and size bounds.
File transfer must reconcile sender offsets, durable receiver writes, chunk integrity, and terminal state on both peers. A full progress bar may show queued bytes, not a complete usable file.
The parts that make the design practical
Turn the important choices into durable contracts: validate inputs, assign state ownership, define cleanup, and specify fallback for older peers. Later optimization must not change those semantics.
- Set minimum, target, and maximum chunk sizes to avoid pathological tiny or unbounded chunks.
- Store a strong digest and ordered manifest for each chunk; the rolling hash finds boundaries but does not prove integrity.
- Enable it where large versions or cross-task reuse pay off, while one-time delivery keeps fixed chunks.
Content-defined chunking is a version-reuse tool, not a universal default. It pays only when saved network and storage exceed hashing, indexing, and complexity costs.
Keep false assumptions out of production
Failure and success must share one state model. An error toast that neither releases resources nor propagates a terminal state leaves dirty work for the next recovery attempt.
- A weak rolling hash is treated as integrity proof, allowing incorrect reuse after collisions.
- Chunk indexes exceed small-file size, erasing the benefit through metadata overhead.
- Rolling hashes run on the main thread, causing jank and background throttling.
What the release gate should inspect
Build golden cases from known inputs and controlled faults, then align production metrics with those results. Verification extends to production only when signals detect the same degradation early.
- Insert the same data at the beginning, middle, and end; compare reuse and CPU time.
- Use zero-filled, highly repetitive, and incompressible data to prove chunk sizes stay bounded.
- Force rolling-hash collisions and verify strong digests reject incorrect chunks.
Completion is not one passing path. Every terminal state reconciles, automation stays below user intent, and every operational cost has an explicit ceiling.