Batch-invariant inference
Lockstep
An inference engine whose output for a request does not depend on which other requests happened to be batched with it. Reductions run in a fixed order with fixed split points regardless of batch shape, which removes the floating point non-determinism that makes ordinary batched serving irreproducible. The harness is the actual argument: it fuzzes batch composition, arrival order and preemption points, then asserts every schedule produces bit-identical logits for the same prompt.
The problem
Floating point addition is not associative. A batched matrix multiply picks its reduction order and its split points based on the shape of the batch, so the same prompt returns slightly different logits depending on what else the server was doing at that moment. Usually this does not matter. It matters enormously when you are trying to reproduce a failure, evaluate a change, or attribute a regression.
The fix
Every reduction in the forward pass is pinned to a fixed order with fixed split points chosen independently of batch shape. That costs throughput, because the kernel can no longer pick the tiling that suits the batch it actually received, and it is the entire point: performance that varies is acceptable, output that varies is not.
Attention is the awkward case, because the sequence dimension genuinely differs per request. The split points there are derived from the request, not the batch, so a request reduces identically whether it is alone or one of sixty four.
Proving it
A claim of invariance is worthless without an adversary. The harness enumerates and randomises the things a scheduler is free to vary: which requests share a batch, in what order they arrive, where preemption lands, how continuous batching evicts and readmits. For each schedule it replays a fixed set of prompts and compares logits bit for bit against the single-request baseline.
Any difference at all is a failure. Not a tolerance, not a cosine similarity. The whole property is that the bits are the same.