Why Temperature 0 Isn't Deterministic in LLM Inference ── Batch Invariance, 80 Divergent Outputs Out of 1,000 Runs, and the 34.35% Speed Trade-off
YouTube (Japanese audio): https://youtu.be/vZIHDULWNF0
Executive Summary & Key Takeaways
- Core Thesis: Every AI engineer learns that setting
temperature = 0(greedy decoding viaargmax) eliminates sampling randomness and guarantees identical LLM outputs. In production multi-tenant inference servers, that assumption is false. In a landmark investigation by Horace He and Thinking Machines Lab (Defeating Nondeterminism in LLM Inference), queryingQwen3-235B1,000 times attemperature = 0with the exact same prompt ("Tell me about Richard Feynman") produced 80 distinct completions, with the most frequent output appearing only 78 times (7.8%). Even more strikingly, all 1,000 runs were 100% identical up to token 102 ("Feynman was born on May 11, 1918, in"), before splitting at token 103 into"Queens, New York"(992 runs) vs."New York City"(8 runs). Contrary to the widespread myth that GPU thread concurrency (atomicAdd) causes forward-pass nondeterminism, the true culprit is the lack of Batch Invariance: dynamic changes in concurrent server load alter batch sizes, prompting GPU kernels (RMSNorm,Matmul,Attention) to switch reduction strategies (Split-K/Split-KV), which shifts floating-point rounding order. Enforcing batch-invariant kernels in vLLM (VLLM_BATCH_INVARIANT=1) and SGLang (--enable-deterministic-inference) restores 1,000/1,000 bitwise reproducibility and eliminates RL policy collapse (true 0 KL divergence at Step 318), at the cost of a 20% to 34.35% throughput penalty. - Who Should Read This: LLM inference & infrastructure engineers running vLLM or SGLang; ML researchers training models with RLHF / GRPO / RLVR; engineers designing regression evaluation (Eval) harnesses; and CTOs in regulated industries (finance, healthcare, legal) requiring auditable reproducibility.
- Key Architectural Takeaways:
- Why floating-point non-associativity turns
1024 + 0.4 + 0.4into either1024.0or1025.0infp16depending on reduction order. - Why single-request GPU forward passes are already 100% deterministic (
atomicAddis not used in forward kernels), and why multi-tenant Dynamic Batching + Split-Reduction is the real source of drift. - How fixed-size split reductions (
Split size = 256) makeRMSNorm,Matmul, andPagedAttentionbatch-invariant. - How to navigate the 20%–34.35% latency overhead (26s vs. 42s for 1,000 requests on
Qwen3-8B) across production serving, CI evaluations, and RL training.
- Why floating-point non-associativity turns
Background & Primary Sources Verified

This technical deep dive synthesizes four primary engineering sources that solved one of LLM infrastructure’s longest-standing mysteries:
- 1. Thinking Machines Lab (Horace He et al.): Defeating Nondeterminism in LLM Inference (Thinking Machines Official Blog)
- 2. Daily Dose of Data Science (Avi Chawla, Oct 27, 2026): Why Temperature 0 Isn’t Deterministic in LLM Inference (Daily Dose of Data Science)
- 3. vLLM Official Documentation (v0.32.0): Batch Invariance (
VLLM_BATCH_INVARIANT=1) (vLLM Docs) - 4. SGLang Official Documentation: Deterministic Inference (
--enable-deterministic-inference) (SGLang Docs)

The Richard Feynman 1,000-Run Experiment
- Model & Configuration:
Qwen/Qwen3-235B-A22B-Instruct-2507(non-thinking mode),temperature = 0(greedy decoding), generating 1,000 tokens per request. - Prompt:
"Tell me about Richard Feynman", sent 1,000 times to an inference API endpoint. - Observed Outcome: Instead of 1 unique completion, the model returned 80 unique completions. The single most common completion occurred only 78 times.
- Exact Divergence Point: Every single one of the 1,000 completions was 100% identical for the first 102 tokens (
"Feynman was born on May 11, 1918, in"). At token 103, 992 runs chose"Queens, New York", while 8 runs chose"New York City", triggering a butterfly effect across subsequent autoregressive steps.

1. Debunking the “GPU Thread Concurrency” Myth: Why 1024 + 0.4 + 0.4 Changes the Winner

1-1. The Mathematical Root: Floating-Point Non-Associativity
At temperature = 0, argmax deterministically selects the token with the highest logit. If the chosen token changes at token 103, it means the raw logit values for "Queens" and "New" were nearly tied, and the computed floating-point numbers shifted in the 3rd or 4th decimal place.
In IEEE-754 floating-point arithmetic, addition is not associative:
Equation:
(a + b) + c ≠ a + (b + c)
Avi Chawla illustrates this clearly using 16-bit floating-point (fp16), which has 11 bits of significand precision (10 stored mantissa bits + 1 implicit leading bit). Between 1024 ($2^{10}$) and 2048 ($2^{11}$), the minimum representable step size (ULP) in fp16 is exactly 1.0. Fractional values like 1024.4 cannot be represented and round down to 1024.0.
Consider adding three fp16 numbers: 1024 + 0.4 + 0.4:
import numpy as np
a, b = np.float16(1024), np.float16(0.4)
print((a + b) + b) # Order A: Left-to-right -> 1024.0
print(a + (b + b)) # Order B: Small numbers first -> 1025.0
- Order A
(1024 + 0.4) + 0.4: First,1024 + 0.4 = 1024.4, which rounds down to1024.0(the0.4is swallowed). Adding the second0.4again rounds down to1024.0. - Order B
1024 + (0.4 + 0.4): First,0.4 + 0.4 = 0.8is computed accurately because both operands share a small exponent. Then,1024 + 0.8 = 1024.8, which rounds to the nearest even integer:1025.0.
1-2. Why the “GPU Thread Race Condition (atomicAdd)” Explanation Is Wrong
For years, engineers blamed GPU thread scheduling: “Thousands of GPU threads finish in random order and accumulate partial sums via atomicAdd, changing addition order.”
Horace He proves this hypothesis completely false for LLM inference:
- 1. Zero
atomicAddin Forward-Pass Kernels: While backward-pass gradient accumulation often usesatomicAdd, standard LLM forward kernels (GEMM,RMSNorm,FlashAttention) use deterministic tree reductions or ordered semaphores within thread blocks. - 2. Empirical Proof (Run-to-Run Determinism at Fixed Batch Size): Running
torch.mm(A, B)1,000 times on the same GPU with the exact same input tensor shapes produces bitwise-identical results (0.0 max difference) 100% of the time.
2. The True Culprit: Dynamic Batching Breaks “Batch Invariance” Across 3 Core Kernels

If a single GPU kernel call is 100% deterministic for a fixed input shape, why do API responses vary? Because in a production inference server (vLLM, SGLang, TensorRT-LLM), your request is dynamically batched with other users’ concurrent requests.
When you send a request at 14:00:01, the server might process a batch of B = 1 token. One millisecond later, 63 other users arrive, and token 103 is processed in a batch of B = 64 tokens. To maximize hardware utilization (SM occupancy) across different batch sizes, GPU libraries dynamically switch kernel reduction strategies:
| Kernel Stage | High-Batch Strategy (Data-Parallel) | Low-Batch Strategy (Split-Reduction) | Why Reduction Order Shifts |
|---|---|---|---|
| 1. RMSNorm | 1 row (token) assigned per SM core; sequential reduction inside SM | Row split across multiple SMs to avoid idle cores; inter-SM reduction | Splitting a single token vector across SMs changes the reduction tree topology |
| 2. Matrix Multiplication (Matmul) | Standard 2D Tile GEMM (1 tile per thread block) | Split-K GEMM / Stream-K (inner dimension $K$ split across blocks) | Accumulating partial $K$-slices across blocks alters the summation order |
| 3. Attention (Decode / Chunked Prefill) | Standard FlashAttention over sequence length | Split-KV (FlashDecoding) + variable prefill chunk sizes | Splitting the KV cache into variable chunks changes softmax rescaling & denominator sums |

3. The Solution & Its Cost: Fixed Split-Size (256), 34.35% Overhead, and True On-Policy RL

To achieve Batch Invariance—guaranteeing that token $i$ produces the exact same bits regardless of whether batch size is 1 or 256—Thinking Machines Lab, vLLM, and SGLang redesigned the three kernels so that the reduction tree for any single token never depends on batch size:
- Batch-Invariant RMSNorm: Always use a fixed reduction topology per row regardless of batch size (disabling inter-SM split reductions for small batches).
- Batch-Invariant Matmul: Disable dynamic
Split-Kswitching and enforce a fixed split size (e.g., 256) in Triton/CuTe kernels so partial sums are accumulated in an identical sequence at every batch size. - Batch-Invariant Attention: Standardize KV cache layout prior to kernel launch and enforce a fixed Split-KV chunk size of 256 rather than dividing KV length by the number of currently idle SMs.

Quantitative Performance Cost & RL Training Impact

- 1,000-Run Reproducibility: With batch-invariant kernels enabled, the Richard Feynman test drops from 80 unique outputs down to 1 unique output (1,000/1,000 bitwise match).
- Throughput & Latency Overhead (20% to 34.35%):
- Thinking Machines Benchmark (
Qwen3-8B, 1,000 sequences of 90 tokens): Standard vLLM takes 26 seconds; adding deterministic attention takes 55 seconds; optimized batch-invariant vLLM takes 42 seconds (+61.5% time / ~38% throughput drop). - SGLang Official Benchmark: Enabling
--enable-deterministic-inferencewith FlashInfer incurs a 34.35% throughput penalty; with Triton backend, the overhead is approximately 20%.
- Thinking Machines Benchmark (
- Eliminating Silent Off-Policy Drift in Reinforcement Learning (RLVR / GRPO):
- During RL training, rollouts are generated by an inference engine (vLLM/SGLang) at dynamic batch sizes, while log-probabilities for policy updates are computed by a training engine (FSDP/Megatron) at fixed batch sizes.
- Without batch invariance, the inference log-probs and trainer log-probs diverge (non-zero KL divergence), causing complete reward collapse at Step 318 unless patched with truncated importance sampling.
- With batch-invariant kernels, KL divergence between inference and training remains identically 0.0, enabling true on-policy RL without importance-sampling hacks.
4. Actionable Engineering Playbook: How to Configure vLLM and SGLang Today

# 1. vLLM (v0.32.0+): Enable Batch-Invariant Inference (NVIDIA GPUs, compute capability >= 9.0)
export VLLM_BATCH_INVARIANT=1
vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507 --dtype bfloat16
# 2. SGLang: Enable Deterministic Inference with FlashInfer or Triton Attention Backend
python3 -m sglang.launch_server \
--model-path Qwen/Qwen3-8B \
--enable-deterministic-inference \
--attention-backend triton
- 1. Separate High-Throughput Production Serving from Deterministic Eval/Audit Tiers:
- Do not pay a 34.35% GPU tax on casual chat traffic where
temperature > 0is already used. Keep standard dynamic Split-K enabled on user-facing clusters, and spin up a dedicatedVLLM_BATCH_INVARIANT=1worker pool specifically for CI regression evaluations, financial/medical compliance audits, and reproducible bug replay.
- Do not pay a 34.35% GPU tax on casual chat traffic where
- 2. Turn On Batch Invariance for All RL / GRPO Rollout Workers:
- If you run RLVR or GRPO pipelines where vLLM/SGLang generates rollouts, enable
VLLM_BATCH_INVARIANT=1(or--enable-deterministic-inference) and align precision/kernel configs with your training loop to prevent Step-300+ reward collapse.
- If you run RLVR or GRPO pipelines where vLLM/SGLang generates rollouts, enable
- 3. Enforce Identical Tensor Parallelism (TP) Sizes Across Reproducible Environments:
- Batch invariance guarantees reproducibility across varying batch sizes on the same GPU architecture and Tensor Parallelism (
TP) degree. ChangingTP=2toTP=4alters cross-GPU AllReduce summation order; always pinTPsize in your evaluation lockfiles.
- Batch invariance guarantees reproducibility across varying batch sizes on the same GPU architecture and Tensor Parallelism (