Why Temperature 0 Isn't Deterministic

Setting an LLM's temperature to 0 does not guarantee deterministic output due to floating-point arithmetic inconsistencies, particularly how the order of operations can change results based on factors like batch size. A derived formula suggests the probability of a token choice flipping is a function of the gap between the top two logits and the amount of numerical noise. This risk is not evenly distributed across tokens but is concentrated on specific "knife-edge" tokens where the model's choices are nearly tied. This phenomenon explains why long, identical completions can suddenly diverge after many consistent tokens.