How Many Guesses Does It Take to Crack an AI's "Random" Number?

Large language models exhibit significant bias when asked to generate a "random" number, often favoring specific values. This analysis moves beyond simple frequency counts by examining the output logits of open models to calculate the true probability distribution for numbers between 1 and 100. Using security-focused metrics like Shannon entropy, min-entropy, and guessing entropy, the predictability of a model's choice is quantified in terms of bits of randomness and the expected number of guesses to crack it. The study compares base models to their instruction-tuned counterparts, revealing that post-training can amplify these biases, with one tested model choosing the number 47 over a third of the time.