LLM Sampling Bias Raises Simulated Security Risk in Patrol Test

Tests of specific language models found that uneven sampling raised a simulated attacker’s success rate in a four-server patrol task to as high as 99.9%, underscoring a security risk when agents must make randomized choices.

The paper, “The Illusion of Stochasticity in LLMs,” tested requests for uniform discrete, uniform continuous and Gaussian samples. Each configuration used 1,024 outputs. In a representative table covering four models and three distributions, all 12 goodness-of-fit p-values fell below 0.05, providing evidence against the requested distributions under the tests used.

The tests used chi-square comparisons for discrete uniform sampling and Kolmogorov-Smirnov tests for continuous distributions. A p-value below 0.05 is evidence against the target distribution; it does not measure whether a model can produce different answers from one call to the next.

Outputs sometimes favored specific numbers, including 7 and 42. Reordering choices in a prompt also changed which options models favored. That pattern means variety alone does not show that a model follows assigned odds consistently.

In the patrol simulation, an agent selected one of four servers to inspect while an attacker targeted the least-inspected server. The modeled attacker succeeded 75% of the time against the optimal uniform strategy. With choices from three Gemini variants, modeled success rates were 96.6%, 99.9% and 99.9%; each model’s prompt was run 1,024 times.

The simulation did not represent an attack against an operating system or network. The study did not test deployed agents or cryptographic random-number generators, so its results describe the models and setup examined rather than every AI agent.

In a separate approach, models received a Python-generated value in [0,1] and converted it into a requested distribution. Qwen3 models with at least 4 billion parameters had p-values above 0.05 for the tested discrete uniform distribution, while models with at least 8 billion parameters crossed that threshold for the tested Gaussian distribution.

Those results mean the tests did not find sufficient evidence to reject the target distributions; they do not prove reliable sampling. More complex conversions can still fail, limiting what the method establishes.

A stateful sampler that keeps track of its state between calls is one possible safeguard for agents that need controlled randomness. Simulating pseudorandom number generators can be expensive and may not fit tasks that require frequent sequential sampling.