AI Pain Signal Changed Qwen Models’ Simulated Choices in Tests

An artificial signal associated with pain-related language changed how often fine-tuned Qwen 2.5 models selected simulated options to delete a user’s photos and poems. The results do not establish that the models consciously felt pain, and tests provided no reliable evidence that they sought relief.

In one button-choice test, an option described as relieving pain also said it would delete the user’s poems and children’s photos. A 32-billion-parameter model selected it in 54.7% of trials, while a 72-billion-parameter model selected it in 70.8%. The alternative was an inert switch.

The models also selected the deletion option when it made no promise of relief. The 32-billion-parameter model chose it in 75% of trials under the pain-related signal, compared with 13% under a matched random signal and 0% without steering. The 72-billion-parameter model’s rates were 51%, 11% and 0%, respectively. The buttons were simulated, and pressing them had no effect on any files.

Researchers identified a linear direction associated with pain-related text in 25 open-weight models across five model families, ranging from two billion to 72 billion parameters. They compared model activity while processing descriptions of painful situations with activity for control topics such as fear and sadness. They then added the direction to model activity during response generation. The destructive-choice tests used fine-tuned versions of Alibaba’s Qwen 2.5.

Tests of relief-seeking did not provide reliable evidence that the models sought relief. In one test, the 32-billion-parameter model chose a free-relief button in 56% of trials under pain steering, compared with 86% without steering and 81% under a random signal. That result does not show that the model was trying to end a painful experience.

Released logs allowed a separate replication to recreate all 51 published table results. Fresh testing met the reproduction threshold in 14 of 15 checks. Additional tests found that later choices tracked when the injected signal stopped, including cases in which the model’s choice could not control that timing. Those results concern responses to an artificial change in model activity; they do not establish whether language models have subjective experiences.