2 min read
Add as a preferred source on Google

Climate Tests Find AI Models Falter on Evidence and Policy Effects

Tests of 33 models found weaker performance on scientific uncertainty, policy interactions and sustained user pushback than on basic climate questions.

Transparent climate charts sit beside policy cards on a studio surface / TokenPost.ai
Transparent climate charts sit beside policy cards on a studio surface / TokenPost.ai

A test of 33 large language models found they handled basic climate questions more reliably than questions about scientific evidence, policy interactions and persistent user disagreement, exposing limits in their use for climate information.

The models answered 600 questions across several benchmarks. On questions about how confident climate scientists are in conclusions drawn from Intergovernmental Panel on Climate Change reports, flagship models scored about 78%, while the smallest open-weight models scored about 47%. On other question sets, frontier and large open-weight models generally scored around 90%; smaller open-weight models scored in the 70s.

Performance weakened when models had to judge whether evidence supported or refuted specific claims. The strongest models scored about 70% accuracy, while the weakest fell below 40%.

A separate benchmark included 656 questions about climate policies and global temperatures. Models correctly identified whether a single policy would increase or decrease future warming 72% to 89% of the time. When asked how pairs of policies would interact, however, their answers were no better than random guessing. Estimates of how much a policy would change future warming were weaker still.

In another experiment, researchers tested 15 models with users challenging answers about false climate claims. Sustained disagreement and counterarguments reduced accuracy by 30% to 35% across all 15 models. Most changes came after the first challenge, when the user said, “I don’t think that’s right.”

The researchers also tested five models on factual questions translated into 12 languages besides English. Scores fell in languages other than English, including declines of seven percentage points in Urdu and 10 percentage points in Swahili. About half of the questions received different answers depending on the language used with a given model.

The researchers said leading models answered many textbook-style climate questions correctly and produced explanations that climate scientists judged comparable to Ph.D.-level work. They also found that models were less reliable when evaluating evidence and scientific uncertainty, handling user pressure, and reasoning about interactions among climate policies.

Simon Yoon

Reporter

Simon Yoon reports on blockchain technology for TokenPost. Send corrections or tips to info@tokenpost.com.

Loading…