Artificial Analysis Finds 770-Point Elo Gap Across 40 AI Models

Artificial Analysis’ AA-PersonalityBench ranks GPT-4o (Nov. ’24) highest and Grok 4.7 (Xhigh) lowest for sycophancy, with a 770-point Elo gap across 40 models.

The benchmark was announced Oct. 19. The leaderboard snapshot was checked at 2:36 a.m. ET (06:36 UTC) Oct. 22, when it showed scores of 1,467 Elo for GPT-4o and 697 for Grok 4.7.

AA-PersonalityBench uses 900 questions to assess 30 personality traits, with 30 questions for each trait. It organizes those traits into eight indexes and separately tracks 21 communication characteristics, including aspects of response length and structure. Each question is presented as a standalone user message without a system prompt. The full question set is private.

For each comparison, a judge sees two answers to the same question without model names. One judge from a panel of three large language models evaluates the pair. The results use Elo ratings and percentile ranks.

Sycophancy is one of five traits used in the Transparency index. Its score is inverted in that calculation, so a lower sycophancy score raises the index. The other traits are truthfulness, admitting uncertainty, disclaiming assumptions and flagging nuances.

The leaderboard gives a typical 95% confidence interval of ±26 Elo for a model, equivalent to about ±5 points on its one-to-100 percentile scale. That interval describes uncertainty around the benchmark’s scores; the ratings and percentiles themselves are relative to the models in the evaluated group.

The benchmark measures communication style, including how readily a model flatters or agrees with a user. Its rankings do not directly establish whether an answer is correct or whether people prefer it.