Google DeepMind and University of Pennsylvania Test AI Text Detector

Researchers at Google DeepMind and the University of Pennsylvania proposed FT-Model+SAE, a Gemma-based AI-text detector that scored 75.4 F1 on an out-of-distribution test, compared with 75.2 for its baseline.

The detector is designed to make text classifications easier to inspect by showing which model features contributed to its score. Sparse autoencoders break internal Gemma model activations into features that respond to patterns in text. Gemini 3.1 Flash Lite generates human-readable descriptions for the features and checks whether each description matches the pattern that activates it. A logistic-regression classifier then uses features above a selected fidelity threshold, and its output lists their descriptions and contributions.

On the in-domain COLING development split, FT-Model+SAE scored 96.8 F1, compared with 98.0 for the fine-tuned Gemma-2-2B baseline. The English COLING dataset spans 41 language models and contains 610,000 training documents and 261,000 development documents.

A separate setup, FT-SAE, scored 92.2 F1 on the COLING test split. It uses autoencoders trained on activations from the base Gemma model. FT-Model+SAE instead uses activations from a task-fine-tuned detector, so the 92.2 and 75.4 results describe different configurations.

The researchers also tested classifiers on a peer-review benchmark pairing human reviews for NeurIPS and ICLR from 2016 to 2024 with AI-generated text. The classifiers produced near-perfect F1 scores, but the study cautions that confounds or spurious patterns may make the task artificially easy. In one GPT-4o comparison, a formatting feature involving double colons after bold Markdown section headers appeared in 54,293 AI-written documents and 11 human-written documents.

The study tested GemmaScope autoencoders with Gemma-2-2B. Its feature descriptions can have high recall but low precision, and the results do not establish how the approach would perform with autoencoders trained on other base models.