DeepSeek Model Completes 21 of 31 Tasks in Vals AI Hacking Test

DeepSeek V4.1 Flash 1017 completed 21 of 31 objectives in Vals AI’s cyber benchmark, six fewer than OpenAI’s restricted GPT-6 Sol, at a fraction of the per-attempt cost.

Vals AI released an updated CyberOffense benchmark Oct. 23. In a simulated environment on the Lighthouse network, models worked through sequences of offensive hacking objectives. GPT-6 Sol completed 27 of the 31 objectives, while DeepSeek’s model completed 21.

A DeepSeek attempt cost less than $3, compared with about $22 for GPT-6 Sol. DeepSeek V4.1 Flash 1017 is an open-weight model available to download. GPT-6 Sol is restricted to OpenAI’s Daybreak program.

Vals AI researcher Nikil Ravi said frontier models available through limited-access programs “are now starting to be matched by open models that anybody can download.” The shrinking performance gap has raised questions about whether access controls still serve their intended purpose.

The updated benchmark builds on Vals AI’s June work on vulnerability discovery and patching. The newer test measures performance across linked objectives in a simulated operation, extending the evaluation to multiple steps.

The comparison comes amid debate over whether limiting access to U.S. frontier models keeps them at the forefront. Those policy discussions have often treated the U.S. gated-release approach as setting the pace for advanced models.

Anthropic declined to broadly release Claude Mythos Preview in April, citing its ability to find and exploit software vulnerabilities. The company limited access to more than 40 partners.