Modified DeepSeek Harness Uses 87.93% Fewer Tokens in Coding Test

A modified DeepSeek Harness used 87.93% fewer model tokens per task and completed more assignments on a coding benchmark, pointing to lower resource use for AI agents.

The Oct. 23 preprint tested 98 HumanEval coding tasks. With HarnessMM and other changes to the agent setup, average combined input and output token use fell from 96,280 to 11,620 tokens per task. The success rate rose from 71.43% to 85.71%.

HarnessMM stores actions the model has already selected and attaches conditions based on a later tool result. If the result meets those conditions, the agent can perform the stored action without another reasoning call. A step that needs a new decision still goes back to the model, whose underlying weights remain unchanged.

The 87.93% reduction reflects more than stored-action reuse. The tested setup also changed how DeepSeek Harness managed context and finalized tasks. It made 539 fewer paid model calls, but only 147 were certified cases in which a stored action replaced a reasoning call.

Output tokens alone rose from 3,830 to 4,530 per task. The decline in the combined total came from lower input-token use.

On the same HumanEval tasks, Codex used 29.54% fewer tokens with HarnessMM, averaging 47,020 per task instead of 66,720. Its success rate stayed at 90.82%. Each comparison kept its underlying model and task scorer fixed.

Results varied on a second benchmark of 257 MBPP-sanitized tasks. DeepSeek Harness averaged 10,610 tokens per task with HarnessMM, down from 53,980, while its success rate rose from 90.27% to 91.83%. Codex also used fewer tokens, but its success rate slipped from 91.83% to 90.66%.

The DeepSeek Harness team also built a continuously running internal instance for multitask and team collaboration. The HarnessMM test examines a separate part of the agent’s workflow: whether it can safely carry out a step chosen before a tool returns its result.

The findings measure token use on coding benchmarks. They do not establish dollar savings, shorter run times, lower energy use or the same performance in production. Costs also depend on provider accounting and how often an agent has an earlier decision it can reuse.