Legal AI Test Records 70% Correct Value Extraction After Revisions

Revised instructions for a large language model coincided with a rise from 52% to 70% in values correctly extracted from driving-under-the-influence verdicts, showing why clear guidance matters in legal data work.

The test formed part of a process for turning criminal verdicts into structured datasets. The process explores which details to capture, identifies supporting passages and converts extracted information into standardized values. Legal experts can refine the instructions that guide those steps.

In the driving-under-the-influence cases, the share of correctly extracted values reached 70% after the instructions were revised, compared with 52% under the earlier guidance. Human annotators also agreed more often after the revisions.

Repeated runs of the same model agreed with one another more often than they agreed with the gold annotations used for comparison. Consistent output therefore did not establish that an extracted value was correct. Disagreements among human annotators and among different models helped reveal where the instructions needed clarification.

The study also examined verdicts involving driving despite a ban. In that evaluation, the instructions specified fixed categories and output formats, and agreement was high for most attributes.