How we score
The same rules for every benchmark, open or private. Nothing is published that was not run.
A task, scored by its lines
Every task carries a rubric: the lines a correct answer must contain, each with a weight.
task score = weight of lines met / weight of all lines
A line is one checkable claim: the mean gap is 8.6%, these three categories trigger an assessment, the date is the one in the notice. Weights say which claims matter most to the team.
Code first, a judge only where needed
Numbers, dates, lists and names are checked by code, with the tolerance written into the rubric.
When a line needs reading, such as whether a memo states the reason for a decision, a judge model checks that one line and nothing else, and a person checks a sample of its verdicts.
The open pay-gap benchmark has no judge lines at all: every one of its lines is checked by code.
Three runs, three numbers
Each model runs every task three times.
Mean is the average over every run: what one call to the model scores on a typical task.
Best of three keeps each task’s best run. Every run keeps its worst, which is what you get when the job has to be right each time.
An interval on every score
Scores come with a 95% interval, from resampling the tasks 2,000 times.
When two models’ intervals overlap, the report calls them close rather than ranking one above the other on noise.
The bar
A model clears the bar when its mean is at or above the share the team set.
The bar belongs to the people who do the job. It is agreed before the first run and printed in every report, next to the scores it was applied to.
What counts against a model
A call that fails and a reply with no readable answer both score zero.
In production a failed call is a job not done. The report counts both separately, so a flaky provider is visible and not hidden in the average.
Cost and time
Cost per task is what each call cost at the provider’s price, averaged per task.
Time per task is the wall-clock time of each call. Models run on your own hardware show no cost, and the report says so rather than guessing one.
What every report carries
Enough to check any number in it.
- The hash of the tasks file that was run
- The version of the benchmark and of the runner
- Every reply, and the verdict on every line
- Start and finish times, and the models’ exact names