Open benchmarks
Benchmarks on real jobs, open to read. The method and rubric are published, every reference answer is computed by code, and the task set is held back so it cannot leak into training.
Set the bar. See which model clears it.
Hurdle builds benchmarks from your team’s real work, runs every model worth testing on them, and hands you the answer and the files.
What Hurdle builds
Benchmarks written from real cases, graded line by line against a standard your people agree, and run on every model worth testing.
Private benchmarks
Your job, your cases, your reviewers’ standard. Every model runs it, and you keep the benchmark, the scores and the cost per task.
Environments
Real jobs packaged as sandboxes with a grader, built for model labs to train and evaluate on.
A finished benchmark you keep, not another platform to log in to
Four hurdles to an answer
A fixed scope, your people on the rubric, and a benchmark that stays yours.
- 1
Send the cases
50 to 150 real examples of the job, with the answers your team accepted.
- 2
Agree the rubric
Each case becomes a task. Your reviewers agree, line by line, what a correct answer contains.
- 3
Run every model
Each model runs every task three times. Lines are checked by code wherever the answer can be.
- 4
Keep the answer
You get the report and the files. When a model worth testing ships, we run it again.
invoice-matching/
benchmark.jsonname, bar, version
tasks.jsonl120 tasks from your cases
rubric/what a correct answer contains
grader/runs with any model
runs/latest/
report.htmlscores, costs, every miss
attempts.jsonleach reply, each verdict
What you keep
The benchmark
Tasks, rubric and grader as plain files. Run them in your own pipeline, with any model.
A plain answer
Which model to use for the job, or a clear statement that none is ready yet.
Every miss, traced
Each missed line sits next to the reply that missed it, so you see why, not just how often.
Switch or stay
One page each time a model worth testing ships, with the new scores beside the old.
Research
Open benchmarks on real jobs. The method and rubric are published, and every reference answer is computed by code.
Questions
Is Hurdle a platform?
No. There is nothing to log in to. You get a finished benchmark: the tasks, the rubric, the grader and the report, as files you keep and can run yourself.
What do you need from us?
Between 50 and 150 real cases of the job, the answers your team accepted for them, and one person who can say what a correct answer contains.
Which models do you run?
Every current model worth testing for the job: the large hosted models, the open-weight families, and models you could run on your own machines.
Can we run the benchmark ourselves?
Yes. The files are plain JSON, and the runner works with Amazon Bedrock, any OpenAI-compatible gateway, or a model on your own hardware.
Where do our cases go?
Only to the model providers you approve for the run. Nothing is published, and cases can be anonymised before they leave you.
What does it cost?
Each benchmark is scoped with you, and the price depends on the number of cases and models. We say it in the walkthrough.