AI evaluation benchmarks. Written new, kept private.
If a test is on the internet, it is in the training data. eeko commissions evaluation items that have never been online from credential-checked experts, has a second expert solve every one, tiers them by difficulty, and licenses them exclusively to you. A score you can defend.
What is a private AI evaluation benchmark?
A private AI evaluation benchmark is a held-out test set that has never been published: items written new by domain experts for one buyer, each with a verified reference answer or a grading rubric, delivered under licence and kept off the open web. Because the items never appear in a crawl, a model cannot have memorized them, and its score reflects what it can do rather than what it has read.
Public benchmarks lose that property within a few crawl cycles. The items, their answers, and their paraphrases leak into pre-training corpora, scores climb, and the gains stop showing up in production. Teams then add more public benchmarks and the cycle repeats. The alternative is not a cleverer public test. It is a test nobody else has.
eeko evaluates the RAG systems and agents we put into production the same way, with items our own models have never seen, so the benchmark service is built to the standard we hold ourselves to: expert authors, a second expert on every item, deliberate difficulty tiers, and a signed record of exactly which items a reported score was measured on.
Held-out evaluation sets, item by item.
An evaluation item is held to a higher bar than a training item: unambiguous, correctly keyed, tagged, and solved twice before it counts.
- Never catalogued, never published
- Text fingerprints checked against registered work
- Key or rubric per item
- Author justification stored with the item
- Checkable criteria, not adjectives
- Pilot grading before release
- Skill and tier tags per item
- Mix rebalanced against pilot pass rates
- Blind solve, then compare
- Ambiguous items rewritten or dropped
- Scheduled new items, same bar
- Expert grading with agreement reported
Benchmark contamination, and what it does to a score.
Four ways a public test stops measuring anything, and the design choice that answers each.
Failure modes, and the fix for each
- Memorization — items seen in pre-training are recalled, not solved. Fix: items that were never online, fingerprinted at intake against everything registered.
- Ambiguity — an item that admits two answers punishes good models by coin flip. Fix: a blind second solve on every item, with disagreements rewritten or removed.
- Saturation — once every model scores in the high nineties, the set cannot rank them. Fix: deliberate difficulty tiers, rebalanced against pilot pass rates.
- Unprovable results — a reported number nobody can tie to a specific item set. Fix: a signed receipt that names each item by ID and hash; the exact set behind a score is then on record.
None of this needs a bigger set. It needs a set that is correct, private, tiered, and on the record.
A benchmark item that nearly had two right answers.
An illustrative item from a commissioned set of organic chemistry problems, followed from draft to sealed delivery.
From scope to a sealed set.
Fixed scope, fixed price, pilot slice before the full set. Refresh orders repeat the loop on a schedule.
Expert-written evaluation sets by industry and discipline.
What a correct answer is, and who is qualified to say so, changes with the field. So does the author and the second solver.
AI evaluation benchmark questions.
How do you keep benchmark items from leaking into training data?
By never letting them out. Items are written for your engagement alone, never listed in the eeko catalogue, never published, and delivered to you under an exclusive licence as the default. Authors sign confidentiality terms. At intake, each item's text fingerprint is compared with every registered work to catch reuse or close paraphrase. Once the set is in your hands, access control is yours; we recommend treating it like a signing key.
How is benchmark difficulty calibrated?
Each author tags an item with the skill it tests and the tier they intend, and the second expert confirms or changes the tag while solving it blind. If you share outputs from current models on a pilot slice, we use the pass rates to check that the tiers separate strong models from weaker ones and rebalance the mix. The target is a set with headroom: hard enough that the best model you have still misses some.
Can you build rubric-graded evaluations, not just exact-match?
Yes. For open-ended work, whether legal analysis, clinical reasoning, or long-form writing, we write a rubric of specific, checkable criteria, pilot it with several graders, and rewrite the lines where they disagree. The finished rubric works with expert graders from the eeko network, with your own team, or as instructions for a model-based grader, and every item ships with its author's justification.
How much does a private AI benchmark cost?
Higher per item than training data, since every item needs a verified key or rubric, a written justification, and an independent second solve. The price depends on the domain, item length, difficulty tier, whether items are multimodal, and whether you want grading afterward. Sets are smaller than training sets, so the total is usually modest, and the exclusive licence is part of the quote, not an add-on.
Do you grade our model's outputs as well?
We can. The same expert pool can score your model's answers against the rubric, logging score, rationale, and grader ID per item, with enough overlap to report agreement. It suits open-ended tasks that automatic scoring handles badly. Responses you share stay under your confidentiality terms, out of the catalogue, and are deleted or returned on the schedule you set.
What is benchmark contamination?
Contamination is what happens when a benchmark's items, or close paraphrases of them, end up in a model's training data. The model then answers from memory rather than ability, scores rise without any real gain, and improvements stop transferring to live use. Public benchmarks are contaminated by default within a few crawl cycles. The only durable defense is a set that was never public to begin with.
How many items does a private eval set need?
Enough to separate the models you care about with confidence, which is usually a few hundred to a few thousand items spread across skills and tiers, not tens of thousands. We size the set from your use case: how many skills, how many tiers, and how small a difference you need to detect. A refresh schedule then adds new items over time so the set stays ahead of model progress.
Related data services.
Measure the model on something it has not read.
Five minutes to send a dataset request. Within one business day a data lead comes back with questions, a draft skill and tier plan, and a price. No deck. A ranked list of what to test first.
Experienced within
Markets served.
As an enterprise AI agency, eeko systems delivers production AI systems remote-first across the United States and internationally — including these markets: