Evals / QA · retainer
We put numbers on what your AI gets wrong
Test sets built from your real cases, accuracy metrics you can quote, and regression checks on every release. It doesn't matter who built the system.
You shipped AI. Now the hard question: is it right?
Most teams can't put a number on it, and they hear from users when it isn't.
“It worked in the demo”
Real inputs are messier than demo inputs, and without measuring it nobody knows how much quality dropped between the demo and your users.
Every prompt tweak is a gamble
You improve one answer and silently break three others. Without regression checks, your users find the breakage before you do.
No numbers for the people who decide
Is the assistant right 95% of the time or 80%? “Seems fine” doesn't hold up in a budget meeting, or an incident review.
What you get
Quality control that keeps running. A one-off report is already stale by the next release.
- Test sets built on your data
- Real cases pulled from your own logs and users. Synthetic benchmarks flatter the system.
- Accuracy metrics you can quote
- An agreed definition of “correct”, then measured and reported as numbers you can take to stakeholders.
- Regression checks on every change
- New prompt, new model, new data. The eval suite runs before any of it reaches users, so a fix to one answer stops quietly breaking others.
- A growing eval suite
- Every production incident becomes a test case. The suite gets stricter as your system ages.
How it works
Collect real cases
The questions that come up, taken from your logs, your support tickets and your users.
Agree the metrics
Together we define what “correct” means for your system and set the bar to clear.
Baseline current quality
The first measurement of where the system stands today, on cases you chose rather than we did.
Guard every release
Regression runs on each change and reports over time. That's the retainer part.
We hold our own work to this standard
Projects we shipped behind eval pipelines, plus open results anyone can check.
In production
Expert Blockchain Chatbot
Production RAG with quality we measured. Retrieval goes past plain vectors (SQL, entity lookup, real-time data), and an evaluation framework (RAGAS) scores the answers.
Read the project →In production
LetAI: Nutrition Estimation Agent in Production Chat
The agent only reached production chat after its evaluation pipeline was expanded, with new datasets and broader case coverage added before release.
Read the project →In production
Market Analyst for a Chemical Manufacturer
Recommends what to produce next on evidence a buyer can re-check: every figure traced to its source row and cleared by named quality gates.
Read the project →Open code
ARLC 2026 Legal RAG Challenge
3rd of 155 teams in the challenge's warm-up phase, scoring 0.954/1.0. An agentic RAG pipeline that answers questions over real DIFC legal documents, with every answer graded by AI judges. The code is open, so anyone can check it.
View the code ↗Open code
BitGN Personal Agent Challenge
We built an AI agent for a safety challenge that hides prompt-injection attacks inside ordinary tasks. The agent has to get the work done and turn down the traps, and it's scored deterministically by what it actually changed. Our defense layer checks 40+ attack patterns across 14 categories. The code's on GitHub.
View the code ↗Publication
zebra_simple: Zebra Puzzle Test for LLMs
Our published LLM reasoning benchmark. Open evidence of how we measure model behaviour, and anyone can check it.
Read the publication →
Frequently asked questions
Eyeballing doesn't scale, and it misses regressions. You check the answer you just fixed and not the thirty you didn't touch. An eval suite checks all of them on every release and gives you a number rather than an impression.
Evals are vendor-independent. We measure how the system behaves from the outside, on your data. You get an honest picture of what you bought, whoever built it.
No, and be wary of anyone who promises it. What evals give you is a number. You'll see the current error rate, watch whether it's improving, and catch regressions before they ship. Evals won't make mistakes impossible. They make them visible.
Unit tests check for an exact output, and language models don't give you exact outputs. Evals score behaviour across a whole dataset (accuracy, groundedness, whether it refuses when it should) and track those scores over time, which is what you need for a system that isn't deterministic.
Eval runs on every change, regular quality reports, and growing the test sets as new cases show up in production. Price depends on your system and how often you ship, so we quote after a scoping call. The form below starts one.
No AI in production yet? The two-week agent build starts with the task that pays for itself first, and ships it.
Vendor-independent
Discuss your case
Tell us what your AI does and what's worrying you about it.