Your AI pilot succeeded and it told you nothing
A pilot run on curated data, with the team watching, against no baseline, is a demonstration. It predicts almost nothing about production behaviour — and the gap is where most AI budgets go.
Thomas Reinhardt, Head of AI Services ·
What the pilot actually measured
Most AI pilots are evaluated on a sample that someone chose. The choosing is the problem. A curated set is, by construction, the part of the distribution the system handles well, and performance on it tells you about the curation rather than the system.
The tail is where production lives. It is also where the cost sits, because the difficult cases are the ones that consume tokens, trigger retries and land on a human anyway.
Measure the baseline before you build
If you cannot state what the current process costs per unit of work and how long it takes at the median and the 95th percentile, you cannot claim an improvement later. You will be comparing a measured system against a remembered one, and memory is generous.
Six weeks of baselining feels like six weeks of not building. It is the cheapest part of the programme.
Build the evaluation set first
Have the people who own the process write questions with known answers, before any system exists to answer them. Several hundred is usually enough. That set becomes the acceptance bar, the regression suite and the thing you run on every deployment.
It also settles arguments. When the system changes and someone says it feels worse, the set says whether it is.