Testing AI before trusting it

Before AI goes anywhere near real work, a team can find out whether it will hold up on that work. The material for the test is already sitting in the business: the tickets already answered, the invoices already coded, the reports already checked. Somebody knows what the right answer was in each of those cases, and nobody has to be paid to produce it again.

Run the AI on 20 to 30 of them, read what comes back against what actually happened, and most of what a team wants to know before committing is answered in an afternoon.

Why a demo does not answer the question

A demo is a run on inputs somebody else chose. The vendor’s cases are clean, the prompt was tuned against them, and whatever the tool is weak on was not on the list. That is not dishonesty. Anyone showing a tool shows it working.

Published scores have a different problem. They measure standardised tasks, they age in months, and the arithmetic of what a leaderboard claim licenses is worked through where the model families live. None of it is about one company’s work.

And the gap is not something a vendor could close. Nobody outside the business has the business’s inputs: the abbreviations 3 people use, the supplier who sends scans, the status code that means one thing in one warehouse and something else in the next. That material only exists inside the company, which is also where the test has to happen.

Teams do eventually find all this out. The usual route is to switch the tool on and discover it over a few months of live work, on real customers. Which model or prompt is best does settle by somebody running real work through it, and this is that same settling done on purpose, in a day, before anything depends on the answer.

The test cases are work the company already did

Collect 20 to 30 real cases from the past where the right answer is known and already recorded. A small fixed set of cases like this, kept and re-run, is called a test set.

Past work is what makes it cheap. Somebody already coded that invoice, sent that reply, checked that figure, and somebody already signed it off. The right answers cost nothing to produce because they were produced months ago as part of doing the job. They also make the reading fast, for the same reason checking any AI output is fast when the right answer already sits somewhere the checker can reach.

Which cases go in matters more than how many. The set is built to cover the kinds of input the work actually contains: mostly the ordinary ones, because that is most of the work, plus the awkward ones people remember without being asked. The scanned one. The one in another language. The one from a customer with no history. Picking 25 cases at random will fill the set with the easy majority and miss exactly the kinds that break, and a set of nothing but hard cases measures something the work never asks for.

Two things are worth checking about the cases before the run. Whether the work has changed since, because a set drawn from before a policy or a product line changed tests a job nobody does now. And whether any of the cases have been used as examples inside the prompt, because a case the tool was shown the answer to is not a test of anything.

Example

Let’s say a finance team codes supplier invoices to cost centres, around 40 a week, and wants to know whether AI can do the first pass. The past year holds thousands of invoices already coded by a person and approved, so the answers exist.

Somebody picks 25. 18 are routine: familiar suppliers, one line, an obvious department. 7 are the ones the team would name if asked what makes this job annoying: 2 arriving as scans, 3 from suppliers who bill through a reseller, 1 credit note, 1 in a currency the team sees twice a year. Building the set takes about an hour, most of it spent pulling the approved codes into the same sheet as the invoices.

Reading the outputs

Run the AI on the cases the way the real system would run: the same prompt, the same source material in front of it. A tool that would have the supplier list and the chart of accounts available in production, tested in a bare chat window without them, produces numbers about a system nobody is going to build.

Then read every output against what actually happened, and record a verdict on each one: right, or wrong and in what way. Reading every case is what makes 20 to 30 the size. 500 cases would need scoring by sampling, which is the thing this test exists to avoid.

Two habits keep the reading honest. What counts as right is written down before the reading starts, not settled case by case as the outputs arrive, because a bar that moves while it is being applied always ends up in the same place. And the person reading knows the work, because judging whether a cost centre is right is the job itself, not a clerical check anyone can do.

Sometimes the AI and the record disagree and the record is wrong. That is a finding, and a useful one, but it is a finding about the records rather than about the model, and it belongs in a different column.

Example

Let’s say those 25 invoices go through, and 20 come back with the code that was approved last year.

Of the 5 that disagree, 1 turns out to be an old keying error: the AI’s code is the right one and the approved record is wrong. Of the remaining 4, 3 are the reseller-billed software subscriptions. Every one of them goes to the reseller’s own cost centre instead of the department that uses the software, because the only place the using department appears is a line in the invoice body that nothing in the prompt told the model to look for. The 4th is the twice-a-year currency, where the total is read off the wrong line.

So 4 real misses out of 25, and 3 of the 4 are one mistake wearing three coats.

What the reading shows

Three things come out of the reading, and only the first is a number.

Where it is right. The parts of the work the tool can carry are now named, in the team’s own cases rather than in general.

Where it is wrong, and whether the wrongness matters. A clumsy sentence and a wrong figure are both misses and they are not the same finding.

And whether the wrong ones cluster. This is the one worth slowing down for. Wrong answers scattered across all kinds of input are a review problem: whoever reads the output catches one here and one there, and no single customer collects them. Wrong answers concentrated in one kind of input are the same mistake every time, landing on the same customers every time, and a sample pulled at random after rollout is built to under-represent it.

A cluster usually names its own fix, which is why it is worth finding before launch rather than after. Something the model was never told, a source it was never given, a distinction that lives in somebody’s head. Fix it and re-run the same cases.

What the reading does not produce is a rate. 4 misses out of 25 is not “16% wrong in production”, because 25 cases chosen by hand cannot measure a rate and were never meant to. What they show is the shapes the wrong answers take, and the shapes are what a team can act on.

Two models, one set of cases

Once the cases exist, every later comparison is cheap. The second model runs on the same 25 inputs and gets read the same way. So does the alternative prompt, the version with the supplier list added, the cheaper tier.

Two rules keep the comparison meaningful. Change one thing at a time, or a difference cannot be traced to anything. And treat a gap of 1 or 2 cases as a tie, because at this size that gap is noise. What separates the options is usually not the count but which cases each one gets wrong.

Example

Let’s say the team tries a second model on the same 25 invoices. It agrees with the record on 21 where the first agreed on 20, which settles nothing, and it misses the same 3 reseller invoices, which settles rather a lot: the gap is in what the model was given, not in which model it is.

The run itself is small money. 25 invoices through 2 models is well under a dollar even on the expensive tier. The reading is about 2 hours of somebody who knows the cost centres, and it happens twice.

So they change the prompt instead, telling the model where the using department appears on a reseller invoice, and re-run the same 25. The 3 reseller cases come back right. The currency case still does not.

What passing means

There is no general pass mark, and any number offered as one is arbitrary. The bar comes from what one wrong answer costs once somebody acts on it.

Where a person reads every output before it leaves the building, a set that comes back mostly right, with the misses scattered and cheap, is enough to start. The wrong ones are going to land in front of a reader whose job is to catch them.

Where the output goes straight into a system, or to a customer, or moves money, the count matters less than one question: would any of the wrong answers in this set have cost something that cannot be taken back? One case like that is a reason to keep a person in the loop, whatever the other 24 did.

And a cluster is a no on its own terms, in either arrangement, until somebody has fixed what causes it. Not because 3 wrong cases out of 25 is a lot, but because those 3 are a promise about who gets the wrong answer next.

Passing is a decision to start under a chosen arrangement, on a chosen amount of checking. It is not a verdict that the AI is right, and the set is too small to support that verdict anyway.

The set keeps working after launch

Nothing about the set gets thrown away on the day the tool goes live. It is what gets re-run when the prompt is edited, when the vendor ships a new model version, when somebody proposes moving to a cheaper tier, and it is the only thing that turns “the new version seems fine” into a comparison. Cases that go wrong in real use get added to it, so the set grows in exactly the places the work is hard.

The day the cases are collected is also the cheapest day to write down how long the work takes now, because after the rollout there is no honest way to reconstruct it. That reading, and everything else that only becomes measurable once the tool is in use, is how to measure AI results.