Building trust in AI: humans, audits, gates

AI gets things wrong sometimes. A better model gets things wrong less often, but no model gets things wrong never, and a company cannot buy its way to never.

What a company can decide is what happens to a wrong answer next. A wrong answer that lands in front of a person as a draft costs 10 seconds: the person reads it, fixes it, moves on. The same wrong answer acted on unchecked costs an apology, maybe money, maybe a story that travels. The model was equally wrong both times. The difference was everything around it, and everything around it is the part the company designs. Checking is one of the 5 parts built around the model, and the one that decides what a mistake costs.

The 3 arrangements

There are 3 ways to arrange what stands between the AI and the outside world.

Arrangement	The AI	The person	Software
Human-in-the-loop	drafts	approves every output	none
Human-on-the-loop	acts	checks a sample afterwards	none
Human-out-of-the-loop	acts	none	checks everything

The arrangement is a choice about the workflow, not about the model. The same model can sit inside any of the 3. So can any of the 4 ways AI enters a company: a chatbot someone uses at their desk, an internal tool, a self-hosted model, a vendor product.

A person approves everything

The AI drafts. A person reads each draft, fixes what is wrong, and only then does anything leave the building.

A mistake cannot get out this way, because a person was always going to read it first. That protection is real only while the reading is real. If someone is approving a draft every 3 seconds, nobody is reading anything, and the protection is gone even though the process looks the same.

The price is that a person stays in every single loop. The company has not automated the work. It has made the drafting fast and kept the judging, which is often a big saving, and a smaller one than “the AI does it” suggests.

Example

Let’s say a support inbox takes 400 emails a day and 6 people answer them, about 65 each. With the AI drafting, each person reads their queue, fixes what is wrong, and sends. The typing is gone and the judging stays. A draft with a wrong delivery date costs one correction, because someone was always going to read it before the customer did.

A person checks a sample

The AI sends its answers on its own. A person reads some of them afterwards.

A single mistake will get through. What the sampling buys is that a repeated mistake gets noticed in days instead of months. It also matters which sample. Reading random replies catches the mistake that runs through everything. Reading the replies customers complained about catches the worst single ones. Teams that do this well read both.

Example

Now say the routine half of that inbox (order status, address changes, invoice copies) goes out unread. 1 person reads 30 sent replies a day picked at random, plus every reply the customer pushed back on. If the AI starts quoting last year’s returns policy on Monday, the random reading surfaces it by midweek.

Software checks everything

The AI sends its answers, and a second system checks each one against a source of truth before it goes out. Whatever fails the check goes to a person instead.

Vendors call this layer guardrails. The word is loose. It covers software checking answers on the way out, and it also covers the standing instructions that tell a model what to refuse, which is a different part of the system doing a different job.

This is the only arrangement that runs at full speed with no one reading, and it is the hardest to build, because the check has to be at least as reliable as the person it replaced. If the check is weaker than the human review it replaced, the workflow has quietly become more fragile, not more efficient.

Example

Now say nobody reads at all. Before a reply goes out, a second system compares it against the order record and the policy page, and holds back any reply that promises a date, a refund or a price it cannot find in those sources. Held replies go to a person. Programmers already work this way with AI: the AI writes the code, and the tests decide whether it gets used.

How to pick

For each action the AI takes, ask: if this goes out wrong and nobody notices, what does it cost? Sometimes the answer is a follow-up email. Sometimes it is money that does not come back. That cost has a name, the blast radius of the action: the cost (in money, reputation, customer harm, time) of one wrong action that nobody caught.

Whether the mistake can be undone matters as much as how big it is. A wrong answer someone can correct tomorrow is cheap even if it is embarrassing. Money sent to the wrong account may never come back, so it deserves heavy checking even if the AI is almost always right.

And the answer is different for different actions inside the same workflow, so one workflow can run different arrangements side by side.

Example

Staying with that inbox, say a wrong answer to “where is my order” costs one more email, so those replies can go out unread. Issuing a refund moves real money and is awkward to reverse, so refunds wait for a person even while order-status replies flow. Releasing a payment to a supplier’s new bank account may be impossible to reverse, so that waits for 2 people.

Everything above assumes mistakes happen by accident. But the emails the AI reads are written by whoever sent them, and a sender can put instructions for the AI inside the email itself. An AI that can look things up or act on accounts may follow those instructions. This is called prompt injection, and it means that wherever outsiders write the input and the AI can take actions, the company should use a tighter arrangement than the error rate alone suggests, because the worst case is no longer random. Where that tightening gets applied is the connection itself, which is what an AI is wired into.

Keep records

Whichever arrangement a workflow uses, keep a record of each run: what the AI was given, what it produced, what the person or the checking system did with it, and what got changed before it went out.

When a customer, a regulator or a manager asks what happened with a specific case, a team with records answers in minutes. A team without them has to say “we don’t know”, which is the worst possible answer in exactly those conversations.

Read across a few months, the records show which tasks it gets right, which it gets wrong, and what people always fix before sending. The cases people keep fixing are worth adding to the test set the tool was tested on before launch, so that whenever the prompt or the model changes, someone can re-run the same cases and compare. How to measure AI results takes the measurement from there.

When employees use personal chatbot accounts for work, none of this gets recorded anywhere the company can see. That, more than any safety worry, is why companies move regular AI work onto company accounts: so there is something to look back at.

Let the AI earn more

A policy written before launch cannot say whether the AI is doing the job well. A month of records can.

So start small: one workflow, a person approving everything, records on. After a month, read the records and act on what they say. Where the AI was right every time, reduce the checking, for example moving order-status replies from approve-everything to sample-afterwards. Where the mistakes mattered, tighten. Where people were approving without reading, fix that, because the protection was already gone. Actions with a big blast radius stay gated no matter how good the record looks. Then take the next workflow through the same steps.

As this spreads, write down which tools are approved for which kinds of data, who is allowed to put AI into a live workflow, and who reads the records.

A company that works this way ends up, a year later, with AI in many places, each with checking that matches what a mistake there costs, and the records to show why. A company that skips the earning and deploys broadly tends to end up with an incident it cannot explain, and then everything gets switched off at once.

When the answer is no

Some jobs should not have AI in them. If one wrong action causes serious harm that cannot be undone, and no check can be trusted to always catch it, there is no safe arrangement, and the honest conclusion is to keep AI out of that decision. The same goes when the checking a job needs costs more than the AI saves: the work is safe, and pointless. Saying no to a use case on these grounds is a perfectly good outcome, and a rarer one than most AI caution suggests. Most jobs, with the right arrangement, work.

None of this needs new technology. It is deciding in advance what happens to a wrong answer, writing down what the AI did, and letting it do more only when its own record says it is ready.