Human in the loop and guardrails: how much checking AI needs

How much checking AI needs depends on what one uncaught mistake would cost. Where a mistake is expensive or cannot be undone, a person approves every output before it goes anywhere, which is what human in the loop means. Where a mistake is cheap to fix, a person checking a sample afterwards, or software checking every answer, can be enough.

AI gets things wrong sometimes. A better model gets things wrong less often, but no model gets things wrong never, and a company cannot buy its way to never.

What a company can decide is what happens to a wrong answer next. A wrong answer that lands in front of a person as a draft costs 10 seconds: the person reads it, fixes it, moves on. The same wrong answer acted on unchecked costs an apology, maybe money, maybe a story that travels. The model was equally wrong both times. The difference was everything around it, and everything around it is the part the company designs. Checking is one of the 5 parts built around the model, and the one that decides what a mistake costs.

The 3 arrangements

There are 3 ways to arrange what stands between the AI and the outside world.

Arrangement	The AI	The person	Software
Human-in-the-loop	drafts	approves every output	none
Human-on-the-loop	acts	checks a sample afterwards	none
Human-out-of-the-loop	acts	none	checks everything

The arrangement is a choice about the workflow, not about the model. The same model can sit inside any of the 3. So can any of the 4 ways AI enters a company: a chatbot someone uses at their desk, an internal tool, a self-hosted model, a vendor product.

A person approves everything

The AI drafts. A person reads each draft, fixes what is wrong, and only then does anything leave the building.

A mistake cannot get out this way, because a person was always going to read it first. That protection is real only while the reading is real. If someone is approving a draft every 3 seconds, nobody is reading anything, and the protection is gone even though the process looks the same.

The price is that a person stays in every single loop. The company has not automated the work. It has made the drafting fast and kept the judging, which is often a big saving, and a smaller one than “the AI does it” suggests.

Example

Let’s say a support inbox takes 400 emails a day and 6 people answer them, about 65 each. With the AI drafting, each person reads their queue, fixes what is wrong, and sends. The typing is gone and the judging stays. A draft with a wrong delivery date costs one correction, because someone was always going to read it before the customer did.

A person checks a sample

The AI sends its answers on its own. A person reads some of them afterwards.

A single mistake will get through. What the sampling buys is that a repeated mistake gets noticed in days instead of months. It also matters which sample. Reading random replies catches the mistake that runs through everything. Reading the replies customers complained about catches the worst single ones. Teams that do this well read both.

Example

Now say the routine half of that inbox (order status, address changes, invoice copies) goes out unread. 1 person reads 30 sent replies a day picked at random, plus every reply the customer pushed back on. If the AI starts quoting last year’s returns policy on Monday, the random reading surfaces it by midweek.

Software checks everything

The AI sends its answers, and a second system checks each one against a source of truth before it goes out. Whatever fails the check goes to a person instead.

Vendors call this layer guardrails. The word is loose. It covers software checking answers on the way out, and it also covers the standing instructions that tell a model what to refuse, which is a different part of the system doing a different job.

This is the only arrangement that runs at full speed with no one reading, and it is the hardest to build, because the check has to be at least as reliable as the person it replaced. If the check is weaker than the human review it replaced, the workflow has quietly become more fragile, not more efficient.

Example

Now say nobody reads at all. Before a reply goes out, a second system compares it against the order record and the policy page, and holds back any reply that promises a date, a refund or a price it cannot find in those sources. Held replies go to a person. Programmers already work this way with AI: the AI writes the code, and the tests decide whether it gets used.

How to pick

For each action the AI takes, ask: if this goes out wrong and nobody notices, what does it cost? Sometimes the answer is a follow-up email. Sometimes it is money that does not come back. That cost has a name, the blast radius of the action: the cost (in money, reputation, customer harm, time) of one wrong action that nobody caught.

Whether the mistake can be undone matters as much as how big it is. A wrong answer someone can correct tomorrow is cheap even if it is embarrassing. Money sent to the wrong account may never come back, so it deserves heavy checking even if the AI is almost always right.

And the answer is different for different actions inside the same workflow, so one workflow can run different arrangements side by side.

Example

Staying with that inbox, say a wrong answer to “where is my order” costs one more email, so those replies can go out unread. Issuing a refund moves real money and is awkward to reverse, so refunds wait for a person even while order-status replies flow. Releasing a payment to a supplier’s new bank account may be impossible to reverse, so that waits for 2 people.

Everything above assumes mistakes happen by accident. But the emails the AI reads are written by whoever sent them, and a sender can put instructions for the AI inside the email itself. An AI that can look things up or act on accounts may follow those instructions. This is called prompt injection, and it means that wherever outsiders write the input and the AI can take actions, the company should use a tighter arrangement than the error rate alone suggests, because the worst case is no longer random. Where that tightening gets applied is the connection itself, which is what an AI is wired into.

Keep records

Whichever arrangement a workflow uses, keep a record of each run: what the AI was given, what it produced, what the person or the checking system did with it, and what got changed before it went out.

When a customer, a regulator or a manager asks what happened with a specific case, a team with records answers in minutes. A team without them has to say “we don’t know”, which is the worst possible answer in exactly those conversations.

Read across a few months, the records show which tasks it gets right, which it gets wrong, and what people always fix before sending. The cases people keep fixing are worth adding to the test set the tool was tested on before launch, so that whenever the prompt or the model changes, someone can re-run the same cases and compare. How to measure AI results takes the measurement from there.

When employees use personal chatbot accounts for work, none of this gets recorded anywhere the company can see. That, more than any safety worry, is why companies move regular AI work onto company accounts: so there is something to look back at.

Let the AI earn more

A policy written before launch cannot say whether the AI is doing the job well. A month of records can.

So start small: one workflow, a person approving everything, records on. After a month, read the records and act on what they say. Where the AI was right every time, reduce the checking, for example moving order-status replies from approve-everything to sample-afterwards. Where the mistakes mattered, tighten. Where people were approving without reading, fix that, because the protection was already gone. Actions with a big blast radius stay gated no matter how good the record looks. Then take the next workflow through the same steps.

As this spreads, write down which tools are approved for which kinds of data, who is allowed to put AI into a live workflow, and who reads the records.

A company that works this way ends up, a year later, with AI in many places, each with checking that matches what a mistake there costs, and the records to show why. A company that skips the earning and deploys broadly tends to end up with an incident it cannot explain, and then everything gets switched off at once.

When the answer is no

Some jobs should not have AI in them. If one wrong action causes serious harm that cannot be undone, and no check can be trusted to always catch it, there is no safe arrangement, and the honest conclusion is to keep AI out of that decision. The same goes when the checking a job needs costs more than the AI saves: the work is safe, and pointless. Saying no to a use case on these grounds is a perfectly good outcome, and a rarer one than most AI caution suggests. Most jobs, with the right arrangement, work.

None of this needs new technology. It is deciding in advance what happens to a wrong answer, writing down what the AI did, and letting it do more only when its own record says it is ready.

Common questions

What is human in the loop in AI?

Human in the loop means a person reviews and approves each piece of AI output before it is sent or acted on. The AI drafts, and the person checks it, fixes what is wrong and releases it. It is the safest way to use AI where one mistake would be expensive or impossible to undo, and it only protects anything while the person is genuinely reading, not approving a draft every few seconds.

What is the difference between human in the loop and human on the loop?

With human in the loop, a person approves every AI output before it goes out. With human on the loop, the AI acts on its own and a person checks a sample of its work afterwards. The first stops every mistake but keeps a person on every task. The second lets a single mistake through, needs far fewer people, and catches a mistake that keeps repeating within days rather than months.

What are AI guardrails?

AI guardrails are checks built around an AI system to stop bad output before it reaches anyone. The word covers two different things: software that tests each answer against trusted records, such as the order system or the policy page, and holds back anything that fails; and standing instructions that tell the model what to refuse. Software checks let AI work at full speed with nobody reading every answer, but only if the check is at least as reliable as the person it replaces.

How much human oversight does AI need?

As much as one uncaught mistake would cost, and more where the mistake cannot be undone. A wrong answer that is cheap to correct needs only light checking, such as a person reading a sample afterwards. Anything that moves money, commits the company to a customer or cannot be reversed is worth having a person approve every time. The safe way to reduce oversight is to start with a person checking everything, keep a record of each case, and loosen the checking only where the record shows the AI was right every time.

Key takeaways

• AI gets things wrong sometimes. The useful question is not how often, but what happens to a wrong answer next.
• There are 3 arrangements: a person approves everything before it goes out, a person checks a sample afterwards, or software checks everything and holds back what fails.
• Pick by asking what one uncaught mistake costs. That cost is called the blast radius. Cheap and easy to undo: light checking. Expensive or impossible to undo: a person approves every one.
• The same workflow can need different arrangements for different actions. A wrong delivery date and a wrong refund do not deserve the same level of checking.
• When outsiders write the text the AI reads, mistakes stop being accidents. An email can carry instructions aimed at the AI, so anything that can act on an account needs the tighter arrangement.
• Keep records: what the AI was given, what it produced, what a person changed. The records answer questions later, and they show which tasks the AI is ready to do with less checking.
• Let the AI earn more slowly. Start with everything checked, read a month of records, and only then reduce the checking where the records show it was right every time.