What AI is bad at in business operations
The failure that costs a business money is not the obvious one. Output that is plainly nonsense gets caught by the first person who reads it. The expensive failure is output that is right in form, right in nearly all of its content, and wrong in one part that a person reading at normal speed has no reason to stop on. The cost is not the error. The cost is that the error passes.
Where models break
Section titled “Where models break”Facts with one right answer. Specific numbers, specific names, specific dates, anything that happened recently. If it was not in what the model saw during training, or was there inconsistently, or has changed since, the model still produces an answer, and it produces the most plausible-looking one. That is hallucination, and it is the mechanism running normally rather than breaking.
What the business knows. The CRM, the wiki, last quarter’s numbers, one customer’s history. None of it was in training. A model on its own knows nothing about the company it is being used inside until something puts that material in front of it, which is the work of tools and retrieval.
Doing anything. A model produces text. It can write the email; sending it needs a connection to the mail system. It can write the database query; running it needs a connection to the database and permission to use it. It can describe how to book a flight; booking one needs an account with an airline. Everything that turns text into an action is separate engineering, with its own permissions and its own record of what happened.
Remembering. The products do remember now. The major assistants keep short entries from past chats and put the relevant ones in front of the model before the new message, which is why a chat can open already knowing the business. The model itself does not remember: what it can hold is what sits in front of it at that moment. And what memory holds is a summary the product chose to keep. It does not travel to another product, and it is not access to the company’s systems.
Arithmetic. Current reasoning models write out their working before the answer, and on multi-step problems they are far stronger than the generation before them. Strong at maths is not the same as reliable at arithmetic. A slip in one step carries forward into every step after it, and long chains are where slips happen. Anything you would open a spreadsheet for belongs in a calculator, a database or a code runner, with the model choosing which one runs rather than working the answer out itself.
Anything safety-critical. Medical diagnosis without a physician. Legal advice without a lawyer. A financial decision without a compliance review. Not because the model can never get these right, but because the cost of being subtly wrong is too high to accept on output nobody checked.
The same answer every time. If one input has to produce one output, always, that is a rule and not a model. Randomness is built in. Two runs of the same prompt can differ, and at volume they will.
Plans that have to hold once the work starts. A model can write a 5-step plan. Watching how step 1 actually lands and changing step 2 before running it is what an agent does, and agents now stay on a task far longer than they could a generation ago. They stay fragile in the same place: the more actions run between a person looking at the work and the outcome, the more rests on the model noticing that things did not go the way it expected.
”Almost true”
Section titled “”Almost true””A model’s wrong answer does not look wrong. It is not shorter, vaguer or less confident than a right one, and it is usually correct in every part except the one that matters. Almost true is the name for output like that: not that a model is sometimes wrong, but that its wrongness is invisible at the speed people check things.
That error comes from a document holding two plausible numbers. The same failure has another end: a subject where the model has nothing at all.
Both are one problem for the person checking. In the invoice row nothing is invented and one true number is picked over another. In the answer above everything about the company is invented and it reads like a briefing note. Neither looks wrong.
It survives review for a specific reason. A person checking work they did not write reads for whether it looks right, because reading for whether it is right means going back to the source, and that costs nearly as much as doing the work by hand. Looking right is exactly what a model is best at producing. So the review step a company counts on is aimed at a property the model satisfies even when it is wrong.
The expensive versions all have this shape. A summary that is accurate except for one sentence that shifts what the source said. Code that runs and carries a quiet bug. A confident answer about a competitor’s pricing that is plausible and wrong.
The defence is to make the output checkable against something other than itself. Ask the model to cite the sentence each claim came from, and reading for plausibility becomes reading for accuracy, because now there are two things to compare. Code comes with tests that run. On the invoice sheet, one extra field fixes it: the model returns the label it took the figure from, so “Contract value” appears in the row next to a number that was supposed to be the balance.
How much checking the work deserves
Section titled “How much checking the work deserves”Not every task earns the same amount. What decides it is the cost of a missed error, which the hub calls blast radius: the cost of one wrong thing that nobody caught.
A support ticket put in the wrong category costs one re-route. Someone notices within a day, moves it, and nothing else follows. Checking every ticket against the model’s category would cost more than the errors do, so reading a sample is the right amount of checking.
A figure pulled from an invoice enters a payment run. A wrong one sends real money and gets found weeks later by an accountant, if at all. It is the same extraction task as the ticket with a different cost of being wrong, which is why the figure needs a second source rather than a second reading: tax totalled against the line items, the balance compared with the purchase order.
The list above is where models break. What decides whether it matters is not on the list. It is whether the work has something to check the answer against, and what it costs when nobody does. What AI is good at is the same test read from the other side.