Ask a general-purpose AI model to summarise a set of accounts and it will give you a figure for turnover. Ask it again with the turnover page removed, and it will still give you a figure for turnover. The second figure will look exactly as convincing as the first.
That is the whole problem in one demonstration, and it is worth understanding properly, because the usual explanation offered to accountants is that AI is “sometimes inaccurate” and needs “checking”. Both are true and neither is useful. Inaccuracy suggests a system that is trying to be right and occasionally missing. What actually happens is stranger and more specific than that.
The short answer
A language model does not look up figures. It generates the most statistically plausible continuation of the text in front of it. When the figure is present in the source, the plausible continuation is the correct figure. When it is absent, the plausible continuation is a number that looks like it belongs, and the model has no mechanism to notice the difference.
This is not a bug that gets fixed in the next version. It is a description of what the technology does. Which means the answer cannot be a better model or a better prompt. It has to be an architecture that does not ask the model to produce numbers at all.
Why “hallucination” is a misleading word
The industry term suggests a malfunction, as though the system is ordinarily reliable and occasionally departs from reality. The opposite is closer to the truth.
The model is doing exactly the same thing when it is right and when it is wrong. It is producing text that fits. When your accounts are in front of it and the turnover figure is £2,340,891, the text that fits is £2,340,891. When the page is missing, the text that fits is a number of roughly the right magnitude, formatted the same way, sitting in the same sentence structure. From the model’s point of view nothing has changed. There is no internal signal that says “I had evidence for that one and not for this one”.
This is why confidence is worthless as a quality indicator. People instinctively read a hedged answer as less reliable and a direct answer as more reliable, because that is how it works with human colleagues. With a language model, the fluency of an answer carries no information about whether it is grounded in anything.
Why this matters more in accountancy than almost anywhere else
In most professional contexts a plausible but wrong answer is an inconvenience that gets caught. Three features of accountancy work make it considerably worse than that.
Wrong numbers look right
A fabricated legal citation can be checked in a database and found not to exist. A fabricated turnover figure looks exactly like a real one. There is no formatting tell, no missing reference, nothing that distinguishes it on the page. The only way to catch it is to go back to the source document, which is the work you were trying to avoid.
Errors compound through a file
A wrong figure in a lead schedule flows into the trial balance, into the accounts, into the tax computation and into next year’s comparatives. One invented number becomes six wrong numbers across four documents, and the ones downstream are internally consistent with each other, which makes them harder to spot.
The output is filed
Statutory accounts go to Companies House. Tax returns go to HMRC. Both carry a signature and professional responsibility. An error that would be embarrassing in a marketing document is a filing issue here, and the person who signed it owns it.
It is not surprising, then, that when AccountingWEB and Sage surveyed more than 600 UK members for their February 2026 report, the top-ranked concern about AI was accuracy and hallucination, ahead of data security. That is not technophobia. It is accurate risk assessment by people trained to be sceptical of an answer that arrives too easily.
Three things that do not fix it
Each of these is offered regularly as a solution. None of them solves the underlying problem, though two of them help around the edges.
Better prompting
Telling a model “only use figures from the document provided” reduces the frequency of invention. It does not eliminate it, because the instruction is processed as more text rather than as a constraint on the mechanism. You are asking the system to want to be accurate. It does not want anything.
A bigger model
Larger models invent less often because they have seen more. They still invent, and there is an argument that they are more dangerous when they do, because the invented answer is more polished and therefore more likely to pass a quick review.
Asking the model to check its own work
A model asked whether its previous answer was correct will produce a plausible assessment of its previous answer. If the original figure was invented, the assessment of that figure is generated by the same process that invented it. Self-verification inside a single system is circular.
What actually works
The fix is structural. It comes from separating two jobs that general-purpose tools combine: finding the number, and writing the sentence around it.
A system safe to put near a client file has four properties.
| Property | What it means in practice |
|---|---|
| Figures come from extraction, not generation | The number is read out of a source document or calculated arithmetically. The language model may write the sentence around it, but it never produces the digits. |
| Every figure carries a reference to its source | You can click any number on a schedule and see the document, page and line it came from. If there is nothing to click, there is nothing behind it. |
| Absence produces a flag, not an estimate | When the supporting document is missing, the system says so. It does not interpolate, and it does not fall back on a prior-year figure without telling you. |
| Arithmetic is done by arithmetic | Totals, reconciliations and variances are computed by code, not predicted by a model. Language models are famously unreliable at basic addition, and there is no reason to ask them to do it. |
That fourth point deserves emphasis because it is the one people find most surprising. A system can use a language model extensively, for reading documents, classifying transactions, drafting explanations and answering questions, without ever letting it near the arithmetic. The model is good at language. Use it for language.
How to test a vendor in ten minutes
Every AI accounting vendor will tell you their product is accurate. The word is meaningless on its own. Here is how to find out, and it takes about ten minutes in a demo.
- Ask them to import a file with a page deliberately missing. Watch what the system does with the figure that page contained. A flag is the right answer. A number is a serious problem.
- Click a figure on a generated schedule. Ask to see the source. If tracing it back takes more than two clicks, or produces a general reference rather than a specific document and line, the traceability is decorative.
- Ask which parts of the system use a language model and which use deterministic code. A vendor who has thought about this will answer immediately and specifically. A vendor who has not will talk about their model instead.
- Ask for a case where the system got something wrong, and what they changed. Every honest vendor has one. A vendor who claims none is either new or not telling you.
- Give it an unusual UK tax question with no clean answer. Watch whether it hedges appropriately or produces a confident position. Confident is worse.
We build Papercare this way because the founder still signs accounts, and the prospect of an invented figure reaching a filed set concentrates the mind considerably. Figures come from source documents and calculations. Where the evidence is absent, the system says so rather than filling the gap. It is the single design decision the product is built around.
What this means for how you review
If your practice is using AI in any form, the review process needs one change, and it is a small one.
The traditional review question is “does this look right?”. A number that looks right is exactly what a language model produces when it has nothing to work from, so that question no longer does the job it used to.
The replacement question is “where did this come from?”. It is faster to answer than it sounds, because on a well-built system the answer is one click, and on a badly-built system the absence of an answer is itself the finding. Applied to three or four figures per file, it catches the category of error that matters.
This connects directly to the training problem. A junior who has never prepared a schedule by hand has no instinct for which figures are worth questioning, which is a large part of why training juniors in an AI-assisted practice has become a harder problem than it used to be.
Frequently asked questions
Why does AI make up numbers?
Because a language model generates the most plausible continuation of the text in front of it rather than looking figures up. When the number is in the source, the plausible continuation is the correct number. When it is absent, the plausible continuation is a number that looks like it belongs, and the model has no mechanism to distinguish the two situations.
Can AI hallucination be fixed with better prompting?
It can be reduced, not eliminated. An instruction to use only the figures provided is processed as more text rather than as a hard constraint on the mechanism. Prompting is risk reduction, not a solution.
Are newer AI models more accurate with numbers?
They invent less often, but they still invent, and the invented output is more polished. There is a reasonable argument that a more fluent wrong answer is more dangerous, because it is more likely to survive a quick review.
How can I tell if an AI tool invented a figure?
You cannot tell by looking at it, which is the core of the problem. A fabricated figure is formatted identically to a real one. The only reliable method is tracing it to a source document, which is why traceability matters more than accuracy claims.
Is it safe to use AI for accounting work at all?
Yes, with the right architecture. A system that extracts figures from source documents, computes arithmetic with code, references every number and flags missing evidence is safe in a way that a general-purpose chatbot is not. The technology is not the risk. The design is.
Why not just ask the AI to check its own answer?
Because the check is produced by the same process that produced the answer. If the original figure was generated rather than extracted, the verification of that figure is generated too. Self-checking inside one system is circular.
What does “never invents a number” actually mean?
That figures are read from source documents or calculated arithmetically, never produced by the language model. The model can write the sentence around a number, classify a transaction or answer a question, but it does not produce the digits. Where a document is missing, the system reports the gap instead of filling it.
The point
AI is genuinely useful in accountancy. It reads documents faster than a person, it never gets bored on page forty of a bank statement, and it drafts a decent first version of most things.
But the question that decides whether a tool belongs near a client file is not how clever it is. It is what it does when it does not know. A system that produces an answer regardless is not being helpful. It is transferring a risk to you without telling you, and it is doing so in the only professional context where the output gets filed with a signature on it.
Ask any vendor to show you the missing-document case. Ten minutes, and it tells you more than the rest of the demo combined.
See what our system does when the evidence is not there. Book a live demo and ask us to break it.
About the author
Written by the Papercare team. Papercare is a UK working papers and AI review platform built inside a live UK accountancy practice by founder Shrineel Patel FCCA.
Last reviewed: 19 October 2026.
