You do not need to know how an engine works to drive. You do need to know that it will not run without fuel, that it makes an unhappy noise when something is wrong, and that it cannot swim.
The same is true here. Nobody is asking you to understand the mathematics. But a rough working model of what a large language model is doing will save you from the two mistakes that cost businesses the most: trusting it when you should not, and abandoning it when it would have been useful.
The short version
- A large language model predicts what text should come next, one small piece at a time.
- It has read an enormous amount and remembers none of it as facts, only as patterns.
- Fluent and correct are separate things. It is always fluent. It is usually correct.
- It knows nothing about your business unless you tell it, every time.
What it is actually doing
Strip everything away and a large language model does one thing: given some text, it works out what is most likely to come next, adds it, and repeats.
Type "the capital of France is" and it produces "Paris", not because it looked Paris up, but because across everything it has read, that is overwhelmingly what follows. Type a half-written email and it continues it. Type a question and it produces the sort of text that follows questions, which is an answer.
That is genuinely the whole mechanism. What makes it remarkable is scale. Trained on a large enough body of text, with enough capacity to store the patterns, "what comes next" stops being autocomplete and starts being something that can summarise a contract, restructure an argument, or spot the flaw in a plan. The behaviour is much bigger than the mechanism, which is why intelligent people disagree about what to call it.
The three things that follow from that
It is confident whether or not it is right
The model produces the most plausible continuation. Plausible and true usually overlap, because most text it read was broadly accurate. But when they come apart, nothing in the mechanism flags it. There is no internal voice saying "I am guessing here". You get the same measured, well-punctuated prose either way.
This is why an invented case reference or a slightly wrong VAT threshold reads exactly like a correct one. It is not lying, which implies knowing better. It is producing likely text. We wrote about how to catch this in practice in AI hallucinations: how to check what AI tells you.
It knows the world in general and your business not at all
It has read a great deal about running a plumbing firm. It has read nothing about running your plumbing firm. It does not know your margins, your suppliers, your terrible customer in Buckley, or that you stopped offering boiler servicing in March.
Every useful answer you get will be one where you supplied that context. This is the single biggest gap between people who find AI transformative and people who find it disappointing, and it is not a skill gap so much as a habit gap. The disappointed group types a question. The other group types a paragraph of background and then a question.
It has no memory unless someone gives it one
By default, each conversation starts from nothing. Some tools now keep notes between chats, and business versions can be connected to your documents, but the underlying model is not learning from you. Tell it your company voice on Monday and, unless the tool is specifically storing that, it will not know it on Tuesday.
The practical response is to keep your context in a document you can paste, or to use the saved-instructions feature most tools now offer. Ten minutes setting that up is worth more than any prompting trick.
The words people will use at you
| Term | What it means to you |
|---|---|
| Model | The engine. GPT, Claude and Gemini are families of them. |
| Prompt | Everything you type. Context plus instruction. |
| Token | A chunk of text, roughly three quarters of a word. Usage is billed in these. |
| Context window | How much it can hold in view at once. Large now, but not infinite. |
| Training data | What it read. Fixed at a point in time. |
| Hallucination | A confident, fluent, wrong answer. |
| Fine-tuning | Adjusting a model on your own examples. Rarely needed by a small business. |
| RAG | Letting the model look things up in your documents before answering. |
There is a longer list in AI jargon explained if someone is about to sell you something.
What this changes about how you use it
Once the mechanism makes sense, a few habits follow naturally rather than needing to be memorised.
Give it the background first. Who you are, what the business does, who the reader is, what good looks like. Then the request. The difference in output quality is not subtle.
Ask it to show its working. "Explain your reasoning" and "what would change your answer?" both surface soft spots that a bare answer hides.
Check anything with a number, a name or a date in it. Those are exactly the details a plausibility engine gets plausibly wrong.
Use it where you can judge the output. It is at its best in the territory where you know enough to spot a bad answer. That is not a limitation to work around, it is where the value is: a good first draft you can correct beats a blank page and beats a perfect answer you cannot verify.
Context: I run [business] in [town]. We do [what]. Our customers are [who]. Our tone is [how we sound].
Task: [what you want].
Format: [length, structure, what to leave out].
Then: list anything you assumed, and what you would need from me to do this better.
What it is not
It is not a search engine, although several now search the web as a step. It is not a database, and it will not reliably recall a specific fact on demand. It is not a decision maker, and treating it as one is how businesses end up defending something no human read. And it is not learning about your company in the background, which is reassuring and also means the context is your job.
What it is: a very widely read, very fast, slightly overconfident colleague who will draft anything you ask, has no stake in whether it is right, and needs telling everything about your business every single time.
Used with that in mind, it is the most useful thing to arrive in business software in twenty years. Used as an oracle, it is a liability with good grammar.
If you would rather see this worked through live than read about it, the free AI Breakfast Club webinar runs online every other Friday morning. Gary builds things on screen, in front of you, and leaves the mistakes in.