How large language models work

A language model takes text, splits it into tokens of roughly three quarters of a word, and predicts the next one. Then it appends that token and predicts again. That loop produces everything: essays, code, translation, summaries, answers.

The model is a network of billions of numbers called parameters, adjusted during training so that its predictions match real text. Nobody writes the parameters; training discovers them.

The transformer, and what attention does

The architecture since 2017 is the transformer. Its key mechanism, attention, lets the model weigh how much every token in the input should influence its interpretation of every other token. That is how it resolves which noun a pronoun refers to across three paragraphs, and it is why these models handle long-range structure that earlier architectures could not.

The three stages of making one

Pre-training. Predict the next token across an enormous body of text. This is where nearly all the compute goes, and where the model acquires grammar, facts and reasoning patterns as a side effect of prediction.

Fine-tuning. Further training on curated examples of the behaviour you want: answering helpfully, following instructions, staying in a format.

Preference training. Humans or another model rank outputs, and the model is trained toward the preferred ones. This shapes tone and refusal behaviour, and it is where most of what people perceive as personality comes from.

Why they make things up

The model was trained to produce plausible text, not verified text. It has no database to check against and no internal signal that distinguishes a fact it learned from a pattern that merely sounds right. A confident, fluent, wrong answer is not a malfunction; it is the same process working as designed on a question where the pattern was misleading.

Retrieval helps: give the model the actual source text in the prompt and ask it to answer from that. It reduces fabrication substantially and does not eliminate it.

The context window

The amount of text the model can consider at once. Everything, your question, the conversation, any documents you paste, must fit. Larger windows allow whole codebases and cost more per request, because attention cost grows faster than linearly with length.

What they are and are not

They are extremely good at transforming text, drafting, summarising, translating, explaining and writing code. They are unreliable at arithmetic without tools, at anything requiring current information they were not given, and at knowing what they do not know.