Type half a text message and your phone offers three words. Ask a chatbot a question and its answer appears a few characters at a time, as if someone were typing. That streaming is not a visual effect. The program really does produce its answer one small piece at a time, and each piece is a weighted draw from a list of possibilities that it has just computed.
The earlier piece How Large Language Models Actually Work (29/02) tells that story from the outside: guess the next word, repeat, and tune with human feedback. This course is the inside view. A graduate of a language-model course can say what a token is, why the model looks at earlier words the way it does, what number training pushes down, what the dials on a chat product mean, and why a given budget buys a model of a given size. They also know that the same machinery, run on noise instead of words, makes pictures.
Everything here is a general description. Specific systems differ in detail, and where a company or model is named it is only to date a fact, not to recommend anything.
- Who takes it
- Year 3 of the AI Development specialization of Computer Science, and as an elective for data science, linguistics and physics students. Typical courses: Stanford's CS224N (Natural Language Processing with Deep Learning) and the Hugging Face course; most begin at word vectors and end at large-scale pretraining and alignment.
- What it is for
- Understanding how a transformer turns text into probabilities, how it is trained, and how its output is sampled.
- Comes after
- Deep Learning (36/35), and behind it Machine Learning (34/32).
- Leads to
- Building AI Applications (36/78) and AI Safety, Alignment & Governance (36/79); also Computational Linguistics (36/30) and Data Ethics, Privacy & Fairness (36/37).
- What this piece gives you
- You will not be able to train a large model after this. You will be able to tokenise a sentence by hand, read an attention table, compute a loss, predict what a temperature dial does, and do the arithmetic that sizes a model. It goes further than 29/02 and 36/35 and does not repeat them: if you have not met embeddings or backpropagation, read those first.
Tokens: the pieces a model actually reads
A model does not read letters or words. It reads tokens: pieces of text from a fixed vocabulary, usually tens of thousands of entries long. Common words are one token. Rarer words are split into familiar pieces, and strange strings fall back to single characters or bytes, so nothing is ever unreadable.
The vocabulary is usually built by byte-pair encoding (BPE), borrowed from data compression and adapted for translation by Sennrich, Haddow and Birch in 2015. Start with single characters. Count which adjacent pair is most common in the training text. Glue that pair into a new symbol. Repeat. The number of merges you allow decides the vocabulary size.
Training text (a classic toy corpus): low ×5, lower ×2, newest ×6, widest ×3. An underscore marks the end of a word.
Slide to the end and "lowest", a word never seen in training, is still built from pieces the tokenizer learned. That is the point: a fixed vocabulary that covers everything. The price is that text is not treated equally. A vocabulary trained mostly on English text has fewer merges for other languages and scripts, so the same sentence can cost many more tokens in some languages than others (Petrov and colleagues measured this across languages in 2023). More tokens mean more cost per answer and less room in the context window. Modern vocabularies range widely; the Llama 3 family, for one, uses about 128,000 tokens.
Attention: every word asks, every earlier word answers
The transformer, introduced by Vaswani and colleagues in 2017, has one trick that earlier designs lacked. Each token makes three vectors from its embedding: a query ("what am I looking for?"), a key ("what do I offer?") and a value ("what do I hand over if chosen?"). A token's query is compared with every key by a dot product. The scores go through a softmax, which turns them into weights that add up to 1, and the output is the weighted mix of the values.
A text generator also needs a causal mask: when the model predicts what follows word 5, it may not look at words 6 and 7, or training would be cheating. Masked, each token sees only itself and earlier tokens. Below, the vectors are small and hand-made (the numbers are invented, so you can follow the arithmetic). A real model learns them, and has dozens of such "heads" in each of many layers.
Head A looks for nouns; head B looks mostly one place back, using the position of each token. Both read the same keys. Hand-set, so that you can check the sums.
Notice what the table cannot do. Pick "it" with head A: the weight splits evenly between "trophy" and "suitcase". Nothing in those vectors can say which one "it" means, because the clue ("big") comes later. In a real model, later layers let the representation of "big" and "it" meet, and the answer comes from many layers acting together. Notice also the cost: every token is compared with every earlier one, so the number of comparisons grows with the square of the text length. This is why long contexts are expensive, and why much engineering goes into making attention cheaper.
Pretraining: the loss is a surprise meter
Pretraining has one objective. Show the model a stretch of text, hide the next token, and ask for a probability for every token in the vocabulary. The score for that guess is the negative logarithm of the probability given to the token that actually came next. Give it probability 1 and the loss is 0. Give it a tiny probability and the loss is large. Average over trillions of tokens and you have the number that training pushes down.
A model that knew nothing and called every one of 50,000 tokens equally likely would score log₂ 50,000 ≈ 15.6 bits per token (a perplexity of 50,000). Perplexity 8 means it is, on average, as unsure as a fair eight-sided die. Below is the oldest possible language model, a bigram that looks only at the single previous word. A transformer is the same idea with thousands of previous tokens and a vastly better way to use them.
Trained on: "the cat sat on the mat . the cat ate the fish . the dog sat on the log . the dog ate the bone ."
Test it on "the fish sat on the cat". The model never saw "fish sat", so it gave that pair probability zero, and −log of zero is unbounded. One impossible word ruins the sentence. Smoothing hedges against this, and big models do the equivalent by never giving any token exactly zero. Lower loss on text the model has not seen is the only honest sign of progress: a model can reduce its loss by memorising, which is why training text is kept apart from test text.
Sampling: how a distribution becomes a word
The network's output is a list of scores (logits), one per token. To get a word, those scores become probabilities by a softmax, and then one token is drawn. Three dials change the draw. Temperature T divides the scores first: low T sharpens the distribution toward the top choice, high T flattens it. Top-k keeps only the k most likely tokens. Top-p (nucleus sampling, from Holtzman and colleagues in 2019) keeps the smallest set of tokens whose probabilities add up to p. The rest are discarded and the survivors are rescaled.
Next word after "The capital of Australia is" (scores invented for the demo; real models differ).
At T near 0 the model almost always gives the top word: stable and repeatable, and sometimes dull. At T = 2 the long tail wins more often, including words that are wrong. The wrong answer "Sydney" is not a bug in the arithmetic. It is a plausible-looking continuation that holds real probability, and the sampling dial decides how often it is picked. This is part of why the same prompt can produce different answers.
Scaling laws: how much model, how much data
Kaplan and colleagues (2020) found that a language model's test loss falls as a smooth power law in model size, data and compute across many orders of magnitude. Hoffmann and colleagues (2022, "Chinchilla") revised the recipe. For a fixed compute budget, they argued, model size and training tokens should grow together; the compute-optimal point is around 20 training tokens per parameter. They tested it: Chinchilla had 70 billion parameters and used the same compute as the 280-billion-parameter Gopher but about four times the data, and it did better on a broad set of tasks.
Putting the two together gives N = √(C ÷ 120). Chinchilla's own budget is a check: C = 6 × 70 billion × 1.4 trillion ≈ 5.9×10²³, and √(5.9×10²³ ÷ 120) is 70 billion.
Chinchilla did not end the argument. A model that will be run billions of times is cheaper to serve if it is small, so it can be sensible to train a small model far past the "optimal" 20 tokens per parameter, spending more on training to save on every later answer. Several current models do this. Loss is also not the same as ability: how well a model does on a particular task can move more abruptly than the loss does, and whether those jumps are real or an artefact of how tasks are scored is itself disputed (Schaeffer and colleagues, 2023). Newer systems also spend extra compute while answering, not only while training. Whether the curves keep paying off, and for how long, is an open question that labs, academics and sceptics answer differently.
After pretraining: preferences, invented facts and the memory bill
A pretrained model is a text continuer, not an assistant. Three further stages are common, though recipes differ between organisations.
| Stage | What it uses | What changes |
|---|---|---|
| Supervised fine-tuning | Written examples of good answers | The model learns the format of helping |
| Preference tuning (RLHF, DPO) | Humans or models ranking pairs of answers | It moves toward answers people prefer; RLHF trains a reward model first (Ouyang et al. 2022), DPO skips it (Rafailov et al. 2023) |
| Reasoning training | Problems with checkable answers | It learns to write longer chains of steps before answering (e.g. DeepSeek-R1, January 2025) |
Hallucination is the confident invention of facts. Kalai and colleagues (2025) argue it is partly a consequence of training and scoring that reward guessing over saying "I am not sure", much as a student who guesses on a multiple-choice test beats one who leaves blanks. A smaller part is simply that rare facts appear too few times to be learned. Retrieval from a trusted source, covered in Building AI Applications (36/78), reduces it but does not remove it.
Context windows are the amount of text a model can read at once. One reason long contexts are costly is the KV cache: to avoid recomputing, the model stores every layer's keys and values for every token so far. For an example configuration (32 layers, 8 key/value heads of 128 numbers each, 2 bytes per number, the shape of an 8-billion-parameter model of the Llama 3 family) that is 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 128 KiB, per token.
The same rule explains why a 1,000,000-token context, such as one announced for a Gemini model in February 2024, needs special engineering and a lot of memory.
| Place | Models and developers, factual list |
|---|---|
| United States | GPT (OpenAI; GPT-3 was 175 billion parameters, 2020); Llama (Meta; Llama 3.1, July 2024, up to 405 billion, open weights under a licence); Gemini (Google DeepMind); Claude (Anthropic) |
| Europe | Mistral 7B (Mistral AI, France, September 2023, Apache 2.0); BLOOM (BigScience collaboration, 2022); Falcon (Technology Innovation Institute, Abu Dhabi, 2023) |
| China | DeepSeek-R1 (671 billion, MIT licence, January 2025); Qwen (Alibaba Cloud, Apache 2.0 for recent versions); GLM (Z.ai); Kimi (Moonshot AI) |
| India | Sarvam (Sarvam AI; Sarvam-105B, February 2026) |
The list is illustrative, not complete, and licences vary: "open weights" means the trained numbers can be downloaded, which is not always the same as open source.
Diffusion: making pictures by removing noise
Most image generators use a different trick. Take a real picture and add a little random noise, then a little more, until nothing remains but static. That forward direction is easy and needs no learning. Train a network to do the reverse for one step: given a noisy picture and how noisy it is, predict the noise that was added. Then generate by starting from pure static and removing noise step by step. Sohl-Dickstein and colleagues proposed the idea in 2015, Ho, Jain and Abbeel made it work well in 2020 (DDPM), and Rombach and colleagues (2022) ran it in a compressed "latent" space, which made it cheap enough to run on a home graphics card. Text prompts steer each step through the same attention mechanism you met above.
Uses the schedule of the 2020 DDPM paper (noise variance rising linearly from 0.0001 to 0.02 over 1,000 steps) and a fixed random noise pattern. The picture is invented.
Watch the correlation figure: the picture is not erased all at once, it fades. The reverse network is trained at every level of fade, from "nearly clean" to "nearly nothing", which is why generation can start from nothing and still arrive somewhere. Whether training such models on scraped images is lawful, fair to artists, or both is being decided in courts and legislatures in several countries as of this writing, with different outcomes likely in different places.
What a graduate just sees
- A model never sees words, only tokens, and the tokenizer's choices show up in cost, in other languages and in oddities such as trouble with spelling.
- Attention is a table of weights that add to 1, and the mask decides who may look at whom. The cost grows with the square of the context.
- Training pushes down one number, the average surprise at the next token. Everything else is a consequence, or a patch.
- The answer you read is one draw from a distribution. Temperature and top-p are dials on the draw, and a plausible wrong answer usually holds real probability.
- Compute, parameters and data trade against each other, roughly C ≈ 6ND; "bigger" without "more data" wastes budget, and cheaper serving can justify training a small model for a long time.
- Fluent is not the same as true. Hallucination is partly a product of how models are trained and scored.
- Image generators and text generators are close relatives: both learn a step-by-step conditional distribution and sample from it.
Where it shows up next
- Building AI Applications (36/78): context length, tokens and temperature become cost, latency and reliability decisions.
- AI Safety, Alignment & Governance (36/79): preference tuning, the compute thresholds in law, and open versus closed release all follow from what you have just seen.
- Data Ethics, Privacy & Fairness (36/37): models absorb patterns, errors and private text from training data.
- Computational Linguistics (36/30): the long line from n-gram counts to transformers.
- In the world: coding assistants, translation, search summaries, captioning, image and video tools.
If you want to go deeper
- 📘 Textbook. Daniel Jurafsky and James H. Martin, Speech and Language Processing (3rd edition draft, free at web.stanford.edu/~jurafsky/slp3), the standard NLP text; and Simon Prince, Understanding Deep Learning (MIT Press, 2023), which has chapters on transformers and diffusion.
- ▶️ Gentler. Andrej Karpathy's Neural Networks: Zero to Hero video series, which builds a small language model and a tokenizer from scratch; and Grant Sanderson's (3Blue1Brown) chapters on transformers.
- 🌐 Free course. Stanford CS224N: Natural Language Processing with Deep Learning, with public lecture materials; or the Hugging Face LLM Course.
A language model is a table-lookup of probabilities built by attention and trained by one rule: be less surprised by the next token. The text you read is a draw from that table, and its size, quality and cost follow from a few dials: vocabulary, context, compute, data and temperature. Image models run the same logic on noise.
Try this: ask any chatbot the same factual question three times, and once with "answer in one word". Notice what stays the same and what changes. Then guess which tokens carried the uncertainty.
Sources. Vaswani et al., "Attention Is All You Need" (arXiv 1706.03762, 2017). Sennrich, Haddow and Birch (arXiv 1508.07909, 2015). Petrov et al., NeurIPS 2023 (arXiv 2305.15425). Kaplan et al. (arXiv 2001.08361, 2020). Hoffmann et al., Chinchilla (arXiv 2203.15556, 2022). Brown et al., GPT-3 (arXiv 2005.14165). Grattafiori et al., "The Llama 3 Herd of Models" (arXiv 2407.21783). Hugging Face, "Llama 3.1" blog post (July 2024: 128K context, over 15 trillion tokens, about 15.6 GB of KV cache at 128K tokens). Holtzman et al., "The Curious Case of Neural Text Degeneration" (arXiv 1904.09751). Ouyang et al. (arXiv 2203.02155); Rafailov et al. (arXiv 2305.18290); DeepSeek-AI (arXiv 2501.12948). Kalai et al. (arXiv 2509.04664). Schaeffer, Miranda and Koyejo (arXiv 2304.15004). Google, "Gemini 1.5" announcement (15 February 2024). Sohl-Dickstein et al. (arXiv 1503.03585); Ho, Jain and Abbeel (arXiv 2006.11239); Rombach et al. (arXiv 2112.10752). Wikipedia, "List of large language models" (consulted November 2026 for release dates and licences). All numbers in the interactives were computed in code and checked.