Computational Linguistics: From Grammars to Language Models

Ask a phone to translate "the trophy would not fit in the suitcase because it was too big" into French or Spanish, where the pronoun must be masculine or feminine to match one of the two objects, and the system must decide in a fraction of a second what a human does without noticing. Change one word at the end ("too small") and the answer flips. Nothing in the spelling tells you; the knowledge is about trophies and suitcases.

Computational linguistics sits where that problem lives. It asks how to represent words, structure and meaning in forms a program can use, and how to tell whether the program got it right. Its practical side, usually called natural language processing, produced search, spelling correction, translation, speech recognition and, lately, chatbots. Its scientific side asks what these systems reveal about language itself. A graduate keeps both questions in view.

Tokenisation: what counts as a word?

Before any model can read, text must be cut into units. English spelling makes it look easy: split at spaces. But "don't" is two words in one, "New York" is one in two, and many languages never put spaces in the places English does. Mandarin writes no spaces between words. Turkish and Swahili pack a whole English sentence's worth of structure into one written word. German glues nouns together.

Language	Written word, split into morphemes	Gloss	Free translation
Turkish	ev-ler-iniz-den	house-PL-2PL.POSS-ABL	from your (plural) houses
Swahili	ni-na-ku-pend-a	1SG-PRS-2SG.OBJ-love-FV	I love you
Mandarin	wǒ xǐhuan kāfēi (我喜欢咖啡)	1SG like coffee	I like coffee

Leipzig-style glosses: PL plural, POSS possessive, ABL ablative (“from”), SG singular, PRS present, OBJ object, FV final vowel.

Modern systems avoid choosing words by hand. Byte-pair encoding (BPE), borrowed from data compression (Philip Gage, 1994) and brought to translation by Sennrich, Haddow and Birch (2016), starts from single characters and repeatedly merges the most frequent adjacent pair into a new unit. Common words become single tokens; rare words are spelled out from pieces. Tokenisers for large models typically have tens of thousands to a few hundred thousand pieces (GPT-2 had 50,257). The try-it below runs the real algorithm on a tiny invented corpus.

Without JavaScript: the corpus starts as 191 character tokens from 12 symbols. After 10 merges the vocabulary is 22 and the corpus 59 tokens, with walk, talk and the endings -ing, -ed having emerged as units; after 18 merges every word is a single token (30 tokens).

Notice the trade: a bigger vocabulary shortens every text but leaves rare pieces under-trained. And tokenisers learn mostly from the languages in their training text, so the same sentence can take several times as many tokens in some languages as in English (Petrov et al., 2023; Ahia et al., 2023), which affects cost and quality for those speakers.

N-grams and Zipf: counting your way to a language model

The oldest statistical language model just counts. An n-gram model predicts the next word from the previous n − 1 words by looking at how often each continuation followed them in a corpus. Claude Shannon showed in 1948 how sequences generated this way start to look like English. The model is crude, but it is a real language model, and its mathematics (probabilities of sequences, perplexity, smoothing) carried into neural models.

There is a catch hiding in any word count, first described by the linguist George Kingsley Zipf (Human Behavior and the Principle of Least Effort, 1949). A few words are extremely common and most are rare, and the pattern is regular: the second-commonest word is about half as frequent as the commonest, the third about a third, and so on. Plot rank against frequency on log axes and you get a roughly straight line of slope near −1. The consequence for language modelling is harsh: however much text you collect, many word sequences will have never appeared, and the model must still give them a probability. The demo below runs on the first 1,235 words of Lewis Carroll's Alice's Adventures in Wonderland (1865, public domain, from Project Gutenberg).

Without JavaScript: in the passage “the” is 4.5% of all words, then “to”, “she”, “was” and “and”; the fitted log-log slope is about −0.8; 246 of the 420 different words occur exactly once. After “the”, the commonest next word is “rabbit”.

In 1957 Noam Chomsky used “Colorless green ideas sleep furiously” to argue that grammaticality cannot be reduced to probability, since the sentence is grammatical yet has presumably never occurred. Fernando Pereira (2000) answered that a statistical model with word classes does rate it far likelier than the same words reversed, so probability and grammar are not as opposed as the argument assumed. Both sides learned something: the models needed classes and smoothing, and the grammarians needed to say what a probability is a probability of.

Parsing: finding the structure, and its ambiguity

A parser assigns a structure to a sentence. In constituency parsing the structure is a tree of phrases; in dependency parsing it is a set of word-to-word links. The big shift of the 1990s was to learn grammars from annotated text: the Penn Treebank (Marcus, Santorini and Marcinkiewicz, 1993) gave thousands of hand-parsed sentences, and today the Universal Dependencies project (release 2.18, May 2026) offers over 200 treebanks in over 150 languages with one shared annotation scheme, which is how a parser trained on one language can be compared with another.

The core problem is ambiguity. The grammar of English lets one string have several trees, and a parser must choose. A probabilistic context-free grammar (PCFG) attaches a probability to each rule and scores a tree as the product of its rules. The demo uses a tiny hand-made grammar and a real chart parser that finds every tree.

Without JavaScript: “she saw the man with the telescope” has two parses. With a 10% preference for verb attachment, the telescope belongs to the man; at 30% the telescope belongs to the seeing. The break-even is 15%. “Old men and women left” has two parses of equal probability: the grammar cannot choose, and a human would need context.

The second sentence shows the limit of structure alone. Choosing between “old (men and women)” and “(old men) and women” takes knowledge about the world, which is why syntax-only parsers hit a ceiling. See Syntax (36/23) for the linguistic side of ambiguity.

Word vectors: meaning from company

J. R. Firth wrote in 1957, “You shall know a word by the company it keeps.” Zellig Harris's distributional hypothesis of the 1950s is the same idea: words that occur in similar contexts have similar meanings. To use it, represent each word by a list of numbers describing which words appear near it, then compare lists. Words whose lists point the same way, measured by the cosine of the angle between them, are similar. Mikolov and colleagues' word2vec (2013) and Pennington, Socher and Manning's GloVe (2014) made such embeddings dense and learnable, with a few hundred numbers per word instead of one per vocabulary item.

Without JavaScript: in the toy corpus the nearest word to “cat” is “mouse” (0.36), then “dog” (0.27); to “bus”, “car” (0.69); to “bread”, “baker” (0.67).

Look at the results. “Mouse” and “dog” come out near “cat”, but “baker” comes near “bread”: closeness records association, which mixes similarity of meaning with being the same sort of thing and being found together. Large embeddings capture more, and the famous “king − man + woman ≈ queen” example works for some words. Later analyses (Rogers, Drozd and Li, 2017; Nissim, van Noord and van der Goot, 2020) found that such analogies are less general than the examples suggest, partly because the input words are excluded from the answer, and that embeddings also pick up the biases of their text.

Attention: how a transformer looks things up

Embeddings give every word one fixed vector, but “bank” in “river bank” and “bank loan” should differ. The 2017 paper Attention Is All You Need (Vaswani and colleagues) built the transformer around a mechanism that rebuilds each word's vector from the words around it. For each position the model forms a query (“what am I looking for?”), and every other position offers a key (“what do I contain?”). The match scores go through a softmax, which turns them into weights adding to 1, and the position's new vector is a blend of the other words' values, weighted by those scores. Stack many such layers and heads, train on text to predict the next word, and you have the architecture behind current large models; for the full picture see How Large Language Models Actually Work (29/02). Here only the lookup matters.

Without JavaScript: at temperature 1 with “too big”, “it” attends about 56% to the trophy and 8% to the suitcase; with “too small” the numbers swap. At a temperature of 5 the weights spread out (trophy 14%, others 7–9%).

This is why long-range dependencies stopped being hard: the first word can attend directly to the last. What the heads actually compute in trained models is studied (probing work such as Hewitt and Manning, 2019, finds tree-like syntactic structure in internal representations), but a single attention weight is not a full explanation of a model's behaviour.

Evaluation: scoring language is hard

A system's output needs a grade, and for language there is rarely one right answer. For classification tasks (is this name a person or a place?) the standard measures are precision (of what it flagged, how much was right), recall (of what was there, how much it found), and their harmonic mean, F1. For generation the measures are rougher. BLEU (Papineni, Roukos, Ward and Zhu, 2002) compares a machine translation with human reference translations by counting matching word sequences, with a penalty for too-short output. It made cheap, fast comparison of translation systems possible and it correlates with human judgement over a whole test set. For a single sentence it can be badly wrong.

Without JavaScript: an identical candidate scores 1.00; a light paraphrase 0.58; reordering the same words 0.91; a paraphrase in entirely different words scores 0 (and so does a string of “the”s); a too-short answer 0.19. At a 0.50 threshold the name-finder has precision 0.64, recall 0.70, F1 0.67.

The pattern repeats across the field. Perplexity (how surprised a model is by held-out text) cannot be compared across models with different tokenisers. Human judges disagree, so annotation studies report agreement beyond chance (Cohen's kappa). Benchmarks saturate and leak into training data, so a high score may reflect memorisation. Newer learned metrics (BERTScore, COMET) track meaning better and have their own biases. The skilled habit is to ask what a number would look like if the system were bad in a way the number ignores.

What language models do and do not show about language

Large language models are neural networks trained on trillions of words to predict the next one. They write fluent text in dozens of languages, produce grammatical sentences with long-distance agreement, and answer questions. What does this show about human language? Careful people disagree, and here are the main positions in their own terms.

The models refute, or at least strain, the generative view. Steven Piantadosi (2023) argued that these models learn rich syntax and semantics from input alone, without the innate grammar that Chomsky's tradition proposes, and that a theory of language should be judged partly by how well it supports such models. Usage-based researchers such as Michael Tomasello have long argued that structure emerges from experience, and see the models as consistent with this.

The models show little about human language. Chomsky, Ian Roberts and Jeffrey Watumull (New York Times, March 2023) wrote that the models work equally well on possible and impossible languages and so explain nothing about what makes human language what it is. A reply is the study by Kallini and colleagues (2024), which found that models learn artificially "impossible" languages less easily than natural ones,. The sheer amount of data is a second point: the BabyLM challenge limits training to 10 or 100 million words to ask what can be learned at a child's scale, noting that models of the Chinchilla class saw 1.4 trillion words, over ten thousand times what a 13-year-old has heard. If a child learns from far less, the models' success does not settle the poverty-of-the-stimulus argument, whichever side you favour.

Form is not meaning. Emily Bender and Alexander Koller (2020) argued that a system trained only on form, with no connection to the world, has no basis for learning meaning in the sense of communicative intent. Their picture of an octopus listening in on a cable became the standard thought experiment. Bender and colleagues (2021) added concerns about scale, environmental and social cost, and calling fluent output understanding (“stochastic parrots”). Defenders such as Christopher Manning have argued that much of meaning is use, so a system trained on use can acquire a good deal of it.

Language is not thought. Mahowald, Ivanova, Blank, Kanwisher, Tenenbaum and Fedorenko (2024) distinguish formal competence (rules and patterns of a language), at which the models are strong, from functional competence (reasoning, world knowledge, social understanding), at which they are uneven. They cite evidence from human brains that these two capacities rely on different systems. The point is not that models are failures, but that a fluent sentence is weak evidence of the underlying thought.

Where there is agreement: neural networks handle many syntactic patterns once thought to need explicit rules (Linzen, Dupoux and Goldberg, 2016, found this early for LSTM networks), they fail in ways that vary by language, and the data is badly uneven. Of roughly 7,100 languages, a small number get nearly all the attention; Joshi and colleagues (2020) found that most are barely represented in language technology. That is both a scientific gap and a fairness one. Whatever one thinks of the theory, anyone building language technology must choose whom it serves.

What a graduate just sees

• A "word" is a convenient fiction; tokenisers cut text into pieces by frequency, and some languages are cut more kindly than others.
• Counting already gives a language model, and the long tail of rare words and unseen sequences is what makes it hard.
• Zipf's law: a few words do most of the work, and most words are rare.
• Sentences are ambiguous; a parser needs both grammar and knowledge of the world.
• Embeddings measure association by shared company, which is not the same as meaning.
• Attention is a weighted lookup: every word rebuilds itself from the words that match it best.
• Every metric is a proxy; ask what it ignores.
• Fluent output is not proof of understanding, and failing a test is not proof of its absence; the debate is real.

Where it shows up next

• Psycholinguistics & the Brain (36/28) and Language Acquisition (36/27): models of reading and learning, and the data gap between children and machines.
• Semantics & Pragmatics (36/24) and Philosophy of Language (36/07): what meaning is, which the models test and sharpen.
• Machine Learning (34/32): training, overfitting and evaluation, all of which apply directly.
• In the world: search, spam filters, automatic captions, translation apps, dictation, writing assistants, and the machine-translated page you read in a language you don't speak.

If you want to go deeper

• 📘 Textbooks. Daniel Jurafsky and James H. Martin, Speech and Language Processing (3rd edition, a free online draft that is updated regularly), and Jacob Eisenstein, Natural Language Processing (MIT Press, 2019). Bender's Linguistic Fundamentals for Natural Language Processing (Morgan & Claypool, 2013) is the linguist's companion.
• 📘 Gentler. Melanie Mitchell, Artificial Intelligence: A Guide for Thinking Humans (2019), a clear and sceptical account of what these systems do.
• 🌐 Free text. Jurafsky and Martin, Speech and Language Processing, third-edition draft online (the draft dated August 2026 begins with large language models, then covers tokens, n-grams and embeddings).
• 🌐 Free courses. Stanford CS224N, Natural Language Processing with Deep Learning (slides free; the 2024 lecture videos are on YouTube), and MIT OpenCourseWare 6.864 Advanced Natural Language Processing (Fall 2005, Collins and Barzilay), which covers the classical statistical methods.