Machine Learning: Programs That Write Their Own Rules

Your phone groups your photos by person. Your inbox hides the lottery scams. A bank flags a card payment at 3 a.m. from another continent. Try to write the rules for any of these by hand and you hit a wall within an hour: what, exactly, makes a face a face, or a message spam?

Machine learning gives up on writing the rules. Instead you show a program thousands of labelled examples and let it adjust itself until it gets them right. This is powerful, and it quietly changes the questions you have to ask. The hard part is no longer "what is the rule?" but "how do I know the rule it found will work tomorrow, on people it has never seen?"

That question, and a handful of simple tools for answering it, are what a graduate of a machine-learning course keeps for life.

Learning from data, and the data you hold back

Take a toy problem. Each email gets a "suspicious words" score, and you must decide spam or not. You could write a rule by hand: spam if the score is above 3. But why 3? Better: try every threshold on a set of labelled examples and keep the one that gets the most right. That, in miniature, is training: a rule with an adjustable knob, a way to score the rule on data, and a search for the best setting of the knob.

Here is the trap. The setting that fits the examples you used will almost always look better on those examples than on fresh ones, because you tuned it to them. So practitioners split their data three ways:

• Training set: the examples the model adjusts itself on.
• Validation set: held back to compare choices (which model, which settings).
• Test set: touched once, at the very end, to estimate how it will do in the real world. If you keep tuning against it, it stops being a test.

Linear regression and gradient descent

Many problems are about predicting a number. The simplest model is a straight line: prediction = w × input + b. The two knobs are the slope w and the intercept b. To score a line, measure how far it misses each point, square the misses so that over and under both count, and average them: the mean squared error (MSE). The method of least squares was first published by Adrien-Marie Legendre in 1805; Carl Friedrich Gauss claimed to have used it earlier, which started a priority dispute.

For a line there is a formula for the best w and b. For huge models there is not, and we use the idea that carries the whole field: gradient descent. Stand on the error surface, feel which way is steepest downhill (the gradient, from Multivariable Calculus 34/02), and take a small step that way. Repeat. Augustin-Louis Cauchy suggested the method in 1847.

The learning rate is a real trade-off: too small and you crawl; too large and you overshoot the valley and bounce out of it. Try it. The 12 points below are made up around a line; the least-squares answer is w = 2.465, b = 8.289, with MSE 2.58.

Overfitting and underfitting

A line is a rigid model. A more flexible one, say a polynomial with more bends, can follow the data more closely. Flexibility is a dial, and both ends of it fail:

• Underfitting: the model is too simple to capture the pattern. It is wrong on the training data and wrong on new data.
• Overfitting: the model is so flexible that it memorises the noise in the training data. It is nearly perfect there and bad on new data.

This is why the validation set exists. Training error only ever goes down as you add flexibility. Validation error goes down, then comes back up, and the bottom of that U is where you stop.

Classification and neural networks as layered functions

Classification predicts a label rather than a number: spam or not, which digit, which animal. A simple classifier computes a weighted sum of the inputs and squashes it into a probability between 0 and 1, and its decision boundary is a straight line (or plane). That is enough when the classes can be separated by a line.

Some problems cannot. The classic example is XOR: the answer is 1 when exactly one of the two inputs is 1. No single straight line separates the four cases correctly. In the late 1960s Marvin Minsky and Seymour Papert's book Perceptrons analysed exactly this kind of limit for single-layer networks, and is often credited with helping to cool interest in neural networks for years (historians debate how much).

The escape is to stack models. A neural network is a chain of simple functions: each layer takes a weighted sum of its inputs, applies a bend (a non-linear function such as tanh), and hands the result to the next layer. One layer draws a line; a layer built on top of lines can carve out regions that no single line can. Training a network means finding good weights, and the tool is backpropagation, which in words goes:

1. Run an example forward through the layers and get a prediction.
2. Measure how wrong it was.
3. Walk backwards, layer by layer, asking each weight "how much of this error was your fault?", using the chain rule from calculus.
4. Nudge every weight a little in the direction that reduces the error. Repeat on the next example.

The idea was derived several times; the 1986 Nature paper by David Rumelhart, Geoffrey Hinton and Ronald Williams is the one that made it popular. Gradient descent plus backpropagation, run at scale, is still how neural networks are trained, including the large language models in 29/02.

Evaluation: accuracy is not enough

"The model is 99% accurate" can be meaningless. Imagine a rare disease affecting 1% of 10,000 people. A model that always says "healthy" is 99% accurate and finds nobody. Count outcomes instead: true positives (TP), false positives (FP), true negatives (TN), false negatives (FN).

The two pull against each other. A model that flags only when it is certain has high precision and low recall; one that flags at the slightest hint has high recall and swamps you with false alarms. Where to set the threshold depends on what each error costs: missing a cancer is not like sending a good email to the spam folder. And when the real condition is rare, even a decent test produces mostly false alarms.

Bias: the model learns the data you gave it

A model has no opinion about fairness. It finds whatever pattern best fits the examples it was shown, including patterns in who is in the examples. If one group is a small part of the training data, the model's single rule is tuned to the majority and does worse on the rest.

This is not hypothetical. In the 2018 Gender Shades study, Joy Buolamwini and Timnit Gebru found that the two benchmark face datasets they examined were 79.6% (IJB-A) and 86.2% (Adience) lighter-skinned subjects, and that three commercial gender-classification systems misclassified darker-skinned women at error rates of up to 34.7%, against at most 0.8% for lighter-skinned men.

More data from the same source does not fix this; different data, or a model checked group by group, can. Bias can also come from the labels (past human decisions the model copies) and from how a system is used. Checking performance for each group, not just on average, is the habit to keep.

The same discipline applies to the newest systems. The Stanford HAI AI Index 2026 report describes a "jagged frontier": one model earned a gold medal at the International Mathematical Olympiad, yet the top model it tested read analog clocks correctly only 50.1% of the time. An average score, or a headline result, says little about how a model will do on the case in front of you; that has to be measured.

What a graduate just sees

• Learning means tuning knobs to fit examples; the rule is found, not written.
• Performance on the data you trained on is not evidence. Only data held back is.
• Gradient descent is "step downhill"; the learning rate is the step size and it matters.
• Too simple and too flexible both fail; validation error has a U shape.
• A network is stacked simple functions, and backpropagation is the chain rule applied layer by layer.
• Accuracy hides rare cases; ask for precision, recall and the cost of each mistake.
• When a thing is rare, most alarms are false alarms.
• A model is only as fair as its data; check each group, not the average.

Where it shows up next

• Linear Algebra (34/04) and Multivariable Calculus (34/02): weights are vectors and matrices, and the gradients are partial derivatives.
• Probability & Statistics (34/05): the train/test idea is the statistician's out-of-sample check, and precision is Bayes' theorem in disguise.
• Algorithms (34/24): training is optimisation, and its cost is why hardware like GPUs matters (see 29/12).
• Computer Security (34/33): learning systems can be fooled by crafted inputs and poisoned data.
• In the world: fraud screening, search ranking, recommendations, translation, medical imaging, and the chatbots in 29/02. As of the 2025 Stanford AI Index, the compute used to train notable models was reported to be doubling roughly every five months, which is why training is now an industrial activity.

If you want to go deeper

• 📘 An Introduction to Statistical Learning (James, Witten, Hastie, Tibshirani; 2nd edition 2021, with a Python edition in 2023): free from the authors, the standard first university text; clear and light on maths.
• 📘 Deep Learning (Goodfellow, Bengio, Courville, MIT Press 2016): the standard graduate text on neural networks, free to read online.
• ▶️ Gentler: 3Blue1Brown's neural-networks video series builds the ideas above with pictures.
• 🌐 Free courses: MIT OpenCourseWare 6.036 Introduction to Machine Learning and Stanford CS229.