Econometrics: How to Find What Caused What

A city opens a new tram line, and a year later house prices along it have risen. Did the tram do it? Maybe. Or maybe the city built the line where it already expected the area to boom. Or a new university opened nearby the same year. Every headline that says "X is linked to Y" hides this question, and most of the time the headline cannot answer it.

An economist sees the same pattern in a minimum-wage law, a school-class size, a tax change, a drug trial. The data show a link. The job is to say what the link would have been if nothing else had been going on. That is econometrics: statistics built for questions where you usually cannot run the experiment, and where the people in your data chose their own circumstances.

Regression is the best-fitting line

Take twelve students: hours studied, and the score each got. Plot them and the dots drift upward. A regression draws the one straight line that sits closest to all the dots. "Closest" has a precise meaning: for each dot, measure the vertical gap to the line (the residual), square it, and add up the squares. The line with the smallest total is the ordinary least squares (OLS) line. Squaring means one big miss counts far more than several small ones; the picture shows each squared miss as a literal square.

The best line here is about 47 + 3.5 × hours (the readout gives it exactly). The coefficient b says: students who studied an hour more scored about 3.5 points higher. Notice what that sentence does not say. It compares different students. It does not say that making one student study an extra hour would raise her score by 3.5. That gap, between a comparison and a cause, is the rest of the course.

Omitted variables: add the confounder, watch the effect change

Suppose you regress wages on years of schooling in a big survey and find that each extra year goes with higher pay. Is that what schooling does? People with more drive or talent may both stay in school longer and earn more anyway. If so, the schooling coefficient is also soaking up the talent effect. This is omitted-variable bias, and it has a tidy rule:

In the lab below, the true effect of one more year of schooling is exactly 3 (thousand dollars). Talent also raises pay, and a dial sets how strongly talent pushes people toward more school. Switch the model between ignoring talent and including it.

Without JavaScript: with talent pushing schooling at strength 1.2, ignoring talent gives a schooling coefficient near 4.7 instead of the true 3.0; controlling for talent recovers about 3.0; randomising schooling also recovers about 3.0.

Ignoring talent at strength 1.2 gives about 4.7, over half again too large. Control for talent and it falls to about 3. The cost of this fix: you must measure the confounder. Most real confounders (motivation, family connections, how well-run the firm is) sit in no dataset. That is why economists spent decades arguing about what regressions could prove.

Randomise: the coin that cuts every hidden link

Press the third button. When schooling is assigned by a process unrelated to talent, the link between them is broken, and even the naive regression that ignores talent lands on 3. A coin flip is independent of everything: motivation, family, luck, things nobody thought to measure. That is why a randomised experiment is the gold standard, and why 2019's Nobel prize in economics went to Abhijit Banerjee, Esther Duflo and Michael Kremer for experimental approaches to reducing poverty.

But economists rarely get to flip the coin: you cannot randomly assign years of schooling, a recession or a minimum wage. The next three ideas are ways to find a coin that nature or a rule has already flipped. Together they are called the credibility revolution, a phrase from Joshua Angrist and Jörn-Steffen Pischke's 2010 essay, answering Edward Leamer's 1983 complaint that econometrics needed to "take the con out" of itself.

Difference-in-differences: borrow the other group's trend

In 1992 New Jersey raised its minimum wage and neighbouring Pennsylvania did not. David Card and Alan Krueger surveyed fast-food restaurants in both states before and after (American Economic Review, 1994). Looking only at New Jersey before and after would confuse the law with anything else changing that year. Instead they used Pennsylvania as the stand-in for "what would have happened anyway". Their finding, that employment did not fall in New Jersey, contradicted the textbook prediction at the time. It has been disputed ever since: Neumark and Wascher re-analysed it with payroll records and found job losses, and the field's overall reading is still argued over. The design is not in dispute. Card shared the 2021 Nobel prize in part for it.

With a common trend of +1.5 and no true effect, the treated group still rises by 1.5, so a naive "before and after" would claim a policy success. The difference-in-differences correctly says zero. The design's weak point is the third slider: it assumes the two groups would have moved in parallel without the policy. If the treated group was already on a different path, the estimate is off by exactly that gap. Good papers show the two lines tracking each other for years before the policy.

Instruments and cutoffs: two more ways to find a coin

An instrumental variable is something that nudges the thing you care about but has no other route to the outcome. Angrist (1990) studied whether military service in the Vietnam era changed later earnings. Men who chose to serve differ from those who did not, so a plain comparison is confounded. But the draft lottery assigned numbers by birthday at random, and a low number made service much more likely. The lottery number is a coin that pushes people into uniform. Its effect on earnings, divided by its effect on service, isolates the effect of service for the people who served because of the lottery. His study found that service lowered earnings for the veterans concerned.

A worked number, invented to show the arithmetic: suppose 35% of low-number men served and 20% of high-number men, a first stage of 0.15; and low-number men earned $300 less a year on average. Then the estimated effect of serving is −300 ÷ 0.15 = −$2,000 a year. Dividing by a small first stage blows up noise, which is why weak instruments are dangerous.

A regression discontinuity uses a cut-off. Israeli school classes are capped at 40 pupils under a rule attributed to the twelfth-century scholar Maimonides, so a school with 40 pupils has one class and a school with 41 splits into two small classes of about 20. Angrist and Lavy (1999) compared schools just either side of those thresholds, where enrolment is close to a coin flip, to estimate what class size does to test scores. Slide the window below: narrow windows are fairer but noisier.

Standard errors: how much would the answer wobble?

Every estimate comes from one sample, and a different sample would have given a different number. The standard error is the typical size of that wobble. For a regression slope it shrinks with the square root of the sample size, grows with the noise around the line, and shrinks when x varies a lot:

The rule of thumb: an estimate more than about two standard errors from zero is called statistically significant at the 5% level. Significant means "hard to produce by sampling wobble alone if the true effect were zero". It does not mean big or important, and it does not mean true. Run 300 studies of a real but modest effect below and see who gets stars.

At n = 50 and a true slope of 0.3, a little over half of the studies reach significance, and those average about 0.40, a third above the truth, because only the lucky overestimates clear the bar. Drop to n = 10 and a true slope of 0.05 and the few stars (about 8%) average roughly 0.7, fourteen times too big. Published results are mostly the significant ones, so they exaggerate. That is one reason newer replications so often find smaller effects.

P-hacking: test enough things and something will glow

A test at the 5% level is wrong by chance 1 time in 20 when nothing is going on. Test 20 unrelated outcomes and the chance that at least one glows is 1 − 0.95²⁰ = 64%. A researcher who tries several outcomes, subgroups or control sets and reports the one that "worked" is p-hacking, whether on purpose or just by wandering a garden of forking paths. Simmons, Nelson and Simonsohn (2011) showed how easy it is: using flexible, ordinary-looking choices they "found" that listening to a song about being older made students younger by about a year and a half. Economists are not exempt: Brodeur and co-authors (2016) found an unnatural pile-up of results just below the 0.05 line in top economics journals.

Defences: decide the analysis before seeing the data (pre-registration), report every test you ran, adjust the threshold when you run many, and above all prefer designs, like the ones above, where the answer does not depend on which controls you picked.

What a graduate just sees

• A regression coefficient is a comparison between different people, not a lever you can pull.
• Before believing "X raises Y", ask what else moves with X and also moves Y. The bias is its effect times its link to X.
• Adding controls can only fix confounders you measured. A coin flip fixes the ones you did not.
• Every natural-experiment study is a hunt for a coin: a lottery, a border, a cut-off, a law that changed for some and not others.
• Difference-in-differences needs parallel trends; ask to see the years before the policy.
• "Significant" means hard to get from wobble alone. It says nothing about size, and a small-sample star probably overstates.
• If you can run twenty tests, one will light up. Ask how many were run.

Where it shows up next

• Development Economics (34/48): randomised trials and natural experiments are how economists test what actually helps poor countries.
• Behavioral Economics (34/46): the replication record in that piece depends on this one, from effect sizes to p-hacking.
• Marketing Research & Analytics (34/59): "did the campaign cause the sales, or did summer?" is difference-in-differences.
• Machine Learning (34/32): prediction asks "what will happen"; econometrics asks "what would happen if I changed it". They need each other.
• In the world: central banks and tax agencies, tech companies' A/B test teams, public-health trials, and every think-tank report that says "causes".

If you want to go deeper

• 📘 Textbook. James Stock and Mark Watson, Introduction to Econometrics (Pearson). A standard undergraduate text, strong on causal thinking and applications.
• 📘 Textbook. Jeffrey Wooldridge, Introductory Econometrics: A Modern Approach (Cengage). The other standard choice, more algebra and more worked data.
• 📘 Gentler. Joshua Angrist and Jörn-Steffen Pischke, Mastering 'Metrics: The Path from Cause to Effect (Princeton, 2015). Written for people who are not specialists, built around exactly the designs above.
• 🌐 Free course. MIT OpenCourseWare 14.32 Econometrics (Spring 2007, Joshua Angrist): undergraduate course with lecture notes and problem sets.