Research Methods in Political Science: How Anyone Knows Anything About Politics

In 1998, a pair of political scientists, Alan Gerber and Donald Green, did something unusual. They took about 30,000 registered voters in New Haven, Connecticut, and chose by lottery who would get a knock on the door, a phone call or a mailer urging them to vote. After the election they checked who had actually voted. People who were reached by canvassers turned out at a rate about 8.7 percentage points higher than comparable people who were not; mailers added about half a point each; phone calls did nothing detectable.

The striking part is not the result but the method. Before that, most claims about what gets people to vote came from comparing people who were contacted with people who were not, and those two groups differ in countless ways. The lottery removed the problem. Almost every question in political science has this shape: we see two things go together, and want to know whether one causes the other. The methods course is a toolkit for that question.

The fundamental problem of causal inference

Suppose a town starts a youth-training programme and a year later its participants earn more. Did the programme do it? To know, you would need to see the same people in the same year without the programme. That world never exists. Every person is observed under one condition only. Statisticians call this the fundamental problem of causal inference: the effect is the gap between what happened and what would have happened, and the second is never seen.

The natural workaround is to compare people who got the treatment with people who did not. That fails when the groups differ in a hidden way that also affects the outcome, a confounder. If more motivated people sign up for the programme, and motivated people earn more anyway, the comparison credits the programme with the motivation. Try it below. The hidden trait raises the outcome; you choose how strongly it also drives who gets treated.

Without JavaScript: with the defaults, self-selection makes the treatment look like it adds about 3.9 when the truth is 2; assigning by coin flip brings the estimate back near 2.

With self-selection and a strong hidden trait, the estimate runs well above the truth. Switch to the coin flip and the answer lands near 2 whatever you set, because chance cannot be correlated with anything. That is the whole logic of the experiment, and everything else in this piece is a way to approximate it when you cannot flip the coin.

Experiments and natural experiments

A randomised experiment assigns the treatment by lottery, so on average the groups are alike in every way, measured or not. Gerber and Green's 2000 study in the American Political Science Review was an early large field experiment in political science; since then, researchers have randomised leaflets, observers at polling stations, community-development funds and information campaigns, in places from Benin to Brazil to Uganda. Their limits are well known: you cannot randomly assign a country's constitution, many treatments are unethical or impossible, and an effect found in New Haven in 1998 may not hold in another place or year.

When you cannot randomise, you can sometimes find a place where the world did it for you: a natural experiment. Daniel Posner's study of two African peoples, the Chewa and the Tumbuka, is a classic. A colonial border split each group between Malawi and Zambia. Their cultures are essentially the same on both sides, yet in Malawi they are political rivals and in Zambia allies. The difference, Posner argued, is size: in Malawi the Chewa are roughly 28 percent of the population and the Tumbuka 12 percent, big enough to form rival blocs; in Zambia each is under 7 percent, too small, so politicians appeal to them together as easterners. The border, drawn without regard to politics, did the assigning. A natural experiment is only as good as the claim that the assignment was "as if random", and that claim has to be argued case by case.

Difference-in-differences

Another approach: find a group hit by a policy and a group that was not, and compare how each changed. The comparison group's change stands in for what would have happened to the treated group anyway. The classic example is David Card and Alan Krueger (1994). In April 1992 New Jersey raised its minimum wage from $4.25 to $5.05 an hour, while neighbouring Pennsylvania did not. Surveying fast-food restaurants in both states before and after, they found employment in New Jersey did not fall relative to Pennsylvania; it rose slightly, by an estimated 2.75 full-time-equivalent jobs per restaurant. The finding was contested, and economists still argue about it, but the design became a standard tool. Card shared the 2021 economics Nobel prize partly for this work.

Slide the extra trend away from zero and watch the estimate become wrong by exactly that amount. The groups start at different levels, which does not matter; what matters is whether they were already moving apart. Researchers check this by plotting several periods before the policy. If the lines run parallel before, parallel afterwards is believable; it is never proved.

Regression discontinuity

Some rules create sharp lines: a scholarship for scores above 80, a programme for towns above 5,000 people, a seat for the candidate with more votes. Compare units just above and just below the line, and they are almost identical, so the line works like a lottery for those near it. The design was introduced by Donald Thistlethwaite and Donald Campbell in 1960, for scholarships and students' career plans. In political science, David Lee (2008) used it for the US House: in districts where the Democrat won or lost by a hair, who won was close to a coin flip, so comparing these districts shows what it is worth to win once.

You must decide how wide a window around the line to use. A narrow window gives clean comparisons but few points, so the estimate is noisy. A wide window gives many points but includes ones far from the cutoff, where the outcome curves for other reasons. Try both extremes.

Experts use refined versions (weighted fits and data-driven windows), but the trade-off is the same. Two cautions: the estimate speaks only for units near the line, and it fails if people can sort themselves across it, as when officials manipulate a vote count or a test score.

Case studies and process tracing

Not every question has a hundred cases. Why did a particular revolution succeed? Why did one country avoid civil war? Here researchers study a few cases in depth. Process tracing is the discipline of this: you lay out the chain of events that your theory predicts, then look for evidence at each link. Stephen Van Evera (1997) sorted evidence into four kinds, depending on how a clue behaves if the theory is true or false. David Collier popularised process tracing in political science, defining it as the systematic examination of diagnostic evidence.

• Straw in the wind: nudges belief a little, neither strong support nor strong refutation.
• Hoop: the theory must pass it. Failing counts heavily against; passing proves little.
• Smoking gun: strong support if found. Not finding it proves little.
• Doubly decisive: confirms the theory and eliminates rivals if passed; badly hurts it if failed. Rare.

The main dispute here is old and still live. In 1994 Gary King, Robert Keohane and Sidney Verba, in Designing Social Inquiry, argued that qualitative research should follow the logic of quantitative inference. Henry Brady and David Collier's reply in Rethinking Social Inquiry (2004) argued that case-based work has its own strengths, and process tracing is part of that answer. Many political scientists now combine the two, using a regression to find a pattern and cases to check the mechanism.

Measuring a concept like democracy

Before you can test whether democracy causes growth, you must decide what "democracy" is and how to count it. Dahl-style definitions stress competitive elections and broad participation; others add the rule of law, a free press and civil liberties. The big datasets make different choices. Polity, begun by Ted Robert Gurr in the late 1960s, scores states from −10 to +10, with 6 to 10 counted as democracy. The Economist Intelligence Unit's index, first published in 2006, adds up 60 indicators in five categories into a 0 to 10 score and sorts countries into four types (full democracy, flawed democracy, hybrid regime, authoritarian regime). V-Dem, launched in 2014, publishes more than 470 indicators across seven principles of democracy, back to 1789. Freedom House publishes its own yearly ratings. They agree about the extremes and disagree in the middle, and the disagreements are not mistakes; they reflect different definitions. Gerardo Munck and Jay Verkuilen (2002) criticised several indices for being too narrow or hiding choices about weights. Some scholars prefer a simple yes/no rule, such as one that counts a country as a democracy only if power has actually changed hands through elections.

Without JavaScript: a country with competitive elections and universal suffrage but no free press, courts or turnover of power counts as a democracy under a minimal rule and not under the others.

Try Country C: elections are competitive and power has changed hands, but not everyone may vote. The rules disagree. Any finding about "democracy and X" depends on which was used, which is why careful papers show their results under more than one measure.

The replication debate

Social science has had a reckoning over whether findings hold up. In 2015 the Open Science Collaboration re-ran 100 psychology studies and reported that about 36 percent of the replications produced a statistically significant result, with effects about half the original size. Political science has had its own cases. In December 2014 Science published a study by Michael LaCour and Donald Green claiming that a short conversation with a gay canvasser durably changed views on same-sex marriage. In May 2015 David Broockman, Joshua Kalla and Peter Aronow showed that the data were irregular and appeared to be fabricated, and Science retracted the paper on 28 May 2015. The case was not about a weak method but about a fraud, and the lesson drawn was that data and code should be shared so that others can check.

The quieter problem is flexibility. A researcher with many outcomes, subgroups and ways to code a variable can almost always find something with p below 0.05. Studies of published results in several social sciences, political science among them, have reported more findings just under the 0.05 line than chance would give. The remedies are pre-registering the analysis plan before seeing the data, journals that accept papers for their questions rather than results ("registered reports"), and sharing data and code. Some qualitative scholars have argued that transparency rules built for statistics fit interviews and archives badly, and the field is still working that out.

Without JavaScript: with 14 outcomes that are all truly zero, the chance at least one looks significant is about 51 percent.

Fourteen tests of nothing give about even odds of a "finding". The tighter bar, dividing 0.05 by the number of tests (the Bonferroni correction), keeps false alarms near 5 percent in total, at the cost of missing smaller real effects. None of this makes a single study worthless; it means one significant result among many tries deserves less trust than the same result from a single planned test.

What a graduate just sees

• A correlation is an invitation to ask what else could produce it. Ask who selected into the group.
• Every causal claim hides a counterfactual: what would have happened otherwise?
• Randomisation works because chance is correlated with nothing. Everything else is an attempt to imitate it.
• A natural experiment is only as good as the argument that the assignment was "as if random".
• Difference-in-differences needs parallel trends; check the years before the policy.
• Near a cutoff, units are alike; far from it, they are not. Narrow windows trade bias for noise.
• A few cases can test a theory if you ask what each clue would look like if the theory were false.
• "Democracy" is a definition before it is a number. Ask which measure was used.
• Many tries inflate false findings. Plans written before the data, and open data, deflate them.

Where it shows up next

• Elections, Parties & Public Opinion (35/15): field experiments on turnout and RDD on close elections are standard tools there.
• Public Policy (35/16): evaluating a programme is a causal-inference problem; see the counterfactual in the Progresa example.
• Comparative Politics (35/13): cross-country comparisons depend on how concepts like democracy are measured.
• Econometrics (34/44): the same methods with the economist's emphasis.
• In the world: reading a news report on a "study finds", pollsters and campaign teams running A/B tests, and think tanks and ministries asking for evidence before spending.

If you want to go deeper

• 📘 Textbook. Kevin Kellstedt and Guy Whitten, The Fundamentals of Political Science Research (Cambridge University Press): a standard first methods text for the discipline.
• 📘 Textbook. Alan Gerber and Donald Green, Field Experiments: Design, Analysis, and Interpretation (Norton, 2012), and Thad Dunning, Natural Experiments in the Social Sciences (Cambridge University Press, 2012).
• 📘 Gentler. Joshua Angrist and Jörn-Steffen Pischke, Mastering 'Metrics (Princeton, 2015): the logic of these methods with almost no algebra.
• 🌐 Free course. MIT OpenCourseWare 17.871, Political Science Laboratory (Charles Stewart, Spring 2012): learn quantitative political research by doing projects.