Research Methods in Psychology: How to Not Fool Yourself

A headline says that people who stand like a superhero for two minutes feel braver and take more risks. It was published in a good journal in 2010, taught in business schools, and watched by tens of millions in a talk. Then bigger studies tried it and the hormone and risk results did not appear. The problem was not a villain. It was a method: small groups, many ways to analyse the numbers, and a system that printed only the exciting result.

Every psychology graduate has this story somewhere in their head, and it changes how they read every claim about the mind. They do not ask "is it true?" first. They ask: how was it measured, who was in it, was anyone assigned at random, how big was the effect, and would it happen again? That habit is what a methods course sells.

Measuring what you cannot see: operationalising, reliability and validity

You cannot put "stress", "intelligence" or "attachment" on a scale. So the first move of any study is to operationalise: to pick something you can count that stands for the idea. Stress becomes cortisol in saliva, or a score on a questionnaire. Intelligence becomes the score on a set of puzzles. Every such choice is a bet that the number captures the thing, and the bet can be wrong.

Two words judge the bet. Reliability is whether you get the same number again: the same person, the same scale, a week later. Validity is whether the number measures what you meant. A bathroom scale that always reads 3 kg too heavy is perfectly reliable and not valid for weight. Reliability is necessary but not enough, and without it nothing else works, because noise in a measure shrinks every relationship you try to find with it.

With reliability 0.7 on both measures, a real link of 0.6 shows up as about 0.42. A study of a shaky questionnaire can fail to find a true effect simply because its ruler is made of rubber.

Experiments and correlations: only random assignment breaks the confounder

People who exercise report better wellbeing. Does exercise cause it? Maybe. Or people with more free time, better health and less money stress both exercise more and feel better: a confounder. A correlational study records what people already do, and cannot tell these stories apart however large it is.

An experiment does one thing differently: the researcher decides who gets the treatment, by chance. A coin flip cannot be related to free time, health or anything else, so on average the groups differ in nothing except the treatment. That is why randomised experiments sit at the top of the evidence ladder, and why a study that merely compares people who chose different things should say "associated with", not "causes".

With a true effect of zero and a confounder of 0.6, the observed link is about 0.35, and a naive reading says exercise helps. Flip the switch to coin-flip assignment and it falls to roughly zero, which is the truth. Experiments are not always possible (you cannot assign people to grief or to poverty), so psychologists also use careful natural comparisons, but they say so when they have not randomised.

Effect size and power: how big, and could this study even see it?

"Statistically significant" says only that a result would be unlikely if nothing were going on. It says nothing about size. A huge study can make a trivial difference significant; a small one can miss a large one. So papers now report an effect size. The common one for two groups is Cohen's d: the gap between the group averages, measured in standard deviations.

At d = 0.5 the curves overlap by about 80%, and if you pick one person from each group at random the one from the higher group is ahead only about 64% of the time. Cohen suggested 0.2, 0.5 and 0.8 as small, medium and large, and he meant them as a last resort when you know nothing better about your field. A tiny effect can matter when it is applied to millions of people; a medium one can be useless for one person.

Power is the chance a study detects an effect that is really there. It depends on the true effect, the sample size and the cut-off. Try it. Below, thousands of invented studies are run on the same true effect; significant ones are shaded.

With d = 0.5 and 20 people per group, power is only about a third. With 64 per group it is about 80%, the usual target. Now notice the second number. When the true effect is small and the study is small, the few studies that reach significance are the ones that overshot: at d = 0.2 and 20 per group, power is about 10% and the "winners" average an effect around four times the truth. Journals that print only the winners print inflated effects. This is one reason replications so often come back smaller.

P-hacking: the forking paths

A p-value below .05 is meant to happen by luck one time in twenty when nothing is going on. That promise holds only if you decide what to test before you look. A researcher with ten outcome measures, four ways to exclude "bad" participants, and the freedom to stop when the result looks good has dozens of tests to choose from, and reports the one that worked. Simmons, Nelson and Simonsohn showed in 2011 that such ordinary flexibility could make almost anything "significant". Andrew Gelman called it the garden of forking paths: nobody has to cheat, because each fork seems reasonable at the time.

With ten independent outcomes the chance that at least one is "significant" is 1 − 0.95¹⁰, about 40%. With twenty it is about 64%. Peeking at the data after every ten participants and stopping the moment p dips under .05 gives a false-alarm rate of about one in five (18% in the simulation) across ten looks. Real outcomes are correlated, so the real numbers are a little lower, but the lesson stands. The cure is not more careful people. It is committing to the plan first.

The replication crisis, and the reforms that followed

Around 2011 several things collided. Daryl Bem published nine experiments in a leading journal claiming evidence of precognition, using ordinary methods, which suggested ordinary methods could prove the impossible. The Dutch social psychologist Diederik Stapel was found to have fabricated data. And the false-positive paper appeared. Then people began to re-run studies. In 2015 the Open Science Collaboration, led by Brian Nosek, repeated 100 published experiments: 97 of the originals had significant results, while only about 36% of the replications did, and replication effects averaged about half the original size. Cognitive studies fared better than social ones. That does not say 64% of psychology is false, since a failed replication can itself be underpowered, and studies differ in how hard they are to repeat. It does say published effects are routinely inflated.

The Many Labs projects asked a different question: do the same results appear across many sites? In the first, in 2014, 10 of 13 effects replicated consistently across 36 samples, one weakly, and two, flag priming and currency priming, did not. Test your own sense of what held up.

The reforms are practical. Preregistration means writing down hypotheses and the analysis plan before data arrive, which removes the forks. Registered reports go further: a journal reviews the question and method first and promises to publish whatever comes out, which removes the pull to print only winners; the Center for Open Science counts over 300 journals offering the format. Open data and materials let others check and reuse your work. Larger samples, shared multi-lab studies and reporting every effect size with its uncertainty are now common advice. Not everyone agrees on how far to go: some argue that strict rules suit confirmatory tests and fit badly with exploratory or qualitative work, and that fixing the incentives matters more than any checklist. Few dispute that the field is better checked than a decade ago.

Who is in the study? WEIRD samples

Most experiments recruit whoever is nearest: university students. In 2010 Joseph Henrich, Steven Heine and Ara Norenzayan reviewed the evidence in The Weirdest People in the World? and pointed out that the participants were overwhelmingly from societies that are Western, Educated, Industrialised, Rich and Democratic, a small slice of humanity, and that on several measures such people were outliers. The classic example is a visual illusion (the Müller-Lyer, in Sensation & Perception, 35/21), which fools people in industrial societies more than it fools people in some small-scale societies. Results about fairness games, moral reasoning and self-concept differ across cultures too.

The remedy is plain: say who was studied, write the claim to match, recruit more widely, and be wary of "humans do X" from a campus sample. Some effects are universal, some are local, and without other samples you cannot tell which.

What a graduate just sees

• A study is only as good as its ruler: a noisy or invalid measure hides real effects or measures something else.
• "Linked to" is not "caused by". Only random assignment removes a hidden third factor.
• Significant is not important. Ask how big the effect is, and how uncertain.
• Small studies are noisy, and the ones that reach print overshoot. Treat a lone small result as a hint.
• Flexibility in analysis quietly raises false alarms. The question is what was decided before the data.
• One result is a starting point; a result that repeats across labs is evidence.
• Who was studied limits who the finding is about.
• A failed replication is information, not an accusation.

Where it shows up next

• Social Psychology (35/25): obedience, priming and group studies are where the replication story is most dramatic, and the tools above decide what you can still trust.
• Personality & Individual Differences (35/26): reliability and validity are the whole business of a test, and heritability claims depend on design.
• Econometrics (34/44) and Marketing Research and Analytics (34/59): the same logic of confounders, A/B tests and natural experiments in other disciplines.
• In the world: reading a news story about a "new study", judging a health claim, running a product experiment, or deciding whether a therapy has evidence.

If you want to go deeper

• 📘 Textbook. Beth Morling, Research Methods in Psychology: Evaluating a World of Information (W. W. Norton, 5th edition). Widely used; built around judging claims in the news as well as designing studies.
• 📘 Textbook. Frederick Gravetter and Lori-Ann Forzano, Research Methods for the Behavioral Sciences (Cengage). The other standard syllabus text.
• 📘 Gentler. Stuart Ritchie, Science Fictions (2020). A readable account of fraud, bias, hype and the reforms.
• 🌐 Free text. Daniël Lakens, Improving Your Statistical Inferences: a free online book on effect sizes, power and open practices.
• 🌐 Free course. MIT OpenCourseWare 9.00SC Introduction to Psychology (Fall 2011, John Gabrieli), a full introductory course.