As I was driving home from work tonight I managed to listen to a really great episode of Ideas on CBC radio. The show talked about how a study claiming psychic powers were real managed to get published in a reputable psychology journal because the results were statistically significant at the 95% confidence level.

This led to a crisis in the field and the realization that we can’t be 95% confident in the 95% confidence level because scientists cherry pick which way they’re going to analyze the data, so that 95% is a biased sample of what they’re trying to measure (a phenomenon known as p-hacking). Kind of reminds me of how people only report their highest ever score when telling their IQ.

It turns out that only about one third to one half of all psychological claims proven at the 95% confidence level can be replicated by independent researchers. In other words, there’s regression to the mean.

And it’s not just psychology but almost the entire field of science is afflicted by this replication crisis.

You can listen to the full episode here.