How do p-hacking, type M errors, and the “winners curse” affect the research findings that make weekly news? Or the research findings published in academic journals? In this episode, Stanford University’s John Ioannidis and host Russ Roberts discuss the surprising frequency and enormity of the problem as well as some causes of false research findings.
1. What concerns you most about the extent of the false research findings surrounding us, and why?2. Referring to meta-analysis, why does Ioannidis suggest that even when these studies are completely flawed, they might give more information than a single study or observation?
3. Roberts refers to flaws in the research question of many minimum wage studies which are then compounded with an 8.5% median power of even detecting an impact on unemployment. Do you think this is depressing or revealing about the limitations of research design/studies?
4. To what extent might requiring increased transparency of information reduce skepticism and/or improve accuracy of research findings?
5. What advice does Ioannidis offer to young scholars in scientific fields? To what extent would you find this advice valuable?
READER COMMENTS
Luke J
Feb 1 2018 at 4:01pm
1) These studies seem to cause much hand-wringing, self-doubt, and resentment between persons in workplaces, churches, and legislators. And for what? To be on the side of “science?”
2) For the same reason listening to Econtalk weekly is more informative than reading one person’s blog.
lauristan
Feb 3 2018 at 9:18am
These discussions make me wonder about the utility of a whole host of quantitative inferential techniques compared to qualitative techniques. If you can’t sufficiently power your analysis, why not just do a qualitative study instead, where at least you can be more confident in the validity of your observations? (Or if you want to just explore the data and generate further hypotheses, that’s cool, too!)
Of course, in some areas of inquiry, there are no substitutes for under-powered quantitative analysis. For example, when your research is about cross-national trends in something squishy like democracy. If we treat all such observational research as “exploratory,” then one of the logical next steps for such research is rigorous qualitative case studies to explore the how and why of the findings in the exploratory analysis. Then, at least we could get information from these studies that can be used in applied work. For example, a rigorous process tracing where cases of a successful outcome are dissected so we can learn what made them successful.
Pete Harvard
Feb 5 2018 at 12:20pm
I found this podcast confusing. If you reject a null hypothesis that there is no effect from a policy, but the power of the test is only 18% on average, what does that mean? What’s the % for low-power vs. high-power tests? How does the fact that the test was low power lead to the true effect being so much smaller than the measured effect? There was a lot of talk about meta-data, but that often seemed tangential to the problem that measured effects were probably not really there. Are you saying that the effect was probably picked up by chance? I’ll give this a second chance, but where is the follow-up discussion showing graphs of these concepts clarifying all of this?
Pike Smalls
Feb 6 2018 at 3:07pm
A link to the original PLOS|medicine essay.
http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124
Is there a layman’s shortcut to determine the power of a given study? I’d like to see what some studies I know reveal. Thanks.
Comments are closed.