Brian Nosek, professor of psychology and Executive Director of the Open Science Project, joined EconTalk host Russ Roberts for a follow-up conversation on the results of his meta-analysis of 100 psychology studies, seeking to replicate their findings.
So what did they find? The answer is complicated…But we’re most interested in what you think of the results and this week’s conversation.
1. Nosek and Roberts make reference to the recent news that there’s a link between bacon consumption and cancer. What does this story have to teach people about p values and statistical significance? 2. Are the results of the Reproducibility Project good news or bad news for psychology?
3. At the end of the conversation, Roberts asks why this issue is getting so much attention today. Nosek’s answer, in part, is “everybody is working on problems that they don’t really understand the answers to yet.” In other words, to what extent is Nosek’s project suffering the pretense of knowledge?
4. Did this episode reduce your confidence in news stories about research findings or books that draw on peer-reviewed psychological studies? Should it?
READER COMMENTS
jw
Nov 20 2015 at 5:40pm
Full disclosure: I haven’t listened to this podcast yet (but I have heard Russ discuss it before and read up on it). If any of the following is changed after I listen (I haven’t missed one in years), then I will post a correction.
1. The WHO study and others like it have already been thoroughly and repeatedly debunked, p-value or not. (See: World Health Organisation, meat & cancer and Is red meat killing us? to start. There are many more.)
In fact, the main source of most nutritional data on humans, the food questionnaire, has been deemed so inaccurate that it should not be used going forward. This also should call into question almost all nutritional research to date.
Remember that the effect of smoking increases risk by 600%. Anything less than say double (100%) may be subject to a healthy degree of doubt (risk is not an exact science), but that won’t get papers published. You will still see headlines screaming “18/23/82/12/42% Increase in Risk!”, because news sites and newspapers always need scary headlines. It is a very cozy and symbiotic relationship.
The prestigious Lancet has also published a “study” on how reducing red meat consumption will reduce climate change. Nonsense piled upon nonsense.
4. No, it has not reduced my confidence because, having read thousands of “studies” (the original papers, not the popular press, if you really want to understand things, you MUST read the source material, it really isn’t that hard), my level of skepticism can hardly be much higher. Well done science is rare not just in psychology, but nutrition, finance, climate/social/political/educational “science” and (sorry Russ) economics, just to name a few.
jw
Nov 23 2015 at 1:29pm
Firstly, congrats on 500!!
Secondly, after listening, I should have remembered from past podcasts that Russ recognized that his profession has some of the same issues covered in the podcast.
Thirdly, besides the issues with the red meat study outlined above (and more available to anyone with just a little Googling…), the other big problem is that the media only ever reports the headlines released with the abstracts from a study. Rarely do they look at the confounding factors and even more rarely do they try and provide perspective on the risks vs the benefits.
So say that red meat actually did increase the risk of cancer by 18%. Do they act like statins, which can reduce hearts attacks, but DON’T reduce overall mortality? Does the increased risk of cancer offset the benefits of cheap protein, healthy fats, and difficult to otherwise come by vitamins?
Lastly, it turns out that even in physics there are controversies. I’m reading “Spooky Action” (Muller) and although after time everyone accepts the mathematics behind non-locality, I had thought that everyone had also accepted the non-locality hypothesis, but it turns out that many physicists can’t stand the idea and have “alternative” theories to maintain locality (naturally, this is super-simplified explanation).
So even when the math and experiments support the hypothesis, people still can’t accept the conclusion.
Amy Willis
Nov 24 2015 at 11:13am
@jw, thanks for your comments! (And the new reading suggestion…)
Comments are closed.