Russ Roberts

Yong on Science, Replication, and Journalism

EconTalk Episode with Ed Yong
Hosted by Russ Roberts
PRINT
Larry White on the Clash of Ec... Jonah Lehrer on Creativity and...

Ed Yong, science writer and blogger at "Not Exactly Rocket Science" at Discover Magazine, talks with EconTalk host Russ Roberts about the challenges of science and science journalism. Yong was recently entangled in a controversy over the failure of researchers to replicate a highly-cited and influential psychology study. He discusses the issues behind the failed replication and the problem of replication in general in other fields, arguing that replication is under-appreciated and little rewarded. After a discussion of the incentives facing scientists, the conversation turns to the challenges facing science journalists when work that is peer-reviewed may still not be reliable.

Size: 32.8 MB
Right-click or Option-click, and select "Save Link/Target As MP3.

Readings and Links related to this podcast

Podcast Readings
HIDE READINGS
About this week's guest: About ideas and people mentioned in this podcast:

Highlights

Time
Podcast Highlights
HIDE HIGHLIGHTS
0:36Intro. [Recording date: May 29, 2012.] Russ: You recently got entangled in a controversy over replicability in psychology, which is our first topic. Guest: That's right. Russ: We're going to talk about scientific results and how that translates into journalism. And I want to start out with your experience in the psychology area, and a particularly well-known study that got complicated because of replicability. That happened there? Guest: So, the original study was published in 1996. It is by a man called John Bargh, who is a very well-known social psychologist, and it showed that people who are primed with words related to old age--so, if they see, if they are unconsciously exposed to these words--then they will walk more slowly down a corridor. Russ: This was after the experiment. The experiment was nominally about one thing, but actually it was about something else. What the experimenters were interested in, when the experiment was over, if they'd seen words that related to being old, they'd walk more slowly. And I have to say, Ed, it just doesn't pass the sniff test for me, to start with. Guest: Right. Russ: It was an extremely renowned study, correct? Guest: Highly renowned. It's been highly cited, thousands of times. It's there in most of the most popular psychology books of our time. It's very well known. But the amazing thing is it's been very seldom directly replicated, by which I mean very few people have taken exactly what was done in the original experiment and reproduced it. People have done tweaks on the original and done other studies to show whether similar trends would happen: so when other unconscious exposures to different types of words could actually affect people's behavior. So, this takes us up to--no, actually, the start of this year--when I saw a paper which was a replication attempt of this seminal study. Which failed to repeat the original experiment. So, this group in Belgium led by a man called Axel Cleeremans, tried to reproduce the study. They tried to hew to the original design as far as possible, and they just couldn't see an effect. What they did see was that people only walked more slowly down the corridor if they had been, if the experimenters in the experiment had been specifically told that that was what was going to happen. So the idea was that if in the experiment it is assumed that these people are going to walk more slowly, then that's what they see. Russ: Let's back up for a little bit. This phenomenon--of course, in psychology jargon is often everything--is called priming. Was this the first priming study, or were there others before it? Guest: No, I don't think it was the very first priming study, but I think it was a big moment in the field. And since then there have been hundreds, if not thousands, of studies using the same techniques. So the idea is basically that if you unconsciously expose people to certain stimuli, whether it's words or pictures, then it affects their behavior. And I don't see anything particularly problematic with that concept. Russ: But the example--let's go to the actual Bargh experiment, make sure people understand it. So, you'd have two groups. One group would be primed; the other group not primed. In the primed group, they'd be given some exercise. In the exercise, words like "old," "aged," "senior"--I don't know what else they used--but words that denoted being old would be used in the exercise. And the other group would not get those words. They'd get different words that didn't suggest being old. Guest: That's right. Russ: The image I have is of the primed group, then, when they leave the experiment they are videotaped going down the corridor; they shuffle. What's the magnitude, roughly, of the difference the two groups as they exit the experiment? Guest: So, in the original two groups, they had one data person with a stop watch timing them at the end of the corridor. And in the new versions they had infrared sensors, which are meant to, which were used to hopefully give a more accurate reading. Russ: And in the original experiment, the Bargh experiment, what were some of the magnitudes of the measurements between the two groups? Guest: I can't remember the size of the effect off the top of my head. Russ: But it was large, right? It was dramatic. Guest: Well, by the standards of these types of these experiments, yeah. It was a pretty big effect. Russ: So, what this replication study found was no effect, unless you told the guy with the stop watch, before they used the infrared? Guest: No, the guy who, the person who ushers the volunteers into the experimental room gives them a sort of envelope with all the test materials in it. If that person thinks that the experiment is going to show that priming reduces walking speed, then they do walk more slowly. Russ: Weird, right? Guest: It is weird. Russ: Slightly alarming. Guest: Right. But I think what was interesting about this wasn't so much the specific details of the replication attempt versus the original experiment. It's what it opened up for me in terms of the sort of cultural norms of psychology. So I'll tell you what happened afterwards. Russ: Yeah, go ahead. Guest: Well, a few months after the paper came out and my post came out, John Bargh, the author of the original study, wrote a very irate post on his own blog, which appears on Psychology Today, and he slated the research team behind the replication attempt. He criticized me for covering it; and he criticized the journal it was published in, PloS One. And it all sort of kicked off from there. And then I wrote a post in response to that. Several other commenters on Bargh's post pointed out certain inconsistencies in his analysis of the situation. I wrote a response to that. And this kicked off a long discussion in the comments of my own blog about the role of replication in psychology. I was getting emails--I was getting quite a few comments, but also lots of personal, private emails saying: Oh, our labs tried to replicate this experiment and couldn't, or we know other labs who have also tried to do these replication experiments and failed. And it sort of opened up this world for me, this undercurrent of failed replication attempts that are being done--that are rarely being done and when they are being done, they sort of circulate around through the gossip and never really see the light of day. And that seems to be really important, because replication is an act that should be a backbone of science. It's our way of checking to make sure that the stuff that enters the public literature is actually going to be true.
8:50Russ: Yeah, well they call it social science. And my field, economics--we'll talk about that in a little bit--has the same issue. And we're not alone. This problem, as we'll see, leaks into so-called real sciences, physical sciences, etc. But the reaction--it's not surprising that the author of the study defended his own work. What was the reaction of other people who weren't what we might call, what Adam Smith might call, the impartial spectator? You had a stake in this; you'd thought thoughtfully. I'm on your side. But Bargh accused you of being gullible or whatever was his attack on you. What kind of response did you get from other folks? Guest: So, a couple of things. I think in general one of the strongest responses, one of the most frequent responses I got, was: I'm glad that people are talking about this issue; I'm glad that we are having a chance to bring this issue to light. And following on from this discussion, I taught a bunch of people and I did a bunch of moving around, and I think there is this sort of seething undercurrent of unease about the current state of psychology. And not from people looking in, going: social scientists are not doing proper science. These are psychologists; these are people within the field worrying about the state of their own discipline. And I find loads of these conversations going on. A few papers analyzing the issue; a lot of blog posts, a lot of conference presentations, all talking about this. It does seem that there is--I don't know if there is a majority, but there is certainly a growing movement of people within the field who are discussing this problem. And then there were people who said this is all a waste of time, because priming has been demonstrated again and again in lots of different studies; it has been "conceptually replicated," is the term; and we can talk about what that means and what the issues are later. Russ: No, you can talk about it now. What does that mean? Conceptually replicated? That's a strange thing. Guest: Right. So the idea is that in psychology, which doesn't have the benefit of dealing with very hard, reliable concepts--so it's not looking at a gene or an animal or even a planet. It's looking at people's minds, in all their hazy, nebulous glory. And because it works on abstract constructs, it often runs into trouble repeating experiments across different populations in different times. So, you do the same experiment on different groups of people and you might get very different results, not because your original experiments were crap, but because you were just doing it on a very different group of people. Now, one way of getting around this is to do sort of different experiments that test the same underlying idea. So, in terms of priming, for example, we've talked about an experiment showing that primed people with age-related words makes them walk more slowly down a corridor. Now, that in itself isn't particularly exciting. No one started off going: What we really want to know is what conditions make people walk more slowly down a corridor? That is not an exciting research question. What is exciting is the underlying idea: Can unconsciously-presented stimuli affect people's behavior? So, the slow-walking study is one instance of that, but you could do lots of other tests, and people have done lots of other tests. And in psychology, or at least in certain underlying disciplines, it seems to be common practice to take all these disparate attempts to test the same underlying concept and go, well, they effectively replicate each other. So, they support each other. Does that...? Russ: Yeah, that explains it. As a person who has run a lot of regressions in my youth, I know about the tendency, both consciously and subconsciously, to convince yourself that this is the right version of the test; this is the right specification of the equation, in the case of regression analysis. And in the case of these shadowy things, what I suspect happens--of course, there's fraud. Occasionally. But fraud is not the real problem. Fraud, you can find fraud and you can actually see fraud when you do find it. The real problem is subconscious fraud. You run the equation--and they don't get any effect. And you think, well, priming is true, of course; so you use the right words: I had a funny population; these were sophomores; I should have had seniors; I should have had a wider group; they knew each other. It's very easy to convince yourself that it didn't go right here. So then you re-do the study, and you change the words. Instead of saying "old," you say "very old," or "a septuagenarian," instead of "senior citizen." And then the problem is, of course--as Ed Leamer in economics has pointed out; we've interviewed him on this program--once you start altering the terms of the experiment that way, the classical tests of significance no longer hold. And so, by pure randomness you are going to get significant results if you run 50 or 100 studies. And then that's the one you convince yourself--that was the right one, because of course it showed priming is true, as everyone knows. Guest: Right. There are a couple of problems here. I ended up writing a feature for Nature about this, the wider issue of replication and publication bias. And there are definitely a couple of big problems here. One is that it seems almost culturally commonplace for people to do these tweaks, these things that you've just mentioned. So, things like: checking your significance level during your experiments and then stopping at the point where you get a significant result. Or not deciding a priori how big your sample size should be or what your statistical test should be. Russ: Or throwing out an outlier: oh, that guy, he didn't count. Guest: Right. So you are making on-the-fly decisions about how to run the experiments in ways that virtually guarantee a positive result. And firstly we know that this is incredibly easy. So there is a guy called Joe Simmons; he's done a really interesting study on this. He outlines these, what he calls "researcher degrees of freedom." And he applied them to a genuinely collected data set that he had and used it to "show" that people who listened to "When I'm Sixty-Four," by the Beatles, actually became one and a half years younger. Not felt one and a half years younger. Actually physically de-aged. Russ: It's a miracle. I always said the Beatles were a great musical group. But it turns out they are even better than that. Guest: How amazing. So, this was published. It was done in a very tongue-in-cheek way. Joe's point is that it is very easy to produce these statistically-significant, apparently positive results by kind of playing around with your data set. Now, it can happen. And then there was another paper by Leslie John, who showed that it does happen. And she did a survey of different psychologists and many of them admitted to exactly these things. By many, we are talking, I don't know, 40%-50% admitting to a lot of these tweaks. So they are commonplace. They are defensible. This came out a lot in discussion that followed my tete-a-tete with John Bargh, where people were saying things like: I've tried to replicate experiments before, you know a lot of these replications attempts fail, and I don't really think that much of it. For all these reasons. So, that's one thing.
17:30Guest: And the second thing is perhaps more understandable, but still a little problematic: which is that psychological experiments are often very difficult to conduct. Russ: Yeah. They are expensive. Guest: Actually, they are probably not that expensive. You do them on grad students; you use grad students to do them. But they are difficult to conduct because people are different. People are weird and very variable. Russ: All kinds of subtle things matter--the intonation of the voice. Unless you had a videotape of how this was gathered and watch the eyebrows of the researchers and the people running the experiment, you might miss subtle clues that encourage certain results. Guest: Exactly. Russ: Very hard to truly replicate an experiment. Guest: Exactly. And small things matter. Now one of the people I interviewed for my Nature piece was Daniel Kahneman, who is a Nobel-winning economist. Russ: Yes; and I have to mention that my wife just finished his book; and she mentioned to me that he cites the Bargh study approvingly. Guest: Absolutely. He's very supportive of that line of research. I spoke to him about it. He felt, and he said to me, he wouldn't try and replicate Bargh's studies because he doesn't think that he has the skill or know-how to do so. So Kahneman would argue that in priming studies, because the entire idea rests on the fact that you are unconsciously experiencing these stimuli, if you in some way draw people's attention to the fact that they've been primed, the effect no longer works. So, you need to very carefully orchestrate the studies so people aren't aware of what is actually happening to them. He talks about a knack that Bargh has and his students have to pull these sorts of things off. Russ: That's one name for it. Confirmation bias would be the other. Guest: Right. Russ: It's a big challenge. Guest: Yeah, it is a big challenge. This is the thing. There is something to be said about that. I am perfectly happy to accept that there is a degree of experimental skill and artistry in pulling these things off. Russ: No doubt. Guest: But on the other hand, that's a very easy out. Russ: It's dangerous, too. Guest: Yeah, exactly. For someone who doesn't have that and is actually pulling a false one, that gives a very easy excuse. Now I'm not by any means saying that Kahneman or Bargh is guilty of this. But the fact that this idea is out there creates this environment in which it's very easy for these sorts of dodgy cases to happen. You talked about fraud. One of the things I mentioned in my Nature piece was the case of Diederik Stapel's. He is a social psychologist, he was a social psychologist at the University of Tilburg, by all accounts a rising star in his field. And we now know that he was guilty of fraud on a massive, massive scale. He was fabricating data. He did so in at least 30 papers, probably more. And he was claiming to find these effects that weren't real; he just wasn't doing the experiments. Russ: It's handy. Guest: Right. Russ: That certainly lowers the cost. As you talk about the artistry of this--and again, there's a certain culture that accrues through graduate school, and it certainly happens in economics; I'm sure it happens in all the social sciences and all the sciences--there's a certain respect for that artfulness. There's a certain skill. Running an experiment, doing statistical analysis, is not a recipe-driven, manual-driven activity. There's an art to it. And certain artful techniques are applauded and perhaps a thoughtful person realizes they are dangerous. And as you talked about Kahneman's hesitancy to replicate some of these studies because that's not his skill set of this kind of artfulness, I am reminded of Richard Feynman's quote, which is: the principle is you must not fool yourself, and you are the easiest person to fool. It's such a dangerous activity for a so-called scientific researcher to trust in his own artfulness and to claim--for himself to claim or herself to claim--that others can't replicate my work; I'm too skilled at it; just stand back and applaud. Guest: No, I should clarify here that I'm actually spoken to Bargh about this, and he that's not what he says about his own work. He very clearly said to me, he doesn't want there to be any secret knowledge; and perhaps what this means is that method sections in psychological papers should be given more space, so that authors can more clearly lay out all the conditions that need to actually be followed in order to replicate a study. So, in fairness to John, he's not the one claiming he's got some special knack. What's dangerous is when people attribute this to other scientists. So, when you look at the report from the investigation committee that ruled on the Stapel incident, they very clearly say, when very few people try to replicate a study and when they did they failed, they assumed it was because he had some skill that they hadn't. And there you have it. People who are actually doing the work necessary to find if this stuff is true, they are not finding it; and they are thinking: Well, it's probably my fault. And this nicely tied back in to what we talked about at the start of this, about conceptual replication. So, in a theoretical sense, you can see why it's a powerful idea because you are showing that individually acquired pieces of data and results, in very different settings, different countries, different groups of people are finding similar thematic relevant things. And that bolsters the underlying concept. But that only works if each individual piece of data is solid in itself. If it hasn't been acquired through these degrees of freedom that we talked about. If they haven't been published because of the desire and the bias towards sexy new results. And assuming that all these things happen, which we know they do, then what you've got are lots of studies which could have appeared because of publication bias then supporting each other. Which, as you say, is just the very epitome of confirmation bias. Russ: And if you are on the other side, it's a feature, not a bug. Because, hey, look how rich the idea is. It applies everywhere! Guest: Exactly. Right, right.
24:36Russ: So, when you talk about publication bias, you are talking about the fact that there is not a lot of--there's a huge incentive to get published. And in general it's easier to publish positive results than negative results, correct? Guest: Exactly, yes. This is for many different reasons. It's because of pressures from the journals, academic pressures and so on just from their careers. It's by no means a problem unique to psychology, although psychology may have certain issues with it. I mean, psychology has an interesting history of publication bias because one of the first real attempts to quantify came within psychology--a statistician called [Theodore] Sterling, who found--I can't remember the actual date now; I think it was some time in the 1950s--and he found that the majority of papers in some big psychological journals--here we go, 1959, he found 97% of studies in 4 major psychology journals were reporting statistically significant positive results. By anyone's measure, that is too many. And when Sterling repeated his analysis several decades later, in the 1990s, it was virtually the same. It is a problem. I think individually all of these things are a problem. The over-reliance on conceptual replication rather than direct replications, the degree of freedom issue of tweaking studies on the fly, the degree of publication bias, the fact that when you actually replicate a study it can be very difficult to know that that means. But synergistically, all of them work, I think work together, to enhance each other's failings. Russ: Yeah, no, I agree. I like to pick on psychology. When I do, by the way, it's not because it's a sister social science that I look down on. Although that's true--I have to confess; occasionally, when I was younger especially, I had the arrogance of economics over the other social sciences. And now I've evened things out. I'm less confident about economics insights statistically, in terms of data. But it's not just a psychology problem. The reason it bugs me is these books get published, and my friends read them and they talk about how cool these results are; and I always say: Do you really think those are true? Have you ever thought about whether they are reliable? And of course they don't, because they confirm their own biases. We all love novel results or counterintuitive results; and so it's a huge problem in economics as well. But I think much more troubling than the psychology literature's problems are those in economics and those in epidemiology. A recent study found--and I have not looked at this carefully--of course you have to be skeptical of all studies, even those that are skeptical of studies. But this was a study that identified 53 landmark studies in cancer research and allegedly 47 of these 53--47!--could not be replicated. And the part that I found--that really of course confirmed my own biases, you have to be careful here--is that the author, Begley, was talking to one of these studies that couldn't be replicated, and he said: We went through the paper line by line, figure by figure; I explained that we re-did their experiment 50 times and never got their result. So, I'm interrupting here to remark: 50 times; they gave it a good shot. And then, the quote continues: he said, he'd done it--he meaning the author of the original study that had made the claim--he said he'd done it 6 times and got this result once but put it in the paper because it made the best story. And that really sums up the problem. I mean, good grief! We're talking about cancer research. It's one thing to say people can be primed by language, or you put people in a blue room and they're more creative--I don't believe these kinds of studies until they've been replicated. But cancer research? People read these things, and they get scared, and they actually change their lives about what they eat and what they do. It's kind of more important. And they've got the same problem. Guest: Yes, absolutely. I used to work in a cancer charity, so this stuff weighs heavily on my mind. And we used to talk about epidemiological studies. Now, I think you are right that one has to be a bit wary about studies that look at other studies, and especially in this case, because it's industry-funded stuff. We still don't know which of those 53 studies have or have not replicated because that data have not been publicly released--which I find wonderfully ironic. Russ: Yes, true. Guest: But you are right; it does point to perhaps some systemic problems. There's the famous work by John Ionnidis, who claims that most published scientific findings are probably not true; he's talking about medical research and he's basing his argument on statistical logic. But you do see these pieces of data in psychology and research in all these fields which suggests that there are problems there. I would hate for listeners to come away from this thinking science can't be trusted. This has kind of come out of the woodwork. What I'm interested in in situations where there are cultural norms that make it much harder for that to happen. Science is meant to be a famously self-correcting process. Under what conditions does it fail to correct itself. That's why I've focused on psychology because I think it provides some interesting case studies there. Russ: You raise an interesting issue which fascinates me. I have noted a large defensiveness on the part of scientists on these issues, because they are afraid that if we find that science isn't as scientific as we thought it was, maybe that opens the door toward religion and superstition, supposedly. And I think the alternative is much worse--to fool people or to be dishonest about the limits of scientists as opposed to the limits of science. They're very different things. Science is one of the great achievements of the human mind; but what it consists of that we can act on and rely on and use is much more complicated. But I do think there's a cultural pushback from the scientific community to be very careful about publicly criticizing scientific results because they are afraid of the cultural implications. Guest: Yes and no. We should definitely come back to this because I think it ties into the issue of science journalism, which I know you wanted to talk about. It feeds in quite nicely to that. But one thing I did want to point out is another reason why I decided to write this piece about psychology is not because I wanted to write a hit piece about this field or because I wanted to slate psychologists, but because one of the ways in which I think psychology stands out is that there are people who are talking openly about this. Russ: Yeah, that's true. Guest: Every single person I quoted in my piece, including all the ones saying we have big problems here, are psychologists. They are not outsiders having a go; they are people inside the field taking a more introspective look. And there does seem to be more of this stuff going on. I have quotes which didn't actually make it into the piece that sum it up well. One guy--I'm not saying these people's names because I'm not sure--but one person said something like: If someone handed me a result that I had no idea about and it was statistically significant, my probable default state at the moment would be to question whether it was true or not. And someone else said: I like to put my graduate students with a thought experiment, which is if all the scientists in the world suddenly died, if all the psychologists suddenly died, and all we had to go on to restart the field was the published literature, would we be in good shape? And the answer is: Probably not. Russ: That's a great thought experiment. Well, a macabre thought experiment. Guest: But instructive. Russ: But economics has the same problem. And by the way, that semi-quote, opinion, that you'd be skeptical of anybody telling you some result--well, that should be the attitude of any scientist. Right? It should never be: Oh, thank goodness, you've finally confirmed. Or: I'm so glad that's true. It should be: well, that's interesting. Let's see if it stands up. That's the scientific attitude, it seems to me, in any field, whether it's psychology, economics, epidemiology.
34:17Russ: Let's segue--I want to mention one epidemiology study in particular and we'll use this as a segue into the journalism issues. A study came out in the last few years about the perils of alcohol consumption for women. And it was a massive study; it was done in the United Kingdom; if I remember correctly it was over 100,000, well over 100,000 people in the sample, which I think means in some people's minds: Well, then it must be reliable. Which of course has nothing to do with it. It's nice. But it was a massive study of women's health. And the researchers discovered that even small consumption of wine and liquor--I think as little as one drink a week--raises your risk of an enormous list of types of cancers. Now, that study was on the front page of numerous newspapers when it broke. And the journalists of course covered it with great zeal. The highest level of skepticism that I found was toward the end of the article you'd sometimes see an interview with someone who would say: Well, yeah, but drinking wine is probably good for your heart; that's been shown; so they probably cancel themselves out, so if you like to have a nice glass of wine with your meal you are probably okay. That was the highest level of maybe we should consume this with a grain of salt. So, I went and actually looked at the study; I'm not an epidemiologist. There were a number of strange things in the study, one of which was: they had data on prevalence of cancer in the family, which they did not use--strangely to me--in the statistical analysis. But the strangest was--this is one throwaway line or two lines in the middle of the long paper--they said: We threw out the people who didn't drink. Guest: Right, right. Russ: That's a strange thing to do. Why would you throw out of your sample the people who are dramatically going through the non-treatment effect that you are looking at? They are sort of the perfect group you want to look at. They are the people that don't drink at all. Obviously you are also interested, among drinkers, is it true that the more you drink, the sicker or healthier you get; but you'd certainly want to look at non-drinkers. Well, they threw them out. Why? Well, they said it's a retrospective study. When you ask people: Are you a drinker, how much have you had to drink in the last week or last month--I can't remember exactly how it was worded--well, some of those people who said zero would have been drinkers before. So they didn't drink the last month or the last year, but maybe they became teetotalers; maybe they gave up alcohol. And so by including them in the study we are contaminating the results, so we'll throw them out. Well, that's true; of course it's also true for people that say they drank like a fish. Those people could be people who didn't used to drink but maybe now they've got a bad health situation or a thousand other things and they've started drinking. And they really don't have a lifetime of heavy drinking behind them. So you can't partially justify. And of course, strangely enough, the people who didn't drink had different cancer rates. They were higher than the modest drinkers! But they justified throwing them out--because they weren't reliable. To me, as soon as I saw that I thought: this is junk; this is not acceptable. But of course it's on the front page of every newspaper--probably in the world. Certainly not every, but many, many prominent newspapers featured it; didn't have any caveats. And so women either reduced their drinking or got less pleasure from it because they were afraid they were killing themselves. It's incredible to me that the journalism community can't get to that--and again I'm not an expert. Guest: Let me interject here. It's an interesting one for you to pick because I used to work in a cancer charity. And our bread and butter was to talk about epidemiological studies like this to the press. So, I don't know which specific one on alcohol and cancer you're talking about, but chances are I've probably been interviewed about this before. Now, my take is--I get what you are saying about the non-drinking category of people; there has been an ongoing debate in this field for a while. There is the so-called sick quitter effect. So you get an odd tick on the left-hand side of the curve if you include people who used to drink really heavily, got very bad disease, and then cut down to nothing. But my main point for the alcohol stuff would actually be it reflects to me one of my big bugbears about science journalism which is the treatment of every new individual discovery as an isolated point. Russ: Yeah, the first one. Guest: And this is particularly bad for epidemiology. Which, as we all know, relies on strength in numbers. You've got lots of different studies; do they find the same thing? If they do find the same thing--and maybe this ties into the conceptual replication stuff--but if you test the same hypothesis again and again and they find the same thing, then that tells you something. And with alcohol and cancer, the International Agency for Research into Cancer (IARC), which are a French organization linked to the World Health Organization; and they hand out the sort of gold-standard rulings about whether something convincingly causes cancer, is a probable cause, a possible cause--that sort of thing. So, they grade strength of evidence. They were saying that alcohol causes cancer in those terms in that stark language back about 20 years ago. And ever since then there have been dozens, if not hundreds, of large epidemiological studies which all find the same thing. And it's not just that--you have studies looking into [?] mechanisms, so is it plausible that alcoholism causes cancer. Russ: That's real science, by the way. Guest: Right. Russ: It's one thing to say we see a correlation; but to understand the mechanism, that would be better. Guest: Exactly. So, is there a mechanism. And there are mechanisms. So, this is actually a pretty strong link. And if you look at the epidemiology, based on observational studies, which are not randomized control trials--there's always going to be a certain degree of uncertainty there about the direction of causation, but I think most people would be happy about saying smoking causes cancer, and that's based on observational evidence. Russ: Yep. Guest: And the evidence for alcohol, while not as strong as for smoking, is certainly stronger than most other risk factors out there. It's just that people don't know about it. Russ: Fair enough. Guest: People don't know about it because every new report fails to mention this massive cumulation of data that's gone on since the 1980s.
41:32 Russ: Fair enough, but let me push back a little bit, which is based on my conversation with Gary Taubes and his research on diet and the relationship between diet and health. Well, study after study shows that fat is correlated with heart disease. But maybe those studies are just the product of group think, and other incentives that we haven't thought about--a desire to get along with what's thought to be the received wisdom in the field. And if I really wanted to push it: A lot of doctors I know think drinking is just evil. It's a toxin and you shouldn't be putting it in your body, and they don't accept what an economist or others might accept as a tradeoff between health and joy, or other issues. It's like, they think it's immoral to ride a motorcycle. I don't ride a motorcycle; I think it's nuts. But I had a friend who rides a motorcycle. He broke his leg, and he went into the hospital; and the doctor said: I'm uncomfortable setting your leg; I'll do it because I have to, but the truth is I know you are just going to go do it again. And I thought: That kind of misunderstands the nature of the human enterprise. But doctors look at the world differently sometimes, not all of them, but they don't always look at the world the same way as other folks do. And so I'm very comfortable with the possibility--of course, I like to drink now and then, I have to confess; I have a scotch occasionally on the weekend; I don't think I have a drinking problem. But I feel often that it's possible to me that some of this research is, shall I say primed, by the cultural attitudes on this position. Guest: I'll tell you what, though. If that hypothesis were true, then the country where you would get studies saying that small amounts of alcohol increase your risk of chronic disease would not be Britain. I don't think that our culture is really compatible with that viewpoint. Russ: Yeah, no. Guest: But I take your point. The general thing here is that scientists are people; and I think that people forget that. I think sometimes journalists forget that even though it's really their job to sort of iron that out. And I think scientists forget it, too. What's fascinating to me is there are endless arrays of scientist-versus-journalist debates--scientists have a go at journalists but not having science journalism correct, and science journalists having a go at scientists for, I don't know, demanding too much or being nitpicking. And I find that when this happens, a lot of very sensible, very well-informed scientists that I know suddenly take or forget that their peers are full of it. Not something they know in their daily lives, because they review terrible papers and they are on the blogosphere and on Twitter criticizing terrible papers, so they know there are other scientists out there who are not agents of truth. But I think this gets forgotten. I think it gets forgotten on both sides, both scientists and journalists. Russ: But let's put the details of this particular study of alcohol and cancer aside for the moment. I brought up the example, yes, to take a potshot at how epidemiology works; but also the fact that, whether the study is reliable or not, the way these are frequently treated in the media is as truth coming down from Mount Sinai. This is just a divine pronouncement, a study has been done, a statistically significant result has been discovered, it's been refereed, ergo we have notched another mark in the search of truth. Guest: Right. Russ: And I think the thing that bothers me--and I'll let you talk about you, because you are in your profession on the inside; I'm on the outside; it's a cheap shot on my part, maybe, but what bothers me isn't just what you've written. It's not just the lack of statistical sophistication. It's the willingness to suspend disbelief and skepticism in the face of these publication biases and other human biases we've talked about. And journalists stand between the public and the--they're the bridge between the public and the scientist. And I wish they did their job a little differently. And I think journalists, being human, are sometimes under the same sometimes unhealthy incentives of publish quickly, publish splashy. And I'd like to hear from you, as a person with a great deal of experience in the field, published a wide array of publications, how those incentives, how they work. Because they must be real. Guest: Yeah. So, one thing I can speak to is the pressures that a newsroom journalist would be under. I think we are all familiar with that. They are under a lot of pressure and under very tight deadlines. They need to sell ideas to editors, so they might need to glam it up to people who have no scientific knowledge whatsoever. They might have no scientific knowledge whatsoever. All these sorts of things. I've fortunately never had to deal with this. My background is in freelancing, and I write for a lot of places which I think have a lot of very good editorial sense. But I think that one of the issues is that--and it applies not just to science journalism but to the wider area of science writing or science communication--is that a lot of people get into this because they love science. Russ: Sure. Guest: That's how I got into this. I love science; I wanted to talk to people about science. And I think when you go in with that mentality, when you start off, you don't really--the idea that a lot of this might not actually be true doesn't weigh upon your mind so much. You are interested in the business of explaining complex concepts to a lay audience. You are trying to make science accessible and interesting and cool rather than playing that watchdog role. Russ: Yeah. And your blog does it very well, by the way. Guest: Thank you. Russ: There's a lot of intellectual excitement there that has nothing to do with these issues of reliability of statistical analysis. Guest: Sure, sure. Thank you very much. When I started with the blog, that was very much my mindset: I'm going to effectively translate these papers and present them in a very interesting way. And I think that the journalistic side of it has sort of appeared over many years. And it wasn't there to start off with. The act of actually verifying whether something was true, calling around, getting different people to comment on something--I always tried to take a skeptical eye to what I read. But I think that's becoming more and more prominent with time. And I think it's a hard job, actually; and I think it's becoming harder the more complicated science gets. Russ: You bet.
49:01Guest: So, I'll give you an example. One of the reasons why I take this a lot more seriously than I used to is me just getting sick of having covered papers which end up being fraudulent or retracted. Surely I could have done something beforehand to check that out. And one case involved a genetic study of variants that are associated with longer life. And this was published in Science, one of the world's top flagship journals. And it was later retracted for, without going into the nitty-gritty, basically not being true. Now, at the time, I actually sent this paper out for comments to a bunch of people, to a few gerontologists. I'd actually written about long life, and I knew a few people. And they all came back saying, yeah, great paper, very interesting results. And the minute it came out, people who are actually specialists in these specializations, GWA studies, just tore it apart. They immediately said: Well, hang on; there's something really fishy going on with your stats and your results here. And that's what hard, I think. Science is becoming very complicated. It's becoming very interdisciplinary. And more and more we are getting papers where you can't just hand them to a couple of people and say is this kosher or not, because, firstly, they may not have the skills. They may only know only one side of the field. They may not have the statistical [?] to know what's going on. Or, worst case scenario: an entire field may be in collusion with a sort of received line of thinking. So, I don't want to make accusations here, but we talked about the conceptual replication issue. So, lots of people are buying in to the same concept, and you get a new paper that's involved in priming and you send it out to the priming researchers. Russ: Yeah. Guest: Well, what are they going to say? So, this act of verification, which I think is very important, involves asking the right people the right questions. And involves becoming incredibly knowledgeable about these fields yourself. And that is increasingly hard as science becomes increasingly networked and cross-disciplined. Asking the right people the right questions, and involves becoming incredibly knowledgeable about these fields yourself. And that is increasingly hard, as science becomes increasingly networked and cross-disciplined. Russ: Let's talk about the Internet for a minute and how that's changed things. I've recently noticed that, one thing that always intrigues me, is things that people know are true that turn out not to be true. One of which is that the life expectancy of a football player in the United States is 58. If you google "life expectancy nfl 58"--I didn't have to add 58, I shouldn't have, but if you google that, you'll find article after article that says, and this is the quote: According to the National Football League (NFL) Players Association, the average life expectancy for an NFL player is 58. Now the NFL Players' Association, of course, is not a scientific organization. They have a bias. They have an incentive to exaggerate the dangers of football to some extent, to justify programs the NFL might provide for older or unhealthy players; reminds me of the time I hear from people when they are angry about it is that America's infrastructure gets a D. Roads, etc., are very unsafe; and how do we know that? Because the American Society of Engineers or whatever their official names says so. Yeah, they would. Doesn't mean it is safe or that it should get an A. But if the people who profit from the result--you kind of need to be a little bit skeptical. So, I think a Sports Illustrated looked recently at the life expectancy of NFL players. It's not 58. They seem to be healthier than the average American. I don't know if they live longer, but it's not 58. And they couldn't track down where that number came from. But that number now is truth. It gets repeated over and over and over again. Guest: Right, yeah. Russ: The Internet makes that easier to discover and somewhat easier to refute; but it's an interesting effect. Guest: It does. What I am actually very grateful to the Internet for is plugging me in to a network of criticism. Which was possibly always there, but maybe a bit invisible. So, you've got scientists talking to each other, criticizing pieces that appear in the media, criticizing other papers, criticizing each other. Now I think that this is spectacularly healthy. Because from my point of view it makes me a better journalist. I know more about--ever since I joined Twitter, ever since I became part of the blogosphere, I know more about the debates going on in the field. I know more about the methodological flaws to look out for. I know more sources. I know more people who are really solid on all these things, who I can contact and ask for a second opinion. And you learn so much about all this stuff, that I think is really essential and really interesting. And that may have been completely invisible before. But I think it's out there now for people who want to see it. The question of course is whether critiques actually reach as far as original mistakes. And I think it's pretty clear that they actually don't. So that to me is the big thing. How do you make sure that the truth actually reaches as far as the dodgy stats that you were talking about. Russ: Just wanted to check. Not about confirming my bias. I just re-googled "nfl expectancy", "nfl life expectancy", and it pulls up the same studies that show it's 58, or 53-59. I think the real problem, of course, is it's not the Internet. The problem is you and I are the problem. Most of us don't really care about the truth; which is human. We are interested in grinding an axe, feeling good about ourselves, feeling angry about something. So, if you want to write an article about the NFL or if you hate football or if you are worried about football players, that little fact, whether it's 58, isn't really the point. The point is we need to help these people. And that's just a weapon. I grabbed it; it was laying there; I picked it up, that statistic. And if told me it wasn't true, I'd feel bad about it; I'd find a different weapon, which was lying around elsewhere. So, I think the real problem with the delay between correction and the truth, or, excuse me, the correction and the lie, is that people don't care. And I don't blame 'em. I understand that. Guest: And I can understand why the average Joe doesn't care. I find it absolutely galling when people who are supposedly journalists don't. Because that is--you know, that's the mission. That's the job. Professionally we are people who are meant to care about that sort of stuff. And what really irks me is that a lot of the mistakes that get made are really tedious mistakes. Things like--all the things we talked about: reusing stats without checking where they came from, not looking at where the vested interests might be, not calling people up. Russ: Not interviewing a critic. Guest: Not interviewing a critic, right; not presenting the context for a study. All these things have been talked about again and again and again. If you look at any common piece on science journalism, that's what people talk about. Those are the obvious mistakes. And I think they are actually, like I said, science is becoming very interdisciplinary and you don't actually know who the right experts are. Russ: True. Guest: And you don't know the methodological things you should be looking for. There are plenty of really interesting ways to screw up being a science journalist. There are many, many ways of making mistakes that I think are, maybe not more excusable but certainly, I don't know, more advanced, more interesting; not the really boring things that people keep on saying all the time, things that could be avoided with a bare minimum of effort.
57:36Russ: Well, I'm going to challenge you a little bit on those points. I agree with 99% of what you said. The 1% though I want to focus on, because I think it's interesting: I think most economists, most academics, most medical researchers, most physicists see themselves as searchers for truth. But they don't act that way. That's how we see ourselves. That's a form of self-deception. There's an element of truth to it; it's not a lie. But we of course are affected by hundreds of other things--our incentives to be published, to get tenure, to be famous, to be lauded, to be respected. And these things clash, obviously, and I think it's true of journalists, too. I've taught a lot of journalists; and they say what you just said with such fervor: Our job is to seek the truth. That is your job, in some way, I guess. It's certainly the way many journalists see themselves. But of course it's not exactly how they act. And that's because, as an economist I see their incentives act truthfully are not always so strong. So, to take an obvious example, in political coverage, if you tell a journalist that he's got a political bias, I think it's like telling him he's a child molester. It's one of the most horrifying things you can accuse a journalist of. And their response isn't: No, I'm not biased; that would be a violation of my ethical code. They get angry and they yell at you. And of course, it's very important and it took me a while but I understand it: journalists want to feel they are searchers for truth, just like we economists and academics want to feel we are. But we need to sometimes step back and realize the incentives working on us consciously and subconsciously are sometimes not so much pushing us in that direction. And we ought to be maybe just aware of the fact that what we say about ourselves are not actually the same. Guest: True. And I think this ties into the stuff that Jay Rosen and others have been about for ages, that rather than having this rather false veneer of impartiality or objectivity, journalists should embrace the idea of more transparency and fairness. And I think that's absolutely true. I think you are kidding yourself if you think you are entirely apolitical or impartial or objective. But that doesn't stop you from being as fair as possible or of doing all the necessary bits of the job that allow you to produce truthful pieces of work. I think as long as one acknowledges one's biases, that can be more powerful than sort of striving to achieve some sort of weird neutrality, which probably will never actually happen. Russ: Yeah. I've been trying that in economics. It's not very popular. Guest: Yeah, right. Russ: My fellow economists get very angry when they suggest I am biased, as I am. They say: Well, you are biased; okay, you've admitted it. But I am not like you. I am an impartial seeker of truth; and you are despicable. I'm carrying on. I'm going to keep claiming, admitting my bias rather than trying to hide it. Guest: Cultural norms bounce off, shield neutrality. My bubble of perfect objectivity, no bias can enter. Russ: It's bizarre. But good for them, I say. I have to go buy one of those, find one of those bubbles or shields because obviously mine isn't working. I got a defective one.
1:01:10Russ: Let me ask you a couple of things, and you can talk about either one or both. One is: What do you think might change the incentives if anything that journalists face? Scientists, I think it's very hard to change those incentives. There is some pressure on journals to offer replication, to publish data at least. It's a start. But actually, I think people like you and others who are using the internet to remind people of the possible unreliability of results puts social pressure on folks to be more open and honest. But in journalism, I'd be interested in your thoughts. And then I'd be interested in what do you think would be the interesting issues in science in the next few years that you are thinking about studying. Two disparate questions. Guest: Well, in journalism, how can we change the incentives? So, I think broadly speaking you could think about it in two ways. You could praise the good stuff, and you could denigrate the bad stuff. I think both of those are happening to some extent. There's certainly a lot of bashing the bad stuff going on, increasingly more so. That's quite good, actually. I don't particularly want my stuff bashed more than any other journalist; but I think it keeps us on our toes, and that's good. It makes us better at what we do, if we get told why we screw up. But I think certainly in some circles, there is a bit of defensiveness against that. I think that has to go. And I think there's a lot of almost clique-y mutual defensiveness, a lot of journalists forming a kind of protecting ring around each other. And I think we should actually as a profession probably be more willing to point out each others' failings. And I see that happening on things like Twitter. There are watchdog sites like the Knight Science Journalism Tracker. I think they do a very good job, and that sort of thing is very good. I think the danger is when you stop thinking of it is what we are trying to do and think of it as a day-to-day grind, when you are a tradesman rather than like a professional. And I think keeping a focus on the latter is probably quite important. But I think really, ultimately, it comes down to readers. Russ: Yeah, good point. Guest: I think the biggest thing that would change what's going on would be if people became more discerning readers--if they paid more attention to material that actually went that extra mile and gave more context and did more of a job of verifying things; if they shout out, or if they just ignored the stuff that just rehashes press releases without so much as a quizzical eyebrow raise. That's the thing that is going to change stuff. Otherwise we get the media that we deserve. And if we are not selective and if we are not demanding enough, then we end up with a poor quality of stuff. And I think it's not just the general public, whoever they may be, who are guilty of this. I think scientists and science writers a nd science communicators are as well, because of that natural tendency we have to love science and want to see it being made fun and accessible. I think that one of the most dangerous ideas in the science communication industry is this idea that you can only make--the only way of promoting science to people is by kind of presenting it as this exciting, happy clappy--you sit around the campfire singing Kumbaya and talking about how wonderful science is. Russ: There's a name for that: It's called a religion. I'm a big fan of religion, but I think we should keep science out of that cultural norm. Guest: I'm glad you've gone for totally noncontroversial subjects in the broadcast. I mean, we can just talk about that and get that over with in a minute. But yeah, I think it's equally interesting to show where scientists disagree, and where studies may be flawed and what else needs to be done. I think that's an incredibly valid and actually fascinating way of getting people interested in it. But anyway, I'm rambling a bit. So you asked about-- Russ: What excites you? What's coming science? I want to close on a cheerful note. So, I want you to talk about what excites you in science these days and what sorts of issues you are keeping an eye on. Guest: Okay. The things that really excite me kind of creep up on me. They are never really the types of things I expect. So, I can give some obvious answers. There have been a lot of studies recently showing breakthroughs--and I use the term very carefully, but I think it's fair to call them breakthroughs--that almost belong in the realm of science fiction. So, you know, paralyze people, gaining control of robotic limbs, showing that you can regrow things like optic nerves, you can convert scar tissue and damaged heart muscle to beating heart muscle. This is incredible stuff. And it may not be there yet in terms of widespread human use. Lots of research, yadda yadda. But it's a first step, and it's very exciting; stuff we couldn't do before. But then a lot of the things I write about on my blog are nothing to do with that. And actually, if I look at my traffic, the posts that get the most hits are never really the ones involving the big breakthroughs that are going to save loads of lives and helps loads of people. They are usually the odd bit of science. Russ: The quirky stuff. Guest: The quirky stuff, that just makes you kind of go Wow at the natural world. So, an example. I blogged last week about whales, showing that the biggest whales, the norwhales, blue whales, humpbacks--they have a sense organ at the front of their lower jaw that no one had known about, and it helps them coordinate the movements of their jawbones and the muscles in their mouths. When they open their mouths to swallow big gulps of food. A finwhale can swallow as much water as another fin whale--so swallow another whale. All the big movements you need to do that--it's got this thing that about the size of a volleyball with nerves and blood vessels in the front of its mouth that takes in information from the bones and the muscles and the cartilage and helps it with how it opens its mouth. This is amazing. These creatures are familiar; they have been the subject of endless natural history programs; their skeletons are on display. People who have hunted them for decades; and we have only just discovered that they have this thing the size of a football in the front of their heads that we just didn't know about. And to me that is really exciting. That's the type of stuff I live for: new and exciting things that show how much we have left to understand about the stuff around us. Russ: And how was that discovered? Guest: Um, it was, well, it's an interesting story. It was a bunch of scientists who were trailing a Japanese whaling vessel. And you know, you've got all these remains that are then discarded. And they used that opportunity, they were going to conduct a very thorough study of the musculature and the anatomy of the jaw, which no one really had done before. And they found this thing. And they went: Well, hang on, that's a bit strange. Russ: It's cool. Guest: There's stuff like that. The odd discoveries that take you by surprise. In a similar vein at the end of last year people had discovered that elephants have a sixth toe in their feet that kind of acts like a heel. And so an elephant foot is like a bit platform shoe. They walk on tiptoes, but they've got this gigantic fat pad which the heel sits on which cushions their footfalls. I don't know if you've actually seen an elephant in the wild. They are like ninjas. You can't hear them. Even though they are massive and you expect them to quake the earth when they walk, they are actually completely silent. Unless they step on a twig, you'd never know one was there. And it turns out there is a little toe which juts into this footpad, fat pad, and gives it extra support. Again the thing we didn't know about despite the fact that these animals are incredibly well studied and have been dissected many times over. It's completely new. And I find that sort of thing endless fascinating. Russ: So do I.

COMMENTS (21 to date)
Greg G writes:

So a powerful priming effect is confirmed.....just not the one the researchers were originally looking for. They are primed to find whatever they hope to find. This explains a lot. Thanks for another excellent podcast.

Justin P writes:

Bias is very powerful.

"People who are actually doing the work necessary to find if this stuff is true, they are not finding it; and they are thinking: Well, it's probably my fault."

I think the problem is summed up right there. The researchers have been "primed" or brainwashed to not put it nicely, to assume that it must be true unless...

This is exactly wrong when it comes to science. True science assumes it's false...but the experiment shows it holds true for now.

I say brainwashed because the school system has, over the years, conditioned students to believe what the experts say, no matter what. The textbook is always 100% true! The teacher (nevermind having maybe 10-15 credit hours in the subject) is an expert and can never be questioned...etc.

We aren't teaching kids to think critically, just parrot what they are told and memorize "facts."

Is it any surprise that when they get older and go into "research" they do the exact same thing?

The problem is how we teach science to kids...we are doing it wrong.

Tim D writes:

Great show, Russ.

You should check out this Radiolab piece featuring Jonathan Schooler, a UCSB psychologist who tried to replicate the finding that launched his career and encountered a great deal of difficulty:

http://www.radiolab.org/people/jonathan-schooler/

RonB writes:

I long ago came to the conclusion that its a "poor researcher that can't get the results he wants". That is not to say its intentional, it's rarely a case of intentional manipulation.

Its not an easy problem to solve, but full disclosure about the process(es) used and a release of all data as a requirement for publication would help.

As Justin points out there in too much of a belief in experts by the general public, as well as by experts themselves.
At least in the social sciences if not the physical science definitive ideas or concepts should be based on the weight of the evidence of multiple studies. Humans have too much variability to rely on one or even a handful of studies.

For my money any study that is based on grad students is suspect.

rhhardin writes:

No, the guy who, the person who ushers the volunteers into the experimental room gives them a sort of envelope with all the test materials in it. If that person thinks that the experiment is going to show that priming reduces walking speed, then they do walk more slowly.

The classic Clever Hans effect was that the horse could only get the right answer if the person asking knew the right answer.

The horse couldn't add but he could read body language.

Jim Feehely writes:

Hi Russ,

I am with Ed Yong on the unjustified reverence for 'research' in the social sciences, particularly psychology, sociology, economics and other fields of study of social and community systems.

I have long held the view that the foundations of these 'sciences' that arose, particularly in the 19th century, is the assumption that with enough research and observation, we can come to understand the fundamentals of these enormously complex systems.

A truthful person, now, must admit that that all the observation and research has actually shown that the individual human, society as a whole and even modules of society and economic relations are so profoundly complex that it is probable that we will never properly understand the fundamentals.

However, the research still seems to be based on the assumption that it is a step in inevitable illumination of the fundamentals of great and complex systems. And worse, the increased and increasing segmental specialisation actually distances this research from its context; i.e. the whole complexity of social systems. This, of course, is the expert problem that the more enlightened thinkers readily identify.

Conversely, economics seems to eschew proper behavioural research that surely must contain the essence of understanding economic activity. Instead, economics seeks, in the main, to measure the data it assumes is emitted from the system, then creates models to explain this data and then assumes behaviours of individuals and social groups as an extrapolation of the macro data distorted through invented models. On an individual level, economics seems to assume that self-interest is the exclusive engine of economic behaviour because it finds altruism and collaboration too difficult to either price or account for in its models. The origins of economics was very concerned with explaining the behaviours of individuals. But in the 20th century economics seemed to abandon that pursuit to concentrate on studying the flows of rivers of money and then working back from conclusions drawn about those flows.

Is it not time for a return to generalist education, an education that admits the impenetrable complexity of our world, compared to the relatively recent obsession with atomised speciality and crude vocationalism?

Cheers,
Jim Feehely.

RandomReal[] writes:

Great podcast.

This conversation reminded me of a wonderful term that physicists have come up with: The Look-Elsewhere Effect. Essentially this is a statistical correction that takes into account how wide of a search you perform looking for rare events. The data analysis that is being performed in the search for the Higgs boson is one example. Since a priori you don't know exactly what the mass of it will be, you have to look at many different masses. Since other known processes can mimic the signal, they are trying to detect an excess of events over and above the expected background. If you only computed the statistics for an excess at a specific mass, you can be fooled. The statistical adjustment with the look-elsewhere effect reduces the statistical significance of your findings depending upon how many different places that you looked.

Russ brought this up when he said

And so, by pure randomness you are going to get significant results if you run 50 or 100 studies.
It's nice to have a name attached to it.
John Dyer writes:

Russ, another excellent podcast! I recently listened to Dr. Moira Gunn interview Howard Rheingold where Howard remarks that the internet is not an "answer machine" but a "clue machine". That statement made me pause and think, it's so true.

It's about 8 minutes into the conversation:
http://itc.conversationsnetwork.org/shows/detail5292.html

[html fixed--Econlib Ed.]

NormD writes:

Great podcast.

I am surprised that you did not discuss the interaction of science and politics.

People with opinions always seek support for their opinions, ways to show they are correct and their opponents are wrong.

Science is a very powerful tool in this regard which leads to two things:

1. People with strong opinions become scientists or science writers so they can foist their opinions on others as science and cast their opponents as anti-science.

2. People with strong opinions use their power to fund/support science that arrives at conclusions that support their opinions and defund/suppress science that conflicts with their opinions.

I subscribed to many science magazines over the past 30 years, read many science books and still listen to many science podcasts, but slowly I noticed the content became more political, less interesting, more full of hype and more self important.

It just got a lot less interesting.

There seems to be less room for oddballs that study stuff that might lead to surprising conclusions.

Its like a giant bureaucracy.

Seth writes:

Nice podcast.

I thought Taubes discovered that many of the studies that have been used to support the fat/heart disease connection either did not actually support that connection (instead concluding that it will only be a matter of time before some study does)or had potential problems in design that were not addressed. Am I wrong about that?

Is it typical for researchers to publish a) what attempts they have made to disprove their own results or b) what other things could explain their results? If not, this seems like an obvious step that could be added to help prevent the reporting of random outliers as actual findings.

Rufus writes:

I thought this was a great podcast on a topic that deserves ongoing attention. Another web site to keep up with flawed science is Retraction Watch

http://retractionwatch.wordpress.com/

Another trend that will hopefully grow is the integration of various scientific disciplines when performing research. If my Electrical Engineering degree taught me anything, it is that some people are much more gifted in math than others (me not included). After some exposure to skeptics like Matt Ridley and Freeman Dyson (other great Econtalk episodes) I have become increasingly skeptical of climate change / global warming because I simply don't think there are that many true geniuses who can understand atmospheric modeling, weather, chemistry, biology, and complex mathematics all at once. Just because you can slam some numbers into a statistical program and produce a graph does not mean you understand how the math works and the inherent limitations of these models. Luckily, the Internet seems full of people who have both the expertise and time to voice their concerns.

While this may draw a few arrows of criticism, another example is in The Hockey Stick Illusion by Andrew Montford. That book illustrates that there are serious problems that need to be solved in the way science and scientific journalism is operating today. Some may disagree with the message of that book, but it does a great job of detailing "lost data", unpublished data, unclear rationale for not using available data, questionable mathematics, and limits of the peer review process.

I do think the Internet and blogs are going to make a huge difference in the life of academics who have been sheltered in professional journals that rarely get read outside of their profession. With access to these studies, bloggers often do a great job critiquing both the media announcements and the studies themselves. Overall, that is good for all of us.

Brian writes:

Guest: And I can understand why the average Joe doesn't care. I find it absolutely galling when people who are supposedly journalists don't. Because that is--you know, that's the mission. That's the job. Professionally we are people who are meant to care about that sort of stuff. And what really irks me is that a lot of the mistakes that get made are really tedious mistakes. Things like--all the things we talked about: reusing stats without checking where they came from, not looking at where the vested interests might be, not calling people up. Russ: Not interviewing a critic.

Russ: Well, I'm going to challenge you a little bit on those points. I agree with 99% of what you said. The 1% though I want to focus on, because I think it's interesting: I think most economists, most academics, most medical researchers, most physicists see themselves as searchers for truth. But they don't act that way. That's how we see ourselves. That's a form of self-deception.

Russ, as I listened to you interview Mr. Yong, I could not help recall your interview of Brian Deer. When you did the Deer interview, it was the bias confirmation for me that I knew had to be out there, and I was so excited. However, as I always do, I went searching for the response from the other side of the debate. What I found cast a great deal of doubt over the motivation of Brian Deer and the substance of his book and interview. I sent you information on this hoping you would present the other side, but as yet you have not.

Russ, I think your treatment of the Deer/Wakefield/vaccination story is directly applicable to the topic of this interview with Mr. Yong. It seems to me, Russ, it may have been an example of you promoting a story because it confirmed your bias about research, as arcticulated in this interview. I challenge you to take the time to review the other side of the Deer-Wakefield matter. Dr. Wakefield has published a book titled "Callous Disregard." Recently, a well-known U.S. government whistleblower, Dr David Lewis, weighed in on the subject. I encourage you to listen to this interview

David Lewis interview

and/or explore the information available here

Lewis investigation

Of course, there is a great deal more info available.

As always, I very much enjoyed this interview, and I appreciate the very high quality of the EconTalk podcasts and interviews. I continue to recommend EconTalk and EconLib to everyone I encounter who has an interest in economics, etc. Thanks.

Rufus writes:

Brian,

While I did not read the entire Lewis Investigation paper, which apparently gives support to the link between vaccines and autism.

"Last September, Columbia University
published a major study supporting the link Dr. Wakefield established between autism and
enterocolitis." (however, not vaccines as we'll see below)

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3174969/

Here 2 quotes of the references to that paper.

Reference 27: There was no evidence that onset of autistic symptoms or of regression was related to measles-mumps-rubella vaccination.

Reference 86: CONCLUSIONS/SIGNIFICANCE: This study provides strong evidence against association of autism with persistent MV RNA in the GI tract or MMR exposure.

I'm not sure they support the argument you're making. I may actually read the whole report to see how or if the 2 references that apparently don't show a link between vaccines and autism provide support for the argument Lewis is apparently making.

Honestly, I think there are some cases where these authors cite papers without having read them fully and then get by because the peer reviewers don't have time to read all 86+ supporting papers.

Andrew Britton writes:

Russ,

Although I agree with much of what you and your guest discussed, I do have worries that it could end up exacerbating the very problem it intends to address.

It is certainly good for researches and scientists in a given field to be extremely careful and skeptical of results, confirmation bias, etc. For the lay public however I think there is a great danger of "lazy skepticism" which can counterintuitively lead to greater bias, rather than less.

For example the person that gets the message from the podcast that none of the science is valid and so they can just go on believing whatever they want comfortable in the knowledge that they can never be disproven (obviously research opposing my beliefs is just a sham! No need to investigate further).

Along similar lines is the conspiracy angle, i.e. an over willingness to believe any claim that confirms your belief that the system is corrupt. The bankers, pharmaceutical companies, government, etc. Is out to deceive us and so any crack pot theory that comes along confirming this must be true.

Another point I am interested to get an opinion on is whether you think this is a good example of a market failure? It was pointed out somewhat, but not elaborated upon, that there are certain incentives which seem to be primarily market driven contributing to these problems. I.e. The desire to sell more magazines, papers, etc. the desire for journals to print new exciting discoveries, the desire to make a successful career, etc. These all seem to be problems intrinsic to the market aspects of science. Thoughts?

DougT writes:

Has anyone here read Dorothy Sayers' essay, "The Lost Tools of Learning?" It's here: http://www.gbt.org/text/sayers.html. This was all predicted so many years ago. And yet confirmation bias and the interactions of people with their (infinitely) fallible human nature is always fascinating.

I think the *best* exploration of confirmation bias was penned by Sophocles 2500 years ago with Oedipus Rex. The poor man didn't want to believe that he'd killed his father and married his mother. But then, with the Prophet's oracle, he can't avoid the truth.

The amazing thing is that science actually makes progress, even with all the failings and foibles of scientists.

Mort Dubois writes:

@ DougT:

"The amazing thing is that science actually makes progress, even with all the failings and foibles of scientists."

I humbly submit that it's not scientists who make progress, but engineers. They are the ones who take the theory and implement it. False theories get washed out at this step. This is why the "science" of some disciplines seems so much more rigorous than others. It's exceedingly difficult to implement measurable systems in the softer sciences, while something like physics allows for actual, testable, manifestations of the theories. This is the reason we've made such great technological strides as a species - the scientific method works great for that kind of thing. And not such great strides in the social and economic sciences, as there it's almost impossible to set up controls and to isolate the effect of inputs in complex systems.

Joe Kash writes:

Russ,
Do you have a reference concerning this study:

"A recent study found--and I have not looked at this carefully--of course you have to be skeptical of all studies, even those that are skeptical of studies. But this was a study that identified 53 landmark studies in cancer research and allegedly 47 of these 53--47!--could not be replicated."

I am an oncologist and I would like to review this.

thanks,

Joe Kash

Russ Roberts writes:

Joe Kash,

I meant to put up a link and forgot.

Unfortunately, that study (which generated a great deal of news coverage, unsurprisingly) is a bit hard to get at. The authors of the study summarized it in Nature which is behind a paywall:

And the study itself, because of confidentiality, did not identify which studies failed the replication test. Which makes this critique of replicability somewhat unreplicable....

Jed Trott writes:

I enjoyed this podcast quite a bit. Thanks to the interviewer and interviewee. I think that part of the problem explored here comes from a fundamental misunderstanding about the nature of the human endeavour. Scientists, experts, and planners are often given to thinking that humanity progresses in some sort of linear path. They think a steady accumulation of knowledge leads to tangible advances. If one looks at the history of humanity, whether in medicine, physics or technology, this is rarely the case. Progress is driven by happy accidents usually made by smart and knowledgeable people. The intelligence and knowledge help these people to stumble on breakthroughs but they usually don't set out to solve the problem that they end up solving. Unfortunately this is something that for many reasons the layman and the expert have trouble believing. It scares the layman and deflates the expert. If we did believe it I think it would help reduce many of the expert problems we have.

John Bell writes:

I was listening to this podcast and it reminded me of a book I'm reading - "Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society" by Jim Manzi. So I was pleasantly surprised to see that you interviewed him just two weeks later.

There's an interesting description he provides of how a theory can have wide acceptance, even when there are observations that appear to falsify it. He gave an example of the perturbations in the orbit of the planet Uranus that did not conform to Newtonian mechanics. Since so many other things did conform to Newtonian mechanics, these results were "set aside" or treated as anomalous. Eventually, the presence of another planet, Neptune, was predicted (as a cause of the orbital trajectory discrepancies) and verified, and Newtonian mechanics "marched on".

So far, so good. However, sometimes something comes along that overthrows the entire paradigm... and here is where he makes an interesting statement:

Empirically, when high-level paradigms come into direct competition (e.g., Copernican versus pre-Copernican astronomy, or relativity versus Newtonian mechanics), almost nobody switches camps, unless it’s very early in his or her career. What happens is that one paradigm stops getting new recruits, And over time the stalwarts of that paradigm retire or die.

When there is an existing paradigm or worldview for "how things work", (especially in an area that is not a science and not an art but a discipline like Economics or Psychology) I can see how easy it can be for experimenters to get wrapped around the axle in the design, execution, and interpretation of the results of their experiments.
People's viewpoints do tend to ossify.

AC writes:

"Cargo Cult Science" by Richard Feynman

http://www.youtube.com/watch?v=yvfAtIJbatg

Comments for this podcast episode have been closed
Return to top