Noah Smith on Whether Economics is a Science
Dec 28 2015

nsmithimage.jpg Noah Smith of Stony Brook University and writer at Bloomberg View talks with EconTalk host Russ Roberts about whether economics is a science in some sense of that word. How reliable are experiments in economics? What about the statistical analysis that underlies much of the empirical work in modern economics? Additional topics include the reliability of the empirical analysis of the minimum wage, the state of macroeconomics, and the role of prejudice or prior beliefs in the interpretation of data and evidence.

John Ioannidis on Statistical Significance, Economics, and Replication
John Ioannidis of Stanford University talks with EconTalk host Russ Roberts about his research on the reliability of published research findings. They discuss Ioannidis's recent study on bias in economics research, meta-analysis, the challenge of small sample analysis, and the...
Russ Roberts on Wealth, Growth, and Economics as a Science
EconTalk host Russ Roberts talks with reporter Robert Pollie about the basics of wealth and growth. What happens when the stock market goes down or the price of housing? When wealth goes down, where does the wealth go? How do...
Explore audio transcript, further reading that will help you delve deeper into this week’s episode, and vigorous conversations in the form of our comments section below.


Dec 28 2015 at 10:47am

But even the Levitt school choice study Noah cites is not the definitive paper on that topic. How many papers have been written since then that show positive benefits from school choice? We should be skeptical of drawing conclusions from one study and should instead examine the overall literature.

Great talk overall.

Dec 28 2015 at 11:05am

[Comment removed pending confirmation of email address. Email the to request restoring this comment. A valid email address is required to post comments on EconLog and EconTalk.–Econlib Ed.]

Dec 28 2015 at 11:13am

Economics is a science but a science that can prove only a few things beyond reasonable doubt.

BTW: I used to think vouchers would significantly improve outcomes like skills and knowledge, the evidence convinced me otherwise. I still support privatization of schools because I think they will be cheaper and better for parents and students. I no longer think that they will make students know more or have better skills.

Greg G
Dec 28 2015 at 12:38pm

I thought this was an excellent discussion in the best tradition of EconTalk so thanks to guest and host for that.

I do want to challenge Russ on one point though. Russ, when describing your biases, you put a lot of emphasis on the fact that you are “very prone to believe that people respond to incentives” which you went on to describe as a “methodological ideology.”

How does this help us to identify you? It strongly implies that you are opposed to some group of people who DON”T think that people respond to incentives. Who, if anyone, do you think believes people don’t respond to incentives? And if everyone believes they do, how does this help to identify your biases?

Often word meanings are contested in EconTalk discussions. “Socialism, free markets, central planning, liberty” etc. are all words of this type. “Incentive” is not. Doesn’t everyone describe “incentive” more or less as “a thing that people respond to”?

Noah did challenge you on this point and you said you would come back to it. Perhaps you came back to it in a way that was too subtle for me to notice or perhaps I am misunderstanding your point from the beginning. Please clarify why you think that us knowing that you think incentives matter helps us to identify your biases.

Dec 28 2015 at 2:00pm

All discussions of economics as a “science”, including that of Russ and Noah, suffer from the same definitional problem.

I would argue that there is a general consensus that an intellectual discipline qualifies as a “science” if it systematically organizes data, information and knowledge such that testable predictions can be made. Add to this Popper’s notion of falsifiability, and I believe most people would find this generally acceptable.

And therefore many economists assert that, indeed, economics is a “science” because it seems to meet the above criteria in their eyes, often citing a handful of well-known examples as evidence.

But in turn many natural scientists scratch their heads, as economic techniques and methodologies bear little resemblance to their own activities.

This talking past one another is simply a result of definitional vagueness. If we add more precision, and propose that an intellectual discipline qualifies as a “science” when the set of predictions, tests, and attempted falsifications has survived some threshold, then it becomes instantly apparent how economics is not a science, and in fact nothing remotely like the natural sciences.

We can argue about what that threshold should be (ten cases?, a hundred?, a thousand?), but it is inarguable that the difference between the natural sciences and economics is measured by many orders of magnitude. As an example, when we assert that a falling object accelerates at 32 feet per second per second, note that every instant a plane successfully flies, or a projectile follows a predicted path (say a fly ball in baseball), then the assertion survives the falsifiability test. This particular natural science assertion has survived falsifiability trillions of times (actually an uncountable amount).

Speaking in orders of magnitudes then, let’s set the threshold for “science” as low as we can imagine. Certainly no one would say one survival of falsifiability should be enough, so let’s set the number at ten. By the way, I think almost no one would agree that this is sufficiently high, but that very fact illustrates the point. Can someone point to the economic study or experiment that has been confirmed, or more accurately not falsified, at least ten times? That is, what economic study or experiment has survived falsifiability without having to bring in after-the fact adjustments, similar to Ptolemaic epicycles, or other post-facto equivocations, in a way that modern natural sciences rarely resort to? And even if we could all agree on some such studies existing (I am not aware of any, but grant that I certainly have a limited knowledge base), would anybody reasonably compare such a study to the millions and billions and trillions of tests met by theories in the natural sciences? No, these disciplines occupy absolutely different realms.

Rigorous economic thinking, and coherent economic theory can indeed be very informative and valuable for interpreting our world. (Russ, keep up the good work!). But alas, in economic analysis ceteris paribus virtually never holds, thereby preventing conclusions and predictions that are comparable to those of the natural sciences.

Economics is not a science.

Sash A
Dec 28 2015 at 7:11pm

Thanks for the discussion Russ and Noah!

I would like to add a point to Noah’s objection to the significance of single studies providing counterintuitive results, such as Nosek’s Reproducibility Project. I agree that it is the accumulation of knowledge that leads us to rethink our priors, however, one should acknowledge the insights those studies often provide on the very flaws in some methods.

An amazing example in the making is Alwyn Young’s working paper on the treatment effect in experimental analyses. He is applying R.A. Fisher’s randomisation statistical inference techniques on 2000 regressions and concludes that only 25-50% of experimental papers are able to reject the null of no treatment effect (Here is the link to the working paper).

I think Alwyn’s work shows that we should rethink some methods we currently perceive as standard.

[html fixed–Econlib Ed.]

Brian Mason
Dec 29 2015 at 6:03am

School choice is a farce. You can choose a slightly better class room experience with bureaucrat mandated text books … You may not choose apprenticeship for example. “Tiny deviation which can only possibly result in small differences” is an honest term.

The minimum wage debate is a good example of economics not being a science: a search for reliable non-obvious rules. The university degree is in trouble: discounted completely at Google for example. It looks better and better to (re)start one’s career with low wage work vs negative wage work aka tuition. So some academics concoct studies with ever more fanciful math to make minimum wage mandates look ok. If people are prohibited by law from (re)starting a career with low wage work then more will choose negative wage work and support fat lazy academics.

Thomas K
Dec 29 2015 at 8:08am

Hey Russ,

great talk.

Although not triggered by a specific well identified empirical paper but rather by his experience and the 2007 financial crisis, Alan Greenspand personally admits that his ideology/model of the world had flaws. For those who can’t wait, skip to minute 4:58 🙂

I found this to be very astonishing and honest acknowledgment.

Karol P
Dec 29 2015 at 12:36pm

What is Science then? Yes, we actual need the definition we can agree on to answer this weeks question.
If share the view of Karl Popper( and apparently Russ’s), that it is impossible to ensure that an Economic theory is true than the following definition seems reasonable:

Science is a rigorous attempt on falsifying reasonable set of Hypothesis.

Growth of knowledge is is attained trough creation of new hypotheses and falsifying already existing Theories as new evidence arise.

I think Ideological priors stem from diversity of viewpoints and we should not get rid of them. They allow us to see the world in different light and confront our viewpoints for the sake of future generations. They motivate us to question the other side arguments thus contributing to the knowledge held by the society.

Dec 29 2015 at 2:23pm

I’m a novice economist but a trained scientist (chemistry) at the PhD level, so I’m sympathetic to both points of view about defining economics as a “science.” However, just because we can agree on the mass of an electron doesn’t remove the politics, bias and special interests from the field of what we might call “hard” science. It has always existed and I suppose it’s a consequence of our fallen human nature… 🙂

I think Russ got into a quagmire with Dr. Smith because his experiential claim that there were “no” converts to the opposing economic side, was easy to knock down. On the other hand, I found Dr. Smith’s rebuttals fairly superficial and– not to be unkind– childish, being very pleased with himself to offer up only the barest of counter-examples and going no further. The irony of course is that Dr. Smith wasn’t willing to give up his views either.

I think Russ is right in his broader point – the two big diametrically opposed views of economics are both full of acolytes that will never be converted, no matter how many examples they see of the failure of their dogma. That’s a significant strike against the “science” of economics.

Russ Roberts
Dec 29 2015 at 4:02pm

Thomas K,

I have a different take on Greenspan’s “conversion.”

Malcolm C. Harris, Sr.
Dec 29 2015 at 4:08pm


Fun discussion.

The NBER does include a cyclical peak on February, 1945 with a trough 8 months later. Germany surrendered May 8th and Japan August 15th (formally on September 2nd aboard the U.S.S. Missouri.)

The unemployment rate was 1.1% at the NBER cyclical peak. The highest it rose to in 1946 was 4.26%.

Looking at key series from their peaks to troughs, industrial production declined 35.5% from August, 1944 through February 1946. Payroll employment dropped by 3.3 million jobs from November, 1943 through September, 1945. Almost two million of that drop was in September, 1945, more or less VJ Day.

The 37 month expansion that followed ended in November, 1948. Whether the 1945 cyclical episode is called a recession or a peacetime adjustment is up to the observer. My memory of earlier discussions of business cycle history either ignored it or refereed to it as a wartime transition or demobilization.

The cyclical trough (October, 1945) corresponds to the first post war month. I do not remember the textbooks of my youth (Keynesian all) dwelling on either the 1945 contraction nor the 1945-48 expansion. They seemed to treat this as wartime distortions rather than a regular business cycle.

One excuse for their curious silence is that quarterly GDP data start in 1947. Still the widespread prediction of a return to the Great Depression by the secular stagnationists was not mentioned in polite company.

Aside: I enjoyed your talk in Wichita a couple of years ago.


Malcolm C. Harris, Sr.
Friends University

Steve Hardy
Dec 29 2015 at 10:00pm

How can you have a study of school choice when there is very little to choose from in an industry that is monopolized by the government (90% of students) and the only other choice is a small number of mostly not for profit private schools. Where is the competition and profit motive needed to create an innovated, dynamic free market? Only when the majority of parents are able to shop for their children’s education will we see real progress in K through 12 education

Russ Roberts
Dec 30 2015 at 12:39am

Malcolm C. Harris, Sr,

The recession in 1945 started before the war ended but the important point is that the Keynesians of the day felt compelled to explain the lack of a recession when wartime spending plummeted. We’ve linked to some of that information in the links above. Try this one for some of the post-war Keynesian angst

Malcolm C. Harris, Sr.
Dec 30 2015 at 6:15am

Thanks for the references.

My interest was primarily from a business cycle point of view: I am writing a history of the business cycle. I expanded my comments appropriately.

This whole incident is one I had not even paid attention to before.


Michael Munger
Dec 30 2015 at 9:51am

Just saw an interesting article posted at NBER, and thought I’d share. The basic point is that physicists do NOT read Aristotle, but moral philosophers do. Science advances through funerals. But then what is economics? We can still profitably read Adam Smith…

Does Science Advance One Funeral at a Time?

Pierre Azoulay, Christian Fons-Rosen & Joshua Graff Zivin
NBER Working Paper, December 2015

We study the extent to which eminent scientists shape the vitality of their fields by examining entry rates into the fields of 452 academic life scientists who pass away while at the peak of their scientific abilities. Key to our analyses is a novel way to delineate boundaries around scientific fields by appealing solely to intellectual linkages between scientists and their publications, rather than collaboration or co-citation patterns. Consistent with previous research, the flow of articles by collaborators into affected fields decreases precipitously after the death of a star scientist (relative to control fields). In contrast, we find that the flow of articles by non-collaborators increases by 8% on average. These additional contributions are disproportionately likely to be highly cited. They are also more likely to be authored by scientists who were not previously active in the deceased superstar’s field. Overall, these results suggest that outsiders are reluctant to challenge leadership within a field when the star is alive and that a number of barriers may constrain entry even after she is gone. Intellectual, social, and resource barriers all impede entry, with outsiders only entering subfields that offer a less hostile landscape for the support and acceptance of “foreign” ideas.

Greg G
Dec 30 2015 at 11:30am

It is a vanishingly rare thing when someone in any field really fully renounces their priors and strikes out in a new direction. The best example of that in the Social “Sciences” that I can think of is Ludwig Wittgenstein.

He became the brightest star in the field of language study by promoting the picture theory of language. This was the idea that language use failed or succeeded to the extent that it matched some objectively correct representation of the world.

Wittgenstein later totally rejected this idea and again became the brightest star in his field by promoting the opposite idea – that language was conventional all the way down. There is no objectively correct language that conventional language has as its foundation.

This is not to say we can’t make mistakes and create a lot of confusion with bad language use. Of course we can. It is to say, there is no better standard for whether or not language was used correctly than whether or not the speaker’s words had the effect that he intended.

I think it is fair to say that this example is so rare that you can regard it as the exception that proves the rule if you like.

It is a lot more useful to think of various methodologies as being more or less scientific on a broad spectrum rather than simply falling into one of two approaches.

Steve Hardy
Dec 30 2015 at 1:51pm

A more recent example of someone reversing their priors is Jerry Taylor who for years at the Cato Institute was a global warming skeptic but now in his new organization, The Nisckanen Center is promoting a carbon tax.

Morgan Dubiel
Dec 30 2015 at 5:35pm

On school choice, there really isn’t any. Even with vouchers government still determines what is or isn’t an “education”. That means that no matter how much “better” a choice is, you’re still limited to following the education narrative dictated by central planners – curriculum, text books and focus. So best practices aren’t practiced if they fall outside the ruling narrative.

So there really isn’t any customization by student, nor is there really any competition driving price down and improving outcomes. In Amnerica if you want to order a meal a la carte you can. It’s illegal to do that in education for the most part.

Lastly, if government schooling is the best then it ought to retain that position with or without a monopoly, no?

Frank Taussig
Dec 30 2015 at 7:00pm

Early in the talk, Noah mentions ” EPSILON BALL ” , can someone please explain that reference to this math illiterate? Intuitively I thought he meant the perceivable universe, the physical world around us as we percieve it being as a receptor suspended in the center of a ball. Am I on the right track at all?

Morgan Dubiel
Dec 30 2015 at 10:18pm

On school choice, there really isn’t any. Even with vouchers government still determines what is or isn’t an “education”. That means that no matter how much “better” a choice is, you’re still limited to following the education narrative dictated by central planners – curriculum, text books and focus. So best practices aren’t practiced if they fall outside the ruling narrative.

So there really isn’t any customization by student, nor is there really any competition driving price down and improving outcomes. In Amnerica if you want to order a meal a la carte you can. It’s illegal to do that in education for the most part.

Lastly, if government schooling is the best then it ought to retain that position with or without a monopoly, no?

D. F. Linton
Jan 1 2016 at 10:05am

Good podcast, but I have one quibble with you Russ.

Your off-repeated proposition is that you have never found an empirical econometric study that caused a real change of mind (my paraphrase, of course). You allowed the topic to shift to whether economists ever change their minds by discarding one theory for another. That’s what Posner did by embracing Keynes. His conversion was not due to some regression-laden report from the CBO or elsewhere, but rather seemed to stem from a preference for a narrative that he felt better fit the circumstances of the day.

Greg G
Jan 1 2016 at 12:32pm

Another example of events actually changing minds was the so called “Minsky Moment” after the 2008 crisis.

Minsky had argued that “stability is destabilizing” because it rewards risky finance. Risky finance then comes to be seen as safer than it really is with those rewards cited as evidence. Risky finance then builds up until the system becomes so fragile a financial panic ensues. The times when investors feel safest are actually the most dangerous.

This has some obvious parallels with the Austrian view that economic boom and bust cycles are driven by unwise credit expansions. To be sure, Minsky was not an Austrian overall and was very heterodox.

The point is, Minsky’s influence was at a low point during “the Great Moderation.” The fact that later events fit so well with his theory of the business cycle really did result in a significant increase in his influence.

Robert Swan
Jan 2 2016 at 4:53pm

An entertaining, but rather futile discussion.

As Mike Munger points out in his comment, even the hard sciences have influential people who, faced with contrary evidence, avert their eyes rather than change their minds. It’s human nature; even scientists have some vestiges of humanity in them, deep down.

But has the “hardness” of a science anything to do with its usefulness. I would submit that physics, generally viewed as the hardest (i.e. most tested) of the sciences, is not directly useful. Most of physics today seems to involve particle accelerators or telescopes or just sitting in a cloistered hall musing and writing.

It is engineering which takes the results of the physicists’ contemplations and tries to apply them in the real world: the world where billiard tables have friction, particles are never zero size and turbulence happens. It has had to blend hard scientific theory with a lot of empirically derived results (e.g. material strengths, flow rates, etc.). Look what it has achieved: dams and bridges, PCs and iPads. Not to mention Large Hadron Colliders. I don’t think engineers generally want to be considered scientists, nor do they (nor should they) feel inferior.

If economics is to have any value, it seems to me that it should aspire to being neither a theoretical science like physics, nor a descriptive one like biology. I think it should have two broad departments: theoretical economics (a branch of mathematics: game theory and the like) and applied economics (an empirical field, akin to engineering, where theoretical results are confronted with the real world). Neither side is what I would call a science.

Enjoyed the comments, particularly Brian Mason’s on wages. I have long lamented the sheer number of people wasting their time and money on a dubious university education. I had viewed it as a triumph of marketing: re-branding fusty old academia as the vocational gateway, but it was new to me to consider a student as just someone working for a negative wage. Nice perspective, and illuminating when it comes to minimum wage arguments.

john penfold
Jan 3 2016 at 5:03pm

We all change our views all the time on all kinds of things,changed interests, new evidence, and experiences, but our thoughts and views are also emergent systems, individually and collectively. Of course after spending ones youth and tying ones income to a PhD in economics, it’s more difficult.

Calum Davey
Jan 4 2016 at 3:34pm

I was surprised that Noah Smith, a strong proponent of the econometric revolution, did such a terrible job of explaining the problems of identification.

In Smith’s example, Galileo makes incorrect identification when the law that he thinks he has identified is ‘true only in a special part of the galaxy’. Smith confirms that he thinks this sort of problem is what identification when he then recounts the famous ‘grue’ problem from philosophy-of-science 101.

The trouble is, this is not what mis-identification is. This is not what the ‘third thing’ problem is. This is the problem of generalisability: that a correctly identified causal effect may not hold in all circumstances and at all times. This is an extremely important problem in social science, which deserves to be given its proper name.

The identification problem is what other fields call ‘confounding’: alternative explanations for the analytical results. In causal terms, this is the third variable problem; that some third variable may the the cause of both of the variables we are trying to show have a causal effect between them. Perhaps Galileo’s balls were not only different weights but made of different material too, so that in fact they rolled at different speeds down the ramp. His result would be misidentified, and the conclusion that weight caused the balls to fall at different speeds would be false.

Jan 5 2016 at 10:09am

Thanks for this excellent guest and interview.

I think when someone self-identifies as a pragmatist they should give us a distribution of where their conclusions fall – else it is just self serving. The President thinks he is a pragmatist. Its not a useful descriptor to someone who does not know you.

And just a niggle on one of his views about the size of government. He argues that we should be cautious because successful governments appear to have large governments so any size changes should be small. That is a very powerful argument if all our reality had sprung into existence in 2015. Otherwise that is absurd. The current nations he identifies were relatively better than most nations when they had much smaller governments. Should I really be scared about rolling back the size of government to the ancient and archaic days of 2000, 1990, 1980 or 1970? Ridiculous.

Ok, on topic. My field, epidemiology, is very much like economics in many of our approaches except with the corrective of medical randomized experiments. As such our field has experienced a number of moments where the majority of the field changes their opinion because a well controlled RCT provides very strong evidence. These experiments too frequently reveal that we were wrong as a field, but it is wonderful that they can be done. Economics has little that is comparable on the macro side. Epidemiology has been deriving data from “natural experiments” since forever – its not enough. You need real experiments.

This discussion was all about macro. However, economics does pretty well in predictions and results in micro economics.

I like Mr. Swans comment above about how to treat economics. Engineering is probably the most important discipline in the world for real human progress. Maybe, properly imagined, economics could be for some social policies.

School choice is a great example of how economics and priors mix. The question do vouchers produce better X, Y, and Z academic outcomes is largely irrelevant to me except as a means of moving policy. I am interested in money that goes with students to whatever education they want (community college, home school, private school, etc) in hopes that long term it result in lower costs and more freedom for me and my children. Do they break up the teachers unions so they are weaker politically and stop hiking my taxes? Do thousands of administrators get let go? Does it result in heterogeneity of education allowing for the break down of advocacy education (not teaching about the environment but teaching them to fight on behalf of the environment). The question about whether they make an academic difference to other peoples children is of very passing interest. I say this to illustrate why so many policy debates and the supposed role of economics as an arbitrator is so difficult. I have different priors not on the question of the economics but on questions no one is even asking.

Jan 5 2016 at 6:27pm

Noah is a socialist in denial, but he is right about Economy being a science as much as any other.

Russ, you cannot compare Macro-Econ with Micro-Physics. Both Micros are repeatable to the extent anything is repeatable. Both Macros are one-time experiments that we cannot set-up as desired.

The ideas that Science predicts and is falsifiable are bogus:

Who cares if people change their minds or not? Some might, but hanging on to one’s ideas while trying to reconcile the new Observations with the prior beliefs has its value too. It’s all in the assumptions. I can’t believe the person becoming a Keynesian did so on the basis of the “evidence”. If the “evidence” is so strong, how come you and me are still not convinced?

Jan 7 2016 at 9:34am

Thanks for another great podcast. I agree Economics is not a science….that science is “not as sciency” as many believe. Am also baffled by Russ Roberts attachment to any fuzzy notions including Libertarianism or free market values at all: his understanding of this belief system is commonly hijacked and distorted by various politico economic hegemons.

Still, am thankful for a podcast which is is more inquisitive than adversarial; that’s underpinned by philosophic enquiries into morality and the significance of research.

Jan 7 2016 at 4:29pm

@ Frank Taussig

I believe Noah’s “epsilon ball” refers to a concept in topology called an ‘open ball in a metric space.’ This is really just a way for mathematicians to take our regular concept of a hollow disk/sphere and abstract it to fancier math spaces (e.g., 17-dimensional space) that might have different definitions of “distance.” In math-speak, “epsilon” is slang for a positive number tinier than any other positive number*, so an epsilon ball is a ball whose radius is very very tiny (some might say “infinitesimally” tiny). Anyway, your basic interpretation is close, it looks like Noah is saying our perceivable universe is an infinitely tiny part of what is actually out there.

I’m surprised not to see discussion of Dani Rodrik, whose most recent book (Economics Rules) directly addresses the ‘is econ a science?’ issue. Rodrik argues that economics cannot produce the well-supported universal theories we see in physics or chemistry. Nonetheless, he claims economists can develop highly context dependent models (e.g., if we assume complete markets, no information constraints, etc.) that, in the right contexts, provide practical intuition to real decision makers.

*whether such an “infinitesimal” number makes sense technically depends on what you assume in your math system, but in mainstream calculus we use a neat trick to reproduce the concept of an arbitrarily small positive number in the definition of a limit.

Bryce Fisher-Fleig
Jan 7 2016 at 7:20pm

Love this episode! Convivial and thoughtful dialog like this keeps me coming back again and again.

I wanted to share with the guest Noah Smith the famous history of science masterpiece of Thomas Kuhns, The Structure of Scientific Revolutions. The central argument of this book is that science specifically does NOT proceed by accumulation. Instead, science goes through phases of “normal science” where accumulation of facts happens, and in tandem theories (which he sometimes calls “paradigms”) also accumulate puzzles and inconsistencies. Eventually the inconsistencies build to the point that normal science breaks down. In these moments, science becomes revolutionary. Scientists are desperate for some insight that will make sense of the all inconsistencies of the theories they were trained in. Eventually, through mysterious historical forces, some experiments and models (again called “paradigms”) gain traction within the scientific community, and normal begins again trying to rebuild itself.

While Kuhn’s work has come under significant scrutiny, I’m surprised to hear anyone claim with certainty that science is essentially accumulation in the post-Kuhns era.

Gene Banman
Jan 7 2016 at 10:01pm

Russ, Great show.

Your question ~”Are there any papers that are so definitive that they changed anyone’s mind?” has two parts and you two kept mixing them together:

1) Is anyone able to change their mind in the face of new data, despite their biases?

2) In economics, is there ever a conclusion with a high probability of being correct?

I think your assertion is that the answer to 2) is mostly “no” and so the answer to 1) is “unlikely” just because its takes high probability data to change someone’s mind in the face of their biases.

However, by mixing the two issues we had this whole discussion about people who have changed their minds: even their strongly held beliefs, being held up as counter examples of your assertion. Those weren’t counter examples as they only addressed issue 1). The real question that should have been focused on was 2).

Even on issue 2), I’m not as pessimistic as you, Russ. There is very wide agreement among economists about basic economic principles. There are very very few socialist economists these days, much less Marxist economists. The data overwhelmed them.

Todd Kreider
Jan 8 2016 at 12:04am

I believe Noah’s “epsilon ball” refers to a concept in topology called an ‘open ball in a metric space.’ This is really just a way for mathematicians to take our regular concept of a hollow disk/sphere and abstract it to fancier math spaces (e.g., 17-dimensional space) that might have different definitions of “distance.”

Yeah, we math guys know what an “epsilon ball” is but also roll our eyes when someone says things like this on a podcast. Just silly.

Some people name drop; a very few math drop. At least Smith didn’t try to throw in Eigenvectors. Kudos to his restraint…save that for the next cocktail party..

Jan 8 2016 at 7:19am

I don’t think the information of whether something is classified as a science is helpful. In my mind science is a process that may be applied to any inquisition.

I see attempts to classify something as a science as an attempt to associate the thing with a high status word, “science”, in this case. I haven’t seen this concept elsewhere, but I call it “status parasitism”.

As an aside, It is refreshing to see someone in your position, Russ, take such a self critical and unassuming stance in this and the superforecasting episode, thanks, and keep up the nice work 🙂

Todd Kreider
Jan 8 2016 at 3:27pm

I forgot to add that I hope Noah is a guest again within a couple of years. I have had my disagreements, but I think his blog is one of the more interesting econ blogs out there.

Mark Webb
Jan 11 2016 at 5:16pm

The real problem with many modern macro hypotheses is that they have little or no predictive power. The benefits of science (however it’s defined) are realized in their ability to predict the outcome of some future event.

Take for example an airplane falling out of the sky. Simple physics equations predict with reasonable accuracy the rate the plane will fall. After these were developed, later physicists discovered you can get more precision by accounting for friction and other variations. But even the unadjusted equation at least gets the SIGN right in predicting that the plane will move toward the earth instead of out into space. When you have experts (in any field) arguing opposing theories that predict results that vary wildly with no clear observational tells either way, there’s clearly insufficient evidence to make a call. That doesn’t mean you’re not engaged in a scientific pursuit. But when you begin to claim your hypothesis is valid regardless of whether it has proven its predictive power, your not engaged in science, just propaganda.

Perhaps the problem is not whether economics is a science, but whether current methods have the precision to answer some of the experimental questions being asked.

Jan 11 2016 at 7:55pm

Thank you, Russ, for the discussion with someone who has a different point of view (different “priors”, in your terminology) than you do. Too many of your guests share your basic assumptions. I find the give-and-take when you have someone you disagree with to be interesting and valuable.

Also, thank you for the recognition that your prediction (in a couple of the interviews of 2008) of massive inflation as a result of TARP spending turned out to be a false alarm. It would be interesting to hear an interview with one or two of the monetarist who at that time also predicted large inflation to see if they would now admit to a flawed understanding of the economic consequences of the spending.

Comments are closed.


EconTalk Extra, conversation starters for this podcast episode:

About this week's guest:

About ideas and people mentioned in this podcast episode:Articles:

Web Pages and Resources:

Podcast Episodes, Videos:



Podcast Episode Highlights
0:33Intro. [Recording date: December 1, 2015.] Russ: We're going to talk today about a number of issues related to economic philosophy and methodology you've written on, and that you and I have disagreed on at Twitter, and a number of people have suggested we should have an EconTalk episode on. So this should be fun. I want to start with the role of ideology in economics. Do you think it plays a role in economic policy debates among academic economists? And do you think it should? Guest: I mean, sure it does. You know, we don't always know exactly what policies work; we don't always know which of the evidence is right. Things like that. And so you're going to have to rely on priors a lot. And people a lot of times get their priors from what they want to be true--wishful thinking. Which is really just, I think, what ideology is. And so obviously it's out there, it's playing a role. Bias exists. And I'm not going to try to claim it doesn't. It's obviously a thing[?]. And I think in the media it's more than in the halls of academia. It's still there in academia, but in the media you get more of it, because the people who really are strongly ideological, who really want to push a point of view are more likely to go to the media than people who just want to, you know, assess the facts as neutrally as they can. Those people are probably a little less naturally drawn to the spotlight. Russ: And of course they get rewarded--I think if you are more ideological you are more likely to be chosen to get that spotlight than if you are kind of a mere truth-seeker. Guest: Absolutely. Absolutely. There's politicians who have an agenda; there's very ideological people out there. And then there's just normal people who have confirmation bias, and they hear something that's sort of--an economics media person saying something that supports what they want to be true, and they want to trust that person or believe them more than someone who says the opposite. So you've got huge incentives for ideological people to get the spotlight. Russ: And my listeners are pretty aware what my biases are. I'm going to lay them out and I'm going to invite you to do the same, just as a prelude to the rest of the conversation. I am a free marketer; I call myself a classical liberal, which means I believe in limited government, personal responsibility. I'm not an anarchist; I think government has a role to play but it's much, much smaller than the role that the government plays in the United States in our economy or in our daily lives. I have another ideological bias though, as I prepared for this conversation that I want to confess. I don't think it will be a surprise to listeners, but I want to get it on the table for you to think about; and you can respond to it with your own position on these issues. I realize I'm very prone to believe that people respond to incentives. I'm very--so this--this is not a philosophical ideology or a political ideology. It's a methodological ideology. I'm prone to believe that incentives are important, that people respond to them; and in particular nonmonetary incentives, not just money. And along with that, I have a strong belief in the power of emergent order--that a lot of problems get solved by individuals either individually, working voluntarily together or sometimes not in any planned way, but that market forces or incentives push them to solve those problems. Of course, I'm aware that there are many problems where that doesn't work very well. Environmental issues being an obvious example. But I have a strong bias, or ideology, or methodological belief that incentives are very powerful. And of course they can be counterproductive for achieving certain ends if they are set poorly. So, that's my take. And I just would also add, before turning it over to you, that when I was younger, meaning--I'm 61--when I was, say, in my 30s or 40s, nothing made me madder than being identified as a conservative economist. First of all, I'm not very conservative. I'm much more of a free market or a libertarian or classical liberal, whatever you want to call it. But second of all, I hated the idea that somehow I could be labeled--that my views would be, then, well-known and that I was just some rigid ideological. But I've come to embrace the reality that I am, to some extent, an ideologue: I have priors about the way the world works. They can come from a lot of places. They can come from empirical evidence, of course. But I think a lot of it is complicated and not easily justified to an objective observer. And I think it's important to admit that, so when people who want to trust me or not trust me, they should know just how much salt, how many grains of salt they should take my positions with.
5:46Russ: Okay. So, your turn. How would you describe yourself? Guest: All right. Well, first of all before I do that, I don't really think that thing about incentives mattering and emergent order and things like that, I wouldn't even call that an ideology, myself. I mean, you know--I have strong priors that differential equations can accurately predict the motion of heavenly bodies, but I wouldn't call that an ideology. That's just sort of an assumption, a prior, about what kind of, you know, scientific things really predict the world and explain the world. Russ: Well, I'm going to come back to that. I'll come back to just why I think it's important: even though I like to think what you're saying is true, I'm not as convinced. But go ahead; I take your point. Guest: Because I think that incentives matter a lot, too. You know, I mean, that's pretty obvious to me. And lots of things seem emergent, at least we might as well describe them that way sometimes. And in terms of ideology: I think of myself as a pragmatist. I don't have a very self-consistent, well-defined ideology to myself. I can't really--I think of myself as a Humean [David Hume]. I have these moral instincts--I think, 'Oh, well, you know, this is too much a restriction on individual liberty' and 'I don't like that.' And then in some other case I'll think, 'Oh, look at these poor people,' you know, 'We've got to have the government do something to help those poor people.' And there's not necessarily a rhyme or reason to that. It's just sometimes I have one instinct and sometimes I have the other. And I think of those moral instincts as sort of the primitive, primal, basic elements of morality, and they don't always work--they don't always form a self-consistent whole. Now, you know, when I look at what really works for nations, what I think looks good for nations, you know I think the rich countries of the world are all doing pretty well: the United States is a great place to live; Canada, Japan, France, Germany, Singapore--I don't really know as much about Singapore. But these are all really good places to live. Every place I have seen is good. And there are some small taste differences. I mean, life in France is not going to be quite like life in Japan. But they are all doing pretty dang good. And, you know, I mean, maybe a radical overhaul where we dramatically decrease the size of government would be even better. And maybe some country will try that. You know, I'm for experimentation; maybe someone will try that. But so far all the countries we see with these great qualities of life, these very successful countries, all have big government in them. And to me, that says that we should at least be pretty cautious before we toss out big government completely. These [?] radical experiments. Russ: And you view that as a pragmatic conclusion. Guest: Yeah. Pragmatic conclusion. I mean, you know: Be careful. Cross the river by feeling for the stones. Go with what works and tweak it a little bit in the direction you like, and don't just launch a radical experiment. I mean, that's what got us in trouble with communism--these people came up with a theory about how everything can be so much better; and it was just a calamity. And it was the radicalness of it and the huge break and radical departure from what's worked in the past. Maybe that makes me a Burkean. I haven't read Burke [Edmund Burke], but-- Russ: Scary thought. Guest: [?] the other day. Yeah. Russ: [?] It might. Guest: I don't know. I mean, I think there's so much we don't know, that we'd better be careful. You know? We only have knowledge about the little bubble of the world close to us, the little epsilon ball right around us. Huge experiments are more like to fail than succeed, so maybe some small countries who do them first, or maybe some country in an extreme situation can be pushed to do that experiment. Like medicine. We don't just go doing crazy medical experiments on human beings because most of the time you are just going to kill someone. Once in a while you'll find some cure for something, but most of the time it's going to be horrible. But I think of it in a similar way.
9:58Russ: Okay, so let me--this is sort of off topic in a way, because now we've gotten into the question of: What's the appropriate role for government, especially when we are thinking about small changes versus large changes? But it's really interesting. So I want to comment on that. So, I think people on my side of the fence, the free market types as well as conservatives, I think fail to notice what you've pointed out, which I think is really important. Which is: Life is pretty good in America, say, where you hear people complaining that it's all going to hell in a hand basket, and how horrible the country is and we have to change things because it's awful. And I agree with you that it's basically pretty good; and in particular, I'd agree with you that it's really good for me. And it's really good for my kids. So, my kids who grew up in a house of educated people with lots of books and limited access to television and devices until they got older and their dad and mom got worn down a little bit--my kids have a pretty good shot at economic success--true happiness, moral success: those are different questions. But in terms of just thriving materially, I think my kids will do well even in an economy where government is 40% of GDP [Gross Domestic Product] measured across Federal, state, and local. Which is not much different from France or the United Kingdom or Germany or the places you also mentioned. So I think that's true. And I think that people who bemoan the size of government often ignore that very uncomfortable fact, despite what you'd think-- Guest: Right. Russ: If you think government is part of the problem and not part of the solution, you do have to deal with the fact that it's going pretty well. So, my response to that--I agree with that part; I think you are right. I think that should give pause to people who want smaller government. There are two things, though, I think that are still worrisome. One, I would call the 'boiling the frog' problem--that, it's true that it's going pretty well right now, but something akin to Hayek's Road to Serfdom, which I think has been misunderstood as saying 'We're going to be communists because we have big government,' or 'We're going to be fascist'--that isn't what he said. He said--it was a warning bell. And I think that people like myself who are thoughtful about--who try to be thoughtful, at least--about the size of government and the fact that the economy is going pretty well, I think there's a worry that it's going well now, but if government continues to get bigger there will come a point where it's not as sustainable. And when we look at countries on your list, such as Western Europe or Japan, we do see some very disturbing signs, in the labor market in particular for young people who are struggling to get access to this engine of prosperity that you and I are doing okay on. Or our kids might do okay on. The other point I'd mention is that, while it's okay for my kids, there are a lot of people's kids I don't think are doing very well. They are handicapped by a really lousy school system. They are handicapped--and we'll come back to this; I want to talk about it in more detail because you've written about it--by a minimum wage that I think makes it harder for them to get experience. Doesn't help my kids--my kids' market values are already above the minimum wage: as 18- and 19-year olds they know Excel--one of my kids knows Excel so he got a job with a great set of learning and opportunities because he was good at math. And so for my kids it works fine, because they're not fighting at the $9-dollar-, $7-dollar-an hour slot. But for the people whose skills are mediocre, they are handicapped by the fact that they don't have good skills; they are handicapped by the fact that their labor market opportunities are cut off by, I believe, a minimum wage that makes them too expensive to hire relative to their skills. So, when I decry the size of government, I am worried about people who are less skilled, who have fewer chances, fewer connections. And their life is not so good. And I worry about how that's going to change over the next 10, 20, and 30 years if government continues to make some bad policy mistakes. What are your thoughts on that response? Guest: Well, first I am pleasantly surprised that the more libertarian person was the first to bring up inequality. That's kind of cool. But anyway, I completely agree. I think a lot of people in America are not doing well, or at least not as well as they'd like. You know, they have a better standard of living than Americans 200 years ago, but that may not be enough to make them happy. It may not resonate. They may have had higher expectations. And they have problems that certainly didn't exist 200 years ago. You know? Like heroin, or something like that. Family breakdown. And so I completely agree and I hear you on this. And I think that, you know, I think about this. I'm not a huge fan of minimum wage. I think that something like wage subsidies, which Jim Pethokoukis of AEI (American Enterprise Institute) has been pushing, and I've been pushing for--it's similar to the EITC (Earned Income Tax Credit)-- Russ: That's the Earned Income Tax Credit. Guest: That's right. That, you know, this third way compromise. I think Milton Friedman was a big advocate of negative income tax, and you know, Bill Clinton pushed it as well. I think that's the sort of best short-term measure for these people. In the long term there's lots of institutional tweaks I think we need to look at. I think there's things we don't understand yet about how, you know how things like corporate ownership and globalization and these big changes may have changed the job environment for people and may have resulted in inequality, may have made some rigidities in our system that were previously tolerable now more severe. And so I think that it's a complex, difficult problem that we are dealing with and I don't have simple, easy answers. My own simple, easy answer is kind of the EITC wage subsidy increase; and I think it's mainly a stopgap. Russ: So, let's go back to the ideology. You described yourself as a pragmatist. What does that mean to you? The way I would take it is you go on a case-by-case basis. You look at what works here, what works there. Is that an accurate description? Guest: Yeah. That's right. Case by case. Russ: So, I think that's unusual, but I don't think it's rare. Right? I think there are economists who are not heavily ideological, who would describe themselves even as scientists--as I think one EconTalk guest said: 'I do have an ideology but I take that hat off when I do science.' And some people may be--I don't think that's possible, but [?] that's the problem with that. And there may be people, and you may be one of them, who basically just look at what's effective.
16:53Russ: So, let's turn to the question of science. Because you and I have written some, had some fun exchanges back and forth on Twitter. So, my claim is that economics isn't much of a science in the way that people in everyday English think about it. I'm not going to get into a heavy philosophical debate about what a science is. But what I mean when I say economics is a science: It doesn't produce reliable empirical relationships that can be used as the basis for economic policy. So, do you agree with that? And if not, why not? Guest: Well, when you talk about science it's really a question--and I am going to get into the philosophy here, briefly--it's a question of how do you know how the world works. In lab sciences you do experiments, and you control a bunch of stuff with experiments. And usually in economics we can't do that. A few times we can, and then we get really, really good results. But usually you cannot. And of course you can just look at correlations and you can say, 'Look, you know, people who got this kind of job ended up doing better in the economy in this, or any millions of billions, or this kind of policy happened and then also the economy did worse at that time so maybe that policy is bad.' And then between those you have this approach of natural experiments, also called quasi-experiments. And this has become much more popular recently. You can often exploit works of how policies happen, of the real-world policy-making process, to get clean identification. Right? And there is stuff that you can't control in a natural experiment that you can control in a lab. And most importantly, the most important difference is that with natural experiments, you can't replicate them. You can't repeat them, do them again. Because each natural experiment only happens once. So it's not as good as lab science. But, it is better than simply looking at correlations. And this is just exploding in the economics profession, this way of evaluating policies. And a lot of people have complained about it: some of the advocates for this kind of approach have been a little, you know, zealous and messianic in their advocacy. But I think most people agree at this point that what they call the 'credibility revolution,' which is really just these natural experiments, has really been a huge boon to our ability to understand the effects of policy. Russ: Let's take an example of that, so that listeners have a better idea what we're talking about. And we had Joshua Angrist on the program a few months back, and he's one of the--I'd say he's one of the two or three most outspoken advocates in favor of this, and a champion of what you've called the credibility revolution. So, just to clarify things: I was trained a lot earlier than you were--and I don't mean in the morning. I got my Ph.D.--I didn't get up--I tried to schedule afternoon classes, as many of us did. Guest: I went to Macro at like 7:30 in the morning. I'm still mad at Michigan about that. Russ: Well, you probably paid a terrible price for it. Your macro probably isn't as good as it could have been if you'd taken it at a civilized hour. Guest: Certainly not. Russ: But, I was trained in the late 1970s, and my econometrics professor was Arnold Zellner. And I was taught all the standard techniques of the day. And that was a long time ago. There have been some improvements. But I wonder, and of course there are many skeptics on this, how effective those improvements can be, have been. So, when we say a 'natural experiment'--I'm going to give two examples. Okay? And then you can tell me if I'm on the right track. So, one would be one that was exploited by Milton Friedman and Anna Schwartz in their book A Monetary History of the United States. So, looking at inflation during the Civil War, and there comes a point in the Civil War where there's high inflation in the South, in the Confederacy. And then the printing press in the Southern States, the Confederacy, gets overrun by the Union Army. So they can't print money any more. And inflation stops. So that's one kind of natural experiment: a dramatic change in the policy regime that allows you to be pretty sure--can't be 100%--that what follows on that is caused by that. There's still uncertainty about that, of course. The second would be, a state increases its minimum wage, but neighboring states don't. And I might have towns that are near the border of the states that change its wage; and I can look at see how the labor markets in those towns did compared to towns that were nearby. So, we might think of them as being in the same labor market or having roughly similar skills; and yet only on one side of the border is there a policy change. Are those good examples? Guest: Those are probably good examples. Depending on how they are done. Right. That's right. So, you have your diff-in-diff [difference-in-differences] approach; you have your regression discontinuity design. And these are some of the standard methods for dealing with natural experiments. Do you know, I've never actually read the Monetary History and didn't know that example about the Civil War? But that's really interesting. That's kind of cool. Russ: I think it's true. [Original source: Milton Friedman, "Prices, Income, and Monetary Changes in Three Wartime Periods," 1952, American Economic Review. Referenced in Friedman and Schwartz, A Monetary History of the United States, p. 59-60, n. 64.--Econlib Ed.] I read it a long time ago, and I have to confess even though I went to the U. of Chicago, I have not read the entire book. So I may be remembering that incorrectly. Or I'm probably remembering it second-hand. Go ahead. Guest: Right. Well, so, that's exactly that kind of thing. Natural experiments are very hard to find in macroeconomics. They are extremely hard to find. Chris Sims wrote a reply to Angrist where he gave reasons why one of the most popular things that people think is a natural experiment, via the Romer-Romer paper about the Fed and monetary policy changes--why that might not actually be a natural experiment. I think Sims is very convincing. Basically, with macro, you don't have a lot of these boundaries where the policy is working in some places and not working in other places. You just have this--because it's macro, it's kind of global. And that can even be true when you look between countries, there's a lot of linkages between countries. So it's just extremely hard to find these clean, natural experiments in macro. And that's why I think macro is always going to be more of the domain of priors than, you know, what we might call micro. In fact, that's almost the definition of macro versus micro. Macro is almost defined by the places where we have worse data. And that's why I think you see ideology start to play a larger role, because it's one of the things that determine people's priors. You start to see it play a larger role in debates. Certainly in the public sphere. Because you can't just point to some of these clear natural experiments and say, 'Well, look, we know this because this. Here's the evidence; very clean identification here.' Several different natural experiments all showing the same thing, so we know what works and what doesn't. You can basically not do that in macro. Very, very rarely. And so that's why I think we see this kind of jadedness in macro debates--simply because the evidence is worse.
24:50Russ: Well, let me push back on that a bit. And when I disagreed or challenged Angrist, I used that Sims article, and Ed Leamer also wrote a similar critique of some of the combinants[arguments?] that the credibility revolution claims. Maybe it's misplaced. Guest: Can I interject a second here? Russ: Yeah. Guest: Leamer's paper is showing why natural experiments are not as good as lab experiments. Which we know. We definitely know that. And he's right about that. And that doesn't mean they are no good. It just means that they are not as good as lab experiments and never really will be. But anyway, okay. Russ: I think he's less--he's much less friendly than that, I think, to the Angrist claims. His claim is basically: You think you've done something to insulate yourself from the problems that he identified in his "Take the Con Out of Econometrics" paper. But he basically concludes by saying, 'Not really.' Well, we'll post those articles and people can take a look at them. Much of them are readable to a non-technical reader. You can enjoy the fireworks I think, especially, no matter what. Guest: Right. And those are some great papers. Those are two of the best philosophy of science papers I think ever in economics. Those two papers: the credibility revolution paper and Leamer's critique of that are super-good reading for anyone-- Russ: Yeah. I agree. They are really interesting. But the point I want to challenge you on is this issue that you can't get a clean experiment. So, it's true you can't get as clean an experiment across space, but you still can get in theory a clean experiment across time. Which is why I think the printing press example is a good one. The one I like to point to, being a non-Keynesian, is the conclusion of WWII. A lot of Keynesian economists, and most prominent being Paul Samuelson, said the end of WWII would lead to a devastating depression if the United States let military spending fall dramatically, as it normally would at the end of a war. It had reached a very high level, you know, an historically high level; and his worry was that the end of the war would lead to a disastrous economy. He implied it would be worse than the Great Depression. He was writing this in 1943. The war ended in 1945. There was no attempt to cushion the blow; nobody listened to the Keynesians, or if they did they couldn't get their policies implemented. And as a result, the end of WWII brought a massive decrease in government spending, an enormous increase in the number of people suddenly looking for work. And there was basically--nothing happened. There was no real dramatic anything. Guest: That's not quite true. There was a very deep, very sharp recession right after the end of WWII. Russ: It was a recession that was not called a--I would say it differently. There was a sharp drop in measured GDP that the NBER (National Bureau of Economic Research) did not call a recession. There was no sharp increase in unemployment. There was a recession I think in 1947 or 1949 but nothing of the magnitude that had been predicted and certainly not even clearly related to the change in military spending. The change in measured GDP--you have all kinds of problems. You've got changes in the removal, I think, at that point of wage and price controls. All kinds of other things going on. I don't remember what the reasons were that the NBER didn't classify that drop as a recession. And maybe I'm wrong about that. We can certainly look it up in more detail and fight over that if we had the charts in front of us. But just sticking with the theoretical question now: that was a natural experiment. It didn't shake, interestingly, very much of the Keynesian religion that people held. And I went back for fun and looked at the American Economic Review and the Journal of Political Economy, two of the more prominent economic journals of the day, and what people wrote at the time. And it's amazing how easily they shrugged that off, if they were on the Keynesian side. And similarly--I'll pick on my side now--the largest increase in the balance sheet of the Fed--my side being I'm something of a monetarist, trained in Chicago--the balance sheet of the Fed went from $800 billion to $4 trillion and every economist on "my side" said that's going to cause massive inflation. And nothing happened. Now, they would say, 'It just hasn't happened yet.' And that may be true. I'm willing to cut them some slack. But I think they should also consider the possibility that their understanding of the relationship between the Fed's balance sheet, velocity, and the level of prices might be imperfect. Guest: Right. And I think--you know, these are good. And this is how knowledge in macro really does progress, to the extent that it does. You have some theory where some people make some pretty bold claims. And I think monetarism makes some pretty bold claims; and Keynesianism makes some really bold claims. And a lot of times you have some situations where the bold claims make some strong predictions, and they don't pan out. In fact, I can't really think of any theories that made these bold claims that ended up not having such cases where they didn't pan out. They all have big problems. And if you try to fit these things to past data and you try to do some rigorous statistical tests, they would be rejected. Even if you let your theory be the null hypothesis--which is cheating--you still can't get these things. They'll all be rejected. So, macroeconomists have retreated mostly to using these things as thought experiments, or the sort of weasely phrase that people will use is, 'Ah--they organize our thinking.' They are a map between the assumptions and conclusions. Russ: But I'm a big fan of that phrase. And I think that's where economics has most of its power. But do you really think it's true that most macroeconomists concede--let's take the more prominent Keynesians and monetarists and anti-Keynesians and anti-monetarists. Do you think they'd say, 'Well, you really don't have much evidence for abuse[?]; it's really too complicated to try to tease out any of these effects.' So we're just kind of stuck with--what? I don't know. Guest: Well, there's always going to be true believers. Are there always going to be true believers? I think that I have observed that there are often true believers. My Ph.D. adviser, Miles Kimball, really believed that if you just take interest rates negative enough then you'll get your big output boost. And I think that so far we haven't really seen much evidence in favor of that: interest rates are going somewhat negative in Europe right now, and there's no obvious boost from it. But, it could just be-- Russ: I don't know his work, but you know what he'd say. And I know what he'd say. Guest: More? Russ: 'They just didn't go negative enough.' Guest: Just do more? Right. And I can't tell him that he's wrong. I'm highly skeptical that he's right. And of course Miles is an extremely good guy. He's a true believer in this kind of theory without being ideological about it at all. So, actually that's what's interesting. Miles is sort of my counterexample, one of my counterexamples; the most extreme counterexample to this idea that everyone is really ideological when you get right down to it. Miles is almost pathologically non-ideological in his ability to separate this sort of desires of how the world should work with his assessment of how the world really does work. He's really good at that, better than most. And I think that he would be a great guest for EconTalk in the future. You can sort of see what I'm talking about. Of course, that degree of personal objectivity, if you want to call it that, is rare. I haven't seen that. He's an extreme example; but I think he's a big counterexample to this idea that everyone in the end, you know, or maybe just in macro, really just comes down to their ideology. You can have strong priors: you can be a true believer in a theory without have a political ideology like small government, big government, you know, help this group of people, help that group of people, etc. You can still have very strong belief that this theory will work. Russ: Absolutely. Yeah. Although, having said that, there aren't--well, I'm not going to go down that route. I was going to talk about interface between macroeconomic ideology and philosophical ideology. I think that's--let's stay away from that swamp.
34:02Russ: But, I want to go back to our original question. So, I don't think economics is much of a science. You seem to think it is something of a science. But you've now said that in half of the field--some would say 80%, some would say 10%, but I'll say half, just to make it simple--which is macroeconomics, it's really--there's not that many natural experiments that allow us to tease out causal relationships and design policy accordingly. So that leaves us with microeconomics. So, what are the natural experiments? I want to pick on the minimum wage, partly because it's such a hot button topic and also there's a huge amount of empirical work on there. Do you think we have learned much that is reliable about the impact of a minimum wage increase on employment? Guest: I think we have--I think that we see that very rarely are the--if we take the size of minimum wage increases that are usually done, they very rarely have a perceptible significant, economically significant, impact immediately on employment levels. So, what employment effect they do have is probably over the longer term, in terms of things like employment growth. And people can argue about that. But we see that--and of course if you hike the minimum wage to a billion dollars per hour, you would obviously get huge, distorted--the above-ground economy would immediately collapse and everything would become black market. But, for the types of increases that we tend to see, very rarely is there a large, big sudden drop in employment. I think that's been a pretty reliable conclusion. That doesn't minimum wage is good. And it doesn't mean minimum wage doesn't have long-term employment effects. And it doesn't mean that very big minimum wage increases wouldn't have an effect. And it doesn't mean any of those things. But it is something. You know. It is interesting and it is something. Russ: Well, until the Krueger and Card paper of 1994 that suggested that the minimum wage would have little or no effect and I think possibly even a positive effect on employment, there had been dozens and dozens, maybe a hundred studies that, in the minds of the people who wrote those studies and in the minds of the people who believed them, very powerfully showed that the minimum wage is bad for poor people; that it reduces employment by a non-trivial amount. And there's disemployment as a result: people lose their jobs or will not get jobs. And then there was a change. That paper came along. And it launched a whole literature saying, 'Well, maybe it's not so bad.' So, as a Martian without--just a truth-seeker, 'I have no ideology; I just want to know the truth'--which of those literatures do I accept? And of course there have been many critiques of the modern literature that finds little or no effect. How do I know which one's right? Guest: I mean, you have to look at the methodology and you have to look at the identification strategies. That's what you really have to do. And, you know, better identification is better. How do we tell which is better identification? Well, we've got to make a guess. But, you notice that's always true in all of natural science as well. If you are going to believe the results of an experiment, you've always got to make the leap of faith that all the reasonable stuff has been controlled for. Right? That the experimenter got good controls. And that's an assumption and a leap of faith you have to make in every natural science experiment. And it's also something you have to look at in natural-experiment literature. You just have to say, 'Look, this is good identification and this is bad identification.' If you go to seminars, that's mostly what people are doing. That's what they are talking about. Russ: Could you explain that for the non-technical--or try? It's a tough question, not an easy thing to explain. When you say 'good identification' or 'bad identification,' what do you mean? Guest: I can explain a super-super-extreme version of that. Right? So, suppose we roll a ball down a ramp like Galileo did, and we see that balls of different weights will still fall at the same rate--will roll down these ramps at the same rate. Okay? We find that out and we conclude there is a law of the universe: Balls always roll down ramps at the same rate regardless of their weight. And let's say that that is only true in a certain region of the Milky Way Galaxy, and tomorrow, our planet, which is always whirling around the center of the galaxy, leaves that region and goes into another reason where the laws are somewhat different, and balls actually roll down ramps at slightly different rates. And then suddenly we re-do the experiment tomorrow and everything is completely different. Now, to believe that the first experiment, you know, really told you a law of the universe, you've got to believe that there is such a thing as a law of the universe. And you've got to believe that there was no other funky thing going on in the cosmos that was making your experiment come out one way that might make it come out a different way tomorrow. And that's really the leap of faith you have to make. Now, that's the fun, goofy example. In these economic experiments what you are really talking about are basically exogeneity. You are talking about: Is the experiment I did basically just a purely random variation or was there some omitted variable that's causing the effect that we see? Russ: Or is my friend Don Cox, who was the first guest back on EconTalk back in 2006, said: 'We're talking about the dreaded third thing.' That there's this thing you didn't observe--you think there's some correlation between these two that's causal, but there's a third thing that you either didn't have data on or didn't notice or didn't think about that's really driving things. Guest: Yes. That's right. That's exactly right. Russ: But there's always a dreaded third thing, isn't there? Guest: There's always a--it just depends on how much dread you have. You know? In rolling balls down ramps, there still is a dreaded third thing. And I realize this--do you know the emeralds or Green or Grue paradox? Do you know that one? Russ: No. Guest: The philosophy thing they ask first year philosophy kids and undergrads. It's the idea that emeralds could be green, or they could be another color called 'grue,' which is green until January 1st 2025 and then blue afterwards. No experiment we do today will be able to distinguish green from blue emeralds. But it might make a difference, because come Jan. 1, 2025, all of your emeralds might suddenly turn blue. If you want to be afraid of that, you can be afraid of that. You can never rule it out. There is always the dreaded third thing if you have enough dread. So it really comes down to a bunch of sort of grumpy economists in a seminar room yelling, 'I don't think that's clean identification because what about this and this?' and trying their darnedest to think about what that dreaded third thing might be. How plausible those things might be.
41:44Russ: So, I think, if you ask me to gamble on whether my emerald is going to turn blue on January 1st 2025, I'd ask you: What's the underlying causal relationship between the data on the calendar and the color of the emerald? And you'd say, 'Well, I don't know what it is, but there could be one.' And we're just unaware of it. We don't have a full understanding of color, etc. But we have a pretty good understanding of color. I think that's part of the reason that I--I think it's hard for these empirical techniques to reliably establish causal relationships with any precision about the magnitudes that are involved. So it's interesting. I would suspect that every economist who is in favor of increasing the minimum wage would concede the possibility and even the likelihood that there would be some employment loss in response to that. They just think it's perhaps smaller than I think it would be. Do you think that's a fair summation? Guest: That's absolutely fair; and by the way, as an aside, I'm not really very big on the minimum wage. I don't think it's that great of a policy at all. I'm not really a fan. I think national minimum wages are an especially bad idea. Local minimum wages certainly not nearly as bad. But I think that there's just lots better stuff than the minimum wage out there. And, so that's what you're talking to: I'm talking about this empirical literature, but at the same time it's not really a policy that I'd do if I were in charge. Russ: So, let me ask the question a different way. Guest: Yeah. Russ: The way I sometimes ask it. Can you think of a study that was so decisively performed in terms of the crossing of t's and dotting of i's that the identification and all the econometric challenges were met with such impressiveness that people on the other side of the debate had to throw up their hands and say, 'Well, I guess I was wrong. I've got to change my view.' Because I can't think of one. I can't think of one. And if that's true, then I would suggest that economics has some serious problems in claiming it's a science. And I would say, especially--and you can rule me out here if you want--but especially about macro. We just had an $800 billion dollar stimulus package that--forget 1945 or the end of WWII--we just had a very nice natural experiment. And we can't get people to agree on whether it was successful or not. There are people who say it was fantastic: it saved us from the Great Depression. There are people who say it wasn't big enough: if had been, we would have it bigger, we would have had a better recovery. And there are people who say it's the reason we had a mediocre recovery. They are all really smart people. Most of them. The ones I'm thinking of. They have great credentials. They are at great institutions. Some of them have Nobel Prizes. And yet they can't convince each other. Why not? What's going on there? Guest: Well, first, before we go on to there, I want to say that--you know, when you talk about things being a science, my whole point earlier was that even science isn't a science in the way that some people like to imagine that it is. Russ: I agree. Guest: You have the smartest people--the thing I said about maybe balls roll down things differently in a different region of space: That's a real theory. I'm not kidding. I didn't make that up. Okay? On the spot. That's a real theory, and you had some really smart people saying, 'Well, maybe this theory is right.' You have some of the smartest physicists in the world right now who think that in different regions of space, different laws of physics might actually prevail. And it's an active debate. You see people on both sides. So, even science is not quite as sciency as people may hope--this shining idealized version of it. You have the smartest people in the world radically disagreeing on things in physics, too. So maybe the bar for being a science isn't quite as high as people think. Russ: Fair enough. Good point. Guest: But as for economics, I think that the thing about the stimulus speaks to the point I was making before, which is that when you--it's not at all clear. The stimulus was not a natural experiment at all. It was not randomly decided upon. Okay? Nor are the variations from state to state necessarily random--which are often used. And even the effect of those variations depend on some assumptions about the national level effect, too--which again is not a random thing. It's not as if Obama just woke up one day and some quantum state flipped in his brain and said, 'Let's do an $800 billion dollar stimulus' and then it happened. It wasn't like that. There was a reason why it happened. So if you want to say, oh, we did this whole stimulus and the economy didn't recover, that's proof; someone else can say, 'Oh, it would have been even worse.' And given how bad the Great Depression was, maybe they are right about that. Or some people say, 'Well, that's the reason'-- Russ: But they could say the same thing about any empirical study. Very rarely do we get a truly random imposition. Certainly that would be true of the minimum wage. Guest: Sure. Russ: It's very possible that states that have lots of poverty are more likely to pass them, or vice versa--that there are states that are particularly healthy so they pass them to look good but they are not really going to have much effect because actually the market-clearing wage is well above the wage that was passed. Guest: Sure. Russ: But going back to my original point, can you name a study that you think people go, 'Well, wow, that one was so good I have to throw up my hands. I give up.'? Guest: Well, I don't know about throwing up your hands and giving up. I think a lot of people who change their minds will not be despairing or giving up. They'll think, 'Wow, this is cool. Today I learned something.' Russ: Well, that's what I meant by giving up: I'm giving up my old position-- Guest: But I can definitely draw a contrast-- Russ: I'm doing[?]-- Guest: I think that definitely happens all the time-- Russ: I'm doing the old Keynesian thing, which was evidently not really said by Keynes, which is: 'When the facts change, I change my mind. Don't you?' Guest: Right. Russ: Evidently that isn't necessarily Keynes's quote, but it gets attributed to him. So, has there ever been a study where people said, 'Yep, I have to change my views now.' Guest: Well, I don't know because I haven't done those before-and-after surveys of economists' opinions. I've seen snapshots through IGM [Initiative on Global Markets] or some of the other surveys of economists' policy positions that have been done. But I just don't have the data to answer your question. One study that's been quoted to me as a study that made people change their mind was the Cullen and Levitt--and I think there was someone else; I hate leaving one person out but I can just remember two [Cullen, Jacob, and Levitt] --Cullen and Levitt's study on the Chicago education lotteries when they randomly let some people go to good schools and other people not; and they showed that the people who went to the good schools don't really show any better academic outcomes, but they do show better behavioral outcomes. And there have been a couple more studies like that, that basically show the same thing since then. That was quoted to me by one of my Econ teachers as a study that really made him re-evaluate his ideas. You can easily see a qualitative difference--immediately and easily see a qualitative difference between that kind of lottery experiment and the 2009 Stimulus Bill Russ: [?] Guest: There is a qualitative difference in that. Russ: Fair enough. Guest: And the idea-- Russ: I'll concede that one. Guest: Great. The idea that you'll never convince everyone, I say, well, even in physics you'll never convince everyone. I mean, not-- Russ: No, for sure. Guest: Even in physics you'll never convince every single person. Russ: Yep. Guest: You'll still have that crazy guy, maybe at the Steklov Institute. Anyway, then, you have--it's on a sliding scale that I don't have data with which to measure, of convincingness. You know? And the stimulus is pretty low on that convincingness scale because it's so non-exogenous. It's obviously so non-exogenous. And those lottery studies are just much higher on the convincingness scale. On the human convincing power of its identification assumptions. And I wish I had really good evidence on a whole bunch of studies on how they made economists change their opinion and I don't. And someone needs to do that kind of study where they measure a bunch of people's ideas and then they wait till after some studies and then they measure their ideas again. I don't know how they'd do that. Someone should be working on this. But I think you'd see big differences. Russ: That's the difference between you and me. I'm pretty confident that that wouldn't decisively determine whether I was right or wrong. I'd rely on less formal evidence. But I like the claim. I think it's an interesting point. Guest: Well, I can prove to you that you can never convince everybody. I can program a robot that will not be convinced by evidence, okay? Russ: That's not the question. Guest: I can program you a robot that just says, you know, 'Stimulus is awesome.' And I can program this robot and I can even have it tweet 'Stimulus is awesome' and I can program another robot that says 'Stimulus is terrible' that I can have that one tweet. So you know that there's obviously some sort of being out there, maybe an artificial intelligence that can't be convinced of things. But what does that prove? It proves nothing. It's trivial.
51:36Russ: I'm making a much weaker claim. In the aftermath of the stimulus package and the Great Recession, Richard Posner wrote a piece that got a lot of attention, where he said, 'All my life I made fun of Keynes, but it turns out he was right.' More or less. Guest: I saw that, yeah. Russ: It's not exactly what he said, but it was close. And, that's one person--he's not an economist but he does a lot of economic writing. I'd[?] call him an economist even though he doesn't have a Ph.D. in economics. And that was interesting. It was brave. I didn't find it convincing--I wouldn't. So, I'm not talking about a case that a study comes along that convinces everyone. I'm talking about a study that convinces anyone. Because most studies, my claim is, don't convince anyone. They find what's wrong with it; they easily dismiss it because they have priors against it--whether they are ideological or methodological. Which suggests--what I'm really saying--natural experiments-- Guest: Well, that's a hugely strong claim. That's not a weak claim. Russ: Noah, here's what I'm really saying: Natural experiments are really good for getting publications in economics journals. They are not so good for convincing people. So it's not that they are not as good as, not quite as good as real experiments. It's that they are really only good for generating journal articles. As opposed to getting people to go, 'Wow. I never knew that.' That's my claim. Guest: Wait a second. Okay. Back up your claim, now. How are you going to back up that claim? Russ: Because I don't see a lot of people writing about the fact--other than Posner, which is kind of like the exception that proves the rule. Someone that writes an article, says at a workshop, blogs, that says, 'You know, I always thought that the public school system was awful. But now, after that study of its efficacy, I'm wrong.' Or, vice versa: 'I've always been a big critic of the public school system, but now that I see these results, I have to admit I'm wrong.' I don't see that. Ever. And especially I don't see it-- Guest: Well, what about me? I have changed my mind in response to some of these studies. Including the lottery study I just talked about, the Cullen and Levitt--and possibly someone else, I forget. Russ: So, what did you change your mind about there? Guest: I thought that being able to go to a really good school, before, or a school of your choice, would result in improved academic performance. I really thought that that would have been true. I was astonished when I saw that result. And it has changed my thinking on the topic. So, you've just seen n of 1. Russ: Okay, it's a start. Guest: That changed my mind. It changed my teacher's mind. I think he said. And it definitely changed my mind. I was like, 'What? Really?' I just couldn't--you know. Russ: So, he's going to buy a house in a poor part of town because it's foolish to pay a premium to be in a good school district? Guest: No. Because you know, it might look bad, you might get robbed, people might not come there. Russ: It's a cheap shot. Sorry. Guest: Yeah, okay. Sorry. I was in serious mode. Russ: I was, too. But I'm making the point. I understand it's not a true test, convinced or not. Guest: Are there people who are really going to change their behavior toward raising their own kids because of this? Maybe there are. I think, following seeing that study, I've certainly thought more about the importance of self-education, self-study, parental guidance for teaching, and natural ability--to be honest. So, yeah. Russ: No, I concede your point. And I'll make your point a little stronger. I'll use myself as an n of maybe 2--[?] to your 1--which is, I always thought that education was really important. Being exposed to the ideas of Bryan Caplan as a colleague and here as an EconTalk guest, I'm more open to the possibility that school is mainly signaling; that the higher wages associated with schooling are really a result of uncontrolled variables that are person-specific rather than related to the education itself. So, I'm making a similar point to what I think your teacher is agreeing to. It's not a core position, though. It's not really--my worldview doesn't hinge strongly on that. So, I can live with that. That one's not so bad. I'd like to see a Keynesian say, 'I'm really disappointed in this.' Or a monetarist say, 'I'm really disappointed in this.' Or an Austrian say, 'You know, the evidence just isn't there.' It's just hard. It's hard for us to concede those things. And some of it just is psychological. Guest: Well, you just talked about Posner, though. Russ: That's one. Guest: You just--I heard that right. So, your incredibly strong claim that no one changes their mind has been disproven by Posner. Russ: Three times. You, me, and Dick Posner. Guest: I would cite--I don't want to put words in his mouth here, but I would cite Steve Williamson as another example. You know, Steve Williamson in 2011 was talking about how Quantitative Easing put us in great danger of inflation; and by 2012 had essentially invented neo-Fisherianism, which is now gaining a little bit of credence in some corners of the macro world. And so, so, Williamson had a pretty big change based on the fact that inflation didn't materialize after Quantitative Easing. And he even came up with a new theory to explain that change, that has now become at least somewhat popular. Some people have latched on to that--John Cochrane and some others have expounded on it, expanded upon that. And so, I would--I don't want to put words in his mouth but I would offer him as n of maybe 4, here. So we're already accumulating a number of people who have changed their minds around stuff, even in macro, where it's hard, where it's a lot harder. I keep thinking these micro experiments are a lot better and more convincing than typically the macro stuff. But, you know, even in macro, we're looking at some big changes--of people who didn't obviously change their ideology. Steve Williamson didn't go from hating inflation to loving it overnight. Russ: Correct. Guest: You know. Posner didn't go from wanting small government to wanting big government overnight, for some other reason. These aren't big ideological or political ideological changes driving these changes of opinion. On macro, I never held very strong priors on anything, ever before, so you won't find me changing those non-strong priors very much. I was so uncertain to begin with. I guess hence the name of my blog. But, anyway, but I think you do see some of these things where these people really do change their mind based on something they see occur.
56:47Russ: So, let me try to clarify what I'm saying here. And then we can move on. Guest: Yeah. Russ: We all understand that when you have a vested interest in a particular intellectual viewpoint that it's hard to concede that your life has been a failure. It would be very unusual for an economist who had spent his or her life as a Keynesian to say it was all a lie, or vice versa, not a monetarist say it's all a lie. Just for psychological reasons. We understand that. What I'm really--so, the fact that most people don't change doesn't really prove my point. What I'm trying to say--and I'm not saying it terribly well--what I'm trying to say is that it depresses me greatly how hard it is to provide empirical evidence that would cause a person to change his or her mind. Because I think it's almost by definition in macro--and I'm going to push, I'm going to make an even harder case for you to defend. I don't think it's just a macro problem. Let's take the example of deworming, which has been a big issue recently. We talked about it in the episode with William MacAskill on effective altruism--big, scientific claims about the way to make the world a better place is to deworm African children and other poor children who have parasites in their system. And that was based on a natural experiment. And then it turned out maybe it wasn't so reliable; it didn't scale; it didn't here, it didn't work there. It got some benefits. But the benefits in terms of improved educational and school performance are very much in question. So I think it's just a general problem that in a complex system it's very hard to find reliable results. That's really my [?]. Guest: Because of that example of one experiment that didn't end up holding up? When people looked at it more carefully? That one? Russ: Well, I guess I-- Guest: I think one-- Russ: Touché except-- Guest: I mean, most aren't going to hold up. It's not going to hold up-- Russ: Touché except for the fact that Brian Nosek was on the program a few weeks ago, and it's 60% of psychology experiments that don't hold up. So, one experiment is-- Guest: That's Social Psych, yeah, that's right. I would protest--my Dad's a cognitive psychologist and apparently only 40% of cognitive psych experiments didn't hold up. He was very proud about that. Russ: It's true. Guest: Okay. So, I've seen--people are starting to do the same tests with economics, and they find similarly large ratios of studies not holding up. That's fine because science is a process and evidence proceeds by accumulation. That's also true in physics and chemistry. You had, even on like the charge of the electron you had to get lots and lots of experiments before people fixed on what they really believe in, was the charge of the electron. People were probably fudging [?] the time and people were making some mistakes. So, it's an iterative process. If you talk about these single thunderbolt studies that really change your thinking, yes, they exist. But really what's much, much more common is an accumulated weight of studies that all seem to have a kind of consistent result. You make a little tweak, you study it; you use a slightly different data set; you use some slightly different identifying assumptions or methodologies; and you look in a slightly different environment or for slightly different numbers; and you see something different. Or, you see basically the same kind of thing. And so, sometimes you don't see the same kind of thing and you just say, 'Well, we don't really know what's going on. Let's go work on something else. Who knows? Let's go for a beer.' Or then sometimes you get this big accumulated weight of evidence where everybody says, 'Okay, now we all sort of agree that this is right and this is wrong because we've all seen a million studies saying it.' So, don't rely too much on these thunderbolt studies, even though sometimes they do exist. But they are pretty rare. Oh, I wanted to show you--n equals 5 on macro: I have a really good example that I forgot about completely that I just remembered. Can I give it? Russ: Yeah, sure. Guest: Bob Lucas--Robert Lucas, most influential macroeconomist possibly ever, I'm going to say-- Russ: Certainly of the last 40 years, 35 years. Guest: Certainly, certainly. A very intellectually aggressive and sometimes even polemical guy, if you read his papers or if you meet him--he said some things in a Q&A once that might have to be edited from your show. But Robert Lucas wrote--and I can get you the cite on this very easily--that up until the early 1980s he had believed that recessions were caused by monetary shocks. By nominal shocks, he called them. And that was supported by his own, Lucas's islands model, Lucas supply function, which I'm sure you know. And then he said, however, Prescott managed to convince him that most recessions were due to real shocks. So, basically real business cycle guys convinced him, and changed his mind. And he said, then, after the Great Recession, he decided that financial shocks must be a really big important thing as well. And he wrote this down. And I'll find you the cite for this. And he chuckles, because he's looked at how his own opinions have changed--Robert Lucas, whose life's work was definitely invested in some of these models. It was first the Lucas islands model; Lucas supply function was a huge part of his life's work. Then later on, he really did a lot to help the real business cycle people; even though he's not always cited as one of the pioneers of it, he really was. And then, those are models without financial shocks. And now he comes out and says, 'Well, okay, after the Great Recession I believe that financial shocks are a big deal.' So, even Robert Lucas, who had his life's work tied up in some of this stuff has changed his mind now twice and is willing to admit it.
1:04:58Russ: I think it's more than twice. I think when he was younger, he was in a different place. But I take your point. And I would say, although Lucas gets a lot of criticism from people on the other side of the ideological fence, I had him for my second macro class, and third macro class, in graduate school; and I didn't become a macroeconomist. But his passion for understanding how the world works was very inspirational to me. And I think his intellectual integrity is very inspiring. So, I certainly would never want to suggest--I think your example is a great one. And I think he's actually deeply interested in how the world works. And I think most academics are. I just think we have a problem: which is that our philosophical viewpoints and the very nature of economics just makes it difficult to see what's actually going on. And to some extent I think we're all in the dark. We have some basic principles that are reliable in various settings about incentives and markets and how the world works. But it's very imperfect. And a part of what we're disagreeing about here is just a question of degree. Let's come full circle, because we're out of time. You started off by saying that the most ideological people get most of the attention, and enjoy the attention. And I think that's true. I think deep down, most economists when pressed--and I have no empirical evidence for this; don't ask me for a cite; it's armchair theorizing--I think deep down we will admit when pressed that our knowledge is not as reliable as we sometimes talk about it as being. I'll give you the last word, and then we'll close. Go ahead. Guest: I think that if deep down, if when pressed we will admit that deep down we are less certain, that's a good thing. Russ: I agree. Guest: And that means that ideology hasn't totally blinded us to the facts and evidence. Russ: Hasn't totally blinded us to what? Guest: To the facts and to the evidence. If deep down--our egos and our natural level of arguing and our vestedness in our life's work, whatever--all these things, media attention--press us toward expressing great certainty, putting forth things with great force, smacking down the enemy, but if when pressed in private deep down we will admit we are really less certain than we think, that's a good thing. And that shows that maybe people are looking at the facts and the evidence, in private a little more than they might be willing to admit in public when engaged in these sort of contentious debates.