Austin Frakt on Medicaid and the Oregon Medicaid Study
May 13 2013

Austin Frakt of Boston University and blogger at The Incidental Economist talks with EconTalk host Russ Roberts about Medicaid and the recent results released from the Oregon Medicaid study, a randomized experiment that looked at individuals with and without access to Medicaid. Recent released results from that study found no significant impact of Medicaid access on basic health measures such as blood pressure and cholesterol levels, but did find reduced financial stress and better mental health. Frakt gives his interpretation of those results and the implications for the Affordable Care Act. The conversation closes with a discussion of the reliability of empirical work in general and how it might or might not affect our positions on social and economic policy.

Jim Manzi on the Oregon Medicaid Study, Experimental Evidence, and Causality
Jim Manzi, founder and chair of Applied Predictive Technologies, senior fellow at the Manhattan Institute, and author of Uncontrolled, talks with EconTalk host Russ Roberts about the Oregon Medicaid study and the challenges of interpreting experimental results. Manzi notes a...
Adam Cifu on Ending Medical Reversal
Why do so many medical practices that begin with such promise and confidence turn out to be either ineffective at best or harmful at worst? Adam Cifu of the University of Chicago's School of Medicine and co-author (Vinayak Prasad) of...
Explore audio transcript, further reading that will help you delve deeper into this week’s episode, and vigorous conversations in the form of our comments section below.


May 13 2013 at 2:28pm

Great Podcast! Very informative, in this day and age seems crucial that each of us has some understanding of statistics and economics since they are so relevant to policy, since politicians, scientists, and we all love using them to validate and justify our beliefs. Seems like a basic form of literacy to be able to think for yourself and not rely on blind trust of scientists in the sense that in such complex phenomena the room for bias is greatly amplified. Thanks for improving the quality of our democracy Russ, by stretching the minds of the electorate. Keep having such interesting and varied guests.

May 13 2013 at 6:20pm

This was a very well done and timely interview!

I have one comment regarding the study power that did not come up directly in the interview. Even with the same number of participants, the study would have had a greater chance to detect an effect if it had included two sets of measurements – one near the time of the lottery, and then once more two years later. That would have given the opportunity to see how each patient’s measures changed after two years on Medicaid (or not on Medicaid). Instead, the study was, in essence, looking to see whether the proportion of poorly controlled diabetics was lower among people who had been on Medicaid than among those who had not.

Second, it is worth mentioning the convoluted way that we do statistics. We usually consider our null hypothesis to have been rejected when a ststistical test gives us a p value that is 0.05 or lower. But p=0.05 actually means something quite different than what many interpret it to mean. For example, in Oregon Medicaid study, the authors compared the proportion of poorly controlled diabetics in the control group vs. the treatment group. A p value of 0.05 would have implied that probability of seeing the observed difference between the two groups even if the null hypothesis was true (i.e. there was no difference) was 5%, or 1 in 20. But many people interpret a p value of 0.05 as meaning that there is a 95% that the finding is correct, which is not the case at al.

It’s also worth noting that accepting p=0.05 (or less) as “statistically significant” is merely a convention. If p=0.06, your research finding is not significant, your drug would not get approved, your work may not be published, etc. But the difference between p=0.05 and p=0.06 is 1% greater probability that the result could have been due to chance.

May 13 2013 at 8:03pm

Perhaps one thing to consider is how significantly a particular symptom improved among those who received Medicaid. Take blood pressure, for instance. Perhaps of the 1500, only 20 people experienced a lowering of their blood pressure. (I’m just making this number up – I don’t know how many truly showed improvement.) I’m wondering by how much their blood pressure decreased? Was the decrease large enough to significantly improve their quality of life? In other words, even if the number of people with lower blood pressure was not statistically significant, was the size of the decrease in each individual significant? I’m not sure whether this is true. I just think it’s of interest to know whether only a few people were affected but in a very substantial way. Does this affect whether we view Medicaid as worthwhile program?

May 13 2013 at 9:22pm


The study design didn’t allow for measuring change in individuals.

It was basically:

1. Randomize to Medicaid and Control groups.
2. Wait 2 years.
3. Test blood pressure.
4. Are there fewer patients with high blood pressure in the “Medicaid” group? Yes. Is the difference statistically significant? No.

May 14 2013 at 1:01pm

At one point the guest said that the effect of the $1,200 spent per person on medicaid provided the (i think mental health only) equivalent benefit on mental health of equal to something like an increase in income of $30k (or maybe double I can’t remember, lets say $30k for argument’s sake).

Basically, I think what he was saying is that poor people with medicaid were as mentally “healthy” as people without medicaid but who were making $30k more (or whatever the number was) in income.

If you assume that the people who make $30k more (but don’t have medicaid) are choosing a level of spending on healthcare rationally (questionable assumption i know) then doesn’t that imply that the $1,200 of healthcare is what the higher income person should choose to allocate(assuming healthcare is a premium good)? As a rational person’s income is marginally reduced from poverty plus $30k, they’d consume less on healthcare and transfer that spending to something else.

Said another way, if instead of giving the poor people $1,200 in healthcare medicaid just gave them $1,200 in cash, wouldn’t they according to this logic rationally spend less than $1,200 of it on healthcare and the balance on something else that benefits them more than an increased level of mental health?

I apoligize for not remember the exact detail, but I think this logic is sound.

Stéphane Couvreur
May 14 2013 at 1:28pm

Near the end of the podcast, Austin Frakt says he loves to be refuted when in error. This reminded me of a nice Socrates quote in Plato’s Gorgias:

´I am one of those who are very willing to be refuted if I say anything which is not true, and very willing to refute any one else who says what is not true, and quite as ready to be refuted as to refute-for I hold that this is the greater gain of the two, just as the gain is greater of being cured of a very great evil than of curing another.´

I’s rather be proved wrong then prove you wrong: neat utilitarian argument!

Thanks for this great podcast,
Stephane from Paris

Eric Falkenstein
May 14 2013 at 5:41pm

Stephane: I suspect that 99.9% of the people who say that mean it. Sure, they will admit to trivial errors, but never profound ones.

May 15 2013 at 2:15pm

Very good discussion.

One question that Russ Roberts brought up and Dan reiterates above is why not just give them the cash? This is certainly a valid response to many aspects of the welfare system. Food stamps are better than sending poor people baskets of food rations. Housing vouchers are better than public housings. Both might be better as cash transfers that allow people to choose their levels of spending on various categories.

There are two problems with applying this to healthcare: one philosophical and one practical. First, unlike our needs for food and housing, our need for healthcare is not uniform. Any level of benefit would give some people more than they need and others less. They couldn’t compensate for this on a private insurance market because an individual’s expected healthcare costs are too predictable to work like normal insurance.

Second, health insurance provides a collective bargaining aspect in additional to the insurance aspect. An individual is going to get a much smaller basket of healthcare goods and services on their own than an insurance company could with the same amount of money. Because Medicaid has the lowest reimbursement rates, the increased prices poor people would have to pay would swamp the benefits of letting them choose how to distribute their benefit. This could be dealt with by setting universal prices.

May 15 2013 at 5:50pm

Dan wrote, “Said another way, if instead of giving the poor people $1,200 in healthcare medicaid just gave them $1,200 in cash, wouldn’t they according to this logic rationally spend less than $1,200 of it on healthcare and the balance on something else that benefits them more than an increased level of mental health?”

Yes. I think you are correct. Where are you going with this?

May 15 2013 at 6:29pm

1) Ryan wrote, “This could be dealt with by setting universal prices.”

Please take a moment to recall the economic outcome of price setting for gasoline under Nixon. (Shortages, Long lines, Black markets.) Very inefficient. Now imagine those same outcomes in medicine. You probably don’t have to work very hard to image those same problems since they already exist. For example, over the last year I have seen a remarkable spike in the number of medication shortages across the USA. Every week my hospital sends out an updated newsletter that tells us which medicines are not obtainable. It is now over 10 pages long. Even the most widely used and critical medications are affected. I think it is safe to assume that the Government’s meddling in the market for drugs has finally reached a critical mass. Along those same lines, there are shortages of doctors. I scheduled a patient to see an endocrinologist this week and the wait time is 5 months. These dreadful shortages are happening at a time when government control of healthcare is only partial! If government oversight were to become “universal” through price controls, a one payer system, or any other type of widespread regulation, then those problems get MUCH, MUCH worse. Recall Communist Russia after the fall of the Berlin Wall. If you transplant that horribly depressed economic picture in to the healthcare system in America then you will have a very accurate approximation of the predictable outcomes.

2) Ryan also wrote that health insurance companies can lower costs of healthcare through collective bargaining. He reasoned that individuals who don’t have collective bargaining—like the poor—would see higher prices than patients under a collective bargaining umbrella and therefore receive a smaller basket of healthcare goods, relatively speaking.

On the face of it, your logic appears sound. However, that logic is refuted by reality. Consider the fact that most people who see a physician have insurance. Consider also that the rates insured patients pay for medical care are remarkably high. Finally–and this you may not be aware of–cash-pay patients have the lowest costs of any patient population. Cash-pay patients—the ones who actually pay–get the biggest discounts and the best rates because the prices quoted to insurance companies are inflated prior to any negotiated discount. Thus, collective bargaining has not reduced the costs for insured patients. They pay more!

3) Ryan also wrote that the “…need for healthcare is not uniform. Any level of benefit would give some people more than they need and others less.”

There are very few words which have less utilitarian value in the English language than “need”. What people think they “need” varies so enormously from one person to the next that it is impossible to generalize what a generic “need” stands for. Many patients, for example, feel they “need” addictive, narcotic pain medications, even when they don’t help their pain. Ditto for people who “need” cigarettes, in spite of the widespread knowledge that cigarettes cause cancer and heart attacks.

Since what people feel they “need” is not uniform, then it follows logically that the optimal level of healthcare each person “needs” is equally disparate. Given that reality, it is not possible to legislate an optimal level of healthcare.

4) Finally, Ryan said “Food stamps are better than sending poor people baskets of food rations. Housing vouchers are better than public housings.”

I agree with that assessment. I disagree when he suggest that this same voucher system would not work for healthcare. For the reasons stated above, individual “needs” for healthcare cannot be generalized. The only way to address an individual’s needs, therefore, is to allow that individual to choose how to distribute his resources in a competitive market system. Thus vouchers would work better than [almost] any other method for addressing healthcare needs for the poor. Vouchers even work better than straight cash distributions, since much of that money would be diverted to “needs” outside of medicine [see econtalk conversation about chickpeas @18:02]

The only thing better than vouchers for addressing the needs of the poor is the Free-Just-Competitive-Informed market—where there aren’t any government redistribution systems and people are left alone to prosper in an environment where prices are minimized, quality is maximized, and innovation is rewarded.

May 16 2013 at 10:34am

Excellent podcast, my favorite of the year thus far. To me this study complies with the law of common sense. Access to health care isn’t the end all for good health, so many factors go into it. And of course providing people assistance by covering the cost of something will make them feel better about their financial situation. Nothing better than when a study comes out and gives people the opportunity to expose their bias.

One of the positives of the Affordable Care Act is the market getting the chance of reacting and coming up with better alternatives. Their are some places in the US who have stopped accepting insurance all together and are cash-pay only. As one could imagine the cost of procedures are considerably less.

Jusitn P
May 16 2013 at 10:48am

Cash-pay patients—the ones who actually pay–get the biggest discounts and the best rates because the prices quoted to insurance companies are inflated prior to any negotiated discount. Thus, collective bargaining has not reduced the costs for insured patients. They pay more!

You speak the truth. I had it explained to me this way. If a procedure costs $100 (Labor, exuipment, expenses) they submit it to Medicaid at $500. Medicaid is required to negotiate a price lower…so they both agree to $400 for the procedure. Since law forbids private insurance companies from getting any lower rates than medicare/medicaid, all insurance companies have to pay $400 for the same procedure. It’s very simplistic and there is a lot more going on, but that’s the big picture going on.

Surgery center provides free-market medicine

Russ, Dr. Keith Smith, co-founder and managing partner of the Surgery Center of Oklahoma might be a good person to interview for a future episode.

May 17 2013 at 10:45am


I was simply pointing out that giving someone who is very poor $1,200 in healthcare is probably over-allocating their spending to healthcare. They’d be better off with a check for $1,200 they could spend how they want.

Granted, the issue of “real” insurance does matter. If the very poor person gets cancer and as a society we aren’t going to doom them to death because they don’t have insurance then giving them $1200 in cash is really giving them $1200 cash plus state sponsered insurance anyway (albeit horribly innefficient show up at the emergency room, state sponsered, insurance). It is a tricky business, because someone who is truly very poor would probably rather take their chances on cancer and lead a better life in the meantime. “We” just won’t let them do it.

May 18 2013 at 12:45pm

This is completely off the subject: I’d like to hear a conversation with Russ and Eileen de Neeve about her book, Decoding the Economy: Understanding Change with Bernard Lonergan.

Always a great podcast.
Thanks for honestly confronting ideas with their alternatives for a true discussion without the heated rhetoric so frequent on TV.

May 18 2013 at 1:00pm

Or maybe a discussion with Paul Hoyt-O’Connor about Bernard Lonergan’s Macroeconomic Dynamics.

May 18 2013 at 10:54pm

If I’m not mistaken, studies have demonstrated that Medicaid patients have worse outcomes because of limited treatment options and follow-up. It’s actually better to be “self-pay” (commonly understood as usually “non-paying”) because then the treatment options aren’t limited by the government plans.

David Zetland
May 19 2013 at 6:16am

I think that you missed something in your discussion of creating a similar reduction in anxiety via income subsidies (doubling people’s income to make them equally happy). Why not mandate insurance for everyone and give subsidies to the poor. That’s how the Dutch do it, and it’s cheap and effective.

Cowboy Prof
May 20 2013 at 1:40pm

Overall, this was a pretty good podcast although the guest tended to skirt around the main findings. There was more methodological discussion, which I’m interested in, but it would have helped to have a bit of meat on those bones.

One of my frustrations with this study is that all the findings seem quite obvious. If you give people bags of money, they will have less financial insecurity and some of the stuff they purchase with it (e.g., cholesterol test) might have some positive utility. So, when people took bags of money to the psychologists, we found out that more people get diagnosed with depression than had they not gone to the psychologist with a bag of money. That seems obvious.

It also seems obvious that this two-year study did not produce dramatic insights into health outcomes (as Russ notes). If I have poor health habits, getting a bag of money to take to someone who will tell me I have poor health habits is probably not going to have a big impact on my health habits unless there is some major shock that shifts me off my current equilibrium. I don’t have Medicare and my doctor keeps telling me to lose weight and eat less meat and I look at him quizzically and ask how he can use the words “eat,” “less,” and “meat” in the same sentence.

WHAT WOULD BE MORE USEFUL is to do a comparative study with some other policy other than Medicare to see if there is an effect. The question framed in this study is “Medicare versus nothing.” I would be interested in comparing Medicare to a private charitable program (perhaps run by churches) or some other incentive structure.

Bottom line: The Oregon study is not “controversial,” it is simply banal.

May 21 2013 at 12:39pm

I was glad that Russ pressed the point about bias toward the end. It seemed to me that Mr. Frakt was more than happy to point out the limitations of the study where they didn’t fit his biases while being no less than neutral on those points which supported his expectations. This impression led me to think that his position is driven more by emotions (i.e. feeling that providing universal health care is right) than by facts. I would be surprised if any objective measure of the influences and/or benefits of Medicaid swayed his opinion. That being said, it doesn’t seem to me that a two year study with such limited samples can really prove much one way or the other.

To me, the issue of benefits concerning these programs draws attention from the true moral question: is it right to force anyone to pay money, time or other resources for the needs of others? Since I find it unlikely that I could be persuaded of this being right, discussions about the benefits derived from immoral actions carry far less weight.

Becky Yamarik
May 21 2013 at 1:48pm

Great discussion followed by many interesting comments written here.

It really made me think a lot, and I came up with three real life examples from my own doctor life it brought to mind

1. A Mexican immigrant, age 52 who runs a landscape business in Santa Ana, California has a swelling in his knee. Uninsured, he goes to charity care, is diagnosed with malignant sarcoma of leg, recommended to get amputation. He refuses b/c of cost and also fear of losing his ability to work. Two years later he turns up in our hospital with lung metastases, dying. Wife is disabled, family so poor that the son pushes her in a wheelchair 0.75 miles each way to hospital.

2. An African American patient, age 68, suffers weight loss of 80 lbs and chronic cough. Although insured with medicare, he doesn’t see a doctor b/c he doesn’t trust them due to the Tuskegee study and a feeling that they “just want to experiment on us”, finally his family brings him in to our hospital. Metastatic lung cancer, he dies 2 weeks after discharge on hospice.

3. A hospitalist and palliative care physician, I haven’t done office based primary care since residency ended in 2003. I recently did some per diem work for a doctor on vacation. For the two weeks I covered, I felt that at least 50-60% of the patients that I saw didn’t need to be seen. Things like “I had the flu two weeks ago and I am still weak”, “I got rear-ended a month ago and my neck is still sore”, “I have a cough and I want antibiotics” (this was NOT Russ! 😉

Lastly, back when the AIDS epidemic was just getting started, the govt decided to cover all people with HIV in order to ensure that the virus didn’t spread. Also b/c the people who mostly got AIDS, often prostitutes and drug users, often were uninsured. It’s been pretty un-controversial as a govt program. What if we did sthg similar with people who got something catastrophic like cancer?

May 26 2013 at 10:27am

There is this bizarre myth that permeates the lay public. the myth basically goes: “Send everyone to the doctor, who will discover all manners of diseases in early stages to be successfully treated with large savings to the system”

In reality:
1. The vast majority of people are healthy. Any interactions of these folks with the healthcare system is a waste of money. The singular exception is Pap screenings in appropriate populations. NB: “appropriate population” is different from “ALL women”

2. In the unhealthy population, a large percentage of treatments will fail. Most commonly because of non-compliance. Obesity is the most common example of such. Although there are potential savings in this patient pool, they can never be realized, wasting even more money.

3. The next group is those with untreatable conditions. Again, although they are truly sick, any interactions with the healthcare system for these folks is a waste.

May 26 2013 at 7:15pm

Becky leaves out much more common stories like:

Patient goes for a “routine” check up. Has no complaint except “Sometimes I get tired”. Doc orders a unneeded thyroid panel and an even more useless thyroid ultrasound (38% of us have thyroid nodules-Horlocker, et al). A subcentimeter nodule is detected. Biopsy is performed and low and behold papillary thyroid CA is found!!! The pt undergoes risky surgery and radio-iodine therapy. Pt is relieved to learn he is CURED!!!!!

Ha! What the sucker (I mean patient) doesn’t know is 40% of the whole population has papillary CA and 98.8% of papillary CA’s don’t need ANY Rx (Ito, et al)

Insurance (particularly govt supplied) WASTES money!

becky yamarik
Jun 2 2013 at 10:48am

buttons, that is a great example. . . even more common and wasteful are the millions of men who’ve gotten PSA testing, early stage prostate cancer is found, and the poor man undergoes treatment leaving him impotent and incontinent, and his prostate cancer, which never would have killed him anyway, is gone. Most medical groups, except urologists who will lose lots of money from not unnecessarily treating prostate cancer, are recommending that we STOP testing PSA altogether!
And yet people are completely in an uproar that we should stop testing for prostate cancer. . . it’s crazy. . .

Comments are closed.


About this week's guest:

About ideas and people mentioned in this podcast:Articles:

Web Pages:

Podcasts and Blogs:



Podcast Episode Highlights
0:33Intro. [Recording date: May 8, 2013.] Russ: Given your interests and background, I thought you'd be the ideal person to talk about the recent results that have come out of the Oregon Medicaid Study. And for those of you who haven't heard of that study, it's a very unusual and potentially influential study done by some very high profile health economists. Austin, you've been a very thoughtful commentator on that study as its results have been coming out. I want to start with some background on Medicaid, the program, Federal government program that also works with the states. How does Medicaid work? Guest: Well, let's start one half step before that, which is to say the first thing to know about Medicaid is that it's not Medicare. And I know a lot of people who aren't well versed in the U.S. health system can get confused between the two. So, just really quick, to a first approximation, Medicare is a program for retirees, people 65 and older. It does also cover some people with disabilities and other health conditions. But basically if you remember one thing about Medicare, it's for the elderly population. Medicaid, if you only know one thing about it, the thing you'll know is actually not true; but that thing would be that it's a program for poor people, people you can think of as near or around or below 100% of the Federal poverty level. Which, by the way, I have a 2008 figure in front of me, was about $10,400 for a single individual. So, Medicaid is a program for poor people. However, I think the next thing you should know about Medicaid is that-- Russ: You said that's not true. Guest: Yeah, I'm getting to it. The next thing you'd want to know is you can think of it as really two programs put together. One is that it is a Federally mandated program for people who meet certain conditions in addition to being poor, and these are people who are old, blind, or disabled, pregnant--specific, 'categorically eligible'--specific categories of people. So these people can get on Medicaid if they are both poor and meet one of these categories. And that's not everybody. There are a lot of poor people who are not blind or disabled or pregnant or old. They are just poor. And so the second thing that a Medicaid program can do, and this varies by state, is expand to include those other people who are poor but don't meet one of those categories. And so this will vary by state; and sometimes it involves certain waivers from the Federal government to expand in certain ways; but it's separate from the mandated core of the program. So one mistake people often make is they think: Medicaid; we've got a program for the poor; it's called Medicaid; everyone that's poor is on it; no problem; they are taken care of. Well, that isn't true. In many, many states it's only people who meet certain categories; and this is one of the things that the Affordable Care Act and health reform law is supposed to address. Or was supposed to address. We can get into whether it will. It was designed to include all poor people regardless of whether they are categorically eligible, right now. Russ: So, if you are on Medicaid right now, either because you meet one of the categories or if you are in a state that expands coverage beyond those special categories, you are "merely poor," what are the benefits? What happens to you when you use health care? Does that vary by category, by state? But once you are on Medicaid, what happens to you? Guest: Mostly, your health care is paid for through the state program, and it varies by state what the details would be. There may be some cost-sharing, a few dollars in a co-payment for a drug or a doctor visit. There may be a small premium. And these things can vary by income, so that if you are very, very poor, way down well below the poverty level, maybe you'd have no cost-sharing and no premium. If you are at the poverty level or maybe a little above--some states expand above--maybe you'd have some of those things. But it's basically a health care benefits program, so it would cover hospitalizations and doctor 's visits, preventative care, prescription drugs. What it typically doesn't include though is dental or vision. So you will find many Medicaid beneficiaries or Medicaid enrollees who are getting much of their care paid for but they have horrible problems with oral health, for example--their teeth are just in horrible shape. And that's an issue for them. Russ: So, if I'm a poor single parent and my kid is running a fever and I'm worried that she's got an ear infection, and I'm a Medicaid recipient, what do I do? In my case, I've got health insurance coverage through my employer, and I take my kid to her pediatrician and maybe get a prescription; and that office visit is--I don't know what it is, maybe $20. And I pay a very large premium to have that privilege of a $20 visit; and my employer pays part of that as well. Now, give me that story when I'm a Medicaid recipient. Guest: It's the same story, except, as with many people's plans, there would be a network of doctors and hospitals that would accept the plan; and some may not. And that's true of many people with private coverage as well. So, you could go to one of the providers that accept the coverage and have your care provided there, and largely or entirely taken care of cost wise by the program. Russ: Depending on your state, whether there's a co-pay, the things you talked about before, right? Guest: Yes. So there could be small copays, and that's going to vary by income, by state. So one of the standard lines about Medicaid is: If you've seen one Medicaid program, you've seen one Medicaid program. There are 50 plus D.C., and they are all different in some way. So it's very hard to generalize without basically lying a little bit. And an advantage of that diversity, however, is that it can be studied. Studies have looked at that variation.
8:03Russ: So, let's get to the Oregon Study. It's been a great week, week and a half, maybe 2 weeks for empirical economics. We had the Reinhart-Rogoff Study, which we'll be doing something on in a few weeks here. But the Oregon Study has created a firestorm of comments in the news and the blogosphere. So give us the background on that study and what it's trying to understand. Guest: So, this study, there's so many great things to say about this study. But let's just start with how it came about. So, Oregon had closed a part of its Medicaid program that was for people who were just poor. It's got the mandated Federal categories--that arm of Medicaid has got to keep going. But for budgetary reasons they had closed new enrollment into their expansion of Medicaid. This is in the mid 2000s. And then they found, the elected officials decided and found some funding that they thought they could open enrollment to about 10,000 people in 2008. But they did something that to my knowledge no other state has done, or at least is not common, which is they offered enrollment to people on a lottery basis--they opened a lottery. And people could apply to the lottery and put their name on the list. And then they selected some number of people off the list, about 18,000--no, 30,000 off the list out of the 90,000 who applied, to be permitted to enroll. And this was a random selection. And while this was going on, some investigators at Harvard and MIT heard about it, on National Public Radio (NPR), I think. And they said: Oh, wow, this is a randomized trial. They are randomizing people into Medicaid, or they are randomizing them into the privilege or applying for Medicaid. Russ: So, the 90,000 who applied, they didn't take the 30,000 sickest or the 30,000 richest or poorest. Guest: No. Russ: They just took a random 30,000. Guest: It was random. Correct. And they said: Okay, out of the list of 90,000, you 30,000 can apply. Now not all of those 30,000 were, as it turned out, by the time they could apply, eligible. Some of them lived out of state; some of them, their income was too high at the time of application; or for other reasons they weren't eligible. Some just chose not to apply, maybe because they found other insurance, or for some reason. Actually, about 18,000 who were eligible to apply actually did. And then ultimately when you get down to the study we're going to talk about where they did some follow-up surveying and so forth, the study actually only has about 6,000 in the control group--those are people that are randomized that couldn't apply; and then 6,000 people in the randomized to-be-able-to-apply group, of whom only about 25% actually enrolled in Medicaid. So, it's a little bit confusing. Out of all these numbers, what's at the end of the day when you look all at the data collection, you've got 6000 people in a control group, 6000 people in, call it a 'treatment group', but I want to say what 'treatment' means in this context is were randomized to be permitted to apply for Medicaid. Russ: But how many of them enrolled in the program? Guest: Right, so then you have the enrollment group, which is a subset of that treatment group, and that's 25% of the treatment group, or about 1500 people. So that's what the latest study is looking at. Those are the numbers, the basic underlying numbers, on what they have. Russ: And a little methodology here: So, we've got the control group and we've got people who are enrolled in Medicaid. How do we, the students of this program, the investigators, monitor or assess their data, health outcomes, demographic variables, etc.? How many times? What's the format? How do we learn about them in these two experiences? Guest: There's a number of data sources. So, first you have information that was provided just in the application process. That's very basic information. I'm not quite sure what it all was, but age, things like that. Russ: Sex. Guest: Yeah, just very basic. And then the investigators--they had a lot of cooperation from the state of Oregon, where this took place, and they got administrative data. So, these are things from the Oregon Medicaid system itself. When people enrolled they provided a lot more information and they could look back over time to see if these people had enrolled in the past and get the whole history of involvement with the Medicaid program for the sample they had. Russ: And they know when they visited a doctor, etc. Guest: Yeah. I don't believe they had access to all those kinds of medical records, but what they did have hospital records. I'm not sure they had outpatient records. I didn't see that mentioned. But they did have hospital system records. Hospital discharge data. And they also pulled in, for reasons that are kind of a detail, credit data. Actually, that's not a detail. What is a detail is they got some kind of food stamp and cash/welfare system data from the state as well. Russ: For the non-enrollees, the control group, how did they know stuff about them? Those 6000--so there's 6000 people who didn't win the lottery; and now they get contacted presumably by some health economists or the state of Oregon saying: oh, you've been chosen to participate in this survey. How did we get information about them? Guest: Right. So all the things I just mentioned are administrative data. And they have that on everyone. They had all that stuff I just mentioned on everyone. Then they went to both the treatments and control groups with mail and phone surveys. The mail and phone surveys--this group published a paper last year and this was done prior to that, and I don't remember the date but let's just say it was in 2009, 2010 or something like that. So they surveyed everyone by mail and phone and they got a 50% response rate in that survey. Russ: Which is very high. Guest: Yes. Russ: For a mail survey it's ridiculously high. Did they follow up by phone? Guest: They followed up by phone. Yes. And this is where they got additional data on health care use and cost and financial strain, health status and demographics; but this is all self-reported. So, they weren't going to these people's doctors and asking them; they were going to people themselves and saying: Have you visited the doctor in the last number of months, or year? What happened there? It's all structured, but this is how it's done. Russ: But they didn't show up with a blood pressure cuff and measure things directly from them. Guest: Not in the mail and phone survey. Russ: Correct. Keep going. Guest: So, the data I just talked about, the administrative data and the mail and phone survey data, that was all available a year or more ago; and this group published a paper a year or more ago. The most recent paper includes that plus in-person interviews. And the in-person interviews was where they collected biometric data. So, this was blood pressure--they actually measured it. Blood samples, so they could get cholesterol levels, blood sugar levels. So this is a lot of really granular detail on the actual health of the individuals. I don't know quite how they did it, if they did it with nurses and so on, but it's almost like having a doctor visit, in a way. They just measured some basic things. Russ: Reminds me of the National Football League (NFL) Combine--they did a 40-yard dash, and then an IQ test. So, they met them face to face; they gathered more data. And this is ongoing. Is that correct? So this experiment is still going? Or is it over now? Guest: The data collection is over because I believe the state has subsequently expanded the program so the lottery is not necessary; they are not randomizing any more and everybody who wants in can get in. Russ: Did the economists protest? Hey, you're ruining our study. Come on. Guest: Yeah, I know, it's true. Russ: That human subject regulation is a bummer, isn't it. Guest: What is forthcoming is there was some more data collection and analysis beyond what is reported in the latest paper. That's ongoing, and they expect to have more results. One thing they could do, I believe they are going to do but I wouldn't swear by it, is: you can monitor, for example, mortality down the road because you can get public death records. And so even though these people were once randomized and now the study is over, you can look 3, 4, 5 years down the road and see if a couple of years' experience on Medicaid, versus not made a difference on things like mortality. That could be done. Whether they are going to do that precisely, I'm not sure.
18:10Russ: So, why has this recent set of results that has come out in the last few weeks, why have they been so controversial? What were the findings? First, rather than talking about why they are controversial, why don't you summarize what the most recent study found, the most recent study of these data. Guest: Okay. So, there were a number of findings that were statistically significant, and I think, to my knowledge are not controversial. And these are areas--I'll just give you the broad categories and we can dig into them. So, they did a battery of analysis on financial protection, how much does Medicaid protect you from high bills and devastating high costs of health care and so forth? They have a battery of results on access and prevention, so: How much easier was it for you to see a doctor and get preventative screening and so forth? They have some results on health outcomes, both mental and some physical health outcomes that were statistically significant. And they have results on cost, in this report. And so, we can go into some details on those. Those are not generally controversial. There's then a set of results on some additional[?] physical health outcomes, physical health measures that were not statistically significant. And it's those that are really the subject of debate. Russ: And summarize those. Guest: So, those are things like the effect of Medicaid on blood pressure, changes in blood pressure, cholesterol level, blood sugar control--which is related to diabetes. And medications for these things. And so when you look at all these things--and they are all, several of them are measured in a few different ways--but when you look at the table in the paper there's maybe a dozen or so results that are staring at you, all about physical health. In one table. And none of them are statistically significant. So it appears as if the study is saying that Medicaid is not able to have an effect, across all of these health measures. And that--the discussion over whether that's what the study is saying--is what the debate is about. Russ: So, we'll talk about that and then we'll talk about why it matters. But let's first talk about these results. I've argued, as have many people, that we ought to be more worried about health care than health insurance. Obviously, health insurance has certain aspects to it that are distinct from health care. But they are positive. Like comfort and not fearing financial distress. But it's also clear that health care insurance doesn't keep you literally healthy. There are other ways to get healthy; there are a lot of factors that aren't related to medical care and the use of the medical system--nutrition, genetics, stress, lifestyle, etc. So this is a big, messy area. But my general bias, which I'll get on the table, has always been that I really don't want to expand the current system that we have, that allows people to spend other people's money, which then pushes up the use of medical care often without value and makes the whole system more expensive for everybody. Now obviously there are a lot of pieces to that that I know you don't agree with. But I want to start with the point that it's not so shocking to me--even though, I'm not a big fan of Medicaid--it's not surprising to me that in a 2-year study that it didn't find very much of an effect. That's not the claim I would think of the value of having health insurance, that over 2 years their blood pressure is going to drop, your blood sugar level is going to drop if you are close to diabetes or have diabetes. Are these really the measures that we want to judge the value or lack of value of federal and state subsidy to poor people? Guest: Well, um, should we use these measures? Or are you asking, are people, did some people think? Russ: Both. Guest: So, I'll take the second one first. I think that there's a great diversity of claims about this study, and in fact, though I haven't looked back, myself, carefully, I probably suggested a year ago that the proof was in the pudding of whether health insurance affected health in results just like these. In other words, I was looking toward this study to put to rest this discussion whether health insurance facilitated an enhancement of health or not. Now, when I said such a thing, I was not aware--I could have been aware but I wasn't aware--of the precise measures they were going to look at. I was certainly not aware of what some of the baseline rates I think were and how much power the study had to detect changes if there were any. And so--now seeing the results and looking at the actual numbers that they had, I'm actually not surprised that they weren't able to show an impact. Now, are these the right things to look at? No, I don't think so. I'd say some of them are and some of them aren't. But I'm not a physician, so it's a little hard for me to judge. Look at diabetes in particular. We know clinically, from clinical evidence, that taking certain medications if you have high blood sugar or diabetes really affects those things. I mean, it really moves the needle on your blood sugar, in terms of blood sugar control. And if health insurance facilitates greater access to those medications, and people follow through, then you ought to see a result. As for blood pressure and other things, maybe you can make the same argument. That's how I think the basic causal chain would go.
24:57Russ: And what's the bottom line for you of these latest findings? Does it change any of your priors? What does it do? Guest: It changed my prior on what I thought this study was designed to detect. So, I spent the better part of this week digging into details on the design of this study and how it was powered or how many individuals were actually involved, and what that meant for what it could measure. Russ: Talk about what you mean by 'powered.' Because that's a statistical term that most people aren't familiar with. Guest: Right. So, 'power' is the, strictly speaking, it's the probability that--the way it works is, you are supposed to say, before you do a study: we're going to do a study of, you know, insurance--of course, not insurance--and we hypothesize in advance that insurance will, let's say, move someone's blood sugar from, let's say, whatever it is, the baseline, to some lower value. If it's elevated, it will bring it down. Insurance will have that effect; it will act through people who obviously have high blood sugar and take drugs; not everybody is like that, but on average it will have this effect. And you say what that effect is in advance. And you design your study so that it has enough sample to capture that effect. If you design a study with three people and you are expecting everybody to be cured from cancer, you are just not going to find that. Russ: Because that's too big an effect to expect likely. Guest: Too big an effect. Combination--too big an effect and too small sample. If you instead say, I have three people with certain-stage cancer and the intervention is I'm going to give them a certain drug, a certain radiation treatment, and one out of the three will be cancer free in 2 years, say--a 30% response rate. Well, maybe that's reasonable to expect. But you wouldn't expect to be able to measure it with very much precision with a sample of three people. Russ: You want 3000, or 300. Guest: Yeah. That's exactly right. So, what is that number? You can calculate that number in advance. It's actually very simple. You can do it with online tools or you can do it in physical software. And this is a standard calculation for any application for any application to NIH, National Institute of Health, or any study within the Veteran's Administration (VA) that I do. You have to demonstrate that you have "power" to detect the hypothesized size that you think is reasonable. So you suggest that this intervention will have such and such an effect; then you do a calculation saying: If it has that effect, we have enough sample, we are going to do enough data collection, or our data base is already big enough, such that conditional that it has that effect, we will be able to distinguish that from no effect. We have the error margin small enough. Russ: So, just to take an everyday example for people who aren't used to these arguments. Let's suppose my hypothesis is that men are taller than women. Which most of us are pretty sure is a true statement. So, let's say we want to show that. If we take a sample of 6 people, it could be that the 3 men you choose just happen to be short men and the 3 women you choose just happen to be tall women, and it wouldn't be a very reliable finding if you found that the women were taller. But as you go to 6000 men and women, the odds get larger that the finding that men are taller than women is more reliable. I gave a bad example at first--I should have said if you chose tall men and short women, you wouldn't know if that was true by chance. It could be that you just chose tall men and short women. But if you chose 6000, 3000 men and 3000 women, the odds that that's true by chance, your finding gets a lot less likely. So, one way to talk about--you've written that this study was "underpowered." Another way to say that is the investigators presumed that the effects would be larger than they turned out to be. Is that correct? Guest: Uh, they could have done that. I'm not sure that's exactly what happened. They could have either presumed very large effects. The problem with that hypothesis is that the effect sizes for which they are powered are enormous. I did actually do this calculation. Russ: What does that mean? Guest: Well, for example, one that I did was, what this study found, the point estimate for the proportion of individuals whose blood sugar dropped below a value--there's a value of blood sugar above which you are considered diabetic or near-diabetic. And they found that the proportion of people on Medicaid who had a blood sugar below that level, so they dropped from a diabetic or pre-diabetic level, down, was 20%. So, this is the point estimate. So, 20% of the people--fewer people in the Medicaid group had elevated blood sugar than in the control group. That's the point estimate. but, big error margin. That's the point estimate. 20%. I computed that, if that number had been 4 times larger, so 80%, if it had an 80% effect, then they would have been powered to have distinguished that from happening by random chance. Russ: It would have been statistically significant--it could have been statistically significant. Guest: Right. They would have been powered for that. But an 80% effect rate from just giving someone insurance--I don't think the investigators were thinking in advance: We expect 80%. What I think is much more likely--and much more likely in economics: You know, economists don't generally do power calculations. In fact, before a year or two ago, I had never done one. Because what's more typical is for an economist to say: We're going to study this issue; we have some data, which is fixed in size. It's the size of the data base; we don't do surveys, generally. We're stuck with it. You know, in macro--there's only so many countries and so many years. I'm not going to suddenly cook up 10,000 more country-year combinations. So, I'm just going to go in and see what I can find. And some of it will be statistically significant, because it has a big effect; and some of it won't be. That's just end of story. We don't do a power calculation because we can't change the effect size. That's just "nature." Or the system. We can't change that. And in economics if you can't change the n, the size of your sample, there's no point in a power calculation. Except one point of the power calculation to do is: Is this even worth investigating in the first place? Am I likely to come up with informative results? Are my results of any value? You could know that in advance, if you thought you had a reasonable guess at an effect size. Or at least a bound on it. It's just hypothesis. What I think is more likely in this circumstance is: Investigators had a great opportunity. It's a great study that did a lot of good things, the design is fantastic; we could go into some of those issues, but I think we could. And they actually pre-specified all their analysis. So, this wasn't a fishing expedition. They put online: Here's what we are going to do. Russ: Yeah. Guest: Exceptionally impressive. And then they just went out and did it. If they did a power calculation in advance, I've never seen it. I don't think it exists anywhere publicly. And chances are, I'm not sure I would have.
33:03Russ: But the bottom line is that on a number of predetermined, important health measures--and I would actually call them proxies, and maybe we'll come back to that later, but it's not actual health. It's things, like you said, biometric measures. They are things like blood pressure, blood sugar, etc.--they didn't find any effect. Now, they did find some effects, though. So, why don't you talk about what they found that was significant. Guest: Well, the headline big effect was financial protection. So, out of pocket medical expenditures that were above 30% of income--so, this is their definition of a 'catastrophic expenditure'--that was basically dropped to zero. I think it dropped to 80%, from a baseline rate of 5.5% down to 4 point something percent. So, Medicaid virtually wiped out the chance that, you know, medical expenses are going to clobber you. That's completely predictable, given what Medicaid is or health insurance in general. The likelihood of medical debt came down by 20%, and the proportion of people borrowing money or not paying off medical bills was cut in half. So, a lot of financial protection. That's actually to be expected and those are reasonable. Big increases in access; and this kind of relates to financial protection. You know, if you don't have to pay for care and it's "someone else's money"--it is--you are going to much more readily go in and get preventative screening. So there are big reductions--or big increases, I'm sorry--in women getting Pap smears (Papanicolaou tests) and mammography. A 20% increase in women getting "all-needed care." I'm not sure if that's self-reported or if they defined what they meant by 'all-needed care.' But, big increases in access. And then there's some significant increases in health outcomes. The big one there is depression diagnosis. So, right after the lottery, they could look at who was diagnosed for depression. And they found that there was an increase in probability of depression diagnosis right after the lottery. But over time, the people on--so, why would there be an increase in depression diagnosis? This is just an access thing, so more people are going to the doctor and they are getting diagnosed. There's higher rates of diagnosis. One hopes, but I don't know, whether all those additional depression diagnoses are real--these people are actually depressed--or whether it's just an increase in diagnosis. Russ: Yeah, we've spent some time on this program, as I think you know, talking about the challenge of correctly diagnosing depression; the interests of the pharmaceutical industry, and encouraging diagnosing of depression, etc. So that's a tricky thing. But they did find an increase in diagnosis. And then a decrease over time in people screened for diagnosis. So the increase in people diagnosed, and then with their interviews they could do a screening for diagnosis later, through a series of questions that were validated for this purpose, and found that the screening-positive rate came down 30%. So, people were diagnosed with depression and then it came down by 30%, whether they were actually by this measure depressed later on. So this is an improvement in depression from, you know, post-lottery diagnosis rates. Russ: Which we don't know exactly what the source of that is. But it appears to be, it's correlated with being in the Medicaid group. So it could be related. Guest: Right. The design here--I think the investigators would be comfortable saying it's a causal effect based on the design. But we could get into whether you are comfortable with using causal language. But just to finish out the mental health thing, they got a statistically significant improvement in self-reported mental health, and in the proportion of people saying their health was good or better than the prior year. And we talked about diabetes. Well, there was an increase in probability of diabetes diagnosis and medication, and those were both statistically significant. And you might wonder about cost of all this. There was an increase in cost, expenditure to the state, of $1200. This isn't all expenses. Increase in cost overall. So, people who were not enrolled in Medicaid, they were just spending out of pocket. But the increase in overall cost of care due to Medicaid from all sources is $1200. Russ: Per person? Guest: Yes, per year. Russ: And that's from their increased use of mammography and all their other tests, etc., that they had access to, presumably. Guest: Right.
38:20Russ: So, let me play the skeptic here. A while back, and you'll tell me when, there was a famous study that was vaguely like this, not exactly like this. 'Vaguely' is not the right word. But it was designed to see how people respond to health insurance. It was done by the RAND Corporation, and it was shocking and controversial. And remains controversial. They found that people who faced--correct me if I'm wrong--lower prices used more medical care. As economics would predict. But their health outcomes were basically the same. People did not get the benefit of those programs. You could say it's not surprising that people who win the lottery get in a better mood eventually, that they have fewer financial problems. That's not a very good test of Medicare's [Medicaid's?] efficacy. It's a result of giving people more free stuff. It means they are going to have fewer financial problems. In other words, maybe than expanding Medicaid, what we ought to be doing is giving people money. So, as a skeptic--which I am--the RAND study, and how this study--there have been other studies, Levy and Meltzer, 2008; Kronick in 2009--that seemed to suggest that health insurance does not have much of an impact on health. Maybe it makes you feel better, maybe it helps you sleep better at night. That's not that is not important, but insurance is a relatively expensive way to get those outcomes. And so the question is, as someone--and you are not alone; there are a lot of people who believe that we should be expanding health insurance and availability in the United States, and it's either through public programs like Medicaid or other ways we could do it--where's the evidence? Guest: So, I'm going to disagree with one nuance in what you said, and then agree with a lot of it, and then focus in on where the action is. So, the disagreement--and this is all going on when this latest study came out in the last week--the disagreement I have with what you said, or maybe with what you implied--you may not have quite said it, but if you said it I'll disagree with it--I'll put the words in your mouth-- Russ: It's fine. Guest: The disagreement would be: This latest study, because of the power issues, basically the low sample size combined with the low reasonable expected effect rate, effectiveness basically of Medicaid, is uninformative on key physical health measures. It's just, the error bars are just too wide. Now, that doesn't mean it's possible that Medicaid had zero effect, that it's possible. I think what it more likely means--well, it means what I said. It means it's uninformative. But I think what's more likely true, Medicaid may have a small positive effect and this study could not detect it. However, you are correct to point out that there's lots of other work we can look at. And, including the RAND Health Insurance Study; including the Levy-Meltzer study. And others. And I think when you look really carefully at those--and in fact the publications bear this out, and what the authors say is that, you know, where the action is--for most people, health insurance doesn't do much. Because most people are healthy. Health insurance doesn't do much for your health. But it does a lot for spending; it does a lot for access; it does a lot for financial wellbeing and peace of mind. So, all of those things--I agree with that. That's the part I agree with. Or people who happen to be sick. Or poor. And/or poor. Both, really. Public assistance for taking care of their health care does have an impact. The RAND study showed; Levy and Meltzer point that out. And so that's where the action is. Now, I think you are right to say that there are other ways to help those people. And the way Medicaid is currently configured may not be the right approach. And I think that's a completely valid discussion. But just on the evidence alone, I think the evidence is consistent with the idea that there are people for whom health care is helpful. But that's not really most people when they are healthy. And in our current system, health insurance does facilitate access to that help.
43:03Russ: Let me just add one more troubling piece to this, which is: You hear a lot about the value of preventive care. And I had Eric Topol on this program a few episodes back. And he made a shocking observation to me--this relates to this study, which is the Oregon Study--which is that the statin drugs which reduce cholesterol, they don't reduce it for most people. And for the ones that have reduced cholesterol, it's not very well correlated with better health care, better health outcomes. So, we have all these drugs that reduce cholesterol, but they don't, it's not clear that they reduce your risk of a heart attack, even though cholesterol has something to do with a heart attack. Which is weird. But that's the reality. And so, to me--and I'm going to put this issue back on my foot in a minute, because I'll be done picking on you in a second, but for me, the challenge is: Shouldn't we just be taking care of catastrophic health care risks rather than pushing the country toward what to me is similar to what we do with Social Security, which I think is nuts, which is: Everybody gets it. Rich and poor, we all get it. We all contribute, well get it back. And that allows us to do some subtle redistribution within donors, payers, and recipients. And similarly, we have a bunch of people who have good health, and bad health; we have rich people and poor people. Everybody's going to get free health care. If you use the system without any free worries at all, even the most of us, most of the time, don't have that much value from that privilege. And we depend a lot on it, because we've lowered the cost artificially, the price artificially, and encouraged usage as a result. Which pushes up the cost. So, what's the--many of the people who designed and executed this study are major proponents of the Affordable Care Act. Including Jonathan Gruber of MIT. Where's the evidence for their viewpoint, given what we've found so far? Guest: Um, well, apart from Jonathan Gruber, I'm not actually sure what the position on the Affordable Care Act is for everyone on the study. He was involved, but he wasn't the principle investigator. Russ: Fair enough. Guest: And this study--I in large part agree with what you said about providing public benefits for everybody, rich, poor, sick, healthy, so forth. But when you have finite resources--even accepting, which you may not, but even accepting that you are going to public funds for support, those funds are finite. Society is only willing to bear so much. Witness the debate over tax rates, so forth. And given those constraints, it's perfectly reasonable to say: Well, we're not going to provide free health care to millionaires; but everybody gets Medicare. Russ: Not just that. We are not going to give my wife, and myself, an incredibly low price for our pregnancies and deliveries, which, we had some control over, strangely enough. Guest: Yes. Russ: It's not catastrophic; it's not unexpected. It's not insurance. I go to the doctor every year, every two years, depending on my schedule and whether I like it. That's a checkup. That shouldn't be part of insurance. Guest: Right. So there's a lot one can debate. There's a lot of things bound up in all this and that would take maybe 2 more hours, if not 1. But where I was kind of headed, was, so, we spend a lot on Medicare, and that's where everybody wants to reach a certain age regardless of their income and assets, so there is some means testing of the premium. But you know, nevertheless. We also spend a lot on, or don't collect taxes on, employer-sponsored health insurance. Russ: Which is nuts. Guest: And there it's even more perverse. It's not only that rich and poor people alike are getting benefits. Rich people are getting a bigger benefit. Russ: Correct. Because they have higher marginal tax rates. Guest: Correct. Very strange. It's hard to imagine why you would design it this way. Now, what we are spending out of public funds on Medicare per person is about on the order of $10,000 per person. And what we are forgoing in tax collection on employer-sponsored insurance is about $5000 on average for each insured worker in a family. That includes family plans, too. Now, what this study, and what really the debate is about in states right now, is about poor people. I mean, legitimately poor people. Objectively poor people. It's not a mix of poor and rich. It is a mix of healthy and unhealthy. But this is a mix of poor people. And the cost, under the Medicaid program of providing a benefit to them that, well, we can argue is it like employer-sponsored health insurance, is it like Medicare? That could be a debate; maybe it's worse, maybe it's better, maybe it's the same; but it's akin to it. Maybe it's basic, apart from dental and vision perhaps, but it's pretty fundamental, pretty standard benefit package. The cost of that, for a variety of reasons, is extremely low--$3000, $4000 per person. And these are all poor people. And the debate right now is whether states right now, well, with help from the Federal government, expand the programs to cover everybody under that so that all poor people have at least this level of protection. And, you know, is this study informative on that question? Um, well, I think it's more informative on the question of: Do poor people benefit from assistance? And in this case the assistance was a specifically designed Medicaid program in Portland. In and around Portland. I didn't mention that. All the in-person stuff, so all the data in this study was in and around Portland. Even though the expansion was state-wide, this focus, this study was focused on the Portland area. Anyway, it was designed around that program. But states have some flexibility to design different varieties of Medicaid. They may not have as much flexibility as you or I would like. But broadly I think it does address the question whether poor people benefit from some assistance. And I think it's clear they do. And that's what you'd expect. Is the nature of that benefit worth the cost? You know, that's a point one can debate. Could that benefit be reconfigured or delivered in a different way that's more efficient or helpful? No doubt it could be.
50:35Russ: So, let me put the issue on my foot--I don't know what the right metaphor is, but some of this discussion--and I'm not talking about our discussion; I'm talking about the general discussion on the blogosphere of the reactions to this. There's a lot of jumping up and down by one side; and a lot of 'oh, it's no big deal,' on the other's. So, the people, like me--and this is not my reaction, as I've made clear, but a lot of people who don't like the Affordable Care Act or who don't like Medicaid generally, they've been jumping up and down saying: See, this proves we've been right all along; Medicare [Medicaid?] is a waste of money. The people who like the Affordable Care Act, like Medicaid, want to see it expanded, say: Oh, it's only two years, it's underpowered, it's one survey. It reminds me a little bit of the recent empirical work that's come out about the minimum wage that says it doesn't have any effect: doesn't reduce employment, doesn't hurt low-wage, low-skilled workers, it just gives them a nice raise. And I have to say, when I see that literature my first thought is to explain it away. Because I believe that there is an incentive effect to employers of making workers more expensive. So I say things like: Well, you know, when you have a relatively low minimum wage, as we do, where only a few people are affected by it, it's not surprising that when you raise it by a relatively small amount it's still only affecting a small number. It's going to be very hard to tease out the effects because most people simply aren't affected by the law. Obviously if the minimum wage goes from $7.25 to $9.15, it's not going to affect your salary or my salary or my employment or your employment. And so, an econometric study that tries to evaluate the impact will often, I would say, might not find any impact. Of course, all the studies that found a big impact, which is what the literature was--my side, those around, said: See, see; it has a big impact. And when these new studies come out, they say: Well, you know, it's a small population, it's a phone survey, the methodology is wrong. So I just want to reflect on the fact that it's very easy to over-exaggerate the significance--I'm using that in the non-statistical sense of the word, the importance--of any particular finding. Because of confirmation bias. So, I'm just curious if you want to just react to that, in terms of the people you know and have talked to, the people you blog with. Is there some hunkering down? And we saw the same thing with Reinhart and Rogoff. The people who have been talking about debt: Oh, well, the result still holds; those who are worried about debt say the result still holds. People on the other side said: See, we told you all along it's a sham; we don't have to worry about debt. So, just reflect on that. Guest: Well, I think that's right. I think there is some hunkering down and shifting of emphasis, just focusing on the statistically significant findings and sort of explaining away those that aren't. I've seen that. I don't like it. I don't like to see that. That's not how I've approached it. The way I approach things is, one study is not definitive; you need to look at a body of work. And one would hope that body of work, using different methods, different data, different people--certainly different people, hopefully people of different ideological persuasions if possible--if they all are kind of pointing in the same direction, maybe not all but a preponderance in the same direction--it really increases you confidence that that's the right way to think about it. Having said that, I think you also have to weigh the methodological strengths of each study. So, in this case, it's a randomized trial, randomized control trial. Albeit with some leakage and crossover and so forth that they address with a statistical approach, quite reasonable and accepted. And I think for that reason, this study carries a lot of weight. However, one of the limitations of this study--and this is something that I was a little bit distressed to see; very few people recognized, on either side of the debate--one of the limitations is: It just didn't have enough sample for certain questions. Had enough for some; not for others. And, you know, I don't think anybody, no rational person, would want to base a decision on under-sampling. You get some examples earlier. If you are going to try to assess something, you want to make sure you sample enough of the world to be confident that you are not just reacting to noise. So on some questions here, what is reported is not that much better than noise. Now, it is better than noise because they had some sample. But the error bars are really big. And so, and this is something you can just compute--how big a sample would they have needed, and therefore this one is underpowered and that one isn't; and it's objective. You can do that on every study. You can go back to every study and do that if you want. And so I was a little bit, I've been a little bit uncomfortable with some of the responses to this, either accepting the results that aren't statistically significant as informative--accepting them as informative or more informative than I think they are. I think some of them are relatively uninformative.
56:27Russ: So, here's how we could imagine life working. It doesn't work this way, right? But we could imagine the following. And again, I'm picking on you a little bit; but I could easily pick on myself. Guest: I've been much more picked on. Russ: Yeah, it's mild. So, here's the world we could live in. So, you said the sample is underpowered. Again, even thought I like--this result confirms some of my biases, I think 2 years is a very short time. I know some people are confident it would be a long enough time. But let's suppose I said to you: Okay, we are going to expand the time span of this study; we are going to run it for 10 years, not 2. Maybe 30 years. Let's run it for 30 years. Let's quadruple or sextuple the sample size, make it as big as you want. Make it 50,000. We've got 50,000 people here, 100,000 people, in each group. Run it for 20 years. And we'll make a deal. Along the lines of former EconTalk guests Robin Hanson or Bryan Caplan--they are both big on bets. And we'll say: Look, if it comes out that even then it doesn't have any effect, would you change your mind? Guest: Oh, yes, I would. Russ: Well, you say you would. And again, to put the shoe back on my foot: Well, if you raise the minimum wage to $25 an hour, well then you'd find an effect; and I have to be honest with myself. When they raised it from $5-something to $7, I think it was $5.15 to $7-and-a-quarter, I would have thought there would be a big effect. Some people claim there is. I think it's pretty hard to tease out of the data. So, the question I think for most of us, when we get these kinds of results that don't confirm our priors, you usually find a way to say: Yeah, I think the methodology; looking at the wrong measures; they didn't do this or that right. It's very hard to find a definitive study. It's not the way the world is. Guest: Well, I don't know how many--I think you are right. And probably everybody, almost everybody says they are not biased. Russ: Yeah, they do say that. I like to admit; it's one of my thrills in life, admitting I'm biased. Guest: I will admit, and this has happened to me: I actually like it when what I thought was true is overturned by some evidence that I'm convinced of. I actually really like that. I would have been more pleased to have this study come out and say: Well, we had the power to detect minuscule improvements and we couldn't even find that; look at this error bar. Russ: But you're unusual, perhaps. Guest: Well, a number of people reacted that way. If you just look around in the early, first few days after this study came out, a number of people said: Okay, we're just basically willing to agree that maybe Medicaid doesn't have a big effect on these health measures or on some basic health measures that we thought it would; but look at the financial benefits, and look at the access, and look at mental health. The mental health result, that is really--well, provided you believe what it's saying, and many people do, it's really big. Even if half of it is true. It's a big result. There's a lot of well-being there. In fact, in a prior paper, a year ago, the authors estimated--they did some back of the envelope calculation using some other work--that the improvement in mental well-being, if you wanted to get that level of improvement from an income enhancement alone, you'd have to double income. There's a big effect. Russ: The only problem with that result is it suggests that, for people who are switching jobs and are going to double their salary, they would still take the job if the employer said: Hey, we'll give you free access to Medicaid. But I'm being facetious. Obviously, if that's a real effect, that's quite extraordinary. There is the question of whether there are other ways to achieve it. But that statement about doubling incomes suggests it would be very expensive to achieve it in other ways. We'd have to check if that's reliable; I don't know. Guest: Oh, sure, I'm not saying we should base policy on that alone. But it's just a way of interpreting the result. In any case, as it turns out, I just don't think this study is as informative as some people think on certain measures. I did this power calculation--it's on my blog--in fact, I have two posts about it; and I was really worried I did it wrong. Because I don't do power calculations very often; and it can be tricky, getting the statistics right and so forth. And so I did it, and what I calculated was-- Russ: I read the posts; you are really cautious. Very cautious. Guest: Well, the result was surprising for me. It said the sample would have had to be 3.5-5 times bigger. And I'm looking at it saying: How can that be? Five times bigger--that's a really strong statement about how underpowered this was. And so I calculated a couple of different ways. I went online to get a different tool so that--maybe I was using data incorrectly. And I had some biostat people look at it. And they all checked off. You know, they do this like for a living. And another professor I found on Twitter had done his own calculation; he said that it was right. And another guy--this is a great thing about blogging--another fellow in the comments--I didn't just do this by formula, I just plugged it into, you know, these online calculators. This other guy said: Here are the formulas. And I just worked through the math, and I'm getting a different result; and we went back and forth and figured out why he was actually using the formulas to calculate something different than I was; and so we worked it all out. The point is, it is really a valid, objective statement that it was underpowered on these questions. I was surprised by that. I didn't want to believe it myself. But it's just what it is. Russ: And when I express my skepticism about our ability to deal with bias, I don't want to suggest that people can never confront things honestly. Obviously, we do sometimes face evidence that forces us to change our mind, or sometimes when we are skeptical about the importance of a result, we should be. Maybe it's not informative. And of course, again, for those of you who haven't been following this, you might [?] start to notice it in the paper and in the blogosphere, it's going to be an issue that continues to get discussed.
1:03:11Russ: I want to close with some factual stuff, because I don't know. What does the Affordable Care Act have to say about Medicaid? Because I know that's going to interact with these findings. People are going to be yelling about them in the next election and elsewhere. So, what does the Affordable Care Act encourage or require at the state level to do? Guest: I'm really glad you are coming around to this, because this is the key question. And do you mind if I just take a minute to do the history on this? Russ: No, go ahead. Guest: So, the designers of the law, and as it was passed, the whole idea was every state was mandated to expand Medicaid up to 138% of the poverty level. You'll see 133% written, but there's a 5% disregard, so it's effectively 138% of the poverty level. And every state had to offer Medicaid to everyone with that income or below. And then, this was debated in the Courts as possibly too coercive. So, there was a claim by some states that this coerced states into doing things they don't want to do, and the coercion here was that if states didn't make this extension, the Federal government could withhold all funding for all Medicaid, even the existing part of the program. So, either a state does this extension or they have to wipe out their Medicaid program, was basically the stick behind this. And if that's the deal, there's probably no state that going to walk away from the extension. It's very coercive. Not only would it affect a lot of people in the state, but it would devastate the health system. There's a lot of money flowing to hospitals and doctors, and a state legislature is not going to walk away from that. So, this went all the way to the Supreme Court; and the Supreme Court came out with a ruling last summer that said: Well, everything's fine--you know, they rule about a lot of things about the Affordable Care Act, including the mandate and so forth--basically, everything's fine with the Affordable Care Act, except we don't like this Medicaid coercion thing; we don't like the way the extension is done. Let's make it optional. States have the option to expand under the way the law specifies; or they could not extend, and just keep the existing program, leaving many people without the option of Medicaid. And so now every state is deciding whether to expand or not. Now, what 'expand' means is not just one thing. There's actually quite a bit of room in terms of how Medicaid is specifically designed. Arkansas, for example, instead of expanding in a traditional public-program way, they've decided to expand by just having all of the expansion population be eligible to go get private insurance through the health insurance exchanges that are going to be set up. So instead of having a separate public Medicaid program, they are going to put all those people on the exchanges. And there's a whole debate about: will that cost more, and what are the advantages of this and the advantages of that. But the point being, states have the option to not expand Medicaid at all, or expand it in some--there's a range of options they might consider in how they design their expansion. Russ: And let's close with your thoughts on how this study is going to affect that outcome. Guest: My prediction is this study will be used in part of the debate. It's kind of like the Reinhart and Rogoff work, where you can say: Did this study influence policy or was it just used to justify positions that would have been taken anyway? But I think you'll have this study cited by state legislators and others debating whether they should expand or not, and some of them may cite it and say: Look, this study showed that Medicaid didn't improve physical health; or they might even say it didn't improve health at all--here's the study. And I'm sure some people will use it to say: Look at the result on depression; look at the financial benefits; this is hugely valuable. And I think very few people will say what I would say, which is: This study showed some positive benefits for Medicaid and this study was uninformative on some others. And meanwhile, the actual choice here is not some other thing like giving people cash--that's just not on the table. The choice is whether people get some assistance or none--poor people. And I come down on the give-them-some side rather than none. But I think very few people are going to use this study in that way. Russ: Thanks for helping us understand it.

More EconTalk Episodes