Intro. [Recording date: September 7, 2023.]
Russ Roberts: Today is September 7th, 2023, and my guest is Elie Hassenfeld, Co-founder and Chief Executive Officer [CEO] of GiveWell. Elie, welcome to EconTalk.
Elie Hassenfeld: Russ, it's a pleasure to be here.
Russ Roberts: Our topic for today is GiveWell, the nonprofit you started with other folks in 2007. Let's start with how it came to be and what it does.
Elie Hassenfeld: Yeah. So back--first, in 2006, I was working at a hedge fund where I'd gotten a job right out of college. And, after having been there for a couple of years, a friend of mine, Holden Karnofsky, and I both wanted to give to charity. And when we went looking for information on what charities do and how well it works--essentially what can you get for the dollar that you give?--we were really surprised that we couldn't find useful information that would help guide our decisions.
As an example, I was interested, at the time, in giving to clean water in Africa, and I remember charities telling me things like, '$20 provides a child water for life.' And, I would ask them, 'Well, how do you arrive at that number? What exactly is the money I'm going to give buying? And, how do you know? And, when you say for life, that sounds like a long time. How do you know it lasts that long?'
And they really didn't have answers to those questions.
And so, after a while of struggling and being frustrated with the information we were getting, Holden and I found ourselves fascinated by the questions that we were trying to answer. And after about a year of this work part-time, decided to start GiveWell as a full-time project in 2007. And, our goal, then, was to be a resource to donors like us: people who were working in the private sector, trying to give away money.
At the time we were thinking not a whole lot of money, people who probably didn't have staffs of their own. This has now changed. And, you know, we launched with that in mind.
And now, we evaluate--we've been around for 15 years--we evaluate many organizations every year. Our goal is to identify outstanding organizations who use funds to do a lot of good with the funds they receive. Over the last 15 years, more than a 100,000 donors have given more than a billion dollars to our recommendations.
And, something that was very important to us at the beginning and is very important to us now, is enabling people who aren't at GiveWell to understand what we do and why. Something that was very frustrating to me back then was not understanding how a charity arrived at the claim it made about $20 provides a child water for life, or how foundations decided which organizations to support.
And so, we put all of our research and the reasoning behind the research on our website so that--I think that imposes some rigor on ourselves in thinking about how we do our work, but also enables outsiders to understand what we're doing and why and critique it where they have disagreement.
Russ Roberts: And, of course, we'll put a link up to your website. It's pretty easy to find, GiveWell, one word, Google it.
I have--as you probably know as a listener to EconTalk, Elie, that I have some unease with some aspects of the Effective Altruism movement, which is part of what motivates, I think, GiveWell's philosophy.
But I want to start by saying that your website is spectacular: the transparency and the care and the openness, it's really unparalleled, not just in charitable organizations--in most organizations, any kind of organization. If you're actually doing what you say you are, and I lean that way. So, I'm a--I think you are, and it's quite, quite impressive; and it's really wonderful. And it's a--in many ways, a model, I think for, not just, again, charities, but other organizations in your openness of both expressing uncertainty and imperfect confidence. There's a lot of humility in your website. It's very, very impressive.
Russ Roberts: But I want to start off with a different kind of question. We'll get to many of those issues you raised in your description, but thinking about you and Holden as individuals back in, say, 2006, and most people sitting at home listening to this conversation, if I said to them, 'You want to spend your charitable money well, what would you do?' And, I think there are all kinds of different answers people might give.
One standard answer: We've had Dan Palotta on the program. He hates this answer: but many people would look at overhead and what proportion of giving goes to the actual recipients, etc.
There are all kinds of different, again, ways you could attack the challenge: I want to do good with my money. I don't just want to feel virtuous, I want to be virtuous. And, that's my favorite thing about the Effective Altruism Movement and what you're doing at GiveWell, which is to actually make a difference, not just feel like you're making a difference.
But, I'm curious what you had in your head--long time ago, throw your mind back there and you think, 'Well, I'd like to spend this money wisely.' Where do you start? It's a tough question. How do you begin to tackle that question. And how did the answer to it evolve over time?
Elie Hassenfeld: All the way back at the beginning, I think we recognized the exact challenge that you're describing: that we could be out there essentially trying to boil the ocean. And so, very early on we said, 'Well, we're never going to be able to get the perfect answer. Let's pick an area that we can dig into.' And, that's why I chose clean water in Africa, way back when, in 2006. And, it felt like a big and challenging area, but one that was tractable enough that I could learn something.
As time moved on and certainly when GiveWell started full-time, we took a more serious look at how to address this question and started very broad. In GiveWell's first year, we looked both at international causes, but also organizations helping people in New York City, where we were living at the time. And, it was in that process that I think we learned something that probably is obvious to many people, but was not obvious to me when I was 25, which was that: the needs and then the opportunities to help people in low-income countries are just so different than the possibilities for helping people in New York City.
And, that's not to diminish the problems that people face in New York City, but the magnitude of the problems overseas are so, so large.
I think, speaking personally or psychologically, we started with what seemed to me--maybe even a different question than the one we're describing today--it was, 'I want to give some of my money to charity. I want to help people. How do I know that it's really helping anyone? How do I know that it's accomplishing a lot of good, nevermind the most good?' But, as we dug into that question, we started to see how broad the differences were. And that, to me, was one of the most interesting questions I'd ever been faced with. And, eventually, over time led to more and more work to try and get a better and better answer to the question of how can we direct these funds where they'll do the most?
Russ Roberts: And, right now, you recommend four charities on your website, which is very unusual, right? There are websites that they'll grade charities--typically have things like overhead, again, not efficiency or efficacy or impact; but just: they seem to be fairly well run--whatever that means. But you actually--GiveWell actually--picks four charities that you hope have the biggest bang for the buck. And, 'biggest bang for the buck' is obviously a serious philosophical and statistical challenge. But you take it very seriously, which is spectacular. And, tell us what those four charities are, and give us an idea of what the bang for the buck is.
Elie Hassenfeld: Yeah. So, let me describe them and let me just clarify one thing first, which is we do recommend these four organizations. Now we also have donors who give to something different. We call it an All Grants Fund, which gives money to things that don't qualify as these top charities. You could think about them as maybe the blue chip organizations; and I'm happy to talk about those other things, too, in a minute.
But, just what those four are. So, those four, two focus on malaria. One is Malaria Consortium and its Seasonal Malaria Chemoprevention [SMC] Program. One is Against Malaria Foundation and its insecticide-treated net program.
The former, SMC, is--the acronym is Medical Treatment to Prevent Malaria that's given to children during the rainy season when they're between the ages of three months and five years. Malaria nets are probably very well known. It's giving people nets to put over their beds while they're sleeping so they're not bitten by mosquitoes at night. And, both of these programs have very strong evidence behind them that they reduce malaria cases and then malaria deaths.
A third organization is Helen Keller International's Vitamin A Supplementation Program, giving vitamin A supplementation to, again, children under the age of five, twice a year has been shown to significantly reduce child mortality, especially in populations that are vitamin-A deficient. So, this is sort of very different than what we think about in a high-income country with respect to vitamin supplementation. These are people who have significant vitamin A deficiency, and this program reduces child mortality by about 25%.
And then, finally, an organization called New Incentives, and it actually got started with some funding from this All Grants Fund that I mentioned. But, today what it does is provides small cash transfers to caregivers to incentivize them to bring their children for routine immunization visits in northwest Nigeria. This is an area where immunization rates are among the lowest in the world. And so, this program brings more people in.
Just very roughly, we would estimate about $5,000 to avert the death of someone who would have otherwise died from malaria, lack of immunization, other childhood illness that vitamin A supplementation protects against or supports better health. And so, about $5,000 per life saved is a way of just having a benchmark for what we're talking about here.
Russ Roberts: And, you qualify that. It's obviously your best estimate. It's obviously an estimate. You qualify it because you're aware that--give some examples of the reasons you qualify it. For example, one being that not everybody is going to use the nets correctly, say, in the case of the malaria. What are some of the other issues that you try to take into account in that quantification?
Elie Hassenfeld: So, I'm trying to take a lot of issues into account. Well, let's just go through malaria nets as an example and show all the qualifiers. So, the first one is: we have data from randomized controlled trials [RCTs] that when you distribute nets, people use the nets; it reduces malaria cases. Randomized controlled trials are conducted very differently than national-level malaria net distributions. In randomized trials, at times, researchers were trying to determine if someone using a net would prevent malaria and reduce death. And so, they would, at times, go to people's homes daily and check if they were using the nets. This doesn't happen when you distribute nets nationwide in Democratic Republic of Congo. Similarly, the underlying levels of child mortality of malaria--of poverty--were much higher 25, 30 years ago when many of the malaria net RCTs were conducted. So, there have been changes in the underlying--the question is, how valid are the results from those studies to today's experience?
One particular way in which using nets has become more challenging and net manufacturers have adjusted is insecticide resistance. Mosquitoes are building up resistance to the insecticide used in the nets. The nets that were being used 10 years ago would not be as effective today as they were 10 years ago because of this resistance. In some cases, we're still using older nets. We try to take that into account in our analysis. And, in other cases, there are newer nets that have been proven more effective against mosquitoes and their current resistance.
I'll just say one last one, though. There's numerous considerations to take into account, but another question we have is, we don't--I'll give a little bit of a lead in to help people understand this--we don't care what GiveWell donors' money, the sort of literal dollars that we direct, accomplish. We care about, ideally, the causal impact that our work has on the world.
So, for example, if we give $100 to the Against Malaria Foundation, but that just causes some other donor to not give that money, well, they end up in the same place. We haven't actually changed what has happened in the world.
And so, we try to take this into account in our analysis. We call it a Fungibility Adjustment, from the idea that money is fungible; and we try to have an estimate of the likelihood that our giving displaces money from someone else, and an estimate of what that someone else would've done with those funds. And, I think this probably gives an idea on one hand of the---I don't know--the intensity that we try to bring to the quantification, but also the, admittedly inherent, massive uncertainty in trying to pin down some of these parameters.
Russ Roberts: Did you have any formal philosophical study that led you to think about these problems in a utilitarian way or any other way, or were you just feeling your way in 2006 and going forward?
Elie Hassenfeld: Really just feeling our way. No formal or advanced training. Took a philosophy class in college, but that doesn't count. You asked earlier or you pointed out that GiveWell is unusual in recommending such a small set of organizations. But I think the reason we've done what we've done, it's all driven by GiveWell serving, I don't know, the person I was or Holden, my co-founder, was. And so, it's just constantly been a movement to try and get the answers we were looking for. And so, it's coming more from a customer perspective rather than an expert philosopher or economist perspective.
Russ Roberts: Yeah, yeah, yeah. I mean, the reason I ask is that utilitarianism says we should try to do the greatest good for the greatest number of people. And, I used to think that was a plausible idea. As I've gotten older, I've started to think that doesn't have a lot of content. But, it gets at something we do care about, which is we don't just want to do good. We care about how much good, and we care about opportunity costs--what we might achieve if we hadn't chosen this outcome or that outcome. Which you obviously are very aware of.
But, you are focused on saving lives of desperately poor people from certain causes. And, my first thought would be: Well, they're not going to die of malaria, but they're so poor, they're going to die of other things. And, if you don't get at those underlying aspects of their lives, one, you're not really saving their life to ensure a longer lifespan of any large amount. And secondly, the life that they have, the quality of it is still maybe quite low, even if the quantity in terms of years--the thing we can't measure is the quality. And so, your methodology pushes you toward those kind of outcomes. Comment on that.
Elie Hassenfeld: Yeah, and this is a great question. It's one that I think is still in our minds, that bothered us a lot early on.
And so, when we tried to look into it, we found two things, both of which were surprising to me at the time. The first is: a huge proportion of deaths that occur in low-income countries occur in those very early years. A shocking amount in the first month, first year, first five years. Once people get past those years, they tend to live long lives--not as long as people in high-income countries, but still relatively long lives.
And then separately, it is true that people's self-reported life satisfaction is lower in low-income countries than in high-income countries on average. But, it's also--I don't know, I don't have the way to describe it off the top of my head--but it's not terrible. People are living lives that they're happy to be living. Their lives could be improved significantly. But, I would say that overall the people whose deaths we avert in childhood go on to, on average, live long, relatively happy lives.
Averting deaths in childhood is not the only thing we do. We've supported organizations that aim to directly reduce poverty via, could be direct cash transfers, via an organization GiveDirectly that I know you've talked to in the past; but also other programs. When we've tried--and this is where the quantification becomes even more challenging--but when we've tried to look at what we get with those poverty-reducing programs versus these childhood-mortality-averting programs, we've generally felt that the childhood mortality averting programs look better to us--like, we are getting more bang for our buck. Though, just to be clear, this is one of the ways in which in the realm of philanthropic recommendations, transparency seems absolutely essential because there's no possible way that I would claim that we have the, quote, "true answer" to sort of the question of how to trade off between these things, as much as, 'You know, in our view, this is what it is.'
And, there are many donors who are big fans of GiveWell and use our work and also give a significant portion to other things, either based on our work or not, because they disagree with some of the underlying philosophical judgments that we're making. You know, we would love to know how to give money to end poverty now. If we could identify the sort of root cause of poverty and then direct money there, that would be amazing. And, I think we would undoubtedly want to support that. I think, unfortunately, we just don't know the answer. We don't know how to do that, as far as I know, and that's the reason. So, I completely agree that in some sense it would be better to alleviate the cause rather than the symptoms, but I don't know how to alleviate the cause.
And then, finally, just one quick final point is that I think that sometimes the disease-reduction programs get short shrift. Reducing serious illness in childhood probably leads to better outcomes for that child itself by having a healthier young early childhood development. Nevermind the effects on the family and the community, by reducing the deaths that are occurring and all the other illness that is not as severe but is just removing people from their ability to do other things. These don't play a huge role in the way we actually calculate the benefits, but they're, I think, an important consideration with these disease-reducing interventions.
Russ Roberts: It's an interesting question about whether a thoughtful person who wants to spend their money wisely--and I'm thinking of a thoughtful person like me, meaning of comfortably financially but not wildly wealthy. I try to give about 10% of my income to charity that is not going to fund a program to end poverty. It will not fund a program to figure out how to end poverty. So, large donors and large organizations like yours have an opportunity--somebody should have an opportunity--to think big. Of course, that's happening in economics departments around the world that are trying desperately to solve these problems. They're not easily solved, obviously. Our understanding of the process is imperfect; our understanding of how to implement the process, to improve the process, is imperfect.
So, even if you have a simple answer that it might even be true, like more economic freedom or better education or whatever you think is the thing that might work, that's not enough. There's a implementation--it's sort of like innovation and entrepreneurship: having a good idea is not really the same as having a good product. And I think we don't have a lot of good products in the poverty-fighting field; and we should probably spend some resources trying to answer that question.
Elie Hassenfeld: Yeah. And so, I think we've tried to play two roles there. The first is--to use your analogy--when we see something that looks like a good product, even if it's not exactly the thing we normally look for, we support it.
And so, just let me give one example. There's a center at Yale called Y-RISE run by a Professor Mushfiq Mobarak, who, I think I know fairly well from work at GiveWell and have always found him specifically to be one of the people who brings together academic research and implementation in a very unique way. His center is focused on research into the scaling of programs, the so-called Randomista Movement, which started in economics 25 years ago, 30 years ago, and was focused on randomized control trials to determine what programs work in economic development at sort of a micro level, has done a lot to push the field forward. And he's asking, 'And, what can we learn about the, quote, "science" of bringing programs to scale?'
And, this is not answering the whole question of how to end poverty, but it is one area where better answers would lead to better outcomes. And, this is a case where--some of--this is not, you know--'We supported this program,'--it's not a case where we had a quantified estimate of the good that would come from supporting this individual and their team at Yale. But it is a case where I think this person's track record makes it a, quote, "good bet" to contribute to the fight against global poverty. And so, we've done that.
I do think--maybe secondly, the niche GiveWell fills and why I think some of the we do what we do and not something else is: We're really committed, and I think where we're good, is setting things up so we can learn empirically about how things went.
And, that's not because that's the only approach, and it's not because it's necessarily the best approach, but I think it's an approach that we're good at. It's what we do.
And so, we tend to, with the vast majority of the funds we direct, put them in places where we will be able to know if we were right or if we were wrong. We will try to learn and then we will try to improve. And that means that we tend to put less energy and less time and less attention on high risk opportunities.
GiveWell would, I think, be very unlikely to fund think tanks in Africa to try to identify pro-growth policies in sub-Saharan African countries. Not because it's obviously a bad idea by any means, but because it's really not what we're good at. I think where we fit in the broad set of people and groups trying to improve the outcomes for people living in low-income countries is trying to give in a way that we can learn and improve over time, that we create feedback loops that we can learn from. And that is fairly unusual in the philanthropic sector. And so, it's a niche I'm happy to try and fill.
Russ Roberts: So, you have four charities you're recommending right now. Why four? Why not 15? There's a million to choose from--probably more than a million. Curious how you narrowed it. What's the process by which a charity enters your space to be considered? And then, how does that decision get made to those four? Why not six? Are there two that are right on the edge, that just didn't make it? Why isn't Shalem College in Jerusalem one of the four--we're important for bringing leadership to a pivotal country in the Middle East that is often a source of conflict in the world? Just saying. Seriously, how do you get to four?
Elie Hassenfeld: So, the process essentially starts with a very wide funnel. And, the way that an organization would get on our radar is one of a few ways.
Early on, I literally read through thousands of charities, tax forms, went to thousands of websites--and this was just to determine what do they do and what questions should we ask them. We decided to focus that within low-income countries. So, the starting point--and I'm happy to go back to that, if you want--but, the starting point was, on some level: What are all the organizations that work to help people in low-income countries and are large enough that they could plausibly be able to engage with us, answer some questions, have some data? And, you're talking in the range of one to 2,000 at that point.
The main way, though, that we filter that list down is via academic research. There's a huge amount of research that has been done on what works to help people in low-income countries. And so, we start with the academic research on what programs work, how well do they work, what do we know. This is mostly relying on randomized trials, though certainly not entirely. There's a lot of them both in the field of economics and public health.
And so, we're looking for programs that, via that process, have strong evidence and could plausibly be implemented by a nonprofit organization--you know, there could be evidence for--anyhow. So, that's where we're starting. And, we're using that to narrow the list to the group of organizations that meet the following criteria. Number One: the evidence is strong enough that we believe it is significantly more likely than not that they're having a lot of effect. So, we want to, in this top charities list, remove organizations where there's, I don't know, a 25% chance of a very large effect. This is supposed to be high confidence, in this short list.
Number Two: it's programs whose estimated impact per dollar exceeds our current threshold. So, to explain that, right now we are raising approximately $500 million per year, and then we're just taking that money and essentially applying it to this--you can imagine a ranked order of all the possible programs we could give to. And so, as we go down that list, one criterion that sets the bar that top charities have to clear is how much money we've got and how far down the ranked order list we would go. Right now, we put things in our model in terms of multiples of direct cash transfers. So, we'll say: We have an estimate for how good it is to just give a very poor person $1,000. That gets you a value of 1. And then, we're going to fund our list all the way down currently to 10. That's our estimate. Of course, very debatable; but that's how we're doing it.
Russ Roberts: Excuse me, meaning--down to 10? What do you mean, 'down to 10'?
Elie Hassenfeld: Some are grid[?], some are 50, some are 30, some are 20, some are 10. But, when we find something that is six, right now, we'd say that's not above our funding threshold, so it couldn't be on the top charity list.
And then, the third part, before it gets on the list, we want to have provided significant funding. Right now for us, we say $10 million at least, for least a year, just so we have some level of experience with the organization. We follow them and we see what they do before they get on that list.
Those are the main--those are the three criteria. And, those four organizations account for approximately two-thirds of the funds we direct. So, this is not everything. Obviously, you have to provide funding in order for an organization to even get to the stage of being a potential top charity. But, the vast majority goes to organizations which are above our funding threshold, high confidence, and we've funded significantly for at least a year.
Russ Roberts: And, who makes that call? Who is making that call? Is it a vote, is it a committee, is it you? How does that get done internally?
Elie Hassenfeld: Yeah. There's a lot of debate and discussion internally, and I think normally there's ultimately a fair amount of agreement among, let's say, the three or four people who are senior members of the research team on a particular opportunity about how to treat it. And so, it's not a formal vote or anything like that, but it tends to be, via the analysis, debate, and discussion, reaching a conclusion about what we think about the opportunity.
Russ Roberts: A consensus emerges. What enters that conversation besides the hard numbers? I think the general feeling people have is that it's just--it's all science: you just sit down and the numbers speak. Is that true for you? It's not true for other places, but for you, is that true?
Elie Hassenfeld: No, it's not true. Look, the numbers do a lot of the work, but what else enters the conversation at the end? And, let me explain this just concretely for a second. I'm talking about this funding threshold and I'm giving a number of 10. Sometimes we reach the end of an investigation and the number in the spreadsheet is 7 or it's 13. Seven and 13 are not that different from 10, given all the uncertainty that's baked in.
And so, what do we do then and how do we move forward? I'd say there's a few different types of considerations that play into that analysis. Number one is: to what extent is there upside or opportunity that we're not taking seriously here? That could come from supporting a young person, a young organization, with a small amount of funding that could lead to something bigger and better in the future.
And, we've done that in the past when the numbers have looked bad. And, in hindsight, those were great decisions, and happy to talk about examples if you're interested. So, one would be--I guess our term for it would be 'unmodeled upside,' is a consideration.
Another one is--and again, I think on some level we feel fairly uncomfortable about all of these because they're soft; but at the end of the day they're important, too. So, another one might be: To what extent do we have high confidence that the person that we're dealing with--the organization or literally the individual--will be very open with us in the future about what's working and what's not? I said earlier that a major consideration for us is ability to learn. We believe by hope, even though GiveWell is 15 years old, we're still at the beginning of a long journey and an opportunity where we will learn is much better than one where we won't.
And so, having confidence that the person will be transparent about what's good, but importantly what is bad , in the future, is very important.
And then, finally, to some extent I'd say track record of the person in the organization itself plays a bit of a qualitative role. So, let's say we have two opportunities in front of us. One, the number is 7, the other number is 13. But, in one case, let's say the one with the lower score, the lower number, that person or organization has successfully delivered a wide variety of programs over the last five years. And, we've seen them overcome challenges both programmatically, organizationally--for lack of a better word, we have confidence that they know what they're doing and we want to support them. That's something that would push that up.
And then, on the other hand, let's say there's an organization we'd give funds to and we don't really feel like we understand the track record as well as we'd like. We're not as confident as we'd like to be that the programs have had the effect that the spreadsheet describes, the spreadsheet estimates; we estimate via the spreadsheet we'd be less likely to fund.
And so, yeah, I think those are some of the things: ability to learn, track record, confidence in the person or the individuals that we are taking into account at the end and that kind of thinking. And, I think they all play into some of this unmodeled upside or maybe unmodeled downside that is important to consider.
Russ Roberts: So, a strange analogy that comes to mind, which I think is probably sometimes relevant for your decision-making, is when a small business gets adopted by Walmart as a customer. So, you've got a small business, you're making some product; and you show up at Walmart and they like it a lot and they decide to buy X billion of that. And, you're totally unprepared for that because you've never done anything like it. And, actually, this great contract which has you celebrating and drinking champagne could end your company because you're not going to be able to live up to it and it's going to be a devastating impact. And, on top of that, you're now listening to them at a level of detail and compliance that is extremely unusual for you. They're not just your largest customer: they're your largest customer by an enormous amount. So, you've essentially strapped yourself to their engine.
So, I'm thinking about--again, I was being facetious before, but we'll play with it for a little longer--let's say Shalem College made your list. You know, we have a $12 million dollar budget. There are things we're not doing, I'd like to do if we had a little bit more. If we had a lot more, there's another set of things I do. But, if all of a sudden I was on your list, we'd be flowing with money because there are a lot of people who trust you and you'd made that call.
And, how do you make sure that--it's one thing to say, 'Well, we distributed 1,000 bed nets last year. We think there's 4 million--a demand for 4 million. So, if we had more money, we'd distribute more bed nets.' A lot of organizations aren't that straightforward. Many of yours are: they're literally, 'We need more capital to expand and cash to do more of what we already do.' But, how do you interact with them about strategic planning and their budget? And, do you cut them off at some point, because you feel like 'They've got plenty of money now; let's take them off the list'? How do you deal with that? Are you strategizing with them along the way, helping them grow? Are you giving them a pace at which they're growing?
Elie Hassenfeld: So, there's a wide variety of ways that this goes. The first thing that we take very seriously is a question that we call, or a topic we call, Room for More Funding, which is: 'How much funding do we think they could use well?' And, we wouldn't put an organization on our list that could only use $100,000 well--or we thought they could only use $100,000 well--if that might lead them to get $10 million dollars.
We're also watching this over time. And so, sometimes organizations have brought in a lot of money from the outside, either because we put them on or it happened independently. And, that affects what we'll give them. We want to ensure that the funds are doing a lot. And so we watch it over time. We're tracking them, we're in touch with them, and that will cause us to give less.
Mostly, the type of support we're providing most programs is direct programmatic support. So, in using one of the examples from before, the organization New Incentives that does conditional cash transfers for immunizations in Nigeria, they might say, 'We work in this area of this state of Nigeria. We have succeeded. We want to expand to this other area. We need X dollars to expand.' And, that type of engagement is the most common. We'll be looking at their past success--their track record--in order to have a sense of how well future growth will go.
But, it's fairly, not to say--linear or organic--in building upon past work for future work. There are some organizations and people where we've provided unrestricted support. Which is really just kind of what you're talking about with Shalem College, saying, 'Ultimately we don't really know what you, organization, will do, but we have a lot of confidence in you. We want to give you the flexibility and the ability to see what needs you have, whether it's hire fundraisers or run a strategic planning process or open a new office in a new country without a program in mind.' And so, we've done that. That tends to be rarer and really happens when we both think organizations could really use it because they are constrained. And also, we have an extremely high degree of confidence or trust that they are going to do great things with the funds we direct.
Russ Roberts: I'm sure you've made mistakes and have regrets about things you funded. You can talk about them specifically if you want, but if you don't, I'm curious what you learned. How much has your process of evaluation that we're talking about--which is pretty subtle; again, it's not a simple number about overhead or what proportion goes to salaries or whatever it is--what are some of the lessons you've learned that might apply generally to organizations and how have they changed over time? Like you said, you're still young, but I'm sure learned a lot.
Elie Hassenfeld: Yeah. I think there's a ton of lessons and a ton of mistakes and happy to talk about them very specifically.
Going all the way back to the beginning: When we were starting, we were very frustrated by a lack of detailed information from charities about what they did. And so, one of the first organizations that we recommended way back when was an organization called PSI [Population Services International]--I think they now are literally called PSI, but it originally stood for Population Services International. And, I should say as a caveat to this, all of what follows, all of what I'm talking about is roughly now 10 to 15 years ago. So, none of this is a statement about what PSI does today. But, back then when we first came to them, they were able to provide a huge amount of data on what they had distributed, where it had gone, what the effects were.
And, we supported them in large part because, I think, we took those numbers at face value in a way that I think was overly enthusiastic about quantification as a path to getting the best answer.
And over time--and this is 2007, 2008, 2009, 2010--we started to ask: To what extent should we believe these numbers? Do we know that the numbers we're getting back--if they say we distributed 1,000 nets to this location in Zambia--well, how do we know those nets got there and that people used the nets? And, as we dug in further, I think all the numbers that they reported were--sort of--true. They were not fraudulent. But also I don't think they were gathered with the most rigorous method.
Russ Roberts: Kind of like GDP [Gross Domestic Product] statistics in certain countries. It's an episode on that we'll link to. But, yeah, general problem.
Elie Hassenfeld: And, they've done a lot of work on this over time to improve.
But, I guess one mistake that we made early on that we've learned from is, I think, being overly enthusiastic about data, in and of itself, at the expense of--I don't know--critical evaluation of information to try to make good decisions.
And so, it moved us more towards, I think, a place we are today, which is certainly quantifying a lot, gathering the data we can. But then trying to go further.
Another type of mistake that we made is we recommended an organization or a program called No Lean Season and made it a top charity--gosh, I don't know what year it was. But in the 2000-teens at some point. And this was an organization that was actually set up by research that this professor, Mushfiq Mobarak, had done at Yale.
It focused on providing small incentives to, largely young men to migrate from rural areas of Bangladesh to urban areas during the lean season. So, people work in an agricultural economy, there's some time of year when there's a lot of food and there's a lot of work in agricultural locations. And then there's a time of year when there isn't; and people don't have work, they can't earn income. And, he ran a small randomized trial where a very small incentive encouraged more people to migrate. They migrated to the cities, they earned a lot more money, they sent money back home. They were even more likely to migrate in subsequent years when they didn't receive the incentive.
And, ran subsequent trials. We worked with others to sort of set up this program.
And, I think we ended up making--ultimately we funded them. They operated for a couple of years. Alongside the funding we gave their operations, we also wanted to run another trial to see how well this worked at scale. And, in fact, the program did not work at scale, when we first funded it. We had different theories--and by 'didn't work,' I mean we didn't see that the treatment group in this randomized trial, they were not more likely to migrate than the control group. And, there are a lot of potential explanations why. Anyhow, I won't go into those. Happy to go into those if they're interesting.
But, I think we actually made two mistakes there.
I think first--I mean, on some level, I think that's the process working. We made a grant, I think we made a good bet, and we were wrong; and we ended our support. Looking back, I think we didn't take seriously enough, at the time, the challenge of bringing this small-scale, fairly complicated program, to scale. In the original trial, it was people going and finding folks who were otherwise not going to migrate. And, this happened at small scales--1,000, 2,000 people. As it got larger, it had to go through large institutions. And through microfinance banks in Bangladesh, which I think made it harder to deliver.
So, on one hand I think we didn't realize how hard it was. But even looking back, I think in some ways in hindsight, I wonder if we made a different mistake, which was: We also exited too quickly.
And, I think that we went in, and on some level--I don't think I would've said this at the time--but looking back, I think on some level we thought it would be easier than it was to deliver. And, that was a mistake.
And then, because we thought it would be easy, when it didn't work, we were ready to end the program, as was the organization implementing it. And I think that was also a mistake. And, I think, in both ways, the better prediction to have made back then would have been: If this is going to work, it's going to be hard. It's going to take some time and some iteration to get the model right at scale. Is that a bet worth taking? And then, if it is, be ready to stick with it over a longer timeframe.
And so now, we tend to commit to longer timeframes with programs that are more experimental: to at least say, I don't know, 'Make the initial decision with a longer timeframe in mind, see if it's good enough on that basis, and then be willing to stick with it over time--be willing to stick with it even if it doesn't look good at first,' so it has the opportunity to improve.
Russ Roberts: As an example of the kind of transparency we were talking about, at the beginning, that you ran a contest called Change Our Minds. Really bold thing to do. I really love that. Describe what it is and how you have used it.
Elie Hassenfeld: Yeah. So, I mentioned that we do a lot of quantification for the programs we support. And, all of this information about the quantification, the research, the analysis is on our website. It plays a major role in our decision-making. And so, because it's all public, we ran a contest where we asked people to try and tell us what we were getting wrong. And, we said the winner would get, I don't remember what exactly, $20,000. We had a lot of other second and third prizes for folks. And, people went through our analysis to find things that we had gotten wrong. And there were places--I think on one hand, I was--well, a few things came out of that. First and foremost, I was somewhat surprised and gratified by the number of high-quality submissions. Going into it, I sort of thought we'd get a lot of junk and we wouldn't get that many submissions.
We got more than 50 really intensely-considered submissions, including from people who have academic positions at prestigious institutions. People were really engaged in this project. People found errors. I'm grateful there weren't huge errors that show that we had made a massive error in so much of our funding, but certainly we had made small errors across programs we recommend: water, deworming.
I think, most importantly, people called us out for being insufficiently transparent about the uncertainty inherent in our analysis. And, I think that on some level we've tried to respond to this. I think we always have known that the numbers we're using are uncertain, but we didn't bring that wide uncertainty to the fore in all of our analysis.
So, just to make it concrete, I said before that we take seriously the possibility that a dollar we give displaces a dollar that someone else has given, or let's say it displaces some amount of money that someone else would give.
But, let's say your reasonable confidence interval around that is that maybe our median estimate is a dollar we give displaces 40 cents from someone else, but it could be as high as 80 cents, as low as 10 cents. And so, we've tried to incorporate that more at the front of--every single researcher, as they're doing their work, is looking at that to remember the uncertainty inherent in the analysis that they're doing.
The other thing that I'll just say we learned from that contest, which is a more of a side note than a substantive critique of the research is: say, about 20% of the people who submitted misunderstood what we were doing. So, they said, you're making this error because you're doing X, but you should do Y. And in fact, we were doing Y: the spreadsheets were just convoluted.
And so, one of the things we've really been working on this year is improving our legibility. So, we're very transparent. If you want to spend a lot of time--I mean hundreds of hours--you can get the answer to the questions you have about our work. But, most people don't have hundreds of hours to invest. And, what we'd really like to do is make the reasoning legible enough that people can understand it, critique it, engage with it at a lower level of intensity. So, we saw in this contest that people were struggling to do that. And so, we're putting more energy, hopefully, into making our research easier to understand and critique.
Russ Roberts: It's fascinating byproduct of that kind of experience, right? What's the goal? But, they helped you redo your website without intending to.
Russ Roberts: The phenomenon you're talking about, of some other donors reducing their gift because yours goes up is called typically crowding out in economics. It's actually--the first published paper I had in my career was on this question of crowding out. And, it's a debated topic in economics.
But, there's also crowding in. And as now a fundraiser for an organization, I have a richer understanding than I had when I was a newly-minted graduate student. If you--if GiveWell--puts a stamp of approval on an organization, that could easily increase donations, right? It could cause people to say, 'Oh, I've been giving those people money, but I wasn't always sure about it. But, if GiveWell--who looks at these things more carefully than I do--thinks it's a good idea, I'm going to increase my gift.' Do you notice that ever?
Elie Hassenfeld: Yeah. I think so for sure. I think one example of this--I mean, obviously our goal as an organization is to recommend programs and then raise money for them. And so, we're very explicitly trying to get people to give more.
And, we've seen a lot of that directly and indirectly. So, we might recommend an organization that then has more publicity around it--because of the recommendation, or it just builds on itself over time.
But then, I think there's another way this dynamic could work, and this is very hard to pin down, but I think it's at play. One of our top charities is Malaria Consortium--Seasonal Malaria Chemo Prevention Program. It goes by the acronym SMC, and it's a program that provides medicine to children to prevent malaria. And, it's a relatively new program. And, when we first supported it, it hadn't been rolled out widely. And, I think one of the necessary conditions for large governmental funders--and so, in the world of aid, this could be the global fund which provides funding to malaria, tuberculosis, and HIV/AIDS [human immunodeficiency virus/acquired immunodeficiency syndrome], but also the United States government and others--is: it is helpful to them to see that a program has been implemented successfully, in order for them to have confidence in their technical bodies that decide what programs are in and out of their list of programs that they'll support--to see that successful track record.
But, someone has to support some of that track record. So, we were not the first to support this program by any means. It was an organization called Unit Aid, which provides some very early support to this program. But I think--and, I can't verify--but I think we played some role in growing the size of this program to the point where now we still support it, but also it is very much on the short list of programs that are funded by governments at the global level. And, I think in the same way that we are worried about the crowding-out phenomenon, I also think this is an example where there probably is, to some extent, a crowding-in phenomenon of making something bigger so that it can get more support from others in the future.
Russ Roberts: What keeps you up at night, related to GiveWell? You told me you have some kids, so I'm sure they keep you up sometimes. But--
Elie Hassenfeld: Yeah, the kids definitely keep me up at night. There's a lot of things. Look, there's certainly very specific decisions that are hard when we're weighing quantitative and qualitative factors, and worrying that--and I'm happy to talk about examples if you want, but worrying that the personal bias for me or others is skewing our decision.
But, maybe more methodologically, I think a lot of what we're talking about is worrying to me. It's very challenging.
And what do I mean by that? When GiveWell was very small, I think it was easier--and, by 'small' I mean a research team of three or four people, five people--it was easier to say: We have these numbers, we have this quantification. It's just a part of the story. It's a tool to think critically about the different considerations, but it's not the answer.
And, one of the things I've seen as we've grown as an organization--we're 70 people, about half of whom are working on research: so, that's deciding where to give. So, I see this tendency now that we're a larger organization for staff to want to rely more on the numbers.
And I think I understand this. It's easier, first, to understand the rules. Maybe if the rules at GiveWell are: I build a model, then I get a number, the number is the answer--that is straightforward. That's a problem that people can understand how to deal with.
On the other hand, if it's: Do a lot of work with the numbers and then overlay your best judgment--independent of personal bias to make a good judgment--that's hard to do.
And so, I really--we're trying to find a way to maintain this good judgment/qualitative overlay and input to our analysis, without getting too--while we grow. And, I think in some ways that's what makes--in my opinion--one of the things that makes GiveWell special is that we're intensely interested in quantification while also highly aware of all the problems with quantification.
And, I think it's very hard to maintain that tension--those two considerations--alongside each other as we keep growing as an institution.
So, anyhow, I have endless numbers of things that worry me, but that's one of them.
Russ Roberts: Well, I think that plagues every institution, charity and non-charity. I think there's always a tension between what's measured; and of course, what's measured gets managed. That's generally true, but it's not all you manage. You do bring other things into consideration. And, there's no right answer as to how much one should weigh in and count more than the other.
There is solace from certainty; and there is fear and anxiety from uncertainty. And so, we have a natural bias, I think, to move toward objective measures that, as you say, people, they find them comfortable.
And certain types of people--the kind of people who do research, who are drawn to that kind of area, are going to be likely to be seduced or comforted--or whatever word you want--by an objective, precise quantitative measure. A couple decimal points out, a couple places after the decimal point.
So, it is a really interesting, I think, challenge in terms of management, generally. I don't think it's unique to you, to GiveWell, but I think it's probably pretty front-and-center there. So, I think in other organizations you can ignore it or miss it if you're not careful. You probably think about it more than most people do in most places.
Elie Hassenfeld: Yeah. There's just a research, an investigation that we've been working through, which is supporting an organization called Evidence Action in helping the Indian government roll out a coordinated water program in various Indian states. And, the background is that I think the Prime Minister, Modi, sees piped water--meaning bringing water to people's households in rural areas--as a major initiative, a good initiative for the country. And, the Water Ministry that they set up in India was looking for help bringing in chlorinated water. And so, Evidence Action has done a lot of work on chlorinated water; and the Indian government solicited their input. And, this was an investigation that--it's very hard. You can do all the work to quantify all the different aspects: How likely is it to work and how much money will you crowd in? How likely are you to crowd out? What is the effect of fluorinated water on mortality, etc., etc., etc.? But, it's so uncertain, because ultimately you have to overlay some degree--a significant degree--of qualitative factors and thinking about how to make the decision.
And, this is a case where Evidence Action is an organization we've literally supported over the last 10 years. Its entire history; it sort of omes out of innovations for Poverty Action, the Yale Group that focuses on randomized trials. Evidence Action has a great track record of success. And, ultimately for us, it's that, that really carries the day. In some sense, we can model all we want; when we make our best guesses, the numbers look good. But, if we're excited to support it because of the organizational track record, and in a different--both in successful delivery of programs, but also honesty, transparency, what they've shared with us--if it had been a different organization, I think we wouldn't have supported, we wouldn't support the program because we wouldn't have that qualitative overlay.
And, I think, substantively, it's how we try to operate. Managerially, I think these sorts of examples are helpful to everyone internally--I hope externally, too--in understanding what we do and why. That sounds great, then good. Maybe some people say, 'I thought this was a math equation and they're out.' But, that's what we're trying to do, to make good decisions about helping people.
Russ Roberts: I mentioned earlier, I try to tithe, roughly: I try to give 10% of my after-tax income to things I care about. And not just things I care about: things I care about that I think the money will have an impact--an impact that I care about. Not just the cause generally, but the way that the cause is being supported. So, I'm Jewish; I give to a lot of Jewish organizations. I give to certain artistic activities, tax-deductible stuff that speaks to me aesthetically, that makes my heart sing. I don't measure anything. Obviously.
Am I making a mistake? Would you--if you had to make the case for why I ought to give to your Top Four-- forget whether which four or whether I could divide it equally--should I stop supporting local charities here in Jerusalem--institutions that I actually use, by the way, not just care about, but I get indirect services from them? Is that a mistake? If we were out late at night having a beer and a heart-to-heart, and you say, 'No, I like you Russ, but I can't respect you because you spend your money so poorly.' Do you feel that way? And, if you don't, why not? And, if you do, how would you make the case to get me to do something different?
Elie Hassenfeld: Let me tell you how I think about it. I'm also a member of a synagogue and I support their activities. The way I think about that is I don't bucket it into my mental charitable-giving budget. I don't have a literal budget. I'm also trying to give about 10%. But, I say--you know what? That's external. That's not--when I think about my own charitable giving, I think about it as trying to help people to the greatest extent possible. I do that via GiveWell; and everything else that I give to, I treat as--I don't know--something different.
When I give to the synagogue or my school's PTA [Parent-Teacher Association], I see it as a shared responsibility to support a service that I use.
And, the reason that I give that I think one should seriously consider giving to the organizations we recommend is just the magnitude of the impact that they have. Giving 10% means a lot of funding that can go to help people. And, while it's definitely imperfect, when I think about the impact of about $5,000 per death averted, that is weighty enough for me that it causes me to want to give significantly to programs that are having that sort of effect.
Russ Roberts: Yeah, again--I'm pretty confident my charitable dollars don't save any lives in the sense that your organizations do.
Russ Roberts: The thing I've been thinking about lately--and we can close on this, get your thoughts--I've argued a number of times on the program that the argument for local giving, even though people here are better off than people may be farther away, is that I have better information about a local charity. It's a Hayekian kind of argument. And that I have a principle of giving locally: It's more likely to be effective just simply because I have a better idea of whether it works.
I don't know if I've told the story before, but a reporter once asked me if I was in favor of giving money to education in Africa. And I said, 'No.' And, they said, 'No? How could you be against giving money to support education in Africa?'
I said, 'We haven't even figured out how to help education in the United States.' I know how to help the budgets of schools in the United States. But to add actual knowledge that changes the lives of children--we struggle here in America to do that. And, I know a reasonable amount about that set of issues. And, to think that I could do good in Africa with that money where I know nothing about it is naïve. Foolish. I'm not being cruel. I think I'm being prudent.
But, what I love about what you're doing is it kind of takes away that argument. You've gone there and you've done the research--and I may have quibbles with it or its accuracy--but certainly is to the best of your knowledge, the organizations you give money to, the money achieves what it intends to achieve, and it does it in some level of effectiveness.
And, the other argument that I would come back to--I'm trying to rationalize my own habits here, of course; I don't want to pretend I'm just a truth seeker--is, part of my humanity, part of the essence of my being human is the connections I make with other human beings.
And, when I help a local organization, especially one that I can literally touch and observe directly, I get an emotional--and it's part of flourishing, connecting to other people and helping them.
Of course, there's an advantage to anonymity, not suggesting that I need to see the people that I am giving, say, food dollars to.
But, I do think that the connections we make--and I'm thinking now of sort of the extreme arguments of Peter Singer and others that I should give to the bed nets in Africa instead of throwing a birthday party for my kid, because, a birthday party for my kid, nobody's life is saved and the bed net saves a life.
And, to me, part of being a father is connecting to your child. And, you do that through love and gifts and kindness, and that's part of being a fulfilled human being.
So, I'm thinking, having had this nice conversation, that maybe some portion of my charitable giving, as you suggest could go toward the alleviation of extraordinary hardship and suffering that isn't related to quality of life--isn't related to my direct connections to other human beings. I'll never see those people. I can only imagine how transformative those resources are. But, it's not unimportant. So, maybe that's another way to think about it, that: like you say, you can have some buckets. One bucket is for the things you do that fulfill you as a human being and are not trivial, but they don't save lives. And maybe I should put some money aside for those activities that we think, at least we hope, do save those lives and transform those people in ways that are really pleasant, charitable-dollar in a local place that's doing good work, doesn't manage to match.
Elie Hassenfeld: Yeah. I mean, the reason I somewhat evaded the question of, 'Do I think you're making a mistake?' is: I'm not going to say, You're making a mistake.' Because, I make the same mistake. I spend money on all sorts of things that are not consistent with a Peter Singer-oriented argument of maximizing global utility.
I want to be comfortable. I want my kids to be comfortable, in ways that far surpass the utility that would be offered to someone in, you know, Western Kenya if they received those funds.
But, I think that I also care a lot about my personal flourishing, my family's flourishing, and I put a lot of time and some money into those activities. But then I think that--I don't think more money to those activities would lead to more flourishing. And, I'm able to then direct the vast majority of the charitable resources I have to the opportunities that I think will help people the most.
And so, I do think that that--I'm sort of trying to separate those--not trying to have the complete utilitarian argument and saying every dollar, every minute is intended to maximize global utility, but aiming to, on some level, do a lot on the side of personal and familial and people close to me, their success and thriving.
But then I have this opportunity to just decide where the vast majority of the dollars that I'm going to give go, and I could give that to something local and--I don't know--be a big shot or something, or just give it to, yeah, I don't know, where I think it'll do the most good. That's what I do.
Russ Roberts: My guest today has been Elie Hassenfeld. Elie, thanks for being part of EconTalk.
Elie Hassenfeld: Thank you so much, Russ. It's been great.