Eric Hanushek on Teachers
Aug 29 2011

Eric Hanushek of Stanford University's Hoover Institution talks with EconTalk host Russ Roberts about the importance of teacher quality in education. Hanushek argues that the standard measures of quality--experience and advanced degrees--are uncorrelated with student performance. But some teachers consistently cover dramatically more material and teach more than others, even within a school. Hanushek presents evidence that the impact of these differences on lifetime earnings for students can be quite large. The conversation closes with a discussion of school finance and the growth of administrators within school systems.

RELATED EPISODE
Doug Lemov on Teaching
Doug Lemov of Uncommon Schools and author of Teach Like a Champion talks with EconTalk host Russ Roberts about teaching and education. Drawing on his experience working in charter schools with children in poverty, Lemov discusses what makes a great...
EXPLORE MORE
Related EPISODE
Elizabeth Green on Education and Building a Better Teacher
Elizabeth Green, author of the new book Building a Better Teacher: How Teaching Works (and How to Teach it to Anyone), talks with EconTalk host Russ Roberts about the art of teaching and the history of various reforms, mostly failed,...
EXPLORE MORE
Explore audio transcript, further reading that will help you delve deeper into this week’s episode, and vigorous conversations in the form of our comments section below.

READER COMMENTS

Mads Lindstrøm
Aug 29 2011 at 12:46pm

Russ, the good teacher may not just mean that you learn more. The good teacher may also make you appreciate math more.

Or the other way around, even a kid genetically disposed for math may loose interest due to a bad teacher. This kid will never catch up, as the bad teacher made him loose interest. This kid will never study engineering as he dislikes math due to the bad teacher.

xian
Aug 29 2011 at 2:25pm

students gifted in math and potential engineers:

in the extreme case, students gifted in math would never get the chance the learn math (like in a mumbai slum). this would certainly be a huge barrier to becoming an engineer.

in the more real case, a simple question to ask is: how many gifted math students are wasting away in bad schools and bad neighborhoods where just entering the school lowers the chance of going to college (ie where engineers comes from)?

Bill Oberdick
Aug 29 2011 at 6:00pm

I thought the analysis was flawed in one respect. The data apparently clearly supports the finding that the better teacher has a strong positive influence on the income of the students. The example given was that a teacher at the 75th percentile level (let’s call her Teacher 75) would have a net financial benefit per student of $ X above the mean, while a teacher at the 25th percentile (let’s call her Teacher 25) would have an effect $ X below the mean. It seems , however, that at least part of the reason for Teacher 75’s financial impact on her students is that her students are more likely to get into med school, law school, a good engineering program, etc. If Teacher 25 and all those below her in percentile rank were replaced with teachers who had the same skill as Teacher 75, I do not think it necessarily follows that they would produce the same kind of financial impact that Teacher 75 had. They would not have the same kind of impact because there are only so many openings in med school, etc, and their students would be competing with those of Teacher 75 for those limited slots. All of which is not to say that we should give up on getting Teacher 25 and those below her up to the a higher level of expertise, but it seems hard to quantify what the financial impact of that improvement would be based on the data that was offered in the podcast.

Steve_0
Aug 29 2011 at 6:47pm

Bill Oberdick:

They would not have the same kind of impact because there are only so many openings in med school, etc, and their students would be competing with those of Teacher 75 for those limited slots.

(Ahem), In the short term.

David B. Collum
Aug 29 2011 at 8:53pm

Some random thoughts and an anecdote:

(1) Tenure in secondary education serves a role: It gives the teachers a chance to stray from state mandates without fear of reprisals. Some of those state mandates are the problem. With that said, I could imagine a longer tenure period (5 years) or even a progressive contract system (5 years then 10 or 15 year contracts after that). Of course, you need to buck the teachers’ unions.

(2) Russ: Didn’t one of your guests argue that you could not statistically show that you had control beyond genetics? It’s not a stance I personally would take (sounds Shockley-esque) but I seem to recall the view.

(3) I am at a research university of some calibre, and it is absolutely clear that teaching, while important, is not a full time job. I have to run essentially a small startup company as a researcher as well as serve on committees, consult at corporations, and edit scientific journals. To equate the task of teaching secondary education with that is not valid. With that said, I also happen to teach two-day short courses in industry. It is exhausting, and the PhD scientists are not wiggling and screaming. At the end of two days I am totally spent. So if pain and suffering is compensated, the secondary education teachers should be getting the combat pay.

(4) Anecdote. The school in my little burg has been ranked in the top 100 schools in the nation; I say this only for calibration. So late one academic year, my son laments how his English teacher will never give him above a B on his paper no matter what he does. In disbelief, I ask for an explanation, which had something to do with always being late to homeroom the year before because of a full leg cast. Sounded like baloney. So I do the experiment. I edit the hell out of one of his papers–not enough to reveal the help but to at least get it a notch or two higher. I got a B minus. You might think that is criminal, but I took a different view. This woman, for whatever reason, hands out an assignment that will result in probably 100 five-page papers being turned in all at once. How could she possibly grade them all? Her mistake was to demand quantity not quality. A 200 word essay would suffice if evaluation (and even teaching) was the goal. As Twain (or some smart guy) supposedly said, “Sorry for the long letter; I didn’t have time to write a short one.”

Raja
Aug 30 2011 at 1:59am

is the purpose of public, govt-funded education to educate or to produce compliant voters? i think the evidence speaks for itself, and in that sense our system has been extremely effective.

there’s almost nothing teachers can do to overcome the home environment of students, in either direction. good teachers can facilitate good students though.

siobhan
Aug 30 2011 at 9:14am

In the same day I heard that good teachers contribute mightily to their students future earnings and that parents have little to no influence on their children’s education rates, iqs, and future earnings. It seems odd to me that a teacher can change my child’s trajectory and all the math I did this summer was for nothing.

mitch mclean
Aug 30 2011 at 10:38am

As I understand it, one of the hallmarks of a successful business is understanding the customer. Leadership, management, and employees then use their knowledge of the customer to create a successful business and financial plan. What I am wondering is what value is created by studying teachers if the real black box or mystery ingredient is the student. It seems to me that Hanushek and his hundreds or perhaps thousands of academic and think tank ideologues are focused on the wrong thing. And how many resources and money do these folks waste in their pursuits? I suspect they are making a pretty good living off of the taxpayers or off of private sector institutions that profit from the current state of public education…and there are a lot of them feasting at that table.

Because I have some relevant experience in both research (Ph.D. biochemistry and molecular biology) and teaching on several different levels (college and private K-12 schools and one year in a public school), I have a growing sense that most of the education research crowd, policy wonks (academic and think tankers), politicians (all of them), bureaucrats, and so forth are mostly part of the problem and not the solution. A bit like the behaviorist vein of psychology, they are obsessed on the inputs and the outputs because these can be “measured.” But the real prize is in understanding the processes, mechanisms, motives, etc. that lie inside the heads of the students. After all, it is the students that must convert those inputs into outputs. To be blunt about it, teachers create opportunities for students to convert information into knowledge and skills. That is it. There is not much more to it than that. The catch lies in creating the opportunities that the particular students at a particular time can work with plus managing the students learning experience.

In a research lab I was incredibly particular about the inputs that went into my experiment. I wanted good controls and I wanted “pure” ingredients (e.g. water and chemicals) and instruments that performed the way they were supposed to. When I listened to O’Donohoe the other day discuss the business of snacks, I was reminded of the T.V. programs I have watched showing how modern factories convert raw inputs into some product (output). The commonality between my research endeavors and the manufacturing process was the remarkable degree to which the inputs are controlled and the processing of those inputs is managed in order to produce a reliable, consistent output.

It is abundantly clear to me that the classroom is NOT a factory. And to have business people, economists and others treat it as though it is a factory is simply bad politics, bad economics, bad business, and bad policy. Unlike Frito-Lay, teachers have little say so into what inputs go into their classroom factories. The curriculum and textbook folks do that part…for better or worse. The student as the information processor is analogous to the equipment or assembly line in the factory model. That is, it is the student that is supposed to be doing the work of converting the input into the output, right? Well, only a delusional business man would put thirty different pieces of unknown equipment on his/her factory floor and then expect to get the same consistent, reliable results. Instead, a business man goes to great lengths to make sure that the equipment can perform to pre-determined specifications. This analogy could grow into a thesis of its own, but I hope the reader can “see” where I am going with it. The point being that if you want a consistent, reliable product in either a factory or a classroom then you must understand not only your inputs but also how those inputs are going to be processed. The reason the factory is so efficient at producing measurable outputs is because all of the inputs and the processing are well understood. In fact, they are so well understood that you almost need no one on the factory floor anymore. If this is what you want in education, fine. Then design a policy to achieve it. Yet this does not seem to be what people want out of their school systems.

What happens when the inputs or the factory equipment gets creative and innovative? Unlike factory equipment, human beings have a brain. This brain is not only adaptive (which is what learning is, after all, as those action potentials, synapses, and neural circuits go into action) but a brain also comes into the classroom with some pre-programming and a lot of unknown switches and buttons that can trigger all sorts of interesting output/ noise. There are NO blank slates in a classroom that the teacher gets to write on. Thus, not only are the inputs in a classroom a bit dicey, but the brain systems that will be converting those inputs into an output via their own unique neurobiological mechanism are mostly unknown to both teacher and student.

On a personal and professional level I have a strong bias toward trying to understand the student(s) and what makes them do what they do. Just how much information can working memory hold and process. That varies a lot from student to student. Although you can tweak that, it takes time to figure that out. And about the time you get it figured out the year ends and you start over. Having had the privilege of teaching some students for several years in a row, I can say with some conviction that understanding your students makes a huge difference. The teacher-student interaction becomes much more efficient and productive. Yet when I worked in a public school I was inundated with bureaucratic non-sense, kids that played the system to the hilt, constant policy changes, never ending confusion over needs vs. wants, and so on. The incentive structure in the public school was, in my brief experience, incredibly distorted. And in a couple of the private schools that I have worked in, I found intense distortions directed toward treating students like privileged, entitled customers instead of students. Private schools have their own set of issues as well as interesting market forces to deal with.

In ending, I am convinced that no one has a handle on the complexities of education. But I am convinced that politicians, economists, think tank policy wonks, bureacrats, and the wide assortment of other self-interested “stake holders” are way off track when they get focused solely on teachers and principals. If you want to make the education system better, then I would advise spending more time and resources studying students and how people learn and why they do what they do. Then I would train/ educate teachers and administrators on how to mange the learning process of students. Time and again in my experience a good teacher is someone that knows their content; they just have to since this allows them to create a wide variety of learning opportunities. But the next level is frequently lacking, too. The best teachers are managers. But neither the colleges of education nor the system itself seems to have figured out how to train teachers to be managers of people. And in that environment, i.e., teacher as manager, the student becomes an employee, apprentice, or intern to be trained. Contrary to what I hear constantly, I am convinced that the student is NOT a customer to be catered to. The student becomes the product of his or her own process. And while few students seem to understand this, I find it amazing how few adults in high and low places seem to understand this, too.

wintercow20
Aug 30 2011 at 1:20pm

Russ, I like and respect the work of Hanushek (indeed, his early work inspired me to become an educational economist) … but I have a feeling that we all wish to oversell the importance of teachers qua teachers. I don’t think we can know very well what makes a good teacher (I am trying to push Hayek’s insights into areas I had not before). In particular it is not just that good teachers have to interact with family and other social institutions, but that the teachers are part of the rich web of local and tacit knowledge consumption and production that will remain elusive even for the best of researchers to capture. Furthermore, good teaching interacts with a number of institutional factors that are not often studied, and I don’t have a good sense for how any of these factors can make for improved outcomes. THAT is why competition and creative destruction are important, not just to make sure we have ‘better’ teachers. I know you know this of course.

On a related note, maybe you can interview Vance Fried about the $8,000 university. Just as you argue (correctly) that we need competition and innovation at the K12 level, I argue that we need it as much at the postsecondary level. I remain startled at the inefficiency of what we do here.

Seth
Aug 30 2011 at 2:26pm

Question: How is it determined if a teacher is in the 25th or 75th percentile?

rovesciato
Aug 31 2011 at 2:26am

any discussion about the quality of teachers has always led me to one observation: it is probably not possible to supply enough teachers with the necessary skills to maintain all students K-12 at a steadily increasing standard of education. It could be argued that an unfettered price mechanism would increase the demand for truly competent teachers and pull people with an innate or raw ability from other occupations; more likely, even with the tax savings from greatly reduced state budgets, a large number of people would not be able to afford the necessary fee to support the price that would generate such a supply and would resort to either paying a ‘rubber room’ fee for an education/child care bundle or neglecting to send children to school at all.

the result is that many children fail, or fall well behind. if a true price mechanism would not supply enough quality teachers to raise all boats (rather, prioritizing many children out of education entirely) it would seem to follow that vouchers would primarily shift the current supply of said teachers given that all children are still required to be put up for education. (although quality of eduction would not necessarily be given priority over cultural, religious, and logistic considerations if vouchers were nation wide and widely used; the rubber room would not likely miss its quota of voucher approved teachers)

to me it follows that the best way to supply good teachers to good students is to allow many students to fail. since it is required by law that all children receive an education, fail must mean fail to meet the increasing standard rather than dropping out of the system entirely, and this would require alternate means of education that do not focus on math, science, and reading. The obvious alternative is reviving trade schools or departments which absolve the student of the tedium of math and science and mnemonic vocabulary. In the course of learning a trade, or learning the basic mechanical competence that would leave one capable of pursuing their trade of choice, the student would learn some basic competency in reading, math and the ability to learn that would leave them capable of pursuing later education at the community college ‘high school after high school’ level as well.

another option would be to allow more teenagers to simply choose to take jobs rather than go to school. but probably the simplest way to release the pressure on quality teachers and still keep kids off the streets is to allow students who want to opt out to take the rubber room option. It sounds ridiculous to me even as i write it; why should a bunch of kids who don’t want to go to school show up so they can hang out in a room for 7 hours? I don’t know, only i suspect that relieved of the necessity to learn these subjects to this level in these years, but still in contact with those who are, we would find far fewer students opting out of education permanently than we might expect.

the bottom line, however, is that many kids are not going to reach the level of competence necessary to make a modern economy hum by the time they are 18 and we ought to other, and cheaper, things to do with them than force them through the motions of K-12. voucher or no voucher.

David
Aug 31 2011 at 5:23pm

“There is a political component” – I only heard this mentioned in passing. I think Hanushek vastly under-estimates the political nature of teacher evaluation. If teachers’ careers are going to be driven by teacher performance ratings, then teachers will spend much time on a form of rent-seeking to game the evaluations.

If you start with the assumption that any rating system will be more political (subjective) than objective, then it follows that rent-seeking teachers will be the most effective when rated.

Tenure was created to shield teachers from political and subjective treatment. Hanushek needs to be much more sensitive of this aspect of the teaching profession: administrators can often be petty, vindictive, and short-sighted.

Konrad
Sep 1 2011 at 10:59am

There is one issue that bothered me in the conversation. It was the assumtion that private schools could get rid of a bad teacher and as a result they could get better results. I am from Wisconsin and in Milwaukee we have a school choice program. The non-partisan Legislative Audit Bureau is in the fourth year of a five year study to determine the performance difference between public and private schools in the program. What they have found is no significant difference between the performance of similar students in the Private vs. Public schools. The report is on the State of Wisconsin web site. http://legis.wisconsin.gov/lab/

We are very concerned with the public school performance that we sometimes lose track of the fact that some private schools are good and some are bad just like public schools. In Milwaukee what made the problem worse was the fact that while the public students were required to take the WI Standardised test on performance the private schools were not required to have their students take that test so that there was no way for a parent to evaluate on an apples to apples basis which school is better for their children.

Russ Roberts
Sep 1 2011 at 10:18pm

Konrad,

I don’t know if that’s a good study or not. Could be. Of course parents care about many things. Test scores are one thing, not the only thing they care about. And while the WI standardized test does create a benchmark, parents have many ways to evaluate even the academic quality of a school. Tests are not the only way. My students know who the good teachers are, the demanding teachers, the easy ones, and the dull ones. They don’t need a standardized test of economic knowledge to discover quality.

Yes, there are bad private schools. But they are likely to disappear for lack of customers. Bad public schools persist–it is much harder for parents to turn to private schools, particularly in poor neighborhoods and tax money keeps flowing for a long time. So the incentives for accountability in the public school system are smaller.

Schepp
Sep 4 2011 at 12:25am

I was wondering how Dr. Hanushek’s Ideas work with Dr. Caplan’s. Caplan argues parents make almost no difference after you get rid of the grossly poor parents. Hanushek argures that 1 year of good teaching can cause $200,000 in benefits compared to the average teacher.

These two concepts seem incompatable. These two guys should get together and see if we can generate some learning. I admit my bias, it is hard for me to see one 75th percentile teacher with 23 students making $4.6 million increase to life time earnings each year.

David Taylor
Sep 4 2011 at 4:16pm

Another excellent podcast. Like others, it touches on issues that are playing out in my daily life.

My young daughter actually went backwards in one (new) teacher’s class. The next year a good teacher more than made up the difference and my daughter ended that year ahead.

My son has Asperger’s syndrome (high functioning, nearly mainstreamed) and he is very sensitive to the strengths and weaknesses of his teachers. I suspect that this is true for ADHD and a variety of other “special needs” kids. Last year, the class he was in was OK: it was a compromise between kids that he got along with and a teacher that I might not have picked. But this worked out because the terrific teacher managing my son’s progress through the school that is worth her weight in gold.

Eric
Sep 6 2011 at 7:49pm

What seemed to be missing from the conversation was an analyst of schools abroad and what contributes to the success of students doing so much better than American children. Surely we can learn from other countries who have public schools and continually do better on international tests than the U.S. Maybe another show?

[Hanushek did a podcast on this already at http://www.econtalk.org/archives/2007/08/hanushek_on_edu_1.html. See also a podcast with Banerjee at http://www.econtalk.org/archives/2011/07/banerjee_on_pov.html and one with Srour at http://www.econtalk.org/archives/2008/12/srour_on_educat.html. –Econlib Ed.]

Greg
Sep 7 2011 at 8:52pm

Liked the podcast and then doubled back and read his academic paper. I do teach statistics, among other things, and I expect variation among teachers.The three to one ratio was a surprise, mostly because I hadn’t tried to quantify it. But in retrospect I can think of cases where 4 or 5x differences are the case. There are people who have no business standing in front of other human being and pretending to teach when all they are doing is self aggrandizement.

That said, the 5-8% “bottom blow” of the education system seems like a great idea. The damage is real and that solution fits the problem.

George Winters
Sep 9 2011 at 8:07am

Studies have shown a 10x spread in the productivity of software engineers. As a manager of software engineers I can attest to the 10x and tell you that building a team of great engineers is still very difficult. One would think that such a big spread would make hiring, performance management, and firing pretty straight forward. It isn’t. Dealing with a spread of only 3x sounds even more difficult.

That being said, imagine what would happen if administrators did what is common practice in the software world: stack rank everyone (1 to n) and use that as the basis for pay increases, promotions, and performance communications.

Note that we never share the rankings with the engineers. The purpose is really to calibrate the managers’ perspectives.

lloydfour
Sep 18 2011 at 3:45pm

http://www.gladwell.com/2008/2008_12_15_a_teacher.html

An old Malcom Gladwell article that compares the difficulty of hiring good teachers to the difficulty of picking a good NFL quarterback. Dr. Hanushek’s work is mentioned.

[spelling of Hanushek’s name corrected–Econlib Ed.]

Comments are closed.


DELVE DEEPER

About this week's guest:

About ideas and people mentioned in this podcast:Articles:

      • "The Economic Value of Higher Teacher Quality," by Eric Hanushek. Urban Institute, National Center for Analysis of Longitudinal Data in Education Research, Working Paper 56, December 2010. Published version can be found at Economics of Education Review, volume 30, Issue 3, June 2011, pp. 466-479.
      • "America's biggest teacher and principal cheating scandal unfolds in Atlanta," Yahoo News, July 5, 2011.
      • "The Rubber Room," by Steven Brill, The New Yorker, August 31, 2009.
      • Education, by Linda Gorman. Concise Encyclopedia of Economics.

Web Pages:

Podcasts and Blogs:


AUDIO TRANSCRIPT

 

Time
Podcast Episode Highlights
0:36Intro. [Recording date: August 5, 2011.] Teachers and education. Provocative paper recently published. What we're going to talk about is the importance of the teacher in the educational process and how we can measure it. Obviously good teachers are better than bad ones. Want to start with this general question: What is the relationship between teacher quality and student achievement? The one thing we've found from decades of work now on achievement of students is that the most important aspect of schools is the teacher. There's been a lot of work done, some of it controversial, because what the work finds is that it's not a commonly measured attributes that we call quality, like having a Master's degree or having more experience; but in fact some people are better at it than others, and some are worse at it than others; and that's what I would call the value or importance of teachers. What are some of the magnitudes? Do we have some idea of the magnitude of what a good teacher will do relative to a bad one? Some work I did a while in the past in an inner city school in Gary, Indiana where there are all poor kids, we found that some teachers got a year and a half of learning in an academic year; other teachers got half of a year. So if you think about that, it says that in one academic year, students can come out a year different in what they know at the end. Depending on which teacher they somewhat randomly are assigned to. It doesn't take much imagination to think what would happen if you had a string of three good teachers or three bad teachers. Early on it could actually determine where you end up very dramatically. A year and a half versus a half a year--when I first heard that my first thought was: the better teachers are three times more productive than the worse ones, in that particular setting. Of course, it could be there are even better teachers that could be brought in; that a year and a half is not really--we expect so little of our students they should be accomplishing two years in a year's worth of work. But basically, 3 times is not a 20% difference. It's a huge difference. Mind-boggling. There's another way to put information about the quality of teachers, and that is comparing it to family. We all know that families are really important, and that's been a constant finding since the early Coleman report in the mid-1960s. But if you look at the difference between kids from a poor family and somebody from a better-off family, the data suggest that a good teacher 3 or 4 years in a row can overcome this family background difference. This is really important, particularly in the policy debate, because some people just go into thinking about this and say it's hopeless; these kids are simply from bad families. That sociological demographic issues are really the problem we have with America's schools; it's not the educational system; it's not the teachers, it's not the system--it's cultural problems outside of that. In fact, before we went to No Child Left Behind and started measuring achievement, holding schools responsible in some way, it was common to go into a school and say: We can't do anything with this kid; he comes from a family that hasn't helped him, so what do you expect from us? Do you have some evidence now that that can be overcome? The evidence suggests that we can overcome it. That's what we see. Now, what we don't do is systematically ensure that a poor kid gets three or four good teachers in a row. We give him a good teacher and then, to be fair, we cycle that kid through a poorer teacher. So that other kids can at least get one good year. So, that's why you don't see a lot of movement in achievement from year to year for poor kids. We are not systematically doing that much better. Let me digress for a minute in a way and ask you a question about the school itself. A colleague of ours here at Hoover told me he was visiting a school in a particularly bad part of the Bay area and he was shocked to see how many kids were in the halls. I joked with him, kind of a macabre bit of humor: Well, that really ruined your regression analysis--statistical analysis--because you say a kid's had a year's worth of school and it turns out very little time has been spent in the classroom. They've been in the school, but very little of that is educational. He said: Yes, I think they are there to meet their buddies, hang out with their friends, lunch--the social aspects of the school. If you took a school like that and you put a great teacher in that school, that teacher would struggle to make the kind of differences we are talking about, if the overall structure of the school wasn't improved, if that teacher wasn't prepared and complemented by other good teachers, I presume. I think that's absolutely correct. You have to think of what goes on in schools as more than the teacher going through lesson 14 with their students. In fact, there's a lot about management of the classroom so that kids are paying attention, not distracting everybody in the classroom. A lot about management of the school. And we don't pay enough attention to some of those things, I believe. We let kids stand in the hallway. In fact, that may be a reflection of the quality of the schools directly. If the kids aren't learning anything in class, why should they spend a lot of time in class? There's a bunch of different issues there, obviously, that could explain that.
7:35I want to go back to your original metric though, because I think it's useful to think about that three-fold productivity difference. A teacher who gets a kid a year and a half ahead in a year, versus a teacher who only gets a kid ahead half a year. For many subjects, that's not necessarily--we have an opportunity to measure that. For reading, we have certain guidelines; in math particularly. You and I have talked about this before on the program. It's hard to know exactly what's going on outside of reading and math. Reading and math, we have a lot of intense curriculum discussions; we have guidelines about what should be accomplished; and some of them are pretty measurable. You should be able to read at a seventh grade level; yes, there's ambiguity about what that means; there's ambiguity about what it means to have "mastered geometry" or linear algebra. Not its nothing compared to history or other subjects that we have our kids spend their day at. So, when you talk about quality--and we're obviously going to have to come up against this issue later--some of the quality measures are very hard to assess, I assume, because they are not like reading and math. Absolutely correct. We do have measures of performance in science more often than not; in history to some extent. There are national tests in geography. Kids don't know much about geography, according to these tests. We have these various subjects that are measured. The work I've been talking about is rather focused on reading and math, and that's because we can track how much a student gains over a year. We measure their reading abilities at the end of 3rd grade and at the end of 4th grade and we can see how much it improved. This is really important when we come to empirical work about assessing the quality of a teacher. What we want to know is what's the value added of teachers? How much to they add, given what the kid knew coming into class or knew from families or prior teachers? That assessment has been important as people talk about evaluating the performance of teachers, and maybe even paying and making decisions about their continued placement in school based upon their value added. So, we do focus a lot on reading and math. I don't think there's anything magical about our spending time on that; there's the same distribution of quality when we look at history teachers and science teachers and so forth. We just have a harder time analytically identifying the teacher impact. I think that's generally true. I think the part I would struggle with, and some of this is introspection as I struggle with my own attempts to teach well, and I think about it in an armchair way--it's not always clear what we want good teachers to accomplish. In the case of math, it's a little more straightforward; certain technical skills we expect a student to absorb at a certain point. Similarly with reading comprehension. History is trickier. Obviously you can test factual accumulation. Most of us would argue that's not good history. History is about subtler, broader causal forces you'd like a student to appreciate. Analytical ways, thinking analytically, certainly would be part of your goal as a teacher of economics as opposed to testing your students on the marginal rate of substitution equals a) ratio of px to py, b) py to px, c) px + py. So those would not be--we could easily test those things but the things we deeply care about, I think, are much harder to test. My wife's a high school math teacher and I think she struggles for example with technical competence versus subtle intuition and deeper issues that are very hard to test. I pick up on what you said at the end. I don't think math and reading are all that different from science or history, in terms of wanting people to have deep understanding and generalize from things they read or see. That's the same in math as it is in history. It is harder to test. One of the disappointments to me of the movement toward greater accountability--No Child Left Behind and the accountability system--is the tests haven't improved as much as I'd hoped. I thought going into the Federal push on accountability in 2001 that it was going to be a race between improving the test and the system surviving. It turns out that there hasn't been all that much solid effort toward improving the test to get at things we want. And I think that's one of the things that has to keep going in the future. I don't understand why it's been so slow. There are a couple of consortia being supported by the Federal government to try to develop better tests in the future, but we haven't seen them yet. It's an interesting issue because those of us who are skeptical about the value of testing--and I'm somebody who, I guess I call myself agnostic; I don't have a horse in the race particularly. I think measurement is good; I think some things are hard to measure, so when you debate these issues about accountability and testing, one view says: While some of these things are important, some you can't test. The answer has always been: We need to make better tests, for sure. Let's put the effort in to make them better. I think what the advocates of testing neglected was the political pressures post imposition of testing. In an extreme, we just had a scandal in Atlanta where students were improving their test scores dramatically; and it turned out teachers--it's not that they were giving away the answers in subtle ways, not that they were teaching to the test, things people worried about. These were more creative. They were going into the exams after the tests were created and changing the pencil marks so they would look better. That of course is an incentive problem, one of the problems. You'd have expected things like that; but this was a pretty widespread conspiracy. So, I think the political forces that forced teachers to do that similarly pushed them to advocate for easier tests nationally and to resist improvements in testing. Fascinating issue; would be great to get an investigative reporter to get into the nuts and bolts of that. We shouldn't get too hung up on the testing per se. I think that testing gives us certain information and allows us to compare across schools. But we get very good information from the normal evaluation. That's the way businesses do it. They don't test their employees, or count the number of widgets that employees produce. They can sometimes but often they can't. One of the things that's happened from looking at value-added features and going down this line of whether these tests are too narrow or that lots of people can't be tested, is some suggestion that we should improve evaluations of teachers more generally, say through more frequent and better in-class evaluations and other information.
15:59There is an interesting case in Los Angeles you may have seen about a year ago. The LA Times hired a researcher to produce value-added scores for all the teachers in Los Angeles for all the teachers it could, all 3000 teachers. And they published their results. What does that mean--put in a value-added? You follow the achievement of kids within individual classrooms and you in some sense get an average gain in learning in a classroom and compare that across classrooms. And you use that as a metric. So you are saying there were 3000 teachers; there was some end-of-the-year metric. Right. There were 3000 teachers for whom they could develop these statistical models that provided value added. Then they published these by name in the LA Times. Awkward. Public school teachers. This was not my favorite way of doing personnel policy. Firestorm in Los Angeles; still little repercussions of this going on. One of the things that happened out of that is that the school districts, and the unions to some extent, said: If they are going to publish value-added scores maybe we should have a better evaluation system. And instead of just mumbling, they set out to actually put that in place. I think over the next year there will be a much better evaluation of teachers, not based on test scores but on broader measures. They've done this for hospitals now--survival rates after surgery, infections. And of course many hospitals, correctly, claim: Our rates are low for survival after surgery because we get the hardest cases. You need to control for that. Of course, there are some bad hospitals, too. Surgeons that aren't careful. Similarly a great teacher might get assigned a challenging group of students, have modest or mediocre improvement over the course of the year, and say that's not representative of my achievement. Absolutely true. The underlying idea is to try to separate out the teacher from other influences. That's not always possible. There's actually a lot of current research going on on what is the influence of composition in the class on these value-added scores, what's the impact of measurement error in the tests over time. There are a lot of these issues that are actively being discussed. And I think the technology is improving. It's actually being used where possible in Washington, D.C. now. I think it's a pretty good system, where part of the evaluation of teachers comes from their influence on test scores but at least half or in other cases all of their evaluation comes from in-class observations of trained observers. You could ask parents' impressions of what kind of year their child had. There are obviously many creative ways to do it.
19:37I think you mentioned earlier, and certainly you talked about this in your paper, that some of the standard ways that I think most people think of as determining teacher quality are not very helpful. A lot of school systems pay their teachers not according to value added--there's very little merit pay in the public school system. Typically teachers are paid whether they have extra--a Master's degree or a Ph.D., they are paid extra per year of experience. I think you would claim there is very little evidence that those two things matter very much for outcomes and quality. Is that correct? Yes. Except for the first couple of years of teaching experience, where people actually learn their craft a little more, there is no impact of added experience on average. No impact on average of having a Master's degree. But those are things we buy, things we pay for in the schools. We pay bonuses for Master's degrees. You get about a 20% bump in salary if you have a Master's degree. It has no impact. No measured impact. We pay more for experience of teachers, but again, on average, that doesn't have an impact either. Seems reasonable, both of those, but if you think about it more than your first thought, pretty clear--in the case of experience it could go the other way. After you've taught a certain length of time you could get a little burned out; harder to maintain your excellence in some settings. You haven't gotten--learning by doing is over. Pages get pretty yellow. Let me ask a related but not directly related question. Those first few years can be very challenging; there's certainly a learning curve there. Some of it is in pedagogy, how to get the material across; a lot of it is in classroom management, how do you keep the kids from eating you alive. Fascinating how hard that is to do and how some people have a knack for it and others do not. Raises the question, if you are a principal, it's nice to hire teachers who all have experience; their students have suffered somewhere else. How do you go about hiring a new teacher when it's hard to ascertain quality in advance? Yes, you can look at their grades, whether they have a Master's degree; but those things often don't help much, as you suggest. Do we know anything about how principals successfully pick teachers? Or is it just that they just fire the bad ones? Private system. I think there's a lot of uncertainty in all cases in hiring teachers. You do what human resources departments in firms do. You have an interview with them, try to see how smart they are, how adaptive they are. In schools you might even have them as practice teachers and see how they model a lesson, might ask them to bring in a lesson plan for what they will be teaching. You do all those things and there is still a lot of uncertainty. In my opinion we just have to live with the uncertainty of hiring, but make better decisions about who stays in the classroom. So, in a private school, if it makes a mistake, thinks a teacher is going to be great; after 5 years in their apprenticeship and then 2 more years they still aren't being successful in the classroom, typically you are going to get fired. In the public school system we're kind of stuck with them. Very much so. In California, after two years, teachers have tenure. If you think of the first year as kind of rocky and say: Let's help Mrs. Smith to get better at this job and they spend the second year doing it; and she may improve her margin of what she's doing. But she's got tenure, and she's around for as long as she wants to stay. It's not a decision of the school. Typically, the most common is there is tenure after 3 years. In most states. This has actually been part of the turmoil in state legislatures around many states in the country, whether tenure remains as it has been or whether it gets changed, and what the evaluation system will be. Everybody is focused on Wisconsin and what their activities have been to change labor laws. But there have been 9 or 10 other states that are doing very similar things. A number of them are effectively eliminating tenure. You have to be able to demonstrate good performance; evaluations that are good before you get tenure, something that hasn't been in the past. In the movie "Waiting for Superman," documentary, which I recommend--I enjoyed it. I think you are in it. I was in it. They talk about this shuffling of the worst teachers around the school system. So, the teachers have tenure, but the principals know who the horrible teachers are. What happens? Is that in every state? What's that called? Sometimes called the dance of the lemons or the turkey trot or other things locally, where teachers are moved around from school to school. They are not very effective, but you can't get rid of them. You can put them into another school, but you have to accept some teachers coming from the other school. You hope that the other school's lemons are better than yours. A little sweeter. The other device for dealing with bad teachers is what's been called the rubber room. Shocking to me, actually. Tremendous article in the New Yorker on New York City's rubber room, where you take incompetent teachers and just put them in a big room and just keep them away from kids. You are still paying them, and you have to by tenure. I'm actually a fan of rubber rooms. Better than the alternative. If the other choice is having them harm kids, or paying them to sit around all day in an empty room, I'd rather do the latter. But the fact that the latter occurs once is shocking. There are teachers paid to do nothing. Part of these are people waiting while their court cases are being decided for some felony charge against them. Or they are on drugs, and while they are on detox they are in the rubber room. But more recently, teachers that are just incompetent at their teaching. They've been added in there. And they sit around because their contracts have due process clauses that allow them to stretch out any attempts to remove them.
27:45The empirical work you've done in this paper, bold and provocative, where you try to measure the dollar impact on lifetime earnings of students who get a great teacher versus a really mediocre teacher. Talk about, first, the question you are trying to answer. Obviously a crucial issue for any school system is how much they pay. One way of measuring whether pay is assessed correctly is whether people show up to apply for the jobs. You could argue if there are excess applications for slots you are paying too much. But it's always possible that you are searching in the wrong quality pool; you need to be a much higher salary. What's the value on the demand side from students or their parents for what a good teacher is? You've tried to measure that. Tell us how you did it and what you've found. The overall idea is that people say teachers are good and more achievement is good; but that doesn't have much meaning. If I tell you that in this classroom your kid is going to learn half a standard deviation more, most people have no idea what that means. Including you and me. You know it means more; you can translate that jargon into some slightly clearer measure, but you don't know what it's significance is for your child. What I tried to do is put together information we know a lot about. We know a lot about how much difference there is between a good and bad teacher. We know what a teacher at the 75th percentile does in terms of student achievement growth in a year versus somebody at the 25th percentile. We know that part in terms of standard deviation of achievement. But there is another body of literature that has looked at what is the impact on earnings of individuals if they knew more. What we found is that you earn more on average if you know more. Very systematic; estimates from very different sources have given quite consistent answers. All I did was try to put those two pieces of information together. There's actually a third. A good teacher adds some amount of achievement to a student. That added achievement, part of it carries through into the future and part of it is sort of lost. It depreciates. We forget things we learned. But part of it stays. If you carry that through to when they go into the labor market, you can say: What's the value of the achievement that stayed with the student in terms of earning? And we can project that out over the entire lifetime. I can't tell you: What's the value of a teacher versus no teacher? We don't really observe no teacher. We do occasionally: there are students doing independent study or who might be doing a tutorial online. You can debate whether that's no teacher. But in general what I can compare is a good teacher--a 75th percentile teacher--versus a 25th percentile teacher. I can say how much does the teacher in the 75th percentile add compared to an average teacher? Or how much does a teacher in the 25th percentile subtract compared to an average teacher? There's one other bit of information you have to know, and that is: How many kids is this teacher affecting each year? So, this is basically the class size. Without giving you the number or answers, let me just mention that if you think in these terms it immediately suggest that it means you should give your good teachers big classes and your small classes should go to the poorer teachers because they do less harm or less damage to the lifetime earnings. Of course, that's not the way we do it, because pay has nothing to do with the value in terms of lifetime earnings of kids, as we do it now. So that the good teachers might say: I'm not going to stay if you give me large classes. More work, harder to discipline them, more grading. The typical economist would say: pay them more. Maybe you'd then be willing to take on a few more kids on your class and it would be worth a lot. Big bang. The other way to think about that is often through the grapevine have discovered who the best teachers are. They want their kids to take those classes, and they are told there is no room. That teacher's class is full; you can't switch into that class. Certainly there are many fine teachers who can teach 35 kids almost as effectively as 17, and for the 18 kids extra who get into that class, it's a glorious improvement. Absolutely. I remember when you were teaching intermediate micro when we were together, with 150 kids. You were very good at that. Long time ago. Let me give you just a single number that comes from putting these bits of information together. If I take a teacher at the 75th percentile--rank all teachers, three quarters in--and compare that teacher to the average teacher, each and every year, according to these estimates, that 75th percentile teacher is adding $400,000 to the economy in terms of present value of lifetime earnings of her students in that year. It's more to the economy in terms of those students, because the employers of those students would also get some producer surplus. But the pay gains--not the amounts, these are the incremental amounts that students could earn over their lifetime. Absolutely. So these are not to the producer--there might be more. I have some other estimates talking about how much more that might be. But if you just cement in this number--if you think about this problem, it's symmetrical. So that the 25th percentile teacher, the one that's one quarter in, is subtracting each and every year $400,000 per student relative to the average. It doesn't have to be symmetrical, by the way; but you are saying it is. What it says to me is these numbers are big enough to think about who is in the class. Even if I'm wrong by, say, 100%--so cut these numbers in half: each and every year the 75th percentile teacher adds only $200,000 in lifetime earnings per student versus subtracting, $200,000 from the 25th percentile teacher--it says it's worth our thinking about who is in the classroom and how we manage our teaching.
35:43I want to challenge that $400,000 figure in terms of the methodology. But before I do that, I want you to clarify the claim. So, I have 20 students in a class, say. You are saying that a 75th percentile teacher relative to a 50th--average--teacher, median teacher, that that 75th percentile teacher will improve the lifetime earnings, not an annual number, they are worth an extra $20,000, say, to each student in that class. Correct? Roughly. Does it matter what grade they are in? Here it doesn't matter what grade they are in in the sense that I've allowed for depreciation at each grade. What I'm saying is that 70% of what a kid gets in the 4th grade carries on into graduation. Not in terms of that's a huge portion of their number in terms of mathematical skills, but it leads to what they learn in future years. Graduation that year? Eventually what I'm doing is projecting out to when they leave school and go out to the labor market. So, each year, if from kindergarten on through 12th grade, I through the luck of the draw in my school system get miraculously put in a 75th percentile teacher's class, versus someone--has to be random--has on average my abilities and background from my school, versus someone in the 50th percentile, I get a bonus from my acquisition of knowledge that's extraordinary. The equivalent of $20,000 every year. I can accumulate those, and I'm going to get those something on the order of $250,000 in present value terms? Yes, over a lifetime, so 1 to 1 and a quarter million dollars is the average present value of somebody in the labor market to make. Let's call it a million to make it easy. So the average person might make a million; a person who got the great teachers would be up at the higher end. No other effects, just the fact that they got better knowledge. Obviously you can debate the depreciation rate, which would matter a lot. But the thing I would worry about is that, two issues here. One is: Are teachers randomly assigned? Is the 75th percentile teacher just the really good teacher or does it happen to be the teacher who was assigned to the better students? That we would call a selectivity bias. The other question would be what we might think of as an endowment effect. This is where sort of, passing my sniff test: unintuitive that teachers make that much difference. Let me give you the counterargument and you can react to these two issue. The counterargument would be kids learn what they need to learn. You have a bad teacher, eventually you are going to figure out. Let's say you are gifted in math. If you are gifted in math, a bad teacher will slow you down, but eventually you are going to be an engineer anyway, let's say. Or a finance major, physicist. You have to believe, in your story, that that mediocre teacher in 10th grade or algebra kept this gifted math kid from achieving what they would have achieved anyway. And vice versa--that there are some great engineers-to-be who don't get to be because in 6th grade they never learn fractions, they fall hopelessly behind, not hopelessly but they are staggered by this blow of a bad teacher, and they don't do as well in the job market. A little hard to believe. I don't find it hard to believe at all. That's the idea behind all the intense policy debates about schools now--schools do make a difference. Otherwise: who cares if you go to a good or bad school, the kid will eventually learn. The parental and genetic effects dominate. And let me carry the argument further. So, I get bad luck in the draw for my 6th, 7th, and 8th grade teachers, so I fall behind in my math skills, but I'm good at math. Your story is I never catch up. I think that's in fact what happens. The kid who doesn't do very well in 8th or 9th grade Algebra I. ends up never getting to Algebra II. She gets shuffled off in another direction and never catches up because they are put on a different path. Or the other possibility is they are pushed through, passed; didn't cover all the material because that teacher didn't cover all the material, that 25th percentile teacher. This is where I struggle with it. I don't disagree; I think their ability to absorb the new material is handicapped by the past. Just hard to understand that that is going to change the salary they are going to get when they walk out of high school or college many years later. We know that people score higher on these tests when we measure them in school subsequently earn more. We follow people into labor markets and ten years later those math tests at age 15 have an impact on earnings on average. But there's another piece of information I'll give you that also comes from the same article, that we've talked about before. Nations where kids know more math, measured by these same kinds of math tests, grow faster. Other things equal. And it makes a huge difference. What this is saying is you learn something along the way, and it multiplies and adds up to having a real impact on the economy. I struggle to believe that other things are being held equal. You do the best you can to try to assess these. Crucial question would be when we observe that people with higher test scores do better with lifetime earnings, they are not perfect, they are noisy predictors, but there's a strong average tendency. What are the measuring? Are they measuring the value added or are they measuring innate ability or other factors correlated with income? I think they are measuring all of them. They are measuring innate ability--some people are smarter than others, achieve more. Some families are better at this than others and that adds to their achievement. What we've found from all of this research is that teachers can also affect what kids know at any point in time. In fact, going back to a remark you made earlier: What we see is that the difference in teacher effectiveness is largest within any school as opposed to between schools. Fascinating. It's not that suburban schools get all the good teachers and the city schools get all the poor teachers. It's that the variation within any school is much larger than the variation between schools. I agree--that brings me up short, forces me to reconsider my argument. My skepticism--I was going to say in favor of your argument, certainly parents spend an immense amount of effort and direct expenditure by choosing where they live to make sure their kids get into a "good school." Now, there could be many reasons for that, social and cultural, but what you are saying is that certain schools do better on average than other schools, but even within a school, there is an immense amount of variation in performance as measured by these tests. Absolutely. It's not only the good kids who have teachers who look like they have high value added. All of these estimates are an attempt to take into account where the kids started.
45:56So, what's been the reaction to this finding? It's a funny reaction. Part of the claim is that bad teachers are really hurting kids, and that has been picked up in some of the policy discussions. People in the teachers' unions really dislike this argument that there are a small percentage of our teachers who are really doing harm to our kids. Because this gets to the heart of whether they protect all teachers through tenure or whether they move toward a better evaluation system. I think the reaction has been largest there. There's another reaction that just perfectly puzzles me, and that is, if you talk at all about differences in effectiveness of teachers--and in part this is a tribute to good teachers, that the $400,000 a year being created--there are some people who immediately say: You hate teachers. You are against teachers. And that if you say there is any difference among teachers you are obviously anti-teacher. That completely puzzles me. This is an occupation where some people are better than others, just as economics is an occupation where some people are better than others, and being a lawyer, and so forth. And we just have to recognize that. Since this has such a huge public-policy part to it in how we run our schools, I think we have to pay attention to that in how we discuss public policy. The way our current system works--and it's an emergent system, not designed by any one person or board or committee--an invisible-hand system--in the current world, we could debate whether high school teachers are paid a lot or a little. It's a tricky thing. There are lots of non-monetary aspects of being a high school teacher: summers off, generous pensions; so looking just at salaries is generally misleading. But one thing you can't argue about is that professors at the university level make more than high school teachers. Many reasons for that. But I'd like you to chat about, think about for a second: does that make sense? Is it imaginable that that should be reversed if our invisible hand wasn't being stopped by legal and governmental involvement at both the college and high school level? Because you could argue that the value added, just on the demand side, that the demand for K-12 education should be much more important and much higher than it is at the university level, when we really are capable of teaching ourselves the whole instruction aspects very different. Similarly, you could reverse it and say: When you are 9 years old having a great teacher is not that important. It's somebody who sets you on fire in a phenomenal philosophy class that changes your life; hard to change that primitive, less-developed brain of a 9-year old. Well, that's a really valuable and important question. There is some debate in the education reform along this line. One argument is we try to use the political system to emulate what a market would look like in schools, and try to set up things like accountability systems and so forth to provide the equivalent of how firms manage their businesses. There's another group of people that say this is the culprit; what we have to do is go to a private voucher system or some sort of choice. I guess I come out somewhere in between. I don't think we are going to move to an all-voucher system in the near future, in that even if we opened up to vouchers, 80% would still want to go to their public school. And there are values in public schools. I think more choice is beneficial, and it puts a lot of pressure on public schools. You get some places where a quarter of the public schools are charter schools that have a choice-only system and aren't unionized and so forth. But at the same time, heavy political element to all of this; you have to have good accountability systems, measure of performance, make that transparent, try to build in rewards for better performance. Similar and in between. I think we can do a lot better with marginal changes to the current system. But politics are important, and the bigger politics of what goes on in the California legislature versus choice and so forth.
51:55Discussion of what's going on right now in America at the fiscal level. We have a situation right now where many states have serious financial issues. They have overexpanded, can't meet their bills. May end up being bailed out by the Federal government. Some have been effectively bailed out by the Federal government through the stimulus package which transferred a lot of money to the states and local government, and defenders of stimulus say that kept a lot of teachers from being fired or laid off. My first thought--and of course that's a great way to market stimulus. They always talk about the teachers, firefighters, and police, because who wants to go to a school without teachers or walk the streets without police or live in a neighborhood where houses can burn down. My first thought when I hear that is surely there are a lot of people in the education system, a lot of state employees in the education system who aren't teaching. So, my first question is: What room might we have in these fiscal dilemmas to reduce employment at the state level that wouldn't hurt the classroom? And secondly, in our conversations before we started this interview, you suggested this might present an opportunity in the classroom itself. So, talk about those things. Well, the first response is that our "reform" policy of the last 30 or 40 years, has been to try to put more teachers and administrators in schools per kid, so that class sizes have gone down and administrators have gone up. I looked over the last 20 years: there's been a 20% increase in the number of students in the classroom; there's been a 34% increase in the number of teachers in schools. Meaning smaller class sizes. And there's been a 41% increase in total staff--that includes both teachers and other people around the edge. The fact that that's grown more than the teachers have grown suggests that the non-teacher part of employment has grown faster than the teacher part. Absolutely. And both have grown faster than the number of students. That's exactly the case. So, we've been on this policy of just adding more and more people. And then at any point in time if there is a fiscal crisis, the newspapers immediately say: How could we increase our class sizes? Does that mean we actually have to go back to 2007 levels? Kind of ludicrous at one level when you think of how much we've reduced class sizes and added administrators over the last period. But at another level, this comes back to exactly what we are talking about in terms of teacher effectiveness. All of the estimates we have--which are consistent with what I gave you in the variations of teacher equality--suggest that we could very dramatically improve our student achievement if we could eliminate the worst teachers. And eliminate them and let class sizes be slightly larger. Not much larger. But if we could reduce the bottom 5-8% of our teachers, the international estimates suggest we would jump from below the developed country average in math and science to near the top. This has a huge impact. So, in fact the fiscal crisis could be a tremendous gain for our schools. Now, what's against that of course is the fact that most schools or more districts are under contracts that call for last in being the first out (LIFO). So that if you are going to reduce the number of teachers, you have to get rid of youngest teachers regardless of whether they are effective or ineffective. And regardless of whether you had some dinosaurs who have been around for 30 or 40 years who have been atrocious but they are protected. Precisely. So, some of the debates around the country in different states--now 8 or 9 states, maybe more--that are revisiting their labor laws for schools. Here, some states are trying to cut back on tenure to eliminate the LIFO policies even if there is tenure around, or even court orders in Los Angeles. There is a recent court order that said that very disadvantaged schools that were part of this lawsuit--the district could not lay off an undue number of teachers, which effectively meant that they couldn't just fire all the young teachers in these schools and get away with it. Fiscal problems have different meanings. The newspapers typically get them quite wrong, because they constantly concentrate on the fact that every spring the vast majority of the schools in the nation hand out pink slips to teachers saying you might be laid off, because they don't even know what their budget is going to be next year. And the contract says they have to give notice. So they hand out a lot of pink slips. Until this last year, virtually none of those pink slips have actually been executed. Teachers have stayed around and nobody has been replaced. This last year, this next year and maybe some last year, a few teachers have been laid off. Big difference in who gets laid off.