What is the appropriate relationship between judgments and measurement? Is it not the case that “if it matters, you can measure it?” In this episode, EconTalk host Russ Roberts welcomed historian Jerry Muller to talk about his book, The Tyranny of Metrics.
Is your work evaluated based on metrics? If so, do you find such evaluation reliable? Are you worried about the reliance on standardized tests at your kids’ schools? Is crime over- or under-reported in your area, and how would you know? These are just a few of the issues Roberts and Muller discuss.
1. What is the “tyranny of metrics,” according to Muller? Under what circumstances are metrics useful?2. How have metrics affected the way both teachers and students are evaluated in K-12 schools? What are the intended and unintended consequences? To what extent can or ought judgment be used instead?
3. How has the use of metrics (especially in technologies like CompStat) influenced policing? Do you think the consequences of this particular reliance on metrics is intended or unintended? Why? What does it mean to “game metrics through reclassification?”
4. What are the costs of transparency in government? Are there some areas where transparency is not to be desired? (Note Muller’s concern with what he calls “the cult of Snowden.”)
Bonus: If incentives and metrics matter, how should Liberty Fund compensate Russ for EconTalk?
READER COMMENTS
Jim Thorson
Apr 20 2018 at 8:57pm
IN DEFENSE OF METRICS AND JUDGMENT
Certainly , as in physics there is an observer effect whenever you put a measurement metric in place. You will change the system by attempting to measure it. It takes a good understanding of the system you are measuring and rigorous identification and control of side effects (unintended consequences) to avoid this. The blind use of metrics , without taking steps to minimize the observer effect , is the misuse of metrics.
The right metrics are not always the final goal you desire. In safety programs, measuring and rewarding a reduction in lost time accidents (a term defined term by OSHA) does not necessarily lead to real improvements in safety, and the system can be gamed. A better metric is to measure some action that we know leads to the desired result- for example “Close call safety incidents reported”. This system can of course also be gamed, but if it is gamed it will likely result in meeting the ultimate goal (but at a higher cost than is optimal).
I disagree strongly with the position that the some things cannot be measured. If the buyer cannot measure the difference in quality of two competing items, those items, are, from the buyers perspective, a commodity. They should buy based on the low bidder. Russ stated that there are a lot of mediocre teachers ( and I would add there are also lots of excellent teachers) How can person make these statement without some kind of a measuring stick?
It may be that that measuring stick integrates a number of factors, and that we cannot tease out what those factors are, and their relative weights. This is called judgement. Or, even if everyone could individually come up with their own list and weights ( and I believe this is possible) , there would be no consensus across a number of of people on the list and weights, and thus no generally agreed measurement.
What I believe we do know is that while the judgments of an individual ( yes, even a completely unbiased well informed individual) are not perfect, the judgement of a lot of people are often very good. Thus the efficient market hypothesis, and the relatively decent record of prediction markets.
What I don’t know is; Is one persons judgement batter than a poorly selected ad vetted metric? How many people (market participants) does it take to make a reasonable judgment? And, how do we choose and incentive the judges?
SaveyourSelf
Apr 21 2018 at 3:47pm
Is your work evaluated based on metrics?
If so, do you find such evaluation reliable?
Are you worried about the reliance on standardized tests at your kids’ schools?
Is crime over- or under-reported in your area, and how would you know?
1. What is the “tyranny of metrics,” according to Muller? Under what circumstances are metrics useful?
2. How have metrics affected the way both teachers and students are evaluated in K-12 schools? What are the intended and unintended consequences? To what extent can or ought judgment be used instead?
3. How has the use of metrics (especially in technologies like CompStat) influenced policing? Do you think the consequences of this particular reliance on metrics is intended or unintended? Why? What does it mean to “game metrics through reclassification?”
4. What are the costs of transparency in government? Are there some areas where transparency is not to be desired? (Note Muller’s concern with what he calls “the cult of Snowden.”)
SaveyourSelf
Apr 21 2018 at 5:21pm
Jim Thorson
Your comment is well thought out and written. You wrote some questions that I’d like to try and tackle for my own edification.
Is one person’s judgment better than a poorly selected ad vetted metric?
How many people (market participants) does it take to make a reasonable judgment?
And, how do we choose and incentivize the judges?
Mauricio Lema
Apr 22 2018 at 6:58am
In the war against the guerrilas in Colombia, a few years ago, some soldiers killed innocent men in order to boost their performance. This is an extreme version of a perverse incentive for measuring performance….
SaveyourSelf
Apr 24 2018 at 9:37am
bonus: If incentives and metrics matter, how should Liberty Fund compensate Russ for EconTalk?
This is a challenging question, I think, from both the perspective of Liberty Fund and Russ. The mission of the Liberty Fund is to provide a “contribution…to the preservation, restoration, and development of individual liberty through investigation, research, and educational activity.” They are a non-profit. It makes sense, I guess, that they would set up the remuneration for the host or Econtalk in a way that does not encourage profit seeking. Doing so would probably lose them their non-profit status. Russ seeking profits on the side would probably threaten their tax-exempt status too. So questions of compensation would have to be restrained to non-monetary means.
At some point in college a professor told me that people are motivated by power, prestige, and property. Power games–also called politics–are off limits as they would also threaten a not for profit’s tax exempt status. Of those three, then, only prestige remains as a means of variable compensation. So I will focus my thoughts there.
My father–a career officer in the US Navy–once told me that money does not motivate people. But changes in money do–bonuses, raises, and pay cuts–motivate people remarkably well. The same reasoning applies, I think, to prestige. I wrote a book recently and published it. I thought the measure that would matter to me most would be the number of books sold or the total profit I made–assuming I ever made a profit. But what turned out to matter the most to me was the little star rating people provided who had read the book. I think that’s the modern equivalent of measuring prestige. Russ already has everything he does open to comments. If he wanted to increase his prestige–in this case the number of comments he received each week–he could add a simple star system to each episode so that comments were not the only way that people could express their appreciation or disgust. Facebook uses a thumbs up, thumbs down system but that has proven too clumsy. I think currently the star system with option to comment is the best measure of prestige monitoring in use presently.
Even without the ‘power, prestige, property’ thing, we already know Russ is sensitive to feedback. He’s very open about that fact almost every week. Exposing him to even more feedback, therefore, is likely to make him even more responsive to his audience. At least to the extent possible within the restrictions of his own consciousness, the mission of the program, and the requirements of the Liberty Fund.
Matt Throckmorton
Apr 30 2018 at 8:50pm
When will the audio of this be posted?
[This post is a discussion of the podcast episode Jerry Muller on the Tyranny of Metrics, where a link to the audio file can be found. —Econlib Ed.]
Comments are closed.