Andy Matuschak on Books and Learning
Aug 5 2019

books-in-space.jpg Software Engineer Andy Matuschak talks about his essay "Why Books Don't Work" with EconTalk host Russ Roberts. Matuschak argues that most books rely on transmissionism, the idea that an author can share an idea in print and the reader will absorb it. And yet after reading a non-fiction book, most readers will struggle to remember any of the ideas in the book. Matuschak argues for a different approach to transmitting ideas via the web including different ways that authors or teachers can test for understanding that will increase the chances of retention and mastery of complex ideas.

RELATED EPISODE
Arnold Kling on Education and the Internet
Arnold Kling, economist and teacher, talks with EconTalk host Russ Roberts about recent technological innovations in education and Kling's forecast for their impact on learning and how they might affect traditional education. Examples include the recent explosion of online lessons...
EXPLORE MORE
Related EPISODE
Elizabeth Green on Education and Building a Better Teacher
Elizabeth Green, author of the new book Building a Better Teacher: How Teaching Works (and How to Teach it to Anyone), talks with EconTalk host Russ Roberts about the art of teaching and the history of various reforms, mostly failed,...
EXPLORE MORE
Explore audio transcript, further reading that will help you delve deeper into this week’s episode, and vigorous conversations in the form of our comments section below.

READER COMMENTS

Alston Mabry
Aug 5 2019 at 10:43am

I read several of Ardrey’s books decades ago (still remember the cover of one of the paperbacks, showing primates on the savanna), and that comment made me think about how we “don’t remember” what we read, but I wonder to what extent it *seems* like we don’t remember an author’s key arguments because we integrate them into our own thinking and then simply forget where they came from.

Alston Mabry
Aug 5 2019 at 10:44am

An excellent summary of current learning science, and with a very practical approach: https://www.amazon.com/Make-Stick-Science-Successful-Learning/dp/0674729013/

Arde
Aug 22 2019 at 6:14am

Thank you for the suggestion. I will check out this book. I am sometimes puzzled by how little time of our education is deveoted to learning how to learn. Our learning would be much more efficient if we knew how to do it better. I wish I had known some of these teqniques when I was at the university.

Paul R
Aug 5 2019 at 11:29am

For ideas to be fully understood, they must be grappled with.

I’m reminded of Russ and Mike Munger’s discussion of the middleman in the POW camp. http://www.econtalk.org/munger-on-middlemen/

To really understand why it’s okay for the middleman to earn a profit, you have to understand the service that he provides the parties that he is connecting. You must consider alternatives and relative values of use, among other ideas.

Stories and examples are so useful because they allow a student to walk through a problem themselves, to work from inception to conclusion. Most people cannot understand an idea as deeply when they simply have the answers or conclusions presented to them. A person who can derive a theorem likely understands it better than someone who only knows the final result.

Rereading books may be the the most neglected practice relative to its utility. I am reading Nassim Taleb’s The Bed of Procrustes, a book of aphorisms, which drives this point home. It’s impossible for me to digest an aphorism if I read it once without thinking about it. I must connect it to something tangible, and I must revisit it until it is fully integrated.

Michael Joukowsky
Aug 5 2019 at 2:17pm

Dear Mr. Roberts, Thank you for your podcasts. I learn a great deal from them. My definition of knowing/learning a task is being able to perform an act according to a standard. From your podcasts, I have thanked people more; I have explained to others why they matter, I understand and look for players who have skin in the game. I go back to earlier podcasts and listen. I have not always agreed with your interviewees, or think you have not been rigorous enough with your interviewees. I wish that we had a way to ask a question that you might decide will be followed up with the interviewee. I do read books more than once, I draw pictures at times to help me understand ideas and I really really love your videos. I wish you would do more. I read a book every two weeks. I am slow reader. But I always read a book with a question and that is what can I learn from this book. I am rereading Ayn Rand. So, thank you and to your great team!

Ajit
Aug 5 2019 at 2:25pm

A great discussion – one which ironically reinforced my thinking despite the general disagreement I had with both Russ and the guest. Two major thoughts:

I think I’m more convinced than ever of Bryan Caplan’s point – education is signaling. The fact that its boring slog made super painful is a feature of the system because it helps identify hard working, smart students. I would be in favor of wholesale changes to the system if the goal was to achieve better learning, but I don’t think it is. If school is meant to be an expensive sorting system/ day care center, then it needs to continue to be boring, parochial and a slog.

To that end, I think its a straw man to say books don’t transfer information. Nothing will transfer knowledge if the person does not have the intrinsic motivation to learn. I don’t think podcasts, lectures, seminars etc can ever teach someone who isn’t willing to put the effort into it.

For me personally, I can think of three textbooks that transformed my life and all three I read after graduate school: John Cochrane’s Time Series book, his asset pricing book, and the deep learning book from MIT. I went through every chapter taking copious notes and trying to apply the material I was learning. It wasn’t easy and there were many times I came close to giving up, but I was motivated and determined. Conversely, despite a reasonable motivation to learn Spanish, it simply never stuck with me. On the margins, I made the tradeoff that it wasn’t worth my time. Maybe the material could have been presented better, but no amount of plucky pedagogy is going to substitute for the willingness to learn something hard and to think critically.

So in short – I think the education system could use a lot more innovation in how its delivered, but that’s not what its intended and even if it was, it still relies on internal motivation, which I don’t think comes from how the material is presented.

JK Brown
Aug 5 2019 at 9:11pm

Why books don’t work is because the one thing the school (don’t use the misnomer education) systems won’t teach students is how to do their job, i.e., study.  Nor is the school system set up to train students to follow Bacon’s advice:

Read not to contradict and confute;
nor to believe and take for granted;
nor to find talk and discourse;
but to weigh and consider. – Bacon

If you are reading books fast, or a lot of books in the year, are you really taking the time to think about what you’ve read?  Kids who take time to think in class are admonished for not completing their assignments, which often only means to read to find some answer to the “reading comprehension” questions, or falling behind the class.

Several years ago, I came across How to Study and Teaching How to Study (1909) by F. M. McMurry.  McMurry advocated to start teaching students how to study, which he broke down into eight factors, starting around 3rd grade.  First and second being focused more on the vocational skills of reading, writing, numbers, etc.  But in 3rd grade, students could start learning how to apply the factors of studying instead of being left with the command to study and only advice on finding a quiet place, etc., and nothing for what to do with the book in front of them other than the obvious re-reading blindly and without advice on how to parse the text.  I’ve a theory that the “good students” are the ones who rediscover how to process a book.  It explains why kids from an educated home environment often do better than those not simply because an educated parent will have also developed study skills and thus are a good example if they don’t teach it overtly.

 

The factors of studying:

1. Provision for Specific Purposes

2. The Supplementing of Thought

3. The Organization of Ideas

4. Judging the Soundness and General Worth of Statements

5. Memorizing

6. The Using of Ideas

7. Provision for a Tentative rather than a Fixed Attitude toward Knowledge

8. Provision for Individuality – develop your own opinion before you consider those of others.

You may notice that a textbook is organized so that the student is not required to do factors 2, 3, and even to an extent 4 for themselves.  As for factor 1, the syllabus provides the purpose.  So most student skip to factor 5 and concentrate on memorization without the thought required in 1-4.

Recently, I’ve been reading the concluding discussion of the weakening impact of teacher-led instruction and the need for the student to be pushed to develop the initiative to learn on their own, developing their own questions and their own plan of attack.

But there is a very striking difference. When the two studied together, the teacher not only did a part of the work, but she was the leader; the pupil was a follower, doing only the subordinate part. Now, being alone, he must do the principal part, in addition to the other. He must divide his topic into parts, and conceive all the questions that are worthy of attention; in brief, he must determine the course of procedure himself, or take the initiative. Herein is found the great difference between studying with a teacher and studying alone, and it is a fundamental one. Capacity for self direction or initiation is not necessary in the usual class instruction; but it becomes indispensable the moment one undertakes independent study.

 

Nick Ronalds
Aug 29 2019 at 8:16pm

Beautiful Bacon quote.

James Gladstone
Aug 5 2019 at 11:56pm

Interesting topic. Perhaps the guest should write a book about it.

Thomas Hockman
Aug 6 2019 at 3:40pm

Good episode.

I am keen to get together with listeners every month to discuss Econtalk.  I am in Orlando.  We can meet in one of the coffeehouses so popular with the local MeetUp.com crowd.

I like Andy’s experiments with increasing time between subject review.  Most of my study methods came from the back of Evelyn Wood’s book on speed reading from 1930 something?  Aka the “Where there’s a will there’s a way” course advertised by Three’s Company’s Jack Ritter.  I am eager to learn a method that is less than 80 years old.

 

I have only read Robert Ardrey’s The Territorial Imperative, 1968.  It argued that the main motive of animals was territory, not sex.  He was widely ridiculed as too uptight to understand that everything is about sex, including a book to refute his foolishness.  Territory as the primary motive is now taken as obvious.  Which of his other books should I read next?  By the way,  The Territorial Imperative is very easy reading.  It is mostly notes of observing animals acting funny.

Bob Uva
Aug 6 2019 at 4:26pm

Hi Russ,

This was a good episode which could take us in many directions of inspiration and challenging methods. I do want to comment on your experience reading Robert Ardrey’s work. When I was in high school, I think it was 1973, I read African Genesis. It completely absorbed me and was probably the best book I had ever read to that time. I’m not sure how I had heard about it — that would be a memory I would like to recall. When an alumnus of a local college, Harvard, came to our school (in a working-class inner suburb of Boston), I had an enthusiastic response to his question of what book I had recently read. We, the alumnus and I, seem to connect after that. I don’t know if was instrumental or not, but I did, unexpectedly mind you, get into Harvard College starting in the fall of 1974.

On the topic of gleaning more from EconTalk, I love the idea Andy suggested for circling back on show comments in later episodes. I would also like to participate in EconTalk discussion groups, preferably in person in the local area (I’m now in Portland, OR).

Thanks,

Bob Uva

Matt Austin
Aug 9 2019 at 12:17am

Another great episode, and filled with great insights!

I would love to participate in EconTalk discussion groups, with a preference for in-person meetings in the Bay Area.

Cheers,

Matt

Paul A Sand
Aug 9 2019 at 6:26am

Good episode. I haven’t read it yet, but the new book Creatures of Cain: The Hunt for Human Nature in Cold War America appears to have some discussion of Ardrey. The book is reviewed in the current issue of Reason.

Michael Bromley
Aug 9 2019 at 11:42pm

Every pedagogical fad, from ELOs and other “Standards Based Learning” systems, cooperative and differentiated learning, technological platforms, etc. etc. is but an attempt to get around the intractible problem in education that outcomes are measurable and thereby manageable. 

In education, sadly, the cart is ever before the horse.

Once we get past the idea that every kid can learn every thing, we are free to work on what’s possible. And, since what is possible is contingent upon each student’s conditions and choices, and, since we cannot change those conditions, we must work with student choice. When educators address purpose, everything flows therefrom.

Textbooks, for example, can be enormously efficient learning tools, except that teachers rarely if ever teach students what is and how to use a textbook and why.  It doesn’t have to be that way.

Teach purpose and learning will follow, regardless of the technique or environment.

Mauricio Lema
Aug 11 2019 at 8:26am

Hemoglobin has four units that have the ability to carry oxygen. It is harder for the first oxygen to be uptaken by the first unit, and it is very easy to uptake the fourth oxygen by the last one. That phenomenon is called cooperativity, and it explains the oxygen-hemoglobin dissociation curve which happens to be sigmoidal. When I read something for the first time, I learn nothing. If I read about it the second time (even from another source), then I will grasp it better, in a cooperative-like fashion. That’s when I realize that I did get something from the first reading. Some seemingly unreachable neural plasticity is stored somewhere that can be accessed under some circumstances that can not be easyly testable.  In a sense this same concept may help to also explain why people who read more, appear (to me, unscientifically) to be able to solve problems more effectively, without explicitly invoking the knowledge-set that facilitated such solution…

 

Great show, as usual, I hear you here, in Colombia, and it makes me feel as a part of a community interested in “things” that are utterly ineresting by themselves.

bogwood
Aug 14 2019 at 10:40am

The Italian philosopher, Msgr Sarducci,  would go a step further.  If you remember the average amount of material from a college level material, you get full credit for the course.  “Hola” would suffice for Spanish 101. Maybe one other phrase for the second semester.

 

Erika Koster
Aug 14 2019 at 12:21pm

I absolutely loved this episode. I plan to listen to it again at a decent interval, and check out all the associated links. Thank you for introducing me to Andy Matuschak!

Glenn Craig
Aug 14 2019 at 7:03pm

Hello Dr. Roberts,

I’m a devoted listener, and with two daughters leaving for college in September I found your discussion with Mr. Matuschak about good learning habits  (my #1` takeaway)  most stimulating.  I’m going to try to get them to listen to the podcast again with me, but it may be a hard pull.

I’m writing to volunteer to host one of the groups you thought of starting to discuss Econtalk episodes.  The podcasts are invariably stimulating, but like most, I suppose, I don’t retain much.  I’m willing to host a virtual group, but would prefer one in real place.  So if you have listeners in Wellington, NZ, I’m your guy.

 

 

SaveyourSelf
Aug 16 2019 at 8:12am

Language, both spoken and written, are constellations of different forms of symbols. Since symbols don’t actually exist—they are artificial constructs—being exposed to them does not typically trigger an emotional response. Since so much of our memory functions are tied to and traverse through the emotion centers of the brain—the Limbic System—, lack of an emotional response may actually prevent remembering, at least in the long term. (Strong emotions are flags for our brain to make strong memories. PTSD is a phenomenon where very strong emotions create memories that are difficult to forget.)
The brain—I can’t remember where I read this, go figure—takes in sensory data from all five major sensory systems—vision, hearing, taste, sound, and smell—simultaneously. The more of those senses that are actively involved in a given experience, the higher the likelihood that, at some future time, the brain can accurately triangulate its search efforts to recover where it stored a memory. If all you use to make a memory is your eyes, like when reading, then that memory has only a single point of reference to identify it for recall in the future.
(I think this one is from The Brain that Changes Itself by Norman Doidge). You all may be familiar with the fact that if you don’t use your muscles for some reason (you are in a coma or in a cast, for example) your muscles will atrophy. But if you use your muscles regularly, they will get stronger. There is a similar phenomenon—it may actually be the same phenomenon—in the brain where repeated use strengthens the dendritic connections between nerve cells in the brain. Lack of use of those connections leads eventually to the connections breaking down and disappearing. Memories may literally disappear due to brain atrophy. Regular repetition may be required for remembering.

 

If forgetting is so profound and predictable, then what is the point of school? You are going to forget (nearly) everything you were taught. Isn’t that a colossal waste of time? Or is school primarily useful because you are forced, year after year, to read. And, even though you forget everything you read, you retain the ability to read and thus retain that most important skill necessary for making and accessing permanent memories recorded in print format in the future. Thus you can find memories you or someone else made in print when you can’t recall them with your brain.

 

I’m really happy Russ and Andy Matuschak did this podcast. The importance of forgetting in how we function as individuals and as a society is…underappreciated. Probably because we don’t remember what we forgot. So we “think” we never forget. Once we recognize that we forget nearly everything over as short a time span as a single year, we can begin to answer the important questions of our age like how best to teach our children so they can thrive in the future and, perhaps, how to structure our environment so we are occasionally exposed to what we think is most important for us to remember. (The actual text of the Constitution, the actual citizenship contract, little tips like don’t drive drunk, don’t text and drive, look both ways before crossing, etc. etc.)

SaveyourSelf
Aug 16 2019 at 8:28am

I think universal forgetting along the lines of the forgetting curve is the strongest explanation of why specialization leads to increases in productivity. Daily re-exposure to the same details gets around nicely the problem of forgetting. That may also partly explain why the most efficient rationing system we know of is networked specialists trading with each other rather than the most educated among us ruling as philosopher kings.

Stuart Berman
Aug 18 2019 at 11:41pm

I was introduced to Robert Ardrey by my stepfather decades ago, I remember both The Territorial Imperative and African Genesis fondly perhaps due to my own early career in theater.

Today I am a professional in information technology (security architect) and I enjoy listening to your podcasts and working my way back into your older podcasts.

 

 

Arde
Aug 22 2019 at 7:08am

I agree with Andy Matushak that normally we absorb and remember little when reading a book. I also agree that often we think we understood the book, but when we scratch a little bit the surface, we realize that we did not understand much. I also absolutely agree that to understand something, you must actively engage with it.

I disagree with the title of his essay because it is contrary to my experience. With all their imperfections, books have been for me by far the greatest source of knowledge and understanding. I have been exposed to some of the alternative learning techniques, such as group discussions, group work, peer-to peer feedback etc. For me the effectiveness of these methods has been far lower than that of the books. These techniques did not add much to my learning (with some exceptions) because often other students were free riding, they were not really interested in the topic, they gave very superficial contributions, they wandered into other topics, professors were bad moderators, many students were ignorant about the subject and I could learn little from them. Often these things have been a waste of time.

I also disagree with Andy Matushak’s dismissal of the tactics of how to learn and read.  I think it is still very worthwhile to learn them. Maybe later in the future there will be new mediums of learning, but while we don’t have them the best we can do is to learn how to read. I would suggest everybody to read a book “How to read a book?” by Mortimer J.Adler and Charles Van Doren. The books is quite old (and in places a bit outdated and too lengthy), but overall it is extremely useful. I applied the techniques from the book and it significantly changed the way I read and what I absorb from a book. I regret that I did not come across this book when I was a student. I could have been much smarter than I am now. Andy Matushak is right when he says that we must ask questions while reading, and this book gives very good advices on which questions to ask for different types of books and for different stages of reading. It gives advice on how to read different types of books (reading non-fiction is different from fiction, reading economics is different from reading medicine etc.), the speed of reading (you should know what to read fast and what to read slowly) etc. Maybe there are more modern alternatives to this books (and thanks to Alston Mabry for his suggestion in the comment above), but I can’t overstate the influence this book has had on me.

Chris Screwtape
Aug 26 2019 at 11:11pm

Russ, I’m under 35. When you said Robert Ardrey’s name, it sounded familiar. I knew I never read his work, but I knew the name; his books, particularly African Genesis, influenced the opening section of 2001: A Space Odyssey. Some of the artwork in the book also influenced the star child sequence at the end. So, in a way, a lot of people have been exposed to his work in a meaningful way — those are some famous scenes. They just don’t know it.

As to the topic, sometimes I think it would be better for me if I just picked twenty or so books and stuck with them, instead of trying to read at least one or two a week. But I think I’m more sympathetic to Umberto Eco’s stance: being educated (I think he says cultured or civilized) isn’t knowing a fact, it’s knowing where to go on your bookshelf to find out.

You and your guest talk about monologues. That’s the culture we live in. In Amusing Ourselves to Death, Neil Postman talks about “serious” television. There’s a section describing a televised discussion on nuclear weapons or something. Postman says the public intellectuals invited, Kissinger, Wiesel, McNamara. Sagan, Buckley and a general,  just give monologues. No one wants to give up their precious airtime to another persons ideas. I actually think the internet is changing this problem a bit — I won’t say it’s for the better.

In the New Statesmen last week, they did a piece on a man that reads and reviews something like 90 books every five weeks or so. He puts out a magazine, reviewing them. I think he would make an interesting guest for you; he not only reads but also engages with what he reads. His name is Ed Needham. He used to be an editor of the Rolling Stone and some other magazines.

Finally, if anyone wants to meet up in Shanghai midweek to discuss Econtalk, I’m happy to do so. Just reply. I’d prefer meeting someplace in downtown Pudong or near Jing’an Temple. But if it has to be the French Concession, I can make that work. Any takers?

Dr. Duru
Aug 27 2019 at 3:20am

I definitely like “Why Books Don’t Reliably Contain Detailed Information” better as a title. “Why books don’t work” sounds too absolutist. I came away from this podcast appreciating books even more, especially the value in returning to challenging books to make sure the material is truly well-understood.

“Effortful engagement with ideas” is also a phrase that will stick with me. I like the way it captures the value of critical thinking and discourse.

Comments are closed.


DELVE DEEPER

EconTalk Extra, conversation starters for this podcast episode:

This week's guest:

This week's focus:

Additional ideas and people mentioned in this podcast episode:

A few more readings and background resources:

A few more EconTalk podcast episodes:


AUDIO TRANSCRIPT
TimePodcast Episode Highlights
0:33

Intro. [Recording date: July 1, 2019.]

Russ Roberts: My guest is... Andy Matuschak.... Our topic for today is a fascinating essay of his called "Why books don't work."... What's wrong with books? I like them. This essay of yours is a little bit of an anti-book screed, and I'm quite, actually, sympathetic to it. But at first glance, the idea that books don't work is deeply disturbing. So, make the case.

Andy Matuschak: Yeah. I think perhaps I need to roll back on the title a little bit. Perhaps a little bit of clickbait there. So let me begin by saying why I love books. Now, first off, and books are [?] the author of our cultural history, and certainly oral history has a great role to play there as well. But certainly insofar as we've been able to build on previous generations' knowledge reliably, and construct cultures that we're proud of, and indeed contribute to future cultures, much of that is occurring through books. Great! Separately, there's also the wonders of fiction: we're also able to live in other people's worlds, experience what could be. Absolutely fantastic. So, perhaps a better title for this essay would have been something like "Why Books Don't Reliably Contain Detailed Information."

Russ Roberts: And, in particular, nonfiction. Although, we learn much from, I'd say, from fiction. But the type of learning we expect to do from nonfiction is different.

Andy Matuschak: Sure. Now, sometimes we read nonfiction for entertainment purposes. But, sometimes we read nonfiction because we genuinely want to learn about the topic, and we really want to understand.

Russ Roberts: And, we want to get smarter. And, I think one of the challenges of your essay, which is a challenge of this whole area, is: What do we really mean by 'learn'? Or, what do we really mean by 'remember'? Or, what do we mean by 'grasp'? But, at the heart of the challenge that you identify with books is something, a framework I had not heard before called transmissionism. Explain what that is.

Andy Matuschak: Sure. It's a bit of a straw man really in that no one actually advocates for it, but essentially transmissionism is the false idea that you can come to know a thing by having that knowledge directly transmitted into you, as if you are a slate that I as the teacher can write upon. And so, in some sense, books are kind of imagining, or an author is kind of imagining that they can write an explanation of a thing, and that a reader can read that explanation, and then they'll come to know it. Now, sophisticated readers don't stop there: they know they have to do more. But it's hard to do it, and it actually the medium itself doesn't do much to convey what else must be done in order to learn.

Russ Roberts: And you make a parallel which I find quite interesting about lectures, because I've given quite a few lectures in my life; and I've attended, mercifully, a smaller number, I think. And I'm struck by many of them as being of no obvious value whatsoever. It doesn't mean that there isn't value because there are subtle things that can be gained from listening to someone speak. But, as you point out, often a very short period of time after that lecture has passed, or if you've read the book, you don't remember a single thing. It's not that you didn't get everything the author or lecturer intended. You get, literally, nothing, at least that comes to your conscious mind.

Andy Matuschak: Yeah. Especially months later. Lectures are maybe an easier thing to attack. I think we kind of have an intuition that, 'Oh, you attend a lecture,' and you might say something like, 'Oh, that was interesting.' Or 'that was helpful.' And then: Did it actually help you understand the material? Well, maybe, if after the lecture you did a whole bunch of stuff: You did some problems that suited some projects. But, if you just listen to the lecture, you probably don't walk away with all that much, except perhaps excitement, emotion, things like that. And that's because lectures, like books--as I was describing before--are sort of founded on this transmissionism notion: the notion is that 'I as the teacher can get up there and say a bunch of words; then you'll know stuff.'

Russ Roberts: But it seems to me, to be fair to lectures and to give them their due, and I think also somewhat to books--and we won't spend too long on this part, saying nice things about them. But it seems to me that much of what we expect to get from them is exactly what we get. We're not really expecting to learn a massive amount. We're not expecting to learn how to master a subject. We want to be exposed to the working of a mind, someone's mind that appeals to us or that we think will inspire us. You know, I think about a weekly sermon as an example where, you're not really expected to learn anything. It's supposed to touch your heart, perhaps, change your behavior--it rarely does, I suspect. But perhaps the accumulation of that contact and that mode has an impact on you. And similarly hearing a lecture of a great scientist or a great thinker maybe inspires you to take your life more seriously, say--not to be fully aware of how the genetic process works or quantum computing, or anything like that.

Andy Matuschak: Absolutely. I agree completely. And perhaps to state a mechanism even more specifically, I think that one idea that these things can convey that runs through those examples is that they can convey cultural norms, cultural values, or perhaps even the single individual's values really quite effectively.

6:10

Russ Roberts: Yeah. And that's a powerful thing. But, where I agree with you is that much of it is, I would call it, a pleasant way to pass time. We call that entertainment sometimes: I think that's somewhat demeaning, I think, to a great lecturer or to be exposed to a great thinker, but it's, as you, I think--you didn't use these exact words, but it was like, 'Yeah; I enjoyed that.' I think you said, 'It was interesting,' I think something we often say after a lecture. And it could be--that things were stimulated in our brain that we didn't, you know, fully appreciate or think about. But, as a way to convey--I would have us try to focus on two things: Information, one; and then wisdom. Obviously wisdom is much harder to convey, and the richest type of learning that I feel I do is--'wisdom' is a grandiose word for it, but, seeing connections between things I didn't see before; understanding that something I thought, I didn't realize applied to something else; seeing an analogy that might help me understand how to think about something--that's what I call, you know, wisdom, deep learning, whatever phrase you want to call it, as opposed to information. You know: The capital of California here is Sacramento. Or: The Boston Red Sox won the World Series last year. Those things, which are all accessible via Google or DuckDuckGo in a very, very short period of time--I view those as different. And I don't go to a lecture to get those facts. I go to a lecture to get something closer to wisdom.

Andy Matuschak: Indeed. And it would be better, perhaps, if lecturers actually operated according to those principles. Now, one thing that we don't know--we, kind of referring to scientists as a field--is the degree to which that kind of declarative knowledge, the information as you call it, has to be deeply internalized before a person is susceptible to the wisdom. Uh, mmnnn, lots of differing views on that: it's kind of difficult to study empirically. But it does seem that to some degree, you need to be familiar with the core components of whatever topic of wisdom is attempted and to be conveyed.

Russ Roberts: Yeah; you know, my wife's a math teacher, as listeners know. And I've been influenced by the psychologist Daniel Willingham. And he's a big fan--as am I--of, it's very hard to get wisdom without information. You need a base level of information: you need, for math, you need your multiplication tables. You need some historical facts. You can't just theorize about history without understanding some really tedious factual things. At the same time, factual things alone are not knowledge. They are just facts. And I think the challenge that we face as our time gets more valuable--and I know you are aware of this, so I'd like your opinion: The temptation to reduce everything to a multiple choice exam tends to force us toward spit-back information and move us away from wisdom, is, I think, part of the challenge. It's certainly the elementary-to-high-school educational problem.

Andy Matuschak: It's a huge problem. Wittgenstein has this wonderful quote, 'a wheel that turns, although nothing turns along with it, is not part of the mechanism.' [Philosophical Investigations, Ludwig Wittgenstein--Econlib Ed.] And so, if you've simply memorized a whole bunch of piece of atomic information and yet they don't connect to each other, you are not able to combine them into anything. You are not going to get that far. Learning science gives us a couple of structures for thinking about this that I find helpful. One of them is Bloom's taxonomy for thinking about tasks. And, it begins with being able to recall things, reasonably; and it moves up through being able to, say, apply things, take some procedural knowledge, perform a procedure. Perhaps initially a very rote procedure and then perhaps one with some flexibility later. Finally it moves through being able to evaluate and make critical arguments. And ultimately to synthesize or create something anew. A different set of structures I'll bring up, and then hopefully that will be helpful for the conversation, is transfer. So, initially when we learn things we tend to apply them in a context where we're told exactly what to do: 'You need to perform this procedure on this data.' And then maybe we remove one of the phrases from the sentence: 'Perform some good procedure to achieve some good outcome on this data.' And then, the challenge becomes just knowing what kind of problem is it, and finally, what we call 'far transfer,' is a problem where you are--you the student, or you the learner--are forced to combine skills, things you are learning, in a completely new-to-you environment, that definitionally cannot be practiced via drills.

10:48

Russ Roberts: Yeah; I've probably mentioned this before: My favorite course evaluation I ever received was, I got a 1 out of 5 from a student who said, 'Professor Roberts is a terrible professor. He expects us to apply the material to things we've never seen before.' Of course, I warn people on Day 1, that's the essence of the class. And I think that is great teaching, when you can pull it off. It's not easily done. Not every student can do it. And I'm ashamed at how the techniques that I used when I was in the classroom to convey that process--my standard technique was: 'Here are some really hard problems. Think about them.' And I was not able to give the feedback that they needed often to help them see why their incorrect answers were incorrect. And that's extremely important, obviously is, to make progress. And we'll talk a little bit about that general challenge of feedback, because I think it's a fascinating one. And it's increasingly challenging as people's time gets more valuable. But, I want you to just talk a little bit about that open-ended versus multiple choice level of understanding. At one point in your essay you talk about textbooks. You'd think textbooks would be written and designed to be--to acquire knowledge, wisdom, etc. People write me sometimes, 'What econ textbooks should I read?' And the answer is, 'None of them.' They are not designed to be read. They are designed to complement an often extremely dry, multiple-choice-driven class. And they are explicitly not designed to convey information. And I think that's--it bothers me.

Andy Matuschak: Nyeah. I'm happy to at least try to speak to that. Unfortunately, we don't have all the answers there. But, what we do understand of how what you call wisdom is acquired, or maybe the ability to, mmm, apply far transfer or to flexibly and fluidly take skills and use them in different contexts, that comes about through a variety of different mechanisms. For instance, explaining things that one has not seen before--using those concepts. Or, making critical arguments using those concepts. When I was a Khan Academy, my R&D group worked on this lengthy project to try to, instead of having multiple choice questions for everything, actually have open-ended activities for things. We were very focused on AP [Advanced Placement] Humanities. For instance, in history if you want to learn to make an argument for what historical causation is, that's a pretty complex thing. Historical causation includes things like, 'It couldn't have been otherwise.' It includes a sense of 'How important was this cause,' versus a different cause. And, so, if I were to ask you, 'What was the most important cause of World War I?' that's something that, mmm, if you were to give that as a multiple choice question, it wouldn't teach you very much in terms of wisdom for historical causation.

Russ Roberts: I would say, 'None.' A typical question like that--not on the AP exam, although I will say, as I've ventured[?] before, that I wouldn't have done very well on the AP History exam, even in areas that I understood the history I think fairly well. And I certainly wouldn't have done well on the AP Economics exam, recently, as my kids struggle with them. And I said, 'These are terrible answers.' But on the history exam, even if I said, 'The assassination of Franz Ferdinand by Gavrilo Princip is the cause of World War I,' and then the next three were related to, let's say, Walt Disney movies--well, clearly a). is the right answer. But you've learned nothing. Now, there are subtler answers that you could give. But they wouldn't be right, either, even though they might apply to World War I. And the challenge is, as you point out, isn't teaching people what the right answer is. It's teaching them how to think and weigh evidence related to answers that are inherently not well-defined as correct or false.

Andy Matuschak: That's right. And indeed that was actually the approach with this project. Before, you mentioned that most textbooks are defined not to be read on their own, but in fact to be used alongside a class. And, if that class is a multiple-choice oriented, very rote class, that's an unfortunate thing; and probably you are not going to walk away with a lot of wisdom. But, if that class is a seminar run by a competent facilitator, then you might get something pretty wonderful, where, the contents of that class, day to day, are discussions. And debates. Some one person raises an argument for what indeed was the cause of World War I; another person refines that and says, 'It's interesting that you point out this cause, because I think actually if that cause had been removed, nothing would have changed, for this and that reason.' And, together the class thing co-creates an understanding of what truly is important for causation. There's modeling happening. The teacher is interjecting, perhaps with a really good example of an argument at a given moment. A teacher might ask a probing question that is meant to highlight a way in which a student's conception is unproductive. Or a way in which it might be enhanced. And, indeed, the students will surprise each other. And so, it kind of got me--we kind of tried to create that kind of thing online.

15:59

Russ Roberts: And you wrote a paper about, which I read, and it's a fantastically provocatively idea--

Andy Matuschak: Thank you--

Russ Roberts: The reason I found it fascinating is a lot of it is built around the idea of peer-to-peer reaction. So, describe what that, how that model works, at least in this case, and whether you think it has, you know, potential to be more widely adopted.

Andy Matuschak: Certainly. So, first I want to kind of describe what it's not. So, other people have tried to do open-ended learning activities online before. And often what that looks like is a peer-to-peer interaction. But, it's grading. So, it's back to the right-versus-wrong. It's: I wrote an essay, you grade my essay, and I grade your essay; we give each other scores. Maybe there's a rubric. The rubrics can help; it's not nothing. But we are really focused not on assessing understanding, but developing understanding. Those two things are different. And, we think that developing understanding looks a little bit more like that classroom, where there's an exchange of ideas and exchange of values. Now, we don't want, kind of the blind leading the blind type thing happening. So, we don't--actually many teachers do--just create a discussion forum--and say, 'Students please discuss between classes.' And sometimes that works. The evidence suggests that mostly it doesn't. So, we want some kind of facilitation, some kind of orchestration. When the teacher is there and they are skilled, they can supply it. But, if they are not there, or, indeed, if you want a conversation happening between every pair of students, then they can't really supply it. So, the solution that we build kind of offers structured scaffolds where students are presented with each others' arguments and then they are given almost like the little hand of cards, for, like, things to try. 'Oooh. I'm going to not disagree with you or point out a limitation of your argument, but actually one of the things I'm allowed to, card I'm allowed to play on your argument, is: I'm going to add a piece of evidence.' Or, 'I'm going to suggest a different implication of the argument that you are making that actually makes it even more powerful.' And, students are not only doing that with each others' work, they are also doing it with model work, that either the curriculum authors or the teacher supplied, so that they can interact as well, possibly without knowing, with examples which are structured to highlight either, mmm, particularly good manifestations of some facet of the skill, or faulty ones.

Russ Roberts: Now, is anything happening with that project?

Andy Matuschak: Ah, right. So, we finished that project mid last year. And, it's kind of on the queue. But, Kahn Academy is a pretty small nonprofit. At the moment my understanding is that they are focusing mostly on middle-school math, where this particular set of skills is not quite the priority. Which is interesting. Because, actually, in the new Common Core set of standards, there is this wonderful list of mathematical practices, ways of thinking like a mathematician that students are supposed to acquire, that are usually de-emphasized relative to the 'you should be able to multiply two digit numbers together.' And, my favorite one--of 8--is to be able to construct viable mathematical arguments and critique the reasoning of others. And, I wish luck to those constructing platforms using only multiple choice for that.

Russ Roberts: Yeah; that's kind of tough.

19:10

Russ Roberts: So, one of the most interesting pedagogical insights I was ever exposed to was, I had the opportunity to spend an hour with Orson Scott Card, science fiction writer. And, he teaches creative writing. And at the time I was teaching economists how to convey economic ideas in writing. Which was, of course, I was teaching at George Mason, I think it was the only course in the country [?] like I'm pretty confident. And, I asked him for his advice. I was just starting to teach that class. And I didn't know quite how to do it. Or at least I thought I didn't know how to do it, but I thought he might know better. And he did. His suggestion was--and I didn't implement it as well as I might have, but I did the best I could--his suggestion was to have the students give each other feedback. Not literally grade, but critique the writing of the other students. And, the reason that's so deep--one thing is it saves the teacher work, which, of course, is always appealing at a certain level. But, I think it's better than that. It's a win-win, because, I think being a great writer requires being a great editor. And, it's very hard to learn how to edit one's own work. And, by editing the work of others, you learn how to edit your own work. And that, I think, is an incredibly powerful thing. Because, what you get instead is a teacher who writes in the margin in red pencil, 'Awkward,' or 'Confusing,' or 'Bad example.' And the student gets it back, and they never look--first of all, they rarely look at it other than to say, 'Boy, there's a lot of red marks here. I thought this was a better essay than my teacher thought.' But I think the idea of learning how to fix an awkward sentence--and of course, in an even better class would be, would be, just write 'Awkward,' you write your better version of the sentence that would improve it. Because then it teaches you how to fix your own sentences. And I just think that is--that's an incredible insight. And I just want to take it one step further. Which is, to pretend that EconTalk is an educational activity. And one of the things I like to think of is that your dialog with mine is echoing a dialog that a listener would have with you. Or with me. And, that process is one of the ways we learn--is through the challenge and the back-and-forth. And I think how little of that is in our current education system, how little of it is in our books, lectures. It's all--it's transmissionism is all the way down. Now, for better or for worse, in high school classes, transmissionism is, I think, getting reduced because kids can't even sit still to pretend they are being transmitted information to. And so, it is much more interactive. It seems to me that interaction and grappling with the material is the only way we really learn.

Andy Matuschak: You'll get no disagreement from me there. Indeed, if you want to learn how to learn how to edit prose effectively, editing others' prose and your own prose, editing the best prose in the world, it's difficult to imagine better ways to do that. And, to the great credit of so many [?] of teachers, particularly recently, there's been a huge shift to interactive classrooms. Yes, because attention spans are shrinking and it's just more interesting and fun; but also because there's a growing recognition that even when one is learning math, this kind of interaction is essential. People are not going to really understand what, say, a negative number is simply by learning how to perform a set of procedures related to it. They need to get a feel for it, the sort of term of art is sense-making. And that involves experimentation and play.

Russ Roberts: You know, or worse--they are not going to learn what a negative number is by hearing a definition. And I think this idea of grappling with problem-solving--and by problem-solving, it isn't just getting the right answer. In math it often is, and I think that's extremely important. But in economics it might be just to understand the factors that might be the ones you want to consider. And there's no "right answer" as to why something is happening in the world right now. That, to me, is the economic way of thinking. You know, I read today that fair-trade coffee isn't helping the workers; it's helping the farmers. And I'm thinking, 'No kidding! That's what my problem set would have challenged the students to think about.' Right? If your skill level is only worth x in the marketplace, the fact that your boss has more money doesn't necessarily mean they are going to share it with you. They pay what the market can bear. So, those kind of insights--which, you know, as you point out, could be applied to a much wider array of things other than, say fair-trade coffee--that's the challenge of getting people to think that way. And, telling them that answer, which I just did, for you listening out there, I don't think is nearly as valuable as arguing about it for half an hour with somebody about whether that's a good idea or not.

Andy Matuschak: Absolutely. Maybe a way to summarize the whole approach is: Effortful engagement with ideas--that's going to be criticism, okay. If you are able to criticize effectively. Now, you may not understand the skill and question well enough to do that effectively. And that's fine. There's different kinds of effortful engagement. For instance, listening to you and another economist having a debate on that topic may be a really great way to slightfully effort-fully engage how you can't stop there. But it may be one great step on a scaffold to being able to enter that argument yourself and to debate with Russ.

24:39

Russ Roberts: And of course, a lot of you out there listen to EconTalk more than wants to each episode. Which is fascinating. Let's talk about that--not about EconTalk per se, but I want to go back to books. I read a book of Jewish theology called God, Man, and History. I found it utterly fascinating. After I put it down, I realized I couldn't tell you a single thing, 6 weeks later. And I went and asked--it's written by both[?], Eliezer Berkovits, in, I want to say in the early 1950s, 1953 or so. And I went around to people I knew who were interested in theology; and I said, 'Have you read Berkovits?' And they'd say, 'Oh, sure.' And I'd say, 'What can you tell me about him?' And the answer was: nothing. Every time. Now, part of this was due to the nature of theology. It's hard to remember theology. You don't deal with it every day. Part of it is Berkovits's style. Some of it, a lot of it is the human brain. Right? Which is prone to forgetting. But I decided--I was not happy about this. So I decided to re-read the book, with one of my children. I'd actually read it the first time with one of my children. But I decided to re-read it with a second child--following along, underlining in the margins, summarizing. And I've now read it, I think four times. It's one of the few books I've read more than once. And I have a pretty good idea of what he's talking about. And I could probably tell you right now. But, it was a lot of work. It was a huge amount of work. I mean, it's a challenging topic, obviously. It's not--actually, that's not really true. Every nonfiction, serious nonfiction book, is usually grappling with a complex set of ideas. So, what are your thoughts on that? On multiple readings? And, you talk about repetition? So, talk about what you are advocating there in online learning.

Andy Matuschak: Sure. So, one lovely thing is that I think that scientists actually kind of know a bunch of what's necessary to form memories reliably. They're just not necessarily what we'll naturally do, day to day. It's a little hard to pull off. Memories are formed more reliably, say, if you are exposed to things multiple times; if they anchored to powerful emotional experiences; if you have a variety of encodings for that memory--for instance, you know, both visual and auditory and perhaps relating to places. But, um, you know, the repetitions thing is a particularly easy and mechanistic approach. So, you know, we're probably all familiar with the notion that if you study a thing a few times, maybe it sticks somewhat better. There is sort of an efficient way to do that. If you study things a few times very, very far apart from each other, it's maybe unlikely to stick so effectively. It also perhaps unintuitively for many, college students especially, turns out not to be the case that--mmm, let me phrase that in the positive--if you study things back to back, say, within a couple of hours of each other, that's also not really going to work all that way. This is called the spacing effect. It turns out to be very helpful to space your study out. It's also possible to build systems which take advantage of that spacing effect. And this has been known for a long time, all the way back to shoeboxes[?] for language learning, a system by Leitner. But, modern systems are also being used. It's a little niche right now. Often by, for instance, med students trying to prepare for the MCAT [Medical College Admission Test] and still in particular by language learners. But, my colleague Michael Nielson and I have been working on trying to integrate these systems more deeply into kind of any informational nonfiction texts you could possibly imagine.

28:12

Russ Roberts: And let's talk about that a little bit more. I'm--I've probably revealed on here that, you know, I don't--the way I think about it is you read about 50 books a year if you read a lot. And, it's a book a week. And you might read for 50 years, as an adult. Maybe a little more. But, it's about 3000 books if you are lucky. And so, you should pick them wisely. And if you are not productive, you should put them down. The idea of reading a book twice--or three or four times--now, I did it with one of my children, which made it special; and that's a different--it adds something like--it's like the thrill of the lecture in person. But I do think it's an interesting question about how often we should return to material, and how we should--and how that process should take place.

Andy Matuschak: So, one of the things that's sort of shocking about the spacing effect and the way that it's instantiated, is that normally, when we do things--say, professionally as knowledge workers or as thinkers--we get diminishing returns. So, if we're not taking any notes at all, and we start taking notes: Okay, some good stuff's going to happen. And then if we take more notes, somewhat more diligent, we'll do a little better. But we're going to get diminishing returns. It's going to be some kind of a logarithmic curve; and then the same is probably true for, say, you know, having a good discussion about that thing with others: if you have twice as many discussions you are probably not going to have twice as many returns, indefinitely. But the weird thing of the spacing effect is it's the opposite of that. It's actually exponential. So, both prior theory and actually our present data with a current prototype, with almost 600,000 data points, is showing that, uh, marginal time actually leads to exponential returns. So, in order to explain that I have to get a little more concrete about what this project is.

Russ Roberts: Reminding listeners that 'exponential' doesn't just mean, like, 'enormous amount.' It means more than linear. So, go ahead.

Andy Matuschak: Sure. Yeah. Okay. So, we made this book on quantum computation. And we've chosen that topic in part because that's a topic that has a lot of notation, a lot of details. You need to know that, you know, okay, the bottom left-hand corner of this particular transformation matrix has this value, in order to really make sense of the topic. So, the first chapter takes about 4 hours to read. And, it contains 112 details that we track your memory of. And, we kind of follow up; and it's an interactive reading experience where as you are reading, you get these little prompts--like, 'Are you remembering this?' If you are not, we are going to kind of come back to you a little sooner. And then we come back the next day and we kind of jog your memory on the stuff you are having trouble with. And so on. You kind of start this practice. What's interesting about it is that the practice actually spaces out over time. So, initially the every day; then every 3 days; then every week, and every month, and so on.

Russ Roberts: When you say every day, you mean the prompts to check your knowledge, right? The feedback.

Andy Matuschak: That's right. And that isn't a cost which is equivalent to re-reading the book. In fact, it's very quick. It's 5 or 6 seconds per question, so there's about 112 of them. And so, it costs, ehh, 10-15 minutes for readers to do each of those repetitions. And actually, just last week I did the last first, mmm, large analysis on the efficacy of this thing. And by the time people get to the fifth repetition, which costs, mmmnn, 60, 70, maybe 80 minutes, kind of depends on where you are--people are able to leave the book alone for a month, have a month pass where they are not engaging with it. They come back. And, after a month engage with those 112 questions and get them right. And, that's really exciting, really interesting. But maybe what's even more exciting about it me is that the marginal costs from the 4th to the 5th repetition is actually the same as the marginal cost from the 3rd to the 4th [third to the fourth). As the marginal cost from the second to the third. Actually, it's slightly less. So, not only does it take slightly less time to get from a week to two weeks, and two weeks to a month; sorry, not only does it take the same amount of time, to get from a week to two weeks and two weeks to a month, it actually takes slightly less time for each of those steps.

Russ Roberts: Now, I looked at that material. And--because I was intrigued by the pedagogical aspects of it. I have no interest in quantum computing, except as a thoughtful member of the 21st century revolution of knowledge. And the questions at the beginning, at least, were things like: What's the space called that a cubic vector is in? And, it's a state space. I forgot that term. Okay; but I got the--the first thing I got right, the answer was 2; and then I got the second one. These are spit-back questions, which are not unimportant. When you say 112 things--that's jargon, I assume. It's--may be some connections of things. But it's not the deeper understanding. Am I right?

Andy Matuschak: You're right. Now, some of the questions were--some of them were involved. They would ask things like: 'Why is it difficult to store a qubit in this particular kind of matter?' for instance. And so you kind of have to come up with a justification. Now, after you practice that a few times, it'll become a spit-back thing.

Russ Roberts: Now, a qubit [also spelled qbit--Econlib Ed.] --I just want to show off and say a qubit is like a bit in classical computing; but this is in quantum computing. Correct?

Andy Matuschak: That's great. Yes.

Russ Roberts: I learned something already.

Andy Matuschak: Wonderful. So, I guess, first off, some of those 112 details are somewhat less spitback. But I think the really important thing is what those 112 details let you think next. My colleague has this metaphor that I really enjoy, so I'll share it here. Reading a challenging technical textbook is often a little bit like beginning by reading a book in English, and then--let's assume you don't know Spanish--Spanish words start creeping in. And by the time you finish the first chapter, like everything is in Spanish. And you turn to the second chapter, and you're like, 'Whoa. Like, I thought I'd picked up a book off the English section. What happened?' And so, you're going to struggle with that second chapter. If you have those 112 details, which[?] we have a second chapter, you are going to have a lot easier time learning about the quantum search algorithm.

34:18

Russ Roberts: Yeah. But that's--Let me take you to a harder example. I'm going to give you two. Just for fun. Because they are both former EconTalk guests. So, I interviewed Yuval Harari a while back about his first book, which is Sapiens. My wife is reading it now. And, let's pretend you ask me: What do you remember about Sapiens? Which my wife taps into a little bit because we are talking about it. And it turns out we are remembering three things. Three things! Not good. It's a long book. I remember that he thinks that agriculture was a mistake and I didn't agree with him; it didn't sit well with me. I remember that his view of money is based on trust and I thought, took that idea a little too far even though there is a sense on which money is based on trust. As an economist I found that a little simplistic. And third, he's anti-religion. Those are the--that's the three things I remember. That's weird! That's depressing. Now, let me take a different book: Fooled by Randomness, by Nassim Taleb. Now, when that book came out--I read it--didn't write when it came out. I read it somewhat after, The Black Swan, his second book came out. And there were two views of that book, which I've mentioned here before. One view was: 'There is nothing original in this book. This guy is a fraud. He pretends he has figured out all this stuff.' And I said, 'You know, I agree with that. ' I didn't learn anything that I didn't know in this book before. I knew that probability is difficult. I knew that risk is a hard thing to wrap your mind around. I understand something about most of the ideas he talked about in the book. So, in some sense, I learned nothing. On the other hand, I learned something incredibly deep. That book really grabbed me by the guts and jerked me around. It forced me to confront some things that I "knew" but didn't really internalize. And I would put that as another category of learning. These are things that, um, I'm just going to give you another example, another EconTalk guest, A.J. Jacobs, who writes a book called Thanks, A Thousand. It's about being grateful. Being grateful is a really good idea. I already knew that before I read the book. But the book made me a little more grateful, maybe. But, even if I remember that that's idea of the book, and even if after reading, I thought, 'Yeah. I should be more grateful,' to get me to be more grateful--that's a very high level. And so, those are the sort of 3--you know, those are all nonfiction books. They are all kind of trying to convey some understanding that the author has of the world around us. And I have really, really, different grasps of all of them.

Andy Matuschak: That's wonderful. And, to some extent, it illustrates the variety of purposes for which books are intended. If we look at, classical rhetoric, only a small part of that information piece. But, if, for instance, in the second book where you were yanked around--I'm not actually sure if it was maybe ethos or pathos. It kind of could have been either depending on your predilections around Taleb. But that's something that's not going to come just from a flashcard.

Russ Roberts: Exactly.

Andy Matuschak: And indeed, that's part of why, for quantum computing, we didn't just make a box of flashcards. We are experimenting with this thing we call the mnemonic medium, which is different from a bunch of flashcards; and it's also different from a book insofar as it's structured information embedded in a narrative. Because narrative can have ethos. Narrative can have pathos. And it can do those jobs that those wonderful books you described did. And, perhaps, also, do the job of leaving you with some detailed knowledge of what you read.

Russ Roberts: Yeah. I'm a big fan of narrative, obviously. You know, I've written three economics novels. And I do think that narrative--story-telling--is an important way that human beings' brains absorb information. Do you think about that, much?

Andy Matuschak: Absolutely. So, at a very simple level, I mentioned earlier that memory comes from emotional salience. But, jeez--I mean, like, let's leave that behind for a minute. Let's talk about meaning. Why are we reading any of this stuff, anyway? What's the point of learning all this stuff, anyway? For me, it's--it's about the moments that spark joy. The moments that fill the delight behind my eyes, where I feel connected to the eternal. I don't know--standing here in front of the Hoover Tower, I felt a moment about that; and I asked you about peace. That's like a heady question for a Monday afternoon. I felt connected to peace. And, I don't want to learn about quantum computing to learn about quantum computing. I want to learn about quantum computing because it could, you know, bring us to the cosmos or something.

38:58

Russ Roberts: But there is something beautiful about it all. So, I do think--I remember as a young man--I was probably 22--and I was reading a book by Robert Ardrey, who I really liked. I'd be amazed if anybody out there under the age of 50 has heard of him. And I'd be interested if anyone over the age of 50 has actually read one of his books. I read almost everything he wrote. He was a playwright, and he dabbled in anthropological creativity. His most famous book was African Genesis, which's theme was, as human beings we came from the savannah and we hurt each other. We were violent animals, not just this sort of peace-loving image that I think a lot of people have about human origins. But he wrote a bunch of books: African Genesis; another book called The Social Contract; and I can't remember the other one. But in The Social Contract, I remember reading that book and putting it down and thinking--it made my hair stand on end. So exhilarated to be in the presence of that experience, and to have captured something of what he was trying to convey. And it made me want to be a writer, actually, more than almost anything else. I thought, 'Boy, if I could make somebody feel this, that would be deeply gratifying.' I don't know if I ever have. But that was such a powerful moment of human connection with this man's brain. It wasn't like I walked out thinking, 'Oh: I know now why people interact the way they do.' It was, 'Wow. It's hard to figure stuff out, and that's a really interesting try.'

Andy Matuschak: What a powerful thing. And, ideally, we're all learning things for this reason. We're not learning things--ideally, I hope--in order to check off a box for the Common Core State Standards. We are learning things because we feel connected to the [? neumonis? gnomonis?].

Russ Roberts: Yeah. For sure.

40:52

Russ Roberts: So, let's talk about how to make things better. You know, one of the things I learned from a really interesting part of my life when I was interested in time management was: When you learn something, you should tell it over to someone. Okay. So, that's a version of your idea of--what's the name of it again?

Andy Matuschak: Transmissionism.

Russ Roberts: No, no, the stretched out--

Andy Matuschak: Oh, oh, oh, oh, right, right--the spaced repetition curve.

Russ Roberts: Right. It's one example of spaced repetition. And, when I talk about--I don't actually probably do this, but I talk about it with my kids, when I try to give them advice on how to give a presentation, I say, 'Tell them what you're going to tell them; then tell them; and then tell them what you told them.' Now, I don't really lecture that way, partly because not all my lectures are informational like that. And I tend to be more of a stream of consciousness guy--which I'm not--it could be laziness. I don't know. It probably is laziness. But also part of it is I'm not really trying to teach them something. I want them to be thinking in a particular kind of way. So, there's a big difference between--lectures have different purposes, obviously. But, let's talk about how, in designing a lecture or writing a book or having a podcast, how we might improve what people get from it other than just the passage of time.

Andy Matuschak: I love it. So, in the essay I talk about ways in which it's possible to intentionally design media so that their grains bend in a particular way. And when you conduct a class--

Russ Roberts: By grain, you mean like in wood.

Andy Matuschak: Yeah. Like the grain of wood.

Russ Roberts: Not like rice.

Andy Matuschak: Yeah. Thank you for clarifying. And, when you are conducting a class that is not so much informational but which you want to use to help students think in a particular way, you are shaping the grain of the discussion in that class by asking questions in a certain way. By using both your body language and your responses to tamp down certain lines of inquiry, steer them in a certain ways, to amplify others. You do that in your podcast all the time: You conduct the grain of this podcast in a certain way. And books can do this as well. And in creating new media we can do this. It's very interesting to think about constructing such environments in educational contexts. One example that I'll share that I'm just totally in love with: When I talk about seminar-oriented classes, people often say that, 'That's great for college,' or, 'That's really great for the Humanities; it's not going to work for Arithmetic.' And, there's this wonderful professor, Constance Kamii, who wrote a book called Young Children Reinvent Arithmetic, wherein she documents her experiments of having students do exactly that. She had a class of 5-year-olds in First Grade where she simply gave them a large library of games they could play, which she invented, such that the grain of the games kind of involved the students' learning; and in fact inventing for themselves arithmetic. And then they, you know, they took a standardized test at the end of the year; and they did fine. Etc., etc. It's a very inspiring story for me.

Russ Roberts: Yeah. And I think there is a challenging mix between engagement and content acquisition. I think about it all the time, in the more edgy stuff that I've done--example being the Keynes-Hayek rap videos, with John Papola. So, we tried to make them entertaining, but we really tried hard to make them educational. And, by having them rhyme--which of course Lin-Manuel Miranda also understood when he did Hamilton--it changes the ability of people to remember, and to have a shorthand form of communication. I probably used the word 'jargon' earlier in our conversation, and I probably put it in sort of sneer-quotes. But jargon is incredibly valuable, as a way to do shortcuts in communication. I think a lot of these innovations in terms of teaching education and, there's a huge challenge in figuring out--you know, games are great. Kids love games. If you can get them involved in games, they are going to get more engaged. The more time you spend on the gaming part, the less time they spend, maybe, learning. But maybe what they do learn is acquired more deeply. So, I think there's just a lot of tradeoffs there that we really haven't started to think about enough. We'll probably need to worry about it a little bit.

Andy Matuschak: Absolutely. It's really understudied. And one way I think about this--I hope you'll forgive me for using the term--is, basically I think about efficient frontiers, where, for a given amount of enjoyableness, how effective can we make the thing--the kind of learning interaction? And likewise, if we want a given degree of efficacy, how enjoyable can we make it? And, films, for instance, at a reasonably low level of efficacy are reasonably enjoyable. You can kind of sit back and watch an educational film; it doesn't take much effort, but you probably won't get a ton out of it. It's very enjoyable. Likewise--but books are a little bit different. It's hard to not engage all that effort fully with a book. But, kind of, at the maximum level, it goes somewhat higher. If you are willing to kind of have a not-so-enjoyable experience with a book--maybe it becomes enjoyable because you are fascinated--but you kind of really delve deep, then you might get something that's much, much, much more effective than a short YouTube video can convey.

46:40

Russ Roberts: I'd be interested to know if you can talk about this; if you don't want to, just say so. But, you know, Kahn Academy, I think, in the early days made a conscious decision to have what I would call low production values--not fancy, very straightforward, not particularly entertaining. The main entertainment opportunity was to increase the speed for some kids who were kind of already getting it, maybe, or who could acquire it a little more quickly. One thing we haven't talked much about is how people differ a lot in what they get out of something: The three things you are going to get out of Sapiens maybe aren't the same three things I'm going to get. And maybe aren't the three things I want to get out of it. Which is another challenge. But, I'm curious, Kahn Academy's impact on mathematics, I would say, is by far its biggest impact. The more ambitious stuff that you have worked on is uncertain as to how it's going to turn out, but I think in math, it's been extraordinary. Is it measurable in any sense? And, the extraordinary part is that feedback. I think that students don't just watch it and hear someone tell them how to do it. They are actually forced to interact with the material. What are your thoughts on that?

Andy Matuschak: Sure. That notion has been part of Kahn Academy from the start. It famously began with him as Uncle Sal, tutoring his nieces; and he was doing it over Skype or some equivalent, with a tablet. And, after doing the session with drawing on the screen, you know, give the kids problems online with like a little website he wrote. This idea of instant feedback--it goes way, way, way back. And, it's very valuable for this kind of declarative knowledge and rote procedural knowledge. And, it's been hugely helpful. There actually are great efficacy studies on Kahn Academy, demonstrating pretty interesting benefits. It's not a savior; it's not a silver bullet, at all. It's done some good. It's great. I think that model is very valuable. I think there's a lot of work to do to scale it to things like we discussed earlier, like, you know, 'What was the primary cause of World War I?' Getting there, rather than how to multiply two-digit numbers, is going to be an extra thing. One other challenge for that model that we engaged a lot with over the years and haven't yet cracked is building the conceptual understanding. Like, you might understand how to multiply those two-digit numbers. But, do you really know what's going on? I'll share this example that really struck fear into my heart from research a couple of years ago. We were doing some kind of explanation-oriented stuff, where we wanted kids to generate explanations for how stuff worked. And, not just, you know, use a procedure. And so, we chose a bunch of kids who had gotten every single question right about a particular topic. Like, as far as the system was concerned, they knew everything there was to know about this topic. And then we asked the simple follow-up question like, 'Can you explain, you know, why this formula gives you the area of this kite?' A kite is like a geometric shape. And, we got this, like, horrifying set of answers: 'IDK': 'I Don't Know.' 'Math is weird.' 'I dunno[?]' 'Blank, blank, mmm, blank.' Zero students--it was several dozen students who had answered every question correctly--could explain that.

Russ Roberts: So, the question there that I think is so interesting, and I talk about this with my wife all the time, which is: Getting all the answers right is great, as long as you don't have to apply it to something new. And then, you might be--it's possible you have the underlying understanding that lets you apply it to the new stuff. But if you don't, you're cooked. And you won't be a good engineer. You're going to struggle. You know, some of it--I think part of the advantages of learning math, which is just such an interesting question, just because so little of it actually applies to everyday life, and yet, in terms of the amount of hours that are spent just, say, on geometry, that it causes some people to say this is a waste of time. I wonder about building grooves[?] in the brain: I think there's probably neural connections, probably a good thing. But I think there's also a feeling of triumph and achievement that comes from mastering math that's different, that's special, that's different than "mastering the cause of World War I." It's just a very, to me, a very deep and interesting area to think about. But the idea of how much underlying stuff you should understand. My wife teaches calculus. And she teaches it very much as a 'let's understand it' class. And her students get to college; they get As in their college calculus class; and they find it unbearably boring because it's all cookbook. It's all punch out the right answer, and not understanding it very well--just not important.

Andy Matuschak: I think what you are accessing now is maybe the greatest tragedy of all to me, which is that: Why learn about math? Let's put kind of political disagreements about forced education aside. Math is sooo beautiful. Maybe I'm not going to force you to learn it; but, oh, my gosh--understanding something really deeply--I'm not talking about arithmetic, but, for instance, why does e show up in so many places? It's so beautiful. And that's because you are connecting to something fundamental and true. These are Platonic objects. Okay--that is a controversial idea. And none of that, none of that is what is accessed in traditional K-12 education. You are learning it because you have to, or because I told you so, or perhaps because it will help you get a job; or perhaps, in a very enlightened environment, because it will help you think analytically and critically and abstractly, and those are skills which are useful in a variety of domains.

Russ Roberts: One of my favorite moments as an undergrad was taking a graduate class in bio-statistics, which was hopelessly difficult. It was a really interesting aside, which was, after the midterm, the teacher called us all together; and he said, 'You all failed.' There were about 40 people in the class. And he was going to change his whole instructional strategy. He was deeply disturbed by this result. Which is interesting, because he could have said, 'Well, just bad students this year.' But he decided that he had not taught the material well, and he changed his classroom interactions. He actually called on us instead of doing pure transmissionism. I don't think he knew any of our names, but we thought he did. He would call out someone's name, and when someone looked up, he would look at them; and we thought, 'Wow, he memorized all our names. That is so cool,' after the midterm. He may have memorized some of the names, but he didn't learn our faces for at least a while. But, it was a very cool emotional technique. But, in that class, we learned to prove the Central Limit Theorem in, I think, three different ways. And, you could argue any one of them was a waste of time. Just: Here's the Central Limit Theorem; just believe it. It was one of the greatest intellectual experiences of my life. I didn't become a statistician. I've forgotten a lot of my statistics. We used, I think, characteristic functions that are called in that proof; I don't remember what they are; they have a capital Greek letter associated with them. I can draw it; I don't remember the name of it. Anyway, it was a beautiful thing. That's all I think I have to say.

Andy Matuschak: That's wonderful. I'm so glad you had that experience.

Russ Roberts: Yeah; no; it's cool. The analogy that you're going to be hung in a fortnight concentrates the mind wonderfully, as, I think Samuel Johnson said. And I thought if I didn't get an A in this class I wasn't going to get into grad school. So, I was pretty motivated. That may have had something to do with it.

54:43

Russ Roberts: Anyway, let's go back to podcasts. And let's think out loud for a few minutes in the time we have left what make EconTalk more valuable to listeners. Now, I know some of you out there get together every week with your friends, or your family, and talk about it. Not very many of you. But that there's more than 2 is deeply wonderful to me. But, would it be useful to have EconTalk clubs? And maybe it shouldn't be every week for that episode. Maybe it should be 3 weeks before. Maybe they'd be staggered, based on what you said earlier. But, what might make--and we don't have to just talk about EconTalk: obviously there are a lot of podcasts out there, and a lot of why people listen to them is just for entertainment--but, a lot of it is to learn stuff. What could we do that would be more useful? Just for fun today I asked--the Adam Cifu episode came out today on Medical Conservatism, and I asked people what's one thing you learned from this episode, drawing on A. J. Jacobs' idea that you should have a 'One Thing' document where you list one thing that you might want to remember. And, you know, this is a great idea. You should do this for every episode. But, of course, your one thing is not going to be the same as other people's. And a list of one things maybe is not so useful. But I don't know. So, what are your thoughts on that?

Andy Matuschak: I'll give it a shot; maybe we can brainstorm together. Here's one variant of 'one thing' you might find interesting. There's kind of a classic textbook on instructional design called Understanding by Design. And it introduces this concept that I found very helpful called Essential Questions. And, it argues that we should try to structure our curricula and structure our courses to really focus on accessing essential questions. What those are, are the questions which are central to the discipline. They are the questions which are alive. They are the questions which are not resolved. They are the questions which you, the novice, and I, the expert, we are all engaging in.

Russ Roberts: Love that.

Andy Matuschak: We can discuss it at all levels; we can apply it, perhaps, outside of the discipline. So, for instance, essential questions around compound interest. For instance, 'Why does it appear so many places?'--let's just stop there--can apply in so many domains. So, one 'One Thing' that might be fun for your listeners is if you or your guests introduce an essential question from their field or from their work, which then the listeners can discuss amongst themselves or think of individually.

Russ Roberts: So, that's an interesting--I love that idea. So, let me try to rephrase it, to see if I understand it, check for understanding here. I love this idea that I ask a guest what in their area of expertise is up for grabs. Right? And, of course, the idea that listeners might know as much as they do is deeply troubling to most experts. The other challenge, though, and this is where I'm going to ask you to try to come up with this, is: What's the forum? So, we have a feature on the website called Extras, where we have--we raise questions related to the episode. Most people don't go to them. Almost by definition. Most people are listening in their car, commuting. They don't go on the website actually at all. I think we've done a--we've not done a very good job in the Internet era of figuring out physical, web-based spaces that are physically designed--user-interfaced spaces that are effective. You know, a comment board is just not great. It's a long stream of monologue. Separate monologues. Occasionally, there's back and forth; we've added that in, in our recent redesign. And it's a little bit better. I think it's a plus. But, a place to chat--maybe it's physical. Literally, in the sense that maybe it's a club. Or maybe it's a place in time online that you know you can come and engage with other listeners at that point about a question that was raised toward the end of an episode.

Andy Matuschak: That's wonderful. I'll share a couple of thoughts. First, one thing we learned from designing the sort of open-ended activity tool at Kahn Academy was that, not all discussions are equally productive or constructive. And, you don't want someone to have to like, wade through those to figure it out. And so, places like Reddit and HackerNews have kind of figured out mechanism which allow the most fruitful or generative discussions to rise to the top. Sometimes that can mean the most controversial, or clickbait-y; and there are ways of dealing with that. But, for instance, if there were a thread: What is the relationship between one's emotional connection to knowledge and one's detailed understanding of that knowledge? That's a question which a learning scientist can engage with extremely deeply. But it's also a question which everyone can engage with. Everyone has experienced, everyone has interesting things to say about that. So, maybe if I go to a thread that includes that question, at the top might be some of the most interesting responses, and I might feel inspired before I come up with my own answer to that question, to engage with one of those responses--because it was very striking to me. And, indeed, the momentum that I built in responding to that might then inspire me to try to answer the question myself.

Russ Roberts: Yeah. I think that's a--let me ask you about the forum for that. Is there a--do you know of any places to interface with material that is, um, better than just--so, there's the comment that just goes down and down and down. Then there's the up-vote, so you get maybe the more interesting ones. But those are the same technology: it's slightly improved. Is there a chance to do something else? Something--maybe it's moderated? I'm not sure.

Andy Matuschak: Yeah. Sure. Just a couple other models that come to mind. So, one of them is moderated-ish. And, it's used by a lot of bloggers. So, many bloggers who write series of essays on their blog will write something; it will include some provocations. The comment section will have a lot of discussions going on. The person who wrote the thing might engage and facilitate those discussions slightly. But the more important thing that they'll do is kind of pull some of those comments, the important ones up into the body itself. Or, to address them in the next essay. So, the equivalent here might be , either to have volunteers for the community be highlighting particularly interesting questions or discussions, or for you on a subsequent episode to kind of react to the ideas from the community that you found most interesting.

1:01:53

Russ Roberts: Yeah; I love that. It's not--and, I've come close to it. I mean, a thing that often will come to me, happen to me, is that, I'll finish an episode and a few days will go by and I'll think, 'Why didn't I ask _blank_.' Or, 'You know, when I said that, that didn't really capture the way I think about it. It's such an incomplete answer, I need to expand on it.' Obviously, it adds an immense amount of work, potentially. It's hard to, you know, get each episode out just already. But, to add to that might be too demanding. But, the idea of using postmortems as a way to enhance--certainly my understanding--is a good thing. But, I love this idea of hearing from--I'll give you another example, and maybe this will be a way we can close. When I used to teach in the classroom, there was a point in my life where--I was teaching in a Business School. And, I got--somebody from some, some business faculty member told me that feedback was a good thing. And, I'd never asked for feedback. 'I know I'm a great teacher. Come on! I don't need any feedback!' Which is, you know, arrogant. I was young. But, based on that encouragement, I would give a one-page feedback form every lecture. Every lecture. And, in that one page was the following: 'How was this lecture on a scale of 1 to 5?'--I think it probably asked, 'How much did you get out of this lecture on a scale of 1 to 5?' 'What was the most interesting thing or most valuable thing you learned?' And, 'What was the most confusing?' And what was fascinating about that is that I didn't anticipate what those were. Often. Both of them. What I thought was the most important or interesting or valuable thing was not what they thought. And what I thought was the most confusing thing was not what they thought. And, that was a--that was a wake-up call. That's--so, I think it would be nice to do something like that for EconTalk. I'm not sure, again, how to visually or physically create that. But, what was the most interesting thing that people got out of our conversation? What left them puzzled, confused, or offended? Which is another thing I would sometimes ask. And then using--not just--I mean, just categorizing, cataloguing that would be interesting. But using it as a way for followup and enhancing learning I think would be fantastic.

Andy Matuschak: That way there are stakes. You'll know that, 'Perhaps Andy or Russ is going to reply to this. It will mean something.' And, I think you are actually also tapping into something that's just really central to this place where we are right now, here in Silicon Valley: being surprised by our users, here, our listeners. Both teach us things we don't expect. And so, we have to go there with an open mind. It will surprise us.

Russ Roberts: You know, one of the things that is strange about being the host of this program is that, you know, I started it as a way to teach economics and to teach what I thought were the virtues of liberty. I don't do either of those in a formal--I don't lecture on that, on this program any more. I probably did in the early days: If you go back to 2006 you'll hear my occasional soap box on both of those things. And, you know, it turns out, what many people get from this is a style of argumentation. Which is unbelievably satisfying to me. And any of you out there who have benefited from that, I'm flattered. And it moves me deeply. Because, partly because I do the best I can. I don't know if I'm particularly good at it. But just that it wasn't expected. It was a surprise. Not what this was about in the beginning. And I think: Boy, that's fun.

Andy Matuschak: Earnest engagement with ideas without necessarily having to agree is a beautiful thing.