John Cochrane on Education and MOOCs
Mar 31 2014

John Cochrane of the University of Chicago talks to EconTalk host Russ Roberts about the experience of teaching a massive open online course (MOOC)--a class delivered over the internet available to anyone around the world. Cochrane contrasts the mechanics of preparing the class, his perception of the advantages and disadvantages of a MOOC relative to a standard in-person classroom, and the potential for MOOCs to disrupt traditional education.

Daphne Koller on Education, Coursera, and MOOCs
Daphne Koller of Coursera talks with EconTalk host Russ Roberts about online educational website Coursera and the future of education both online and via bricks-and-mortar. Koller, co-founder of Coursera with Andrew Ng, explains how Coursera partners with universities, how they...
Eric Hanushek on the Education, Skills, and the Millennium Development Goals
How important are basic skills for economic success and growth? Eric Hanushek of Stanford University's Hoover Institution talks with EconTalk host Russ Roberts about the importance of basic education in math and literacy and their relationship to economic growth. Hanushek...
Explore audio transcript, further reading that will help you delve deeper into this week’s episode, and vigorous conversations in the form of our comments section below.


Ed Hanson
Mar 31 2014 at 8:09am


As an interested friend of a first semester calculus college student who is really struggling, I would really like to know if you now remember the name of that Professor and and his MOOC. It would really help.

I was wondering what might be thought of separating the education from the certification. Perhaps certification could come from from already branded facilities. In other words, you bring your knowledge from what ever source to our testing and that facility determines the extent of your knowledge.


Russ Roberts
Mar 31 2014 at 10:04am

Ed Hanson,

The calculus teacher I liked is Robert Ghrist. His Coursera course is here.

Barry Crowder
Mar 31 2014 at 12:50pm

As someone who has taken several courses via Coursera, it was interesting to listen to this podcast. I enjoyed professor Cochrane’s perspective as to the considerations that went into his course, and also the technical limitations as well as the wishes for future features. And while I haven’t participated in a course that was reverse-taught, I can really see the benefit of watching the lectures on one’s own time, and having the interactive discussions with the professor during classes.

One technique that I’ve encountered in many of the courses I’ve taken is peer review of student submitted projects. In addition to simply taking a quiz, the students also submitted work that was reviewed and graded by other students who were also taking the class. Of course, this can be problematic due to inconsistency of the assessments, but where there is a clear rubric for grading, I’ve found it to be mostly positive.

The type of submissions can vary. I’ve submitted software in the form of source code for programming classes. I’ve also submitted recordings of myself playing a musical instrument for a musical performance course. In another course, we submitted a 5 minute presentation in which we were “teaching” some concept from the course.

I do think that there are some types of material that are better suited for MOOCs than others, mostly due to technical limitations. Material that has more objective criteria, with more straightforward guidelines will work much better than subjects that are more open-ended.

I suspect that MOOCs will increase in relevance and widespread use as people adapt to and enhance the technology that enables the material to be presented to such a wide audience.

Mar 31 2014 at 12:57pm

Several times in the episode, Cochrane and Roberts assume or identify directly what education’s purpose is, but that education doesn’t really do it very well.


Russ: Oh, John, come on. You are going to tell me that just because they have a B.A. from University X that has a nice reputation, I’m confident they know what they are doing? Guest: No. But I think that’s a lot of the function of what universities do. Russ: Yeah. Guest: We don’t necessarily do a very good job of it. Why do you hire somebody with a U. of Chicago or Harvard degree? Well, that’s a brand. And it’s a brand that is pretty careful about its reputation. It wants you to think that our students are good and you can hire them with confidence and that we have certified that they know what they are doing; and you point out, correctly, that maybe we’re not as good at doing that as we say we are; and that we had better shape up.

If education is about ensuring that people know what they’re doing, and educators aren’t largely failing in providing that, why is the premium to education so high?

Education teaches me how to write a 6 page argumentative essay and how to factor a quadratic equation, how does that ensure that I know what I’m doing in my future job in middle management?

Enter Bryan Caplan: Education isn’t about learning how to do anything, it’s about certifying that I’m the kind of person who will fit employer’s needs. I’m very much looking forward to the future Econtalk with Bryan Caplan, and I hope that they cover the implications the Singaling Model of Education has on online education. That is, if education is about signaling conformity, then online education will always be great for learning for it’s own sake, and will pay a modest premium to those who can’t afford to go to “regular” university. But it signals non-conformity, and that will always make it less profitable for employers to hire them relative to those who received a traditional education.

Mar 31 2014 at 1:09pm

Another thing,

What I find most interesting about the conversation about the economics of education, is that it’s possible that economic educators actually have to drop the efficient market hypothesis. You are the efficient market. You have the comparative advantage in leading the frontier. You are the entrepreneurs. This is your market. There is a $100 bill on the street that only you walk on. There is not an, “if it were profitable, people would already be doing it,” assumption.

I find that this leads to a very different kind of analysis than economists are used to.

Ed Hanson
Mar 31 2014 at 1:54pm

Thank-you very much, I have been checking it out and it will help if my friend will find the time. Had to go down a level to match his course.


Mar 31 2014 at 3:21pm

Thanks for Econtalk Ross, I treasure these weekly episodes, I have been listening since 2008. So much so that I would actually HATE IT if EVERYONE started listening to these podcasts too!!!

What makes econtalk special is the feeling that the listener is eavesdropping on a phone conversation between two bright US professors…if too many people are tuning in, well, the conversations are still great, but the listener no longer feels as privileged as before…

Thanks again for the brilliant podcasts, but try to keep it cosy if you can… at least for your emotionally immature listeners, like myself, who stop liking something once it becomes too popular!!!

Russ Roberts
Mar 31 2014 at 3:22pm


Next Monday’s episode is Bryan Caplan on Education, Signaling, and Human Capital. Enjoy!

Mar 31 2014 at 3:31pm

I’m surprised that the following, which I consider to be a realistic threat MOOCs pose to higher education, wasn’t brought up:

What if a post-college standardized test emerged that employers found valuable? Or maybe a series of standardized mastery exams for various subject matters. This is one way to get around grade inflation and other ways in which it is hard to compare job candidates from different schools. In that case, if you’re looking for people with a mastery of computer science, who cares if the student did it at a four year college or through a series of MOOCs? If the MOOC student competes with the college student on the standardized mastery test and wins, the employer gains valuable information about their relative abilities.

I still don’t think that this would replace the top tier schools, where much of the value is in the social and professional connections that you make. But if a student were able to save tens of thousands of dollars from going to a lower tier school, this could be an attractive alternative.

Steve Sedio
Mar 31 2014 at 9:40pm

Good podcast, but I think you are missing a critical “market”. The billions of us that want / need to learn about a subject (or subset of a subject) that won’t go (or go back to) college.

Ask around, very little of what we learn in college gets used in real life. We spend our life collecting a lot more higher education level bits of information. We buy books, take expensive seminars, we wade through the papers in professional journals, and often we re-invent the wheel.

As slow as that process is, it is faster than the video taped classes recorded at MIT or Berkley (I watch for hours and hours to find that one piece of the puzzle).

The competition is You Tube, Wikipedia, forums, and podcasts. That provides me good quality information in minutes. (I haven’t attended any of Russ’s classes, but I bet I wouldn’t have as many ah-ha moments as I get from his podcasts).

I would like to see a higher education version of Kahn academy, easily searchable in approximately 15 minute snippets. If I get an idea, give me the knowledge to make it happen, or to tell me what I missed, so I can try another approach.

There will be cases where a little knowledge is a dangerous thing, but as the man says, fail fast and fail often – that is the best teacher.

Especially now, no one can learn it all, but given the ability to collect the right snippets combined in ways only people can – more break throughs will happen.

John Sallay
Mar 31 2014 at 10:44pm

I’ve got several thoughts on this and previous related podcasts. You guys talked a little about completion rates and how much time people were willing to spend on these courses. It seemed to me that the implication was that a good course would get students to spend more time in it. I come from a different perspective.

I’ve taken several online courses from udacity over the past year or so. My favorite aspect is that I can go at my own pace. I’ve got a busy job and two toddlers. Some weeks I’m lucky if I can get 15 minutes in on a course. It takes me several months longer to finish than it would the traditional student, but I’m still learning. I wouldn’t be able to put in 15 hours a week for the course that Dr. Cochrane taught, but I could probably complete the course over a year or so. To me, that’s a great success. I don’t have time in my life to go to a college campus and take a course, but I can open a browser when I have a spare 30 minutes.

On another note, the one-on-one teaching ideal was brought up many times. I wonder why this is thought of as the ideal. Why would one-on-one teaching be better than having a group? I agree that personalized attention and pacing is important, but I think that there is a lot that can be leveraged in a group setting. I sought out study groups when I was in college because I think that I learned better working in a small group. Much like in markets, other people frequently came up with better ideas than I would have been able to do myself. Also, trying to “sell” people on my way of solving a problem helped all of us to have a deeper understanding of the underlying material.

I don’t think that traditional courses or MOOCs do a very good job of bringing people together, but I would hold this up as more of an ideal. I think that this is what Dr. Cochrane was getting at with his mixed course. He was able to get a level of group engagement with his students after they had watched the videos that he didn’t get without it.

Apr 1 2014 at 7:07am

I really enjoyed this podcast. Russ, you have a very special tone that you make possible in econtalk.

-MOOCs are the new textbooks not the new Universities-

I love the MOOC=textbook analogy. I really hope the startups working on these adopt it. One important point is that textbooks do not displace Universities.

Plato’s writing disrupted Socrates’ style of teaching by lecturing, but did not in the displace it. A Socratic question and answer session while strolling in a Greek garden is better if the students spend an hour reading Plato’s Ring of Gyges in advance? That’s how Plato taught.

Plato’s books didn’t replace Socratic walks for the lucky few that could participate, it added to them. For the many more who couldn’t attend Socrates’ walks (including me, 100 generations later!) Plato allowed get the material a different way.

Stretching my Socrates/Plato Analogy even further, I read and discussed Plato when I was in college. Plato enabled that with his books. Similarly, A teacher at another College could take Professor Cochrane’s finely crafted MOOC and teach a class using it,not competing with it. I learned primarily from textbooks. FOr me, college provided a commitment device, social environment, benchamarks, etc. I did attend tutorials, but I only attended about 25% of lectures. I would have done fine without them altogether. My main mode of learning was always textbooks. A good textbook made or broke a course for me.

-Credentials & Financial Incentives-

On more “social” questions like credentials and financial models for lecturers, it might be preferable for those to come later. Once credentials and finical motivations come in, they tend to dominate. The WWWeb is a good case study. Financial motivations certainly added an incredible amount but the “rogue” period before the dotcom period made the commercial web so dominant was important in its cultural development.

There’s enough of a demand from people who don’t need credentials for and enough supply from people who don’t need direct compensation right now to enable the development of online learning. I think it would be preferable if credentials were fitted to an existing method which is gradually emerging rather than online learning developing around credentials.

To beat my analogy horse one last time, Socrates didn’t have people asking if this would be on the exam.

I think credentials and business models will inevitably arrive if/when online learning gets bigger, but some things are better when incentives are non financial. I’d rather not say, “pure” but I think there is a noteworthy effect financial motivations have on things.

Apr 1 2014 at 7:32am

I loved this episode. Thanks for taking the time at the top to dwell on the slightly more technical aspects of putting up a MOOC. Econtalk at its best: some hands-on discussion, and then some contemplation of the big picture. Many interesting ideas to chew on.

As for the educational enterprise that is Econtalk: I have been a devoted weekly listener since 2009. Econtalk has taught me so much over the years, and absolutely blown my mind quite a few times. The podcast medium seems perfectly suited to Russ’s art as a conversationalist, although his honesty, his curiosity, his civility, and his willingness to trade points of view would doubtless serve his educational mission in other formats as well.

Here’s one interactive evolution of the current format that could be fun: Russ, over the years you have given out a few of those great little econ riddles: the price of coffee, or in this episode the quality of light bulbs. Perhaps you and the guest could feature one such question every week during the show and then there could be a dedicated place for people to discuss just the econ riddle of the week, with maybe occasional audio or video responses to some of the top comments, answers, or questions so far.

Apr 1 2014 at 9:20am

Nice interview, but I winced when Cochrane suggested that Econtalk should be more like TED talks. Christian Lander was pretty much on the money when he said that TED talks are for people who want to feel smart without actually having to think very much.

Some mediums are just better suited than others for explaining complex ideas. The written word is at the top of the list, but long-form audio interviews can get pretty close. The video medium almost inevitably reduces complicated reasoning to soundbites, and shifts the viewer’s focus from the content of the argument to the appearance of the speaker. There’s a reason that the Supreme Court doesn’t allow cameras at oral arguments.

Lots of programs try to boil down complicated ideas and present them to a mass audience in a simplified form, but I don’t think that’s what you’re trying to accomplish here. I wouldn’t mess with success.

Apr 1 2014 at 10:19am

I absolutely agree with Steve.

I wish TED was more like econtalk, not the other way around.

Entertainment value is a good concept. But if you measure and optimize for “more listeners” and aren’t selective about how you achieve that well.. that’s how you can start with beautiful National Geographic documentaries and end with alien bigfoot hunters. We already have plenty of the latter and I have a hard enough time resisting it. If Econtalk devolves into TED while I’m not looking, It’ll get me for sure!

Econtalk has depth.

Apr 1 2014 at 3:58pm

Very interesting podcast.

Caroline Hoxby has a recent NBER working paper on MOOCs that listeners might find of interest:

Apr 2 2014 at 9:26am

Some thoughts on a great podcast

David Longstreet
Apr 2 2014 at 10:44pm

Big university is a Goliath and Goliath never goes without a fight. It will be interesting to see how these “second rate universities” respond when they start feeling the financial impact of MOOC’s and other online education.

John Cochrane suggests that MOOC’s may replace large courses at second rate universities. These second rate universities make a lot of money on these courses. Each 500 student classroom can net universities well over $500,000.

Robert Kennedy
Apr 3 2014 at 10:10am

I’m surprised that Russ did not broach an analogy that he has mentioned in other conversations. In other episodes, Russ has commented on the evolution of audio recording. Before music recording, only the very wealthy could afford to hire the best musicians to play for them. everyone else could only listen to less talented musicians that happened to live close to them. with audio recording, the music of the best musicians become available to everyone at an affordable price. I’m thinking that the same thing is happening here. MOOCs are providing access to the best teachers to a much wider audience at an affordable price. Recorded music did not obsolete live performances for those with means and the motivation. MOOCs will not obsolete live teaching for the same reasons.

S.E. Blackmer
Apr 3 2014 at 11:40am

Why does the quality of light bulbs increase while the price decreases?

Because light bulb makers are price takers not price setters, the markets set the price through competition among light bulb sellers…

Is this the answer? Am I the only one losing sleep over the question?

Apr 4 2014 at 5:32am

S.E. Blackmer, It’s kind of a trick question 🙂

I think Russ mentioned that there is a magic word, which he is looking for. I’m fairly confident that magic word is “technology” or maybe efficiency. Same reason that smartphones keep getting better and cheaper.

*Hmm. Seems like I’ve been listening to this podcast for long enough to get a picture of Russ’ aesthetic. He wouldn’t ask a riddle if the answer required supply curves or special concepts like “price setter” to answer.

Kevin Postlewaite
Apr 4 2014 at 6:28pm

I really enjoyed this podcast about MOOCs. I felt that, though, that there were some potential aspects about MOOCs that were overlooked, or at least not discussed, that can raise their quality/value.

Right now, MOOCs are typically monolithic, bundled as an entire course. At some point, I can imagine that there start to be an ecology of MOOCs where they will be broken up into much smaller pieces. At that point, people will start to pull the best pieces from different producers together. For some students for some concepts, having different presentations of the same material might make the difference between understanding and not understanding.

There was much discussion about automatic assessment. I’ve taken a number of MOOCs, programming and statistics related, where the assessment was at least as effective (I felt) as the classes I took in college. Udacity, e.g., lets you run your program in their UI and get instant feedback. Obviously these are among the easiest types of classes to assess.

People talk about using MOOCs for flipped classes but another possibility is for students to study different material or just go at their own pace. I think it would be a big improvement if we differentiated math accomplishment for high school students not by letter grade but rather how far/how deep they achieved, but insisted on some level of mastery to progress.

Also, Russ mentioned that he was interested in different ways to connect to his listeners. I think it would be interesting if there were a Reddit page where listeners posed/upvoted questions and the best were answered in an ancillary podcast.

Daniel Barkalow
Apr 4 2014 at 7:45pm

I think a educational course has a wide variety of purposes, only one of which is information transfer, and different courses produce value primarily via different purposes. There are some courses, such as Cochrane’s MOOC, or introductory calculus, where the purpose is to learn some information and the skill to apply it, as described in the course catalogue, and that’s about it. The literature class that Russ mentioned is really not the same way: the point of the class was not really to read a few books, or to learn that Faulkner is actually better than Conrad, or to find out what the professor thinks these authors meant to say. It was probably to learn to discuss literature (both finding out what other people thing, and conveying what you saw), and more broadly, to communicate effectively in small groups. I think that class may have contributed a lot to Russ’s ability to make Econtalk worthwhile to so many people, but the syllabus and exams didn’t include anything about podcast hosting. I think MOOCs will probably become the best thing available for learning knowledge and how to apply it, while doing relatively little for the intangibles.

Robert Ferrell
Apr 4 2014 at 11:39pm

My wife and I home school my 4th grade son. It works better than public school for him. One of the great things is we can introduce him to material that is way over his head. And one of the great aspects of living right now is we have access to oodles and oodles of material, via the internet. We’ve been using some chemistry videos from Tyler DeWitt,, (e.g. Hydrogen Bonding and Common Mistakes , definitely NOT 4th grade Common Core material).

My son and I watch the videos together. The way DeWitt has produced them, each of us feels like we are getting one-on-one instruction. It’s very good, not dissimilar to Khan, but spiffier production. DeWitt shows himself during the whole video, so it really feels like I’m sitting across my desk from him. Both my 9yo and I find it very engaging.

With Econ Talk I feel like I’m listening in on a conversation a couple of informed folks are having, maybe over dinner or over a glass of Scotch. But I’m driving, not sitting with them. That’s another great format. Russ, you ask questions I’d want to ask if I were in the room.

Both of these contrast with the videos of Walter Lewin teaching physics. He’s a brilliant, engaging, physics professor. Maybe the best ever. But watching a video of his lectures is not nearly as engaging as being in the lecture hall. As Cochrane said, online teaching is not just a video of a great teacher. It is a whole new beast.

[broken html fixed–Econlib Ed.]

Russ Roberts
Apr 6 2014 at 2:54am


Here’s a hint. If you could give your answer to the question without taking the course, it’s usually not what I’m looking for. Technology is necessary to improve the durability of lightbulbs but that is the answer someone could give without studying economics. I’m looking for what drives the technology.

For those losing sleep, the answer is here.

Ron Crossland
Apr 10 2014 at 9:45am

Thought provoking episode.

I liked the MOOCs = replace textbooks idea, but would tweak it a bit. MOOCs + books + podcasts + search + live participation = enhanced education. Likely other variables to add to the equation.

Technology, like MOOCs, is driving new engagement methods, search is refining the wait time to find what we want, and serial reading/hearing allows for portability and a different absorption rate. I would argue we will lose the need for keyboards as a input device before we will lose the advantages of reading.

A vector for Econtalk that can be done well on surprisingly low budget is the occasional video of the interviews. The Christy/Emanuel debate comes to mind as an example. Good material for later MOOC type engagements as well.

Jeff Potter
Apr 17 2014 at 9:54am

The Cochrane and Caplan transcripts are amazing. (I’d rather read than listen, but I’ve heard these make for educational listening commutes.)

Will college soon have a home-schooling done on it? As with instutitional school such a development won’t take over but might offer competition and a force for correction. Institutions will adapt to MOOCs — but in a good way?

I think it was Caplan who said that gov’t has skewed the data outrageously with funding. A commenter then mentioned the gov’t also skews with degree requirements for work (any degree in many cases).

I would think that banks/loans also skew by likely only funding students who will forward the money to (I mean “learn at”) accredited programs.

The institutions themselves skew the process because of their hold on accreditation.

If tuition were able to go directly to educators — the teacher at the front of the class (!) — and if testing and accomplishment were relevant, then education might be liberated and improved. Radical!

(Yes, also use tuition to pay for equipment and facilities. But other middlemen? I do see that “equipment and facilities” are a wonderful loophole for vast amounts of machinery, buildings, and administration to creep into.)

MOOCs seem like great economizers that let a teacher really teach. (But as Cochrane said: they leave the teacher with the hard stuff: the actual teaching, diagnosing, paying attention to pupils.)

Comments are closed.


About this week's guest:

About ideas and people mentioned in this podcast episode:Articles:

      • Education, by Linda Gorman. Concise Encyclopedia of Economics.
      • Arthur Pigou. Biography. Concise Encyclopedia of Economics.
      • Alfred Marshall. Biography. Concise Encyclopedia of Economics.

Web Pages and Resources:

Podcast Episodes, Videos, and Blog Entries:



Podcast Episode Highlights
0:33Intro. [Recording date: March 10, 2014.] Russ: John, you recently taught your first MOOC--and that stands for Massive Open Online Course. And we're going to talk about that experience--what you learned and what you think of the potential for MOOCs to disrupt traditional education. So, let's start with the course itself. What was the subject matter? And give us a rough idea of what it covered. Guest: So, I decided for my first MOOC to do a Ph.D. level asset pricing class. So, it's the first Ph.D. course in asset pricing. And it covers the standard theory of how we think about stocks, bonds, and how they relate to the macroeconomy. I did that course first because it's very straightforward material--not a lot of interpretation; it's kind of math that you learn and learn to interpret. And I thought that would port[?] well to MOOCs in my first effort, and then I could go on and do more discussion and interpretive stuff once I learned the technology. Russ: And what was the--do you have the formal name of the class? Does it have a formal name? Guest: Asset Pricing. That's the name. Yes. Go to Coursera and look up asset pricing and you'll be able to look at it now. I leave it open for people who are interested. Russ: So, anybody can go there now and take it for free? Or do you have to pay something? Guest: Uh, yes. You can go now and go through it. MOOCs are organized into sessions, where actually taking it you have to do it while it's offered, along with the other community of people who are taking it. And then a professor like me can leave it up there so people can go through it. And most of it I think is still available now. Russ: You should point out, in case those listening don't know what a Ph.D. level finance class on asset pricing is about: it's a very mathematical class. It's very challenging. It doesn't have, as you said, a lot of casual intuition about, say, how to be an investor. Correct? Guest: That's right. This is not, How to make money in the stock market. This is--I think it is useful for lots of people. This is the theory of how we think markets come to equilibrium and all sorts of apparently-interesting ways of making money really are just ways of taking on risk. But the language is mathematical. It's not really so much doing math as staring at math and getting the intuition about it. And it's useful if you want to learn how to read the academic literature on asset pricing. Yeah. Russ: So, how many people took the class? Guest: Like all of these, a lot of people were interested. I got 37,000 who signed up. But in the end 4000 were watching the videos. And about 300 took it all the way through and took the exam. This class took about 15 hours a week outside of class. There was--I ported all my Ph.D. level problem sets to Coursera. So that was a lot of work. One thing I learned was there is a larger demand to watch the videos and take some quizzes than there is to do 15 hours a week of hard problem sets. Russ: Surprise. Was it also offered at the same time as a regular class at the U. of Chicago? Guest: Yes. I based this class on my Ph.D. class. Another reason I did this class was because I had it all pretty well prepared. And I did them at the same time. Part of the thing I was experimenting with was the flip-the-classroom concept, where my on-campus class took the online Coursera class first and then came to the class for discussion and elaboration. Not watching my butt while I put equations on the board. Russ: And how many people were in that Chicago class, in the real, non-virtual class? Guest: We had about 30 people in the Chicago class. Russ: So of the 300 that completed the coursework and did the exams, so close to 30 presumably were your regular Ph.D. students, and the other 270 were people from all over the country or all over the world. Do you have any idea where they are from? Guest: Yeah. A good geographic distribution. A lot from Europe. A bunch of Russians. Some Latin Americans. A bunch of Americans. So, people from all over. We didn't really get that many Chinese, but I think that's because my main method of advertising the class was my blog, and my blog is blocked in China. So they didn't hear about it. Maybe next time. And that's one of the interesting things, is I did get learning who is interested in this. A bunch of Booth (U. of Chicago Booth School of Business) alumns were interested in it. Ph.D. students at other universities. A lot of people in industry who wanted to deepen their knowledge of where the models came from that they were using. So, interesting spread of people. Faculty at other institutions, who wanted to catch up on asset pricing--they might have been experts in something else. Russ: And when you are talking about that survey, that distribution of folks and the differences between them, that would be true of the 4000, not just the 300? 37,000 were interested; 4000 watched some or all the videos; and 300 did all the work. Is that a good way to describe it? Guest: 4000 got through all the videos and 300 did all the work. MOOCs usually have about a 10% per week falloff rate and mine did as well. There is a quick falloff as the shoppers leave. And then about 5-10% per week after that. And that's one of the big issues of MOOCs, is keeping people engaged and getting them to finish. The initial falloff, I don't think is so bad. Because people shop, and that's the whole point. The whole thing about MOOC is it's free to sign up. So the 37,000 look at my first lecture--which was a review of stochastic calculus--and they said, Hmm, that's not for me. And so that's appropriate. Russ: Yeah. I don't understand this idea that if people don't finish there is something wrong with the course. It's like saying you should finish every book. When I was younger I finished every book I started. It's a bad policy. It took me a long time to grow out of that. There's some value to it. But this idea that somehow if you try something and you don't like it, there's something wrong with what you tried seems like a strange idea. Guest: Well, I think there are two aspects to that. One is, you are exactly right. We shouldn't look at that initial drop-off. However, the big question is: Do MOOCs substitute for conventional education? And one of the functions that conventional education does is it's a social and precommitment mechanism. You get in here and even though around week 7 you think, Spring is coming; boy, I'd rather go outside than finish this class--it's on a schedule. The threat of the bad grade keeps you to keep going. So--it would be nice if MOOCs could serve some of that function as well. Russ: Yeah, we'll get into that. So, talk about the technology of what you had to do to actually get the course up online. How much time it took, what the experience was like. Give us a little bit of the flavor of that. Because I think most people think, oh, you just get in front of a camera and they film you and that's nice. But that's not what happens. Guest: Yeah. I thought so, too. Which I quickly learned to regret. MOOCs are, like all technology, an innovation that gives you 0 marginal cost and a very large fixed cost. So, adding an extra student to the MOOC costs nothing, whereas of course classrooms are a limited size and expanding is hard there. That's a good part of MOOCs. The downside of MOOCs is like all technology the fixed cost is high. It is much more expensive in time and effort to get something up as a MOOC, and to revise a MOOC, than it is to just put your notes together and go give a lecture. Part of that is learning the technology, and then part of it is adapting the technology to what you do. Like designing a webpage. It takes a lot of time to do it. So that was my first--not just time and technological expertise. It takes a team. I had a great support team here at the U. of Chicago who not just knew the technology but knew the pedagogy, knew how to do it. I thought: Oh, yeah, I'll just take my class. 'No, no, no, no, no. You have to prepare short videos with in-video[?] quizzes, get that all set ahead of time.' And I was very grateful for this [?]. So, to put together a good MOOC, you really don't just turn on your laptop webcam and start talking. It really helps to have a team of people who know what they are doing, who understand the pedagogy of it, understand how to use the technology. The technology is already getting to the point where you don't just want to sit down with this and put the MOOC on. You want help. I could go on. Russ: So, in thinking about-- Guest: Sorry. I could go on and on about my experience with the technology and how it works. I'll let you probe.
9:44Russ: First I want to go on the other side of the screen. I want to talk about the student experience. And I've looked at a little bit of the class. I was a little surprised. It is you filmed at the blackboard. That is, it's not the blackboard blown up to fill the screen with your voice as a voice-over. The filming was you writing down equations, talking about them, gesturing, pointing like a regular classroom. So if I'm watching this as a student, I'd see that the way I would see you in the classroom. What's different? What role do the quizzes play? Talk about how they were set up. And other assignments. How were they graded? And office hours, to the extent they were possible. Guest: Okay. So the student experience, at least in my MOOC. And there is an interesting question, how do we use this technology? Because there are lots of different ways to set up the student experience. But the student experience in my MOOC was, you would go to the website, which would give you your weekly tasks; and it would say: Do some reading, maybe. And then it would say: Watch the lecture videos. The lecture videos were designed--they were about 5-10 minutes each. They were supposed to be 5 and sometimes they were a little bit longer. This class is about looking at equations, looking at graphs, looking at empirical results, understanding how finance works. One of the challenges is how do you both talk and present visual material? That's a challenge for how do you do that stuff. In my case, I primarily put things on the board, because I have a very visual style. And I don't like slides, because slides, you lose track of what was on a previous slide. So the entire lecture is on the board and then we can point to things. You, the student, can also get a PDF of what the board looks like, so you can see the whole thing in front of you. And we point to various equations and talk about them, and I'll put a table up with numbers and we'll talk about what they mean. Then there's a little in-video quiz to try to keep you awake. So you watch the video. They you go on and do the quizzes, so sort of weekly set of assignments. The first set of quizzes are multiple choice. The quizzes for now are basically multiple choice or numeric entry--really the only thing that works reliably. So, you take some multiple choice quizzes about the concepts we did. And then for my class you go on to another set of harder problem sets, which are like the sheet[?] problems [?] deducing equation derivations and I check that you got the right answer by asking you to fill in numbers to the final equations. So, you get stuck, what do you do? And there are forums. And I think as this thing develops, the social aspect is the part that really needs to develop a lot more. We have that now in the forums. It's a discussion forum. The students talk to each other: Help, how do we do this quiz? How do we do that quiz? What was going on? I had TAs (teaching assistants) monitoring the forums, and they were very helpful. Especially if there were any typos or anything, they were in there quickly. I monitored the forum; it didn't take a lot of time. Once a day I would check in and see how people were and chat back and forth on the forums. We also had Google Hangouts, where we would have sort of like office hours. It worked best to have students email questions; my TAs would go through the questions, find some good ones, ask them, and then I would answer them in real time. Obviously Google Hangouts are a limited technology, and that needs to be spiffed up; and I hope Coursera will put that in its own technology as well. That all develops that sense of community, which is where I think MOOCs really need to go. So that's roughly the experience. Russ: How many people would come to those Hangouts? Guest: That's a good question. Now Google Hangouts are limited to 10 active people, and we found that Google Hangouts, the voice in our direction [?] didn't really work. So the answer is in some sense we don't know. We were getting emails from, I don't know, 20-25 people typically. And then we would post the Google Hangout onto YouTube, so people could watch it either at the same time or then they could watch it whenever--because they are in different time zones. So I don't know how many people were actually watching the Google Hangouts/YouTube thing. But the students reported that they liked them a lot. It gave them an informal time to talk. We talked about things other than the class--you know, how do I become an academic? When the Nobel Prizes came through we talked about all the finance Nobel Prizes, and gave them a chance to know me, and they said they liked that a lot. Russ: That's nice.
14:12Russ: You've written that the MOOC is the equivalent, the modern equivalent of a textbook. What do you mean by that? Guest: Well, so there is a question: what is the MOOC, where will it go? One note that we talk a lot about how it threatens college education. Well, one thing I learned is that mostly MOOCs right now are adult education. They are people who are not going to college or interested in going to college. And mine, too. Most people were not people who were a threat to the university's business. But as far as textbook: people keep saying, is this going to displace us? No one will teach classes again. And my funny story is--yeah, in 1492, Gutenberg invented moveable type, so textbooks will be presented; they'll just read the textbook and no one will come to our lectures. Well, that didn't work out. Russ: Although it does in some classes. I'm sorry to say. Guest: Well, yeah. But I do think MOOCs--finally enough funny stories. How will MOOCs function in the marketplace? One strong function already happening is that the MOOC class is something that a group of students at a university might take under the supervision of a faculty member; and then the faculty member would use it for his or her own flipped classroom. You take class at a university, your assignments are: Go to Cochrane's MOOC for week 1; then come in and we'll talk some more; and you'll take final exam here. So in that sense, functioning very much like a textbook. The same flipped classroom I used, somebody else could use as well. Or, if you like just reading a textbook, read a textbook. But I do think--the MOOC experience is not just a complete substitute for taking a class. It is also a set of tools and materials that are the foundation for somebody else teaching class. Much the same way a textbook is. Russ: Well, let's talk about the flipped part of your class, the live part that you did at the university. Had you done that before? What was it like? What do you think the students got out of it compared to the students that didn't get the flipped part, who just did it online? Guest: Yeah, I thought it worked well. You know, I hate to sound cold, but young people today don't read. And if you give a reading assignment and you say, Come in on Monday and discuss it, they look at you with blank faces. If you say: Watch a set of 5-7 minute videos, do some multiple choice tests, multiple choice quizzes--the quizzes also can refer to the reading assignments; by the way, I'm checking those quizzes and you are going to be graded on them and you've got to get them in Sunday night at midnight. Lo and behold the students come in Monday morning just very well prepared. They've gotten the basic factual material that I present in a me-talk/you-listen lecture and ready to challenge me and really think about it. So--now the flipped classroom is a lot harder for the faculty member. It's really easy to just stand at the board, and I pretend to teach, you pretend to learn, and we write equations down. So, running a discussion of some really thoughtful Ph.D. students, that takes work but I think it's much more satisfactory when you are done. So, bottom line: I think the students in class got a better experience than the students on the MOOC. And also, it freed me up in class to do more advanced material that MOOC students didn't get. But, not everybody can spend 4 years of their life at the U. of Chicago going a Ph.D. program. So I think that people who were taking the MOOC class got a really good half of what the people on campus got. They get less of--so, where does traditional teaching still [?] less, and where do MOOCs have to go? A lot of being a teacher is not know the right answer. It's knowing all the wrong answers. And when you come in and say, 'John, I really didn't get it because I thought that when the expected return went up we would make more money', is I'm: 'No, no, no, no. That's mistake 32b-slash-5.' Russ: You've heard it 7700 times in the last 10 years. Guest: Yeah. And even the ones that I haven't heard before, we can synch through to get it. Now that is the experience that is much harder to replicate on the MOOC. The social part of MOOCs and the forms make it that that's where it goes. But really, that's what distinguishes, that's going to be the challenge. It's easy to put online: How do you pass your pilot's license test? Material that's very cut and dried. You think it's somewhat easy to put online material where we all really know what the material is and there is a well-studied pedagogy. You know, Algebra 101. Putting online things where the actual knowledge is closer to the research frontier and it's less well-studied, that's going to be harder. And getting that experience--what teachers actually do, of noticing your mistakes and helping you to correct them--that's a social part of online that I think is, it's there now, but I think that's what you still get in a good classroom. That's what's coming. Now, on the other hand, what's great about it--I'm snobby. I'm at the U. of Chicago. If you are out in the middle of nowhere, and your faculty members aren't that good, well, I think that MOOC provides the great experience--it can provide a much better experience than a third-rate college education. Russ: I think that's particularly true at the high school level. I think people complain about a video class can't substitute for a degree, face-to-face teacher. No kidding. But most people don't have a great face-to-face teacher. I'll keep my personal bias about the U. of Chicago to myself. But we all would agree that there are teachers at every university--perhaps even at the U. of Chicago--who are not so great in the classroom, not so great face to face. And the chance to learn from a great teacher anywhere in the world is a glorious thing. Let me ask you a tougher question. Do you think the people in the flipped classroom at, in Hyde Park in Chicago, got a better education than the people who just did the MOOC online? How would you compare the experience of the people who did the MOOC at Chicago with people who took the class from you with just the classroom 3, 4, 5 years ago without the MOOC technology, without the flipped part, and watched your rear end, as you put those equations on the board? And let--you know, you could chat about the Nobel Laureates that week if you wanted or as you added. It's a different educational experience. Do you have any feel or idea of what that new flipped classroom was like across those two choices? Guest: I think it was very good. And, even if one doesn't go MOOC using these online technologies for regular classroom instruction is going to improve, can improve classroom instruction a lot. Now, then, once you've created all the online materials, the marginal cost of opening it to a MOOC is zero, so that's part of where the economics methods may go, is that we're going to create these materials for in-class use and then it's free to open them up. But I think it was very good. My students came more prepared. My students told me they liked this. So my lectures, my 5-minute MOOC lectures, are very condensed. When you give a real lecture, you stop; you tell a joke; you let things sink in. You try to get a sense of are people following, are they not. Half of them aren't, and they are just kind of hoping they can go back and review it. And several students said, Look, this is hard stuff, and I loved being able to stop and rewind it, go back, check something, and not feel like I was the dummy who was holding up the class. So, even just the flat lecture part, they said they got more out of--they liked me better online than in person. And of course it let us go into deeper discussions, more intuition, and more advanced material in class. I particularly--one reason I did this is that the modern business school I think can benefit greatly from online technology. We--business schools have gone pretty much to teaching 3 hours once a week, and it's a big strain on the human brain to get 3 hours of new material once a week. Okay? I find it hard to stay away in an hour-and-a-half seminar. And three hours once a week is a lot. Well. And a lot of what we do now is 6 hours every two weeks, or international programs that are 5 concentrated days every 6 weeks. That's very hard in the traditional lecture and problem set style. Now, think of that setup with online materials, where every day you watch a snippet of lecture; you do a little bit of a quiz; you get on the forum; you talk to your professor, your TAs, you talk about the intuition on the forum. And then you come together once a week to review what you've done. If you have a professor who can get a set of exercises, a set of things we do together that really build on those things, you have a much better learning experience.
23:37Russ: Yeah, I think that face-to-face part is--it's not--it's such a subtle thing. And it obviously varies tremendously by subject matter. One of my all-time favorite undergraduate classes was a class on Faulkner and Conrad. And we would read--I came into the class loving Conrad and not having much taste for Faulkner. But I figured I'd put up with the Faulkner so I could enjoy the Conrad. By the end of the class I loved Faulkner. I wasn't such a big Conrad fan any more. And it wasn't because my teacher taught us a lecture that I could have watched on video or face to face about how great Faulkner was. It was the conversation that took place in that classroom about--I think it was Professor Patterson, by the way; thank you. That conversation was, can't be--if somebody watched it, the film of that conversation, they wouldn't get anything close to as much out of it as those of us who participated in it. And I think that's what great learning is about. And I think the ability of the web to deliver that is limited. But positive. And it depends on the material. Guest: But we can't be snobby. Not everybody can have a great professor like you did. And so what I think the web, the online stuff, certainly does, is it brings much better access to that sort of thing to people who are out of college or out in the middle of nowhere. And, I also think this highlights, so I think the next step for MOOCs--MOOCs are now very much Web 1.0. It's one-directional. It's web pages. You are looking through a web page. The beginnings of discussion forums. But you know, we've had discussion groups since the 1990s. I think the challenge and the opportunity for it is to move into Web 2.0, where that kind of social and discussion thing can happen remotely on a fairly large scale. And then I think you'll get a lot better experience. It never will be as good as face to face with a great professor. But certainly it'll be face to face with your fellow students, and some of that discussion thing. And we've got to remember, you know, young people are very good at this. They text each other across the rooms. My kids text each other from different rooms of the same house, and seem to achieve a lot of core[?] communication that way. Russ: That's for sure.
26:05Russ: Let's talk about the potential of MOOCs to disrupt either high school or college or university education. What do you think? Is it just a gimmick? Is it going to catch fire as the 2.0 version comes along? There are a lot of barriers to it, obviously--a lot of institutional barriers certainly at the high school level. And it requires a lot of different skills. I think the skills it takes to be a great flip-classroom teacher are not the same as the skills to be an enthralling lecturer to 500 people in a giant lecture hall. So, where do you see that going? Guest: Well, for right now, MOOCs, their market is adult education primarily. A group of people, and Coursera especially, take lots of classes. And I don't mean that disparagingly. There's a whole group who take finance classes together; and they showed up for my MOOC and helped each other out with the hard equations. Now, in a classroom setting, the hard question for MOOCs is certification. When do you get college credit? When do you get something other than, you know, your certificate from Coursera that you took this MOOC? When do employers start looking at MOOC certificates or groups of MOOC certificates as being a certification equivalent to undergraduate degrees? That's a harder question. Now, that's starting to happen. Universities, some universities, they are sort of using the MOOC as textbook and then granting--their job is to say, here's 10 MOOCs that we think are worthwhile, and we'll give you credit for taking these MOOCs, maybe with an exam that we proctor to see that you really know what you are doing. So, that's the step of disrupting college and high school, is intimately related with the step of when does taking a MOOC count for the kind of credit and credential that we give to people who take colleges and high school. That's a hard step that everyone's thinking about. Russ: It seems to me a lot of it is cultural. I mean, if I had to choose personally between or for my kids, and I have college-age or near college-age children, I think about what advice I would give them about what to do with the next four years of their life, as they are about to go to college. It's hard to imagine saying, Well, what you should do is you should stay home with us because it's cheap--and you love us, of course; nothing you'd like more than living with us for another 4 years as you enter young adulthood. You should live with us and for 4-6 hours a day you should be online watching videos and doing the homework and doing the quizzes. And you'll save--let's say we've solved the certification problem. At the end of this year you are going to fly to the U. of Chicago or wherever and you are going to take the exams for these classes. And it'll cost a tiny, tiny fraction of what it would cost otherwise. It's hard to make that leap, it seems to me. I know there are some people trying to create an all-MOOC university. I don't know how that's going. But socially, the social part of college, is one, is a whole separate question, which is a huge part of what I think people pay for when they go to college. But just on the education, part, if you want a technical degree, a degree in a STEM (Science, Technology, Engineering, Math) field that lends itself to MOOC-style precision. It's an interesting option, but it's hard to make that leap, culturally, I think. Guest: Well, culture, yeah. Let's be economists for a minute. The culture follows the economics. For the moment, if somebody comes to you and says, 'I want to fix your computer and here's my 10 MOOCs that I took,' does he really know what he's doing? And the certification part is: do they really know what they are doing? So, the college part-- Russ: Oh, John, come on. You are going to tell me that just because they have a B.A. from University X that has a nice reputation, I'm confident they know what they are doing? Guest: No. But I think that's a lot of the function of what universities do. Russ: Yeah. Guest: We don't necessarily do a very good job of it. Why do you hire somebody with a U. of Chicago or Harvard degree? Well, that's a brand. And it's a brand that is pretty careful about its reputation. It wants you to think that our students are good and you can hire them with confidence and that we have certified that they know what they are doing; and you point out, correctly, that maybe we're not as good at doing that as we say we are; and that we had better shape up. And that that's the challenge for MOOCs, too--especially if you want to start monetizing them instead of making it for free. To be able to say that you are providing that branding, that service that colleges try to provide--and some do and some don't. Russ: Well, let's talk about that monetizing, right? Because this technology in principle--let's move away from economics. I'm going to go to math. It's like economics in that there's a lot of really bad economics teachers, I think, teaching at the college and high school level. And I know that's true of math also. So, let's take math. I apologize--I can't remember the professor's name who I saw teaching a MOOC on calculus, but I watched the first--I don't know--half an hour or so, and it was fabulous. How many first half hours of calculus classes in America are fabulous? Well, there are a few. There's maybe 100. Maybe there's 50. Maybe there are even 400. But there aren't 40,000. And so when you think about all the high school students suffering through bad calculus teachers, the value of that half hour and the subsequent time is so extraordinarily great. And now we've got a technology that delivers it. And I'd love for that professor to get fabulously wealthy. I'd love for that professor to become a millionaire. Just like a great textbook writer can become wealthy. But this to me is more valuable than a great textbook. I'm not sure I could say why, but it just feels more valuable, and it seems like it's more transformative to do a great MOOC than to read a great--I guess the simple answer is generally textbooks aren't written to be read. They are written to be tasted. So here's somebody who can deliver hours of genuine insight into a very subtle and difficult area. That person should get fabulously wealthy in a market-based transformation of the education field. Can that happen? Guest: It should. But let's go back just a minute and remember. We had the technology to play videotaped lectures for 20, 30 years, and that never really took off. So, a successful MOOC is more than just great lectures. However, I agree with you totally, especially having suffered through lots of bad math classes myself, and that math is not cut and dried. It's about intuition, it's about passion. And even perhaps your bad high school math teacher might be able to sit in the flipped classroom and help students through problems that they encounter after having done the great MOOC. But yes, so, the economics of it? A large base, you can put a lot of pennies together and get a superstar economy, that's one of the monetizing ideas, is that this will make teaching like so many other things where a few superstars earn a lot of money; and other people are there at best to be glorified TAs, and won't make lot of money. That's a possibility. The flip side is that this is a 0-marginal cost business. Hard to charge money for a 0-marginal cost business. Russ: That's correct. But the certification is where I think the monetization comes in. To the extent that if people could verify that you got an A in a very hard class, a genuinely hard class. In fact, the certification could be better with a MOOC because you could actually be transparent about what an A meant. Whereas at many, many universities now an A is much less valuable than it used to be. And as a result-- Guest: It means you are breathing. Russ: What? Yeah. Guest: It means you showed up to the final exam and you can breathe. Russ: Shame on who--I don't know who; that's the problem. It's an emergent phenomenon, this phenomenon of grade inflation. It's an interesting thing. But if you said here's what you had to do to get an A in this class, it actually could have a lot more value. And you wouldn't care if 80% of the people got an A, because you'd know what it meant. You'd know, in a way you can't know about a calculus class at a university level. So I think the potential to do that is very large and I'm not sure what's holding it back. I think both at the university and the high school level. The high school level has an immense amount of material available, say at Khan Academy and elsewhere, at the level that high school students can really teach themselves a great deal of material. The areas where it's weak obviously are writing, and what we might call thinking. I don't think a lot of thinking gets taught in most high school classes, anyway. So--maybe I'm wrong. What do you think?
35:57Guest: I think you are exactly right. There's this danger in discussing things like education, health care, to say, everybody has to have the absolute best. But if the majority of high school students could come out knowing calculus, knowing basic grammar, getting sentences out, and how to program a computer, even without the deep appreciation of literature, it would be lovely if they had--a huge step forward in our world. Russ: Seems that way. Guest: But, what you just said is interesting about the economics. You said that there's a function, the branding one, and that is: so there's 100 calculus MOOCs; which are the good ones? Maybe the economic rent will come to the person who says, Look, I've studied all of these and here's the 5 good ones. I give you the Russ Roberts Degree because you've taken this list of MOOCs, as opposed to the actual--[?] they could split it with the professor himself. The other model is, you know, our universities do a tremendous amount for brand name recognition, and once something is 0-marginal cost, we may end up just giving it away for free because everyone wants to be the big brand name of the global university. We'll see how that works out. Russ: That's going to be interesting. What I'm waiting for, and I don't know if I'll wait in vain, is for John Cochrane and 25 other, maybe 40 other, fabulous business professors around the world to offer an MBA (Master of Business Administration) that is disembodied; it's not in a physical location, except maybe for the exam, and maybe we could solve that problem as well. And that would be a very--ironically, it would be an inexpensive degree to acquire but very lucrative for the faculty who work or are involved in it. Of course, it's a different experience than being a faculty member at a normal university. You don't have the collegiality and stimulus of the research side. But it's an interesting model. We're limping toward that model. I don't see anybody sprinting toward it. Guest: Well, except--so what does the bricks-and-mortar business school do that that doesn't do? A bricks and mortar business school is selective about who they let in. In fact we are often accused at simply being really good at selecting smart people and then giving them a 2-year party. We have connections to employers. We have a fantastic office that gets them jobs. And we have an alumni network. Now I think you can form an alumni network. The people who met online in my class still stay in touch with each other, which I think is a very interesting phenomenon. And I wonder in your model whether you could be picky and say, Look, before you are allowed to take this class or be part of our MBA, you have to score so much on the GMAT (Graduate Management Admission Test) or something of the sort to do the selection part as well. But I think it's a very interesting model. Maybe I should get off my butt and do it. Russ: And the high school model is I think even more disruptive. We can joke about how modern university education has many problems, but I don't think it has as many as the modern high school. And the opportunity to home school--enough people don't like their high schools, for a variety of reasons, that they home school their kids. And this is already happening, of course. People are using these online resources with great zeal. But the opportunity to put together an entire high school experience, be an online education--which, by the way, so much time in high school is wasted. So, you could spend your mornings in class and you could spend your afternoons at museums, or building whatever you wanted in your garage or coding or doing a thousand--learning the flute--there are a thousand things you could do more effectively than sit 30 people in a room on [?] lessons and hope they behave themselves. It seems to me that the opportunity to transform high school education is huge. Guest: Absolutely. And there we've also made a transition. High school education is material that is by and large codified, even where in some cases it's codified wrong. We do not ask high school teachers to be researchers, to think of new things. We ask them to be professional pedagogues and to know how to get a codified set of material in. And there are certainly economies of scale. And, one of the great things MOOCs are doing is learning about pedagogy. At universities we are pretty bad at actually self--we are supposed to be empirical social scientists but we don't really look at how we do that well. And the MOOCs are looking at what works and what doesn't work in the way of all technology. When you get to university, the idea of the research university is that you are at a level where the textbooks are a little bit out of date, and so you get together with a professor and kind of think through things that are still at the fuzzy edge. That's going to be harder to bring to MOOCs, but frankly so little of what we actually have to teach people is at that level, even introductory university classes are pretty cut and dried as well. So, I agree. That could be a great savior of the American high school experience. And you say we'll miss the social side. I'm not sure that the majority of people think the social side of high school is a great thing. Russ: Yeah, it's a feature, not a bug.
41:27Russ: Let me ask you a question that comes from Arnold Kling, who has been on here to talk about these issues as well. I have a couple of insights through him I want to hear your reaction to. One is that education is feedback. So, what Arnold, I think, means by that, is that the way you learn--think about this--we call feedback: I gave him a quiz, or I gave him a homework set. But what he's really talking about is a wide range of subtle things. Which is--does the student know whether the student is catching on? And I think so much, when I think of my failings as a teacher, it's when I just sort of said, Well, it's up to them; they've got to figure it out; and I'm not going to give them--I loved and still do, open-ended questions that don't have precise answers. But my students would say, 'But how am I doing?' And I'd say, Don't worry, you are learning. That's what this is about. This isn't about, I got a 73, I need to study more; I got a 96 so I can slack off. This is an intellectual adventure. Well, most students don't like intellectual adventures. Some reasons they don't like them are not so good. But some are good. They don't really know what they know. And so I think the biggest challenge of the MOOC is that. How do you give feedback? How do you help students understand what they know and don't know, to lead to mastery? Guest: The two parts of education are the social structure that keeps you going and the feedback. With a great teacher, the person who as we talked about before who knows all the wrong answers and can help you quickly straighten out a misconception, quickly see where you are kind of thinking about things wrong and bring you back to the correct path, let's you see how you are doing relative to some standard of mastery. Now, they have their quizzes. And the good part of it, actually: formalized quizzing is in some sense better than: Write an essay, I'll have a graduate student grade the essay to see if you know what you are doing. Well, that graded feedback in the traditional university, and so the old marginal cost thing--you know, once I write a good multiple choice quiz, 100,000 people can take it and get some sense of where they are. The quizzing I found was one of the weakest points of current software that I used and clearly one that can be improved dramatically. And maybe students will get--they'll certainly get better assessment than they often do now. The challenge is to get that feedback to help them get out of bad ruts. And there I think the Web 2.0 social development is going to be key to making it work better. Russ: Yeah, it's going to be interesting to see what happens there. But what I think is bizarre about the modern university is that so many professors give multiple choice exams out of what I have to assume is laziness. I've given a few in my time, but most of my exams were open-ended, challenging questions that didn't have easy, straightforward answers. In fact, they were so not-straightforward I often would grade them myself rather than using my teaching assistants. And that was very time intensive; and I think that was when I was the best teacher that I was, the more open-ended and challenging those questions were. The funny part is, when you are teaching 100,000 students you can't grade them individually. You can't go through them. You want to exploit the economies of scale you are talking about, but I think unfortunately at many modern universities, 200 or even 80 is too many for people to grade thoughtfully. So they just use multiple choice as if they are teaching--they are going to exploit the economies of scale even though they are supposedly in a small class. And I think that's just a terrible tragedy. Guest: Well, what you are saying is that what the MOOC really endangers is the 150-person lecture class at a second rate university. Because that the MOOC can do with a multiple choice quizzing and no real feedback and a graduate student who barely speaks English as your TA. That the MOOC can do a lot better. The 20-person class where you get great feedback from a great professor, that will still be there. But that's like saying, you know, Southwest Airlines isn't as good as a private jet. Well, thanks. But it is true--so one of my biggest frustrations with the current state of MOOCs in doing my quizzes and assessments is that like you, I organize my classes around weekly problem sets. And they were almost all open ended. Particularly a puzzle-type page. Here's a natural misconception; how do you resolve this question? And you've got to go resolve this question. I couldn't figure out how to ask that in machine-gradable form. Or: here's a theorem; prove it. You can't ask that in machine-gradable form. Russ: And when you do, you take away the learning. You take away the experience the student has. If you say, here's four answers, one of them is the right one, three are awful--the student can often tell, yeah, 'I know what he wants; he wants a. because b, c, and d., he never talks about those things.' But you don't learn anything from that. You don't learn anything the way you do when you have to get up in front of your study mates and defend an answer to a puzzle that you think is right and they think is awful. Guest: Now, on the horizon I think they can do a little better. So, you certainly can do, ask a question and write a paragraph, and you can think of ways of grading a paragraph that can mechanically grade[?] or I think. So I often do this, my actual grading--I hate to tell this; my actual students might be listening--I'll ask: Here's a puzzle; the expected return on one strategy looks better than another; how do you solve this? And the way I grade it is if the word 'beta' appears in the answer they get credit, and if it doesn't, they don't. Russ: Oh, John. I'll edit this out. Yeah. The equivalent for me is: Here's a puzzle that I thought of for my kids. This is what I call dinner table economics. The puzzle is: Why do lightbulbs last longer than they used to? Because a lot of people think, lightbulbs wear out because they want to sell you more lightbulbs. It's a really bad answer. So if that was one of the answers--again, if you wrote that down as one of the answers, compared to, say, the correct answer--which I'm not going to say--it's not a hard question but you should think about it out there--the right answer is, you've got to think about it, but there is a key word in that right answer and if I saw it, yeah, I could grade it pretty quickly. Guest: Yeah. Now, on the other hand, I've gotten pretty good at multiple choice questions. Well done multiple choice questions. Russ: Yeah, they can be done well. Guest: You can ask people to think. But how do you machine grade assessments--that's one of the big challenges. Russ: Yeah. I'm not sure it's ever going to be fully solved. But maybe it will just be solved pretty well and that will be good enough.
48:30Russ: I want to give you another, a different kind of challenge. Let's take me. I'm your student. I'm your only student. There's a famous, I think I've talked about this on EconTalk before. I think Pigou--tell me if you know this story. I think Pigou took a class from Marshall where Marshall came to teach the class and Pigou was the only student. I may not have the names right but there's a famous British economist who showed up every day or a couple of times a week. And he had one student; and he taught the class as if there were 80 students there. And he just lectured, and the student, if it was Pigou, took notes; and that was the class. Now, Arnold Kling again raised the following question: he says, I think that the best way to think about teaching is to think about what would you do if you had just one student in the room, rather than thinking about wow, if you get 100,000 students to watch this lecture. So, let's take me. I'm your student, John. I know a little bit about finance, not that much. I'm okay at math; I'm not a star. I can do some of the math; but some of the math in your class is challenging for me; I'd have to tool up. I show up at your door and I want you to create a class for me. Now of course the ideal isn't--you don't have time to create a class for me. So I want you to create an online class but it's for people like me. Is that different from what you do for 100,000 people? That's my question. Guest: Yes. So, two answers to that question. One thing that implicitly you bring up is that MOOCs may allow a lot more customization. You can do MOOCs on subjects--so in the modern university you cannot teach a one-person class. Sorry. We watch the bottom line. But if you are interested in early 13th century poetry-- Russ: We can teach three people sometimes. You're making it sound like-- Guest: But if you want to teach a class in early 13th century Iranian poetry or something of that sort, there are 150 people around the world who might be interested in that. So one thing this might do is to give us classes that are both more specialized in a topic and more specialized to the person. There could be--there are already 100 Introduction to Finance classes. And there is one out there that is exactly right for your interest and your level. So to some extent the MOOC is going to do that. The thing it's not going to do, which I would do with you for a 1-on-1, is of course, I would not give a lecture. We would talk and it would be a lot more of me listening. In my other instruction life, I'm a flight instructor. Which is done one on one, and where assessing the student's competence is really important. And where assessing the student's misconceptions about how things work is really important. And that's what you do when you are one-on-one and the guy needs to learn how to fly the plane. And by 1-on-1 sort of quizzing, I'll pose a puzzle; you tell me the answer; I'll go, is that really how it works? We really explore what you understand and what you misunderstand. That's the way you teach 1-on-1 classes, and that's the thing that's hard to do on a MOOC. Would you really trust a pilot of your plane who said, I learned to fly on a MOOC and a simulator? He might be darned good. And he would certainly have run through all sorts of accident scenarios that the MOOC and the simulator did, but there might be a few remaining misconceptions about things that had gotten through the process that you might worry about. Russ: But to think about me again, and I think your point about specialization, customization, is exactly right; and I think we've just scratched the surface on this. Right now the way it's customized is: Oh, yeah, I know this material; I'll just fast forward through it. Or: I don't know it; I'll play it three or four times. And that's great. But obviously there is going to be a way where you can say, I want to go down this path or that path, and there is going to be menus and options that aren't available right now. And one way to think about this is you have the high school student who is a superstar and is being held back--you just give them extra problems, you give them extra reading, and so on. And I think the MOOC has the potential to do that with extraordinary success. But I'm thinking about just the pedagogy for a minute. Like you said, you wouldn't lecture to me, right? So what would you have me do? Would you have me read? Would you have me watch the videos and then we'd talk? Would you probe and quiz and challenge me to see what I knew? What would be the pedagogical differences between that one-person class and a bigger class? Guest: Yeah. If I had all these materials, I actually would have you watch videos for just straight explanation stuff, rather than me give a personalized lecture, one-on-one. I would save my time. We're also presuming my time--I hope your example wasn't one where we presume my time is completely worthless. Russ: Right; you're just at my beck and call, John, for all my dumb questions. That's what it is. Guest: As with my flight instruction students: I say, Look, I am not going to give you a one-on-one lecture. You are going to read the book, and you are going to come back and we are going to talk about the book. And so I would have you, I think the MOOC materials would be great. But then we would spend a lot of time one on one probing what you know and what you don't know, and helping straighten things out. Working on problems together. That would be the ideal thing to do in the one on one. But, you know, not everyone can have that. I think the economics you set out exactly: How do we leverage the talents of the best possible ones to give something that's pretty darn good to 100,000 people? And maybe more tailored to what they know than even being in the class with 1 great professor and 25 other dumb students isn't that great either. Russ: My hope is that--I want the technology to leverage something you referred to in passing, which I think is under-rated, which is the wrong answers. I've always dreamed about a problem-set based class which could be delivered to lots of people because I've done it so many times I know all the wrong answers. I know the likely mistakes you are going to make, so when you make those, I don't just say, 'Wrong', and you get to guess again. I say, 'Wrong; here's a hint.' And you get to go back into the tree and figure out which branch you should have gone down. I think that's the goal. That's the standard that we could maybe hit some day. Guest: Yeah, that is something in principle MOOCs should be able to do. The quizzing should be able to--when you have questions in a field that's well-enough developed that you kind of know the wrong answers, a good quiz should pose something. There's a wrong answer; even if you enter a wrong equation, oops, I recognize what you are doing wrong, help you find it, and so forth. Right now, at least the Coursera software that I used is quite primitive relative to that. I wasn't even able to allow students to attempt a question, get the wrong answer, and then get a hint and try that question again. This is simple programming. But I certainly think that flexible quizzing that responds to common wrong answers is something that certainly should be and probably will be in version 2.0 of the software.
56:05Russ: So, let's close. I want to talk about EconTalk, this particular form of online education. I've thought about teaching a MOOC or creating a MOOC, and I realized I've got this wonderful massive open online educational experience already, which is that those of you who are out there listening, many of you, tens of thousands are listening every week and over the course of a year or two years it's closer to a hundred thousand, for a particular episode that's interesting to people. And I think we could do much better than that. And I think there's a lot of you out there who would like to engage more than just listening for an hour a week. So, first I want to say that I'm working on those now. I'm trying to find some ways to leverage my time effectively that I can interact with you and provide material for you, as we've started to do this year already a little bit with this essay contest. I need a better way to communicate with you, other than Twitter and Facebook, because many of you aren't on those sites. And the only way you hear me is this once a week for an hour, which is by definition delayed, the way we do it, which I think is fine. But I'm working on that. And I'm curious, John, if you have any thoughts on that. What could I do--so my first thing is, those of you who are out there, please, if you are interested in going deeper into some of these ideas and looking for connections between one episode and another, I'm going to try to start creating those resources for you. But I'm curious, John, having taught a MOOC, do you have any thoughts on what we could do to make EconTalk a little more valuable to the audience? Guest: Yeah. Let me preface by celebrating the question. You are a real out-of-the-box thinker on how do we get economic ideas and economic discussion into a larger framework. MOOCs are part of it; they are one technological format. Blogs are another. EconTalk is obviously another way of getting these ideas out. And, boy, it's an exciting question to start thinking about. Now, what can we do? Sometimes talking goes on too long. My daughter listens to you faithfully, listens while she runs or paints or does something else; and because then she says when it gets boring she can kind of zone out for a while. Russ: Sure. Guest: Yeah. Obviously integrating visual would be an interesting thing to do. Some sort of back-and-forth: we don't want to turn it into Talk Radio--that would be interesting. Something on the range of TED Talks (Technology, Entertainment, Design)--you know, ask me to do a little more homework and prepare something to show for ten minutes and then discuss it; that might be an interesting thing to do. Get into those sort of deeper economic ideas. That would be an interesting thing to do. It's a great question with a hundred different answers, and I hope you keep exploring it. I say to all my students, yeah, my standard answer to all my students when they say something and I don't know what the answer is: Great question. Russ: Well, that's okay. I think we'll head toward a bunch of answers. There obviously isn't one answer. And I think one of the great advantages of audio is the opportunity to zone out, by the way. Or to re-listen to it. I know a lot of you out there listen more than once, which I stand in awe of that fact. It boggles my mind. But what I'm thinking of is some way to test or quiz or challenge people to take what they've heard and take it a little further. And that I think is what would make this experience--for some subset--obviously there are many of you out there who aren't interested in that. You are happy, an hour a week is maybe more than enough. But for those of you who want to go deeper and want to learn more, I think we're going to look for some ways to do that. Guest: Well, certainly, if one wants to use EconTalk as materials in a classroom setting, which I know people do, or as material for a MOOC, the feedback I've gotten from MOOCs is that the occasional quizzes, the little quizzes every couple of minutes in the lectures and the quizzes afterward, you know--what was the main point? How would you disagree with Cochrane's statement about x, y, and z? Those things kept people awake and made them go back and review things and put things in a framework and solidify it as a learning experience. When there is something that is a learning experience, adding that part of the MOOC technology could be quite useful.