Adam D'Angelo on Knowledge, Experimentation, and Quora
Aug 8 2016

Adam D'Angelo, CEO of the question and answer website, Quora, talks with EconTalk host Russ Roberts about the history, evolution, and challenges of Quora. Along the way they discuss the aggregation of knowledge and the power of experiments for improving the day-to-day performance of the site.

Pedro Domingos on Machine Learning and the Master Algorithm
What is machine learning? How is it transforming our lives and workplaces? What might the future hold? Pedro Domingos of the University of Washington and author of The Master Algorithm talks with EconTalk host Russ Roberts about the present and...
Ryan Holiday on Ego is the Enemy
  How does our attitude toward ourselves affect our success or failure in the world of business or in friendship? Ryan Holiday, author of Ego Is the Enemy, talks with EconTalk host Russ Roberts about the role of ego in...
Explore audio transcript, further reading that will help you delve deeper into this week’s episode, and vigorous conversations in the form of our comments section below.


Aug 8 2016 at 12:49pm

Serious question: where did I screw up my Quora personalization so that I mostly get “how can I reliably double $10k in one month” type questions ? Is there a way to mark a question “stupid” ?

Greg Linster
Aug 8 2016 at 12:59pm

If you decide to take Mr. D’Angelo’s suggestion about shortening the intro, I, for one, am going to miss the music.

On a serious note, it seems to me that Quora is not a suitable place for all types of questions. In fact, much like Facebook, the content displayed is manipulated, often subconsciously, by biased human engineers.

I’m curious as to how Quora deals with the problems associated with giving off the pretense of being objective.

Mark Sundstrom
Aug 8 2016 at 4:23pm

Thank you once again for an interesting podcast. Interestingly, I was getting a few “Internal Server Errors” trying to access the Quora website this afternoon, so maybe they did have an activity spike.

I have used Quora casually for a few years but haven’t found much value there yet. Maybe I’ll pay a bit more attention for a while, to see if I’m missing something.

I have to chime in on the music intro/exit and podcast speedups: I have always listened to the entire musical intro and exit; I’m just never in that kind of a hurry that I’d want to skip it. And the entire podcast always at normal speed, but that’s not always true for the other podcasts I listen to.

Aug 8 2016 at 5:26pm

Interesting topic doesn’t make a business viable:
– knowledge is not expanded through democracy – new knowledge is by definition introduced by heretics
– knowledge transfer requires intelligent questions much more than intelligent answers – there is no good answer to a bad question (but a clever answer maybe)
– what do you do with binary answers (where the “experts” strongly disagree)?
– Adam sounded very flat – how he can sell anything is a mystery but perhaps the land of unicorns is getting desperate for action
– too cumbersome to log in (people like anonymity)
– it sounded like it’s out of balance – as expected, most users want to be spectators, few want to ask, and even fewer to answer
– how do you attract the true experts instead of the idle big mouths?
– what did Obama answer to the question: “Why isn’t Clinton in the prison yet?” (leading with these politicians as “experts” is one more reason to distrust the concept)

Charles Hu
Aug 9 2016 at 3:16pm

This is my first time on econotalk. I find the topic discussed interesting. the guest speaker appears to have significant operating experience and I agree with most of his viewpoints, and learned a few things about how the internet interacts with me on a personal level.

I liked his approach valuing both metrics and surveys. I think this is a balance that would be extremely important to achieve, but in the corporate culture currently metrics seems to be the predominant force. Perhaps doing away with the quarterly is a good idea.

I also liked a lot his idea about approaching problems with preexisting hypothesis. I think the basic honesty with ourselves is probably harder to achieve than one care to admit. I probably wasted a lot of time lying to myself.

Regardless a great interview and applicable to what my goals are. I am curious how the speaker deals with personalization leading to perpetuated prejudice. Eg is a white supremacists going to find his views corroborated and widely popular, because that is the contents that will generate clicks? Or maybe it’s irrelevant?

Philip Jollans
Aug 9 2016 at 4:34pm

I had never actually heard of Quora, so I went to the web site to look at it, but I couldn’t access it without registering on the site, so I left.

I’m surprised you didn’t discuss the surprising feature, that you can’t even view the site, without signing up. Why would I sign up to a site I don’t know? How can that be a good business model?

Aug 9 2016 at 9:37pm

The interview was quite interesting, so I went to check out the Quara site and was extremely underwhelmed. I browsed a bunch of topics in cognitive science, philosophy, psychiatry, economics, etc. The questions seem like they were being asked by young children. They are extremely simplistic or just the worst kind of clickbait. The only thing I can compare it to is a much worse version of Reddit.

The fact that this site gets as much traffic as the guest claims and that people are valuing this company at $1B makes me a bit sad for humanity.

Aug 10 2016 at 10:26am

Long time listener, first time commenter.

You mentioned that you thought speeding up the podcast would change the pitch – you’d sound like a chipmunk. Not so – I use a small program called mp3speed (it’s a freebie, try it out if you like) that changes the tempo without changing the pitch. It just sounds like you’re talking in a normal voice, but faster. I have mine set for 145% of the original.

mp3speed is here:
but I have no connection to it, other than as a satisfied user.

I suspect you sound different in person because you have a much more intimate connection with your microphone than you do with a stranger standing a couple of feet away.

Thanks for the podcast! I’m not usually much into economics (I’m a math geek) and every week I say, well, I’ll just look to see who he’s talking to, then maybe I’ll download it. So far, I have *always* downloaded it. 🙂

Aug 10 2016 at 6:44pm

I’m deeply grateful for Econtalk. Thanks for making it happen.

I came here to comment on Quora and found that Ivan had already not only read my mind but also typed it all out. But since I’m here…

If I had to put Quora in one word, it would be a tossup between boring and lightweight. There are two main problems:

  • The questions. Here are the first three on my feed right now: “What are the most common lies told by programmers”; “Are there any informal moments you’ve had with president Obama, that you can share with us” [this one to Josh Earnest]; “Why has George Soros gained super villain status among conservatives”. And these are after I had UNFOLLOWED American politics.

    What a waste of bandwidth. Maybe there is some way I can keep the rubbish off my screen by stopping my feed, and just search for questions that might interest me. But that brings us to the second problem…

  • The answers I haven’t recognized any answerers as people whose opinion I’d want to know. They have titles like “constantly traveling”; “between cultures”; “opinionated enthusiast”. These are put forth as the reasons we should take them seriously. And the answers of the quality that you might expect from such answerers.

All that time, all those people, all that funding, all aiming to “share and grow the world’s knowledge”. Producing in the end an intellectual rubbish heap.

Daniel Barkalow
Aug 10 2016 at 8:42pm

The real potential problem with customization is that you might get a different view of the same topic from other people, and not (a) be aware of the content of other viewpoints or (b) be aware that not everyone shares your viewpoint. The algorithm could be giving you questions about economics that use certain terminology and get answered a certain way, and you’d think that Quora users see those questions and answers when they ask about economics, but they’re actually seeing other questions and answers that phrase things differently and disagree with you. One thing that’s especially valuable about EconTalk is that we all hear the same episodes, regardless of our views, and we can hear what those views we disagree with are.

Of course, a lot of website success metrics naturally lead to factionalization (like a lot of real-world social dynamics): you’re more likely to pay more attention to answers that you agree with, and answer questions that are asked the way you think gets at the topic the right way. You might think that nobody asks questions that are hard to answer in your theoretical framework, when the truth is that Quora just never shows you those questions, because you haven’t been eager to answer them in the past.

Aug 11 2016 at 12:02pm

I was disappointed in this podcast because what I believe are key issues were glossed over: What makes somebody an expert? How are they qualified/disqualified? How can they truly be relied upon/how can we believe what they write? Russ hinted at it at the onset with his comment about politicians, but the issue was never fleshed out.

Likewise, I’m surprised that “The Tyranny of Experts” podcast with William Easterly isn’t linked to in the notes section of this podcast. Granted the overall subject matter is vastly different, but Easterly did illustrate dangers of relying on “experts.” Nassim Taleb has also written on the fallacy of experts, though I don’t recall offhand that subject being discussed during his EconTalk appearances.

In any event, if Quora classifies politicians as “experts” for anything other than knowing how to get elected I think they’re making a huge mistake.

Aug 11 2016 at 4:44pm

I will never use Quora.

D’Angelo certainly came off as a nice enough person, but always remember that the ONLY reason for its existence (and GOOG, FB, Pintrest, Snapchat, and now MS) is to sell your data. The reason that you must log in is not to increase the quality of the questions and answers, but to link you and your habits with the other major miners and increase its value.

As before, the difference between what Google gets in revenue ($80B/yr) for your data and the “free” services that it provides to you is worth $16B (not even considering another $1B for their extravagant R&D boondoggles).

If Quora paid even a quarter for every “expert” answer used and a dime to each user for the use of their data, their entire business model would fall apart and their billion dollar valuation would be toast. (BTW, at the peak of the dotcom bust, by one metric private valuations were $1M/employee, at Quora, they are $7M/employee.)

Also, your CEO expert didn’t realize that if one had to listen to the intro music every time you accessed the site, no one would ever get to the download button. But since it is part of a one hour listening experience, the branding outweighs the time for music by a mile. I look forward to the A/B music length optimization testing over the coming year.

An old man in the sea
Aug 12 2016 at 3:55pm

Quora seems a nice site.

But much better are the forums in StackExchange.

We can share code, write with LaTeX, and ask very specific/specialised questions.

My experience on Quora is very different. The question has to be simple enough to fit in very few lines, no LaTeX allowed, or other formatting…

Aug 13 2016 at 10:26am

As a software developer and economics graduate, I found this podcast incredibly interesting.

One thing that I’ve always questioned was whether or not the competition for discretionary time that Adam alluded to, i.e. Facebook vs Netflix in terms of video consumption, was enough to incentivize companies to work toward at least implicitly maximizing social welfare in a way that tends toward what traditional markets such as canned bean or steel processing companies could achieve in earlier times. At least in traditional competitive markets there was a clear delineation in terms of what different industries actually were. Today it seems that these boundaries change month by month. Furthermore, network effects are supposedly more powerful today than they were in the past (otherwise we would never see companies such as Facebook or Twitter grow as fast as they did), so potential market entrants are less able to compete in the same niche even if they wanted to. This concerns me from an economic welfare perspective.

As a personal example, when deciding to use the messaging application Telegram as opposed WhatsApp I had to contend with a big trade-off, which was whether the value I placed on privacy versus the value I placed on connectedness to my friends was worth not having anybody to talk to (at least this was true about a year ago; WhatsApp has since improved their encryption technology but not as much as I had hoped, and it’s still closed-source (see: vs. Telegram’s open-source code model) . It appears that the vast majority of users prefer WhatsApp, but it’s not particularly clear that they do so for reasons that do not transcend personal privacy considerations. For example, WhatsApp’s user interface and experience are essentially identical to Telegram’s, but Telegram has a much more sophisticated personal privacy architecture (see:, i.e. end-to-end encryption for both one-to-one chats and large groups, as well as a built-in scrambler for protecting sensitive geolocation information, and much, much more. Who wouldn’t juice the juicy as opposed to just the average apple in the grocery store if they really were the same nominal price? The problem is signaling the value of encryption to a market that doesn’t have many choices, and in traditional markets, these sort of things seem to work their way out implicitly.

These are definitely complicated ideas to market to the public but perhaps more people would be better off and also realize that they actually were better off if there more alternatives to choose from, which sadly is not the case. Do people really not care about their privacy? Sometimes I wonder if perhaps they do not, but I don’t think that’s the most important question to ask here. I think they are generally concerned, but people make these complex choices implicitly rather than explicitly, so with more choices they wouldn’t have to explicitly contend with the increasing returns to scale that a service such as WhatsApp has amassed due to early advantages in generating preferential attachment between groups of users. Instead, they would simply choose another service.

Anyway, the core of this issue for me was that there was a strong transaction cost involved in either convincing a large enough group of my friends to switch over to Telegram in order to make it worthwhile to not abandon the service, or otherwise in convincing myself that switching over to WhatsApp was the better choice (a kind of forced cognitive dissonance that made me uncomfortable). By contrast, if I want vegetarian food at a traditionally meat-centric Bavarian restaurant, I can always order a salad, even if isn’t quite up to my personal standard. In the chat messaging example, I am either prohibited from ordering my salad entirely, or must begrudgingly wait for the platform to lift said salad prohibition, which is entirely out of my control, and sometimes just never happens.

It would almost be better if my user data and authentication existed on either a public third-party server that I trusted, much like how I give my financial data to Stripe versus the apps that I purchase things on, or perhaps even my own personal server. That way I could interface with both services as opposed to having to make an explicit choice between each platform, and without the lock-in generated by preferential network effects, there would naturally be more services to compete with one another and, as a consequence to the consumer, more services to choose from. In conclusion, and insofar informal my analysis so far has been, I don’t think D’Angelo made a strong enough case against regulation for industries that are subject to such rampant network effects. Frankly, he and other Silicon Valley CEOS have an incentive to not spend much time on that part of the discussion because it’s possible that they are very, very wrong. The best case scenario, I believe, would involve something more like a publicly-entrusted application programming interface (API) with which most web or mobile application could connect, something like Facebook’s SSO (Single Sign On), but something that doesn’t rest in the hands of only a few firms. Why does everybody only “Log in with Facebook or Google+?”. Twitter and LinkedIn compose a kind of very distant third place, but there are virtually no alternatives beyond these four SSOs.

On a tangential and more positive note, closer to the the end, Adam mentions he has no reason to leave Quora or to work on anything else. The platform certainly provides exceptional value in terms of generating relevant answers to questions that people may have. But the true power of the Internet is only now revealing itself in the form of video consumption, and this will definitely explode as virtual reality becomes more mainstream. It might be the case that people not only enjoy asking questions from experts in the form of text but would also like to listen to answers in the form of pre-recorded or live streaming (virtual) face-to-face sessions with trusted individuals from the comfort of their own living rooms. Quora certainly has a long way to go! I’m excited for the future.

Aug 15 2016 at 8:31am

Interesting podcast.

My favorite part was, immediately after a brief discussion on social signalling, the guest told us that he has Hillary Clinton and Barack Obama answering questions.

Aug 15 2016 at 11:40am


Encrypted communications are only a small part of the problem, my main concern is app permissions and tracking. (ATT is my internet provider and they just sent out their updated “privacy” policy and I am one of the 0.001% who read it. Basically, they give themselves the right to collect and store EVERY web page that you visit, read all of your web based email, build personal profiles and monetize it however they wish. Some privacy.)

How are WhatsApp and Telegram different in permissions and tracking privacy? Do you have any suggestions for countering this?

Aug 18 2016 at 11:35am


That’s very interesting. I would consider myself fairly technically sophisticated when it comes to computers but it’s true that most times I also fall into the 0.999% camp with respect to various online privacy policies, because this is behavioral and so naturally I too am subject to various biases when impatient (or under the probably correct assumption that regardless of the possible alternatives, most policies will be disadvantageous to me anyway). On the other hand, there are VPNs and Tor for that. It’s a shame that even using them sends a signal to ISPs that those users are suspicious – can you use them a problem under ATT?. I’ve heard that most VPN users are flagged as high risk regardless of how they use them.

The problem is especially bad in Canada. We have three viable telecomms to choose from. Each is expensive relative to international markets, and intrusive in terms of personal privacy.

You might find this article interesting:

“Imagine letting your spouse know you are pregnant over WhatsApp, only to find that the next time you log in to Facebook you are presented with advertisements for baby furniture, diapers and college savings plans.”

I don’t know much about the specific differences in permissions and tracking privacy between the two platforms (my concern is largely with encryption anonymity partly because I’m a bit of blanket technophobe when it comes to behaviorally explicit technology versus traditional productivity-enhancing technology) but I am fairly confident Facebook shares most of what you type into WhatsApp across all of its platforms, including Instagram. If you are concerned but unsure, just activate your evolutionary ingrained risk-aversion bias superpower and download Telegram!


Caveat lector: anecdotes incoming.

Proponents of these markets draw on three major arguments:

1) Competition over Discretionary Time
2) Fragility at Scale
3) Public Returns on Private Research

1) Competition over Discretionary Time – Is competition over discretionary time relevant at all? This is not a unique proposition. The use of my free time after work is sought after by fitness centers, movie theaters, restaurants, bars, video games, hanging out with friends online, tweeting, Facebooking, Skypeing, white water rapids kayaking, jogging, travel blogging, playing Rugby, competing in the Olympics for the New Zealand All Blacks, writing a biography on Mother Theresa, managing a manned space mission to Mars, running the United Nations from my office as Secretary General (my name is actually Ban Ki-moon), solving the Riemann hypothesis, etc. In no way are any of one of these firms and activities capable of monopolizing my time compared to the alternatives, and the available alternatives are replete with diverse options that meaningfully differentiate themselves from competitors. This is an outright embarrassing example. I can watch videos on Youtube, Netflix, Hulu+, Facebook, Vine, Instagram, Twitter, HBO Go, Amazon Prime, etc. This is not disputed. The monopoly, however, is not over the content or time-use of the product per se, but rather over the social and behavioral implications of the restricted nature of my interactions on the platform due to network lock-in. I can’t meaningfully use an alternative to Facebook in a way that is essentially the same as Facebook but NOT Facebook. Even when I complain about and criticize Facebook, my best possible audience to receive such a message is on Facebook itself. Ironic? In fact, the largest anti-Facebook page on the Internet is a “group” on Facebook (just type [four-letter expletive] Facebook” into Google and follow the link of the first search result. This is true for other platforms, but not nearly as all-encompassing as it is for Facebook in particular. This argument is identical to saying “We’re a monopoly. We’re not a monopoly. Please ignore the previous sentence.”

2) Fragility at Scale – Does the hypothesized increased fragility of these networks due to the supposed abrupt nature of social change really outweigh the magnitude of losses to consumer surplus compared to alternative possible versions of history during that same time period? Do networks of an increasing returns nature decline as quickly as they grow? Sure, Facebook may not be the hot thing to use in a few years time, but they’ve already made headway into virtual reality with Oculus Rift. Would they experience market advantages in that industry if they were not Facebook?

Does it matter to Facebook if Facebook’s web and mobile applications go out of fashion any time soon? The possible duration of the monopoly of their present services are not distinct from the possible duration of the monopoly of their future services. Facebook still owns and maintains the database of social connections that make up Facebook. What is stopping them from porting these connections to new platforms? Very little. By contrast, when the oil industry dries up, oil companies cannot simply port their existing infrastructure to provide for alternative energy services, not without serious hardware modifications and investments in personnel retraining. The marginal cost of converting a Facebook user to an Oculus Rift user is infinitesimally small compared to converting the end-consumer of an oil exploration company to one who consumes solar power from the same company. Even when the Facebook brand goes out of style, the behavioral profiles they’ve generated from their monopolistically-acquired user-base will retain their value. This is a major distinction.

3) Public Returns on Private Research – Facebook does in fact give back quite a bit in terms of various open-source projects that make life a lot easier for hundreds of thousands of programmers worldwide. React.js is freakin’ awesome. Then again, these projects appear largely in the context of licensed Apache open-source libraries that serve to enhance their existing developer ecosystems, and hence expand their implicit market impact. Twitter developed Bootstrap, which saves millions of web designers hundreds of hours per person per year. I use it myself in nearly every project. I include a license attributable to Twitter every time I boot up a new template in Sublime (I suppose I can get away with not doing so but I’m nice, I think); this not only enhances their public image, but attracts talented developers to their premises. Okay, so these are examples of fairly good deals, but what they’re not sharing is more interesting. This reminds me of that old proverb, “Blind them with silver, for they will not see your gold.”

Still, all this eventually funnels its way back into the companies that support these open-source platforms. They don’t do this out of benevolence, and no firm should feel compelled to do anything out of pure altruism (that’s not the point), but in these cases, the leverage lies squarely with the platform providers as opposed to the users (who are part of the product and not customers) so I’m not convinced that Kaldor-Hicks efficiency is unexploited here in the sense that platform users are not better off than they could be under a more competitive regime. Again, even if the market benefits, what would a possible alternative history look like if instead of Android and the Google PlayStore, for example, another platform rose to prominence in its place? Would the personal privacy policies of his hypothetical alternative differ dramatically as a function of the personality of its founder; would it differ at all? These are questions of historical contingency that are occluded by the impossibility of reversing time. Monopolies generate relative non-ergodicity in the marketplace that makes it hard to judge the comparative effectiveness of their market proposals because there are virtually zero experiments from which to produce a reasonable baseline of comparison. If Facebook did not rise to prominence at the rate that it did, would there have instead been 10 different Facebooks? We will NEVER know. Would VKontake or Orkut (now defunct) even exist as Facebook alternatives without government intervention on behalf of Russia, Brazil, and China? Unlikely.

What’s interesting is that arguably the most important things that large social networking applications should open-source on the basis of their monopolies’ magnitudinous impact on public life stems their their news feed algorithms, because the size (and hence rich social connections) of their databases grants them the unique ability to alter the emotions ( and political orientations ( of millions of users, as just two examples among hundreds, and these are shrouded in total mystery. But how would revealing the nature of these algorithms seriously negatively impact their businesses if their front-facing products were, as they claim, superior to all possible market alternatives? I think they realize that if anybody had access to these algorithms, it would essentially be a crap shoot as to which platforms became popular… and given that these algorithms cannot be produced without databases the scale of which physically requires being Facebook or Twitter or Instagram itself, can only emerge monopolistically.

My hypothesis about these markets, especially for emerging social application technologies, is that they are not competitive on average. Most likely, we have yet to properly identify what succeeds in making such markets not competitive. In the absence of evidence, unfortunately, it is leverage and power that in practice determines what is true.

Yegor Seleznev
Aug 25 2016 at 11:50pm

I discovered EconTalk only recently, and I love it. It is an easy and entertaining way for me to learn important things. But this particular episode, the Quora episode, is disappointing. Adam d’Angelo is not in a league with regular EconTalk interviewees, and Prof. Roberts seems to be too accustomed to complimentary interviews. Early on, Roberts remarked that asking a politician to answer a question is probably not the best approach. But he dropped the issue, possibly because, being a guest of Quora, he developed a guest complex. Mr. d’Angelo should know the difference between political propaganda (which the answers of Hillary Clinton are) and knowledge. In general, the site is biased toward the left, moderate left. This may reflect the audience but this has little to do with digging out and disseminating knowledge. Personally I find the site boring.

Comments are closed.


EconTalk Extra, conversation starters for this podcast episode:

This week's guest:

This week's focus:

Additional ideas and people mentioned in this podcast episode:

A few more readings and background resources:

A few more EconTalk podcast episodes:



Podcast Episode Highlights
0:33Intro. [Recording date: July 19, 2016.] Russ: I'm recording this episode at Quora's headquarters in California, high above Castro St., but not that high; so we may hear the occasional honk or truck sound in the background. Adam welcome to EconTalk. Guest: Thanks for having me. Russ: So, what is Quora, for our listeners who haven't been there? Guest: So, Quora is a knowledge-sharing platform. And we basically want to connect people who have knowledge with other people who need it. And the product takes the format of questions and answers. So, anyone can come and ask a question, and then we try to show those questions to people who are going to be especially qualified to answer them; and then those people can write answers. And over time we try to build up this big data base of high-quality answers to questions that can be useful to everyone. Russ: How did you get here? How did your career path end up in this? I think it's about 6 years old now. Guest: Yes. Let's see. So, I started out--I studied computer science in college, but I was also interested in social science and economics. And after college I went to Facebook-- Russ: That's a website, I think, that also--yeah, go ahead. Guest: I've heard of them. Yup. So, I think Facebook is a very interesting application of both computer science, but also social science and some of the theory about signaling and how users behave in these large-scale social products. Russ: Because there's a weird online community. We talked beforehand about signaling; and I don't usually think of social media as being a signaling phenomenon. For me, it's mainly a place where I can get information, or try to spread information about, say, EconTalk. But I probably did some signaling with realizing it. Why did you mention that? Guest: I think a lot of what motivates people to participate in these social networks, especially things like Twitter, Facebook, Instagram, Snapchat is a desire to signal things about themselves--information that's going to be useful to other people. So, it's one of the first chances that we've had to really apply signaling theory to product design. And so I studied--one summer in college I studied, there was a group at MIT (Massachusetts Institute of Technology) called the Sociable Media Group, doing some research on that, that really opened my eyes to doing some of that. So, Facebook was a great chance to apply it. And then after leaving Facebook I looked around, sort of decided what I wanted to do; and I'd always been interested in knowledge sharing and questions and answers and this idea that there has been--there is a huge amount of knowledge that is out in people's heads, that's not on the Internet; it's not very easy to get access to. There's a huge amount of knowledge that I think is locked in research papers where you need to know a certain amount of very domain-specific language to even navigate, to figure out what you want to get access to. And so I thought that there was just an opportunity to really build a platform that would get all this knowledge out to the world and make something that could last for a really long time. Russ: So, some people would say that's what the Internet is. It's just a knowledge-sharing thing. So, right now, if I wanted to solve a problem I had or figure out something I didn't know the answer to, I would just enter it into Google; and there are other sites--Yahoo Answers--that lets people answer stuff. How does Quora differ from those, and what's the importance of that difference? Guest: So, compared to the Internet in general--one of the problems on the Internet--so, if you want to get knowledge that's already on the Internet, there's a lot of good ways to do that. Google does a very good job of indexing all that information and making it available to everyone. The challenge we're interested in is: How do you get more information onto the Internet in the first place? And so there have been some other attempts to do this, other question and answer products-- Russ: Jeeves. AskJeeves. Guest: Yup.Russ: Is that still around? Guest: It's still around. It's actually more of a search engine than a question and answer platform. So, there's been all these other attempts to make this work. But it turns out to be really challenging to run one of these systems at scale and not have the quality just degrade to this kind of, you know, mess where there's nothing really good going on and there's no experts participating. Russ: Right: I might get a hundred answers to my question and I'm going to have to read all of them to figure out anything. Guest: Yep. Russ: Which would be a disaster. So, how do you try to avoid that? Guest: So, we're able to have things like--we've had Hillary Clinton answering some questions; we've had Obama answering questions. We were able to set up an environment where experts want to participate. And there's a lot of different pieces that go into the puzzle. One of the first things that we did--one of the ways that we differentiated early on was that we required everyone to use their real names. And real names are really important because they mean--a real name means that an expert or a someone with sort of real-world-- Russ: Some reputation-- Guest: real-world reputation-- Russ: Even if they don't always tell the truth. So, I'm not sure politicians are your best example of experts. But go ahead. Guest: Yep. So, it lets someone with a real-world reputation bring that reputation into the platform and start out with this credibility. And so, these other platforms at the time that we started didn't really use real names. And so that just put them at a disadvantage. And it's very discouraging to an expert to have to start from zero in an online system. So, that was one piece of the puzzle. But since then, one of the most important things for us has been personalization. So, we've invested very heavily in getting the right answers to the right people, and getting the questions to the people who are going to be able to write really good answers. And that's something that we've gotten better and better at over time and we're still continuing to invest and get better at it. So, ideally you have this environment where people have seen the content--the questions and the answers that are the right questions and answers for them--and that's usually a totally different experience than what other users are having.
7:18Russ: So, sometimes you go to the Internet--you might to Quora with a particular question because you really want an answer: you've got a pressing question or something you're just really curious about. But sometimes you just go because, 'I'm bored. I'm looking for some interesting content; this might be fun.' Do you have any perception of the mix of users in those two camps? Guest: It's mixed. I'd say maybe half and half. There are a lot of people who have important questions every day that are trying to get an answer to that. And then there's a lot of people who just--they're bored, maybe they are really are digesting [?]; they get provoked by a particular question that they are curious about and then they'll go and read that. And I almost think of the reading behavior as your mind is rewarding you for getting some information that might be helpful to you later. So you get this sort of short-term payoff that it feels good to learn new things and to express your curiosity. Russ: Speaking of signaling: so, before this interview I thought maybe I should--not 'maybe I should'--I went on the site, right, which I've been on before; but I thought to refresh my memory, make sure--I'll explore some things; maybe I'll get some ideas for questions. And then I thought--gee, should I ask a question? And then I realized: 'Do I really want people to know I'm asking that question?' Right? Because that's one thing--one of the questions I ended up asking--I didn't ask this question; I searched on it--was: What should I ask Adam D'Angelo? Which I thought was really unclever--not a bad idea--but obviously the personalization, the non-anonymity changes people's willingness to participate in the site in certain ways, right? Guest: Yeah. So, we actually have a feature where you can ask an anonymous question, if that's important. But the vast majority of users don't ask questions. It turns out that almost all questions that people have are questions that someone else has had in the past. Russ: Nothing new under the sun. Guest: Yeah. And that's actually one of our strategies, that we don't want to have 20 versions of each question. We want to have one version; and that allows us to concentrate all the answers' energy into that one place. And then that helps to generate the best possible answers. Because people know that there's this one place where they are going to answer the question that's going to be kind of definitive for a long time. Russ: So, that's a big challenge. For example, I really like photography, and I'll be on a photography website like DPReview (Digital Photography Review), which is my favorite. And in the forum someone will ask a question that's been asked 273 times. But this [?] person who has got in, just joined the forum, was lazy, didn't search, or they asked the question a slightly different way and they didn't find it. And everyone jumps on them--and sometimes they are kind and they'll put a link that's been asked 40 times: 'Here's one of the better ones.' But obviously one of the challenges is: how do you figure out what question is like another question. So, how do you solve that? Give me, if you can, if you're comfortable, how many questions get asked a day, roughly? I mean, what kind of volume are we talking about here? Guest: So, we don't share numbers on the volume of questions. Russ: But the answer is: A lot. Guest: Yes. It's a lot. Russ: Even though most people don't ask, doesn't matter; I'm sure there's a lot who ask a lot of questions. Guest: Yeah. It's a lot, and it's growing very quickly, at the rate that the rest of the product is growing. So, if you want a sense of scale, we have about 100 million monthly unique visitors--that was the last number we announced. Russ: That's a big number. Guest: Yeah. It's a lot of people. But most of them are looking at questions that someone else asked. Back to your question about how do you [?]-- Russ: How do you merge, how do you decide to merge a question? Guest: So, there's a few things. One is we have a machine learning algorithm that can look at two questions and determine whether, make a guess at whether those are likely the same question or not. And so, if you are going to ask a question that we think is a duplicate then we'll go and show you that question and say, 'Hey, you probably wanted to look at this question.' Some questions get through that filter, though; and so we have another set of systems that try to merge questions later on. There's an offline machine learning process that takes a little bit longer. Then, there's--users of the product can actually go in and merge two questions. We have a process for how those get reviewed and how we make sure that we control the quality on that. Russ: Cool. Guest: So it's not perfect, but it ends up being a much better experience than, say, on the photography feed forum where there were 20 versions of the same thing.
11:59Russ: So, if I could go back and visit the Adam D'Angelo of 2010--you had some kind of vision. How has it turned out? Something like you imagined? Nothing like it? Guest: Pretty much-- Russ: Or just bigger? Guest: Yeah. I mean, it's definitely bigger than I expected. I think everyone in our industry has been surprised over the last few years at just how big all these markets have gotten. One--you know, you have all these people coming onto the Internet for the first time, all these people have phones so they are using the product more. But, there's also another thing I think is a huge factor here is that personalization is a really powerful technology as far as its effect on markets. So, in the offline world, let's say for magazines, everyone reading a magazine has to see the same magazine and the same order of the articles. Russ: A little bit--they might put a different cover on the West Coast versus the East Coast. Guest: Yeah-- Russ: That was a big breakthrough. Guest: Right. There might be 5 variants or something like that. There might be 10, 20. But still, they are the same stories in the magazine; and you have this limited amount of physical space to hold them in. But with the Internet, you can have, sort of an infinite amount of content; and you can have every user getting a totally different experience. And so that's meant that markets that used to be fragmented, like, say, the magazine industry, now in the Internet you can just have a very small number of products that are able to reach a much bigger audience. Because they can show the content to the right people. So I think, you can see this with Google. There's just one search engine: everyone uses the one search engine. There's NetFlix, NetFlix instead of all these different TV channels and different shows on different channels and different cable providers: You just basically have NetFlix. And I think personalization is this--it basically has this effect on markets where it makes the market a lot bigger, because a single company can address this very diverse, needs [?] Russ: It's one, but it's providing a million channels instead of--issues of monopoly where they may not have the incentive to customize it. Because that's not really the case. Because they are constantly trying to customize it to maximize their reach and then their revenue, right? Guest: Right. And though they may not have head-to-head competition every day every way you might have had in some previous industries. But instead they have this very strong incentive to just get people to use, get the existing users to use the product more. Which, you know, effectively Netflix is in competition with Facebook even though you think about them as totally different markets. People have this discretionary time that they are going to spend online; and the better Netflix gets, the more time is going to Netflix; and the better Facebook gets, the more time is going to Facebook. Russ: And the same is true for Quora, obviously. Hulu is a competitor even though it is not a question-and-answer site, because it's in the same category of interesting things I can do on my computer or phone. Guest: Yep. Russ: Absolutely. When I have some spare time.
15:12Russ: You don't appear to have any ads on the site. And yet, according to my research outside of Quora you are worth something over probably a billion--your valuation of the company is over a billion dollars. Do you have plans for monetization that you can talk about? Guest: So, we actually do have ads. We are running ads on a small percentage of our pages. It's just a small test right now. So far the results have been very promising. And we're looking at--we're going to scale that up over the years ahead. Our valuation--you know, I'm not going to take a particular stance on what our valuation is. The valuation that investors give us is based on projections of the future and based on the amount of usage that a product has. You can assume that we are going to monetize in a similar way to other similar products per unit of usage. Russ: But do people worry that there's a big difference between an ad-free site and a site that's full of ads, and how that might affect the culture and the ease and the pleasure that people get from visiting the site? Is there a tradeoff there? Guest: Yeah. I mean, we've been very up front with our users that we will run ads in the future. We don't intend to have a product that is full of ads. We think we can make a lot of money, enough to sustain the business, without having to make the user experience a lot worse. I think a good example of this is Google. You do a search; most of the time you don't see any ads. If you happen to search for something that's commercial, then you'll see more ads. Russ: All the time. And I want to say, 'Who do I call to tell them I already bought that? They can stop.' Guest: Yeah. Russ: It's interesting. They'll get better at that, I assume. Guest: Yeah.
17:04Russ: So, the company is 6 years old. You've had a lot of growth, obviously, because you didn't have any users when you started and now you've got 100 million. But, as an outsider there's a temptation to assume that you're bored. You had this great idea; it came to fruition: Big enough, we're done, the site's up and running, it's growing steadily. Is the thrill gone, or is it still fun to come to work every morning? Guest: Yeah. Well, see, it's still really fun, and I think one of the things that really exciting to me is just the scale that we're getting to. We're starting to reach like a big percentage of the United States and the rest of the world every month. That's inspiring sort of on its own, but it also creates a lot of challenges internally. So, the growth basically follows this exponential growth pattern, so that there's a lot more people using the product every week than the week before. And that creates a lot of challenges internally. So we have to make sure that our infrastructure and our technical systems are able to be ready for the load. We have to make sure that our personalization technology can do better and better as we get bigger and bigger. The bigger we get, we also get more data. And so that lets us do a better job in machine learning, in personalizing. It lets us run more experiments. That's something else that I think is a pretty interesting part of what we do. Russ: And we'll talk about that--but I have to warn you now, because you can have this big spike when this EconTalk episode is released. So you're going to want to prepare for that, probably. But, buy an extra server. Guest: Yeah. Well, we're on Amazon Web Services, so luckily we can just turn on some more servers and just run them by the hour. Russ: Oh, whew. Guest: So I get a warning when it is about to be close. Russ: You have to borrow some money. Um, so, in terms of the product itself, you know, as a user, I see questions; I see the answers. Do you do other things besides questions and answers? Is there product variation or innovation beside just the Q&A part of the site? Guest: We've experimented with some formats outside of questions and answers, but over and over we've just seen that the question-and-answer product is just doing really well and getting bigger and bigger. And it takes--it's almost taken our full energy as a company to just keep up with the question-and-answer product and make sure that the quality stays really high. And just bring it--we've brought it to mobile phones; as new technology has come out we have to keep pace with that. And we have to keep our costs under control. So there's a lot to do just for questions and answers. So that's kept us focused. But I imagine in the long term we will branch out into other formats of knowledge besides questions and answers. Russ: So, besides asking questions or reading questions other people have asked and answered, you can sign up for Topics on the site and see Q&A related to a particular topic. Do you, Adam D'Angelo, do you have topics that you subscribe to yourself, personally? Guest: Yeah. Yeah. I follow a lot of topics. Some of the ones I'm particularly in are related to machine learning. So that's just a personal interest of mine. But also it's super-applicable to what we are doing every day. Russ: It reminds me--I've told this story before, but one of my favorite management stories, Sam Walton supposedly used to fly on a private plane--he was the founder of Wal-Mart. And he'd see a WalMart truck on the road, land in a cornfield--I don't know if it was literally a corn field, but land somewhere ahead of the truck--and then hitchhike--stick his thumb out, get a ride with the trucker into town and chat with the trucker and then go into the actual store. You know, it's called manage by walking around, among other things. But I always like this image of this trucker driving along, trying to stay awake, and seeing the Founder and CEO (Chief Executive Officer) of his company there on the side of the road with his thumb out. And trying not to have a heart attack. But that's the standard way, in brick-and-mortar businesses, are renowned for getting information. They get their hands dirty; they get in the trenches. The kind of [?] of the CEO is up in the executive suite, never really tries the product and just relies on, say, the marketing team, etc. So, you--you've got a ton of information about how the site's doing. But you also have the one data point of you as user. Does that affect you at all? Does that ever lead into the decisions you make? Do you ever, like, slam your fist down on the desk saying, 'Why am I getting these bad answers?' or 'bad questions?' or whatever? Or is it all just driven by the data? Guest: I'd say most of it goes on the data. But I definitely use my own experience. And one thing I try to do that you might find interesting: I actually try to behave more like what I think a normal user behaves-- Russ: Sure-- Guest: so that my own experience is not too distorted from that. So you could say that, as the CEO maybe I would just artificially kind of like force myself to write tons of answers and follow lots of topics and give the system lots of information about me so that it could do a really good job. But then I would have this experience that was nothing like a normal user. Russ: Correct. Guest: So I actually go out of my way with using Quora--and other Internet products--I just try to like act like a normal user so that the information I get isn't so biased. But yeah, you know, in terms of it affecting my--it's good for generating theories and ideas. Usually I'd like to test those theories and ideas on some, ideally on a controlled experiment. But also we look at surveys of our users; we look at things users are reporting. We look at--we have all kinds of sources of information. So I try to make sure that the ideas I generate from my own usage are consistent with the data. Because if you don't do that, you can really just go off in a direction that's crazy. Russ: Destroy your side, effectively. That would be bad.
23:16Russ: Let's talk about personalization, because--I'm fascinated by how this happens in the web in the different sites that I go to. So, in the case of a site like Twitter, I personalize it because I choose who to follow. That's a site I happen to know, somewhat. If I'm on your site and I enter a topic--machine learning, economics, meditation, Boston Red Sox--am I going to get the same stream as other users who have entered those topics? Guest: So, the way it works is we have a lot of different sources of input. We have the topics that you've followed. We have the people who you are following. We have all your behavior from what you've read in the past and questions that you've answered in the past, questions you've followed. And we have an algorithm that takes all that information as input and then uses that to try to predict what's going to be interesting to you and questions you are going to want to answer. And so it ends up in practice being very different for different people even if they are following the same topic. It will depend on where you live, and-- Russ: So, it's that personalized. Guest: Yeah. It's actually--it's very, very personalized. And different users have very different experiences on our product. Russ: But as you mentioned when we talked before we started, a lot of the personalization is on the answer side. So, talk about that. Guest: In terms of showing what answers? Russ: Correct. Guest: So, a lot of what we do--a lot of Quora usage is--we actually, when we started the service, our first version of it, we expected that people would come to Quora when they wanted to write an answer. And those would be the two big use cases. But, we designed the product so that people could read other people's answers. And that very quickly became--we noticed that that was just something people loved to do. Russ: Sure. Guest: And so that has emerged as one of the big use cases for the product. So, we use a lot of different signals to figure out how to personalize what answers you see. Because we have millions of answers. Any given user is going to see a very small subset of--maybe thousands of answers out of the millions. Russ: But you are also going to determine which people will get some questions--so, I've been asked questions. I'm not an expert, but I play one on the Internet, obviously. I might get a question about, say, Keynesianism or Hayek or whatever. So, what determines whether I get those questions as a so-called expert? Guest: So, there's all kinds of factors. There's what answers you've written in the past. There's what--the answers that you wrote in the past, were they on the same topic that the new question is on? If so, that's probably a positive signal. If you previously wrote an answer about economics, then the next time there's an economics question that's probably something you're going to want to answer and be good at answering than a question on some other topic. So, that's one factor. There's other things, like if we feel like a question already got a great answer from someone else, then we might not show it to you, because we want to spend your energy on filling out--ultimately our mission is sharing all the world's knowledge, and so generating a lot of duplicate answers is not good for that. So we want to send the questions to the people who are going to answer them best. Russ: So, viewers rate the answers also, correct? Guest: Yeah. Russ: So there's sort of two levels of rating: there's the viewers' rating but you also have inside information about how long people stayed on an answer and that kind of thing. Do they both matter in terms of determining who gets to see what, when? Guest: Yeah, they are all useful. The votes are a very important signal, especially not just like the number of votes but who was the voter. So, did you get a vote from someone who we know really has authority on some particular topic? That's an important signal to us.
27:33Russ: So, one of my favorite books is The Professor and the Madman, by Simon Winchester. It's about a lot of things, but one of the things it's nominally about is the creation of the Oxford English Dictionary (OED). And you learn early on that there's a handful of people, in particular a particular person who becomes a focal point of the book who discovers a large number of the first usages of words in the English language. Because one of the things that the OED did is one of the first crowd-sourcing information projects. Basically the Editor went out and said, 'Hey, if you know where these words come from, if you saw an earlier version of this, we'd love to have it.' So people would send in postcards with the book and where they saw it. They couldn't copy the page, Xerox the page, so they'd send a page number. Do you have answerers who are, like, crazy common--who are answering an enormous number of questions, that everybody loves, that you literally know as opposed to data knowing, as opposed to the machine and the algorithm knowing? Do you have individuals that you know are just like superbly good at this? And how many are there? Are there only 10, or are there a 1000? Guest: Yeah. There's thousands. But as far as--one person that comes to mind, there's a professor at Berkeley, Richard Muller, who has written a large number of answers about physics and about lots of other topics; but he was one of the top answerers at one point. Russ: Did you do anything to keep him happy? Or just is the thrill? Guest: Hopefully we built a good product and that's going to keep him happy. We have an event once a year where the top writers can come to meet us at the headquarters here. Russ: That's cool. How many people come? More than one? Guest: Yeah, a few hundred. Russ: That's fun. What do they do? Guest: They talk to each other. They get to talk to me, ask questions. It's mostly I think just fun for them. Russ: Sure. I think that's really cool. Guest: This, though, is not really the motivation for them. Russ: No, I understand. It's just a fun--it makes them--one of the challenges that I have with this program is creating a community. Because I've got a community. Right? I've got tens of thousands of listeners who feel a connection to me every Monday morning. And I have some of a connection to them, but we never see each other, unless I do a live event. Which I do every once in a while. I struggle with ways to have online experiences, or face to face, that would give people, that would make it more fun just to be part of this experience. And what you're doing is a great idea. Right? That's one way to do it. I could at least have my guests come to a party. There's roughly 50 guests a year. I don't know how many would be interested in meeting and hanging out with the other guests, but that's an interesting thing. But the listeners--creating a forum for the listeners, either in real space or in virtual space, seems like a good idea. Guest: Yeah. I would probably start with virtual space. Russ: Yeah. I haven't figured out how to do that very well yet. Guest: With skill it's a lot better. Russ: Yeah. Maybe we can talk about that later.
30:45Russ: Any big mistakes you want to share? Regrets? Things you can't believe you did, now that you look back on it? Or just surprises? Things you can't believe that didn't work out and you had to totally change something? Guest: I think in the early days of the company--this was actually before we started doing a lot of controlled experiments--we added a lot of features to the product. And it got really complicated. Russ: A common problem for startups, new products. Guest: Yeah. The complexity tax just has hurt us over and over. And it just slowed us down a lot. You add the feature, and a little increased usage in the near term; but it just means that every feature you are going to add has to interact with that feature. And you are going to have to control for the effect of that feature in some future experiment; and you are going to have to go measure the impact of that feature; and you have know when that feature breaks and fix bugs in it; and you have to explain to users how to use it. And then you get to this point where you have so many features and people inside the company don't even know all of them. It's a huge, huge burden. So that's probably the main thing, when I look back on our product [?] over the last few years. Or the last 6 years, let's say. That's stuff like something that we paid a very heavy price for. Russ: So, the site is more streamlined than it once was? Guest: Yeah, and we're also--it's one thing that's interesting about complexity--it's much easier to prevent the addition of another feature that's going to make things more complex than it is to remove an exiting feature. Because one thing that's very important to us is, we never want to remove anyone's knowledge that they've shared, for example. That would be an absolute No for us. That's very important to our integrity, that we maintain everything everyone's shared forever. And so that means that it's hard to get rid of any feature where people have contributed some information.
32:56Russ: So, you mentioned controlled experiments. Tell me what proportion of your time here is doing those kind of experiments. How much effort is it? Is it one a day, one a week, one a month? And how do you do them? And what's the nature of them? Because usually you don't think of, people often don't think of--what do we mean, 'controlled experiments'? Trying to figure out if the answers are right? But that's not what we're talking about. Guest: Yep. So, one thing we have is what we call an experiment framework. And this is basically a set of software that makes it very easy for us to run controlled experiments. We made this big investment maybe 4 years ago to create this experiment framework. I think it usually companies don't do this until they get much bigger than we do, but we're very serious about the science of this, here. And so we built this experiment framework. And since then--we ran some experiments before the framework, but if you are just counting experiments that have happened since the framework, I think we've run about 2000 experiments. Russ: That's a lot. Guest: And at any given time we have about 30 experiments running concurrently. And we--let's see. So the whole company is about 130 people. There are 13 Data Scientists. And so their job is to analyze data, run the experiments, analyze the experiments--sort of like uphold our integrity in terms of scientific process. And so they are generally looking at data, analyzing data, determining what's true. And then there's a whole other set of employees--most of the company--that's creating the changes that go into the experiments, to them, to be tested. Russ: So, that's shocking--10% of your staff are Data Scientists. Which is not normally what--at least this outsider--would have thought. In my mind, 'Hey, you've got to make the website work; you've got to have some web designers; you've to make it look nice; you've got to make it work correctly. You've got some algorithms running in the background.' But the idea that you are constantly trying to improve the site is--surprising. Like, isn't it good enough? What's wrong with it? It works. I get information. How much better could it be? Guest: Yeah. So, let's see. So, I think maybe one intuition that can help you understand this is that there's sort of a finite surface area that users can consume. So we've built the product and the surface area of the product can't really expand that much. And so at some point you don't want to keep adding surface area: you want to make the existing surface area better. And it's actually very hard to do that without running experiments. People think that they have an idea for something that's going to be an improvement-- Russ: Can you give me an example? I'm not sure what you even mean by surface area. Do you mean the physical size of what I can access with my eyes on the page? Guest: Let's say if you were going to write down in words, if you were going to write down a description of everything that can happen on the product--all the features and how you use them and where they are and how you get to each particular thing you might want to do-- Russ: That's a long list-- Guest: The length of that document you could think of as the surface area of the product. Russ: Okay. Guest: So, you want to--so, we want to make the existing product better as opposed to keep adding surface area. It's maybe not true at the very beginning. At the beginning of a product, it's a totally different experience, evolving a product. Russ: Right. Guest: A new product. Because you have no data, you have no users; you are going entirely on intuition. Russ: Seat of your pants. Guest: Yep. So, anyway, so as we get more and more users, we can then start to use data and we can then start to run experiments. And so we have lots of people coming up with changes. A big category of work is in personalization. So, getting better at showing you the answers that you are going to think are really good and really interesting. And then getting better at showing you questions that you are going to want to answer. Russ: So, you just mentioned--you survey users. But in general, you are not answering that question by asking me. You are going to look at my behavior on the site. Which is going to be answered by--what? Guest: So, if, for example, do you write more answers, do you write better answers? And we have a way of assessing whether we think answers are better or worse. But we look at those two things, as we run an experiment. And that's usually a better, a more reliable indicator of whether we made the product better than a survey. But surveys are also useful. Russ: Yeah. Guest: Because surveys will pick up on things that the metrics can't-- Russ: Qualitative, 'I was annoyed'. But usually if I'm annoyed, I won't come back. You'll see that, right? Guest: Yup. But also--maybe you'll come back more in the short term. Because you--let's say, your objective was to recruit people for your company; and you decide that you are going to use Quora to do that. If we make it very hard for you to find questions to answer you might actually spend more time looking for answers. You might have to write more answers to achieve the same goal. Russ: Yeah. Guest: So it's important to have some qualitative feedback to balance the quantitative. Russ: Sure. Guest: So, the qualitative is also important to understand why something is happening. So you might, well, [?] a change and suddenly people are not using the product as much as they used to. And we don't know why. And it could be something like, 'Oh, well, if you were on a mobile phone with a bad connection then it just didn't work at all.' Russ: Right. Not the content, nothing to do with-- Guest: Right, for example. And then it could be it be something where in some other country there's a filter that the government has installed and we're triggering that with some change that we made, and we never would have detected that. There's things you can never anticipate.
39:14Russ: So, when you have an experiment, you try something--you are going to add a feature, change the color of the logo, whatever it is--do you usually have a hypothesis about what's going to happen? Or not? Guest: Yeah. Russ: You do. Guest: So, we didn't used to have this, but we actually decided to enforce the discipline that every experiment ahead of time, it either has to, either declare a hypothesis, or you have to declare that there is explicitly no hypothesis and you are just doing it to learn and see what happens. And, if you have a hypothesis and the hypothesis turns out to be true, then we'll generally launch the feature. If you have what we call a learning experiment, and you pick up on some--some, some--consequence of that experiment, and you think that that would be a good thing to launch, we require you to run another experiment, where that's the hypothesis. So--because otherwise you run into, you know, statistical significance problems. Russ: Sure. So, how often do you run an experiment that fails--in the sense of, you thought it would turn out this way and it didn't? Guest: It's most of the time. Russ: It's depressing. Guest: It's maybe a third of-- Russ: or not, on their own[?] Well-- Guest: Yeah. Well, I hear numbers from other companies that are actually more negative. So, it's hard to say. It's not depressing. I mean, it's definitely depressing if-- Russ: It's fascinating actually, more than anything else. Guest: Yeah. I can tell you, if you are coming in to work in Quora and it's the first Internet company you've worked in with experiments, it's depressing. Because people-- Russ: They are all excited. Guest: It's just counterintuitive. Everything that you've done in your life before usually has worked. And there's-- Russ: Or at least you thought it did. Now you are going to test it. Guest: Exactly. Right. And I think a lot of other companies are just operating on this mindset where they are just in, like, dreamland, where they think that all these things they are doing are working. But the real reason that the product is working is some kind of network effect, or-- Russ: Time is bad. It's like: I have more listeners than I had 5 years ago. It doesn't mean I'm a better interviewer, better whatever. It could just be more people have access to podcasts. Right? Guest: Right. So there's all kinds of factors that kind of distort people's assumptions about how well they are doing. And there's also those biases like wishful thinking. People want to think that they are good at it, and it's hard to get this evidence that actually most of the time it didn't work. Especially after you've--and just to make a point about how extreme this can get, especially in some organizations, if you worked on this effort for 6 months before running this experiment, and then you need to find out that all the last 6 months of what you did didn't do anything-- Russ: Yeah, it's no fun. Guest: It feels bad. But in some organizations, there could be consequences--that could mean that you are fired. Russ: Yeah, yeah, for sure. Guest: Or that you are going to lose a lot of responsibility. And so there's all these forces that kind of push people toward this irrational or not-objective interpretation of results. Russ: So, a lot of them fail. But some of them don't. How important is that? My first thought would be, 'Well, come on. After 6 years you kind of know most of the things you are going to know.' Now, it's true your listener base--your viewer base--is bigger. It's different. It's more diverse and maybe it's just bigger. How much more is there to learn? Guest: There's tons more. There's--you know, the user base gets bigger but we also get more information about even the existing users. There's--technology is advancing all the time. There's new platforms coming along that we want to support. There's--one of the biggest forces is that machine learning techniques are just getting better and better. And so we want to try out some of these new techniques and see whether they really--someone will publish a research paper and it looks great, where you have to get it into the actual production code where you have to engineer it, it turns out that there are some things that make it not actually work as well as a much simpler technique that was able to be implemented better. So, there's all kinds of things. And then, the other thing that helps drive things forward and make it faster and faster is: So, as we hire more people--so we are 130 people now--we're able to take on sort of more challenging projects that we wouldn't have done when we were, say, 20 people. And so we have a big fraction of the engineering team, all they work on is tools and infrastructure to make the rest of the engineering team more efficient. And so, challenges that would have been too hard to take on before get to be possible and feasible because of the progress--we call this the Platform Team--the progress that they make on blocks, efforts that then other teams can take on. And so you have this really multiplicative effect within the product development team, where you can now take on things that you just couldn't have considered earlier on. So that kind of--we don't see a lot of, like, diminishing returns because of those factors. Russ: 44:39 So, what are some results that you found from experiments that did work, that surprised you about how important they are? Guest: One thing is just the important of speed--of making the product fast for users. It makes sense that a faster product, people will use it more. But just how exactly important that is--how much, down to milliseconds, it will cause an increase in usage. And when you average these things out over millions of users, you can really pick up on these very small differences. Russ: I mean, that's a shocking thing. It shouldn't be, to an economist, because we say, well, lower the price a penny, people are going to want to buy it more. But well, yeah, something more; but not everybody is going to respond to it. Surely a millisecond couldn't make a difference in whether people stay on the site or not. But it does. Guest: Yeah. Milliseconds, I'd say. But I think that--one way to think about it is--so, transaction costs are a huge issue for us. Any friction anywhere is going to greatly decrease the amount of usage something has. And, um-- Russ: I think I should cut out my introduction to EconTalk. Is that a transaction cost? Maybe we should just get to it? Of course, some people solve this in a--which shocks me, again, because of my age--but a lot of people I know listen to EconTalk at double speed, one-and-a-half. Which is--maybe that's why when people meet me they say, 'You sound different.' It's because yeah, you are listening to me at chipmunk speed; and I'm already kind of a chipmunk. So it's kind of bad. But it's hard to realize how powerful, just as an economists--and we know that annoying things on the Internet, you stop. You just go elsewhere. You just--it's why I appreciate my listeners telling me how much they care about audio quality, because I've tried to make the audio quality better. And I think it makes a huge difference, even though it's a subtle kind of transaction cost. Guest: Yeah. I think--my intuition off of the top of my head is if you cut the introduction down to something very short: I think mentioning EconTalk is important for branding and reminding people to go back later. But I think you could do it in a much shorter amount of time. You know, you can think of the, sort of like, what's the payoff that people get for listening per unit of time-- Russ: Yeah. Guest: It's going to go up if you decrease the amount of time to get the same payoff. Russ: Yeah. Yeah, it's a small amount. And people do just kind of fast forward through it. A lot of them though find it comforting to hear the theme music. Others, maybe that's a barrier. Guest: I think you might get somewhere between 1% and 0.1% more usage if you cut it down. Russ: Yeah, that's all good. I don't get much gain from it being longer. Except, I mean, not having to re-record it.
47:21Russ: Tell me about your typical day--if there is such a thing. So, right now, your typical day--this particular day--has an hour plus time taken up for an EconTalk interview. And we hung out a little beforehand; you gave me a tour of the place; so that was very nice of you. But it is not a typical day, although I'm sure you do some media. What percentage of your day is spent arguing with the Data Scientists about how to interpret an experiment versus getting funding for the next place you are going to head to, versus trying to market the site, versus managing 130 people--which must be very different than it was 5 years ago when it was whatever the smaller number was? And how do you teach yourself? You didn't study any of that. So you just kind of learned it by the seat of your pants. You'd like it as a question, but it's not an easy one to answer. Guest: Yup. I think I've just read a lot of books about management. That was where that came from. I'd say most--I'm relatively internally focused, as far as [?] go. We've raised a lot of money in the past; we still have a lot of money in the bank, so we don't have to worry about fundraising very often. We--um--I don't do a lot of press, because I think the product kind of tends to market itself: as we get more content, more users find out about it. And that's a more efficient channel than media, generally. So I spend most of my time internally focused. So, that's things like, let's say, this morning I had a meeting with our head of product. And, he's actually an economist, that's his background. So, we were working on setting up the goals for what the product team are going to be over the next quarter. And then we are trying to resolve the, sort of, an issue where some people in the company have one opinion about a certain topic and other people have a different opinion. So we're trying to figure out how do we get the constructive resolution of this difference. So, you could call that management. But it's also about, a lot of that is about product direction. And it's very deeply tied to what we are doing as a company. I spend time on recruiting. I spend time on just looking at data myself, trying to stay informed. Trying to build good intuitions. I look at all the experiment results. One thing we set up that's been really great is we have this internal mailing list where any time there's a data request or an experiment that gets analyzed, the person who analyzes it will write up the analysis and send it to this mailing list for the whole rest of the company to see. And so that provides a sort of like archive for us to all learn from everyone else's research. And they'll give other people ideas. But it's also good for me to just stay in touch with, say: Hey, what's working? Is our--like, do we need to adjust? Should we have more people working on this aspect, or more people working on something else? I can learn a lot from results of the experiments. Russ: How do you keep your team of Data Scientists tooled up? So, obviously, you are trying to read a lot. You are asking, you are following machine learning on Quora. But you are reading books; you are reading journal articles, etc. And you are sharing them with your staff, I'm sure, when you find them interesting. But how do you keep them--is there any formal way that you keep them educating themselves? Besides encouraging them to listen to EconTalk, of course. Guest: Um, they are fans of EconTalk here. Russ: Glad to hear it. Guest: So, we, let's see. The Data Scientists, um-- Russ: I asked that because machine learning is a relatively--this field must be exploding in terms of its technical progress, experimentation within the field. Like you said, there'll be a paper that comes out; people will think it works; maybe it doesn't. It's a very young and I would think embryonic area with lots of things to figure out and potential for growth. Guest: Yeah, absolutely. So, we don't have anything that's really formalized around this. We have a culture that really values learning and like observing the world and what other companies are doing. And so, for example, we have, like, there's a group that meets to look at the latest deep learning research; and there have been other reading groups in the past. We'll have external speakers come in. We have--you know, there are a lot of mailing lists where people are circulating things that they've found that are relevant. But often a lot of the learning comes internally from running experiments and seeing what happens. And I think we tend not to trust the results of other companies' product changes very much. Russ: May not apply to you. Guest: Yeah, they may not apply to us. Russ: May be mistakes. Guest: Yeah. I think a lot of the time the companies are making mistakes. The companies have different user bases than us; they have different strategy, a different position in the market. They are just--they have things that are done in different ways that might be the right way for that company but it wouldn't be the right thing for our company.
52:40Russ: You used a phrase, 20 minutes ago or so, about, it was close to a motto, about knowledge: the way you see Quora. Do you remember that phrase? Guest: So, the mission is, 'Share and Grow the World's Knowledge.' Russ: Yeah, okay, yeah, that's it. So, how often do you remember that? Because one thing I find interesting about business and life, if I said to you, 'What's your motto?' you just told me. A lot of times we don't always act according to our motto. We think we do. But we end up being pushed and pulled by all kinds of incentives. We talked about that recently on the program with Ryan Holiday--we were talking about nonprofits. But even in the profit-focused business, you forget. How do you keep that front--do you think you do a good job keep it front and center? And how much time do you think about what you are actually doing, as opposed to the day-to-day minutiae of executing the business? Guest: Um, so I think the mission, we remember all the time. I'd say we are a relatively mission-driven company, compared to other companies. So, one of the things we do when we are interviewing candidates, for example, is we look for whether they, whether we think they would be interested in our mission: Is that their motivation to come work here? Or, how much does that compare to--or do they just view us as one of 10 different companies that they could work for, and they are sort of all interchangeable? So I think, I think getting employees in here in the first place who care about knowledge is sort of more important than just sort of reciting the phrase over and over. But the phrase is important because that's how people remember to look for it in other employees that they are hiring. But, yeah: Definitely repetition is very important. Russ: But--I guess I didn't ask the question well. What I'm also thinking of is: it's a wonderful thing to create a site that people enjoy using. Just--that's just great. Right? It's a wonderful thing. There's of course a lot of substitutes about ways to spend my time, as we were talking about earlier. There's a big difference between that and changing people's lives--giving them an answer to a question that's, again, life-saving, life-changing, transformative--versus, whew, now I know that [?]; that was interesting. Right? Knowledge is a really rich, complicated thing. It can be exhilarating, but it also can be practical. Does that get into your, in your guts, day to day, in the company? Do you feel a sense of satisfaction? I mean, it's great that you have all these users and all these people, you know, on the site. But do you have a feel for what's happening in their lives? Guest: Yeah. I mean, we actually, we have an internal process where people who, employees who notice these kinds of things happening in the user-base will share them with other employees. Russ: Give us an example. Guest: There was someone who wasn't able to conceive, in their marriage. And through some answers that they read they were able to figure out-- Russ: That's a good one. Yeah. Guest: There's been--you know, there are questions about what it's like to be, to have depression and how to overcome depression. And then there's these stories about other people who read these; and someone who was going through some pretty bad times in their life, through reading these answers was able to do a lot better. And just wrote me a personal email about it. So, yeah: We have that--we like to--I think the employees especially like, it's great to feel like a higher sense of [?] than just getting your work done.-- Russ: Sure. But you don't spread that in an active way. It's just something that happens from time to time. Guest: Yeah. There was one period where we formalized it a little bit. But I think it's actually almost better to just let it go organically.
56:50Russ: So, we had, um, I had Adam Smith--Abby Smith Rumsey on the program recently talking about cultural memory and how challenging it is to archive the digital world and to deal with the collective memories that we have. And your site is a powerful place where there's some--I'd say there's a lot of information about how people perceive the world. Certainly we'll look back on it in a hundred years and say maybe it was perceived incorrectly. And we can see that, because 'look how people answered these questions on Quora.' But you've got this massive amount of information on the site. Do you ever think about how people in the future, whether it's at the Library of Congress or somewhere else, might be able to access this information? Let's put aside privacy for the moment, which is a whole 'nother complicated piece of this. Especially because you are using real people's names. But, can you imagine ways that we might use the incredible knowledge in a meta-way that's accumulated on a site like yours? Guest: Um, yeah. I mean, absolutely. I think--to be clear, I think archiving for the long term is super-important to us. And so we've--that's one--I think it's actually something that most Internet services today are not really set up with any concern for the future. It's about what's happening right now; and then the content that's even a week later is not really relevant. So, we have a big focus on this. I could imagine--one thing I always wondered about when I've looked at history--you know, you get this interpretation of history that's very colored by the person who is writing the history. Russ: Yeah. Guest: And you don't really have a sense for how people were feeling, or maybe, you know, how some people were feeling; but you don't know how most of the people were feeling, or what the different perspective of views were. And so I think it's--it's really amazing that in a hundred years from now people will be able to look back, and through Quora and other services will be able to get a sense for, really, what did people think of Donald Trump: like, what was the true reaction to this, this candidate? Russ: Answer: Diverse. Shockingly diverse. Guest: Yeah. Very diverse. Russ: Surprisingly diverse, perhaps. But we are not going to get into politics here. But to me it's--the harder question is, something like inequality, which the media talks about all the time. A lot of times, historians get at, 'Well, what do people really think about an issue? We'll look at a diary.' One person's diary. And what we are creating with the Internet, for better or for worse, is hundreds of millions of diaries, in a certain sense. Right? Millions of blogs; billions, trillions of comments. Trillions of photographs. So, what people care about in certain spaces. Now, inequality is a lot tougher. You don't take photographs of it, literally. But these are the questions that, conceivably, we'd have a better understanding of, because of the information we have--maybe. Guest: Yeah, absolutely.
1:00:10Russ: Well, let's close with you. You've been here 6 years; you've built something extraordinary. What's the future of you? Do you think you'll be here for a long, long time? Are you going to go public? Are you going to get sold? Do you have other dreams, to other projects that you hope to get to--maybe they'll be part of Quora, maybe not? Anything you want to share? Guest: So our intent is to go public at some point. We're not in a rush to do it, but we want to be a long-term independent company. I'm personally committed to just sticking with it as long as it takes. I couldn't see any reason why I would do anything else. Russ: Because you're not bored. Guest: Yes. I'm definitely not bored. Things get much more exciting as we get bigger and bigger. We have more scale; we have more data; we are reaching more people; we are getting more knowledge shared into the world. So, I'm totally committed. Russ: We're sitting here in Silicon Valley, in Mountain View, California. What's happening outside this building that you think is underappreciated or under-reported about the future? Right now, if you asked me, as a total amateur who doesn't swim in the circles that you swim in, I would say things like, 'Well, driverless cars, digital health'--there are some obvious trends. What's going under the radar, if anything you think is important? Or under my radar, at least? Guest: Yeah. I mean, I think one thing that's under a lot of people's radar, including people in the industry, is just the power of personalization. It's, um, there's this effect where you, as a user of any of these products, you are only seeing one experience. And you don't--it's not very intuitive to you that every other user is seeing things that are totally different. And there's a lot of hype around machine learning and AI (Artificial Intelligence), I think some of which is justified and some is probably not. But ultimately I think one of the biggest impacts that the machine learning has on the economy is through enabling better and better personalization, which allows these products to get much, much better for any given user than they used to be. So, I think all these companies, all these personalization companies, I expect to get much bigger than they already are; and I think you are going to see personalization in all these other markets where it never existed before. Russ: Do you want to say anything about the--either on Quora or outside of Quora--about some tradeoff that might exist with personalization? So, when I think of Quora, I think, 'Well, that's good. It would be great to have answers and questions that I think are more interesting to me.' But when I think of, say, Amazon--is it really that exciting that they are going to get better at selling me stuff, or, you know--because maybe it will just be stuff that has a better margin for them? I sometimes worry that this personalization is not always going to be, as we say in economics, welfare-improving. It will be improving for the company. And then, it's kind of going to be getting very large--are they going to be able to restrain themselves in ways that might be harmful to their, to the political process then cracking down on it? Guest: Yeah. I think, you can actually--I think one interesting thing recently was Facebook came out and disclosed how they think about personalizing their newsfeed. And I think one of the reasons--other than just a guess from outside; I'm not speaking on their behalf--but I would guess that part of the motivation there is to get ahead of regulators. Because they'd rather have the public understand what they are trying to do than have these misconceptions out there that then lead to regulation which could be much worse [?] and probably also for the public. Russ: Of course, we have no idea if they were telling the truth. Guest: Right. But, at least--they probably have some incentive to tell the truth, because there might be--a regulator[?] to come along later. So, even the threat of regulation I think is good for protecting consumers. I think--you know, in these companies that are selling a product that are not ad-supported--I think personalization enables price discrimination. Russ: Sure. Guest: And I think Amazon has actually taken a very strong stance against price discrimination. But, you know, that could reverse at some point. There could be other companies that don't care to value the user. Russ: Yeah. Or just--I think--I don't know. I'm a big believer in competition. And I don't think, in general, showing me stuff I want to buy is evil. Generally a good thing. I'd rather look at stuff I want to buy than stuff I don't want to buy. But I think the issue of sort of margin and competition in this space is relevant. It's going to be an issue as we move in this direction. Guest: Yep. Yeah. I can see companies pushing things that are higher margin for them. It's hard to think about what's the right, from an economic perspective, what's the right solution to prevent that from happening. Russ: Yeah, I'm with you there. It's not obvious. Guest: It's not clear. Russ: Anything else you want to add? Guest: No, I think that's everything. It's been great.

More EconTalk Episodes