Russ Roberts

Bostrom follow-up

EconTalk Extra
by Russ Roberts
Continuing Conversation... Nic... James Otteson on the End of So...

Bostrom Follow-up

For me, the interview with Nick Bostrom was mind-blowing. I've been thinking about it quite a bit since we did the interview and the thoughtful comments from listeners have continued the process.

I made a strong claim early on that surprised Bostrom so I want to try to give a little bit of background for why I made the claim. I made the observation that Bostrom's view of superintelligence reminded me of the way medieval theologians talk about God. Bostrom responded by pointing out that unlike God, his concept of superintelligence was limited by the laws of physics. That is true, but other than that, in his book he is able to imagine a superintelligence with all kinds of superpowers. By his definition, these powers are beyond human powers, so to be more precise than I was before, his concept of superintelligence is somewhere between man and God but a lot closer to a God-like set of powers than merely advanced human skills.

Here is one description from his book of what Bostrom imagines a superintelligent machine might be capable of doing:

Using its strategizing superpower, the AI develops a robust plan for achieving its long-term goals. (In particular, the AI does not adopt a plan so stupid that even we present-day humans can foresee how it would inevitably fail. This criterion rules out many science fiction scenarios that end in human triumph. The plan might involve a period of covert action during which the AI conceals its intellectual development from the human programmers in order to avoid setting off alarms. The AI might also mask its true proclivities, pretending to be cooperative and docile. If the AI has (perhaps for safety reasons) been confined to an isolated computer, it may use its social manipulation superpower to persuade the gatekeepers to let it gain access to an Internet port. Alternatively, the AI might use its hacking superpower to escape its confinement. Spreading over the Internet may enable the AI to expand its hardware capacity and knowledge base, further increasing its intellectual superiority. An AI might also engage in licit or illicit economic activity to obtain funds with which to buy computer power, data, and other resources. At this point, there are several ways for the AI to achieve results outside the virtual realm. It could use its hacking superpower to take direct control of robotic manipulators and automated laboratories. Or it could use its social manipulation superpower to persuade human collaborators to serve as its legs and hands. Or it could acquire financial assets from online transactions and use them to purchase services and influence.

So it has goals that threaten humanity. But it's so smart and cunning relative to human beings that it can achieve these goals by manipulating humans without humans catching on, even though it has no legs and hands--no mobility, no physical resources at its direct disposal. It will just get some. How? By socially manipulating people using its social manipulation superpower. This is where I started to get a little uneasy. Social manipulation isn't easy to do. Being smarter doesn't make you better at it. It might have to socially manipulate millions of people who can communicate with each other. It's just not obvious that that's possible.

Actually I become uneasy with the first sentence from the quote. Strategizing superpower? The challenge of strategizing and executing a successful strategy isn't difficult because we're not smart enough or don't have enough data. The challenge is a fundamental uncertainty about the future where billions of people have their own plans and strategies. Human beings are really mediocre at foreseeing the future and how various decisions might affect that future. So the United States attacks Iraq and all kinds of things happen that weren't foreseen. Do you think there's a scientific way to foresee the consequences of war more accurately? It isn't obvious to me that that's possible. There's a level of uncertainty that comes with war that isn't about the fact that we don't have enough previous data points about war and its consequences. So I have trouble imagining how a superintelligent entity could become good or great or precise about strategizing. An omniscient god can. Such a god by definition can see the future. Humans can't. So when you assume superintelligent entities can achieve goals through their strategizing superpower, well to me, that sounds like assuming your answer before you get started. It borders on magical thinking. It presumes a god-like omniscience.

A lot of this part of the conversation reminded me of Hayek and his Nobel Lecture. In that lecture Hayek isn't saying that we don't know enough to steer the economy. He's saying we'll never know enough. He writes:

Consider some ball game played by a few people of approximately equal skill. If we knew a few particular facts in addition to our general knowledge of the ability of the individual players, such as their state of attention, their perceptions and the state of their hearts, lungs, muscles etc. at each moment of the game, we could probably predict the outcome. Indeed, if we were familiar both with the game and the teams we should probably have a fairly shrewd idea on what the outcome will depend. But we shall of course not be able to ascertain those facts and in consequence the result of the game will be outside the range of the scientifically predictable, however well we may know what effects particular events would have on the result of the game.

Is it imaginable that some superintelligent entity could design health sensors that would gather all these data along with each player's level of motivation? That it could accurately forecast will, desire, passion? (Below, I speculate whether these will ultimately fall under the sway of the knowable.) Then all outcomes might be foreseeable, strategizing is imaginable, and manipulation doable. As I mentioned in the conversation, this question of the power of the superintelligent reminded me of the Socialist Calculation Debate of the early part of the 20th century where MIses and Hayek argued that top-down central planning was not ineffective because of an insufficiently large computer or data set but rather it was ineffective because the fundamental problem was simply too complex. It is not possible for the man of system or the machine of superintelligence to move the chess pieces of society unerringly when each piece has a motion of its own.

So many intractable social problems are intractable for reasons much greater than a lack of intelligence. They are not engineering problems but of a different nature, unamenable to solution by bigger, faster brains. Certainly this perspective comes naturally to economists who often think that there are no solutions only tradeoffs.

I thought the interview really went off the rails when we talked about justice and making the world a better place. Bostrom seems to imply that the challenge of improving the world is a matter of finding the right way to represent human aspiration in computer code:

The shortcoming is in our current ability to describe, capture, represent human values in computing language. So, this is something we don't know how to do. Maybe we could create an AI today that would want to maximize the number of digits of pi that it could calculate. So, a very simple goal like that would be within our current reach to program. But we couldn't make an AI that would maximize justice or love or artistic beauty because these are complex human concepts that we don't yet know how to represent.

I was arguing that it's not a programming problem. It's a reality problem. These concepts cannot be represented by computer code because they inevitably are a matter of different judgments made by different people. There cannot be a scientific judgment about justice. To return to theology, we can imagine divine justice. But that is because we conceive of God as all-knowing and while we may not understand divine justice, we also understand that God's ways are beyond our understanding. Is it imaginable that we humans could create a machine that would surpass us not only in calculating ability but in moral discernment? I literally cannot imagine it. Perhaps that is Bostrom's point. These machines will be to us, as we are to a cat. But then again, this is a nearly divine vision of super intelligence. (I worry that I may have misunderstood Bostrom here. I am hoping he will respond to clarify.)

Underlying much of the discussion was the question of values or motives or aims for a superintelligent entity. I am not sure about Bostrom but for many thinkers in this area, consciousness is just chemistry. So if, for example, we get to superintelligence through replicating the brain's physical state, it will then be a brain with everything else that goes with a brain such as consciousness. This is not a universally held view. There is another view that says that the mind is more than its physical components. That consciousness, will, drive, and intention or more than the firing of neurons in the soup of the brain. This view is hard to accept or even imagine for some, but some really smart people believe that the mind-body problem will never have a scientific understanding rooted in the laws of physics and chemistry. (I have started Cosmos and Mind by Thomas Nagel on this topic and hope to continue reading David Chalmers on this issue).

While very interesting, put that to the side. The question I want to think about is not whether machines will have consciousness but whether we human beings will treat them as anything other than machines. I'm trying to get at the Bostrom claim that the superintelligent entities will deserve moral standing, whether we will care for them, or whether for example, they should be allowed to vote.

Consider the following thought experiment. A man's wife dies but he does not realize it. A kindly government agency replaces her with a robot whose exterior perfectly mimics flesh and blood and through the accumulation of past data about his wife--her speech, her chemistry, her body, her neurons, her moods, whims and so on--is able to pass the Turing Spouse test. Because the machine so thoroughly understands his dead wife's biology and chemistry, it is able to pass itself off as the wife without the husband realizing it. On his deathbed, the husband is informed that the thing he has been living with, loving, lying to, flattering, being flattered by, insulted by, arguing with, making up with, being consoled by, vacationing with, and so on, was not really his wife, but only a virtual wife programmed to emulate his wife with undetectable precision. What is his reaction? Does he think that it doesn't matter, that because the bot perfectly emulated the wife, it was, essentially, his wife? I don't think so, at least not in our current culture. Maybe someday our culture would view this differently. (And no, I have yet to watch the movie, Her, but I plan to. Right now, I would say that because this ad is funny rather than poignant, we have a long way to go on attributing reality to smart machines.)

Or imagine that instead of wooing a flesh and blood creature with all of her glorious traits and unavoidable human flaws and trying to get her to marry him, a man of the future will design his wife, picking the attributes that please him, perhaps even allowing for it to get annoyed with him, or to betray him romantically. Certainly he could program a random element to surprise and delight him. Would that be appealing? Do I really want my own galaxy, as Bostrom suggests will be possible in a future of superintelligence, a galaxy where I am king and millions serve me and satisfy my various whims? This is quite close to Nozick's experience machine. It is not appealing to most of us living in 2014. Perhaps as we get more experience with smart machines, the line between human and machine will blur and much of what I have wondered about will seem normal. But right now, today, these virtual scenarios do not appeal to most of us. There is something about real life that appeals to us, despite its unpleasant messiness. That something is its realness. Why that should matter is a fascinating question for another time.

Comments and Sharing

TWITTER: Follow Russ Roberts @EconTalker

COMMENTS (12 to date)
David writes:

On how intelligent an entity (whether it's an AI or an alien of some kind) can get:

I grew up with younger siblings and cousins. When they were young children, it was really easy for me to trick and outwit them. (Sorry, older siblings can be cruel.) At the same time, an intelligent adult was still able to outwit me. Even toddlers can outwit apes in many games. There are probably really good manipulators out there who can outwit me and make almost anything sound reasonable.
So the question is; why should we assume that there aren't still several 'levels' of skill at strategy and manipulation above us humans? That for such an AI even highly intelligent and cautious people would be as easy to outwit as a toddler is for us?

I'm obviously not sure that this is true. Maybe we actually are close to the limit of social skills and predictive power. But that's not obvious.
So I think it's a good idea to investigate these ideas and be very cautious with AIs, even if there's only a small chance (rather than certainty) that they could actually be that dangerous.

Woody Belangia writes:


Neither of you mention it, but I wonder if Bostrom is leaning too heavily on the Turing test -- the idea that a machine which could fake human intelligence sufficiently to fool real human actors is functionally equivalent, and therefore *really* intelligent. That seems questionable to me. The Turing test is not sufficient to establish that machines are conscious, for instance, and I wonder if a non-conscious intelligence is a contradiction in terms. It is at best an analogy, in the same way we might talk of an “computer virus" or an "angry sky". Bostrom supposes that Turing-successful computers and humans are equivalently intelligent, and you (rightly) suspect this equivalence.

You write: "So it has goals that threaten humanity. But it's so smart and cunning relative to human beings that it can achieve these goals by manipulating humans without humans catching on, even though it has no legs and hands--no mobility, no physical resources at its direct disposal. It will just get some. How? By socially manipulating people using its social manipulation superpower. This is where I started to get a little uneasy. Social manipulation isn't easy to do. Being smarter doesn't make you better at it. It might have to socially manipulate millions of people who can communicate with each other. It's just not obvious that that's possible. ”

Here’s where I think Bostrom may have a point. I do not think that machines have to be evaluative in the human sense in order to effect social manipulation. Your AI-as-God analogy reminds me of the critics of capitalism who say that the Invisible Hand of Adam Smith is a disguised theology. The price system reflects incentives of many diverse actors. (And in what follows, I want to think of “prices” referring to a phenomenon more generic and ubiquitous than money terms.) The social actors alter their behaviors, both individually and collectively, in response to changes in prices. Prices are bearers of information, with signals running both ways, and machines are adept manipulators of information. LTCM is an example of computers pursuing ends divergent from both their programmers intentions and certainly against the social good. I do not think that the computers making split nanosecond trades are intelligent in any precise sense. But I can imagine such computers performing trades that stimulate counter-trades as a way of creating a more favorable market for future trades from the computer’s standpoint. Isn’t that a form of “social manipulation”? That example seems to be a rudimentary form of Bostrom’s worry. The computer does not need to “understand” the human values being manipulated. It need only be pursuing some maximization function of data that does express human valuation, like prices.

BTW, a science-fiction novel that is somewhat successful in describing such a scenario is Avogadro Corp: The Singularity is Closer than it Appears by William Hertling. It is not a great novel — it tries too hard to be a titillating thriller — but its setup seems plausible to me.

Keith Vertrees writes:

Russ, I agree 100% that some of the problems that Bostrom believes super intelligence can solve are, in fact, insoluble.

In the episode comments someone mentioned Asimov's Robot series. I think his Foundation series is also pertinent. In it, an advanced mathematician uses psychohistory to predict the future centuries in advance. This seems to be the kind of thing that Bostrom believes super intelligence can achieve. I think it's neat science fiction, and entirely impossible.

David Kendall writes:

Another fun and thought provoking novel with an AI character is Robert Heinlein's "The Moon Is A Harsh Mistress."

The notion that an AI will eventually develop a qualitatively different intelligence, analogous to the difference between the intelligence of a dog and a human, is difficult for me to think about in any systematic way. Can my dog think about the qualitative difference between her intelligence and mine? I don't think so. I find myself pretty much in the same boat trying to think of an AI super intelligence that is qualitatively superior to human intelligence.

Cris Sheridan writes:

Russ, once again I really enjoyed this interview and appreciate you exploring these ideas with Nick on your program.

If we broaden our interpretation of AI, we can see that the AI-as-god analogy extends back to the very beginnings of recorded history as humanity has often created objects or images simulating man, animal, or a combination thereof and attributed them with divine power. AI is humanity's most modern and successful attempt in this regards. As you probably know, this was something strongly prohibited by the Judaic, Christian, and Islamic God (for reasons I think we will still discover to be true).

That said, I don't believe superintelligence as AI is properly conceived. As Hayek would say w/respect to spontaneous order, it will be "the result of human action but not of human design," and likely be an emergent byproduct of humans and technology on a global scale. We see this taking shape today w/the advent of the internet and, more specifically, through our modern day financial system. There are many names and ways of describing this: global brain, noosphere, superorganism, etc.

Edward writes:
Bostrom seems to imply that the challenge of improving the world is a matter of finding the right way to represent human aspiration in computer code

I didn't think so.

I understood him as claiming almost the opposite: that there are some common values most humans share and that these are quite difficult to capture in code.

Depending on your inclination and biases these 'common values' might be very general or quite specific. They probably, however, mean that 99.99% of the human population don't want a universe of paper clips.

Robert W writes:

"Social manipulation isn't easy to do. Being smarter doesn't make you better at it."

I would like to know what the source for this claim is. It seems false on its face to me. The smarter people I know usually have much better social skills. Sure I can think of some very intelligent people with poor social skills, but I can think of an even greater share of unintelligent people who can't manage interpersonal relationships well at all. The observation that people are good at either one or the other could spring from this effect:

But regardless of whether it's true for humans as they were designed by evolution, that's not much reason to think that it would be true for a de novo AI. The AI would have to train itself in how to predict human behaviour in response to different stimuli. The 'smarter' it was, almost by definition, the better it could reprogram its 'social module' to learn from experience and accurate predict human responses.

Whether it will achieve enormous improvements on what humans already do is less clear. I expect it could at least become much better than the best humans. If only because a very fast AI could potentially learn from watching, listening and learning from thousands of years of videos/recordings featuring humans interacting.

"I was arguing that it's not a programming problem. It's a reality problem. These concepts cannot be represented by computer code because they inevitably are a matter of different judgments made by different people."

I think you are mistaken here.

Your concept of justice could be represented in some computer code - computer code that simulated your brain. Other simpler computer code would sometimes get your attitude wrong, but might be able to produce very similar judgements to you, good enough for practical purposes (especially if the model it builds is build from many responses you provide to hypothetical situation).

It may be that justice is a real concept, and that there is just a fact of the matter about what is just and what is not. In that case the superintelligent AI would be in a better position to judge than you are now. I doubt this personally.

If that's not the case, there won't be any 'true' way of arbitrating differences of opinion in how we should program the AI to decide what is just and unjust. However, we would still in practice need a way to produce an AI that somehow takes into account a range of different perspectives, so that i) everyone gets some of what they want ii) we don't go to an extremely bad outcome based on one person's outlandish opinions iii) we compromise rather than fight over how the AI is programmed.

Similarly, democracy or liberalism as decision making processes may not be preferable because they necessarily find the unique 'correct answer', but pragmatically they are sure better than constant winner takes all conflict over how things should be organised.

GoVoluntary writes:

To rephrase an old saying:

AI plans, God laughs.

Richard Fulmer writes:

Replace the term "super-intelligent computer" with "average bureaucrat." Congress passes a law that has some goal, say cleaning up the environment. Our average bureaucrat takes that law and transforms it into a number of regulations and then proceeds to impose those regulations on the country at large. Those regulations could do a lot of damage to the economy and the country's citizens.

In other words, we have given the bureaucrat a bad "program" (bill) that does not take into consideration other values.

Now, let's create a super-intelligent computer. We write a program (bill) to tell the computer to do some beneficial thing for us. But we are only human and we don't qualify the program enough. The computer performs its task as mindlessly as did the bureaucrat and with as little regard to unintended consequences.

The computer doesn't have to be "God" or all-knowing to do a lot of damage. It just needs to be given a "prime directive" and the power to carry it out. The fact that neither we, the computer's programmers, and the computer itself are not all-knowing almost guarantees a bad outcome.

John C. Havens writes:

I really enjoyed this interview. Thank you for the great discussion, as always.

I don't agree, however, that we can't map values with technology. We've been measuring wellbeing for years, using objective and subjective criteria. Both basically ask a person to answer the question, "how do you feel right now?" Many surveys use Cantril's Ladder of Life Satisfaction where the question is, "How satisfied are you with your life right now?" People respond with a number between 1 and 10. Their subjective truth them makes up, in aggregate, much of what we call objective data making up wellbeing polls like the ones created by Gallup.

Measuring values can be done in a similar way. Schwartz is a famous psychologist who mapped 12 common values amongst all humans globally, noting areas like family, safety, and so on. While the interpretation of these values varies widely for individuals, their commonality makes it comprehensible for study at scale. How does this individual value family? How about their community? How about their country? It's the scale that can be handled by the brute force algorithmic techniques of AI. I believe this is what Bostrom meant in your interview.

Values with AI, or autonomous robots, also involves the training of said robots to exhibit the elements of values as we perceive them once they're manifest. So, for instance, valuing "family" could be interpreted via the actions of sitting at a table for dinner, or quitting a job with too long a commute. Humans programming an AI directing it to these types of examples, after gathering the massive amounts of examples required, could start to create an algorithm reflecting the repeated actions that represented the manifestation of these values. Likewise, their opposites (violence, ignoring one's family) could also be plotted.

This is where the solutions of the AI would start to look god-like, but they're merely aggregation of mass inputs based on what we exhibit in the world.

Gandydancer writes:

Russ Roberts writes:"Social manipulation isn't easy to do. Being smarter doesn't make you better at it."

Sure it does, everything else being equal. Everything else is never equal, but more detailed awareness of signals and correct calculation of the probabilities they imply has to help. The rest is database. For human beings some of the database is built into the meat, but there is no obvious reason this can't be simulated.

Roberts goes on to deny suggest that the problem of human manipulation is too complex for even a superintelligent machine to construct a robust plan to get what it wants. But this is rather beside the point. For superintelligence to matter it merely has to have some probability of success.

Woody Belangia writes:"The Turing test is not sufficient to establish that machines are conscious..."

There are no tests which establish consciousness in human beings either. Given the experience of one's own consciousness one assumes that the other meat computers have a similar internal life. But it's just a reasonable assumption. And it's just as reasonable when dealing with an Turing-test passing machine.

Matt writes:

One thing that should be remembered (and you discussed in a podcast some time ago with Don Boudreaux) is the fact that the efficient arrangement of a social order is not a matter of having sufficient computational power applied to existing information - the real issue is that social orders themselves generate the very information upon which they depend to continue to emerge and evolve...

No intelligence - neither from man nor machine - has the ability to make rational decisions on the basis of information that doesn't yet exist...

Comments for this podcast episode have been closed
Return to top