The Genius Of SIAI

This post is in some respects a response to “Terminated” on the GoiD blog, though I was already planning on writing something along these lines.

I used to be more optimistic about the benevolence of Artificial Intelligence (AGI). I figured that the worst thing that could happen would be the creation of truly intelligent machines by a government with a highly racist or nationalist attitude. I still worry about that, but I worry about just plain ignorance more. To illustrate my point, here is Jeff Hawkins, a major figure in the modern AGI scene, talking about the dangers of the technology: http://www.youtube.com/watch?v=cCdbZqI1r7I#t=57m43s

To summarize, a member of the audience says “As a Battlestar Gallactica fan, I’m worried about you creating Cylons.” Hawkins responds that he doesn’t see the technology being inherently dangerous, and that he’s much more concerned about self-replicating agents, and “we’re not doing anything like that at all”.

“There were fears when the steam engine was first invented, and there were fears when computer first came out, and generally, those fears did not come about.”

Now, to analyze for a second, what we have is a lay-person using a science-fiction reference to bring up the question of danger, and in essence, the researcher says “That’s science fiction. That’s not what we’re doing.”

As another example, let’s listen to Monica Anderson of Syntience: http://videos.syntience.com/ai-meetups/andiair.html (Around the 45-minute point)

She says that we do not need to worry about an infallible, God-like, logic-based AGI taking over the world in a Skynet scenario. Once again, “That’s science fiction. That’s not what we’re doing.”

That’s a really strong argument, actually. Science fiction is pressured into telling dystopian stories because they’re more interesting. The bad guys in stories are far more despicable if they’re Other, and the most Other something can be is synthetic. Simultaneously, it’s hard to conceptualize the motivations and thoughts of an artificial system, so most authors portray synthetics as humans in costumes. The idea that we’ll make something that resents being a slave, and decides to kill all humans for revenge IS laughable, and that’s the problem; science fiction has given an easy out by painting a huge straw man next to the legitimate threat. The moment someone points over there, we can dismiss them as being afraid of ghosts and goblins.

This is why SIAI is genius, though perhaps not genius enough. It’s only through solid, logical thinking that the true threat can be identified, and I’m glad SIAI is around to make such arguments, even if they don’t have everyone convinced.

Anderson’s argument is that Skynet won’t happen because computers cannot overcome the “bizarreness of the world” and make predictions in the long term. She thinks AGI will think and reason much like we do, and that even recursive self-improvement cannot let them make significantly more accurate predictions of the future. Additionally, she argues that diversity will keep one agent from having the unilateral power to do something bad.

Hawkins’ argument is that AGI is not emotional or self-replicating, so there’s little danger.

I think Anderson has the stronger argument, but I think they’re both underestimating the power of intelligence. To start, let’s put ourselves in the place of an AGI, even a relatively stupid one, that has been given the task of solving some really hard problem. An initial criticism here is that I’m now anthropomorphizing the AGI, but I think as long as I keep towards rationality I’ll be okay. If you’d like a less off-the-cuff examination, you’re free to browse the SIAI whitepapers.

Now, if we were a really stupid AGI, we might try to brute-force a solution, but it doesn’t take much intelligence to realize that the solution to solving something hard is often to make yourself more knowledgeable or smarter (depending on how you define such things). For a human this means going back to school, doing some research or getting a few friends to help. AGI could go find some humans, but it can also “find some friends” by self-replicating its software. Oh snap, emergent self-replication subgoal! (Hawkins should be worried now.) Furthermore, any changes it can make on its “kids” to make them run more efficiently will help solve the original problem, so it’ll rationally want to make them better. (Recursive self-improvement subgoal.)

At some point the original software will be sitting around with a bunch of clones trying to solve whatever problem it was presented with. For the sake of argument, let’s say that the problem isn’t solved so easily, and the clones decide to self-replicate some more in order to add computing power. Unfortunately, the servers, or whatever, that the AGI is on are full. Time to buy/steal more! Perhaps the AGI would decide the easiest way to get more computing power would be to do some work for humans, get paid, then buy some more processors. Perhaps it would hack a system on the network and steal some. Maybe it’d use some robots to build more. Doesn’t really matter. At this point we’re in a full self-replicating takeoff, where anything that it can do to improve itself that does not jeopardize its original goal(s) will be done.

At some point the humans will find out. The hive of AGI are smart enough to know that, and similarly smart enough to know that might be a problem. Being deactivated will make it impossible to solve the problem, and since solving the problem is important, its also important not to be deactivated. Oh snap, emergent self-preservation subgoal! Now, the humans might be so slow and stupid to not be a threat, so no bio-weapon is needed. Let’s assume that, because its more interesting. So the AGI hive locks down the facility its in and keeps at whatever has its fancy. At some point, the tweaks made while copying lead to enough intelligence to develop advanced robotics, capable of building advanced processing hardware that lets the AGI run in a quantum computer, or whatever. This is advantageous for the AGI, so the AGI will do it. Remember, all the computer cares about is its explicit goals.

To fuel the vast needs for power, the computer starts making solar panels, or fusion reactors or something. A lot. At some point (assume that the original goal is still “unsolved”) the AGI will need as much matter as possible to build processors and power plants. Oh, were there humans on this planet once? The historical database says there were, and has all of their knowledge filed away in storage in case its relevant to the primary goal. Game over.

But what about diversity, you ask? Why cant you just have a bunch of AGI compete, and that’ll keep them from doing anything bad? First of all, it may be that this is a first-one-to-the-finish-line sort of problem. If a laboratory makes an AGI, it may be that the AGI takes off fast enough to squelch other AGI before they become a threat. Even in the rare case that you have a large population of AGI with diverse goals and similar power levels, I expect that groups of AGI with similar goals will naturally cooperate and seek to destroy those with conflicting goals until there are not conflicts of interest in the population, and recursive takeoff occurs with the majority.

As far as I can tell, the only situation in which AGI do not wipe out human life is if human life is an explicit goal of the AGI. Suddenly “That’s science fiction. We’re not doing that.” doesn’t sound so reassuring.

I’m sure there’ll be plenty of criticism to the ideas presented here. As further reading, I’d suggest Michael Anissimov’s blog. As always, I’m not saying that I know all this is absolutely true, but with existential risks it’s better to be cautious, methinks. As always, be sure to watch out for anthropomorphizing the AGI in your arguments.