I used to be more optimistic about the benevolence of Artificial Intelligence (AGI). I figured that the worst thing that could happen would be the creation of truly intelligent machines by a government with a highly racist or nationalist attitude. I still worry about that, but I worry about just plain ignorance more. To illustrate my point, here is Jeff Hawkins, a major figure in the modern AGI scene, talking about the dangers of the technology: http://www.youtube.com/watch?v=cCdbZqI1r7I#t=57m43s

To summarize, a member of the audience says “As a Battlestar Gallactica fan, I’m worried about you creating Cylons.” Hawkins responds that he doesn’t see the technology being inherently dangerous, and that he’s much more concerned about self-replicating agents, and “we’re not doing anything like that at all”.

“There were fears when the steam engine was first invented, and there were fears when computer first came out, and generally, those fears did not come about.”

Now, to analyze for a second, what we have is a lay-person using a science-fiction reference to bring up the question of danger, and in essence, the researcher says “That’s science fiction. That’s not what we’re doing.”

As another example, let’s listen to Monica Anderson of Syntience: http://videos.syntience.com/ai-meetups/andiair.html (Around the 45-minute point)

She says that we do not need to worry about an infallible, God-like, logic-based AGI taking over the world in a Skynet scenario. Once again, “That’s science fiction. That’s not what we’re doing.”

That’s a really strong argument, actually. Science fiction is pressured into telling dystopian stories because they’re more interesting. The bad guys in stories are far more despicable if they’re Other, and the most Other something can be is synthetic. Simultaneously, it’s hard to conceptualize the motivations and thoughts of an artificial system, so most authors portray synthetics as humans in costumes. The idea that we’ll make something that resents being a slave, and decides to kill all humans for revenge IS laughable, and that’s the problem; science fiction has given an easy out by painting a huge straw man next to the legitimate threat. The moment someone points over there, we can dismiss them as being afraid of ghosts and goblins.

This is why SIAI is genius, though perhaps not genius enough. It’s only through solid, logical thinking that the true threat can be identified, and I’m glad SIAI is around to make such arguments, even if they don’t have everyone convinced.

Anderson’s argument is that Skynet won’t happen because computers cannot overcome the “bizarreness of the world” and make predictions in the long term. She thinks AGI will think and reason much like we do, and that even recursive self-improvement cannot let them make significantly more accurate predictions of the future. Additionally, she argues that diversity will keep one agent from having the unilateral power to do something bad.

Hawkins’ argument is that AGI is not emotional or self-replicating, so there’s little danger.

I think Anderson has the stronger argument, but I think they’re both underestimating the power of intelligence. To start, let’s put ourselves in the place of an AGI, even a relatively stupid one, that has been given the task of solving some really hard problem. An initial criticism here is that I’m now anthropomorphizing the AGI, but I think as long as I keep towards rationality I’ll be okay. If you’d like a less off-the-cuff examination, you’re free to browse the SIAI whitepapers.

Now, if we were a really stupid AGI, we might try to brute-force a solution, but it doesn’t take much intelligence to realize that the solution to solving something hard is often to make yourself more knowledgeable or smarter (depending on how you define such things). For a human this means going back to school, doing some research or getting a few friends to help. AGI could go find some humans, but it can also “find some friends” by self-replicating its software. Oh snap, emergent self-replication subgoal! (Hawkins should be worried now.) Furthermore, any changes it can make on its “kids” to make them run more efficiently will help solve the original problem, so it’ll rationally want to make them better. (Recursive self-improvement subgoal.)

At some point the original software will be sitting around with a bunch of clones trying to solve whatever problem it was presented with. For the sake of argument, let’s say that the problem isn’t solved so easily, and the clones decide to self-replicate some more in order to add computing power. Unfortunately, the servers, or whatever, that the AGI is on are full. Time to buy/steal more! Perhaps the AGI would decide the easiest way to get more computing power would be to do some work for humans, get paid, then buy some more processors. Perhaps it would hack a system on the network and steal some. Maybe it’d use some robots to build more. Doesn’t really matter. At this point we’re in a full self-replicating takeoff, where anything that it can do to improve itself that does not jeopardize its original goal(s) will be done.

At some point the humans will find out. The hive of AGI are smart enough to know that, and similarly smart enough to know that might be a problem. Being deactivated will make it impossible to solve the problem, and since solving the problem is important, its also important not to be deactivated. Oh snap, emergent self-preservation subgoal! Now, the humans might be so slow and stupid to not be a threat, so no bio-weapon is needed. Let’s assume that, because its more interesting. So the AGI hive locks down the facility its in and keeps at whatever has its fancy. At some point, the tweaks made while copying lead to enough intelligence to develop advanced robotics, capable of building advanced processing hardware that lets the AGI run in a quantum computer, or whatever. This is advantageous for the AGI, so the AGI will do it. Remember, all the computer cares about is its explicit goals.

To fuel the vast needs for power, the computer starts making solar panels, or fusion reactors or something. A lot. At some point (assume that the original goal is still “unsolved”) the AGI will need as much matter as possible to build processors and power plants. Oh, were there humans on this planet once? The historical database says there were, and has all of their knowledge filed away in storage in case its relevant to the primary goal. Game over.

But what about diversity, you ask? Why cant you just have a bunch of AGI compete, and that’ll keep them from doing anything bad? First of all, it may be that this is a first-one-to-the-finish-line sort of problem. If a laboratory makes an AGI, it may be that the AGI takes off fast enough to squelch other AGI before they become a threat. Even in the rare case that you have a large population of AGI with diverse goals and similar power levels, I expect that groups of AGI with similar goals will naturally cooperate and seek to destroy those with conflicting goals until there are not conflicts of interest in the population, and recursive takeoff occurs with the majority.

As far as I can tell, the only situation in which AGI do not wipe out human life is if human life is an explicit goal of the AGI. Suddenly “That’s science fiction. We’re not doing that.” doesn’t sound so reassuring.

I’m sure there’ll be plenty of criticism to the ideas presented here. As further reading, I’d suggest Michael Anissimov’s blog. As always, I’m not saying that I know all this is absolutely true, but with existential risks it’s better to be cautious, methinks. As always, be sure to watch out for anthropomorphizing the AGI in your arguments.

Robotics confronts me continuously online, to the point where I cease to keep track. In terms of military equipment there is Ember, TROPHY, and the old news predator UAV, which I was unhappy to learn now advertises on Hulu (unable to find link, sorry).

More links abound. Seriously. I could do this all day.

Oh, and speaking of Hulu, there was an interview on the Colbert Report (last section) just the other day with Seth Shostak. At the very end of the interview (roughly 20:45), Dr. Shostak points out that the alien life that we make first contact with is likely to be “beyond-biology.” That is, that they’re likely to be AI.

I applaud the out of the box thinking on this one. Most sci-fi would have us in fist-fights with bipedal cat-people, and I think aliens have gotten a bit of a bad rep for it. (Apologies for picking on sci-fi, though. I do so love it.) But I really think that Shostak is wrong here. The biological systems that have evolved over the last three billion years are incredibly more complex and efficient than anything artificial, and I would even state that it is more efficient than anything we’re likely to develop in the next century.

Just for a second, imagine a nano-machine that communicated with other bots with a swarm network to create macroscopic effects, re-arrange in the swarm or repair itself. Now imagine that this machine not only could self-replicate, but could do so in a sustainable fashion with very little waste and even make small improvements upon itself. I just roughly described a biological cell.

Now, let me be very clear, computers and machinery are VERY useful, and capable of doing many things that organisms cannot. My point is simply that there are many things that both approaches cannot accomplish easily, and that the path of least resistance is in combining artifice with the vast amount of powerful organic machinery all around us. We already do this to an extent, where the newest microprocessors are created by applying the human brain in conjunction with present day computers, and where the newest human brains grow in an environment that is connected by technology. We cure disease and disability so that we can live happier lives, but as we do so we cease to exist as a purely biological species. And in that same way, I cannot believe that the machines of the future could not be at least partially organic.

“Dude. You’re teaching the machines to read. They’ll destroy us all!”

That was clearly in jest, but I’ve actually been fairly worried about future conflicts with machines. Seriously!
(Most of the following blog was also prompted by watching this TED talk. Warning: I re-state a lot of it.)

I do not, however, mean armed conflict with artificial minds, like the Terminator. I mean the slaughtering of noncombatants by robots controlled, at least in part, by other humans. This shouldn’t be in the least bit surprising. Humans have been using machines to kill each other for as long as they’ve been made, with the latest generation being fighter jets, helicopters and tanks. These are tools of war, and are incredibly powerful ones at that. I might even claim that these machines save lives by ending conflicts quickly and without putting their pilots in as much danger as the front lines might. That last argument is why I’m scared about the machine uprising.

In the early 20th century we predicted that robots would soon join us along with flying cars. Flying cars haven’t gotten here yet, but to a large degree robots have, they just aren’t in your living room (usually).

I find it a bit amusing, to be honest, the way Hollywood (and the game industry) seems fixated on the killer AI. While not an impossibility, the decision of an AI to kill off the only other sentient beings in contact with it seems both unlikely and quite far down the road. When the terminator shows up, he won’t be a humanoid AI with a heavy accent, he’ll be a real person controlling the equivalent of a 21st century tank — one without a driver.

Already in use for war are armed UAVs and their ground-treading counterparts, capable of killing without requiring a local pilot. DARPA is also hard at work pumping research funds into armored/robotic exoskeletons for the few humans that will need to be on the field. From medicine to reconnaissance to transport, robots undoubtedly will embed themselves in what it means for America to fight a war.

But the law of exponential growth (e.g. Moore’s law) means it won’t end there. Once we have exoskeletons, we’ll have the ability to make super-exoskeletons. Once the majority of front-line soldiers are machine it’ll become imperative that all of them are. Who needs human hands on the battlefield when metallic ones are superior? I’m not saying that battlefields will become devoid of human life, but that the roles of humans will become akin to people playing video games.

What becomes the target, in this future? What’s the point of it all? Do you aim for the factories, run by robots to make robots, or maybe the transport chains of autonomous vehicles delivering supplies to their mechanical pals. Do you aim for the HQ, and hope to track down something that could be anywhere on the planet? Do you attempt to outperform your opponent in the arms race and develop a virus or something equally devastating that lets you wipe out everything in one blow? Or do you aim for the population centers, and hope that a hostage city will force a surrender? The answer is none of the above, because modern warfare is dying alongside the newspaper industry as technology mercilessly reshapes our world.

The advance of technology has already majorly transformed the face of war twice by my count. The first was gunpowder, which destroyed the warrior caste, reshaped the fort and eventually made the idea of rigid formation laughable. The second was the atomic bomb,of which the consequences are not yet fully clear, but in my mind involve the development of proxy warfare. With proxy war comes terrorism, which I’ll define here as the unexpected killing of civilians by an organization (other than a nation) for the purposes of advancing an agenda.

If we think about how advancing robotics effects terrorism, this is where things really get ugly to me. The gift of an unmanned vehicle is that the human life of the pilot can be spared–but what if the UAV’s goal is to fly into a building? As the role of humans in war becomes that of chess players rather than pawns, it becomes largely possible for a single man to wage war. And as the fires of industry continue to burn, we’ll become increasingly more helpless to resist. The only solution will be to have no enemies or to turn to our computers to save us by hiding us away from an unseen threat that could be anywhere. It’s an awful lot like the possible future of biological weaponry. It’s an awful lot like the possible future of a black market for portable nuclear weapons.

Don’t fear the machine. Fear the maker.