Let’s get something out of the way. I’m not worried about a robot apocalypse. I don’t think Skynet is going to launch nuclear missiles in a surprise attack against humanity. I don’t think Matrix robots will turn us all into batteries, nor will Cylons kill us and replace us. HAL’s not going to plan our ‘accidental deaths’ and Megatron’s not lurking behind the moon ready to raid our planet for energon cubes. The ‘robo-pocalypse’ is a joke.

There are a host of errors in Saenz’s post, and I said as much on twitter by lumping him in with people like Jeff Hawkins, Monica Anderson (both of whom I admire for their work in the field) as “people who dismiss AI existential threat by citing sci-fi”. Perhaps it would be more accurate to say that they dismiss the threat while citing sci-fi, as Saenz himself points out that science fiction isn’t even discussing the issue, but I feel like the implication is there: “Oh, that’s nothing to worry about. It’s the product of sensationalist writers trying to tell an action-packed story.”

Yes, it’s the subject of fiction for reasons other than realism, but no, this does not mean the concept itself is unrealistic. (Most stories are incredibly unrealistic in their depictions of robotic minds, of course, but that’s a side-issue.) I’m not sure what logical fallacy that is, but I’m pretty sure it has been identified elsewhere. (Speaking of elsewhere, I’ve talked about this issue before.)

But this post is not supposed to be about Saenz. This post is a response to a very good question I received on Twitter by @notcalledjack:

Well H. sapiens was an existential threat to H. heidelbergensis, but from a larger perspective, was that such a bad thing?

IOW, I question the (applicability of the) word “threat”. :)

This question, if I may be so bold as to interpret, revolves around the concept of evolution. Would it really be such a bad thing if human intelligences were replaced by robotic ones? How would this be any worse than a child outliving a parent?

I believe that humans are goal-pursuing agents. All I mean by this is that we prefer certain worlds and we all work to change the world towards one that we prefer. This preference can come from many places, and can range from being well rested, to having lots of offspring, to making lots of money. I hope this point is totally non-controversial.

A “threat” then is something which has a chance of destroying a preferred world; something which might mess us up and keep our goals from being achieved. To be even more precise, the word threat is typically reserved for things that might cause massive amounts of goal-failure, often involving death.

If I were told that people in the future would wear more glasses than modern people, I wouldn’t care in the least. That’s a trend that doesn’t impact my goals at all. Likewise, if I think about a new species of ant out-competing another species I don’t really care. Trends themselves, including evolutionary ones, are bad things (as far as goals are concerned) and probably not much of a good things, except insofar as we’re curious and like seeing what’s next.

Trends that DO have specific impact on our goals DO have weight when it comes to our choices (i.e. they can be bad or good). A trend where 10% of my cells die each year is incredibly bad for me, and is similarly a threat. A trend where I get a 10% raise every year is good for me, etc.

So, when I think back on Homo heidelbergensis I am not moved. Their deaths were part of history—part of the evolutionary chain—a trend of about as little meaning to me as that of ant species. But if I were put back into that age, and I were a “Heidelberg Man”; my family were being killed by H. sapiens invaders and I was starving because they guarded the hunting grounds, I would surely wish them gone.

If we take the long view, and pretend that humans are tiny aliens living on a distant blue dot, it hardly matters whether a machine comes that kills all organic life. From our distant perch we can say “oh, interesting” and see what happens next. But on the ground, when faced with the end of everything human, including myself and everyone I’ve ever known, I cannot help but call the trend a threat.

I have a problem with science fiction: it’s too pessimistic. I don’t mean this in a “dystopian” sense, either. I mean that visions of the future are haphazard in what technologies exist, and what people will need to do to survive.

In the future, treadmills will suck.

Take James Cameron’s Avatar, for instance. In Avatar there are many vehicles that transport humans around on the alien world of Pandora. These vehicles are all piloted by people. The natives of the planet are actually able to kill off aircraft pilots by throwing spears through the windshield. Now, imagine if these aircraft were piloted by computer, as many military planes are today. No pilot would mean the front of the vehicle could be well-armored. No pilot would also mean that such aircraft could be mass-produced on-site, without needing to ferry pilots in from Earth. There is a scene where the humans attempt to destroy a specific forest by flying a bomb there by hand. Why isn’t the bomb just dropped from orbit? Don’t these people have spacecraft? The worst plot-hole, however, is the justification for fighting the natives of the planet. Apparently there is a rare substance of astronomical value that’s located just below the native’s sacred land. Yes, it works for symbolism, but the substance is needed for, get this, an energy source. So apparently, in this advanced future where they can afford to shuttle people around at 70% the speed of light, humanity has not yet figured out how to get decent energy from nuclear or solar? Pessimistic in the extreme.

“I can go back to playing World of Warcraft now?!”

As another example, let’s consider Pixar’s WALL-E. In the film, humanity abandons Earth for being too toxic, and ends up living for 700 years on a colony ship, where they are catered to by artificial intelligence. These people live each day constantly connected to the computer, and never do so much as get up from their chairs. At one point in the movie, WALL-E (the protagonist robot) bumps into a human, distracting him from his virtual world, the result being that the man is pleasantly surprised. Let me clarify: Pixar is telling us that in the far future, a person who has never known anything but a life of digital entertainment would be pleasantly surprised by being distracted by a dirty little robot. Those video games must SUCK. I understand that the whole thing is commentary on our modern, western lifestyles of excess and entertainment, but that doesn’t change the part of the story where these people are so unsatisfied by futuristic life that they feel a need, at the end of the movie, to return to a life of agriculture. That’s so insanely pessimistic.

The reason for this pessimism, I think, is two-fold:

• people forget that technology actually does improve society (most of the time)
• writers recognize that without problems there is no story

I’ll address these in reverse-order.

Okay, so plots are problems by their very definition. No story ever went “Once upon a time there were some happy people. The end.” We write about problems because, perhaps on a subconscious level, writers are attempting to fix real problems by communicating imagined scenarios (i.e. fiction). The writers at Pixar were hoping to (on some level) make Americans a bit more conscious of their lifestyles in hopes that they’d take better care of the planet and themselves. To do this, the writers tell a story about what would happen if these trends got worse, rather than better. The story of Wal-Mart destroying the world.

My issue is that when you tell a pessimistic story, you confuse the real problem with a hyperbolic one, and run the risk of failing to address the actual obstacles in place. In other words, if your story isn’t realistic then people may throw out the core problem as unrealistic, or they may think that hyperbolic aspects of the plot are actual problems, and focus on the wrong things. The best example I can think of is that of the Terminator, an icon for evil robots everywhere. On one hand, people watch Terminator and consider just how ridiculous it is to have an evil AI (that gets naked when it travels through time) and so they mistakenly throw out the problem as being one of idle fantasy. On the other hand, people get the impression that the mistake was in giving Skynet the keys to the bombs, and so they mistakenly believe that non-military AI serves no threat.

My suggestion: make the problem in a story as close as possible to the actual one, and avoid hyperbole when possible. Millions of people on Earth suffer because of the choices made by affluent people in the West, so tell a story about those people, and how they might come free of their hardship (or not, for a downer-ending).

Now for the second point; people (sometimes) forget that progress is real.

Why are there bartenders in Star Trek, again?

People have incredibly short lives, if you think about it. Most people alive today cannot remember getting their family’s first television, much less the first radio. The scope of many people’s lives are such that things seem only slightly different than they ever were. People forget that infant mortality has dropped by a factor of 10 in the US in just the last 70 years. Not having enough XYZ has been a problem for all of my life, so why wouldn’t it be a problem in a hundred years? Because that’s not how technology works.

Slavery existed for nearly the entire course of human history, and just happened to end at the time when complex machinery began making factories feasible. Coincidence?

Aging. Poverty. Slavery. Energy. Transportation. Education. Communication. Disease. Ignorance. Even work itself. These are all finite problems with points where they will be essentially solved. To think else-wise would be to cover one’s eyes to the lessons of history. Even companionship, empathy, security, intimacy, and fun are goals which I think can be solved with technology. When every story of the future paints a picture that says “nothing substantial changed”, we get the impression that science and technology are good for nothing by making shiny gadgets and robot dogs.

Science Fiction stories are our dreams of the future, and dreams matter. They help pull us past the repercussions of our actions and into a brighter future. I have no idea what’ll happen in the next 50 years, but I can guarantee we won’t be anywhere near where we are today. Civilization really only has two end-points: extinction and utopia. Let’s pay better attention to where we’re going.

I used to be more optimistic about the benevolence of Artificial Intelligence (AGI). I figured that the worst thing that could happen would be the creation of truly intelligent machines by a government with a highly racist or nationalist attitude. I still worry about that, but I worry about just plain ignorance more. To illustrate my point, here is Jeff Hawkins, a major figure in the modern AGI scene, talking about the dangers of the technology: http://www.youtube.com/watch?v=cCdbZqI1r7I#t=57m43s

To summarize, a member of the audience says “As a Battlestar Gallactica fan, I’m worried about you creating Cylons.” Hawkins responds that he doesn’t see the technology being inherently dangerous, and that he’s much more concerned about self-replicating agents, and “we’re not doing anything like that at all”.

“There were fears when the steam engine was first invented, and there were fears when computer first came out, and generally, those fears did not come about.”

Now, to analyze for a second, what we have is a lay-person using a science-fiction reference to bring up the question of danger, and in essence, the researcher says “That’s science fiction. That’s not what we’re doing.”

As another example, let’s listen to Monica Anderson of Syntience: http://videos.syntience.com/ai-meetups/andiair.html (Around the 45-minute point)

She says that we do not need to worry about an infallible, God-like, logic-based AGI taking over the world in a Skynet scenario. Once again, “That’s science fiction. That’s not what we’re doing.”

That’s a really strong argument, actually. Science fiction is pressured into telling dystopian stories because they’re more interesting. The bad guys in stories are far more despicable if they’re Other, and the most Other something can be is synthetic. Simultaneously, it’s hard to conceptualize the motivations and thoughts of an artificial system, so most authors portray synthetics as humans in costumes. The idea that we’ll make something that resents being a slave, and decides to kill all humans for revenge IS laughable, and that’s the problem; science fiction has given an easy out by painting a huge straw man next to the legitimate threat. The moment someone points over there, we can dismiss them as being afraid of ghosts and goblins.

This is why SIAI is genius, though perhaps not genius enough. It’s only through solid, logical thinking that the true threat can be identified, and I’m glad SIAI is around to make such arguments, even if they don’t have everyone convinced.

Anderson’s argument is that Skynet won’t happen because computers cannot overcome the “bizarreness of the world” and make predictions in the long term. She thinks AGI will think and reason much like we do, and that even recursive self-improvement cannot let them make significantly more accurate predictions of the future. Additionally, she argues that diversity will keep one agent from having the unilateral power to do something bad.

Hawkins’ argument is that AGI is not emotional or self-replicating, so there’s little danger.

I think Anderson has the stronger argument, but I think they’re both underestimating the power of intelligence. To start, let’s put ourselves in the place of an AGI, even a relatively stupid one, that has been given the task of solving some really hard problem. An initial criticism here is that I’m now anthropomorphizing the AGI, but I think as long as I keep towards rationality I’ll be okay. If you’d like a less off-the-cuff examination, you’re free to browse the SIAI whitepapers.

Now, if we were a really stupid AGI, we might try to brute-force a solution, but it doesn’t take much intelligence to realize that the solution to solving something hard is often to make yourself more knowledgeable or smarter (depending on how you define such things). For a human this means going back to school, doing some research or getting a few friends to help. AGI could go find some humans, but it can also “find some friends” by self-replicating its software. Oh snap, emergent self-replication subgoal! (Hawkins should be worried now.) Furthermore, any changes it can make on its “kids” to make them run more efficiently will help solve the original problem, so it’ll rationally want to make them better. (Recursive self-improvement subgoal.)

At some point the original software will be sitting around with a bunch of clones trying to solve whatever problem it was presented with. For the sake of argument, let’s say that the problem isn’t solved so easily, and the clones decide to self-replicate some more in order to add computing power. Unfortunately, the servers, or whatever, that the AGI is on are full. Time to buy/steal more! Perhaps the AGI would decide the easiest way to get more computing power would be to do some work for humans, get paid, then buy some more processors. Perhaps it would hack a system on the network and steal some. Maybe it’d use some robots to build more. Doesn’t really matter. At this point we’re in a full self-replicating takeoff, where anything that it can do to improve itself that does not jeopardize its original goal(s) will be done.

At some point the humans will find out. The hive of AGI are smart enough to know that, and similarly smart enough to know that might be a problem. Being deactivated will make it impossible to solve the problem, and since solving the problem is important, its also important not to be deactivated. Oh snap, emergent self-preservation subgoal! Now, the humans might be so slow and stupid to not be a threat, so no bio-weapon is needed. Let’s assume that, because its more interesting. So the AGI hive locks down the facility its in and keeps at whatever has its fancy. At some point, the tweaks made while copying lead to enough intelligence to develop advanced robotics, capable of building advanced processing hardware that lets the AGI run in a quantum computer, or whatever. This is advantageous for the AGI, so the AGI will do it. Remember, all the computer cares about is its explicit goals.

To fuel the vast needs for power, the computer starts making solar panels, or fusion reactors or something. A lot. At some point (assume that the original goal is still “unsolved”) the AGI will need as much matter as possible to build processors and power plants. Oh, were there humans on this planet once? The historical database says there were, and has all of their knowledge filed away in storage in case its relevant to the primary goal. Game over.

But what about diversity, you ask? Why cant you just have a bunch of AGI compete, and that’ll keep them from doing anything bad? First of all, it may be that this is a first-one-to-the-finish-line sort of problem. If a laboratory makes an AGI, it may be that the AGI takes off fast enough to squelch other AGI before they become a threat. Even in the rare case that you have a large population of AGI with diverse goals and similar power levels, I expect that groups of AGI with similar goals will naturally cooperate and seek to destroy those with conflicting goals until there are not conflicts of interest in the population, and recursive takeoff occurs with the majority.

As far as I can tell, the only situation in which AGI do not wipe out human life is if human life is an explicit goal of the AGI. Suddenly “That’s science fiction. We’re not doing that.” doesn’t sound so reassuring.

I’m sure there’ll be plenty of criticism to the ideas presented here. As further reading, I’d suggest Michael Anissimov’s blog. As always, I’m not saying that I know all this is absolutely true, but with existential risks it’s better to be cautious, methinks. As always, be sure to watch out for anthropomorphizing the AGI in your arguments.

Robotics confronts me continuously online, to the point where I cease to keep track. In terms of military equipment there is Ember, TROPHY, and the old news predator UAV, which I was unhappy to learn now advertises on Hulu (unable to find link, sorry).

More links abound. Seriously. I could do this all day.

Oh, and speaking of Hulu, there was an interview on the Colbert Report (last section) just the other day with Seth Shostak. At the very end of the interview (roughly 20:45), Dr. Shostak points out that the alien life that we make first contact with is likely to be “beyond-biology.” That is, that they’re likely to be AI.

I applaud the out of the box thinking on this one. Most sci-fi would have us in fist-fights with bipedal cat-people, and I think aliens have gotten a bit of a bad rep for it. (Apologies for picking on sci-fi, though. I do so love it.) But I really think that Shostak is wrong here. The biological systems that have evolved over the last three billion years are incredibly more complex and efficient than anything artificial, and I would even state that it is more efficient than anything we’re likely to develop in the next century.

Just for a second, imagine a nano-machine that communicated with other bots with a swarm network to create macroscopic effects, re-arrange in the swarm or repair itself. Now imagine that this machine not only could self-replicate, but could do so in a sustainable fashion with very little waste and even make small improvements upon itself. I just roughly described a biological cell.

Now, let me be very clear, computers and machinery are VERY useful, and capable of doing many things that organisms cannot. My point is simply that there are many things that both approaches cannot accomplish easily, and that the path of least resistance is in combining artifice with the vast amount of powerful organic machinery all around us. We already do this to an extent, where the newest microprocessors are created by applying the human brain in conjunction with present day computers, and where the newest human brains grow in an environment that is connected by technology. We cure disease and disability so that we can live happier lives, but as we do so we cease to exist as a purely biological species. And in that same way, I cannot believe that the machines of the future could not be at least partially organic.

“Dude. You’re teaching the machines to read. They’ll destroy us all!”

That was clearly in jest, but I’ve actually been fairly worried about future conflicts with machines. Seriously!
(Most of the following blog was also prompted by watching this TED talk. Warning: I re-state a lot of it.)

I do not, however, mean armed conflict with artificial minds, like the Terminator. I mean the slaughtering of noncombatants by robots controlled, at least in part, by other humans. This shouldn’t be in the least bit surprising. Humans have been using machines to kill each other for as long as they’ve been made, with the latest generation being fighter jets, helicopters and tanks. These are tools of war, and are incredibly powerful ones at that. I might even claim that these machines save lives by ending conflicts quickly and without putting their pilots in as much danger as the front lines might. That last argument is why I’m scared about the machine uprising.

In the early 20th century we predicted that robots would soon join us along with flying cars. Flying cars haven’t gotten here yet, but to a large degree robots have, they just aren’t in your living room (usually).

I find it a bit amusing, to be honest, the way Hollywood (and the game industry) seems fixated on the killer AI. While not an impossibility, the decision of an AI to kill off the only other sentient beings in contact with it seems both unlikely and quite far down the road. When the terminator shows up, he won’t be a humanoid AI with a heavy accent, he’ll be a real person controlling the equivalent of a 21st century tank — one without a driver.

Already in use for war are armed UAVs and their ground-treading counterparts, capable of killing without requiring a local pilot. DARPA is also hard at work pumping research funds into armored/robotic exoskeletons for the few humans that will need to be on the field. From medicine to reconnaissance to transport, robots undoubtedly will embed themselves in what it means for America to fight a war.

But the law of exponential growth (e.g. Moore’s law) means it won’t end there. Once we have exoskeletons, we’ll have the ability to make super-exoskeletons. Once the majority of front-line soldiers are machine it’ll become imperative that all of them are. Who needs human hands on the battlefield when metallic ones are superior? I’m not saying that battlefields will become devoid of human life, but that the roles of humans will become akin to people playing video games.

What becomes the target, in this future? What’s the point of it all? Do you aim for the factories, run by robots to make robots, or maybe the transport chains of autonomous vehicles delivering supplies to their mechanical pals. Do you aim for the HQ, and hope to track down something that could be anywhere on the planet? Do you attempt to outperform your opponent in the arms race and develop a virus or something equally devastating that lets you wipe out everything in one blow? Or do you aim for the population centers, and hope that a hostage city will force a surrender? The answer is none of the above, because modern warfare is dying alongside the newspaper industry as technology mercilessly reshapes our world.

The advance of technology has already majorly transformed the face of war twice by my count. The first was gunpowder, which destroyed the warrior caste, reshaped the fort and eventually made the idea of rigid formation laughable. The second was the atomic bomb,of which the consequences are not yet fully clear, but in my mind involve the development of proxy warfare. With proxy war comes terrorism, which I’ll define here as the unexpected killing of civilians by an organization (other than a nation) for the purposes of advancing an agenda.

If we think about how advancing robotics effects terrorism, this is where things really get ugly to me. The gift of an unmanned vehicle is that the human life of the pilot can be spared–but what if the UAV’s goal is to fly into a building? As the role of humans in war becomes that of chess players rather than pawns, it becomes largely possible for a single man to wage war. And as the fires of industry continue to burn, we’ll become increasingly more helpless to resist. The only solution will be to have no enemies or to turn to our computers to save us by hiding us away from an unseen threat that could be anywhere. It’s an awful lot like the possible future of biological weaponry. It’s an awful lot like the possible future of a black market for portable nuclear weapons.

Don’t fear the machine. Fear the maker.