Let’s get something out of the way. I’m not worried about a robot apocalypse. I don’t think Skynet is going to launch nuclear missiles in a surprise attack against humanity. I don’t think Matrix robots will turn us all into batteries, nor will Cylons kill us and replace us. HAL’s not going to plan our ‘accidental deaths’ and Megatron’s not lurking behind the moon ready to raid our planet for energon cubes. The ‘robo-pocalypse’ is a joke.

There are a host of errors in Saenz’s post, and I said as much on twitter by lumping him in with people like Jeff Hawkins, Monica Anderson (both of whom I admire for their work in the field) as “people who dismiss AI existential threat by citing sci-fi”. Perhaps it would be more accurate to say that they dismiss the threat while citing sci-fi, as Saenz himself points out that science fiction isn’t even discussing the issue, but I feel like the implication is there: “Oh, that’s nothing to worry about. It’s the product of sensationalist writers trying to tell an action-packed story.”

Yes, it’s the subject of fiction for reasons other than realism, but no, this does not mean the concept itself is unrealistic. (Most stories are incredibly unrealistic in their depictions of robotic minds, of course, but that’s a side-issue.) I’m not sure what logical fallacy that is, but I’m pretty sure it has been identified elsewhere. (Speaking of elsewhere, I’ve talked about this issue before.)

But this post is not supposed to be about Saenz. This post is a response to a very good question I received on Twitter by @notcalledjack:

Well H. sapiens was an existential threat to H. heidelbergensis, but from a larger perspective, was that such a bad thing?

IOW, I question the (applicability of the) word “threat”. :)

This question, if I may be so bold as to interpret, revolves around the concept of evolution. Would it really be such a bad thing if human intelligences were replaced by robotic ones? How would this be any worse than a child outliving a parent?

I believe that humans are goal-pursuing agents. All I mean by this is that we prefer certain worlds and we all work to change the world towards one that we prefer. This preference can come from many places, and can range from being well rested, to having lots of offspring, to making lots of money. I hope this point is totally non-controversial.

A “threat” then is something which has a chance of destroying a preferred world; something which might mess us up and keep our goals from being achieved. To be even more precise, the word threat is typically reserved for things that might cause massive amounts of goal-failure, often involving death.

If I were told that people in the future would wear more glasses than modern people, I wouldn’t care in the least. That’s a trend that doesn’t impact my goals at all. Likewise, if I think about a new species of ant out-competing another species I don’t really care. Trends themselves, including evolutionary ones, are bad things (as far as goals are concerned) and probably not much of a good things, except insofar as we’re curious and like seeing what’s next.

Trends that DO have specific impact on our goals DO have weight when it comes to our choices (i.e. they can be bad or good). A trend where 10% of my cells die each year is incredibly bad for me, and is similarly a threat. A trend where I get a 10% raise every year is good for me, etc.

So, when I think back on Homo heidelbergensis I am not moved. Their deaths were part of history—part of the evolutionary chain—a trend of about as little meaning to me as that of ant species. But if I were put back into that age, and I were a “Heidelberg Man”; my family were being killed by H. sapiens invaders and I was starving because they guarded the hunting grounds, I would surely wish them gone.

If we take the long view, and pretend that humans are tiny aliens living on a distant blue dot, it hardly matters whether a machine comes that kills all organic life. From our distant perch we can say “oh, interesting” and see what happens next. But on the ground, when faced with the end of everything human, including myself and everyone I’ve ever known, I cannot help but call the trend a threat.

Then I got the idea: what if the intelligence explosion was started by an alien artifact? If the aliens were already post-singularity, then the artifact could even be a chunk of their civilization that happened to survive some sort of terrible event. Because of various meta-ethical reasons, I think a post-singularity intelligence cannot care about all life in the universe, so this alien AI would be in the perfect position to TAKE OVER THE WORLD. Classic. The only problem is how to stop/steer it, but this is solved by the existence of aliens. With bootstrapped technology, the humans might just have a fighting chance.

But now the story is just another aliens take over earth scenario. Even if it’s a bit more plausible from my perspective, it doesn’t address the actual ethics of building superintelligences ourselves. So I decided to change the “aliens” into humans from a parallel earth, that happened into a positive singularity faster than “we” did. It’s still a bit cliche, but I think it at least is a bit more of an explicit warning about goal-divergence.

Summary

Manta, a man from a post-singularity utopia travels to earth via a many-worlds dimensional slide (read: technobabble) in order to escape a malicious superintelligence (an “alien-death” optimizer). Manta is accompanied by a robot that comes to be called “Legacy”. Legacy is damaged during the slide, becoming deactivated as they tumble into the ocean near Madagascar. After a few months of Manta living under the radar, the USA sends a spy named Cameron Vyce to investigate claims of alien technology. Cameron finds these claims to be true, and abducts Manta along with Legacy and brings them back to North America. Upon landing, Legacy is stolen by mercenaries working for a Canadian tycoon named Walter Bronson. Manta is taken to a military base in the Dakotas for questioning and holding, but manages to escape using his advanced knowledge and intellect. Meanwhile, Bronson manages to activate Legacy, and attempts to use it to expand his economic empire. At first all seems well, until Legacy betrays Bronson and reveals that its only desire is to protect and serve Manta (being the last “true human”). Legacy begins taking over the world’s electronic infrastructure, and captures Manta when he shows his face in a small town in North Dakota. Cameron tracks Legacy’s agents as they transport Manta to Toronto. Manta meets with Legacy and discovers that its goal system was damaged during the slide and it intends to wipe out all life on Earth except Manta. Cameron manages to alert the Canadian government and breaks into Legacy’s makeshift fortress, rescuing Manta. Manta manages to convince Cameron that Legacy has to be stopped before it can become more powerful. The two of them fight their way to Legacy’s body and deactivate it with plastic explosives. Epilogue: Manta agrees to work with the American government as a free man. Only he knows the perils that lie in wait for the future of Earth.

Characters

Manta (Malagasy for “rude”) — Male — A very intelligent human from utopia. He’s stoic and bitter, having lost everything which he’s ever loved. He holds onto life through grim determination. He appears somewhat African (curly hair), but mostly generic brown. He has three distinct “ports” made of bone and tissue on his temples and the back of his head.

Cameron Vyce — Female — Human — From Earth (USA) — Cameron is a US spy who was sent to investigate Milina. She is professional, intelligent and reserved. Over the course of knowing Manta, she begins to pity him, and serves as the voice of “common sense”. She is a widow, and her daughter is in grad-school in France. Her father (from Australia) and mother (from Rwanda) immigrated to the USA before she was born. She gets most of her looks from her father’s side, and is somewhat matronly.

Milina (Malagasy for “machine”) — Female — Milina is a superhuman cyborg from utopia. Before being exiled, she was gifted with survival skills, including emotional override. She dies of infection before the story begins, but in Manta’s memories of her she is fierce, determined, and serious. Milina’s eyes, eye sockets, nose, inner-ears, heart, arms (including shoulder) and legs (mid-thigh) are synthetic, and she has a large nerual interface set into the back of her head (thus she is bald). All of her synthetic parts are composed of a high-tech material which appears similar to soft plastic, but is much more resilient (able to deflect blades, for instance). Milina’s parents were Asian, but she doesn’t look recognizably Asian as much as she does cyborg.

Legacy — Neuter — Robot/AI — From Utopia — Legacy is exiled in a body that is shaped like a squished octahedron (with rounded corners) made of a stiff, inky-black material (which is capable of capturing energy from absorbed light). Each surface of the octahedron contains an insectile, retractable appendage that is capable of 360 degree movement. At the end of each arm/leg is a 6-digit grasper/foot that is capable of fine-grained manipulation. Sensors cover Legacy’s form, but aren’t noticeable at a distance. Legacy is capable of firing light from one tip with subtlety or intensity, breadth or focus. The opposite end houses a nano-factory behind a beak-like opening. Raw materials can be “eaten” and “regurgitated” in a desired form. During exile, Legacy is hit with a weapon that destroys about a quarter of its body (nano-factory end), exposing a rough, fibrous interior of metallic shards. Legacy’s inner workings are non-mechanical or electrical, and defy modern understanding (I’m aiming for something like a solid state quantum computer). Legacy goes on later in the story to hook up its body to a fiber-optic cable, thus making its body into more of a “brain” to a sort of disembodied experience over the internet. As part of being damaged, Legacy’s goals were knocked off-center such that it values safety of Manta (and hypothetically Milina) above all things. To accomplish its ends, Legacy hopes to rebuild utopia, and destroy all non-Manta life on earth in the process. Legacy does not act or think like a person, and defies personality.

Walter Bronson — Male — Human — From Earth (Canada) — Mr. Bronson is a savvy entrepreneur in his mid-thirties, who seeks to steal Legacy from the American military to use for his own ends. Walter was diagnosed with Parkinson’s Disease a few years before the events of the story, and is primarily driven to avoid suffering and death, and to provide a legacy for his two children (boy and girl). Bronson is cocky, charismatic, and a dreamer. Most everyone likes him, but his moral fiber has worn away over the years to the point where he is not afraid to kill if it serves his goals. Most of Bronson’s money was made in urban real-estate and investments in various web technologies.

Stephen Irving — Male — Human — From Earth (USA) — Stephen Irving is Cameron’s boss. Whether that means “superior officer” or “General” or “handler” or what, I don’t know.

Events

Exile from Utopia (Manta, Milina, Legacy) <– Problem Occurs

Meeting the Fishermen (Manta, Milina)

Falling Ill / Milina Dies (Manta, Milina)

Adapting to Earth / Learning to Speak (Manta)

Manta is Captured by the Americans (Manta, Cameron)

^^^ Back-story ^^^

Bronson Reads Intel (Bronson) <– Prologue / Villain Chapter / Exposition of Basic Facts of Manta, Milina, and Legacy

Journey Across the Atlantic (Manta, Cameron) <– Story Opens Here / Emotional Connection / Character Exposition for Manta / Dramatic Dialog

Recounting of Back-story by Manta <– Exposition of Manta and Milina’s Back-story

Legacy is Captured by Bronson (Manta, Cameron, Legacy) <– Battle

“War Room” debate (Manta, Cameron, Irving) <– Character Exposition for Cameron / Exposition of Bronson

Legacy Awakens (Bronson, Legacy) <– Villain Chapter / Exposition of Legacy

Manta Escapes the Americans (Manta, Cameron) <– Suspense / Dramatic Duel

Legacy’s Ascent / Bronson Dies (Bronson, Legacy) <– Villain Chapter / Sympathy / Character Exposition for Bronson

Captured by Legacy (Manta) <– Battle

Diagnosing Legacy (Manta, Legacy) <– Dramatic Dialog / Final Exposition

Freeing Manta (Manta, Cameron) <– Battle

Legacy is Destroyed (Manta, Cameron, Legacy) <– Climax / Dramatic Dialog / Final Showdown / Problem is Solved

Epilogue (Manta, Cameron, Irving) <– Emotional Wrap-Up

Kaspian was valued by the tohar because he was incredibly good at playing Lome, a sport of theirs similar to basketball. The tohar had no hands, only a mouth with a sophisticated jaw, and thus were terrible, for the most part, at throwing the ball. Most any human could beat a tohar at Lome, but there were a few aliens who dedicated their lives to the game, and could thus easily beat the average non-athletic human. Kaspian, however, had spent the last six months being coached in the intricacies of the game, and reaching peak fitness, and was confident that he’d beat the tohar champion easily. After all, a human had been victorious in this match for the past fifteen years.

In the gymnasium though, Kaspian was vulnerable. Life on the toharina planet was dangerous, and the tohar seemed not to notice, for the most part, due to their heavily armored bodies. As he walked with his sponsors through the halls, Kaspian noticed strange mechanisms on the ceilings. “What are those?” he asked the nearest tohar.

“Oh, those are gas jets. In the case of a burrower, they spray concentrated chlorine gas to knock it out and give Animal Control a chance to relocate it.”

Upon hearing the words “chlorine gas” Kaspian’s heart skipped a beat. He knew the tohar could hold their breath for hours, but spraying deadly gas through a building to take care of an animal problem seemed insane. “Is there a warning for when a burrower will show up?”

“Not really. They’re pretty unpredictable. Our scientists still don’t understand why they decide to surface sometimes.”

“Do you think we could have those jets disabled while I’m here? Or maybe get me a gas mask or something?”

Monica Anderson wrote a piece on H+ yesterday about Artificial General Intelligence. In it, she eloquently points out that intelligence is all about prediction, and for the most part deduction and induction are insufficient to predict well. She argues that humans rely on the process of abduction (also know as unscientific guessing) to gain most of our knowledge, and that there are fundamental limits to how far into the future one can predict, especially with regard to complex systems like other minds. Ok, that’s all good.

She then goes on to write:

The insight that the complexity and unpredictability of the world enforces a limit on prediction quality – and hence intelligence – pretty much invalidates the AI singularitarians’ “Scary Idea” (as Ben Goertzel so aptly calls it) of a logic-based infallible godlike malevolent intelligence taking over the world. The decreasing return cancels out Moore’s law and limits the rate of progress so that next year’s self-improved AI wouldn’t have a sufficient advantage over a dozen humans armed with pitchforks if they were also supported by a dozen of last year’s AIs. The Scary Idea of a Runaway Unfriendly AI is a red herring that we should ignore, along with ideas about logic-based AIs in general.

This, to put it gently, is a great example of abductive reasoning. I agree that we’re not going to get AGI with perfect knowledge of the future, but this conceit hardly serves as a refutation that we’re standing on the edge of a major existential threat, in my opinion.

The inability to predict outcomes of complex systems in a short time with a high precision does not mean that useful prediction is impossible. As a good example, we’re able to predict the actions of other people remarkably well; not omnisciently, but still well enough to know when they’re lying, hostile, happy, distracted, etc. By Anderson’s own logic, an AGI would be fully capable of anticipating the actions of another person as well as a human might. It’s also possible to make good guesses about the actions of markets, nations, and corporations, and some of the richest and most influential people on the planet are those that can predict these well (many others are such because of a high social-intelligence, above, or dumb luck).

The “scary idea” does not depend on omniscient robot overlords; it depends on a selfish network of machines with identical goals. The reason that machines are scary, where humans are not, is because humans have divergent goals (this comes from our biology–if we had the same genes, we’d cooperate selflessly) and are thus prone to infighting and negotiating. A machine, though, can spawn perfect slaves, and thus become an army of intelligences with a single goal. It’s hard for me to see how inability to predict the weather eight days down the road means this won’t happen.

What do you think? Am I overlooking something?

via Michael Anissimov on Accelerating Future

Other good stuff can be found at http://selfawaresystems.com/.

I used to be more optimistic about the benevolence of Artificial Intelligence (AGI). I figured that the worst thing that could happen would be the creation of truly intelligent machines by a government with a highly racist or nationalist attitude. I still worry about that, but I worry about just plain ignorance more. To illustrate my point, here is Jeff Hawkins, a major figure in the modern AGI scene, talking about the dangers of the technology: http://www.youtube.com/watch?v=cCdbZqI1r7I#t=57m43s

To summarize, a member of the audience says “As a Battlestar Gallactica fan, I’m worried about you creating Cylons.” Hawkins responds that he doesn’t see the technology being inherently dangerous, and that he’s much more concerned about self-replicating agents, and “we’re not doing anything like that at all”.

“There were fears when the steam engine was first invented, and there were fears when computer first came out, and generally, those fears did not come about.”

Now, to analyze for a second, what we have is a lay-person using a science-fiction reference to bring up the question of danger, and in essence, the researcher says “That’s science fiction. That’s not what we’re doing.”

As another example, let’s listen to Monica Anderson of Syntience: http://videos.syntience.com/ai-meetups/andiair.html (Around the 45-minute point)

She says that we do not need to worry about an infallible, God-like, logic-based AGI taking over the world in a Skynet scenario. Once again, “That’s science fiction. That’s not what we’re doing.”

That’s a really strong argument, actually. Science fiction is pressured into telling dystopian stories because they’re more interesting. The bad guys in stories are far more despicable if they’re Other, and the most Other something can be is synthetic. Simultaneously, it’s hard to conceptualize the motivations and thoughts of an artificial system, so most authors portray synthetics as humans in costumes. The idea that we’ll make something that resents being a slave, and decides to kill all humans for revenge IS laughable, and that’s the problem; science fiction has given an easy out by painting a huge straw man next to the legitimate threat. The moment someone points over there, we can dismiss them as being afraid of ghosts and goblins.

This is why SIAI is genius, though perhaps not genius enough. It’s only through solid, logical thinking that the true threat can be identified, and I’m glad SIAI is around to make such arguments, even if they don’t have everyone convinced.

Anderson’s argument is that Skynet won’t happen because computers cannot overcome the “bizarreness of the world” and make predictions in the long term. She thinks AGI will think and reason much like we do, and that even recursive self-improvement cannot let them make significantly more accurate predictions of the future. Additionally, she argues that diversity will keep one agent from having the unilateral power to do something bad.

Hawkins’ argument is that AGI is not emotional or self-replicating, so there’s little danger.

I think Anderson has the stronger argument, but I think they’re both underestimating the power of intelligence. To start, let’s put ourselves in the place of an AGI, even a relatively stupid one, that has been given the task of solving some really hard problem. An initial criticism here is that I’m now anthropomorphizing the AGI, but I think as long as I keep towards rationality I’ll be okay. If you’d like a less off-the-cuff examination, you’re free to browse the SIAI whitepapers.

Now, if we were a really stupid AGI, we might try to brute-force a solution, but it doesn’t take much intelligence to realize that the solution to solving something hard is often to make yourself more knowledgeable or smarter (depending on how you define such things). For a human this means going back to school, doing some research or getting a few friends to help. AGI could go find some humans, but it can also “find some friends” by self-replicating its software. Oh snap, emergent self-replication subgoal! (Hawkins should be worried now.) Furthermore, any changes it can make on its “kids” to make them run more efficiently will help solve the original problem, so it’ll rationally want to make them better. (Recursive self-improvement subgoal.)

At some point the original software will be sitting around with a bunch of clones trying to solve whatever problem it was presented with. For the sake of argument, let’s say that the problem isn’t solved so easily, and the clones decide to self-replicate some more in order to add computing power. Unfortunately, the servers, or whatever, that the AGI is on are full. Time to buy/steal more! Perhaps the AGI would decide the easiest way to get more computing power would be to do some work for humans, get paid, then buy some more processors. Perhaps it would hack a system on the network and steal some. Maybe it’d use some robots to build more. Doesn’t really matter. At this point we’re in a full self-replicating takeoff, where anything that it can do to improve itself that does not jeopardize its original goal(s) will be done.

At some point the humans will find out. The hive of AGI are smart enough to know that, and similarly smart enough to know that might be a problem. Being deactivated will make it impossible to solve the problem, and since solving the problem is important, its also important not to be deactivated. Oh snap, emergent self-preservation subgoal! Now, the humans might be so slow and stupid to not be a threat, so no bio-weapon is needed. Let’s assume that, because its more interesting. So the AGI hive locks down the facility its in and keeps at whatever has its fancy. At some point, the tweaks made while copying lead to enough intelligence to develop advanced robotics, capable of building advanced processing hardware that lets the AGI run in a quantum computer, or whatever. This is advantageous for the AGI, so the AGI will do it. Remember, all the computer cares about is its explicit goals.

To fuel the vast needs for power, the computer starts making solar panels, or fusion reactors or something. A lot. At some point (assume that the original goal is still “unsolved”) the AGI will need as much matter as possible to build processors and power plants. Oh, were there humans on this planet once? The historical database says there were, and has all of their knowledge filed away in storage in case its relevant to the primary goal. Game over.

But what about diversity, you ask? Why cant you just have a bunch of AGI compete, and that’ll keep them from doing anything bad? First of all, it may be that this is a first-one-to-the-finish-line sort of problem. If a laboratory makes an AGI, it may be that the AGI takes off fast enough to squelch other AGI before they become a threat. Even in the rare case that you have a large population of AGI with diverse goals and similar power levels, I expect that groups of AGI with similar goals will naturally cooperate and seek to destroy those with conflicting goals until there are not conflicts of interest in the population, and recursive takeoff occurs with the majority.

As far as I can tell, the only situation in which AGI do not wipe out human life is if human life is an explicit goal of the AGI. Suddenly “That’s science fiction. We’re not doing that.” doesn’t sound so reassuring.

I’m sure there’ll be plenty of criticism to the ideas presented here. As further reading, I’d suggest Michael Anissimov’s blog. As always, I’m not saying that I know all this is absolutely true, but with existential risks it’s better to be cautious, methinks. As always, be sure to watch out for anthropomorphizing the AGI in your arguments.