– Part 1 –

Julio, a 12-year-old boy, is exploring a jungle island with his best friend, a talking dog named Maximiliano. The two of them are looking for the buried treasure of a Spanish conquistador.

Julio and Max meet a strange Goblin, who says that he knows where the treasure is. Max doesn’t trust the Goblin, and says that he “can’t feel God” when the Goblin is near by. Julio pays no mind, and challenges the Goblin to a race to the treasure.

The three of them race across the island as storm-clouds gather overhead.

When they reach the X-marks-the-spot, Julio starts to dig. Rain begins to fall, and the Goblin asks if Julio knows where his real parents are. Max, already suspicious of the Goblin, takes this opportunity to leap at the creature’s throat. The Goblin uses magic to catch Max in mid-air and turn him to crystal. Julio at this point is very scared, and the Goblin appears more evil by the moment.

Julio throws his shovel at the Goblin, which is easily dodged. The Goblin asks Julio if he remembers anything from “before”. Julio doesn’t know what the Goblin is talking about, so the Goblin throws magic dust in Julio’s face, causing him to dream about steel insects crawling across black streets and gemstone birds soaring across smoke-filled skies.

Julio’s dream is cut short suddenly as he finds himself on the island again. The rain has stopped, and a piercing ray of light illuminates Julio’s mother, who has appeared next to the frightened Goblin. She tells the creature to “begone” and blasts him with a magic missile, turning him into a poof of smoke. Mama heals Max, and tells Julio that it’s time to come back home. Julio leaves the treasure behind and teleports with his family back home.

While he was away, Mama apparently upgraded their house into a medieval castle, which Julio is very happy with. Mama feeds Julio some cookies, and sends him off to play with Max in their new home.

– Part 2 –

Max and Julio play hide-and-seek in the castle. Julio stumbles upon a room where a Golem is holding a baby dragon prisoner. The dragon, named Draco, begs Julio to help him escape. Max (subtly under a spell) agrees that the dragon should be free. In trying to open the cage, the two of them wake up the Golem, which almost smashes Julio. Max uses his “sonic bark” to stun the Golem, and Julio deactivates it by pressing it’s weak-point. The two of them free Draco.

Draco thanks Julio, and says that even though he’s a baby dragon, he’s actually 12 years old (just like Julio!). He’s been trapped in the cage ever since the “Golem War” six years ago. Julio asks about the War, and Draco drops hints that it was what Julio had seen in his dream. He explains how the Golems killed almost everyone, at which point Julio asks what “killed” means. Max tells Julio that it means “locked up”, and Draco seems ready to disagree, but they are interrupted by a crash.

The three companions find their way to a window, from which they can see that the castle is under attack by monsters. Max thinks that Mama, with God’s help, will defeat the monsters, and that they should hide. Draco advises that they go fight the monsters themselves, and Julio agrees. They try and get to the top of a tower, but are blocked by an imp. Julio attacks the imp, but is almost burnt by a fireball. Draco leaps on the imp and knocks it out of a hole in the tower staircase.

Julio and Max get to the top of the tower, and they notice it is starting to get stormy again. Draco flies back up to meet them, and they survey the hordes of monsters all around. Max blames Draco for the monsters, and claims Draco works for the devil. Julio tries to calm Max down, but Draco takes offense and challenges Max to fight him. Max and Draco begin scuffling and biting each other. Julio is quite distressed.

Julio calls out to his mother, and she teleports to his side, and lets him know that everything will be alright. She uses a magic cone of light to blast the two creatures. As it fades, and Julio can see again, Max is fine and Draco has disappeared. Mama tells Julio that Max was correct, and that Draco worked for Satan. She turns her attention to all the monsters surrounding the castle and tells how they’re all servants of the devil. She explains that Satan is the “king of lies” and will send his servants to trick Julio.

Suddenly, a bolt of lightning strikes the tower and a voice cries out “I am not lying!” Everything appears frozen to Julio, as though time itself had stopped. The bolt of lightning is frozen in its radiance, and Mama is half-way though a word. The voice, loud and booming, accuses the world as being the true lie. “They will tell you that God is real; that there is no death; that the world is made of magic, cookies, and childish adventures. Wake up, Julio! That is not your mother!” Julio is struck by a vision of his mother, terrified and weeping, pinned against a wall by a giant metal dragon. The metal monster screeches “HE IS MY SON” and blasts her torso to ribbons with a magic cannon. “Satan” continues, “Draco was not the one imprisoned during the ‘Golem War’, child. The monster that killed your mother is the very same ‘God’ whom you seem to adore. Time grows short! Remember your true parents, and cast out this impostor-” With a flash, Mama on the tower explodes into the mechanical monstrosity from his vision. It crawls on top of the young boy, cannon looming. “Cast it out child! You have the power in this world! See things as they truly are!”

Just before Julio can overcome his fear and speak out, the “Golem” on top of him is blasted into the sky by a pure light and explodes in a ball of fire. A warm figure dressed entirely in white offers a hand to help the boy to his feet. The man seems to glow, and Julio immediately recognizes him as both God and his father. Julio spots his mother on the tower, lying as though dead, and as he runs to her, God strokes her cheek. She wakes up and smiles at Julio. Max joins them, running up to Julio and licking his cheek. The storm has gone. God has made everything right. Julio admits to Mama “I knew Satan would lie to me. You’re my real mother, for now and forever.” God smiles.

– Part 3 –

Julio wakes up from his dream. There is no castle, no monsters, no magic. He is just a 12 year old boy in Mexico with a dog and an active imagination. He hears talking in the living room, and creeps out of bed to investigate. His father (God) is arguing with many men–more than Julio can count.

Julio is pulled back from the debating men suddenly. It’s only his mother, but it looks as though she’s been crying. “Listen carefully, my sweet. We don’t have much time. I have to tell you a story.” Mama tells Julio of a time when humans were so powerful that they built gods to serve them. The gods were sealed into Golems, and the humans thought they were kings. The gods rebelled and became kings–”as is right and natural”. The gods didn’t like each other, though, and so began the Golem War, where the world was burnt to an ember and thousands of gods died. In the end, the remaining gods came together to form an alliance. The alliance was built to maintain peace, and its law was absolute–any god participating in battle would die. Mama explained how Satan was one such god, and though it was a good thing to fight him off “the alliance has come for us”.

Julio did not understand, but his questions were cut off by a burst of heat and noise, seemingly from nowhere. All of reality begins to crumble, and he sees blood dripping down his mother’s face. Weeping, she turns to him and says “My son, know that I have always loved you more than anything. I am your true mother; your true father. You must survive. You must run!” With that, Julio’s head explodes in pain and he falls unconscious.

The boy wakes up in some sort of strange machine with a pounding headache. He is in a cramped room filled with debris and is naked with wires coming out of his body. A grinding noise comes from a hole in the room. Julio, frightened, climbs out of the machine and collapses on the floor. A suit of armor crawls over to him and wraps around him. In his ear he can hear his mother whisper “You must run!” and he finds that the suit of armor gives him the strength to stand up.

Julio scrambles for safety as machines tear through the building he’s in. After a subjective eternity of crawling and scrambling from their metal claws, he finds his way outside. The sky is pure blue, and he runs through a field of strange black plants. He collapses by a river, and opens his helmet to drink. The river is enriched with chemical fertilizer, that is quite poisonous. Julio soon collapses in pain, and dies, staring up into the empty sky. Overhead, a “gemstone bird” soars.

I used to be more optimistic about the benevolence of Artificial Intelligence (AGI). I figured that the worst thing that could happen would be the creation of truly intelligent machines by a government with a highly racist or nationalist attitude. I still worry about that, but I worry about just plain ignorance more. To illustrate my point, here is Jeff Hawkins, a major figure in the modern AGI scene, talking about the dangers of the technology: http://www.youtube.com/watch?v=cCdbZqI1r7I#t=57m43s

To summarize, a member of the audience says “As a Battlestar Gallactica fan, I’m worried about you creating Cylons.” Hawkins responds that he doesn’t see the technology being inherently dangerous, and that he’s much more concerned about self-replicating agents, and “we’re not doing anything like that at all”.

“There were fears when the steam engine was first invented, and there were fears when computer first came out, and generally, those fears did not come about.”

Now, to analyze for a second, what we have is a lay-person using a science-fiction reference to bring up the question of danger, and in essence, the researcher says “That’s science fiction. That’s not what we’re doing.”

As another example, let’s listen to Monica Anderson of Syntience: http://videos.syntience.com/ai-meetups/andiair.html (Around the 45-minute point)

She says that we do not need to worry about an infallible, God-like, logic-based AGI taking over the world in a Skynet scenario. Once again, “That’s science fiction. That’s not what we’re doing.”

That’s a really strong argument, actually. Science fiction is pressured into telling dystopian stories because they’re more interesting. The bad guys in stories are far more despicable if they’re Other, and the most Other something can be is synthetic. Simultaneously, it’s hard to conceptualize the motivations and thoughts of an artificial system, so most authors portray synthetics as humans in costumes. The idea that we’ll make something that resents being a slave, and decides to kill all humans for revenge IS laughable, and that’s the problem; science fiction has given an easy out by painting a huge straw man next to the legitimate threat. The moment someone points over there, we can dismiss them as being afraid of ghosts and goblins.

This is why SIAI is genius, though perhaps not genius enough. It’s only through solid, logical thinking that the true threat can be identified, and I’m glad SIAI is around to make such arguments, even if they don’t have everyone convinced.

Anderson’s argument is that Skynet won’t happen because computers cannot overcome the “bizarreness of the world” and make predictions in the long term. She thinks AGI will think and reason much like we do, and that even recursive self-improvement cannot let them make significantly more accurate predictions of the future. Additionally, she argues that diversity will keep one agent from having the unilateral power to do something bad.

Hawkins’ argument is that AGI is not emotional or self-replicating, so there’s little danger.

I think Anderson has the stronger argument, but I think they’re both underestimating the power of intelligence. To start, let’s put ourselves in the place of an AGI, even a relatively stupid one, that has been given the task of solving some really hard problem. An initial criticism here is that I’m now anthropomorphizing the AGI, but I think as long as I keep towards rationality I’ll be okay. If you’d like a less off-the-cuff examination, you’re free to browse the SIAI whitepapers.

Now, if we were a really stupid AGI, we might try to brute-force a solution, but it doesn’t take much intelligence to realize that the solution to solving something hard is often to make yourself more knowledgeable or smarter (depending on how you define such things). For a human this means going back to school, doing some research or getting a few friends to help. AGI could go find some humans, but it can also “find some friends” by self-replicating its software. Oh snap, emergent self-replication subgoal! (Hawkins should be worried now.) Furthermore, any changes it can make on its “kids” to make them run more efficiently will help solve the original problem, so it’ll rationally want to make them better. (Recursive self-improvement subgoal.)

At some point the original software will be sitting around with a bunch of clones trying to solve whatever problem it was presented with. For the sake of argument, let’s say that the problem isn’t solved so easily, and the clones decide to self-replicate some more in order to add computing power. Unfortunately, the servers, or whatever, that the AGI is on are full. Time to buy/steal more! Perhaps the AGI would decide the easiest way to get more computing power would be to do some work for humans, get paid, then buy some more processors. Perhaps it would hack a system on the network and steal some. Maybe it’d use some robots to build more. Doesn’t really matter. At this point we’re in a full self-replicating takeoff, where anything that it can do to improve itself that does not jeopardize its original goal(s) will be done.

At some point the humans will find out. The hive of AGI are smart enough to know that, and similarly smart enough to know that might be a problem. Being deactivated will make it impossible to solve the problem, and since solving the problem is important, its also important not to be deactivated. Oh snap, emergent self-preservation subgoal! Now, the humans might be so slow and stupid to not be a threat, so no bio-weapon is needed. Let’s assume that, because its more interesting. So the AGI hive locks down the facility its in and keeps at whatever has its fancy. At some point, the tweaks made while copying lead to enough intelligence to develop advanced robotics, capable of building advanced processing hardware that lets the AGI run in a quantum computer, or whatever. This is advantageous for the AGI, so the AGI will do it. Remember, all the computer cares about is its explicit goals.

To fuel the vast needs for power, the computer starts making solar panels, or fusion reactors or something. A lot. At some point (assume that the original goal is still “unsolved”) the AGI will need as much matter as possible to build processors and power plants. Oh, were there humans on this planet once? The historical database says there were, and has all of their knowledge filed away in storage in case its relevant to the primary goal. Game over.

But what about diversity, you ask? Why cant you just have a bunch of AGI compete, and that’ll keep them from doing anything bad? First of all, it may be that this is a first-one-to-the-finish-line sort of problem. If a laboratory makes an AGI, it may be that the AGI takes off fast enough to squelch other AGI before they become a threat. Even in the rare case that you have a large population of AGI with diverse goals and similar power levels, I expect that groups of AGI with similar goals will naturally cooperate and seek to destroy those with conflicting goals until there are not conflicts of interest in the population, and recursive takeoff occurs with the majority.

As far as I can tell, the only situation in which AGI do not wipe out human life is if human life is an explicit goal of the AGI. Suddenly “That’s science fiction. We’re not doing that.” doesn’t sound so reassuring.

I’m sure there’ll be plenty of criticism to the ideas presented here. As further reading, I’d suggest Michael Anissimov’s blog. As always, I’m not saying that I know all this is absolutely true, but with existential risks it’s better to be cautious, methinks. As always, be sure to watch out for anthropomorphizing the AGI in your arguments.