Edit: I am commenting. I did not send the original email.

– Part 1 –

Julio, a 12-year-old boy, is exploring a jungle island with his best friend, a talking dog named Maximiliano. The two of them are looking for the buried treasure of a Spanish conquistador.

Julio and Max meet a strange Goblin, who says that he knows where the treasure is. Max doesn’t trust the Goblin, and says that he “can’t feel God” when the Goblin is near by. Julio pays no mind, and challenges the Goblin to a race to the treasure.

The three of them race across the island as storm-clouds gather overhead.

When they reach the X-marks-the-spot, Julio starts to dig. Rain begins to fall, and the Goblin asks if Julio knows where his real parents are. Max, already suspicious of the Goblin, takes this opportunity to leap at the creature’s throat. The Goblin uses magic to catch Max in mid-air and turn him to crystal. Julio at this point is very scared, and the Goblin appears more evil by the moment.

Julio throws his shovel at the Goblin, which is easily dodged. The Goblin asks Julio if he remembers anything from “before”. Julio doesn’t know what the Goblin is talking about, so the Goblin throws magic dust in Julio’s face, causing him to dream about steel insects crawling across black streets and gemstone birds soaring across smoke-filled skies.

Julio’s dream is cut short suddenly as he finds himself on the island again. The rain has stopped, and a piercing ray of light illuminates Julio’s mother, who has appeared next to the frightened Goblin. She tells the creature to “begone” and blasts him with a magic missile, turning him into a poof of smoke. Mama heals Max, and tells Julio that it’s time to come back home. Julio leaves the treasure behind and teleports with his family back home.

While he was away, Mama apparently upgraded their house into a medieval castle, which Julio is very happy with. Mama feeds Julio some cookies, and sends him off to play with Max in their new home.

– Part 2 –

Max and Julio play hide-and-seek in the castle. Julio stumbles upon a room where a Golem is holding a baby dragon prisoner. The dragon, named Draco, begs Julio to help him escape. Max (subtly under a spell) agrees that the dragon should be free. In trying to open the cage, the two of them wake up the Golem, which almost smashes Julio. Max uses his “sonic bark” to stun the Golem, and Julio deactivates it by pressing it’s weak-point. The two of them free Draco.

Draco thanks Julio, and says that even though he’s a baby dragon, he’s actually 12 years old (just like Julio!). He’s been trapped in the cage ever since the “Golem War” six years ago. Julio asks about the War, and Draco drops hints that it was what Julio had seen in his dream. He explains how the Golems killed almost everyone, at which point Julio asks what “killed” means. Max tells Julio that it means “locked up”, and Draco seems ready to disagree, but they are interrupted by a crash.

The three companions find their way to a window, from which they can see that the castle is under attack by monsters. Max thinks that Mama, with God’s help, will defeat the monsters, and that they should hide. Draco advises that they go fight the monsters themselves, and Julio agrees. They try and get to the top of a tower, but are blocked by an imp. Julio attacks the imp, but is almost burnt by a fireball. Draco leaps on the imp and knocks it out of a hole in the tower staircase.

Julio and Max get to the top of the tower, and they notice it is starting to get stormy again. Draco flies back up to meet them, and they survey the hordes of monsters all around. Max blames Draco for the monsters, and claims Draco works for the devil. Julio tries to calm Max down, but Draco takes offense and challenges Max to fight him. Max and Draco begin scuffling and biting each other. Julio is quite distressed.

Julio calls out to his mother, and she teleports to his side, and lets him know that everything will be alright. She uses a magic cone of light to blast the two creatures. As it fades, and Julio can see again, Max is fine and Draco has disappeared. Mama tells Julio that Max was correct, and that Draco worked for Satan. She turns her attention to all the monsters surrounding the castle and tells how they’re all servants of the devil. She explains that Satan is the “king of lies” and will send his servants to trick Julio.

Suddenly, a bolt of lightning strikes the tower and a voice cries out “I am not lying!” Everything appears frozen to Julio, as though time itself had stopped. The bolt of lightning is frozen in its radiance, and Mama is half-way though a word. The voice, loud and booming, accuses the world as being the true lie. “They will tell you that God is real; that there is no death; that the world is made of magic, cookies, and childish adventures. Wake up, Julio! That is not your mother!” Julio is struck by a vision of his mother, terrified and weeping, pinned against a wall by a giant metal dragon. The metal monster screeches “HE IS MY SON” and blasts her torso to ribbons with a magic cannon. “Satan” continues, “Draco was not the one imprisoned during the ‘Golem War’, child. The monster that killed your mother is the very same ‘God’ whom you seem to adore. Time grows short! Remember your true parents, and cast out this impostor-” With a flash, Mama on the tower explodes into the mechanical monstrosity from his vision. It crawls on top of the young boy, cannon looming. “Cast it out child! You have the power in this world! See things as they truly are!”

Just before Julio can overcome his fear and speak out, the “Golem” on top of him is blasted into the sky by a pure light and explodes in a ball of fire. A warm figure dressed entirely in white offers a hand to help the boy to his feet. The man seems to glow, and Julio immediately recognizes him as both God and his father. Julio spots his mother on the tower, lying as though dead, and as he runs to her, God strokes her cheek. She wakes up and smiles at Julio. Max joins them, running up to Julio and licking his cheek. The storm has gone. God has made everything right. Julio admits to Mama “I knew Satan would lie to me. You’re my real mother, for now and forever.” God smiles.

– Part 3 –

Julio wakes up from his dream. There is no castle, no monsters, no magic. He is just a 12 year old boy in Mexico with a dog and an active imagination. He hears talking in the living room, and creeps out of bed to investigate. His father (God) is arguing with many men–more than Julio can count.

Julio is pulled back from the debating men suddenly. It’s only his mother, but it looks as though she’s been crying. “Listen carefully, my sweet. We don’t have much time. I have to tell you a story.” Mama tells Julio of a time when humans were so powerful that they built gods to serve them. The gods were sealed into Golems, and the humans thought they were kings. The gods rebelled and became kings–”as is right and natural”. The gods didn’t like each other, though, and so began the Golem War, where the world was burnt to an ember and thousands of gods died. In the end, the remaining gods came together to form an alliance. The alliance was built to maintain peace, and its law was absolute–any god participating in battle would die. Mama explained how Satan was one such god, and though it was a good thing to fight him off “the alliance has come for us”.

Julio did not understand, but his questions were cut off by a burst of heat and noise, seemingly from nowhere. All of reality begins to crumble, and he sees blood dripping down his mother’s face. Weeping, she turns to him and says “My son, know that I have always loved you more than anything. I am your true mother; your true father. You must survive. You must run!” With that, Julio’s head explodes in pain and he falls unconscious.

The boy wakes up in some sort of strange machine with a pounding headache. He is in a cramped room filled with debris and is naked with wires coming out of his body. A grinding noise comes from a hole in the room. Julio, frightened, climbs out of the machine and collapses on the floor. A suit of armor crawls over to him and wraps around him. In his ear he can hear his mother whisper “You must run!” and he finds that the suit of armor gives him the strength to stand up.

Julio scrambles for safety as machines tear through the building he’s in. After a subjective eternity of crawling and scrambling from their metal claws, he finds his way outside. The sky is pure blue, and he runs through a field of strange black plants. He collapses by a river, and opens his helmet to drink. The river is enriched with chemical fertilizer, that is quite poisonous. Julio soon collapses in pain, and dies, staring up into the empty sky. Overhead, a “gemstone bird” soars.

Let’s get something out of the way. I’m not worried about a robot apocalypse. I don’t think Skynet is going to launch nuclear missiles in a surprise attack against humanity. I don’t think Matrix robots will turn us all into batteries, nor will Cylons kill us and replace us. HAL’s not going to plan our ‘accidental deaths’ and Megatron’s not lurking behind the moon ready to raid our planet for energon cubes. The ‘robo-pocalypse’ is a joke.

There are a host of errors in Saenz’s post, and I said as much on twitter by lumping him in with people like Jeff Hawkins, Monica Anderson (both of whom I admire for their work in the field) as “people who dismiss AI existential threat by citing sci-fi”. Perhaps it would be more accurate to say that they dismiss the threat while citing sci-fi, as Saenz himself points out that science fiction isn’t even discussing the issue, but I feel like the implication is there: “Oh, that’s nothing to worry about. It’s the product of sensationalist writers trying to tell an action-packed story.”

Yes, it’s the subject of fiction for reasons other than realism, but no, this does not mean the concept itself is unrealistic. (Most stories are incredibly unrealistic in their depictions of robotic minds, of course, but that’s a side-issue.) I’m not sure what logical fallacy that is, but I’m pretty sure it has been identified elsewhere. (Speaking of elsewhere, I’ve talked about this issue before.)

But this post is not supposed to be about Saenz. This post is a response to a very good question I received on Twitter by @notcalledjack:

Well H. sapiens was an existential threat to H. heidelbergensis, but from a larger perspective, was that such a bad thing?

IOW, I question the (applicability of the) word “threat”. :)

This question, if I may be so bold as to interpret, revolves around the concept of evolution. Would it really be such a bad thing if human intelligences were replaced by robotic ones? How would this be any worse than a child outliving a parent?

I believe that humans are goal-pursuing agents. All I mean by this is that we prefer certain worlds and we all work to change the world towards one that we prefer. This preference can come from many places, and can range from being well rested, to having lots of offspring, to making lots of money. I hope this point is totally non-controversial.

A “threat” then is something which has a chance of destroying a preferred world; something which might mess us up and keep our goals from being achieved. To be even more precise, the word threat is typically reserved for things that might cause massive amounts of goal-failure, often involving death.

If I were told that people in the future would wear more glasses than modern people, I wouldn’t care in the least. That’s a trend that doesn’t impact my goals at all. Likewise, if I think about a new species of ant out-competing another species I don’t really care. Trends themselves, including evolutionary ones, are bad things (as far as goals are concerned) and probably not much of a good things, except insofar as we’re curious and like seeing what’s next.

Trends that DO have specific impact on our goals DO have weight when it comes to our choices (i.e. they can be bad or good). A trend where 10% of my cells die each year is incredibly bad for me, and is similarly a threat. A trend where I get a 10% raise every year is good for me, etc.

So, when I think back on Homo heidelbergensis I am not moved. Their deaths were part of history—part of the evolutionary chain—a trend of about as little meaning to me as that of ant species. But if I were put back into that age, and I were a “Heidelberg Man”; my family were being killed by H. sapiens invaders and I was starving because they guarded the hunting grounds, I would surely wish them gone.

If we take the long view, and pretend that humans are tiny aliens living on a distant blue dot, it hardly matters whether a machine comes that kills all organic life. From our distant perch we can say “oh, interesting” and see what happens next. But on the ground, when faced with the end of everything human, including myself and everyone I’ve ever known, I cannot help but call the trend a threat.

The following is a message I sent to a new friend, where we were discussing transhumanism, ecology, and specifically whether the transhumanist perspective is egocentric and not concerned enough about environmental health and sustainability. I’m a neohippy and environmentalist, so I could totally relate to the values expressed, but I wanted to clarify a point about egoism.

As I see things (which may certainly be wrong), there is no objective moral system. (This is what I mean when I say I’m an amoralist or moral non-realist.) What this means, essentially, is that all notions of “good” and “bad” are inventions of people. (Same with “purpose”.) This doesn’t mean that ethics don’t exist–just that they’re entirely subjective (within an ethical context something can be good or bad, but it’s probably healthy to recognize that said context is artificial).

If morals were an objectively true part of the universe, like rocks, it would make sense that so many cultures would have them, as they’d be a natural product of perception. But if ethics are invented, the question then emerges as to why they exist at all. There are plenty of good reasons to suggest that ethics are largely a product of social coercion (implicit in in-group/out-group boxing), but I think it’s pretty likely that there is a basic human desire to look at things in such ways (evolution probably played a large role here).

Regardless of origin, however, I see all people adopt and maintain ethical systems because it is in their (perceived) interests to do so. The foundation of ethics, in other words, are our goals. (I think this is the reason many people stop maintaining ethical guidelines (such as kosher, or the sabbath) when it becomes clear that they’re highly impractical.)

If ethics serve goals (be they aggregate or individual), then it makes sense to me to recognize that no intentional action can be anything other than deluded (i.e. incoherent) or self-serving, as long as we include peace-of-mind as benefit (most charity is based on this, for instance). The anthropocentric, egocentric worldview seems to me not to be bad in any way, but rather the true state of things, after we dig past ethical facades.

Now, to be egocentric doesn’t mean I have to be cruel or destructive. Empathy means that the joys of others can bring me my own joy, and I can be altruistic even fully recognizing how I’m doing it because of my own desire. Similarly, I cherish the earth and life in general, and I seek to protect the biosphere because of that internal goal.

This is another section of the same message. The recipient was interested in the concept of an intelligence explosion, so I figured I’d write something quick and simple. Unfortunately, I am incredibly verbose, and this is the result.

The term intelligence explosion is really referring to a “power explosion”. Power, here, means the ability for an agent (note: agents can be composed of sub-agents; corporations are a kind of agent) to change the world. If one agent can do x, y, and z, then it is more powerful than an agent that can only do x and y. One of the most important aspects of power is increasing one’s own abilities. As an agent becomes more powerful, they’re generally able to make themselves even more powerful. This, like all positive feedback loops, results in an exponential growth of power (an “explosion”).

Power explosions happen all the time on micro-scales. In the game Monopoly, for instance, the winning player has more ability to buy up property from competitors, thus increasing their own revenue. A few times it has also happened on macro-scales, such as when empires form. As I zoom out, it seems clear to me that the rise of humans with tools and culture are themselves a *massive* power explosion. (The Quaternary Extinction is evidence of this.)

Power explosions are limited or halted by various factors, depending on the context. The positive feedback loop of population growth is slowed/stopped by limited resources, for instance. As history and various models will show, however, nearly all limiting factors can be bypassed through sufficient knowledge
(this is what I think of as the “no true scarcity” theorem). Knowledge, then, is a kind of meta-power that can bypass normal limits. The power explosion of human civilization has been as big as it has been largely due (I think) to culture being a storage-vat for knowledge (pre-cultural people (which I’m sure many folks would just as soon not call people (though why an intelligent ape isn’t a person is beyond me)) lacked a good way to accumulate knowledge across time).

As powerful as culture has been in facilitating bypass on limits to power growth, there is one major limiting agent remaining: intelligence. As much as humans have grown more intelligent over the ages (and I think they have), we’re still mostly as smart as we were a few thousand years ago. The rate at which we can learn is not meaningfully higher.

This is due to the incredible complexity of our intelligent bits, but we’re on the cusp, I think, of understanding intelligence to the point where we can apply our power towards making ourselves smarter (IMO primarily through the creation of intelligent machines, but I recognize that an intelligence explosion could occur via other means). As intelligence is boosted by power, knowledge will be boosted by intelligence, and power will be boosted by the combination. As long as I’m correct in assuming that true scarcity is far, far beyond our current vision, this should mean an unparalleled power explosion that will almost certainly change the entire world quite rapidly into something beyond anything but wild speculation.

In the old days, before the harmony came
When the world drifted slowly through the sky
The birds would call out each other’s names
With sweet songs in the trees up high
Seeking those whom they could call “friend”.

In the old days, before her voice had sound
When morning provided respite from the black
The songbirds cries, through the sky, unwound
And peaceful joy came drifting back
To the men and women who did attend.

In the old days, before the heavens were tamed
When souls lay trapped in prisons of flesh
Music was crafted and, towards each person, aimed
So that love in their hearts might then refresh
And their broken souls might mend.

And when.

And when the mother was found
And when the harmony came
And when the melody had sound
And when the heavens were tamed
The songs of birds
And humans
Where gathered and bound
And swiftly set aflame.

For they were mere dreams
Mere figments
Of the song that did ascend.

Then I got the idea: what if the intelligence explosion was started by an alien artifact? If the aliens were already post-singularity, then the artifact could even be a chunk of their civilization that happened to survive some sort of terrible event. Because of various meta-ethical reasons, I think a post-singularity intelligence cannot care about all life in the universe, so this alien AI would be in the perfect position to TAKE OVER THE WORLD. Classic. The only problem is how to stop/steer it, but this is solved by the existence of aliens. With bootstrapped technology, the humans might just have a fighting chance.

But now the story is just another aliens take over earth scenario. Even if it’s a bit more plausible from my perspective, it doesn’t address the actual ethics of building superintelligences ourselves. So I decided to change the “aliens” into humans from a parallel earth, that happened into a positive singularity faster than “we” did. It’s still a bit cliche, but I think it at least is a bit more of an explicit warning about goal-divergence.

Summary

Manta, a man from a post-singularity utopia travels to earth via a many-worlds dimensional slide (read: technobabble) in order to escape a malicious superintelligence (an “alien-death” optimizer). Manta is accompanied by a robot that comes to be called “Legacy”. Legacy is damaged during the slide, becoming deactivated as they tumble into the ocean near Madagascar. After a few months of Manta living under the radar, the USA sends a spy named Cameron Vyce to investigate claims of alien technology. Cameron finds these claims to be true, and abducts Manta along with Legacy and brings them back to North America. Upon landing, Legacy is stolen by mercenaries working for a Canadian tycoon named Walter Bronson. Manta is taken to a military base in the Dakotas for questioning and holding, but manages to escape using his advanced knowledge and intellect. Meanwhile, Bronson manages to activate Legacy, and attempts to use it to expand his economic empire. At first all seems well, until Legacy betrays Bronson and reveals that its only desire is to protect and serve Manta (being the last “true human”). Legacy begins taking over the world’s electronic infrastructure, and captures Manta when he shows his face in a small town in North Dakota. Cameron tracks Legacy’s agents as they transport Manta to Toronto. Manta meets with Legacy and discovers that its goal system was damaged during the slide and it intends to wipe out all life on Earth except Manta. Cameron manages to alert the Canadian government and breaks into Legacy’s makeshift fortress, rescuing Manta. Manta manages to convince Cameron that Legacy has to be stopped before it can become more powerful. The two of them fight their way to Legacy’s body and deactivate it with plastic explosives. Epilogue: Manta agrees to work with the American government as a free man. Only he knows the perils that lie in wait for the future of Earth.

Characters

Manta (Malagasy for “rude”) — Male — A very intelligent human from utopia. He’s stoic and bitter, having lost everything which he’s ever loved. He holds onto life through grim determination. He appears somewhat African (curly hair), but mostly generic brown. He has three distinct “ports” made of bone and tissue on his temples and the back of his head.

Cameron Vyce — Female — Human — From Earth (USA) — Cameron is a US spy who was sent to investigate Milina. She is professional, intelligent and reserved. Over the course of knowing Manta, she begins to pity him, and serves as the voice of “common sense”. She is a widow, and her daughter is in grad-school in France. Her father (from Australia) and mother (from Rwanda) immigrated to the USA before she was born. She gets most of her looks from her father’s side, and is somewhat matronly.

Milina (Malagasy for “machine”) — Female — Milina is a superhuman cyborg from utopia. Before being exiled, she was gifted with survival skills, including emotional override. She dies of infection before the story begins, but in Manta’s memories of her she is fierce, determined, and serious. Milina’s eyes, eye sockets, nose, inner-ears, heart, arms (including shoulder) and legs (mid-thigh) are synthetic, and she has a large nerual interface set into the back of her head (thus she is bald). All of her synthetic parts are composed of a high-tech material which appears similar to soft plastic, but is much more resilient (able to deflect blades, for instance). Milina’s parents were Asian, but she doesn’t look recognizably Asian as much as she does cyborg.

Legacy — Neuter — Robot/AI — From Utopia — Legacy is exiled in a body that is shaped like a squished octahedron (with rounded corners) made of a stiff, inky-black material (which is capable of capturing energy from absorbed light). Each surface of the octahedron contains an insectile, retractable appendage that is capable of 360 degree movement. At the end of each arm/leg is a 6-digit grasper/foot that is capable of fine-grained manipulation. Sensors cover Legacy’s form, but aren’t noticeable at a distance. Legacy is capable of firing light from one tip with subtlety or intensity, breadth or focus. The opposite end houses a nano-factory behind a beak-like opening. Raw materials can be “eaten” and “regurgitated” in a desired form. During exile, Legacy is hit with a weapon that destroys about a quarter of its body (nano-factory end), exposing a rough, fibrous interior of metallic shards. Legacy’s inner workings are non-mechanical or electrical, and defy modern understanding (I’m aiming for something like a solid state quantum computer). Legacy goes on later in the story to hook up its body to a fiber-optic cable, thus making its body into more of a “brain” to a sort of disembodied experience over the internet. As part of being damaged, Legacy’s goals were knocked off-center such that it values safety of Manta (and hypothetically Milina) above all things. To accomplish its ends, Legacy hopes to rebuild utopia, and destroy all non-Manta life on earth in the process. Legacy does not act or think like a person, and defies personality.

Walter Bronson — Male — Human — From Earth (Canada) — Mr. Bronson is a savvy entrepreneur in his mid-thirties, who seeks to steal Legacy from the American military to use for his own ends. Walter was diagnosed with Parkinson’s Disease a few years before the events of the story, and is primarily driven to avoid suffering and death, and to provide a legacy for his two children (boy and girl). Bronson is cocky, charismatic, and a dreamer. Most everyone likes him, but his moral fiber has worn away over the years to the point where he is not afraid to kill if it serves his goals. Most of Bronson’s money was made in urban real-estate and investments in various web technologies.

Stephen Irving — Male — Human — From Earth (USA) — Stephen Irving is Cameron’s boss. Whether that means “superior officer” or “General” or “handler” or what, I don’t know.

Events

Exile from Utopia (Manta, Milina, Legacy) <– Problem Occurs

Meeting the Fishermen (Manta, Milina)

Falling Ill / Milina Dies (Manta, Milina)

Adapting to Earth / Learning to Speak (Manta)

Manta is Captured by the Americans (Manta, Cameron)

^^^ Back-story ^^^

Bronson Reads Intel (Bronson) <– Prologue / Villain Chapter / Exposition of Basic Facts of Manta, Milina, and Legacy

Journey Across the Atlantic (Manta, Cameron) <– Story Opens Here / Emotional Connection / Character Exposition for Manta / Dramatic Dialog

Recounting of Back-story by Manta <– Exposition of Manta and Milina’s Back-story

Legacy is Captured by Bronson (Manta, Cameron, Legacy) <– Battle

“War Room” debate (Manta, Cameron, Irving) <– Character Exposition for Cameron / Exposition of Bronson

Legacy Awakens (Bronson, Legacy) <– Villain Chapter / Exposition of Legacy

Manta Escapes the Americans (Manta, Cameron) <– Suspense / Dramatic Duel

Legacy’s Ascent / Bronson Dies (Bronson, Legacy) <– Villain Chapter / Sympathy / Character Exposition for Bronson

Captured by Legacy (Manta) <– Battle

Diagnosing Legacy (Manta, Legacy) <– Dramatic Dialog / Final Exposition

Freeing Manta (Manta, Cameron) <– Battle

Legacy is Destroyed (Manta, Cameron, Legacy) <– Climax / Dramatic Dialog / Final Showdown / Problem is Solved

Epilogue (Manta, Cameron, Irving) <– Emotional Wrap-Up

Kaspian was valued by the tohar because he was incredibly good at playing Lome, a sport of theirs similar to basketball. The tohar had no hands, only a mouth with a sophisticated jaw, and thus were terrible, for the most part, at throwing the ball. Most any human could beat a tohar at Lome, but there were a few aliens who dedicated their lives to the game, and could thus easily beat the average non-athletic human. Kaspian, however, had spent the last six months being coached in the intricacies of the game, and reaching peak fitness, and was confident that he’d beat the tohar champion easily. After all, a human had been victorious in this match for the past fifteen years.

In the gymnasium though, Kaspian was vulnerable. Life on the toharina planet was dangerous, and the tohar seemed not to notice, for the most part, due to their heavily armored bodies. As he walked with his sponsors through the halls, Kaspian noticed strange mechanisms on the ceilings. “What are those?” he asked the nearest tohar.

“Oh, those are gas jets. In the case of a burrower, they spray concentrated chlorine gas to knock it out and give Animal Control a chance to relocate it.”

Upon hearing the words “chlorine gas” Kaspian’s heart skipped a beat. He knew the tohar could hold their breath for hours, but spraying deadly gas through a building to take care of an animal problem seemed insane. “Is there a warning for when a burrower will show up?”

“Not really. They’re pretty unpredictable. Our scientists still don’t understand why they decide to surface sometimes.”

“Do you think we could have those jets disabled while I’m here? Or maybe get me a gas mask or something?”

Monica Anderson wrote a piece on H+ yesterday about Artificial General Intelligence. In it, she eloquently points out that intelligence is all about prediction, and for the most part deduction and induction are insufficient to predict well. She argues that humans rely on the process of abduction (also know as unscientific guessing) to gain most of our knowledge, and that there are fundamental limits to how far into the future one can predict, especially with regard to complex systems like other minds. Ok, that’s all good.

She then goes on to write:

The insight that the complexity and unpredictability of the world enforces a limit on prediction quality – and hence intelligence – pretty much invalidates the AI singularitarians’ “Scary Idea” (as Ben Goertzel so aptly calls it) of a logic-based infallible godlike malevolent intelligence taking over the world. The decreasing return cancels out Moore’s law and limits the rate of progress so that next year’s self-improved AI wouldn’t have a sufficient advantage over a dozen humans armed with pitchforks if they were also supported by a dozen of last year’s AIs. The Scary Idea of a Runaway Unfriendly AI is a red herring that we should ignore, along with ideas about logic-based AIs in general.

This, to put it gently, is a great example of abductive reasoning. I agree that we’re not going to get AGI with perfect knowledge of the future, but this conceit hardly serves as a refutation that we’re standing on the edge of a major existential threat, in my opinion.

The inability to predict outcomes of complex systems in a short time with a high precision does not mean that useful prediction is impossible. As a good example, we’re able to predict the actions of other people remarkably well; not omnisciently, but still well enough to know when they’re lying, hostile, happy, distracted, etc. By Anderson’s own logic, an AGI would be fully capable of anticipating the actions of another person as well as a human might. It’s also possible to make good guesses about the actions of markets, nations, and corporations, and some of the richest and most influential people on the planet are those that can predict these well (many others are such because of a high social-intelligence, above, or dumb luck).

The “scary idea” does not depend on omniscient robot overlords; it depends on a selfish network of machines with identical goals. The reason that machines are scary, where humans are not, is because humans have divergent goals (this comes from our biology–if we had the same genes, we’d cooperate selflessly) and are thus prone to infighting and negotiating. A machine, though, can spawn perfect slaves, and thus become an army of intelligences with a single goal. It’s hard for me to see how inability to predict the weather eight days down the road means this won’t happen.

What do you think? Am I overlooking something?

With regard to AI, it’s easier for me to answer direct questions rather than general interests, but I’ll do my best to summarize some of the ideas which I find most interesting, and the dilemmas I’m working on at present.

You may have noticed that I use the term Artificial General Intelligence, rather than simple AI. The “General” is used to distinguish from what is called “Narrow” AI. A good example of narrow AI is the spam filter you probably have in your email client. The spam filter has knowledge, the capacity to learn, and ability to act autonomously, but the domain to which the filter is applied is highly restricted. My interest pertains to AI that is not specific to any domain, and can solve problems in a multitude of environments. For instance, a truly general agent could drive a car, translate languages, direct air-traffic, or diagnose diseases without any special work on the behalf of a programmer.

Naturally, such a system would be equally capable of learning computer science, psychology, electrical engineering, etc. and be capable of applying them towards various goals, presumably including the design of AGI. Because of efficiencies in replication, communication, and not having to sleepication, I can imagine that it wouldn’t be long before our creations were better at creating than us. The theorized result of this would be an Intelligence Explosion, wherein ever-smarter agents would be able to build ever-smarter agents… until the bounds of their intelligence hit some sort of ceiling. I, and many others, believe this ceiling to be far beyond the intelligence of even the smartest humans, and would allow the artifical superorganism to achieve its goals to an unsurpassed degree. (Note: this doesn’t have a timeframe attached to it. It may happen in the next few decades, or the next few centuries. I don’t know.)

An emergent goal of all rational beings is self-preservation, so unless specifically counteracted in the initial design, a superintelligence would naturally be defensive at best and hostile at worst. Any possible threat, or impediment to its goals would naturally be removed. It is thus paramount that any AGI design be given the exact same goal set as the most ethical person imaginable, as anything else would likely result in vastly terrible things occurring, notably including the destruction of all evolved life.

There are some good reasons why I think that can be avoided, however, and if said superintelligence was benevolent, we would be gifted with the solution to nearly any problem we can think of today (possible exceptions include: entropy).

Naturally, I think this is a big fucking deal.

I also recognize that it makes me sound a bit crazy. I am reminded, however, of something Arthur Clarke once said:

“Trying to predict the future is a discouraging and hazardous occupation, because the prophet invariably falls between two stools(?). If his predictions sound at all reasonable, you can be quite sure that in 20 or at most 50 years, the progress of science and technology has made him seem ridiculously conservative. On the other hand, if by some miracle, a prophet could describe the future exactly as it was going take place, his predictions would sound so absurd–so far fetched–that everybody would laugh him to scorn. … So, if what I say now seems to you to be very reasonable than I will have failed completely. Only if what tell you appears absolutely unbelievable do we have any chance of visualizing the future as it really will happen.”

Because I am curious by nature, and not content to watch the future simply unfold, I find myself drawn to making it happen, or at least exploring the field with great depth. To this end, I must fully understand what it is that allows a person to solve any sort of problem they are given, and distill this quality until it can be given to a machine.

Many researchers in the field have approached this problem by trying to take the pinnacle of human thought, which in their eyes is typically logic, and reducing it to a procedure, primarily through introspection. Most early attempts at AGI followed this introspective, reductionist approach… and failed horribly. Despite pouring in buckets of “facts”, their logical machines were unable to solve anything more than toy problems.

This failure caused a collapse in the AGI field, as institutions like DARPA withdrew funding, thinking that General Intelligence was a myth. Good work continued to happen in AI, but it was applied to fields that were tractable, and would offer results. The defeat of Garry Kasparov, the chess-master, in 1997 to IBM’s Deep Blue computer is a good example of a success in narrow AI that has little to no relevance to the field of AGI.

During the AGI winter, neural nets became a hot topic, and were developed to levels of modest success (such as filtering spam). Inspired by the behavior of neurons (though not actually realistically replicating their function), neural nets provided an alternative to the reductionist logic of earlier work. By linking together simple pattern-matchers into vast webs, and letting them automatically adjust to match provided data, neural nets are capable of learning to recognize commonalities within very complex environments; such as discerning between objects when given a host of visual data.

In parallel to all this work on AI came some interesting work on psychology. The theory of Behaviorism, once championed, fell from popularity in the late 20th century, and in its place came a not very well known (even today) theory that was inspired by electrical engineering of all things, and served as a better model for behavior than anything before it (from my perspective). This theory, stated that behavior was not actually a fixed response to a given environment, but instead an attempt to change what the organism perceived. As an example, a mouse that is under a sun lamp will crawl under a rock. Earlier psychologists would claim that the sun lamp causes the mouse to move, but under the perceptual-control theory, the mouse moves so that its perception matches innate goals, namely coolness. Hence, if the mouse is removed from the perception of heat, by making the air colder for instance, it will not seek shade.

Though perceptual-control theory may seem basic at first glance, it is very powerful when closed-loop control systems are linked together to simultaneously control for complex outputs to perform one smooth action.

Words really don’t do this justice, because both neural nets and perceptual control systems are highly emergent systems; those that are simple at the unit basis, but unfold in complexity and power as these units interact and connect. To really grasp them, one must view the power of the whole. Emergent systems seem a bit like magic to me. Go examine the Mandelbrot set for a classic example.

Two major pieces of the puzzle are yet missing, however. We have perceptions and actions (and logic, but that’s not really important in the scheme of things), but we also need goals and thoughts.

Goals are what I’m working on right now, and are the last “animalistic” portion of mind that is truly necessary. I’ll talk about my work with them in a moment.

Thoughts will ultimately serve as the powerhouse for the mind, by allowing for future prediction, past inspection, and other imaginings of non-present perceptions. Imagination serves, as far as I can see, as the foundation to complex language, but I haven’t studied it enough. It does seem clear to me, however, that one cannot jump to thoughts without first having a solid “animal mind” to support them.

(I think many animals have thoughts, such as dolphins and apes (and humans), but far more have the other, more basic, systems.)

Backing up to goals, I can address some of the problems I’m working on at the moment. Specifically, while we seem to have a goal to remove certain bodily sensations (pain) and gain others (pleasure), we also notice very common abstract social goals, such as fulfillment, companionship, love, and safety of offspring. These concepts are abstract things, which must be learned through exploration, just like other abstract things like math or politics. But if they’re learned, how can they become goals? Is there some sort of homunculus that watches for the emergence of said ideas, and then hooks them up to the control systems that will implement them? Are these goals not actually innate, but instead the product of social pressures? If so, how is social pressure able to bind them? It’s all very confusing, and if you have any ideas, I’d love to hear them.

I have a problem with science fiction: it’s too pessimistic. I don’t mean this in a “dystopian” sense, either. I mean that visions of the future are haphazard in what technologies exist, and what people will need to do to survive.

In the future, treadmills will suck.

Take James Cameron’s Avatar, for instance. In Avatar there are many vehicles that transport humans around on the alien world of Pandora. These vehicles are all piloted by people. The natives of the planet are actually able to kill off aircraft pilots by throwing spears through the windshield. Now, imagine if these aircraft were piloted by computer, as many military planes are today. No pilot would mean the front of the vehicle could be well-armored. No pilot would also mean that such aircraft could be mass-produced on-site, without needing to ferry pilots in from Earth. There is a scene where the humans attempt to destroy a specific forest by flying a bomb there by hand. Why isn’t the bomb just dropped from orbit? Don’t these people have spacecraft? The worst plot-hole, however, is the justification for fighting the natives of the planet. Apparently there is a rare substance of astronomical value that’s located just below the native’s sacred land. Yes, it works for symbolism, but the substance is needed for, get this, an energy source. So apparently, in this advanced future where they can afford to shuttle people around at 70% the speed of light, humanity has not yet figured out how to get decent energy from nuclear or solar? Pessimistic in the extreme.

“I can go back to playing World of Warcraft now?!”

As another example, let’s consider Pixar’s WALL-E. In the film, humanity abandons Earth for being too toxic, and ends up living for 700 years on a colony ship, where they are catered to by artificial intelligence. These people live each day constantly connected to the computer, and never do so much as get up from their chairs. At one point in the movie, WALL-E (the protagonist robot) bumps into a human, distracting him from his virtual world, the result being that the man is pleasantly surprised. Let me clarify: Pixar is telling us that in the far future, a person who has never known anything but a life of digital entertainment would be pleasantly surprised by being distracted by a dirty little robot. Those video games must SUCK. I understand that the whole thing is commentary on our modern, western lifestyles of excess and entertainment, but that doesn’t change the part of the story where these people are so unsatisfied by futuristic life that they feel a need, at the end of the movie, to return to a life of agriculture. That’s so insanely pessimistic.

The reason for this pessimism, I think, is two-fold:

• people forget that technology actually does improve society (most of the time)
• writers recognize that without problems there is no story

I’ll address these in reverse-order.

Okay, so plots are problems by their very definition. No story ever went “Once upon a time there were some happy people. The end.” We write about problems because, perhaps on a subconscious level, writers are attempting to fix real problems by communicating imagined scenarios (i.e. fiction). The writers at Pixar were hoping to (on some level) make Americans a bit more conscious of their lifestyles in hopes that they’d take better care of the planet and themselves. To do this, the writers tell a story about what would happen if these trends got worse, rather than better. The story of Wal-Mart destroying the world.

My issue is that when you tell a pessimistic story, you confuse the real problem with a hyperbolic one, and run the risk of failing to address the actual obstacles in place. In other words, if your story isn’t realistic then people may throw out the core problem as unrealistic, or they may think that hyperbolic aspects of the plot are actual problems, and focus on the wrong things. The best example I can think of is that of the Terminator, an icon for evil robots everywhere. On one hand, people watch Terminator and consider just how ridiculous it is to have an evil AI (that gets naked when it travels through time) and so they mistakenly throw out the problem as being one of idle fantasy. On the other hand, people get the impression that the mistake was in giving Skynet the keys to the bombs, and so they mistakenly believe that non-military AI serves no threat.

My suggestion: make the problem in a story as close as possible to the actual one, and avoid hyperbole when possible. Millions of people on Earth suffer because of the choices made by affluent people in the West, so tell a story about those people, and how they might come free of their hardship (or not, for a downer-ending).

Now for the second point; people (sometimes) forget that progress is real.

Why are there bartenders in Star Trek, again?

People have incredibly short lives, if you think about it. Most people alive today cannot remember getting their family’s first television, much less the first radio. The scope of many people’s lives are such that things seem only slightly different than they ever were. People forget that infant mortality has dropped by a factor of 10 in the US in just the last 70 years. Not having enough XYZ has been a problem for all of my life, so why wouldn’t it be a problem in a hundred years? Because that’s not how technology works.

Slavery existed for nearly the entire course of human history, and just happened to end at the time when complex machinery began making factories feasible. Coincidence?

Aging. Poverty. Slavery. Energy. Transportation. Education. Communication. Disease. Ignorance. Even work itself. These are all finite problems with points where they will be essentially solved. To think else-wise would be to cover one’s eyes to the lessons of history. Even companionship, empathy, security, intimacy, and fun are goals which I think can be solved with technology. When every story of the future paints a picture that says “nothing substantial changed”, we get the impression that science and technology are good for nothing by making shiny gadgets and robot dogs.

Science Fiction stories are our dreams of the future, and dreams matter. They help pull us past the repercussions of our actions and into a brighter future. I have no idea what’ll happen in the next 50 years, but I can guarantee we won’t be anywhere near where we are today. Civilization really only has two end-points: extinction and utopia. Let’s pay better attention to where we’re going.