I think we tend to deprecate all sorts of self-examining machines, even those we have in our own body, like feedback loops in nervous systems. I meant self-examining to mean “self-checking” or “awareness of what is going on at any level inside.” But this in a mostly mechanical way. Humans have a very faulty modeling system in that they learn values haphazardly (usually from parents or peers) and then run scenarios based on those values or do modeling based on some loose goals or whims that may not meet any specific strategy. You know as well as I do that most of NVC is uncovering and understanding strategies versus whims and blind reactions.

With evil AI’s the problem is goal setting and achievement. If you were to say to an AI, “there are too many people using this resource” would it interpret a strategy as “get rid of the people,” “find another resource,” or “find a resource alternative?” That’s also a human problem. If we design AI’s from the bottom up, they have to be AI’s that have very limited goals, like “move this rock from here to there.” If we can design them from top down, we have to make sure to put in some kind of ethical system so that they don’t wipe out humans or destroy the planet trying to implement that goal, which humans also tend to do. It may be that machines can work in tandem with humans (which they do now) in that the machine puts forth a lot of strategies which the humans can evaluate. Maybe you could have AI’s cooperate in that they cannot implement a goal without permissions from other AI’s (a government!) It is too complex for humans and may be too complex for machines to try not to be evil. The best humans can do is to run stuff in their minds like “do not hurt the other humans” and hope for the best, which may lead to accidentally hurting them or not being able to predict long term problems.

But what is interesting is what is the goal of humanity? Should AI have the same goal? If humans can’t agree to a goal or purpose, why do we think AI would? Competing strategies doesn’t work, because the people with the best values might not have the most efficient strategy.

You’ve got a real pickle, not just with AI’s but with humanity as well. I think AI will develop as “move the rock from here to there” kind of problem solving. “Why is my arm not working–hm, run self-examining program” type of thing. Already computers can do a far better job than humans in many cases (like checking my spelling) but they do not work by themselves. I envision a co-dependency which AI might not evolve out of.

It’s an interesting discussion, even if the solution is beyond the singularity.

mom

Regarding software:

In a way, our current understanding of the mind is pretty magical. It’s a bit like how alchemists viewed chemistry; we have some vague concepts, some of which are true and most are probably naive. Don’t confuse this with the mind actually be magical, though. We’re currently seeing a surge of scientific progress in neuroscience which seems to me like the tip of the iceberg. Instead of talking about human minds, let’s take a step back and pretend the goal was to make software that is nearly analogous to how a cockroach brain would perform. Is this possible? I think it could be done with a good amount of effort with modern technology. Now what if we wanted to replicate a snake brain? Far more difficult, but within reach. Probably no harder than the human genome project (Though nobody would give it that much funding. Who wants digital snakes?). Now imagine the difficulty in building a pigeon brain AFTER having built a snake brain. This is potentially much easier than just the snake, as many of the underlying systems are already well understood. It’d be hard, but no where near impossible. Crows seem like a short jump away, and crows are reasonably capable of problem solving. My point here is not that we’ll bridge the gap with animal “minds” but that the problem seems much more difficult because you’re taking it as one big jump, without considering how ideas might progress in the next half century.

Regarding parenting and recursion:

I think you’re falling into the anthropomorphic trap; you’re imagining that we’d be making a new kind of person, or even a new kind of brain. Biology is limited in a number of major ways, such as the difficulty in making direct edits during runtime. Humans have been trying to correct “faulty” brains for all of history, but these attempts have resulted in minimizing disability at best. Even brain drugs like anti-depressants are pathetically insufficient at accomplishing their goal and usually have collateral damage. Imagine if we were able to see directly into the brain as it runs, and have documentation for how it’s supposed to work. Even better, what if we could also alter specific things on a microscopic level (like chemical receptors), and see how that changes things.

“Growing up” is another nasty biological anachronism. Hypothetically, an artificial intelligence could be copied at various stages, or even have “memories” re-written. Coercion damage isn’t inherent; it’s the product of our need for autonomy. Humans evolved to resist being forced into doing things because it made us less likely to get our genes survive. Similarly, a machine intelligence might find it near-impossible to not try and improve upon itself, much as we find it near-impossible to not try and satisfy our needs. (This is also why it would probably never “bore” of anything we build into it.)

——————–

@Mom

How would a machine “know” about itself / self-examine without modeling itself? It’s hard to get to the idea “I am software” without first having the idea “I am”. We agree on the importance of non-human-ness of AI, and I think that there are quite a few AI developers who understand this. We still need to focus on human intelligence, though, because it’s a functioning example of the goal. It has the components we need to replicate, even if it has many we’d rather leave out, or avoid having to replicate. We humans tend to mentally turn everything else into a human (including the universe) though, so treating AI as a human seems par for the course.

I do want to discuss the dangers of a powerful AI though. First of all, keep in mind that we probably have the potential to imbue desire into an intelligence we make. This means that if, say, the US government were to make it, it might be given the task of “incapacitating terrorists.” That’s not good, because it’s a class of human which is okay to kill, and the AI has to decide who belongs in that class. Additionally, ambivalent AI might accidentally kill off all life, much as we’ve made various species extinct by pursuing our goals. Imagine if it decided to maximize solar collection by tidally locking the earth with the sun and pulling it to a closer orbit. Oops.

I can’t see the complete simulation of a human mind being easier than the creation of a recursive AI seed, so my guess is that uploading is a post-singularity technology. That kinda sucks because it makes it both hard to anticipate and not very interesting to discuss.

——————–

Thanks for the comments!

It also seems that being aware of one’s actions is easier than most humans make it out to be since they tend to be unaware due to running several different “theatres” at the same time. An AI might be better at ethical stuff just because it wouldn’t be obsessing about the recent death of it’s father, etc. Consciousness and emotional reactions to stimulated memories seem to mess up humans more often than help them. Psychological schools such as Buddhism work and work to free humans from the trap of acting in an emotional backlog. Most “bad” human action is due to this–most murders occur out of passion or as a reaction to an emotional state rather than as cold-blooded (note the term) decisions based on logic. Humans find it difficult to murder people if the decision is only logical–look at euthanasia.

I think that longevity is much easier than people make it out. Studies show that humans easily reach ages into their hundreds IF they have a clean environment. Most disease and “age related” problems are simply issues of toxicity. There is way, way too much evidence of people getting better when they cleaned up their diets or environments and way way too much evidence to support that cancers get worse with toxic environments to ignore this simple “cure” to age. Diet and exercise just happen to be really hard for people to do because of emotional problems or habituated problems such as sloth. But fixing these problems is WAY easier and WAY WAY less expensive than drug research that might extend life but not fix the underlying issues.

The AI thing is on the wrong track, in my opinion. I think that AI works best when it’s NOT like anything human. Why human? Especially when dealing with the human mind which is bound up in all kinds of things like biological impulses and chemical dumps that reinforce emotional issues. Even human logic and rational thinking is bound by human ideas that are based on experience as humans. It would be far better and far more efficient to “invest” in AI that is itself and not like anything human. As far as AI getting rid of the humans or attacking them, that’s just not very logical. Why would AI be interested in the humans except for the fact that the humans are too prevalent on the planet? If the humans were not in the way, so to speak, I don’t think AI would care about them. See, even the word “care” implies an emotional attachment.

But what about uploading? Or is that another topic entirely?

good post, Max!
mom

1. hardware issues
2. software issues
3. for lack of a better term, “parenting” issues
4. recursive improvement

For the sake of argument I’m going to assume we will soon meet whatever pure hardware requirements there are for running a consciousness. This point doesn’t interest me that much.

So the first super-difficult issue will be in creating software that is intelligent to any degree (even comparable to a “stupid” human or even one of the higher functioning members of the animal kingdom). This is no easy feat, and current research seems like it may be bogged down on important conceptual and epistemological questions (see: OpenCyc, which is a bit of a bucket theory of the mind approach to AI).

Even getting to this stage will require better theories about how consciousness works. Before we even get to the level of modeling digital synapses firing, we have to have an enormously better understanding of how all the pieces fit together. Developments in neuroscience will help, but this is to a large extent, i think, a *philosophical* problem. And hence its unsusceptibility to technological attack.

There also occurs the issue of parenting. The models of intelligence that we are familiar with, humans, require extensive parenting in order to get fully up to speed as mature individuals. Any AI we created would pose this as a problem to solve — how do you parent an AI? Obviously, lots of the existing tradition on parenting won’t be directly applicable. So you will have to figure out a bunch of it as you go along, probably making lots of mistakes. The AI will have lots of coercion damage. The AI may be irrational in various ways. It may not have lots of interest in recursively improving itself, or doing much of anything at all. Rather than a Kurzweilian paradigm of rational genius recursively improving itself into ubermenschdom, the first AI (or AIs) may be marginally interested intelligences riddled with neuroses.

And why not? Humans have very great potential, and are capable (to a high degree) of self-improvement and learning, but very few have ever fully actualized their potential. it seems pretty baseless to assume that changing hardware will solve the problems that have crippled the human race’s self-actualization *without* solving a bunch of intermediary problems first.

Finally, recursive improvement. The common transhumanist model is that an AI will want to figure out how to recursively improve itself and be able to do so quickly because….because why? Assuming that it has the desire to do so for the sake of argument, it will run into the same major issues discussed above: the “big picture” of consciousness question. Although presumably we will have a somewhat better model of consciousness, having been able to create this hypothetical AI in the first place, success in reverse-engineering version 1.0 of Rational Being hardly transfers into being able to offer improvements in any reasonable timeframe. Perhaps the argument is that the AI can just be run faster, and use its initially middling intelligence to try and improve itself at lightning speed.

This is somewhat plausible, but i think this scenario then demands we consider issues of interest and boredom. Maybe the AI will want to spend 1 million subjective years getting itself to version 2.0….maybe not. But it’s not obvious, and it’s definitely not inevitable.