Let me elaborate on that last point, because I think it’s important. Most of what artificial intelligence, as a field, focuses on (and rightly so) is building a model of an environment. This learning involves taking raw sensory data and turning (or rather, integrating) it into a set of concepts and relationships. But with the simple case of a game, we don’t really need to learn what the rules are, or learn how to take actions; we can focus entirely on how to play well.

Playing well involves maximizing a value, which depends on the game. In poker, this value is “money owned”, while in chess it’s “enemy kings captured*”. (In chess, the game never progresses to the point where any kings are captured, but as far as goals are concerned, checkmating the king is identical to capturing it.) Another way of putting this is to point out that each game has a distinct goal. Playing well involves maximizing the degree to which this goal is met, which I will call “utility“.

* – It’s technically a little bit more complicated than that, because it’s possible to stalemate, and that’s not as bad as an outright loss.

So, when presented with several possible “moves” in a game, we want to pick the move that has the highest utility. I call this searching and choosing “being rational”. To make a game-playing AI then, I needed to build a Rational Agent.

To be a bit more formal about the agent’s task, it must output a move when given:

• A “state” where it currently is.
• A “rulebook” (a.k.a. model) for how the game works, including what moves can be made at the given state.
• A utility function that, given a state, returns the degree to which it meets a goal.

The utility function is conceptually separate for me from the rulebook, as it specifies exactly what this agent “cares about”, where the rules are objective and unchanging regardless of who you are. (Example: the rulebook for chess doesn’t tell you which player you are.) Another reason that I think of them as conceptually distinct is because as I’m defining the utility function, it is agnostic to consequences. This is a fancy way of saying that the utility function doesn’t pay attention to what might happen next; it’s firmly grounded in the “present” (i.e. whatever state it’s given). Because of this agnosticism, it doesn’t actually rely on any information about game dynamics (rules).

The job of considering consequences is exactly what the Rational Agent is built to do. It takes each action in turn and finds the utility of its immediate consequences, then adds the utility of the next state, and the utility of next… etc. In other words, it is the limit of the infinite sum of the utility function from t=0 to infinity.

Or at least, it would be if it were a PERFECTLY rational agent. The unfortunate reality (from a math perspective) is that most systems are chaotic enough that it is damn near impossible (and sometimes just plain impossible) to take such an integral. As an example, consider a game (like chess or whatever) where the rule is that your opponent will play exactly as you would in the other position. If you try and compute the consequences of an action, you have to consider the moves of the opponent, which will involve considering your moves, to an infinite regression.

I wrote a post on this problem several weeks ago. It had some fancy math that lets some infinite regressions be solved, but I couldn’t figure out any fancy tricks that worked for all situations. I intuitively don’t think it makes much sense trying to compute the limits, either. If I’m resigned to building an agent that is not perfectly rational, but is only somewhat rational, than I can simply say that at some point the agent stops recursively “imagining” the future, and works on the approximate sum.

The problem with this, however, is that many games are not won until very late (a.k.a. after many moves). Consider a game I invented to test my agent:

Counting

or perhaps “Lineland Treasurehunt”

You are at a position, and your goal is to get to another position. Possible moves are “walk forward” and “walk backwards”. Oh, and the spot you start at is called “0″ and the goal spot is called “100″. Each time you walk forward your position increases by one, and each time you walk backward, it decreases by one.

The problem is that, as defined, there’s no base utility in being at “99″ over “0″. The only utility in being at “99″ is in the consequence of being able to go to “100″ on the next turn. If each step is a move, then the agent needs to model 100 moves in the future to have any clue what to do. Now, imagine how many moves one would need to look-ahead in order to see a victory in chess. For a perfectly rational agent it’s a mathematical possibility, but for something that can be written in code it’s just impractically beyond the maximum recursion depth.

The only way an agent with a maximum look-ahead of 5 moves can beat the Counting game is to get the hint “high numbers are better than low ones”. According to the rules, this is not intrinsically true, but it’s an approximation of truth that allows the agent to perform under normal circumstances. It is a heuristic, and it is of vital importance.

To be more precise, a heuristic is a component of the utility function that has been added because it is a good approximation of future utility (i.e. the goal). Some heuristics in chess are “control the center of the board”, “capture (non-king) pieces”, and “threaten many squares”. Clearly you don’t want to focus on doing any of these if you have the opportunity to checkmate, but because checkmating takes time, it’s cognitively easier to aim for any of these more apparent mini-goals.

The learning of heuristics seems like the most important aspect of practical rationality, and it’s something I’m looking forward to thinking about more. It seems plausible to me that the emotion of happiness in humans may be a heuristic-encoding signal. When we’re happy or sad, we say “What’d I just do? Remember presence/absence of that as a heuristic!”. Just speculating…

Here’s my rational agent in Haskell: http://raelifin.com/files/code/rationality.tar.gz

(I also wrote a version in Java with many more example games, but I’m working on tuning that, and it’s not really suitable for posting at the moment. The Haskell is also much cleaner/prettier.)

N.B. This is almost certainly old ground. Reinforcement learning or alpha-beta pruning almost certainly would be good things for me to study. My ignorance on the subject is not meant to imply that there’s not already a lot out there.

Fair warning: I don’t know if I like this post. It may be because I’m tired of looking at the math, but it might also be due to incoherence, or a terribly weak ending. I’d rather post it than work on it more, so heads-up.

Let’s say there are two doors. Behind the left door is $500 and behind the right one is $2. You can only go through one door. Which door do you pick?

This is not a trick question.

I actually can’t say that you’ll pick the left door, as you’re a pretty complex (and probably somewhat random/noisy/stochastic) system. But let’s imagine we ask a robot the same question, and the robot is designed to maximize moolah. I theorize that the robot will (if programmed well) pick the left door because if the robot goes through the left door, it will get more money. Duh. This is simple.

Let’s make it more interesting. Let’s say that behind the left door is $10, and behind the right door is a 50% chance at $100 (and 50% of $0). So, which gives you more money? There’s no way to tell. If we want to program our robot to maximize money, we have to decide how it treats uncertainty. I’m going to define the robot as maximizing average dollars collected. In other words, whenever confronted with a probabilistic outcome, it multiplies the dollars earned in each possibility with the probability of said outcome.

AverageDollars(RightDoor) = $100 * 50% = $50
AverageDollars(LeftDoor) = $10 * 100% = $10
$50 > $10, so pick RightDoor

Now, what if the right door offered a (non-overlapping) 1/3 chance of $18, 1/3 chance of $9, and a 1/3 chance of $6. In this case, the AverageDollars is the sum of the products of each amount with their probabilities.

AverageDollars(RightDoor) = $18 * 1/3 + $9 * 1/3 + $6 * 1/3
= $6 + $3 + $2 = $11

If I define “Utility” as the same thing as AverageDollars, this can be generalized to:

Where:
U(x) = Utility of x
A = an Action (like picking the RightDoor or the LeftDoor)
O = an Outcome (how many dollars are behind a door)
P(O|A) = Probability of an Outcome given an Action

Yeah. It’s still pretty basic. You may be bored. Let me make things more interesting. Let’s say I offer you one of two kinds of boxes (Red or Blue), and inside each box is a random prize.

If you open a red box, you have a (non-overlapping) 50% chance of $5, and a %50 chance of $10. But if you open a blue box, you have a %50 chance of $2, and a %50 chance of finding $9 and a red box. Here’s how I work out the math:

In English, I lump the expected value of the red box in with the $9 when computing the value of the second possibility in the blue box.

Now, what if the red boxes were changed to have a %50 chance of $5, a %40 chance of $10, and a %10 chance of just a blue box?

Put these two functions into a robot and it’ll recursively compute for infinity. There will ever be potentially smaller boxes infinitely deep, according to the math. Thankfully, this problem has a simple solution. First, I’ll substitute the first equation into the second, expand the terms, and simplify.

Now, I’ll subtract both sides by U(BlueBox)/20, factor out the U components, and divide both sides by the new coefficient.

Ta-da! By waiting to evaluate the recursive term, we can group and factor it out, leaving a simple numerical output. (I ran into this problem when working on a game-playing AI, and I thought up the solution myself! It’s not all too fancy, but I’m happy that at least my algebra is paying off. ^_^) Here’s the general form of the method being applied:

As a reminder, here is the general form of the utility function from before (this time with each Outcome O passed through a Value function V, so that we’re allowed to use symbolic outcomes that aren’t just real numbers):

Now, if the system in question has any loops, the function cannot be computed in that form. A more computable function would break the function down to the form of the recursive factor method (m*U(A) + b or more simply b/(1-m)).

Where:
Vb(O,A) = The value of an Outcome, given an Action, ignoring recursive terms (i.e. those with U(A) as a factor).
Pb(O,A) = The probability of an Outcome, given an Action, ignoring recursive terms.
Vm(O,A) = The value of an Outcome, given an Action, divided by U(A), ignoring NON-recursive terms.
Pm(O,A) = You get the point.

This function isn’t perfect, but it’s pretty cool. The major edge-case that it fails on are piecewise functions with recursive terms in the condition, but it should work on problems resembling the red and blue boxes.

_ _ _
_ X _
O X _

The “O” player would pick randomly, because the “thought” of blocking X, and playing a drawn-out game was equally terrible as that of a quick death. Essentially, I had made an AI that really hated playing, and so would commit suicide.

Okay. Easily enough fixed. Never mind the half a day spent looking for bugs… Just keep coding…

I dropped the continual punishment/error factor and jacked up the punishment for lost games (keeping the old continual punishment factor for tie games). So now my AI played out to tie games (good)… with completely pathetic moves!

Here’s an example:

_ _ _ _ _ _ _ _ _

_ X _ _ _ _ _ _ _

O X _ _ _ _ _ _ _

O X _ _ _ _ _ _ X

O X _ _ _ _ _ O X

O X _ X _ _ _ O X

O X _ X O _ _ O X

O X X X O _ _ O X

O X X X O O _ O X

O X X X O O X O X

Now, what’s really fascinating is that it’s not playing randomly. If there’s ever a situation where one agent could win, the other blocks it. Even in complex situations where one agent can set up a sure victory in the future (a fork), the other blocks it. In other words, it’s defense is impeccable, but it has essentially no offense.

Thankfully I had stared at the code enough, and the past bug was fresh enough for me, that I didn’t need to debug for hours to find this one. The problem is I told each agent that the other one was perfectly rational. A perfectly rational agent makes no mistakes, and in Tic-Tac-Toe this means the best outcome you can hope for is a tie. Both agents knew this, and so every action (that didn’t lead to certain defeat) was equally futile. I had convinced my AI that the very game of Tic-Tac-Toe was pointless, so it stopped “trying” to win!

I tweaked the AI so that it thinks its opponent is only probably rational, but perhaps a little spontaneous. Now it plays aggressively. ^_^

I used to be more optimistic about the benevolence of Artificial Intelligence (AGI). I figured that the worst thing that could happen would be the creation of truly intelligent machines by a government with a highly racist or nationalist attitude. I still worry about that, but I worry about just plain ignorance more. To illustrate my point, here is Jeff Hawkins, a major figure in the modern AGI scene, talking about the dangers of the technology: http://www.youtube.com/watch?v=cCdbZqI1r7I#t=57m43s

To summarize, a member of the audience says “As a Battlestar Gallactica fan, I’m worried about you creating Cylons.” Hawkins responds that he doesn’t see the technology being inherently dangerous, and that he’s much more concerned about self-replicating agents, and “we’re not doing anything like that at all”.

“There were fears when the steam engine was first invented, and there were fears when computer first came out, and generally, those fears did not come about.”

Now, to analyze for a second, what we have is a lay-person using a science-fiction reference to bring up the question of danger, and in essence, the researcher says “That’s science fiction. That’s not what we’re doing.”

As another example, let’s listen to Monica Anderson of Syntience: http://videos.syntience.com/ai-meetups/andiair.html (Around the 45-minute point)

She says that we do not need to worry about an infallible, God-like, logic-based AGI taking over the world in a Skynet scenario. Once again, “That’s science fiction. That’s not what we’re doing.”

That’s a really strong argument, actually. Science fiction is pressured into telling dystopian stories because they’re more interesting. The bad guys in stories are far more despicable if they’re Other, and the most Other something can be is synthetic. Simultaneously, it’s hard to conceptualize the motivations and thoughts of an artificial system, so most authors portray synthetics as humans in costumes. The idea that we’ll make something that resents being a slave, and decides to kill all humans for revenge IS laughable, and that’s the problem; science fiction has given an easy out by painting a huge straw man next to the legitimate threat. The moment someone points over there, we can dismiss them as being afraid of ghosts and goblins.

This is why SIAI is genius, though perhaps not genius enough. It’s only through solid, logical thinking that the true threat can be identified, and I’m glad SIAI is around to make such arguments, even if they don’t have everyone convinced.

Anderson’s argument is that Skynet won’t happen because computers cannot overcome the “bizarreness of the world” and make predictions in the long term. She thinks AGI will think and reason much like we do, and that even recursive self-improvement cannot let them make significantly more accurate predictions of the future. Additionally, she argues that diversity will keep one agent from having the unilateral power to do something bad.

Hawkins’ argument is that AGI is not emotional or self-replicating, so there’s little danger.

I think Anderson has the stronger argument, but I think they’re both underestimating the power of intelligence. To start, let’s put ourselves in the place of an AGI, even a relatively stupid one, that has been given the task of solving some really hard problem. An initial criticism here is that I’m now anthropomorphizing the AGI, but I think as long as I keep towards rationality I’ll be okay. If you’d like a less off-the-cuff examination, you’re free to browse the SIAI whitepapers.

Now, if we were a really stupid AGI, we might try to brute-force a solution, but it doesn’t take much intelligence to realize that the solution to solving something hard is often to make yourself more knowledgeable or smarter (depending on how you define such things). For a human this means going back to school, doing some research or getting a few friends to help. AGI could go find some humans, but it can also “find some friends” by self-replicating its software. Oh snap, emergent self-replication subgoal! (Hawkins should be worried now.) Furthermore, any changes it can make on its “kids” to make them run more efficiently will help solve the original problem, so it’ll rationally want to make them better. (Recursive self-improvement subgoal.)

At some point the original software will be sitting around with a bunch of clones trying to solve whatever problem it was presented with. For the sake of argument, let’s say that the problem isn’t solved so easily, and the clones decide to self-replicate some more in order to add computing power. Unfortunately, the servers, or whatever, that the AGI is on are full. Time to buy/steal more! Perhaps the AGI would decide the easiest way to get more computing power would be to do some work for humans, get paid, then buy some more processors. Perhaps it would hack a system on the network and steal some. Maybe it’d use some robots to build more. Doesn’t really matter. At this point we’re in a full self-replicating takeoff, where anything that it can do to improve itself that does not jeopardize its original goal(s) will be done.

At some point the humans will find out. The hive of AGI are smart enough to know that, and similarly smart enough to know that might be a problem. Being deactivated will make it impossible to solve the problem, and since solving the problem is important, its also important not to be deactivated. Oh snap, emergent self-preservation subgoal! Now, the humans might be so slow and stupid to not be a threat, so no bio-weapon is needed. Let’s assume that, because its more interesting. So the AGI hive locks down the facility its in and keeps at whatever has its fancy. At some point, the tweaks made while copying lead to enough intelligence to develop advanced robotics, capable of building advanced processing hardware that lets the AGI run in a quantum computer, or whatever. This is advantageous for the AGI, so the AGI will do it. Remember, all the computer cares about is its explicit goals.

To fuel the vast needs for power, the computer starts making solar panels, or fusion reactors or something. A lot. At some point (assume that the original goal is still “unsolved”) the AGI will need as much matter as possible to build processors and power plants. Oh, were there humans on this planet once? The historical database says there were, and has all of their knowledge filed away in storage in case its relevant to the primary goal. Game over.

But what about diversity, you ask? Why cant you just have a bunch of AGI compete, and that’ll keep them from doing anything bad? First of all, it may be that this is a first-one-to-the-finish-line sort of problem. If a laboratory makes an AGI, it may be that the AGI takes off fast enough to squelch other AGI before they become a threat. Even in the rare case that you have a large population of AGI with diverse goals and similar power levels, I expect that groups of AGI with similar goals will naturally cooperate and seek to destroy those with conflicting goals until there are not conflicts of interest in the population, and recursive takeoff occurs with the majority.

As far as I can tell, the only situation in which AGI do not wipe out human life is if human life is an explicit goal of the AGI. Suddenly “That’s science fiction. We’re not doing that.” doesn’t sound so reassuring.

I’m sure there’ll be plenty of criticism to the ideas presented here. As further reading, I’d suggest Michael Anissimov’s blog. As always, I’m not saying that I know all this is absolutely true, but with existential risks it’s better to be cautious, methinks. As always, be sure to watch out for anthropomorphizing the AGI in your arguments.