Kaspian was valued by the tohar because he was incredibly good at playing Lome, a sport of theirs similar to basketball. The tohar had no hands, only a mouth with a sophisticated jaw, and thus were terrible, for the most part, at throwing the ball. Most any human could beat a tohar at Lome, but there were a few aliens who dedicated their lives to the game, and could thus easily beat the average non-athletic human. Kaspian, however, had spent the last six months being coached in the intricacies of the game, and reaching peak fitness, and was confident that he’d beat the tohar champion easily. After all, a human had been victorious in this match for the past fifteen years.

In the gymnasium though, Kaspian was vulnerable. Life on the toharina planet was dangerous, and the tohar seemed not to notice, for the most part, due to their heavily armored bodies. As he walked with his sponsors through the halls, Kaspian noticed strange mechanisms on the ceilings. “What are those?” he asked the nearest tohar.

“Oh, those are gas jets. In the case of a burrower, they spray concentrated chlorine gas to knock it out and give Animal Control a chance to relocate it.”

Upon hearing the words “chlorine gas” Kaspian’s heart skipped a beat. He knew the tohar could hold their breath for hours, but spraying deadly gas through a building to take care of an animal problem seemed insane. “Is there a warning for when a burrower will show up?”

“Not really. They’re pretty unpredictable. Our scientists still don’t understand why they decide to surface sometimes.”

“Do you think we could have those jets disabled while I’m here? Or maybe get me a gas mask or something?”

Monica Anderson wrote a piece on H+ yesterday about Artificial General Intelligence. In it, she eloquently points out that intelligence is all about prediction, and for the most part deduction and induction are insufficient to predict well. She argues that humans rely on the process of abduction (also know as unscientific guessing) to gain most of our knowledge, and that there are fundamental limits to how far into the future one can predict, especially with regard to complex systems like other minds. Ok, that’s all good.

She then goes on to write:

The insight that the complexity and unpredictability of the world enforces a limit on prediction quality – and hence intelligence – pretty much invalidates the AI singularitarians’ “Scary Idea” (as Ben Goertzel so aptly calls it) of a logic-based infallible godlike malevolent intelligence taking over the world. The decreasing return cancels out Moore’s law and limits the rate of progress so that next year’s self-improved AI wouldn’t have a sufficient advantage over a dozen humans armed with pitchforks if they were also supported by a dozen of last year’s AIs. The Scary Idea of a Runaway Unfriendly AI is a red herring that we should ignore, along with ideas about logic-based AIs in general.

This, to put it gently, is a great example of abductive reasoning. I agree that we’re not going to get AGI with perfect knowledge of the future, but this conceit hardly serves as a refutation that we’re standing on the edge of a major existential threat, in my opinion.

The inability to predict outcomes of complex systems in a short time with a high precision does not mean that useful prediction is impossible. As a good example, we’re able to predict the actions of other people remarkably well; not omnisciently, but still well enough to know when they’re lying, hostile, happy, distracted, etc. By Anderson’s own logic, an AGI would be fully capable of anticipating the actions of another person as well as a human might. It’s also possible to make good guesses about the actions of markets, nations, and corporations, and some of the richest and most influential people on the planet are those that can predict these well (many others are such because of a high social-intelligence, above, or dumb luck).

The “scary idea” does not depend on omniscient robot overlords; it depends on a selfish network of machines with identical goals. The reason that machines are scary, where humans are not, is because humans have divergent goals (this comes from our biology–if we had the same genes, we’d cooperate selflessly) and are thus prone to infighting and negotiating. A machine, though, can spawn perfect slaves, and thus become an army of intelligences with a single goal. It’s hard for me to see how inability to predict the weather eight days down the road means this won’t happen.

What do you think? Am I overlooking something?

For example, if I have a stone, I might form the hypothesis that the mass of the stone is 30g. To test this, I might weigh the stone. Implicit here is the idea that what is recorded from the scale is the mass of the stone. Once I read the scale, I have the following ideas:

(A) The scale says “45g”.
(B) The mass of the stone is 30g.
(C) The scale is an accurate measure of the stone’s mass.

If all three were true, there would be a contradiction, so I can conclude that one or more of my ideas must be false. The problem with falsification is that I have no logical reason to favor A & C (false hypothesis), over A & B (bad scale), B & C (bad eyesight), or others. Ultimately, I cannot refute anything with absolute certainty, so I cannot disprove.

This difficulty can be reduced by what I like to think of as the inductivist section of critical rationalism (I’ll show why in a moment). Wikipedia says that, with respect to hypotheses, “differentiation may be made on the basis of how much subjection to criticism they have received, [and] how severe such criticism has been”. In my example, none of the three ideas has been criticised, but it’s easy to imagine a scenario where the accuracy of the scale had been previously tested.

There are, unfortunately, two problems with this reasoning: (1) the scale requires evidence to test, so we still have the “which conjecture do we accept” problem at an earlier point, and (2) we still have two conjectures to decide between (A&B). What most critical rationalists will likely turn to is the difference between unsubstantiated conjectures (B) and those based on observation. It’s important to remember that hypothesis A is still conjectural, but we can grant it a sort of “natural critisism” stemming from our perception.

Here’s the rub: lending a heigher weight to any of our conjectures still doesn’t allow any refutation to logically occur. The only way to do that would be to accept something as true after enough observation, and this is exacly what everyone does (including critical rationalists), but CR brushes off as illegitimate.

At this point, one might turn to belief weights in order to avoid having to assign a binary value to a hypothesis (and the fallacy of inference). Unfortunately, any sort of updates to belief weights requires knowledge that is assumed to be true; it doesn’t actually let you move from a state of unsubstantiated conjecture to one of informed belief. In other words, it requires prior knowledge, which, as discussed earlier, we can’t logically obtain.

As an example, let’s say I have a hypothesis that a zebra exists and then I perceive a zebra. What weight do I give my hypothesis? In order to find it, I must know how accurate my perception is. For example, if hallucinating a zebra is equally probable to seeing a real zebra, there is a 50/50 chance that the zebra actually exists. But let’s say that I am not given a value for how accurate my perception is… how do I determine the likelihood of false positives, etc? The natural answer is to make a bunch of observations, and test to see if they were “correct”… except to do that, you’d need to assume the training labels (“correct/incorrect”) were true! If you want to evaluate the accuracy of the training labels, you have to assume some other input is true. The catch 22 ensures that you cannot logically produce a factual statement (even a probabilistic one) about the world without having been given other (binary) factual statements.

Unless I’m overlooking an infallible source of knowledge, I can conclude that nobody in the entire universe has any knowledge (that is, factual data) of the universe… and never will. Not even an infinite intelligence would be able to know anything about reality.

To escape this agnosticism, I might suggest that when we look like we’re doing logic, we’re actually not (at least, not formally). For instance, if the scale reads “45g”, I might simply accept that the stone is 45g and reject my old hypothesis, not through logic, but through common sense. The problem here is that common sense is a blanket term used to describe mental tasks that are easily done by people, but we don’t understand explicitly. Doing something via “common sense” is a lot like dying from “old age”; it’s just not a useful term. To make things worse, humans generally seem to reject paradox and use deduction, so we can be confident that something very close to formal logic is going on mentally.

My theory is that ideas are not evaluated based on truth, but based on the utility that comes from predictive power. Prediction, here, is based on sensory data, as opposed to objective reality. Unlike reality, we can be sure of our sensors as long as we think of the sensors as “inputs”. Let me give some examples…

I find a stone and decide to weigh it. I predict that the measured mass of the stone will be 30g. I put the stone on a scale, and it says 45g. My prediction had a significant error, so I discard it as being non-useful. Because I’d like to be able to predict the stone’s mass I form a new prediction that the mass is 45g (informed hypothesizing). I can use my memory to test the prediction… it works! This retrospective success reinforces the expected predictive power of that hypothesis. This explains why a hypothesis that matches previously observed data is granted more weight, and why one that doesn’t is discarded (falsification).

Let me give an example. Little Andrea sees a crow that is black. She conjectures that all crows are black (or more simply: “crows are black”). She sees another black crow. Prediction reinforced. She asks her mom what color crows are. Prediction reinforced. She sees a green apple (non-black non-crow). Observation is outside prediction scope; no change. At the age of 46, Andrea meets a street performer with an albino crow. Prediction failed. She notes “Oh, how strange… a white crow!” Prediction weakened slightly, but still retained, because in the vast majority of cases it’s useful to guess that crows are black. After seeing enough white crows she may reject her initial generalization and adopt a more probabilistic one (about 90% of ravens are black), but since a probabilistic idea has intrinsically less predictive power (and is harder for humans to measure*), they are under-weighted and often avoided (leading to accident fallacies and others).

I could wander from here into my theories of semantic memory, but I’ll try to stick with critical rationalism to finish my thought. When Popper started, what he sought was to step away from justification, the practice of trying to support currently held ideas. In the service of this, the claim was made that one can disprove an idea, but not prove one. Though I’ve come to reject this claim, I don’t think that critical rationalism is a bad approach.

Justificationalism comes out of a natural tendency to want to be right, and it appeals to this bias even when a more open mind might find more effective ideas. Critical rationalism avoids this by forcing each person to listen to other arguments in order to determine how they might fail.

Critical rationalism also avoids the trap of adding weight to a theory because of selective observation. For instance, if I have the theory that “proteins are a kind of enzyme”, I might seek to “confirm” it by looking for enzymatic proteins. This will bias my data set so that it appears that the idea is effective, when it actually isn’t. Critical rationalism will naturally disrupt this bias with a second bias of seeking data that doesn’t fit the theory. Because an idea that is predictive only, say, 80% of the time isn’t very useful, this bias is helpful in pushing us towards more consistently accurate ideas.

One might suggest that when people say “truth” they mean “predictive power”. If this is true, I can easily show where Popper’s ideology fails. No prediction will be correct 100% of the time; our sensors are fallible. An idea that fails shouldn’t be rejected as “falsified” if it’s still accurate almost all of the time. F=ma is still a really important piece of knowledge. In this way, induction works.

* – Probabilistic ideas work differently for different forms of memory. Associative memory, the kind of thought that we use when making split-second decisions, is very probabilistic. For example, it is easy to have a gut feeling that a deck of cards is about half black and half red. Semantic knowledge, the kind of idea that we use consciously, doesn’t work so well with such things. I cannot imagine a person being able to tell you what proportion of a pile of cards is clubs unless they do some mental math.

Photo credit: swissbones on Flickr