However, there is a class of problems that cannot actually be “solved”, such as “build widgets”, “make our company more efficient”, “anticipate nuclear attacks”, “make people happy”, etc. Such indefinite problems mirror the top level goals in living creatures (namely “replicate & survive”). To take the case of making widgets, I have no doubt that a majority of the AGI’s efforts would go to widget construction, but as the AGI grows, the amount of effort (even if not the percentage) put towards self-preservation and improvement would also go up.

For your second major point, I think it’s relevant to look at the entire concept of motivation and control. I think it’s highly likely that an AGI would understand that widgets aren’t the primary goal of humans. It would understand that we built it to solve a subgoal of ours, much like we understand that our genomes made sex pleasurable not for the sake of sex, but for the sake of reproduction. The problem is that sex is a top-level goal in people, and widgets would be a top level goal in this hypothetical AGI. It cares about what we want just as much as we care what our genomes “want”.

(The example about sex isn’t a perfect one, because we also have a top-level goal of reproducing, so we don’t typically follow sex to the detriment of reproduction. I think it works as an analogy, though.)

Lastly, you know that I’m no Luddite. I’ve written previously about AGI being the most important thing that the human species work on, and I stand by that. There’s huge potential for AGI to lift us out of suffering.

Ultimately, though, I don’t care if North Korea beats us to it. I care if what’s built helps humanity thrive. Optimally, I’d like it to help the entire world thrive, but I fear that even an American AGI won’t have those values.

Also, it seems unlikely that the whole time it was building advanced robotics, server farms, solar panels and power plants it never gave any consideration to what happens after it solves the problem. “There, done! … … Now what?” It would have to 1) identify that it will need more processing capability, 2) conceive additional processing solutions, 3) plan the implementation of such solutions, 4) identify potential risks that may thwart its efforts (without recognizing that at least one of the “risks” (i.e. us) is the point of solving the problem in the first place), 5) procure the necessary materials, 6) … and would have to do this for countless projects and sub projects at potentially hundreds of levels. Yet it never realizes that delivering the solution to those that requested it is the whole point? I know… if the explicit goal is to reach the solution, not deliver it, then it is possible. Point taken: be careful what you ask for.

Still, i assume that this is not an entreaty to prohibit the development of AGI. I realize my own blog post is held together by a lot of supposition as well, but i still don’t see any way around my conclusion. Either in secret or out in the open, legally or not, people are going to work on this stuff. Who would you prefer get it working first, North America or North Korea?