Courtney Abbott's Key Ideas from Superintelligence
by Nick Bostrom

British computer scientist Alan Turing believed that scientists one day would be able to design a program that would teach itself. This led to his idea of a “child machine,” a “seed AI” that would create new versions of itself. Scientists speculate that such a process of “recursive self-improvement” could one day lead to an “intelligence explosion” resulting in “superintelligence,” a radically different kind of intelligence.

AI may grow as an intelligent software based on the architecture of a particular human brain. This differs from AI and speculatively involves taking a dead person’s brain and creating a digital reproduction of that person’s “original intellect, with memory and personality intact.” This “barefaced plagiarism” is called “uploading,” or “whole brain emulation” (WBE). It would take years, perhaps until the middle of this century, to develop the necessary enabling technologies.

AI could also grow by the path of genetic engineering, which could lead to a population of genetically enhanced individuals who together make up a “collective superintelligence.” This scenario might include cyborgs and systems like an “intelligent web.”

As science’s exploration of superintelligence progresses, more questions arise. For instance, how might different efforts to create superintelligence compete with each other? An ancillary to that question is what the consequences might be if a front-running technology gained a “decisive strategic advantage.” An artificial intelligence project might achieve such an advantage by reinforcing its dominance in a way human organizations cannot.

Bear in mind that the character of AI and, by extension, of superintelligence, is not expressly human. Fantasies about humanized AI are misleading. Though perhaps counterintuitive, the thesis of orthogonality holds that levels of intelligence do not correlate with final goals. More intelligence doesn’t necessarily imply more shared or common goals among different AIs.

In contemplating the specter of superintelligence gone berserk, some suggest reducing AI size, scope and capability. For example, scientists might base an early-term AI on nothing more than a question-and-answer system, an “oracle.” Programmers would have to take great care to clarify their original intentions so the AI functioned as desired.

Relentless doubt about the efficacy of AI always comes back to its motivation and how to refine it. The question remains: how can science create an AI with “final goals” which align with the values that foster peaceful coexistence between man and machine.

Scientists might approach this process of “value loading” in several ways. One would be simply to write values into an AI’s program. That evokes the riddle of how to translate the incomprehensible, sometimes ambiguous, nature of truths and principles into code.

Within the panorama of possibilities for superintelligence, the most practical – and, perhaps, also the safest – the alternative may be whole brain emulation, not AI, although some scientists dispute that notion.

Whichever path humanity chooses, a true intelligence explosion is probably decades away. Nevertheless, be aware that people react to AI like children who discover unexploded ordnance. In this instance, even experts don’t know what they’re looking at.

The oncoming and unstoppable intelligence explosion carries profound “existential risk.” As technology slowly evolves to make this explosion a reality, humans must carefully consider all the philosophical and moral ramifications now – prior to the appearance of superintelligent AI. Once AI appears, it may be too late for contemplation.