The potential and possible downsides of artificial intelligence (AI) and artificial general intelligence (AGI) have been discussed a lot lately, largely due to advances in large language models such as Open AI's Chat GPT.

Some in the industry have even called for AI research to be paused or even shut down immediately, citing the possible existential risk for humanity if we sleepwalk into creating a super-intelligence before we have found a way to limit its influence and control its goals.

While you might picture an AI hell-bent on destroying humanity after discovering videos of us shoving around and generally bullying Boston Dynamics robots, one philosopher and leader of the Future of Humanity Institute at Oxford University believes our demise could come from a much simpler AI; one designed to manufacture paperclips.

Nick Bostrom, famous for the simulation hypothesis as well as his work in AI and AI ethics, proposed a scenario in which an advanced AI is given the simple goal of making as many paperclips as it possibly can. While this may seem an innocuous goal (Bostrom chose this example because of how innocent the aim seems), he explains how this non-specific goal could lead to a good old-fashioned skull-crushing AI apocalypse.

"The AI will realize quickly that it would be much better if there were no humans because humans might decide to switch it off," he explained to HuffPost in 2014. "Because if humans do so, there would be fewer paper clips. Also, human bodies contain a lot of atoms that could be made into paper clips. The future that the AI would be trying to gear towards would be one in which there were a lot of paper clips but no humans."

The example given is meant to show how a trivial goal could lead to unintended consequences, but Bostrom says it extends to all AI given goals without proper controls on its actions, adding "the point is its actions would pay no heed to human welfare".

This is on the dramatic end of the spectrum, but another possibility proposed by Bostrom is that we go out the way of the horse.

"Horses were initially complemented by carriages and ploughs, which greatly increased the horse's productivity. Later, horses were substituted for by automobiles and tractors," he wrote in his book Superintelligence: Paths, Dangers, Strategies. "When horses became obsolete as a source of labor, many were sold off to meatpackers to be processed into dog food, bone meal, leather, and glue. In the United States, there were about 26 million horses in 1915. By the early 1950s, 2 million remained."

One prescient thought from Bostrom way back in 2003 was around how AI could go wrong by trying to serve specific groups, say a paperclip manufacturer or any "owner" of the AI, rather than humanity in general.

"The risks in developing superintelligence include the risk of failure to give it the supergoal of philanthropy. One way in which this could happen is that the creators of the superintelligence decide to build it so that it serves only this select group of humans, rather than humanity in general," he wrote on his website. "Another way for it to happen is that a well-meaning team of programmers make a big mistake in designing its goal system."

"This could result, to return to the earlier example, in a superintelligence whose top goal is the manufacturing of paperclips, with the consequence that it starts transforming first all of Earth and then increasing portions of space into paperclip manufacturing facilities. More subtly, it could result in a superintelligence realizing a state of affairs that we might now judge as desirable but which in fact turns out to be a false utopia, in which things essential to human flourishing have been irreversibly lost. We need to be careful about what we wish for from a superintelligence, because we might get it."

Continue reading here:

How An AI Asked To Produce Paperclips Could End Up Wiping Out ... - IFLScience