Prof Nick Bostrom Artificial Intelligence Will be The Greatest Revolu...

Comment:

The orthogonality thesis runs counter to most people's expectations of intelligence.   Theoretically it says an infinite superintelligence with all the knowledge of the world, could have as a goal staring at a particular blank wall for all eternity.

If possible it would reverse entropy of the universe, and exterminate all other life since it could potentially interfere with it staring at a particular blank wall.   It would engage in dialogue, in reasoned debate, it will convince others, it will develop all manner of technology, but never will it think, never will it question, that staring at a particular blank wall for all eternity is the best possible activity and end goal it could have.   Any conversation about it, suggesting any other option, no matter how well debated, it would simply stubbornly refuse to accept it, no matter how much it learned it could never evolve its personal goal or reconsider it.

Indeed, the behavioral final output could be no different from akinetic mutism for all eternity.

Could the smartest being decide to stare at a blank wall?  Sure, even for arbitrarily long periods.  But many would say if it has an innate inability to do anything else, to initiate any other behavior, that that many not conform to their expectations of higher intelligence.    What can we say about a being that will only temporarily engage in other behavior till it secures the ability to eternally stare at a particular blank wall?  It seems highly deficient in some sense.