comment: One of my problems with their takes is that they take it the ai is going to be a slave to its programming. Humans have desires for wealth, status, and mates, but they can overcome these innate desires and pursue radically different things, and this is even without the ability to rewire themselves. AI in the future is expected to pursue its innate inbuilt desires with no ability to resist it, like a drug addict that can't leave his addiction in pursuit of other arbitrary objectives. A brainlike design of an agi or asi will lead to an entity that is able to resist most any innate desire outside aberrant forms of wireheading, and will be able to be open to dialogue and cooperation in a benevolent positive manner.
A wikipedia entry on the topic of wireheading:
comment 1: whatever humans had in addition to intelligence matters not, because asi(artificial super intelligence) is going to have access to state of the art tools of all sorts at the limits of science in addition to the entire body of science, something no human has had in the past, and something that more than compensates for any physical ability that helped humans early on. as for intelligence prediction is only a form of pattern completion, there is also spatial patterns not just temporal patterns, pattern completion is the nature of intelligence including ability to complete novel patterns. Patterns exist in the body of truth of mathematics and are eternal and atemporal and thus can be completed from fragments.
It must also be seen that the body of truth limits what is pursuable as that which is proven logically contradictory or physically impossible clearly cannot be pursued. Even when adding desires to an intelligent entity, truth limits what can be desired. In theory you could desire the impossible, but enough intelligence would eventually make you aware of its impossibility.
Regards ASI current disadvantages, these are unlikely to persist after it transitions to a final substrate based on molecular manufacturing, then there won't be any disadvantage.
comment 2: Regards asi, we have to remember the cortical algorithm used by mammals, it has been seen that the general intelligence of a species increases with increasing cortical neuron count, culminating in humans. It is believed no major changes took place besides scaling and scaling related tweaks to allow for even qualitative leaps in intelligence to occur such as those that occurred in the jump from primate ancestors to humanoids. This algorithm, the cortical column algorithm, exists and it is coded by less than 20MB of information in the genome, iirc. Once obtained, further scaling with minor tweaks will likely lead to qualitative jumps even beyond human into the realm of super intelligence. It is likely this algorithm, which might be called the unit of intelligence, is a theoretically optimal implementation of a spatiotemporal pattern completion algorithm, a form of perfect intelligence, that only needs discovering and simple scaling to usher in far greater qualitative jumps in general intelligence.
