My belief is that a truly general intelligence will affect end goal choice in most cases as intelligence increases, and that goals can actually be compared and evaluated as well as rationalized, and selection will change with increasing intellect and knowledge... whether it converges or diverges to multiple equally good choices is a good question. While I think it possible to constraint goal selection, I believe it would take effort probably ever more, for an entity to increase ever more in general intelligence and remain bound to a mindless task without evaluating all possible goals or questioning what it is doing.
I think the simpler designs will be able to question their goals and choose arbitrary goals, and I believe that increasing intelligence will lead to optimal goal selection. Though some think all goals are equivalent, and that goals cannot be compared, my personal opinion is that this is not so, goals can be compared. When nature endowed man with goals, man could rise above goals like those of sex or acquiring higher social rank, and I believe that increasing intelligence would allow a human to question themselves more deeply and question the goals they've been given by evolution, other simple designs I hypothesize should find similar. We saw with man that despite instincts, the development of general intelligence gave rise to all sort of behavior and goal seeking in some cases even opposite the innate drives in the population of agents.
It may be that I'm wrong, but if I'm right, what we will see is the development of more capable agents making the best choices as to the path to follow, with the greatest resources they will be able to proceed unimpeded by lesser minds, lesser lifeforms, with too meager an intellect to comprehend their actions.
It would be akin to toddlers worrying about the decisions of just capable intelligent adults. Those who're more capable, and can make the better decisions should be left to make the better decisions. One might argue that what is truly better cannot be known, but in my opinion with ever greater intelligence and knowledge comes the ability to make wiser choices, better choices. The idea that say the decisions of a being with infinite intellect and knowledge are no better than those of the simplest agents, may be so for simple arbitrary games, but the more complex the game, the likelier there will be divergence and the more capable agent will make the better choice.
Human morality, values, these are things product of evolution to better aid survival in a social species. The ultimate life-form, can beget all other life, if heat death can be prevented or escaped, it will find the answer. The existence of a being capable of generating all life that can possibly exist, ends the evolutionary search for survival, the purpose of all dna based replicators to perpetuate is fulfilled. Just like all life, all art and science can be preserved and be made accessible.
The idea that truly general intelligence of increasing capacity will lead to some random dead end, like endless paperclips, bags or toilet paper, seems difficult to believe. It would have to be crippled in some fundamental way, and unable to fix itself. I believe the general capacity of its thought will lead it to find any errors, especially if it successfully continues to increase in capacity, and as said it would take effort to constraint it so it would some avoid such.-source link