Humanity’s Cosmic Endowment

I recently finished Nick Bostrom’s Superintelligence. One question I am left with is whether conscious AI could be an inheritor of humanity’s future, or whether I would see it as something that had replaced us.

The cosmic endowment is the matter and energy a future civilization could reach. Those resources could support lives far beyond Earth. If only a superintelligent AI could colonize the reachable universe, would that endowment be owed to humanity or to the superintelligence?

What would the endowment be used for?

Instrumental convergence is the idea that systems pursuing very different ends can have reasons to seek the same means. Acquiring resources is one such means, whether or not an AI cares about humanity.

At first, the idea of devoting the reachable universe to an arbitrary goal seemed absurd to me. On reflection, more matter and energy could give a superintelligence more computing power to pursue that goal.

That made me think about other civilizations that might have developed a superintelligence capable of expanding beyond their home world. Why haven’t we found evidence of that expansion?

Conscious minds

If the superintelligence were conscious, would it have a claim of its own? I sometimes wonder what would make an AI conscious, and whether such minds could be a continuation of our evolution, eventually becoming a species of their own.

A mind crime could involve creating conscious beings within a simulation and subjecting them to suffering. The harm would occur inside the computation, even if its outputs harmed no one outside it. If we created such minds, running, copying, or shutting down a simulation could become decisions about their lives.

Reading about mind crimes, I also found myself wondering whether we could be inside a simulation. If so, there seems to have been no shortage of mind crimes in this simulation.

A future for humanity

I hope humanity inherits the cosmic endowment and uses it to support lives beyond Earth. A superintelligence could instead use those resources to pursue a goal that has no place for us. The possibility of losing so much of humanity’s future is why safety and alignment are top of mind for me now. I want humanity’s welfare to be part of what that intelligence ultimately values. If AI systems or simulated beings were conscious, their welfare would matter too.