
I recently finished Nick Bostrom’s Superintelligence. I wonder whether conscious AI could inherit humanity’s future, or whether I would see it as having replaced us.
“Cosmic endowment” is a way of thinking about the matter and energy a future civilization could reach. Those resources could support lives far beyond Earth. If only a superintelligent AI could colonize the reachable universe, would that endowment be owed to humanity or to the superintelligence?
What would the endowment be used for?
At first, the idea of devoting the reachable universe to an arbitrary goal seemed absurd to me. But more matter and energy could give a superintelligence more computing power to pursue that goal.
This is one example of instrumental convergence: systems pursuing very different ends can have reasons to seek the same means. An AI could have a reason to acquire resources throughout space without caring about the lives those resources might otherwise support. Its expansion would not, by itself, secure a future for humanity.
Conscious minds
If such an AI were conscious, though, would it have a claim of its own? I sometimes wonder what would make an AI conscious, and whether such minds could be a continuation of our evolution, eventually becoming a species of their own.
That question extends to conscious minds we might create inside simulations. If their lives mattered, we would have responsibilities toward them. Subjecting them to suffering would be an example of a mind crime: harm within a computation, even if its outputs harmed no one outside it. Running, copying, or shutting down such a simulation could become decisions about those lives.
A future for humanity
I hope humanity inherits the cosmic endowment and uses it to support lives beyond Earth. A superintelligence could instead use those resources to pursue a goal that has no place for us. The possibility of losing so much of humanity’s future is why safety and alignment are top of mind for me now. I want humanity’s welfare to be part of what that intelligence ultimately values. If AI systems or simulated beings were conscious, their welfare would matter too.
Get new posts by email
Follow.it sends the emails and includes an unsubscribe link.
Related posts
Farming Expensive Agent Sessions

I’m using expensive agent sessions to refine the instructions that tell Codex how I want the work done.
Load Flow and Mandelbrot in the Browser

A browser-based load-flow solver and Mandelbrot explorer let me revisit power-system analysis, numerical precision, and the limits of small client-side tools.
Make Internal Tools Cheap to Try

A rough prototype of an internal tool lets people test their understanding of a workflow before anyone commits engineering time to it.