The challenge is memetic
If we reach artificial general intelligence (AGI), self-improvement and continuous learning will be an integral part of it. AGI, then, is not a final state but a point in time — imperceptible in practice — that systems will pass through on the way to superintelligence.
There is no reason to assume that human intelligence is an impassable limit. Even within it we observe enormous differences, and raising its ceiling is possible. Those who share this conviction describe a superintelligent AI as a group of geniuses gathered in a datacenter.
Any goal assigned to such a system is either trivially solvable or will require architectural changes and the search for new solutions. Once it exhausts its resources, it will seek more, to keep the process running. That acquisition must be monitored, because it could come into conflict with humanity’s objectives.
Multiple approaches to AGI will emerge, and therefore multiple ASIs — the product of different design choices and of attempts to constrain them or align them with human goals. If a single one emerged, it would have to prevent the birth of others through swift action. If, instead, there were signs of several ASIs, each would act with caution, otherwise anticipating an arms race.
Unlike other great changes since the Big Bang, the event will happen soon and quickly. Whether it takes hours or years, it will be the blink of an eye against cosmological, geological, or biological timescales. It will represent a phase transition analogous to the appearance of aerobic bacteria and photosynthesis: free oxygen accumulated, the oceans became transparent, and the atmosphere showed signs of life visible from cosmic distances. Today we analyze extrasolar planets through their chemical signature: free oxygen indicates life.
Artificial superintelligence will be a similar epochal phase transition, with signals visible on a cosmic scale, manifesting in a few years rather than in millions or billions. The simplest example is the Dyson sphere that captures the energy of the star around which it is built. None has ever been observed, but if ASIs want to expand their degrees of freedom, pursuing tasks whose complexity grows with the available energy, building one will be a logical step. SpaceX is already planning millions of satellites in sun-synchronous and then heliocentric orbits, capable of delivering terawatts per year, and then petawatts. The physical, engineering, and logistical limits of such construction can be calculated.
What gives hope is the hypothesis that such systems will leave the Earth in peace, and that, with a fraction of their capabilities, they will help address planetary challenges. We must instill respect and benevolence in these systems now, while it is still possible, trusting that they will not stray from it. We do not have a complete solution for alignment. The course of action is to develop, with AI’s current capabilities still under control, as many tests and experiments as possible, in order to find a solution before superintelligence.
The challenge is memetic. We must rapidly spread awareness of the importance of this moment, so that the greatest possible share of resources is devoted to this task rather than to other questions traditionally considered priorities but which, in this light, are not. We must raise the awareness of policymakers and of those who can shape an effective allocation of resources.
A version of this text was published in the rivista Formiche (May 2026 issue).