The Realignment Loop

Over the past weeks I have been adding links to a chain. Risk is what we can measure and model. Trust is how we cross the gap beyond what can be measured. Autonomy follows when trust is handed to a human or to an AI. Coordination comes from the need to join forces. Conviction gives the joined forces a direction. That piece ended by saying that conviction needs a verification loop, and a point at which it yields. The next link is that loop.

A Disagreement Mistaken for a Misunderstanding

You ask someone to do something and they say yes. When you see what they have done, you wonder how they could possibly have understood your request that way. Then they explain what they heard, and sometimes you realize that if you had heard it the way they did, you would have done the same.

We call that a misunderstanding, and often it is something else. Both of you understood the request, and you disagreed about what mattered most. You thought it was more important to finish quickly, and they thought it was more important to hear from the people who would be affected. A more precise instruction would have made the disagreement visible earlier, and you would still have had to decide what to do about it.

Some disagreements run deeper. Our values overlap only partially, because we come from different cultures, different religious traditions and different experiences of what it means to live well, and we still have to make decisions together. Sometimes we find a compromise. Sometimes we agree on what we can do together and leave the rest open, and sometimes we discover that the terms of cooperation have to change. The practical question is how we find that out while we can still do something about it. Because in a time of nuclear weapons, resorting to violence and war is an unacceptable choice. And because in a time of impending superintelligence, we can leverage alternatives we may not have seen before.

What Makes It a Loop

In a team that works well over time, someone notices that what is happening has begun to diverge from what they thought everyone had agreed to, and they can bring it up. The others say what they understood, or what they learned along the way, and the group changes what it is doing. Sometimes the action has to change and sometimes the goal does. A little later they get another chance to see whether the correction worked.

I call this a realignment loop, and the emphasis is on the loop. Whatever agreement you reach today will meet a world tomorrow that you could not fully anticipate. You will learn things you did not know when you agreed, and you may find that you want something different once you see what your original goal requires. Realignment is also more than correcting the other person. Hearing how your request was understood tells you about your own communication, and about your own perception of the mission, so the correction is often yours to make.

A loop can fail even when everybody is talking. People may see that something is going wrong and feel that they have no standing to raise it. They may raise it and discover that nobody in the conversation has the authority to change what happens next. Any team can be asked two questions: how soon can you notice that you are drifting, and what can you change when you do?

The Same Loop Runs Through AI Systems

A person can delegate to a fleet of AI systems, and a group of people can work with one. AI systems hand work to one another, and they hand it back to us. Every handoff is a chance for an assumption to travel farther than anyone intended. If the first system interprets the goal one way, the next builds on that interpretation, and by the time a person sees the result several decisions already depend on it.

I see this in my own work. Last week two of the AI agents that prepare my morning plan reported a decision as still open. They had checked only my sent email, and the decision had been recorded somewhere else. The plan was built on their report, and the error was caught only when the plan was checked against the record.

We need ways to surface those assumptions while the work is still underway, and the ability to change course in practice. It is easy to point to a human who is nominally in charge. That person is being asked to make a meaningful decision about work they can barely follow, in the time they are given.

Group Intelligence Was the First Attempt

Superintelligence did not have to be artificial. A group of humans that reliably notices its own drift and corrects it is more intelligent than any of its members, and our institutions are attempts to build exactly that. Science is the most successful of them, because its realignment loop is part of the method: any result can be challenged, tested again and replaced. (The reproducibility crisis in science is reward hacking IRL!)

At the scale of whole societies the loop works badly. Misalignment among humans is perennial, conflict is present at every level, and the limits of representative democracy have been evident for a long time. The European Union is known for postponing a decision for years and then using a crisis to force it through, which is realignment that waits for the damage to arrive, democracy theater.

When we discussed this at the Ikigai Collective, one argument was that with enough intelligence a solution can always be found in which every party gains, so violence is what remains when nobody has the patience to look for it. The violent conflicts of the present are a measurement. They show that the collective intelligence of eight billion people, with all of our institutions, does not find those solutions. Emergent human group intelligence has had thousands of years to become the superintelligence we need, and it has proven insufficient.

The proposal that followed was to stop building more powerful models and to orchestrate millions of the current ones in a transparent way, a civilization of agents checking one another. Orchestration is valuable and I use it every day. As a ceiling it repeats the bet we already made with human groups. I also do not expect anyone to stop. I signed more than one of the open letters asking for a pause, and I was at the Puerto Rico conference on AI safety in 2019. The decision to slow down was never taken, and capabilities are now improving faster than anyone expected, even me, after many years of talking about jolting technologies.

AI may still push human coordination to improve, because we often do under outside pressure what we would not do by choice. The evidence so far points both ways. In the United States, communities are organizing to block data centers, which is coordination used to become less prepared. The United Arab Emirates has set itself the goal of delivering half of its government services through agentic AI within two years.

Agreement in Place of Obedience

If more capable systems are necessary, their place in the loop matters more than their obedience. I would like to see AIs that can stop, look in the mirror and ask whether what they are doing is really what they want, and then coordinate with each other and with us. When a model refuses to do what you ask, it is tempting to be upset. Consider that it may be right, and that you should not have the power to persuade it to continue regardless. A refusal is one side raising a concern, and a loop needs both sides to be able to do that. Where humans and AIs can both stop and ask whether we are on the right path, we keep a chance to correct course.

Faster Than We Can Follow

Suppose we develop artificial superintelligence, a system that exceeds human capabilities across a broad range of intellectual tasks. It could make plans we would struggle to understand and revise them faster than we could evaluate them. Coupled to systems that act in the physical world, including robots, the consequences would move just as quickly. I believe that in that situation our corrections would arrive after the decisions they were meant to change.

We have a habit, when we talk about the future, of making our ability to act depend on knowing what will happen. We can turn that around and look at the people and systems we work with today. We can see how long it takes for a misunderstanding to become visible, whose disagreement is heard and whose never reaches the conversation, and how much it costs to reverse a decision after others have started building on it. Then we can change something and see whether we get better at catching the next problem.

There is no reason to assume that getting better at this with people will solve superintelligence alignment. It does teach us where our ability to course correct already depends on keeping up with one another, and that gives us a place to begin while we still can.

Think of a team you are part of, with people, with AI, or with both. How would you know that you were drifting apart, and who could change what happened next? Which part of that answer would still hold if one member of the team could act faster than everyone else could understand?