Cheaper Verification, and More Valuable Trust

Over the past weeks I wrote about risk between ice and gas and about why every project still needs buyers. Both pieces dealt with what we can measure. This one is about the part of every decision that we cannot measure, and about what we use to act anyway.

Starting and Shipping

For a lot of people two moments in the life of a project are especially hard. The first is to start. The second is to ship, which is different from building. Building is intimate: it is forgiving, and it follows your own rhythm and your own constraints. Shipping means facing the world at whatever stage the project has reached, which is guaranteed to be imperfect, and the world will tell you so eagerly. It is also necessary, because only the world can give you users, feedback, criticism and, in the case of an open-source project, contributions that you then can adopt and incorporate.

Both moments are a meeting between an inside and an outside. So is the moment when two people decide to work together. In each of them we can evaluate all the factors that are measurable, which is what I called risk: a model of reality, a forecast of its consequences, and a comparison with what happens. That is important and it is not sufficient, because there will always be things that we either cannot model or cannot measure. We still need to take action, and to start the momentum that building and shipping require.

Trust Is the Leap

The gap between measurable risk and the imponderable components of a decision is bridged by trust. Without it we would be alone, whether the organization we are building is made of two people or of thousands, and it would not be nearly as much fun.

Trust matters most in the very first moment, before there is any track record to rely on. I am one of those people who deliberately place more trust in the other person than would be necessary, or even reasonable, at the start. I do it because the first act of trust shapes everything that follows, and generous trust at the beginning smooths and accelerates the work together. Then the cycle runs: we open up, verify the consequences, close the cycle, and open it again. Each completed cycle confirms the trust or revokes it. This plays out daily at every scale, between two founders, inside large corporations, and among nations as they compete and cooperate.

Trust can also be misplaced, in two different ways. Interests may diverge, and one side may try to capture all of the value, if possible without the others noticing. Or there may be no malice at all, only mistakes or incompetence, and the other party simply fails to deliver what was agreed. Everyone has experienced both.

What AI Changes

The way I see it, AI makes verification cheap. It radically lowers the cost of measuring what happened after the fact, and therefore of deciding whether a given act of trust was well placed. What can be checked increasingly no longer needs to be trusted. What remains, and what becomes scarcer and more valuable than before, is everything we cannot check: the intent of the other party and their future behavior. Trust is needed for fewer things, and each of those things matters more.

The same AI tools, misused, counterfeit the signals we rely on to decide whom to trust, from a falsified document in a transaction all the way to fake identities, fake values and fake intent. I receive a message like this at least a couple of times a month: *Hi Dad, this is my new number, I lost my phone, I urgently need a thousand euros, please send it.* An unprepared person may panic enough to fall for it. A better prepared one takes a moment, asks how likely it is, and verifies, either with something the scammer cannot know or, additionally, through a second channel.

The additional channel is precisely what failed at much larger scale this year. Fideuram, the private banking arm of Intesa Sanpaolo, lost €95 million after its chairman received a WhatsApp message that appeared to come from the group’s CEO, followed by a phone call in which an AI-cloned voice of a senior lawyer confirmed the instruction. The transfers went mainly to accounts in China and Hong Kong. About €53 million has been recovered, and some €36 million is still missing. The confirming channel had been counterfeited along with the first one. As trust becomes more valuable, mistakes in our most valuable trust relationships become more costly, and we will have to use AI itself to protect them.

When Trust Tools Lower Trust

Since August, Article 50 of the EU AI Act has required providers to mark AI-generated content, and on 5 October OpenAI announced that it will add invisible watermarks to ChatGPT’s text in the European Union, joining Google and Anthropic, which already mark theirs. A lot of people are upset about it, and I understand why. They feel tracked and labeled, and they consider their content theirs regardless of the tools used to produce it.

The lawmakers meant to prevent AI fakes. What they are doing instead is polluting the information environment with a label that cannot be universal. I do everything with AI, so for me the label is both well deserved and a surprise to nobody who knows me. Many others will be taken by surprise, at a time when one polarized group sees AI as positive and a large group sees it as negative. The solutions designed to prevent AI from lowering trust are, paradoxically, lowering it, and we will need something that achieves their goal better.

The pressure will only grow. Video models can already generate a conversational human character that many people cannot tell apart from a person over a short call, and the call will soon be an hour long. Either we accept that many of the people we meet on video will be AIs, or we find ways to recognize who is human, and then each of us will decide whether to trust AIs less than humans, as much, or, for some people, more.

Contracts, Platforms and Agents

Someone in our conversation who used to administer contracts described the hardest part of that job: getting colleagues to trust him, while making sure they did not hand their agency over to him along with the paperwork. They would say that he had read the contract and the AI had read it too, so they must be fine. His answer was that they still had to read their own deal.

I agree that the decision must stay with the human. I would add that contracts are almost universally an abuse of the trust needed to enter them. I do not know a single person who reads an insurance policy, a bank account agreement or a mortgage from beginning to end, in the small print, and I have never met a bank teller who said: take three hours, sir, I will wait. They wave you through and tell you not to worry. In this case AI rebalances the relationship. However many mistakes it makes analyzing a mortgage agreement, those mistakes are irrelevant next to the complete ignorance of the clauses that signing blindly implies. Give every agreement to your AI and ask whether anything in it is strange, whether anything should worry you, and what you should push back on. Often there will be something.

Which AI you ask matters too. Models are trained and instructed differently, and the more your tools and agents know about you, the better they serve you. That is a reason to export what you have given them, periodically and at the finest granularity available, both to check that it is consistent and to be able to switch to another tool without starting from scratch. I am fairly obsessive about exporting my data from every platform, because I do not think the platforms deserve our trust. They provide value, and they care more about themselves than about any individual user.

I would like to know what the platforms think of us. I suspect they are somewhat terrified of humans. We want to use them unfettered, for any purpose or idea we may have, and at the same time we want them to stop criminal behavior, which is the opposite demand. The relationship was complex when platforms were passive. Now that every platform is becoming agentic, it will be harder to decode on both sides: for us, to know what kind of relationship we are in, and for them, to understand human behavior even more deeply.

Trusting the Factory

The same question returns, sharper, when agents do the work instead of reading it. A team I work with has been testing a software factory: agents that take a task, write the code, open a pull request, test it and merge it. With strict rules, the agent declined every single pull request, the correct ones together with the broken ones, and kept asking for a human to review. With looser rules it accepted almost all of them. Neither setting is trust. Asking a human to read the code so that the agent can feel confident enough to merge it would bring back exactly the bottleneck the factory exists to remove. The better answer is another layer of checking: tests verify the code, and a verifier checks whether the decision to accept or reject was right, sending the work back when it was not.

What we decide to keep for ourselves follows the line between what can be checked and what cannot. A bug means that a user cannot do something they expected to be able to do, so the expectation already exists, the fix can be verified against it, and the factory can handle it alone. A feature request asks for something nobody expected, and accepting it changes the direction of the product. Someone who loves a cluttered interface will keep asking to bring every option back to the home page, and the people responsible for the design must be free to say no. Intent stays with humans.

What happens when the factory makes a mistake big enough that we lose trust in it, and have to question everything it ever did? Trust was needed in the first place precisely because there were things we had not foreseen and could not model. When a systematic mistake appears, we analyze it, prevent it from recurring and improve our model of the risk, knowing that we have not eliminated the need for trust, only pushed it further out. It is the same thing two people do when they stop arguing over every clause, sign the agreement and start working together, so that they can find out whether their trust was well placed. I see no version of the future in which we decide that the past ten years of increasingly trusting AI were a mistake and build the world without it.

The most advanced builders are moving along the same path, cautiously. In August, Anthropic reported that Claude leads 26% of its own AI research and development work, up from under 1% in February, while none of that work runs fully autonomously. Humans there still decide what is worth doing. Each completed cycle widens what agents are trusted to carry out, and the last thing they will be trusted with is choosing the work itself.

The Need for Trust Is Here to Stay

Attackers understand the trust cycle as well as anyone. A Microsoft Research paper argued that the famous scam emails are crude on purpose: the bad English filters out everyone who would see through the scam later, so that only the most promising targets reply. Sophisticated operations, including those that place North Korean workers in Western jobs under false identities, are patient. They complete the first cycle of trust, accept more responsibility as the organization grants it, and escalate only then.

None of this makes the leap optional. It is mathematically guaranteed that our models will always leave out things we cannot measure, and if we want to act and create, we will have to take a leap of faith. I was struck recently by a remark about democratic elections: the losing side hands power over to the people who were its enemies until the day before, trusting that it will not be killed, that its rights and property will be upheld, and that in a few years it will be able to run again and win. Judged by older and less enlightened power relationships it looks almost absurd. It is a trust cycle that a whole society agrees to run, over and over.

So we have to be creative and inventive about trust, and use it daily in what creates society today, so that it remains part of the society of tomorrow. A community is one place to practice it. Our group at Ikigai Collective runs that cycle every Thursday, with people opening up, placing trust in one another, and coming back the following week to see whether it held.