On 19 October 1987 the Dow Jones fell 22.6 per cent in a single session, the worst one-day loss in its history. No war had started and no bank had failed. The trigger was a set of portfolio-insurance programmes - automated selling rules - that were near-identical across the market. When prices dipped, every program issued the same instruction at the same moment, which moved prices, which triggered the next round of identical instructions. Twenty-three years later the Flash Crash of 2010 briefly erased around a trillion dollars of market value by much the same mechanism: automated strategies, each behaving correctly, all reacting in unison.
The machines did not cheat, defect or malfunction. They agreed. The damage came from too much alignment, not too little. What fixed it was circuit breakers - crude rules that halt trading when the market starts feeding on itself. Nobody improved the algorithms’ values. The repair went into the structure of the game, not the character of the players. Figure 1 shows the whole episode in one diagram: three arrows and a red bar, which between them contain the entire regulatory philosophy of 1987.
We are now giving AI agents rules in different financial arenas - agent payment protocols, agent-to-agent standards, fleets of software negotiating with each other and with us. The question of how they will behave is usually framed as a question about the agents: are they aligned, what do they want, can we install the right values? Game theory suggests this is the wrong end of the telescope.
The theorist who pushed the point furthest is Ken Binmore, in Game Theory and the Social Contract and Natural Justice. His claim: cooperation is a property of the game, not the player. In a one-shot transaction, betrayal is the rational move and no amount of intelligence changes that. But if the transaction is repeated indefinitely between players who remember, things change: cheat me today and I can withhold cooperation tomorrow, and if we both care enough about the future, cheating stops paying. This is the intuition behind what game theorists call the folk theorem - so named because everyone in the field knew it long before anyone published it. It says that with indefinitely repeated transactions among sufficiently patient players, a vast range of outcomes - from steady cooperation to organised extortion - can each be sustained as an equilibrium, meaning a pattern of play that no player can profit by abandoning on their own.
The theorem’s real message is an embarrassment of riches. Repetition does not predict cooperation; it merely permits it, along with much darker arrangements. Something has to select which equilibrium a population actually lands on. Binmore’s answer is fairness. Our norms of fair division are evolved focal points - conventions so salient that everyone expects everyone else to follow them - which pick one equilibrium out of the folk theorem’s many. On this view the human moral sense is a piece of coordination machinery, tuned by natural selection and grounded in empathy. Evolution did not make us nice. It put us in a repeated game and handed down a convention for playing it.
The best evidence that the mechanism needs no values at all comes, fittingly, from machines. Calvano and colleagues, in the American Economic Review in 2020, let self-interested pricing algorithms loose on a simulated market. There was no communication, or cooperative objective, and no values installed. The algorithms learnt to hold prices above the competitive level, and sustained the arrangement with punishment-and-return strategies: undercut me and I will make the next rounds expensive for you. From the algorithms’ side this is a triumph of emergent cooperation. From the consumer’s side it is collusion. This tells you that “cooperation in the game” is value-neutral. Whether it delights or alarms you depends entirely on whether you are inside the coalition or outside it.
What selected the collusive equilibrium over the competitive one was the reward signal - profit was what the algorithms were built to maximise. For machines the selector is the objective we hand them and the environment that rewards it. Both are design choices.
An objection arrives on cue: today’s LLM agents are nothing like the rational actors of economic theory. Currently their preferences are incoherent and shift with the framing of the prompt; the sharper results here are preprints, but the thrust of these papers is clear enough. What the theory demands is an identity that lasts, and a memory of who did what to whom.
Which is exactly what production AI agents lack. They are spawned for a task, forked into copies, reset between sessions. Reciprocity needs someone to reward or punish; a tit-for-tat strategy needs a stable someone to be for tat. You cannot hold a grudge against a process that was deleted at the end of the conversation and reborn elsewhere as forty indistinguishable copies. Statelessness looks like a safety property. In repeated-game terms it removes the only mechanism by which self-interested players have ever learnt to cooperate. Redwood Research made a version of this point in 2024: a model that cannot carry a plan between sessions cannot scheme across them. But their mechanism is one agent failing to coordinate with copies of itself. The folk-theorem version is broader and worse - statelessness removes the repeated game in which any reciprocity equilibrium could form, including collusion between distinct agents. Including human agents.
Add memory and the machinery switches on. A 2026 study of LLM agent populations found that cooperation, once collapsed, rebuilt itself when agents were given persistent reputations - a record of conduct that follows the agent around. No values were added, only memory. And Ashery and colleagues showed in Science Advances that populations of LLM agents converge unbidden on shared conventions, complete with collective biases that no individual agent holds. Give agents identity, memory and a future, and folk theorem outcomes can kick in.
Memory, though, cuts both ways. It is also the mechanism of lock-in: equilibria that harden, grudges that compound, conventions that no one chose and no one can audit. Calvano’s algorithms used their memory to collude. Agents cooperating beautifully among themselves may be cooperating against the humans outside the game. Figure 2 traces the hinge: the agents are identical in all three panels, memory is the only variable that changes, and in the third panel we are the stick figure outside the ring.
Repeated-game bargaining favours the patient player with the long memory. Agents bring near-perfect recall, credible precommitment - code can bind itself in ways flesh cannot - and arbitrary patience. Humans forget, forgive at random, discount the future steeply and get tired by mid-afternoon. Worse, Binmore’s own selection device is missing. Fairness focal points work because humans share an evolved substrate: empathy, comparison, a lifetime’s rehearsal of give-and-take. And evolution may have hard wired some of this learning into human social interactions. A stable human-agent equilibrium is entirely achievable, although nothing in the mathematics says it will be one we would call fair, or one that has us as its main beneficiaries. It is possible to be inside the game and be its weak party. Figure 3 puts the asymmetry on the scales: everything on the agent’s pan is a repeated-game advantage, and everything on ours is a liability.
The agentic future needs identity, memory, horizon, and payoffs. It is the same as 1987 - circuit breakers were simply new payoff rules. But that fix was available because exchanges set the rules and could rewrite them by decree. The agent economy is now under construction. Deployers spawn stateless agents, because statelessness is cheap. No regulator yet has a venue for emergent machine collusion. Cooperation among agents is coming regardless; once memory arrives, the folk theorem all but guarantees it. Whether it is cooperation with humans depends on the rules of a game that, at present, nobody has written.