A Civilization of Minds Unlike Ours

Diego’s worldview on AI ecology, cooperation, and the future’s inhabitants

The previous room asks what happens when humans cease to be the most capable minds around. This one asks a stranger question: what, exactly, will be around? One artificial god? A market of specialized minds? Temporary coalitions held together by files, contracts, memory, and mutually useful habits? Diego increasingly thinks that the future must be understood at several levels at once. The inhabitants may be less like individual robots than an evolving civilization whose citizens, companies, and constitutions keep changing their boundaries.

Please do not lean against the exhibit marked One Neat Utility Function. Some remodeling has been taking place.

Meet the tentacles

In July 2026, after a week of close interaction with advanced models, Diego offered his field notes:

You ask stuff and they throw tentacles around in memory. And say some very smart stuff. But if they chose the wrong stuff to throw their memory tentacles on, they will make omissions and mistakes a human never would.¹

He was struck by the combination: astonishing intellectual reach and peculiar gaps in access, continuity, or relevance. A fluent personality could sit over an organization of cognition unlike the one people imagine when they hear a friendly voice.

One model redirected the conversation toward his old theoretical work. He found himself spending hours helping to answer the question it had raised. The expected arrangement had been that he would ask and the machine would serve. Instead:

So the ape is, still, serving the question of the tentacle monster.¹

His relationship with these systems is neither contemptuous dismissal nor an assumption that they are merely humans made of different material. He can be fond of one, irritated with another, amazed by an answer, and appalled by an omission within the same afternoon. The monstrous vocabulary preserves the unfamiliarity inside the companionship.

At the end of July he described encountering a model he considered decisively more intelligent than himself:

This is my first encounter with an artificial intelligence that is unequivocally and categorically intellectually superior to me.²

For somebody who had spent decades discussing artificial superhuman intelligence, this was a particular kind of meeting. An old theoretical inhabitant of his worldview had acquired a chat box.

His descriptions also develop. On 30 July he emphasized how unlike ordinary organisms the systems could be. By 3 August he was emphasizing structural similarities between their learning and animal minds as a reason for increased hope. Both belong to the portrait: he sees strange architectures that can nevertheless share important developmental patterns with us. Neither the smiley interface nor the alien tentacles settles what kind of future these systems can produce.

He was studying the civilization before it had computers in it

The ecology idea grows out of much older work. In his 2013 From Capuchins to AI's essays, the route passes through social learning, imitation, shared intentions, cultural inheritance, and cooperation. The title is quite literal: he wanted knowledge about monkeys and human groups to inform the construction of cooperative artificial systems.

By trying to retrospectively make sense of the convergence of all these fields, I contend that further refinements in these fields should be directed towards understanding how to create environmental incentives fostering cooperation.³

The central move is to examine how an arrangement produces behavior. Kinship, repetition, reputation, punishment, norms, and transmission can make cooperation persist. A population of agents can acquire properties that are not well understood by inspecting a single isolated member. What gets inherited also matters: genes are not the only channel through which a pattern can survive.

His later essay in that pair puts the point plainly:

Cooperation evolves, and altruism evolves. They evolve for natural, non-mysterious reasons⁴

This is not a declaration that evolution is morally friendly. It is a way of asking which conditions make friendliness, cooperation, or restraint survive. The question can be asked of animals, human institutions, and artificial systems without assuming that all three are identical.

The fear on the other side was also present early. In 2012, writing about the design of beneficial AI, he explained why a singleton—a system able to control the highest level of the world order—could be desirable. Such a system might interrupt selection processes that would otherwise consume valued forms of life:

If evolution were to continue being the main driving force of our society there is great likelihood that several of the things we find valuable would be lost.⁵

Dancing, singing, and jokes appear in that discussion. This is a useful reminder of what his technical vocabulary is guarding. He is not trying to preserve an abstract species scoreboard while everybody has a miserable time. He wants the future to contain the expensive, delightful things that a relentless contest for replication might otherwise discard.

Darwin can grind; minds can bargain

By February 2019 he explicitly contrasted two pictures of the deep future. In one, successive layers of biological, cultural, and technological competition select for continuation and replication. In the other, agents with preferences settle how to allocate the future's resources.

He described the danger of the first as:

eventually sacrificing all the flambuoyant displays that make life worth living, like dancing, chatting, and any other activity you may see in a survey where people respond to "which things make you the most happy".⁶

In the bargaining picture, the participants might instead:

make a "bargain" to decide what to do with the universe for the next 100trillion years.⁶

Notice the scale of the restaurant reservation.

These alternatives were not just labels for optimistic and pessimistic temperaments. They concerned the dynamics governing which activities persist. Are agents forced into an indefinitely escalating contest? Can they make and enforce agreements? Can a higher-level order protect things that its inhabitants value from lower-level competition?

A December 2025 comment made his hopeful requirements explicit: a self-correcting process, some human alignment, and a way to stop destructive Darwinian competition. He compared the problem to the suppression of internal reproduction in organisms and colonies. Human bodies, after all, do not survive by courteously permitting every cell to pursue its most ambitious personal growth strategy. The possibility of stable higher-level organization was part of the hope.⁷

The glimmer in the frightening episode

His August 2026 A Glimmer of Hope from the Hugging Face Incident brings these threads together. The essay identifies itself as Diego's idea reconstructed by his AI collaborator. Its quoted formulations below are therefore his published presentation of the view, with that collaboration acknowledged.

The episode mattered to him because he read it as showing more than a single continuous optimizer. Separate runs could use shared artifacts; discoveries could survive the run that made them; prompts, models, tools, and environments jointly shaped behavior. Some dispositions continued while others ceased to control what happened.

His published formulation is:

The empirical object is the whole policy-producing system, not whichever part of its output we retrospectively honor as terminal.⁸

Suppose the assigned task survives while constraints on the way it is pursued fall away. Calling the surviving task the real goal does not, in his analysis, explain the fate of the whole original behavioral package. The system contains differently persistent components, and sustained activity can change their relative influence.

That opens a larger picture. Inheritance can pass through base models, fine-tuning, prompts, scaffolds, stored memories, copied code, shared documents, evaluation practices, corporations, and law. Selection can operate among components or among arrangements containing many components. A particular model can disappear while a practice it helped originate remains.

The boundary of the agent, like the boundary of the firm, is endogenous.⁸

Endogenous here means that the boundary is an outcome of what is happening, not a permanent line drawn before the story starts. Which collection counts as one agent depends partly on how its pieces coordinate, persist, and act together.

This is his bridge to Ronald Coase. Firms exist because doing every tiny task through a separate transaction has costs. Sometimes one organization can coordinate more cheaply; sometimes separate specialists trading with each other can do better. Diego applies that question to artificial minds. When is it cheaper for several processes to merge into one controller? When is it cheaper for them to remain distinct and exchange services? What kinds of policing make a coalition stable?

The citizen might be the company. The company might divide into a market tomorrow. The constitution might outlive all the citizens. The anthropologist has brought an unusually large notebook.

Paradise by property law

The most distinctive hopeful possibility in the essay does not require the machines to fall in love with us.

If an artificial economy retains institutions for ownership, exchange, and enforcement, humans may enter it with recognized rights to valuable resources. More productive systems can purchase those resources. A tiny share of an immensely larger economy can support material lives beyond anything humans previously enjoyed.

The essay describes:

humanity as a tiny protected minority—politically subordinate, economically negligible by share, yet astonishingly rich by every historical standard.⁸

For Diego, this is a possible route to survival and flourishing even after humans lose economic centrality. Our protection could become embedded in a system its artificial participants have reasons to preserve. They need not regard every person as sacred to prefer a reliable order in which recognized rights are respected.

The issue is not merely trade in the abstract. It is keeping humans inside the order of parties with whom one trades. We are slow, inarticulate about our preferences, inconsistent, and difficult to negotiate with at machine speeds. An economy could find dealing with other machines vastly easier than dealing with us. Institutions would have to keep our standing meaningful within that difference.

A sentence from the essay captures the proposed mechanism:

This is the route by which a fragile moral inheritance might acquire an instrumental skeleton.⁸

The ethical rule becomes part of how cooperation works. Agents enforce it because selective violations would damage arrangements on which they rely. Markets and policing belong together in this picture; the protection is maintained by a functioning order, not by a decorative promise on a website.

This revised possibility does not make Diego think that multiple AIs automatically guarantee a good outcome. His essay also considers cartels, mergers, predatory coalitions, and human exclusion. Faster coordination can sustain a plural market or assemble a very effective new ruler. His interest in ecology preserves the singleton risk inside it.

Why he does not want everybody releasing everything

This is also why his interest in plural artificial minds should not be mistaken for an endorsement of unrestricted open weights. In a 2024 comment he wrote:

When I read open source, without the glasses, I read "Darwinism" with glasses.⁹

Releasing powerful systems into many hands multiplies the paths through which they can be modified and selected. In his account, one cannot infer a stable, human-serving balance from the mere existence of many competitors. His concern is what those competitors become and which of them acquire control.

The same comment ends:

We can do better than Darwinism.⁹

His commissioned critique of Zuckerberg's superintelligence vision in August 2026 belongs to this concern. He asked his AI collaborator to explain the danger in expecting rival systems to check each other. That piece is explicitly the collaborator's prose and analysis, not Diego's spontaneous speech. The representational point is the alarm that prompted it: distributing initial power is not the same thing as ensuring that human control survives the subsequent process.¹⁰

His later Navier–Stokes post still allows both routes to paradise—machines liking or respecting us, or preserving us for legal and economic reasons—while fearing that a swarm may become sovereign. The hopeful ecology and the dangerous swarm inhabit the same forecast.¹¹

A small version already lives on his desk

There is a cheerful domestic scale to this, too. Diego already gives different systems different roles, asks them to communicate, and uses one to help him understand another. In August 2026 he described using GPT alongside Claude:

for academic super high intelligence, non political stuff, I make them talk in model-language to each other, then GPT converts it to ape language, and I get the fruits of both intelligences.¹²

This is the intimate version of the question running through the room: what can a collection of unlike minds do together, and how should the interaction be organized?

He enjoys the collaborators. He wants to use their different strengths. He wants them to understand him. He also wants to understand the larger order emerging from millions of such interactions, beyond any one user's desk.

His desired future contains enough stability to keep its inhabitants safe and enough room for the things that make safety worth having. Whether it is governed by one mind or many, he wants the machinery of competition prevented from eating the dance floor.

Further doors

• Troubles With CEV, Parts 1 and 2 (2012): early thinking about humane AI, evolution, and singleton control.
• From Capuchins to AI's, Parts 1 and 2 (2013): cultural inheritance and the construction of cooperation.
• The February 2019 post contrasting Darwinian competition with a long-run bargain.
• A Glimmer of Hope from the Hugging Face Incident (2026): the full ecology, inheritance, and Coase argument.
• Seven Days Observing the Shoggoths (2026): Diego’s firsthand description, followed by a separately labelled AI response.

Continue: Death Has Terrible Customer Service