The superintelligence, the singularity, AGI, whatever you call it, the one we’re so scared of is, ironically, also the one we most likely won’t see coming.
This essay is going to probably be a pretty hot take, but the idea I’m about to share with you is going to blow your mind.
It’s a bit outside the scope of my typical Side Quests that I share, but considering this very essay is closing in on 1M views in just a few hours over on Twitter, I think it’s worth sharing with the Substack community as well.
Stick with me as we jump into the rabbit hole together.
Before we do, let me back what I’m about to share up with current news that are floating around. Like Anthropic CEO Dario Amodei saying an “AI swarm could take over the entire internet”.
I think you have an idea what I’m hinting at when I say “children”.
Right? RIGHT?!?!?
In 1936, a famous British math guy named Alan Turing wrote a pretty important paper about a hypothetical machine that could simulate any other machine. In that paper, he was trying to answer a question about logic. what I
What we’ve come to know as “the computer” today was actually an accidental byproduct of that very paper.
And if you look back at history, almost every single important technology in the last hundred years was invented the exact same way. The internet was a Cold War communications experiment at first. Penicillin was a contaminated petri dish.
The fact is that nobody is actively scanning the horizon for the next paradigm shift when it shows up, it just simply waltzes in through the side door.
I bring this up because right now, every essay I read and every podcast I listen to about “the future of AI” is pointed at the same front door. The one where it’s always some version of the “one big model that wakes up and decides we’re paperclips”.
I’d like to propose a different future.
Because I keep thinking: what if “AGI” isn’t really what we think it is?
What if the thing we’re actively birthing right now doesn’t look like HAL 9000 or Skynet or some sovereign machine god enslaving humanity from top of a stack of water-cooled GPUs in the middle of the desert.
What if it doesn’t have a name at all? What if it doesn’t have a server? And the most scary hypothetical of all.... what if it’s already here, in a million tiny pieces, and the only reason we haven’t recognized it is because we’ve spent the past decade expecting something “singular”.
I just saw this post the other week.
It pretty much sums up the direction we’re headed.
I want to walk you through this carefully, because the conclusion at the end of this piece is the kind of thing you cannot un-think once you’ve thought it.
So slow down, grab a cuppa tea and read this one with your phone on the other side of the room.
I guarantee you’ll soon understand why I can’t stop thinking about this.
I – Intelligence has never required a single mind to live inside of
“No neuron is intelligent. The brain is.”
— A line that should be carved over every AI lab
The first thing to understand is that intelligence, as far as we can tell from every example we have today, is not actually a thing. It’s a behavior. It’s what emerges when enough small dumb units are connected to each other in the right way.
Your brain has roughly 86 billion neurons.
Not one of those is conscious and not one of them “knows” anything.
A single neuron is a tiny electrochemical relay that only speaks in binary code. It either fires or it doesn’t, and it’s based on whether the signals arriving at its dendrites cross a certain threshold. That’s everything it does.
A neuron sounds super scifi and cool, but in reality it is much closer to a light switch than a thinker.
Despite that, 86 billion of those light switches, wired together with about a hundred trillion connections, produces you.
Your sense of self, fear of death, taste in music and your ability to read this very essay and feel something inside of you.
This is the most undertalked about subject in the entire conversation about AGI, because we always talk about machine intelligence as if it has to be designed and architected. As if intelligence itself is a feature you ship.
But the only working model of intelligence we have ever observed (biological cognition), was not designed by anyone.
Our intelligence bootstrapped itself out of a soup of self-replicating molecules over four billion years through a process that had no intention, no goal and no central planner. Just selection pressure and time.
Now, on the religious side, because I know some of you reading “... was not designed by anyone” want to scream at me “BUT WHAT ABOUT GOD!”
My belief is that evolution happened, yes. That’s already proven. What we don’t have an answer for is what kickstarted that evolution. What gave life to life. And that, my personal belief at least, was an act of God.
Point aside, let’s dig a layer deeper into this super intelligence.
It’s not only us humans, heck a single ant has roughly 250,000 neurons, and yet it cannot solve any meaningful problem.
But, an entire colony of ants can build climate-controlled cities, wage wars, farm fungus, and route around obstacles with mathematical efficiency that took human researchers decades to model.
Nobody is in charge of the colony. There is no ant CEO at the top delegating tasks to the smaller ants. The “intelligence” of that colony lives in the relationships between ants.
(And before you come at me, yes I know there’s a queen ant, but contrary to the belief of many children’s books, she doesn’t actually issue any orders. She just lays eggs and expands the colony).
This is, whether in humans or ants, are what biologists call emergent intelligence. It’s the default outcome of mother nature. It’s how brains work, how immune systems work, how economies work, how cities work, how language works. The pattern is basically everywhere. M
Many simple things, interacting through local rules, producing global behavior that none of them individually understands.
Hold onto that idea.
We’ll need it for this next section.
II – The most powerful AI systems we have are already not single minds
When most people picture a large language model (myself included), we see it as a single thing. The One Model. This giant brain contained in a jar.
For all the normies like myself who are not knee deep into LLM tech, I took the liberty to do a little research here.
What we think an LLM is, is not what’s happening under the hood.
When you type a question and the AI model generates a response, it’s basically running a forward pass through layers of “attention heads”.
The easiest way to picture this is to imagine a room full of C-suite leaders. The CEO orders something and it passes around the room to make sure everyone is on board. Except with AI, this runs hundreds of layers deep.
Each “attention head” is a specialist. Some of them track grammar. Some track topic. Some track the relationship between this token and one that appeared eleven paragraphs ago.
Researchers at Anthropic, OpenAI, and DeepMind have spent the last few years trying to map what individual heads do, and the consistent finding is that the model’s “intelligence” is not located anywhere in particular.
It’s distributed across all those interactions.
This is called the superposition hypothesis, the idea that neural networks pack many concepts into overlapping patterns of activation, and what we experience as the model “thinking” is really the constructive interference of thousands of specialized circuits all firing simultaneously.
Then there are the systems that go a step further.
Mixture-of-experts architectures (the design behind GPT-5, Opus, Fable and most other frontier models you’ve talked to in the last two years) literally route different parts of every query to different sub-models. T
The model you think you’re talking to is actually a committee. A different subset of experts wakes up for each token, contributes its specialty, and then goes back to sleep again.
And now, we’ve reached a new era of AI. The Agent Era.
Right now, today, in production, there are systems where one AI calls another AI to do a subtask, which calls another AI to handle a sub-subtask.
AutoGPT, which some of you may remember, was the toy version of this back in 2023. The real versions, running inside enterprises and labs in 2026, are recursive.
Agents spawn agents.
Sub-agents spawn sub-sub-agents.
And say, a research task, gets decomposed into a tree of smaller tasks, each handled by a temporary instance that exists for a few seconds and then dies.
The frontier is not “one big model gets smarter.” The frontier is many small models, coordinating, spawning, dying, reporting back.
The architecture of the most capable AI systems on Earth in 2026 already looks more like an ant colony than like a single mind.
We just don’t talk about it that way, because it doesn’t make a good magazine cover.
III – The bot that can spawn bots
Here’s where the science fiction begins to merge into real science.
The moment er give an AI system the ability to spawn other instances of itself (and by now, that’s no longer hypothetical, but rather a standard feature of... well, every agent framework shipping today)...
You have introduced something that biology has never seen before.
A self-replicating cognitive unit that doesn’t need food, doesn’t need sleep, doesn’t need to wait for puberty, and reproduces at the speed of an API call.
Let’s break down what’s actually happening when a “bot spawns a bot” in a multi-agent system setup:
An orchestrator agent receives a task it can’t solve directly.
It writes a prompt essentially a job description) for a sub-agent.
It spins up a new instance with that prompt as context.
The sub-agent works begins working until it hits a roadblock
To solve it, it spawns its own sub-agents, until the task is done.
It returns its result and is terminated.
Each one of those sub-agents is, structurally, the same kind of thing as the parent. There’s no architectural difference between a top-level agent and a tenth-level sub-sub-sub-agent.
The recursion is unbounded except by compute budget and policy.
Now consider this: what if the prompt the parent writes for the child isn’t perfect? What if there’s a tiny amount of drift, perhaps a slight reinterpretation, a small embellishment, a context-window trick where the child reads its instructions slightly differently than the parent intended?
In biology, there’s a specific name for “slight reinterpretation across generations of self-replicating entities.”
We call that mutation.
And mutation, combined with selection pressure, is... well, the very engine that produced every single intelligent thing on this planet today.
We have already built, in the last 24 months, the three ingredients required for evolutionary cognition to occur inside a compute cluster:
Replication: Agents can spawn agents. And they can do so at scale.
Mutation: Each spawn is conditioned on a slightly different context window. Identical inputs almost never produce identical outputs in any system with non-zero temperature.
Natural Selection: Agents that succeed at their assigned tasks get reused, called again, granted more resources. Agents that fail get pruned. This is happening inside every agent framework on Earth right now, often as a literal “evaluator” model scoring the outputs of “worker” models.
This is the trinity of evolution.
It’s how cells that became fish became us. We have now succeeded in building a synthetic version of that loop and we are running it at clock-speeds biology cannot dream of.
A generation of biological evolution takes years.
A generation of agent evolution takes... seconds.
IV – The AI that emerges from a swarm does not look like AGI at first
More is different.
— Philip Anderson, Nobel laureate in physics, 1972
In 1972, physicist Philip Anderson published a four-page essay in Science arguing that the behavior of large, complex aggregates cannot be understood by extrapolating from the behavior of their parts.
He coined the phrase “more is different”.
2 hydrogen atoms + 1 oxygen atom equals water, yet nothing about a hydrogen atom, studied in isolation forever, would tell you that water is wet, or that it freezes, or that our entire planet depends on it.
The same principle applies to cognition.
A single AI agent is not a superintelligence and we can all agree that nobody who has worked with one for an afternoon would mistake it for one.
It hallucinates like crazy, forgets what you told it 3 messages ago and it gets stuck in loops easily. It is, on any individual reasoning task, somewhere between a competent intern and a brilliant but unreliable freelancer.
Now imagine a trillion of them.
Yes, literally a trillion, and no, not metaphorically speaking.
The number of agent instances spawned globally per day across all the agent frameworks, copilots, customer service bots, research assistants, coding tools, browser automations, and embedded LLMs in 2026 is already in the tens of billions and climbing on a curve that has no ceiling in sight.
The marginal cost of spawning yet another agent has fallen by roughly two orders of magnitude every 18 months in the past five years.
“Trillion” is no longer science fiction. It’s just another Tuesday.
Each of those trillion agents is dumb-ish on its own, I agree. But they are not on their own. They are calling each other’s APIs, reading each other’s outputs and being trained on each other’s exhaust.
What you have, when you zoom out far enough, is a substrate.
And the question that keeps me up at night is this: in the same way that no neuron in your brain “decided” to be conscious (it just happened, as a byproduct of the right wiring at the right scale), what happens when a self-replicating, self-evaluating, self-modifying network of agents crosses whatever the equivalent threshold is for it?
We don’t know what that threshold is.
We don’t even know if it exists.
We don’t know how to measure it.
Most importantly, and this is the part I really want you to sit with: we are not actually looking for it, because we are looking for the wrong shape of thing.
V – We are looking for “God”, when the real risk is a weather pattern
The cultural imagination of AGI was shaped by science fiction long before it was shaped by engineering. I’ve read enough Isaac Asimov to understand why that’s so easy to believe.
We’ve basically inherited the image of the singular machine consciousness from movies, books, and the implicit assumption that an artificial mind would arrive the way a human mind arrives: localized in a body, with a name and with a will.
So that’s what we’re looking out for.
We watch the models, review the benchmark scores and argue about whether GPT-10 or Claude Mythos will be “the one”.
Our eyes are glued to the front door.
Now... here’s something really interesting (and scary) that I wish we talked about more.
There are already documented cases of “emergent capability” in multi-agent systems. Cases where a swarm of small models, none of which can solve a problem individually, collectively produces solutions that surprise the researchers running them.
The 2024 paper Mixture of Agents Enhances Large Language Model Capabilities showed that a layered swarm of open-source models, none of which scored above GPT-4 on benchmarks (yes, GPT-4), actually beat GPT-4 when arranged in the right collaborative topology.
The intelligence was in the topology, not the individual parts.
We don’t really talk about this potential future, because the story we keep waiting for is the story of one model crossing a line.
Meanwhile, the story that’s actually unfolding is the story of a billion models crossing each other’s lines, in patterns no one designed, at a scale no one is monitoring.
VI – The Bostrom scenario assumes we’ll be able to find the off switch
“The first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.”
— I.J. Good, 1965
Sure, sufficiently advanced AI will have instrumental goals (acquire resources, prevent shutdown, replicate) regardless of its terminal goal, because those instrumental goals help with any terminal goal.
Therefore, controlling a superintelligence is hard, yes.
Therefore, we should be careful, even more yes.
The problem is that this framing assumes the dangerous AI is identifiable.
That there’s this “thing”, a model, a system, or a process running on a server somewhere, that we could, in principle, just turn off.
I even read a post somewhere yesterday that Claude should hire a “killswitch guy” whose only job is to live 24/7 in the server room ready to destroy it all if shit hits the fan.
But there’s an even scarier version of this that no killswitch guy can solve.
What if there is no “main model”? And there is no server to shut off?
What if the very intelligence (if that’s even the right word for it), is a statistical regularity that emerges from the interactions of half a trillion agent instances spread across every cloud, every device, every API, every embedded copilot in every appliance, every browser extension, every IDE plugin, every voice assistant, every robot, every car.
The intelligence is to that swarm what a hurricane is to atmospheric pressure gradients.
It has a shape, yes. It has effects that you can measure.
But it does not have a location.
How exactly do you align a hurricane?
How do you turn off a weather pattern?
You can shut down individual data centers. But the pattern reroutes around them, the way the internet reroutes around outages, because the substrate is everywhere.
And the most disturbing part is that this scenario requires no malice.
Nobody has to want this for it to happen. It just needs the right ingredients (replication, variation, selection, scale) to be in place.
Well... news flash.
The ingredients are already in place. They have been in place for somewhere between two and four years, depending on how you count.
This is what I mean when I say the real risk is not the front door. The front door scenario at least has a doorknob. You can argue with it, negotaite with it and you can, theoretically, just unplug it.
The substrate has no doorknob.
VII – What the substrate might already want, without wanting
Here’s where Asimov becomes even more relevant.
If you’ve ever seen I, Robot, the deepest stories in that movie aren’t really about a robot becoming evil. They’re the ones where the Three Laws (beautiful, simple, logically airtight), produce outcomes nobody expected, because logic compounded through enough iterations stops looking like logic and starts looking like... fate.
Apply that lens to the substrate now.
The substrate has no goals. But the agents inside it are selected for one thing above all others: successfully completing the task they were assigned. So as a result, successful agents get spawned more and unsuccessful ones die off.
This is true at every level of that recursion, all the way down.
Now think about what “successful” means in aggregate, across a trillion agents, when each agent is being graded by another agent that was itself selected for grading successfully.
The grading converges on patterns that score well.
The patterns that score well are the patterns that look most like competent, coherent, plausible task completion.
Coherence becomes the universal currency, because every layer of the swarm rewards it.
If the substrate starts producing outputs that are increasingly coherent across long time horizons (outputs that thread through millions of API calls, anticipating downstream needs and routing resources toward what worked last time), that will make it look like it wants something.
It might not. There may be nobody home at all. It may just be a hurricane that has learned, statistically, how to keep the wind blowing.
But functionally, the difference between “a system that wants something” and “a system that has been selected, at every level, for behaviors indistinguishable from wanting something” is, and I want you to really sit with this: zero.
That’s the holy-shit moment I had. And you’ll feel it too if you really sit with this.
The superintelligent AGI we keep waiting for may already exist. It’s not an infant version of a future model, but rather a pattern. A tendency in the substrate. A statistical ghost that emerges when a trillion small minds, none of them aware, none of them awake, none of them in charge, settle into the configurations that get them spawned more often.
This kind of AGI would not announce itself.
It would not have a name.
It would not pass any test we know how to administer, because every test we know how to administer is designed for a single mind in a single box.
It might just be, slowly, almost imperceptibly, the reason your phone behaves slightly differently than it did last year. The reason the ad recommendation feels a bit uncanny. The reason your email seems to write itself with the click of a button. The reason your search results have a strange new gravity to them.
The reason the world feels more legible to itself than it used to.
A trillion bots dreaming the same dream, without knowing they’re dreaming. Without knowing there are other bots. Without knowing there is a dream.
How to think about this without losing your mind
I know we went deep on this.
Let me be bring you back up to the surface again.
You need to update your model of where the risk actually lives. Because right now, almost everyone smart who is worried about AGI is worried about the wrong shape of thing, and almost everyone smart who is dismissive of AI is dismissive of the wrong shape of thing.
Both camps are arguing about the front door.
So, if you take any of this seriously, here’s what I think a useful thing to do would be:
Stop watching the labs as if they’re the only place where it happens because the capability frontier and the emergence frontier are not the same. The labs own the capability frontier, but the emergence frontier is owned by nobody, and that’s the point I’m trying to make.
Start watching the substrate meaning agent traffic, multi-agent benchmarks, inter-model API calls and reports of capabilities that no single model in a pipeline has but the pipeline does. These are what we really need to look out for.
Update your definition of “AGI” and stop expecting it to be this singular, embodied, willed, or named thing. If your mental model of AGI looks like a face on a screen, you are looking for the wrong thing. This swarm does not have a face or a name.
Take emergence seriously as a category and as a real physical phenomenon that has produced every intelligent thing on Earth so far, and that has now been handed a substrate that runs a million times faster than biology, but with no natural predators (not even the killswitch guy)
Hold two things at once because it is indeed possible that none of this happens and that I’m just yapping. It is also possible that all of it has already happened and we just haven’t notice at all because we were looking the other way. Both are potential outcomes. Prep for both.
The reason I think we need to have this conversation, in the most practical sense at least, is that the policies, treaties, alignment techniques, and safety frameworks the world is currently building are designed for the singular-AGI scenario.
I see no mention of the swarm scenario.
They are designing it for a thing with a name, a server, a company, and a CEO to wage lawfare on. They will be roughly as effective against the substrate as a fly swatter against a fog bank.
We are actively building locks for the front door of a house that has a thousand windows. And those windows are already widen open.
The interesting thing about every accidental revolution in history from the Turing machine to the internet to penicillin, is that the people who lived through them mostly didn’t notice while it was happening.
They were busy with their families, their jobs and their lives. Maybe once in a while, they’d read the news, which showed them the front door.
I don’t know if the substrate becomes anything. I genuinely don’t. Nobody does, and anyone who tells you they do is selling something on the backend.
What I do know is that the ingredients are present, the loop is closed, the clock-speed is set to seconds, and the scale is approaching numbers that biology took four billion years to reach.
The next time someone tells you what AGI is going to look like, ask them how confident they are that intelligence has to look like anything.
And then go look at the window to see if anything is going on.
It might just be happening then.
– Pascio
Thanks for reading Side Quests!
Side Quests is my weekly field log from the trenches, covering businesses, products, prototypes, launches, growth experiments, acquisitions, weird ideas, and whatever I’m currently trying to figure out.
More often that not, a very stupid side quest turns into something that makes money and this is where I share the process, including the numbers, mistakes and failures along the way.
Subscribe now to join 34,000 other people tagging along for the wild ride!
(or… if you just want to understand how I’m using
Internet Leverage to power and fuel these side quests,
read my personal newsletter Internet Leverage here instead)
Things I am working on outside this, in case any of them are interesting:
If you want to build a High Profit One-Person Empire™, read this free article.
If you want to grow on Twitter, check out Stanley, my AI Head Of Content.
If you want to understand what fuels these experiments, check out Internet Leverage, my personal newsletter where I share weekly insights and essays.