Is the singularity really near? Kurzweil, AGI and a future far less reassuring
Let us play a game. Imagine going back to 2005 and telling a software engineer that within twenty years a program would write working code from a sentence in plain English, pass a technical interview and summarize a contract better than an intern. They would have called you a visionary, or a snake oil salesman.
That same year Ray Kurzweil published The Singularity Is Near, and said things far bolder than that.
His underlying thesis is simple: technological progress does not advance in a straight line, it accelerates. According to what he calls the law of accelerating returns, each generation of computers lets us design the next one faster, and that gain cascades onto artificial intelligence, genetics, neuroscience and nanotechnology. Not a steady improvement, but an improvement that improves itself.
From there came two dates that became famous: 2029, when artificial intelligence should match human beings in most cognitive tasks, and 2045, the year of the singularity, when biological and artificial intelligence would begin to merge. In 2024, with The Singularity Is Nearer, Kurzweil confirmed both.
The uncomfortable part is that 2029 is no longer distant science fiction. It is the day after tomorrow.
How it might unfold, step by step
Anyone imagining the singularity as the moment a machine opens its eyes and says "I exist" is probably looking in the wrong direction. The transition, if it comes, will be far less cinematic and far harder to notice while it happens.
The first stage has already begun: agents that use software, write code, handle communications, read documents and coordinate entire work processes. They are not yet reliable enough to call them AGI, but they do not need to be in order to replace a sizeable slice of repetitive cognitive work.
The second stage arrives with persistent memory and long term autonomy. An agent that does not produce an answer and shut down, but follows a project for weeks, changes strategy when something does not work and coordinates other agents. Here the word "tool" starts to feel too tight.
The real leap happens when these systems learn a new skill without anyone training them for it specifically. Try asking yourself where the tool ends and the collaborator begins: the answer at that point is no longer obvious, and neither is it obvious from a contractual, fiscal and legal standpoint.
In Kurzweil's vision the finale is the direct link between brain and machine: neural interfaces and, one day, nanomachines that connect the mind to the computing power available on the network. It is the most speculative part of all, and it is worth keeping separate from the rest so as not to throw out the whole analysis along with it.
Where Kurzweil got it right
On one thing he is right, and history proves him right with some regularity: we are terrible at grasping cumulative effects.
A ten percent improvement seems small. Repeated enough times across hardware, algorithms, data and the automation of research itself, it shifts the boundary of what is economically possible far faster than our intuition can follow. It is not a technical limit, it is a limit of ours: the human brain reasons in sums, technology proceeds in products.
And here comes the point that often escapes even the skeptics: to change society you do not need superintelligence at all. All it takes is a system even less brilliant than a competent person, but one that works twenty four hours a day, replicates at almost zero cost and reasons at the speed of silicon. It does not have to outsmart us. It only needs to exist in a million copies.
Where the reasoning cracks
The most fragile step in the whole construction is a single one, and it sits at the beginning: Kurzweil jumps far too casually from the growth of computing power to the growth of intelligence.
If computation were enough, how do we explain that a system trained on more text than a human being could read in a thousand lifetimes keeps mistaking a correlation for a cause, failing to notice it has made a mistake, and collapsing in the face of a situation only slightly different from those it has already seen?
Because intelligence is not a single quantity that rises like a computer's RAM. It is a set of distinct capabilities: adaptation, judgment, planning, understanding others, creativity, awareness of one's own uncertainty, relationship with the physical world. Some are improving fast. Others almost not at all. Summing them into a single number that doubles every so many months is a rhetorical convenience, not a measurement.
Then there is a problem with the curve itself. Technologies tend to follow an S: they start slowly, explode, and slow down when they hit a physical, economic or organizational limit. The trouble is that an S curve, seen from inside while you are traveling along it, is indistinguishable from an exponential. Every exponential curve in history has later turned out to be the first half of something else. How do we know that this time is different? The honest answer is that we do not know, and neither does Kurzweil.
The researcher Toby Walsh added a more technical objection that is harder to dodge: for a singularity to happen, very intelligent machines are not enough, you need machines capable of redesigning themselves again and again without ever hitting a new bottleneck. It is a huge assumption, and no one has ever shown that the world works that way.
The real risk arrives well before the singularity
There is, however, a point on which Kurzweil, more than being wrong about timing, seems wrong about human nature: the idea that the benefits will distribute themselves on their own.
Technology can become cheap without power being distributed. The leading models require capital, energy, chips, data and infrastructure that very few players in the world possess. Making inference ten times cheaper takes nothing away from whoever controls its production.
So the right question is not when AI will be cheap. It is: cheap for whom?
Because the first effect of AGI might not be abundance, but an unprecedented asymmetry: some organizations with millions of digital workers at their disposal, and everyone else with bargaining power quietly thinning out, without a single day when anyone announces that something has happened.
And there is a side effect that gets far too little attention. Automation does not strike at random: it strikes the entry level tasks first, the simple ones, the ones you learn on. But if we erase the work through which people become experts, where do we think the experts of fifteen years from now will come from? Nobody becomes senior by skipping the apprenticeship. We are about to find out what happens to a profession that cuts the first rung off the ladder.
Then there is the question of information, which is the most insidious. When texts, images, videos and identities are generated on an industrial scale, we do not only lose the ability to tell the true from the false. We lose something more fundamental: the common ground on which two people who do not trust each other can still agree on what happened. A society can survive many crises. It struggles badly to survive without a shared reality.
The most likely future makes no noise
Perhaps the singularity will not arrive as an explosion. Perhaps it has already begun, and it looks like a slow surrender of control.
First we delegate writing, because it saves time. Then analysis, because it is faster. Then operational decisions, because it makes fewer mistakes than we do in ordinary cases. And at some point we notice that companies, infrastructure and institutions rest on systems no single person is able to fully understand any more. Not because anyone decided it. Because each single step, taken on its own, seemed reasonable.
And here lies the truly unsettling thing, the one that should keep both optimists and skeptics awake: Kurzweil does not need to be right about everything. He only needs to be right about the speed, and wrong about one thing alone, our ability to adapt in time.
So let us stop asking only when AGI will arrive. It is the easiest question and the least useful. The serious questions are three others: who will control it, what goals we will give it, and above all how much power we will be willing to hand over before we have understood what we were handing over.
Because on this we already have one piece of data: so far we have handed it over in exchange for convenience, one piece at a time, without anyone asking our permission. And we have barely noticed.