Second-Order Sterility

On what happens when a system’s inputs become its own outputs.

There is a failure mode in machine learning with a name most people outside the field have not heard, and a shape everyone outside the field will recognise immediately.

Train a model on data. Let it generate. Train the next model on what it generated. Repeat a handful of times. What comes out is not noise. That is the part that surprised everyone, including the people who found it. You would expect degradation to look like static, like a photocopy of a photocopy going grey and illegible.

It does not. It looks like competence.

The output gets smoother. More typical. More confident. More like the average of everything the model ever saw, and less like anything in particular. The strange cases go first, then the rare ones, then the merely unusual, until what is left is a very fluent machine producing the most expected possible version of everything, having lost the ability to produce anything else.

I have not been able to stop seeing this since I understood it. Not in machines. Everywhere else. But let´s start somewhere simple.

The mechanism, plainly

Every generation of the model samples from the previous model instead of from the world.

Sampling is lossy in a specific way: it is lossiest at the edges. The centre of a distribution is dense and survives being copied. The tails are thin. A rare thing, by definition, appears rarely in any sample, and the rarer it is the more likely it is to be missed entirely. Miss it once and it is gone from the training data forever. The next generation cannot miss it, because it no longer knows it existed.

So the variance collapses before the mean drifts. The weird dies before the wrong arrives.

And here is the property that makes this dangerous rather than merely sad: the system’s confidence rises as its contact with reality falls. It is not hedging. It is not producing garbage that a reviewer would catch. It is producing the most plausible possible thing, with total fluency, about a world it is no longer looking at.

I want to give this its own name, because it is not a machine learning problem and treating it as one is why we keep missing it in the places it is doing real damage.

Second-order sterility: the condition of any system whose inputs have become its own outputs.

First-order means in contact with the thing. Second-order means in contact with a representation of the thing. A system can run for a very long time on second-order inputs. It will feel productive the entire time. It will produce nothing new.

The tell is not decline. It is smoothness.

We are trained to look for decay as a drop in quality. That instinct is exactly wrong here and it is why this is hard to catch.

Sterility does not present as bad work. It presents as consistent work. Fewer failures. Fewer embarrassments. Fewer things that make no sense. The edges come off, the outliers stop appearing, and everyone involved experiences this as the field maturing, the team getting more professional, the taste improving.

The diagnostic question is not is the quality falling. It is:

When did this system last produce something that surprised it?

If the answer is years, the system is sterile. It does not matter how much it is producing. Volume is not evidence of life. A sterile system can run at full capacity indefinitely, which is precisely what makes it hard to shut down.

In a life

Start with the smallest instance, because it is the one you can check today.

Your feed is a model trained on your outputs. You click, it learns, it serves you a version of what you already are. Then you read that, and it becomes what you think, and you click accordingly. Within a few cycles you are not receiving information about the world. You are receiving a compressed representation of your own previous behaviour, delivered with the emotional texture of news.

Your beliefs get more confident. They also stop producing surprise. Nobody experiences this as narrowing. Everyone experiences it as finally understanding what is going on.

The same loop runs without any technology at all. It is called rumination, and it is the mind training on its own output until the tails of its own experience are gone and only the most typical, most rehearsed version of the story remains. This is why thinking harder about a problem you have thought about a thousand times produces nothing, and why a single conversation with someone who does not share your priors can produce more in an hour than a year of reflection. The conversation is first-order. The reflection was not.

And the most common version, the one I see in ambitious people constantly: reading about the work instead of doing the work. Books about building, podcasts about founding, frameworks derived from other frameworks derived from a handful of companies nobody has looked at closely in twenty years. All of it fluent. All of it second-order. You can spend a decade in there and come out with an excellent vocabulary and no scars.

In an institution

Now scale it up and the examples stop being metaphors.

Academia largely trains on academia. The unit of contribution is a paper, and the raw material of a paper is other papers. There are fields where the object of study has not been touched directly in a generation, where the literature is the terrain, and where the fastest route to a career is to model the models. The output is fluent, voluminous and confident, and much of it has not been in contact with anything that could have said no.

Finance prices prices. The instrument references an index which references an instrument, and at some depth of derivation the underlying asset stops functioning as a real object with a real yield and becomes a number that other numbers refer to. The 2008 mechanism was not exotic maths. It was a system whose inputs had become its own outputs, running at full confidence, with variance collapsed to nearly nothing right up until it wasn’t.

Management by dashboard is the same disease with better branding. The dashboard is a representation. Decisions get made on the representation. The representation then becomes the thing people optimise, which changes what the representation reports, which drives the next decision. Nobody in the loop is looking at the customer, the product or the shop floor. Everyone is looking at a compressed model of them, refreshed hourly, rendered in a very good font.

I will name my own industry, because it is the purest case I know. In compliance, we produce attestations. An auditor examines the attestation and produces a report. A framework maps the report to another framework. A certification asserts the mapping. At no point in that chain is there a requirement that anyone look at the running system. It is possible to be perfectly compliant and completely insecure, and the reason is not that people are lying. It is that the entire structure has been trained on its own outputs for so long that the tails, the strange failures where breaches actually live, are no longer represented anywhere in it. I sell software into this. I think about it a great deal.

In a civilization

And then the largest scale, where two versions are running at once.

The obvious one: culture remixing culture. Sequels of adaptations of reboots, aesthetics trained on aesthetics, every new interface converging on the same rounded rectangle because it was trained on the last one. Not bad. Smooth. Increasingly typical. Increasingly incapable of producing the thing nobody expected. And now genuinely recursive, because a growing share of what the next models will learn from is what the current models wrote.

The less obvious one matters more, and it is the reason I care about this beyond aesthetics.

We are losing the ability to build physical things, and we are replacing it with the ability to simulate physical things. These are not the same skill and the difference is invisible from inside. A simulation is trained on the assumptions of the people who wrote it. Reality is not. You can produce an entire generation of engineers who are excellent at modelling reactors that nobody has been permitted to build, and the models will get smoother and more confident every year, and the tails, the failure modes that only appear when you actually pour the concrete and run the thing hot for a decade, will quietly vanish from the training data because nobody generated any.

The gap does not announce itself. It shows up the first time someone tries to build, and discovers that the accumulated knowledge was a very high-resolution picture of a thing, not the thing.

What reality is actually for

Here is the inversion, and it is the whole argument compressed:

Reality is not primarily a source of information. It is a source of refusal.

We think of contact with the world as how we learn things. That is the smaller half. The larger half is that the world is the only thing that can tell us no. The concrete cracks. The customer doesn’t buy. The crop fails. The patient does not improve. The plane does not fly. None of this is information you asked for, and all of it is unavailable from any representation, because a representation only contains what its makers already knew to put in it.

A system stays alive exactly as long as something outside it retains the power to reject its outputs.

Remove that and nothing dramatic happens. That is the trap. You do not get punished. You get smooth, confident, high-volume production, forever, in a slowly narrowing space, and everyone inside experiences it as things going rather well.

What to do on Tuesday Monday

I distrust philosophy that doesn’t tell you what to do this week. So:

Audit one week of inputs. What fraction of what you read, watched and discussed was produced by a process that also consumes you? Not a moral question. A measurement. Most people find the number is above ninety percent and are genuinely shocked, because it never felt like a closed loop from inside.

Read the dead. Anything older than fifty years was written by someone who could not possibly have been trained on you. That is its entire value, and it is why old books feel strange in a way new books rarely do. The strangeness is the signal. It is the tail of the distribution, still intact.

Build one refusal loop. One thing per week that can fail in public and tell you so: ship it to a stranger, make a physical object, run the measurement, ask the customer who churned. Not feedback that is filtered, aggregated or delivered as a summary. The unmediated no.

Touch the object. If you are deciding on a report, go and see the thing the report describes, once. You will find something not in the report every single time. This is the highest-yield hour in management and almost nobody spends it.

Ask the sterility question of anything you are part of. When did this last produce something that surprised it? Ask it of your company, your marriage, your field and your own head. Volume will try to answer for all of them. Do not let it.

My honest close

I am not writing this from outside it.

I use these models every day. I run a business on them. I write with them, I ship software that generates attestations for a living, and I have spent whole weeks in the reading-about-it loop, mistaking fluency for progress and vocabulary for scars. Nothing here is a warning from someone standing on dry ground. It is a description of the water I am in.

What I have changed is small. I keep a short list of things that can tell me no, and I make sure something on it gets a chance to every week. That is all. It turns out to be surprisingly hard to arrange, and surprisingly clarifying when it happens, and I notice that every genuinely new thing I have thought in the last year arrived within a day of something refusing me.

The question I am sitting with, and I do not have the answer:

If contact with reality is the scarce input, and we are building an economy whose defining technology consumes representations and produces representations, then what is the most valuable thing a person can have in twenty years? I suspect it is not intelligence. Intelligence is about to be extremely cheap. I think it is contact. Verified, unrecycled, first-order contact with something that has the standing to say no.

And I do not think we are building any of that.

Written for the Anima Mundi community. July 2026.