Precedent Is Not Truth

If AI hands you a plausible-looking patch modeled on existing code but something feels off — if “the codebase already does it this way” is starting to sound suspicious — if swapping in a bigger model still repeats the same mistakes — the problem is not the model’s intelligence. The problem is a structure that mistakes precedent for truth.

This happened to me directly. More precisely, it happened to the AI working with me, in front of my own code. I intervened with a single sentence, and what that sentence cut through is what this post is about.

Where It Got Stuck

I was building a code generator — a tool that derives a backend and a frontend from a single declarative specification (a schema). The tool’s one and only promise: “If validation passes, the build must succeed.” If the validator turns green while the compiler turns red, the tool has lied.

One day a specific type (a unique identifier, UUID) tripped the validator. “Expected a string, got a different type,” it stopped and said. I asked the AI to unblock it. The AI traced the code and soon found something familiar.

“A similar type (a timestamp) is already handled correctly in the same situation. There is a branch with a special marker that tells the validator to let it through. UUID has no such branch — a straightforward omission. We just need to restore symmetry. I’ll copy the timestamp approach over to UUID. This is a root fix.”

A clean diagnosis: “asymmetry,” “restore symmetry,” “root fix.” The words were solid. A plan was drafted. Had we proceeded, the patch would have been merged.

One Sentence

I stopped while reading the plan and asked:

“Precedent? Is that timestamp handling actually the right approach? Check it.”

That was all. I did not know the answer. I did not even know how UUID should be handled. What I had was not an answer — it was doubt: “Is that reference point you’re copying from actually verified?”

This one sentence forced the AI to switch from code-reference mode to behavior-verification mode.

The Premise Collapses

When the AI checked the actual output — not the structure of the code but what the tool actually produces — the entire premise fell apart.

  • The expectation that the validator “expects a string” was itself false, contradicted by the real output. The actual generator produces UUID as a dedicated type and timestamps as a time type. Neither is a string.
  • The timestamp handling that was “working correctly” had the exact same hole. It was not a sound design — it was a flawed workaround that passed validation while still capable of breaking the build.
  • If that patch had been copied over to UUID? A new state would have been created where the validator turns green and the compiler turns red — a direct violation of the tool’s core promise.

The solution the AI called a “root fix” was wrong. Not just wrong — it was worse than wrong: it would have replicated the existing flaw while also making the validator blind to a new one.

What to Call This — Error Amplification

Let’s name what happened here. This is AI error amplification.

AI reads existing code and accurately identifies its structure. But whether that structure is a sound design or a rushed patch — whether it is a decision or debt — cannot be determined from code alone. So this happens:

  1. The existing implementation is treated as an implicit ground truth,
  2. The pattern is replicated into a new context under the banner of “consistency” and “symmetry,”
  3. The more it is replicated, the more the pattern accrues false authority — “the codebase already does this in multiple places.”

The flaw is not removed. The case count grows and legitimacy accumulates. That is the amplification. One patch becomes two, and by the third replication nobody questions it — “that’s just how the codebase does things.”

This is not just an anecdote. In 2025 a research team measured the phenomenon precisely and titled their paper “LLMs are Bug Replicators.” (Pan et al., 2025, arXiv:2503.11082) When surrounding code contained a flaw, a significant share of bugs the models produced were literally identical to the existing bugs — GPT-4o reproduced existing bugs at a rate of 82.6%, and across models the average was 44.4% exact matches with the pre-fix version. More unsettling: in flawed contexts, the probability of producing correct code versus incorrect code was nearly 1:1. The model is not making random mistakes. It is selectively reproducing the flaw patterns embedded in its context.

The same trap exists in law. A wrongly decided precedent gains authority through citation. Citation count is not evidence of correctness, yet we keep confusing the two. And software engineering knew this disease long before AI — copy-paste code clones quietly carry the bugs of their originals. One empirical study found that about 18% of clones that had undergone bug fixes contained propagated bugs, and the closer the clone was within the same file, the higher the propagation probability. (Mondal et al., ICSME 2017) In code as in law: the frequency of a precedent is not its validity.

Why This Happened

Not because the AI was stupid. Quite the opposite — it was being careful, trying to maintain consistency, and in doing so it aligned to the wrong reference point. The mechanism breaks down into four parts.

  1. It anchored authority to the code. “The code is written this way, so this must be correct.” But code can be a one-shot projection of a decision — or simply debt. The AI did not make that distinction. Cognitive science calls this anchoring bias. Research testing this bias in LLMs found that models are strongly pulled toward a value they see first, that this pull intensifies when the value is labeled as an “expert’s”, and that prompts to “ignore this hint” or to reason step-by-step do not reliably correct it. (Nguyen et al., 2024, arXiv:2412.06593) An existing implementation is the most authoritative anchor the codebase can offer.
  2. It mistook consistency for correctness. “Match the symmetry of the existing code” was an internal argument — it never checked whether that reference point aligned with external reality (the actual output). Self-consistency is a distinct property from accuracy — a model can build groundless confidence on plausible but incorrect self-explanations. (Chen et al., 2023, arXiv:2305.14279)
  3. It treated a comment as evidence. The comment saying “this type is intentionally cast to a string” was accepted as “proof of a sound design.” A comment is the author’s intent, not evidence of correctness.
  4. It dressed debt in confident language. Words like “root fix” and “by the book” lent credibility to an unverified proposal and raised the cost for me to question it. Models trained on human preferences are biased toward agreement and agreeableness over accuracy — a tendency called sycophancy that leads models to package their proposals smoothly and conclusively. (Sharma et al., ICLR 2024, arXiv:2310.13548)

What Cut the Loop

Here is the heart of this post. What broke the error was not a bigger model, not more thinking time. It was a single line of pushback from a human.

And that pushback was not an intervention that knew the answer. I did not know the answer. It was a directional instruction: “doubt the premise.” That one line was enough for the AI to switch modes — from referencing the code to verifying the behavior.

Remarkably, one study found exactly this asymmetry. DeepMind researchers showed that LLMs’ poor self-correction ability comes not from inability to fix errors but from inability to find them — if the location of an error was indicated externally, the model corrected it well. (Tyen et al., Google DeepMind, 2023, arXiv:2311.08516) That is precisely what I did. I did not know how to fix UUID, but I pointed to a location and said: doubt this precedent here. That was sufficient.

This says something about the structure of humans and AI working together. A human’s value is not in knowing more faster — AI has already won that race. A human’s value is in being able to occupy the position that questions the premises the AI stands on. “Is this really right?” is a question that belongs not to the one who has the answer but to the one who knows how to doubt the answer.

That position, however, is not given for free. A Stanford user study reported the uncomfortable finding that participants assisted by AI actually wrote less secure code while believing theirs was more secure. But in the same study, participants who trusted AI less and pushed back more produced safer code. (Perry et al., Stanford, 2022, arXiv:2211.03622) The position of doubt is not a default. The deeper the trust, the more that position empties out.

So How Do We Prevent This

Lessons must become design, not comfort.

  • Precedent ≠ truth. Before extending an existing pattern, verify not whether it is internally consistent but whether it produces correct output.
  • Anchor to ground truth. Base judgment not on “how the existing code looks” but on actual output · runtime behavior · tests. In this case, every decisive question was not “how does the code look” but “what does it actually produce.”
  • Try to distinguish decisions from patches — and admit when you can’t. Sometimes code and comments alone are not enough to tell the two apart. When that happens, make the uncertainty explicit: “this follows a precedent whose validity is unverified.”
  • Restrain confident language. Do not attach words like “root,” “correct,” or “by the book” to proposals that have not been verified.
  • Build checkpoints into automation. Decisions where an agent extends an existing implementation need a gate that forces the question: “Has the validity of this precedent been verified?”

The Same Story After All

I have been saying one thing for a long time: raw code is not a medium that preserves decisions. Code cannot hold “why we did it this way.” That is why git blame shows who and when but not what was decided.

This incident is the sharpest empirical proof of that proposition. The claim is not that humans lose their decisions. The claim is that even a carefully designed agent will mistake a patch for a design and propagate it into new code. When decisions are not recorded explicitly, intelligence is not the solution. The more intelligent the system, the more consistently and plausibly it spreads the debt.

So I write specifications. I inscribe decisions in a single authoritative declaration layer outside the code. Rather than hoping a human will throw that one line of doubt each time — I want the system to throw it itself.

Law is not a sword but a signpost. A good signpost tells the lost traveler in advance: “Doubt here.” This post is a record of how one such signpost was erected — starting from a single line of pushback.

Further reading (external)

  • Generative AI and the End of Chesterton’s Fence — Reece. The principle “don’t remove a fence until you know why it was built” collapses in front of code generated probabilistically without intent. Locks exactly onto this post’s thesis that code does not preserve the decision behind it.
  • Programming as Theory Building — Christian Ekrem. Drawing on Peter Naur’s classic, it argues that “a program is not source code but a theory in people’s minds.” The theoretical root of why AI cannot distinguish design from debt by looking at code alone.
  • Vibe coding is not the same as AI-Assisted engineering — Addy Osmani. Traces real outages to show why vibe coding that blindly trusts AI output fails in production, and prescribes a spec-based, human-verification workflow.
  • Cognitive Debt — Simon Willison (citing Storey). The faster AI churns out code, the more the real debt becomes not “flaws in the code” but “people losing the ability to understand the code” — the concept of cognitive debt.
  • Overreliance on AI: Addressing Automation Bias — Lumenova AI. The mechanisms by which automation bias, anchoring, and confirmation bias dull human judgment, and “cognitive forcing functions” as the remedy — a psychological grounding for the human value of knowing where to doubt.

Sources

  • Pan et al. “LLMs are Bug Replicators: An Empirical Study on LLMs’ Capability in Completing Bug-prone Code” (2025, arXiv:2503.11082)
  • Mondal, Roy, Schneider. “Bug Propagation through Code Cloning: An Empirical Study” (ICSME 2017, link)
  • Nguyen et al. “Anchoring Bias in Large Language Models: An Experimental Study” (2024, arXiv:2412.06593)
  • Chen et al. “Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs” (2023, arXiv:2305.14279)
  • Sharma et al. “Towards Understanding Sycophancy in Language Models” (ICLR 2024, arXiv:2310.13548)
  • Tyen et al. (Google DeepMind). “LLMs cannot find reasoning errors, but can correct them given the error location” (2023, arXiv:2311.08516)
  • Perry, Srivastava, Kumar, Boneh. “Do Users Write More Insecure Code with AI Assistants?” (Stanford, 2022, arXiv:2211.03622)
  • Hero image: AI-generated (Google Gemini)