The study was killed by a device I buried myself

Studies usually die one of two ways: quietly, in a file drawer, or loudly, published with claims they can’t support. Mine died a third way. A rule I wrote down before seeing any results shut the study down, exactly as written. And I decided to publish the corpse. This is a post about why.

What I was trying to measure

In 1777, Lessing declared that he could not cross from “accidental truths of history” to “necessary truths of reason,” and called the gap between them an ugly, broad ditch. The metaphor has been quoted and embroidered for 250 years. It has never been measured. I wanted to measure it as a property of a literature: take the inferential chain running from “physical reality has an external cause” all the way to the God of Nicene Christianity, divide it into eight steps, and code twenty-one issue-years (2004–2024) of the peer-reviewed philosophy-of-religion literature — 3,536 articles across eleven journals — for which step each article addresses, in which direction, at what claim strength, on what kind of evidence.

The coding was to be done not by humans but by LLMs from three different companies, independently. A human coder who knows the hypotheses is a bias vector; a machine can be firewalled — the coder-facing prompt never contained a single word of the hypotheses. And because “three models agree” is weaker evidence than it sounds (shared training folklore, memorization, style leakage), the design came wrapped in a validation battery aimed at exactly those artifact classes.

The trap I dug, and documented myself digging

The pilot reliability gate passed: minimum pairwise Krippendorff’s α of .705–.808 across four dimensions. But anyone who has run a coding study knows what’s behind a number like that. It was the fifth round. The previous four had failed or underperformed, and each failure triggered a revision — codebook clarifications, a model swap, a collapsed scale. On round five the number cleared the bar, and we stopped.

Stopping when you clear the bar has a name — optional stopping — and its product has a name too: a selected estimate. The sealed plan said this about itself, in those words. So the pilot pass was declared conditional: at the start of the main run, the instrument would face 150 fresh items it had never touched, judged by a dual statistic fixed in advance (point minimum α ≥ .70 and a bootstrap 95% lower bound ≥ .667). The consequence of failure was also written in advance: halt everything, make no confirmatory claim from anything already coded.

The checkpoint

The result: point minima of .676–.736, with two dimensions under the .70 floor. Bootstrap lower bounds of .390–.650 — all four dimensions under .667. Failure everywhere.

The escape routes had been bricked up ahead of time. Decoding noise? The run-level determinism audits were clean. Two families ganging up on the third? The collusion watch never came close to triggering. A computational artifact? Complete coverage, frozen parameters enforced by the statistic implementation itself. What remains is the explanation nobody tuning a pilot wants: three model families substantively disagreed about what fresh professional prose says. A number built by five rounds of tuning collapsed within days of meeting data the tuning had never touched.

I remember feeling the sentence assemble itself: “.676 is basically .70.” The only reason that sentence lost is that the halt rule had been written down before the sample was drawn.

The second confession

If the story ended there, it would be a tidy methods fable. It didn’t end there.

While preparing the upload, I opened the OSF account to insert the preregistration identifier into the paper. There was no registration. The project’s records said the registration had been completed on the day of the freeze; the web registration had, in fact, never been completed. The freeze itself was real — a SHA256 manifest, machine-verified, sealed into git. But the claim of an externally timestamped preregistration was false.

We corrected it instead of erasing it. The word “Preregistered” came out of the title, replaced by “Prospectively Frozen.” Dozens of instances of “registered” throughout the paper were demoted to “precommitted.” A correction-of-record section now states what existed (a locally sealed protocol), what did not (an external registration), and exactly how much less a reader should therefore believe (git timestamps alone are not tamper-proof). The checkpoint had tested the instrument. This episode tested the procedure’s honesty itself — and the test is always the same question: when you find a fact that hurts you, do you fix the record, or disclose the fact?

Four and a half lessons

One. A pilot reliability score measured after revision loops is an advertisement, not an estimate. If the loop stops when the score clears the bar, the reported number is the maximum of several tries. You can’t fix that by being honest about the loop. You fix it by paying for fresh data the loop never touched.

Two. The checkpoint is absurdly cheap relative to what it protects. Ours cost a fraction of the full run it halted. A study that runs to completion on an unreliable instrument doesn’t fail loudly — it publishes confidently wrong numbers, with three models’ worth of false consensus behind them.

Three. Freeze the consequence, not just the threshold. Our halt happened not because we are disciplined people, but because the consequence was written before the temptation arrived.

Four. Publish the corpse. Negative instrument results are rare in the LLM-annotation literature. I doubt that’s because instruments rarely fail.

And a half. Trust comes from a correction machine, not from flawlessness. If anything in this project deserves belief, it’s not because we made no mistakes — it’s because both of our big ones sit in the record as dated, public corrections.

What we did not learn

Let me be precise. The hypotheses were never tested. Whether the philosophy-of-religion literature shows a type–direction asymmetry, whether its topography tilts by venue — we do not know. “The bias turned out not to exist” is not this study’s conclusion; a scale that breaks before the weighing tells you nothing about the weight. Lessing’s ditch remains unmeasured. What got measured instead is narrower, and I think worth the trade: how far a carefully tuned pilot number retreats when it meets data it has never seen — and what it takes, procedurally, to force yourself to believe the second number over the first.

Preprint (full design, validation battery, checkpoint record): github.com/dankang21/ditch-audit — docs/draft/ has the manuscript and the compiled PDF (preprint-v0.12.zip), alongside the full audit trail (85+ commits, phase-tagged). Sealed-freeze archive: osf.io/rt3zg. The orchestration and prose drafting of this project were assisted by an LLM (Claude); the full disclosure is in the preprint’s back matter.