← Back to portfolio

When two papers disagree — and when they only look like it

Ask a language model whether two scientific findings contradict each other and it will compare two sentences. Sentences differ for reasons that are not disagreement: the populations differ, the dose differs, the instrument differs, the definition differs. Every one of those comes back as a contradiction, and a tool that cries contradiction on a page of ordinary science is worse than no tool, because someone has to check all of it by hand anyway.

This is the layer that sits between “retrieved some text” and “these two papers disagree”. It is deterministic, it calls no model, and its central design decision is that a contradiction is the narrowest verdict it can return, not the default one.

A claim is not a sentence

A claim here is a sentence plus the conditions under which it was asserted. Six axes decide whether two claims may be compared at all: population, intervention, outcome measure, conditions, methodology, measurement. If any of those differ between two findings, the findings are not in conflict — they are answers to different questions, and the layer says so by name.

  two claims
      |
      v
  same subject, predicate AND object?  --no--> NOT_COMPARABLE
      |yes
      v
  every comparability axis recorded
  on BOTH sides and equal?             --no--> DIFFERENT_CONDITION  (names the axis)
      |yes
      v
  any axis NOT RECORDED?               --yes-> INSUFFICIENT_INFORMATION
      |no
      v
  polarity opposed?                    --yes-> CONTRADICTS
      |no
      v
  SUPPORTS

“Not recorded” is a state, not a blank

An unset axis means the paper did not say. It never means “does not apply” and it never quietly matches. Two findings with opposite signs whose methods sections never stated the measurement return INSUFFICIENT_INFORMATION — because a difference in setup would explain them just as well as a real conflict, and nothing in the text rules that out. This is the single rule that separates a comparison you can act on from a pile of false alarms.

The eight verdicts

Two rules that exist because working scientists trip them

• A registered prediction against its own result is QUALIFIES, never a contradiction. Otherwise every honest pre-registration contradicts the paper that reports it, and the practice that most deserves credit scores worst.
• A source withdrawing its own earlier claim is SUPERSEDES, reported with the withdrawn side first. Self-correction is the strongest signal in a literature. Reporting it as a conflict punishes it.

A citation that cannot be found is refused, not stored

Every claim must carry the exact passage it rests on. The system then locates that passage in the source file itself and records the character offsets. A quote that is not present in its source raises rather than returns — that is not a weak claim, it is a fabricated attribution, and storing one poisons every later comparison built on it. A quote that occurs more than once is refused too, because the offset would be a guess.

Whitespace is normalised on both sides first, since a PDF extraction breaks lines where the original did not. Anything beyond that is treated as a different sentence, deliberately.

Run against a real case

The test case is a genuinely hard one from my own speaker-verification study: a pre-registered prediction about pitch direction, a first run contaminated by pitch-tracker octave errors, two defensible corrections that give opposite answers, and a withdrawal of the claim. Five quotes, anchored by the locator in PREREGISTRATION.md and PAPER_DRAFT.md on disk. Ten pairs.

The load-bearing row is the first one. The paper spends a full page arguing exactly that: the two corrections disagree, and the disagreement is about which F0-cleaning rule was applied, not about pitch direction — which is why the claim was withdrawn rather than published at p = 0.013. The layer reaches the same judgement from structure alone, with no model and without reading the prose that argues for it.

The defect it found in itself

An earlier run over five papers returned two contradictions. Both were wrong, and one of them was the code. Comparability was being decided from subject and object while ignoring the predicate — so “discrimination improves with window length” and “discrimination is insufficient for window length”, two true sentences one paragraph apart in the same document, compared as one assertion with opposite polarity and came back CONTRADICTS.

The predicate is now part of comparability, and predicates that are merely worded differently are refused rather than assumed equal. It is carried as a named regression test, not a changelog line. Under-reporting a comparison is the cheap error. Asserting a contradiction that is not one is the expensive one, and the whole layer is tuned in that direction on purpose.

What it deliberately will not do

• It will not infer a condition the paper did not state. The axes are read from the methods section by a human, because a guessed population is exactly the input that produces a confident false verdict.
• It will not let a model decide what may be compared. A model may later be asked to judge a pair — but only a pair this layer has already ruled comparable. Narrowing what the model is allowed to consider is the safety property.
• It will not return a verdict without a reason. Every judgement carries both quotes and the axis that produced it, so a researcher can disagree with the specific step rather than with the machine.

How it runs

It is exposed as an MCP server, so an agent doing the reasoning has to go through it. A rule an agent can skip is not a rule. Three tools: claim_relations() for the taxonomy, record_claim() which anchors a quote in a real file or refuses, and compare_claims() which returns the verdict, both quotes and the reason. On the other side of the boundary the client validates the response against the contract the server actually advertised, rather than trusting the reply.