The number in my paper is not the number my pipeline produces

What actually happened

One of four things, and each is an ordinary afternoon’s work:

• A threshold moved. The draft said above 1% allele frequency because that is what min_freq was in March. It is 5% now. Nothing failed; the sentence simply stopped being true.
• A sample was dropped. The results table says 24. One failed QC and was removed from the manifest, so the pipeline runs 23 and the table does not know.
• A tool was upgraded. The methods section names bwa 0.7.17. The cluster has 0.7.18 and nobody announced it.
• A step was renamed or removed. The reported count came from an analysis that has since been rewritten, so the number in the paper now corresponds to no step at all.

Why nothing catches it

Because the number and the pipeline live in different worlds, and no tool spans both. A workflow engine knows what ran; it has never read your manuscript. A reference manager knows your citations; it has never seen your parameters. Version control sees both files change and has no idea that one change invalidates a sentence in the other.

Even careful provenance stops one step short. Ligase was already unusually good at naming what it could not vouch for — it names a tool with no citation rather than omitting it (D24), names a sample identifier that is really a file name (D26), names a step it cannot account for. All of that runs in one direction: from a result back to how it was produced. The question a reader of the paper asks runs the other way, and there was nothing to ask it of.

Why the answer is plausible rather than absurd

Because nothing is wrong except the sentence. Rerun the pipeline and it succeeds. Check the output and it is correct. Audit the step and it reproduces. Every verification you can run passes, because the computation is not where the problem is — and the number in the paper is in the plausible range, because it was produced by an earlier version of the same analysis.

That is what makes this the most expensive kind of mistake in the catalogue. It survives peer review, because a reviewer cannot see your pipeline. It survives a reproducibility check, because the pipeline reproduces. It is found, if it is found at all, by somebody trying to build on the paper.

What Ligase does about it

You tell it the number, and the step behind it:

pipe = lg.Pipeline(
    call(hxb2, min_freq=0.05),
    claims=[
        lg.Claim(
            "variants were called above 1% allele frequency",
            step="call",
            parameter="min_freq",
            value=0.01,
        ),
    ],
)

Both numbers are stated and ligase methods exits non-zero, so the build that moved the threshold is the build that tells you the manuscript needs editing. The write-up does not guess which of the two should change, because it cannot know whether the paper or the pipeline is the thing that is behind.

For the renamed step, the same mechanism says the other thing:

Or point it at the number itself

The claim above compares a parameter the run recorded. Point a claim at the number in the output and the engine reads it:

lg.Claim(
    "malaria F1 = 0.333 against seven curated targets",
    step="validate",
    value=0.333,
    at=lg.cell(column="f1", row=0),
    tolerance=5e-4,
)

Rerun the analysis and leave the manuscript alone, and the verdict turns over:

This is the form that catches the failure at the top of this page directly: the table in the paper and the number in the artifact have diverged, and the build says so rather than passing (D61).

What it does not claim to do

It does not go looking for a number it was not pointed at, and it parses one format: a TSV with a header, a column by name, a row by position. A claim that cannot say where its number is binds to the step key, the output digest and the date that computation happened — enough for a reviewer with your bundle to find the computation and rerun it, and not a verification of the number. That is a legitimate state rather than a shortcoming, and the guide says which claims are in it.

It is also not a statistics engine. It does not recompute your F1 or check that it is a well-formed F1. The promise is narrower and worth more: the number you published is the number in the bytes.

See also

• Binding a number you published — the task.
• A number in a paper — why the engine owes this.
• D45 — what a claim binds to, and why editing a manuscript reruns nothing.
• D59 — which of a claim’s two identifiers is safe to print in a manuscript.
• D61 — reading the number out of the artifact, and the four things that is not.