Decision Records

D1 — Rust core, Python surface

Scientists live in Python; correctness-critical logic (types, hashing, graph, cache) needs speed and safety. PyO3 + maturin give a single pip install.

D2 — Content hashing, not timestamps

Timestamps lie (copies, clones, touched files). Cache keys hash code, parameters, input content and tool versions.

D3 — BLAKE3 with size+mtime prefilter

BLAKE3 is fast enough for multi-GB BAMs; the prefilter avoids rehashing unchanged files. The rayon feature that buys that speed means hashing a step’s inputs queues work on whatever pool the step is running on, which is why the executor’s pool has to stay usable from inside a step (D27).

D4 — Semantic hashing

Outputs are canonicalised before hashing (sorted records, volatile headers stripped) so scientifically identical results share a hash.

D5 — Dimensions as types (units analogy)

Errors are caught before compute. Conversions are typed functions in a conversion graph, enabling v2 auto-fixing via path search.

D6 — SQLite metadata, content-addressed store

Zero-config, embedded, portable; no server required.

D7 — HIV-1 reference pipeline

One real, validated pipeline drives every feature and ties Ligase to SAWA.

D8 — Out of scope for v0.1

Auto-fixing, public-data fetching, cluster execution, GUI.

D9 — Coordinates are semantics, not encoding

The coordinates dimension records what a step sees, not the on-disk convention. SAM and BAM both present 1-based closed alignment records, so re-encoding between them changes nothing. BED (0-based half-open) and GFF (1-based closed) differ semantically, and that is the difference worth catching.

The rejected alternative — BAM 0-based because its binary encoding is — would make samtools view -b appear to change a dimension. False alarms are not a safe failure mode: a checker that cries wolf gets bypassed, and then the real hg19/hg38 mixup goes through unchallenged.

D10 — References relate by containment

Two references compared with each other must be identical. Data checked against a reference need only be contained in it: every contig the data mentions must appear in the reference with a matching length and MD5, while extra contigs in the reference are fine.

Strict equality everywhere would reject a VCF whose header lists only the contigs it has variants on — correct input, rejected. Contig order still gets its own error, because htslib matches by name but GATK requires matching sequence dictionary order.

D11 — noodles for format I/O, and the MSRV follows it

Format parsing uses noodles rather than rust-htslib. It is pure Rust, so wheels build without a C htslib on the machine — which is the difference between pip install ligase-bio working and not. The cost is that noodles is a pre-1.0 crate that bumps minor versions often, and its MSRV sets ours (currently Rust 1.91).

MD5 arrives via md-5. That is not a security choice: MD5 is what the SAM specification prescribes for the @SQ M5 contig checksum, so we must compute exactly that digest to compare against it. Cache keys use BLAKE3 (D3); the two are unrelated.

D12 — Steps defer; the graph is built from signatures

@lg.step reads a function’s signature once and replaces the function with a Step. Calling it records the call and returns a handle; the body never runs while a graph is being built. The return annotation is therefore the only statement of what a step produces, and an unannotated parameter is refused at decoration time — it would be an edge the graph does not contain, which means an input nothing is ever compared with.

The handle is cast to the data type the step declares, which is a deliberate lie to the type checker and buys static checking of the pipeline’s wiring: a mis-wired call(align(...), calls) is an error in the editor, not only at check(). lg.handle() is the typed way back to the real object.

The rejected alternative was a handle that genuinely subclasses the data types. It would have needed graph identity stored on each dataset, and a dataset that knows which graph it belongs to cannot be used in two.

D13 — The graph decides requirements and blames the origin

The dimension catalogue allows an unknown dimension unless a downstream step requires it. Only a graph can answer that, so required_pair — which reads requirements off the pair of kinds a step joins — becomes the default, and the graph adds what the table cannot know: where the gap started.

An undeclared-dimension error therefore names the input that has to declare it, the step that requires it, and the route between them, and is reported once however many steps the gap travels through. Pointing only at the step would send a user to a file they cannot change; one error per step the value passes through would make the report unreadable, and a report nobody finishes is a report that does not work.

A step handed the wrong kind of data is reported instead of that step’s dimension errors rather than alongside them. Those bindings are not what the step declared, so comparing their dimensions would describe a pipeline the user never wrote.

D14 — Contig order is compared only between sequence dictionaries

OrderDiffers exists because GATK requires two sequence dictionaries to agree on order (D10). A BED’s contigs come from its records, which have an order too — whatever order the intervals were sorted in. Comparing it would report a contig order mismatch for an ordinary BED that happens to list contig 2 before contig 1.

So ReferenceId records whether its contig list is a Dictionary or merely Observed, and order is compared only between two dictionaries. The rejected alternative, sorting record-derived contigs before storing them, would have made the stored list disagree with the file.

The same reasoning keeps reference out of what a step requires of an interval file: its contigs cannot be declared, so demanding them would be an unsatisfiable check.

D15 — An error identifies its node, not just its step

Per-sample fan-out makes one written step into one step per sample, so a report that knows only step names understates every cohort-wide problem. Two consequences follow, and both are about what a user does next rather than about catching more.

The summary counts the places that failed rather than the names they share. “4 errors in 1 step” describes a localised problem you fix by looking at one sample; “4 errors in 4 steps” describes a cohort-wide one you fix by rethinking the pipeline. Reporting the second as the first is not a cosmetic understatement.

An error names its node in the heading rather than on a line of its own, since the name says which step this is and so belongs with the step. It is added only where the step’s own name is ambiguous — more than one node in the pipeline being checked shares it — and then only with something that tells those nodes apart.

Both halves of that condition matter. The golden example fixes the shape as in step call: and the project treats it as fixed, so a suffix on a step nobody could confuse is noise on the commonest path. And the sample is not always the discriminator: two same-named steps fed different alignments of one sample would both read (sample sample_a), which looks exactly like an identifier while being none. Discriminators are therefore tried in order — the sample, then the first file the step was built from — and the first that gives every node in the group a distinct value wins. If none does, nothing is added, because no suffix beats a misleading one. A step spanning a whole cohort likewise gets no sample: labelling it after one of them would send a user to the wrong data.

Identical problems are then stated once, with every affected sample named rather than counted, because a user who must expand something to find out which samples failed has a shorter report rather than a better one. Two rules keep that honest: an error with no name is never collapsed, since there would be nothing to list it under; and grouping is on the problem’s full text, so a sample that failed differently keeps its own paragraph.

Collapsing applies to the rendered report, not to the error list. pipe.errors() stays one error per failing node, because counting affected samples — or retrying one of them — wants that granularity.

D16 — Execution lives in the core; recipes are extracted first

A step’s body is Python; a run is dozens of external processes over tens of gigabytes. If the scheduler had to call into Python to learn what each step does, the GIL would sit in the hot path of every parallel run.

So the body is called once, before the run starts, and returns a recipe — a command template, or a function to call later. The core then owns the whole run (ready queue, worker pool, rendering, hashing, store, metadata) with the GIL released, and takes it again only for the lg.write steps that genuinely are Python.

This is only available because bodies were already deferred (D12). A body that computed something could not be called before its inputs existed; a body that describes work can.

The rejected alternative was a Python thread pool driving the core step by step. It is simpler, and external processes release the GIL anyway — but every task would have to re-enter the interpreter to fetch its recipe, and the scheduler’s correctness would live in the layer with no tests against it. Putting the ready queue in Rust also means cargo test can assert that a fan-out branch is independent, which is the property the whole milestone rests on.

D17 — A placeholder that names nothing is an error, and {{ escapes

lg.sh fills {ref}, {reads.r1} and {out} from the step’s own parameters. A mistyped {refrence} could either be passed through to the shell as literal text or refused. It is refused.

Passing it through would mean a command that fails obscurely much later — or, with an echo, succeeds. The cost of refusing is that a brace meant literally has to be doubled, so an awk '{{print $1}}' is written the way Snakemake writes it, which is already familiar to the people this is for. Anything between braces that is not an identifier is reported as a brace needing an escape rather than as a mistyped placeholder, because that is the thing the user has to change.

Every substituted value is also quoted into exactly one shell word. A reference under /my refs/ is one path, not two arguments, and a value can never end the quoting and continue as code.

D18 — A failure stops its own branch and nothing else

A step that fails is recorded against its node. Independent branches keep running, and steps below the failure are not run and are not reported.

Losing three good samples to one bad FASTQ is the overnight-run problem this project exists to remove, so a cohort-wide abort was never an option. The other half — reporting the steps below a failure — was rejected for the same reason the checker reports one error per missing declaration rather than one per step it reaches (D13): one cause, one failure. A cascade of “its input is missing” lines makes a report longer without making it more useful.

Nothing is written to the cache for a failed step, so a half-written output can never be served as a result later.

D19 — A cache from another version is discarded, not migrated

Two version numbers sit in the cache’s metadata. A change to the directory layout drops the tables; a change to how keys are composed deletes the step entries and keeps the file hashes.

That asymmetry is the point. A file’s hash is a fact about the file, whatever keys are made of. A step entry is a claim that these ingredients, combined this way produced that output — so an entry from another key format is not stale data to be migrated, it is a claim about something else, and reading it is the one failure a cache must never have.

Nothing is migrated because nothing needs to be: every byte in the cache is derived, recomputing is merely slow, and being wrong is not.

D20 — A tolerance is a quantisation

CLAUDE.md allows a numeric type to declare a tolerance for equality. The only way a hash can express one is to quantise: put the value in a bucket of that width and hash the bucket.

“Within ε” is not transitive. a near b and b near c does not make a near c, so it is not an equivalence relation — and a hash can only ever express an equivalence relation. The rejected alternative, true ε-equality, cannot be hashed at all: it would need every value compared with every other, which is the opposite of what a content-addressed cache is for.

The honest consequence, which belongs in the open rather than in a footnote: two values closer together than the tolerance can still fall either side of a bucket edge and hash differently. A tolerance therefore means “the same bucket”, not “within ε”. It removes the common case — a last digit that moved — and does not promise more than that.

Exactness stays the default. QUAL in a VCF declares a tolerance of a tenth of a Phred unit, because it is the field with the motive: a floating-point accumulation whose last digits move with the order the terms were added in. A tenth of a Phred unit is two orders of magnitude below the smallest threshold anyone filters on, so it cannot merge two qualities a user would treat differently. Nothing else declares one, because a tolerance nobody has measured would be a guess baked into every cache key, and the project’s bias runs the other way: a cache that is too cold costs time, one that is too warm is wrong.

D21 — Canonicalisation falls back to the bytes

A file that does not parse as its type hashes as its bytes, tagged as such, with the tag inside the digest so the two can never collide.

Canonicalisation means parsing, and parsing fails on real inputs: a truncated BAM, a .vcf.gz that is not gzipped, a file still being written, a .bam holding something else entirely. Refusing to hash such a file would mean a format problem surfacing as an engine failure, in a layer that has no business judging whether a file is valid — that is M6’s job, and it has a whole milestone because it is not a line of code.

The direction is what makes this safe. Bytes are the stricter test, so the fallback can cost a rerun and can never serve a wrong answer. Falling back the other way — guessing at a canonical form from a partial parse — would mean two files that differ in the part that could not be read hashing the same.

For the same reason every parser here is strict. A canonicaliser that read what it could and ignored the rest would be the dangerous kind of lenient: it would reduce a file to the fields it managed to understand, and two files differing only outside that set would agree. So a FASTA with no > line, a BED row with two columns, a VCF record with seven, and a report row that does not fit its header are all failures that fall back, rather than partial successes.

D22 — A tool’s signature is not part of its result

Every tool in the reference pipeline records how it was run, inside the file it produces. minimap2 writes @PG CL:minimap2 -t 8 -ax sr ...; bcftools writes ##bcftools_callCommand=call -mv ...; Date=Sat Oct 17 21:14:12 2026. These are dropped before hashing.

The reason is stronger than tidiness. The command line a tool records contains its output path; a step’s output path is the staging file, named after the step’s own key. So before M4 a step’s output bytes contained that step’s own key, and any change to the step at all — a parameter, a tool version, a comment — moved its bytes, which moved the key of the step below, whose bytes moved in turn. Early cutoff, which M3 documented as falling out of content addressing, could not get past the first real tool.

The rejected alternative was to make the staging path stable so that the @PG line would repeat. It fails for two reasons: the timestamps remain, and two nodes sharing a staging path could not run at once — which would cost the per-sample fan-out the whole project is built around.

Dropping these fields has a cost worth naming: a cache hit can no longer prove which command produced an object, because the field that said so is not in the hash. That is the right place for the loss. A hash answers “is this the same result?”; “how was this produced?” is provenance, it is M5, and it wants its own storage and its own lifetime (cache).

D23 — Provenance has its own store and its own lifetime

The cache answers “has this been computed?” and is designed to be discarded: everything in it can be recomputed, so a cache written under another key format is dropped rather than migrated (D19). Provenance answers “how was this produced?”, and that answer cannot be recomputed. The machine, the installed tool versions and the exact command that produced an object last year do not come back.

So it lives in provenance.sqlite beside cache.sqlite, with its own PROV_VERSION and nothing that ever drops a row. Cache::open clears tables, not the directory, so a separate file survives every key-format and canonicalisation bump for free.

The rejected alternative was more columns on the cache’s steps table, which already held the tool versions and the duration because it had them. It fails on lifetime: the question that table exists to answer is retired whenever the ingredients are recombined, and a methods section is not. Discarding a year of provenance because KEY_VERSION moved from 2 to 3 would be an unforced loss of the only data here that cannot be regenerated.

A record is keyed on the step key, which is already the identity of the computation. Two things are kept out of it deliberately. The sample label belongs to the pipeline a step was reached from rather than to the step, and Graph::scopes can derive it whenever it is wanted. Citations are not stored at all: a citation cannot change a result, so editing one does not rerun the step, so a stored one would go stale while looking current.

D24 — A citation is curated or declared, never guessed

ligase methods cites minimap2, samtools and bcftools from a table in the engine, anything else from cite= on the step, and reports a tool covered by neither as uncited rather than omitting it.

Two failure modes to avoid, and they pull in opposite directions. A reference synthesised from a tool’s name would be a fabrication in a manuscript — the one kind of error this project exists to prevent, and worse than no reference at all. But a tool silently missing from a reference list lets an incomplete methods section look complete, which is the same failure wearing a quieter coat.

Naming the gap satisfies both. The table stays short for the same reason: every entry is a standing promise to keep a reference correct as versions move, and a long list of half-checked citations is worth less than a short list of checked ones beside an honest admission.

The rejected alternative was no table at all, with every citation declared. It is defensible — there is nothing to maintain and nothing to get wrong — but it leaves the reference pipeline’s own methods paragraph with no references until three are typed by hand, and those three are the ones the project is built on.

D25 — A bundle states a claim and can be checked

“A bundle that reproduces the results on a clean machine” cannot be built as written. The reference pipeline’s inputs are sequencing runs from the SRA; a bundle carrying them would be gigabytes, and fetching them is out of scope for v0.1 (D8).

So the bundle carries everything except the data and names the data firmly enough that nothing else can be substituted: every input by content hash, every reference by accession and checksum. A clean machine is then, to this engine, an empty cache — nothing computed, nothing remembered, every tool probed again — and the claim is that a run there produces outputs whose canonical hashes are the recorded ones. That is testable, and tests/test_demo.py tests it on the real pipeline by writing a bundle from one cache and verifying it against another.

verify matches rows by step and sample rather than by key, because the key is the thing that may legitimately have moved, and that is what makes the verdicts worth having:

• identical — the same key and the same result.
• reproduced — a different key and the same result anyway.
• differs — a different result.
• missing — this pipeline no longer produces that step.

reproduced is the one that justifies the mechanism. Keys are content-addressed, so a check on a machine with the same data and the same tools is identical and tells you little; a check where the aligner is a different build is reproduced, which is a stronger statement than a cache can make and is what a reviewer on their own machine actually gets.

Inputs are checked by content too. That is not a bonus feature: without it a differs is uninterpretable, because a changed input explains a changed output, and a bundle that could not distinguish “irreproducible” from “different experiment” would be misleading in the one situation it exists for.

The format is eight flat TSV tables rather than JSON. No dependency to add, greppable, openable in a spreadsheet, and consistent with the engine’s one structured store being SQLite (D6). Tabs, newlines and backslashes in a field are escaped, so a recorded command containing a tab cannot shift a column.

D26 — A field that cannot be dropped is noticed instead

bcftools names a VCF’s sample column after its input BAM’s path when the BAM carries no @RG SM, and a Ligase run hands it a file inside the content-addressed store. The calls then carry something like /Users/kelyn/ligase/.ligase/store/62/7db9….bam where a sample identifier belongs, and the consequence is not cosmetic: the canonical form of a VCF includes its sample names, so such a file’s canonical hash depends on where the cache directory lives. A reproducibility bundle written from one cache cannot verify against a clean one. M5’s bundle verification is what found it — call came back differs on every sample with byte-identical variant records, the only difference being the #CHROM line’s last column.

This is D22’s pattern in a field that cannot be dropped. A command line says how a file was made; a sample identifier is part of what the file is about, and a sample set is one of the dimensions this engine exists to keep straight. So the hash stands and the engine says what it saw: a Note, carried beside the hash into the run report, provenance, the bundle and the methods section. A note is not an error, never changes a hash, and the fix is one line in the pipeline the user is the only one who can make.

Both louder alternatives were rejected, and for different reasons.

Refusing it at canonicalisation time is what D21 forbids: a VCF whose sample column holds a path is perfectly valid VCF, and turning a format problem into an engine failure is the thing that layer must not do. Refusing it at ligase check time instead cannot work, because the file does not exist yet: the checker runs before any compute, on inputs, and the file with the wrong identifier is an output. A check-time rule would miss the failure that prompted this one and fire only on hand-supplied files that are not wrong.

Normalising it — reducing a path-shaped identifier to its basename before hashing — is the silent repair of a wrong file that this project refuses on principle, and it risks collapsing two genuinely different samples whose basenames agree. It also would not have worked. The basename of a file in the store is the hash of its bytes, and an alignment’s bytes hold the staging path that samtools sort -o recorded, so the basename moves with the cache root exactly as the whole path does: the bundle would still have reported differs.

The rule for what counts is deliberately narrow, because a rule that guessed would report good identifiers as mistakes. An identifier is a file when it holds a path separator or ends in an extension naming a format this engine reads. K03455.1 has a dot and is not a file; SRR8525904 and sample_a have neither.

Three consequences worth stating. The notes are stored with the canonical hash rather than recomputed, because a cache hit does not reopen the file and a warning that stopped appearing on the second run would be worse than none. CANON_VERSION does not move: the rule changes nothing about what is hashed. The cache’s SCHEMA_VERSION does, for the new column, which clears the cache’s tables — the same practical cost, for a reason that can be stated honestly.

D27 — A worker that waits is a worker rayon cannot use

A ligase methods run on the HIV-1 demo stopped for forty minutes with no child processes alive and under ninety seconds of CPU burned, while a second full run of the demo was competing for the same machine. Nine rayon workers: two inside BLAKE3’s parallel hashing of an input, asleep in a nested rayon::join, and seven asleep on the executor’s own Condvar waiting for a step to become ready.

That is the whole bug. The scheduler ran jobs long-lived worker loops, each inside a scope.spawn job, and a worker with nothing ready blocked on a condition variable of the runner’s own. Rayon has no idea such a thread is idle — it is inside a job, so it is busy — and a blocked worker cannot steal. Meanwhile hashing a step’s inputs happens inside a step, and BLAKE3 hashes a large file across whatever pool it is called from (D3). So a step queues work on the executor’s own pool, and that work had two possible takers instead of nine. Whether the final stall is a strict cycle or a lost wakeup between those two is not something the sample can settle, and it does not matter: a pool whose spare capacity is invisible to its own scheduler is the defect, and heavy oversubscription is only what made it show.

So the rule is that a worker never blocks on anything that needs the pool to make progress. Concretely: no worker loop and no waiting for work. A step is one rayon job, and a job that finishes starts the steps it released before it ends — so the run is over exactly when the scope is empty, and a worker with nothing of its own to do is idle as far as rayon is concerned. The two places that used to block are gone. The ready queue is no longer waited on, and a node that wants a key another node is producing is parked on that key and handed back when the claim is dropped (the scheduler).

The rule is narrower than “a worker never blocks”, which would be unachievable: a step is a child process, and a worker sits in wait for its whole duration, as it does for the GIL inside a callback. Neither of those needs the pool to finish, so neither can close a cycle. The ready queue and a claimed key did.

Two alternatives were weighed and rejected.

Hash single-threaded inside the executor — update_mmap rather than update_mmap_rayon. One line, and it removes the nested parallelism completely. It was tempting because the prefilter means a warm run rehashes nothing and the executor already hashes jobs-ways across concurrent steps, so the loss is confined to a cold hash of a very large file in a narrow graph. It was rejected because it treats the symptom: the pool would still be unsafe to reuse from inside a step, and the next thing to want nested parallelism would reopen the same hole with no test to catch it.

A dedicated hashing pool, so the two pools cannot starve each other. It keeps D3’s parallel hashing and removes the cycle, but it makes the real thread count jobs plus that pool’s. --jobs is a promise about machine load — the situation that produced this bug was two runs on one machine — and quietly exceeding it is the wrong direction for a tool people leave running overnight on shared hardware.

What this costs is the scheduler itself: the worker loop, the condition variable and the active count are gone, and M3’s termination argument is now the scope’s rather than a count of busy workers. That is a real change to closed code, which is why it is pinned by two tests that fail in ten seconds against the old scheduler rather than hanging (the scheduler). It also pays something back: a parked twin no longer occupies a --jobs slot doing nothing.

D28 — The rules are vendored, and enter a key by content

Interpreting a mutation needs a ruleset. Stanford publishes the HIVdb algorithm as ASI2 XML in its own repository, hivdb/hivfacts, under the Unlicense — a dedication to the public domain. Two versions are vendored byte-for-byte and pinned by SHA-256: 9.8, which is what the published results M6 validates against were produced with, and 10.2, which is current.

Calling Stanford’s web service was the alternative, and it fails on the project’s own terms: an answer that depends on the day it was asked for cannot be put in a cache key or carried in a bundle. D8 puts public-data fetching out of scope for v0.1 in any case.

Each algorithm is committed twice — the XML as published, and a .ruleset derived from it by scripts/vendor_hivdb_algorithm.py. The XML is what you audit against upstream; the derived file is what the engine reads, which means no XML parser in the core and no new dependency. tests/test_rulesets.py re-derives every .ruleset on every run, because a derived file nothing re-derives is one that can quietly stop matching its source.

What the rules enter the key as took longer to settle. A callback step’s recipe describes as the constant "callback" and code is the step function’s source, not what a library it calls reads, so they had no home at all. The obvious fix — an algorithm="HIVDB_9.8" parameter — pins by name, which this project refuses everywhere else: the dimension catalogue says reference identity is never by filename alone, and load::variants distrusts ##reference because it holds a path. Upstream ships both HIVDB_10.0.xml and HIVDB_10.0-1.xml, so a correction republished under a near-identical name is not hypothetical, and a key that moved only with the name would serve a result computed under rules that have since changed.

So tool= carries it. That field already means “something whose version changes the answer”, and its version string is already free-form. A built-in list, one entry long, keeps the exception visible instead of hiding it in the probe, and hivdb resolves to every vendored algorithm by version, date and the BLAKE3 of its bytes. Doing it this way picks up the key, provenance, the methods sentence and D24’s citation table at once — and HIVdb needs a citation, which a parameter gave no home.

Every shipped algorithm is named rather than only the one in use, because the probe runs before any step body has said which it wants. Editing 10.2 reruns a pipeline pinned to 9.8. That is over-conservative in the direction D20 already commits to: a cache that is too cold costs time, one that is too warm is wrong. The string reads “as shipped” because a methods paragraph renders it after “ran with”, and a step that named 9.8 must not appear to have run with 10.2.

And the engine that reads them. The rules are half of what a hivdb step is; the other half is the engine’s own judgements about reading them — which codons count as read, how much of a gene has to be (D34), what a mixture satisfies. Those move between releases, and a key naming only the rules would serve an interpretation the engine in front of the reader would not produce: exactly what the scored-codon floor would have done to every cache holding the earlier answer. So the tool version ends read by ligase <version>, over- conservative in the same direction — a release that touched none of this still reruns every hivdb step.

D29 — An interpretation never invents a susceptible result

An interpretation can be wrong in two directions and they are not equally bad. Reporting resistance that is not there costs a patient a drug they could have taken. Reporting susceptibility that is not there costs them a drug that will fail. Three judgements could each have gone either way, and all three go the same way.

A mixture is every amino acid it could be. Where the reads support both the reference and an alternate above the threshold, the codon holds both, and a rule is satisfied if any of them satisfies it. An ambiguous position is two viruses, not an average of one, and taking the majority would discard the minority virus — which is what deep sequencing is for.

A position that was not read is not a wild-type position. bcftools writes only the sites it has something to say about, so a position missing from a VCF is ambiguous between “agrees with the reference” and “nothing aligned here”. Coverage is read from the alignment; a codon not covered to depth is unread, and a drug whose rules mention one reports its score as a floor and names what it could not see. An item already refuted by a position that was read is settled, so its unread positions are not reported — otherwise every drug would list most of its positions and the warning would mean nothing.

A gene nobody sequenced is not assessed. HIVDB 9.8 scores lenacapavir against capsid, which is in gag. A pol pipeline has nothing to say about it, so the report says so. This is D24’s reasoning about citations applied to a drug class: a named gap is honest, a gap filled with a cheerful default is not.

The test that pins all three is a no-template control. Sixty of its thirty-two thousand reads map to HXB2, and a pipeline that read “no variant called” as “agrees with the reference” would report a tube of water as fully susceptible to twenty-seven drugs.

A fourth judgement joined them with D31, and it goes the same way. A codon is only the combinations the reads carry. A VCF says what is at each position rather than what travels together, so enumerating two ambiguous bases in one codon yields up to four amino acids and some are carried by no read at all. The alignment settles it, and what happens where it cannot is D31.

D30 — A consensus caller cannot answer a resistance question

The demo’s call step ran bcftools call -mv --ploidy 1 through M5, and its docstring defended the choice: bcftools call is a consensus caller, a threshold below a half rarely removes anything, and a low-frequency caller is a change of tool rather than of threshold. All of that was true.

M6 measured the cost. On SRR32217726 the reads at HXB2 3204 are A 238, G 192 — a 45% K219E, a zidovudine resistance mutation — and bcftools call -mv emitted no record for it, because the majority allele is the reference and there was no variant to call. The report came back susceptible to all twenty-seven drugs.

Given that input it was right: it matches Stanford’s own majority-consensus report for the same sample exactly, which is useful evidence that the interpreter is sound. But Stanford’s 5% report for that sample also finds L100I and reads high-level resistance to four NNRTIs. A pipeline whose answer flips from “fully susceptible” to “resistant to most of the NNRTI class” on a threshold it never exposed is not doing what it says it does.

So the caller is gone. The pileup’s own allele depths are the evidence, min_freq is applied to them directly, and the parameter now means what its name says. What counts as present for interpretation is a separate parameter, mixture, because the two are different questions: one is about what the reads show, the other about what you are willing to act on.

The rejected alternative was to keep the caller and document the limit. It is cheaper and it is what the old docstring did — but a limitation documented in a docstring is not visible in a report, and this one changes the clinical answer.

D31 — A codon no read spans keeps its per-base reading

A codon was assembled from the VCF: the alleles the caller credited at each of the three bases, then every combination of them, translated. A VCF says what is at each position and not what travels together, so the combinations include codons no read carries.

That is not hypothetical. On SRR32217726 the caller credits G at the first base of RT codon 219 in 43% of the reads it counted there, and at the second base in 2.4%. At a 2% threshold both pass, and the four combinations read K, R, E and G — where of the 4,347 reads spanning the whole codon, 54% carry AAA and 45% carry GAA, and AGA and GGA are under two tenths of a per cent each. Two of those amino acids are arithmetic rather than virus. Six of M6’s nine published discrepancies were this, and although none of them changed a score, it is the kind of wrong the project exists to avoid.

So codons are read from the alignment, which was already an input for coverage. Reads spanning all three bases of a codon are counted as the triplets they carry, and the codon holds the amino acids whose share of those reads reaches mixture.

Three things about that had to be decided, and each could have gone the other way.

Phasing narrows a reading and never widens one. Which alleles are present at a position stays the caller’s judgement, because bcftools mpileup weighs base quality and a raw read base does not. Sample 30 is the case that proves it: at HXB2 2773 the raw bases carry the V75A base in 3.2% of the reads there, where the caller credits it in 1.7% of the reads it counted. Taking the raw count would “find” a mutation at 2% — and would agree with the published report, by accident, on evidence the caller had already judged too poor to report. So phasing only chooses among the combinations of alleles the caller credited; it never adds one.

The threshold applies to the amino acid, not to the spelling of it. Two synonymous triplets each below mixture can hold an amino acid well above it, and the report is about amino acids. Thresholding each triplet on its own would lose one that is there, which is the one direction D29 forbids.

A codon no read spans keeps its per-base reading. Paired-end reads can cover a codon between them without either mate spanning it, and a codon with an indel across it is not read whole by anything. Three answers were available:

• Report it unread. Honest about the phase, but wrong about the rest: the per-base evidence does establish which alleles are there, and only their phase is unknown. An unread codon makes every drug whose rules name that position report a floor, so discarding what is known would spread a hedge across a report over one missing read.
• Guess a phase, for instance by pairing majority with majority. This invents exactly what the change exists to stop, and it can drop a resistant amino acid — D29’s forbidden direction.
• Keep the per-base reading, which is what Ligase does, with a note saying how many codons of which gene fell back and why. The reading is then wider than the truth rather than narrower: it can over-report resistance, never under-report it, which is the direction D29 commits to everywhere else.

That note is a Reading::notes entry, which is where the “no alignment was supplied” note already goes. Both reach the person reading the result: the step hands them back to the engine, which says them in the run report, in provenance, in a bundle and in the methods section (D32).

The same fallback covers a codon the reads span but say nothing useful about — fewer than depth spanning reads, or no credited combination among them. All of those mean one thing, “the reads cannot settle this codon”, and they get one answer.

In practice the fallback does not fire. On each of the validation’s three covered samples, all 1,178 covered codons are spanned whole by at least a hundred reads, because the window is three bases and the reads are a hundred and fifty. It is a guarantee rather than a path, which is the right shape for a hedge: it exists so that a shallow or short-read dataset degrades towards over-reporting rather than towards silence.

The cost is reading sequences where coverage alone needed only spans. On the deepest run of the validation — 95,000 reads — the resistance step goes from 0.6s to 1.1s, and the three-threshold validation run from 14s to 21s. Doubling the cheapest step in the pipeline to stop it inventing amino acids is not a close call.

D32 — A step says what its own work rested on

Three of the assumptions an HIV resistance reading makes have a clinical direction, and all three were silent. A reading made without an alignment takes every position the caller did not report to be reference, which manufactures susceptible results out of dropouts (D29). A gene whose codons did not all reach the depth asked for has drugs whose scores are floors rather than answers. A codon no read spans is read base by base and can name a mutation no single read carries (D31). Each was recorded on Reading::notes, returned through interpret_resistance, and dropped by the Python wrapper that called it. Nothing rendered any of them.

The engine already had the channel. A Note is carried beside a step into the run report, provenance, the bundle and the methods section, in the same words each time (D26). What it lacked was a way in from a step’s body, because canonicalising a resistance report cannot find any of this: a run with full coverage and a run with none can write the same bytes. The finding is not a fact about the output, which is exactly why the output cannot be read for it.

So Host::call returns what the body noticed, and lg.write lets a body hand back pairs of a note’s code and its value. The wording stays in Rust with the rest of each note’s advice and consequence, so a body names the kind of finding and the engine says the sentence.

Three alternatives were considered.

A column on the report, which is where the brief for this change started. It travels with the file, which is a real advantage over a line in a terminal, and it loses on three counts. The report’s columns are part of what a report is — the canonicaliser keeps the header as a row — so adding one changes every report’s canonical hash and makes the diff against Stanford’s own reports that M6 rests on harder to read. The sentence would repeat on all twenty-seven rows, because the hedge is about the reading and not about any one drug. And it would reach only the person holding that one file, when the question “what did this result rest on?” is asked by a reviewer reading a bundle.

Returning them from lg.hivdb.interpret, which cannot work: the function is called before anything runs and returns a recipe, so there is nothing to return notes about yet.

Carrying them without storing them, which fails on the second run. D26 settled this for the other kind of note and the reasoning is unchanged: a cache hit does not rerun the body, so a hedge that appeared the first time and vanished the second would be worse than none. They are stored against the step’s key rather than the output’s hash, because they are not facts about those bytes — and the key already carries what they depend on, since the alignment is an input. The cache’s SCHEMA_VERSION moves to 3 for the column, which clears its tables; D19 discards rather than migrates, so that is the whole cost.

Two smaller things fall out of it.

A code no Note names fails the step rather than being dropped. The codes come from this engine’s own notes, so an unknown one is a mistake in a body — and a note quietly swallowed here is precisely the silence this change exists to end. For the same reason a body that returns something other than notes is told what to return: lg.write(lambda out: out.write_text(…)) hands back a character count, and that is the mistake a user actually makes.

Note::explains_a_difference separates the two kinds for bundle verification. A sample identifier that is a file explains a differs with no science behind it, because such a hash depends on where the cache was. A coverage hedge explains nothing of the kind — it is the same on every machine — and offering it as an excuse for a hash that moved would send a reader after the wrong thing. It still appears everywhere else a note does.

What this does not do is make a shell step able to hedge. A command cannot be asked what it assumed, so a step that shells out has to say it in the file it writes. That is a limit of the mechanism and not a decision against it.

D33 — The frame a reference is numbered along is declared, not guessed

codon::GENES states the HIV-1 coding regions as nucleotide ranges — protease is 2253..2549 — and those numbers mean nothing on their own. They are positions in HXB2 K03455, counted from the first base of the genome. Nothing checked that the reference handed to a reading was numbered that way.

A pol-relative reference is the obvious thing to have when the amplicon is pol: K03455 2085..5096, so every position in it is 2084 lower. Handed one, the engine read what is really integrase as protease and reported intermediate protease-inhibitor resistance, with no error and no note (SAWA’s evaluation found it). That is the class of bug this project exists to prevent, so finding it here settles what the fix has to be: loud, and before any compute.

A reference now carries the frame its positions are numbered along, as part of its identity — a name and the base of that frame its own base 1 is. A step annotated -> ResistanceReport has said in its signature what it reads, so the graph compares the two and refuses a reference that is not that frame.

Why the frame is declared and never inferred

It cannot be read out of a file. A 3,012-base slice of K03455 and a 3,012-base genome of something else are identical from the outside — same kind of defline, same kind of bases, no record anywhere of where base 1 sits in anything else.

One case is an exception and it is the common one: a reference whose single contig is 9,719 bases is whole-genome K03455, or something deliberately numbered along it, and needs no declaration. Length is the one fact the gene windows actually depend on. Checksum equality would have been stricter and wrong, because it would refuse an HXB2-coordinate consensus of another subtype, which is a legitimate thing to interpret against.

Everything else is asked about rather than guessed at. The reference demo declares nothing and is unaffected.

Why this is not the coordinates dimension

D9 keeps coordinates to semantics: whether a position is counted from zero or from one. A frame is one axis further out — what a position is counted along — and folding the two together would make one dimension answer two questions.

Nor is it an eighth dimension. Dimension checking is pairwise: two inputs that both state a value must agree, and a value is required only where a step joins two inputs on it. Only a reference can state a frame, so an eighth axis would have had nothing to be compared against and would never have fired. Reference identity is where it belongs, and it is what the task asked for in the first place: a frame is part of what makes two references the same reference.

Why a declared slice is refused rather than shifted

Once a reference says it starts at K03455 2085, the engine knows the offset and could add 2084 to every window. It does not. Shifting coordinates is a conversion, and conversions are typed functions in a conversion graph — D8 puts auto-fixing out of v0.1’s scope, and doing it here would be auto-fixing by another name. The error names the offset and the padding that fixes it instead.

What this does not catch

A declaration is taken at its word, because there is nothing in a FASTA to check it against. A pol-relative reference declared as a whole genome passes the check. The reading is the floor under that: the windows reach base 5093, and a file that does not is refused when it is read, which is late but not silent.

D34 — A gene barely read is not assessed

D29 says an interpretation never invents a susceptible result, and the report already honours that per position: an unread position makes every rule naming it undetermined, the score is a floor, and the unread positions are listed.

What that does not survive is a gene mostly unread. Nothing there refutes resistance and nothing there shows it, so every drug scored against the gene lands at zero — which the algorithm’s own global range reads as level 1, Susceptible. A clean clinical negative, from a gene the run never covered. SAWA saw it: a sequence covering 32% of integrase came back as five susceptible integrase inhibitors, where Stanford’s service declines to report the gene.

So a gene read below gene_coverage of its codons is left out of the reading, which makes its drugs not assessed — the same state as a gene the pipeline never supplied, which is what it effectively is. The reading says which gene, how much it read and how much it needed, as a note the engine carries into the run report, provenance, a bundle and the methods section (D32).

Four fifths by default, and a parameter. How much of a gene a surveillance programme is willing to call susceptible on is its own decision, and a judgement belongs in a cache key rather than in a constant nobody can see — the same reasoning as mixture and depth.

The share is of the scored codons, not of the gene

The first form of this counted the share over the gene’s own codons, and that made the default close to unsatisfiable for the records clinical labs hold. Reverse transcriptase is 560 codons. HIVDB 9.8 scores 40 of them, the highest at position 348. Four fifths of the gene is 448 codons, so a partial-pol genotype that read protease and RT 1–348 — every position any drug is scored on, and the standard extent of a clinical genotype — had covered 62% of the gene, and the whole of RT came back not assessed. The floor was refusing readings for not covering codons nothing would ever consult.

So the share is counted over the codons the chosen algorithm scores: the positions some drug’s rules name, restricted to the gene’s own window. A position no rule names cannot change an answer however it was read, so reading it is not evidence that the gene was read well enough to score — and a position a ruleset names past the end of a gene is left out, because no reading could cover it and a floor nothing can clear is not a floor.

It is not simply a lower floor. Counting the right codons tightens the check where it was loose. A dropout across RT 40–120 leaves 479 of 560 codons read — 86% of the gene, which the old floor waved through — and takes 21 of the 40 positions the rules score, leaving 19 against a floor of 32: the gene was being scored against under half the evidence that could have refuted it. That reading is now refused. And the failure this decision exists for is unchanged: 32% of integrase is five of the twenty-six scored positions against a floor of twenty-one, still not assessed.

The note’s second count moved with it — the scored codons rather than the gene’s, written scored=. A note stored under the earlier codons= is still read, with the number that run used: a stored note is a record of what that run noticed, and dropping it would lose the record rather than correct it.

What this does to a cache. The engine now answers differently for the same calls, the same reference, the same parameters and the same rules, and a hivdb step’s tool version named only the rules — so an existing cache would have gone on serving the old refusals. The engine’s own version is now part of that tool version, which is D28 applied to the engine as well as to the rules: a cache too cold costs time, one too warm is wrong.

The sharper rule, and why it is not here yet

The share is now over the positions the rules name, but it is still a share over a gene. The exact question is per drug: whether the positions this drug’s rules name were read, and if not, whether what was missed could change the level. The engine has the floor already; comparing it against the ceiling the unread positions allow would let a drug report “indeterminate” rather than a level, with no threshold at all.

It is deferred because it would have to live in the scoring layer, and the scoring layer is pinned to 3,348 published Stanford expectations whose fixtures name only the positions that differ from consensus B. Every other position was sequenced and found wild-type, but the fixture does not say so, so the engine reads them as unread. A ceiling computed from those would turn nearly every published expectation indeterminate, and the oracle would stop being an oracle. Teaching the fixtures to state their coverage is the prerequisite, and it is a change to the validation suite rather than to the engine.

D35 — A kind must bring a dimension that catches a documented silent failure

The engine knew eight kinds of data when this was written, and every request for one more arrives the same way: a project has files in a format Ligase does not know, and teaching it the format looks like a morning’s work.

A kind earns its place only if it brings a dimension whose mismatch is a documented silent failure in real research — not a conceivable one, but one somebody has published, or chased for a week, or had pointed out in review. A kind that is a file format with a schema is a parser, and Ligase does not need more parsers: the libraries that read those formats exist and are better at it than a side project will be.

The cost a kind carries is not its loader. It is a canonical form, a row in the applicability matrix for every dimension, a share of the vocabulary every error message is written in, and all of that kept correct for as long as the engine exists. A kind that catches nothing spends it to make Ligase a worse version of a format library. So the answer to a kind that fails the test is no, however convenient it would be for one project — including this one.

The rule also says something permissive that is easy to miss: a kind need not hold sequence. A curated gene list is a text file with one column, and the identifier space those genes are named in is as sharp a dimension as coordinates and more widely botched. What disqualifies a kind is bringing no dimension, not being unlike a BAM.

D36 — Release is its own dimension; Reference keeps its meaning

Data that came from a versioned external resource — a ChEMBL version, an Ensembl release, an HGNC download date, a snapshot timestamp — wants a dimension, because two inputs built from different releases of one database are the hg19-against-hg38 failure in a different costume. The tempting place to put it is reference, generalised from “which genome build this is on” to “which pinned external version this is relative to”. A ChEMBL version is a pinned external version. The phrase even fits.

It is rejected, and the reason is the comparison rule rather than the phrasing. reference compares by containment (D10): contig names, lengths and MD5s, with extra contigs in the reference allowed, because a VCF whose header lists only the contigs it has variants on is correct input. A database release has no contigs. There is nothing for one release to contain of another, and ChEMBL 33 against ChEMBL 34 is a mismatch outright — equality, with no subset case to make room for.

The generalised dimension would therefore carry two comparison rules and pick between them by the kind of data it happened to be sitting on. That is the quiet complexity this engine exists to refuse: the same dimension passing check() under one rule and failing under the other, with no error message able to say which rule it applied or why. So release is a dimension of its own, meaning which versioned external resource this data came from, and reference keeps the meaning it had.

Two dimensions may share one rule

The converse is not forbidden, and the implementation rests on it. One dimension may not carry two rules; two dimensions may share one.

release and subject are both “one label, and a different label is a different thing”, so they are one type. organism and source_set are both “which of these things is the data about”, where order is noise and a missing member is the whole point, so they are the other. Four axes, two value types, and each rule written exactly once — which is the point, because a rule implemented twice is a rule that will eventually be implemented differently.

What differs per dimension is the wording a user reads, and that lives on the leading question rather than on the value: the same set comparison asks “did you pass data from the wrong organism?” on one axis and “was a source added or removed upstream?” on the other.

The sharing was settled for organism and provisional for source_set, and source_set has since been split out. The set type collapsed members that differ only in ASCII case, keeping the first spelling. For Latin binomials that is exactly right. For what source_set holds it is mixed: GSE12345 and gse12345 are the same study, but s3://bucket/A.vcf and s3://bucket/a.vcf are two different objects, and the second would have been dropped without a word. Nothing could go wrong while no kind applied the axis. The right to split it into a type of its own, with its own equality, was reserved for the milestone that gave the dimension a kind, and the obligation was written into M8, M10 and M12, each of which declined the axis with reasons.

M15 is the milestone that applied it, and the split landed there: the set type is generic over how two members are compared, Organism folds ASCII case and SourceSet does not. The algebra is still written exactly once, the two rules are each written once, and the two axes are now two types — so an arm of check::compare that read organism where it meant source_set no longer compiles, which is stronger than the test M7’s standing requirement asks for. Release and Subject remain one type, so the requirement still stands for those. The rule itself was settled earlier, by M10a, on the asymmetry of the two failures: folding case can only drop a member and produce a false agreement, while not folding it can only add one and produce a refusal a user can read. Splitting it was the decision recorded here being carried out, not reversed.

D37 — Organism is set-valued

organism was a single string, which assumes a dataset is about one organism. A host–pathogen sample is not: plasma from a person with HIV-1 holds the person’s nucleic acid as well as the virus’s, and the ordinary reference for it is a combined FASTA carrying both genomes. A metagenome is about many organisms. A xenograft is about two. A disease knowledge graph joins human genes to evidence from model organisms.

Given one string, every one of those gets written down as whichever organism the analysis is about — the pathogen, the host — and the declaration is then true of the analysis and false of the data. Two such declarations agree with each other, so nothing objects, and a later step treats everything in the file as one organism’s. That is the failure written up in my pathogen results include host reads.

So organism is an ordered set, and a single organism is a set of one: organism="HIV-1" means what it always meant and renders as it always did. Comparison is set equality — the same members, in any order, each trimmed and compared without regard to case — because ["Homo sapiens", "HIV-1"] and ["HIV-1", "Homo sapiens"] are the same dataset, and the order somebody typed two names in is not a fact about the sample. Insertion order is kept so a message can echo what was written; it is not part of the identity.

An unequal comparison then names the symmetric difference instead of only reporting that two sets differ: ref also lists Homo sapiens., or, where each side holds something the other lacks, aln also lists HIV-2, and ref also lists Homo sapiens. The two value lines already state both sets in full; what they cannot do is point at the member that makes them unequal, and for a metagenome of a dozen organisms checked against one of eleven, that member is the only part worth reading.

Naming the difference rather than choosing a reading of it is D26’s habit: the engine says what it saw and leaves the rest to the person who knows the sample. There are three readings here and the engine cannot tell them apart — the narrow side may be declared too narrowly, the wide side too widely, or the two files may simply not belong together.

An empty list is a gap, not an empty set. Two datasets that each declared no organism would otherwise agree, which is exactly the silence the dimension exists to break. So the Python loaders refuse a list naming no organism, and the core, which could raise but chooses not to (load::reference returns a Result, and an empty list is not among its errors), records the dimension as unknown with a reason rather than as a set of nothing.

There is still no alias table. ["human"] and ["Homo sapiens"] are two different sets and are reported as a mismatch, because equating them means guessing, and a wrong guess passes check() silently. Declare one spelling and use it.

D38 — Identifier namespaces never coerce

TP53, ENSG00000141510, 7157, P04637, NM_000546 and HGNC:11998 are six correct names for one gene, and no two of them are equal as strings. A gene list records nowhere which of those spaces its names are in: a TSV has a column called id, and which space its contents belong to was settled by whatever exported it — featureCounts gives Ensembl IDs, a GEO supplementary table usually gives symbols, a proteomics search gives accessions.

So joining two lists is a string comparison between two naming conventions. It does not fail. A join of two columns with nothing in common returns the rows they share, which is none, and an empty result is an ordinary thing for a join to produce. The failure is written up as my gene-set overlap is empty, and what makes it the worst kind is that the answer is rarely zero: a few identifiers collide, or one list was already mixed, so the overlap is small and non-zero and reads as a measurement of poor concordance rather than as a mismatch.

So identifier_namespace is a dimension, compared by equality, and two lists in different spaces are refused.

Why the obvious convenience is refused

The tempting thing is to coerce: recognise that one side is symbols and the other Ensembl IDs, map between them, and let the join succeed. Every ad-hoc df["gene"].str.upper() and every hand-written dict lookup in the field is that convenience, implemented once per analysis.

It is refused because mapping between spaces is a lookup with failures, not a rename. Going from TP53 to ENSG00000141510 has four outcomes, not one. Most identifiers find a row. Some find none, because they were renamed or retired since the mapping table was downloaded. Some find several, because one symbol legitimately covers more than one gene. And some arrive where another identifier in the same list arrives, because the table holds both a deprecated symbol and its replacement.

A dimension that coerced silently would have to decide what to do with the last three on the user’s behalf, invisibly, and every available decision is bad. Drop them, and the output is quietly smaller than the input — the thing D26 exists to refuse. Carry them over unchanged, and a name from the old space is sitting in a set that declares the new one, which makes the output’s own declaration false. Take the first of several, and the result depends on the order of rows in somebody’s file. Let two identifiers collapse onto one, and the output names a single entity twice with two sets of attributes, so it is not a set and anything counting it double-counts.

What the rule buys

The only way across is a map_identifiers step, which turns all four outcomes into things a user can see. Its output declares the space it has arrived in, so everything downstream is checked against a true statement. The identifiers that did not cross are named, and counted, in the run report, in provenance, in a reproducibility bundle and in the methods paragraph, so the output being smaller than the input is a recorded fact rather than a discrepancy somebody notices six months later. Counted as well as named because a conversion that leaves four thousand identifiers behind needs a number before it needs four thousand sentences — so the named ones stop at twenty, which is enough to see a pattern, and the total carries the rest. And its mapping table is a typed input, so the table’s contents join the cache key and the table’s own space is checked against the list being mapped — an Entrez table handed to a list of symbols is refused before anything runs.

This also makes the conversion the first edge of the conversion graph (D5) rather than a special case beside the checker. liftover and rename_contigs are later rows in the same registry: each declares the one axis it crosses and reads its destination from the same parameter, and the engine’s planned automatic fixes are a path search over those rows. See conversions.

Inference, and four refusals to guess

The space is the one dimension the engine can usually read without being told. Six of the seven spaces it recognises carry a marker — a prefix, an underscore, a colon, or being digits throughout — and the seventh, HGNC symbol, carries none, which is why the shapes are tried most specific first and the symbol is the fallback. P04637 reaches UniProt accession rather than being read as an unusual gene name.

Inference then succeeds only when every identifier in the file resolves to the same space and that space carries a marker, and declines in four cases:

• a file that mixes two spaces — precisely the corruption the dimension exists to catch, where taking the majority would hide it;
• a file with no rows — every shape fits all zero identifiers, so an answer would declare a space for a file that names nothing;
• a file holding one identifier nothing claims — 2-Sep is in no space, and a space declared over such a file would be a claim about names the engine could not read;
• a file whose identifiers all fit the marker-less shape — bare gene symbols, which are also the shape of ChEMBL’s component synonyms and of most in-house panels, so the file has not stated a space (D56).

Where it declines, the dimension is unknown with a reason. And unlike release, subject and source_set, this axis is demanded where two gene lists are joined: a user can always satisfy it, and two lists that each say nothing would otherwise agree by silence, which is the failure itself in its purest form.

A declaration is recorded, not verified

Where the user declares the space, the identifiers are not consulted. That looks like a missed check and is deliberate. The catalogue knows seven spaces and there are more than seven: a hand-curated resistance panel, a lab’s internal accessions, a consortium’s own numbering are each a real space, named in a way the engine has never seen, with the same right to be compared as Ensembl gene has. If a declaration had to match the catalogue, the dimension would be unusable for exactly the in-house truth sets that most need it, because those are the files nobody else can validate for you.

So the engine cannot distinguish an unknown-but-correct space from a wrong one. What it can do is refuse to connect two sides whose statements disagree, and read the space off the file when nothing is declared — which is the stronger position wherever the identifiers are recognisable.

And there is still no alias table

HGNC symbol and HGNC symbols are two different spaces; so are Ensembl and Ensembl gene. They are reported as a mismatch, on the same grounds that keep human and Homo sapiens apart (D37): equating two spellings means guessing that they mean the same thing, and a wrong guess passes check() in silence. Declare one spelling and use it.

The spreadsheet case

A list of symbols opened in Excel comes back with some of them converted to dates: SEPT7 to 2-Sep, MARCH1 to 1-Mar. This is measured across the supplementary files of published genomics papers at rates high enough that HGNC renamed the affected gene families to stop it happening.

Such a file loads, because it is valid TSV and refusing it would turn a format problem into an engine failure (D21). What the loader does instead is notice it: the identifier '2-Sep' is a date, not a name. A note, not an error, because the file is what it is and the fix — re-exporting with the column typed as text — belongs to whoever has the original. There is no repair option, because 2-Sep could have been SEPT7 or SEPT2 and the engine will not guess which gene was meant.

The other corruption, a symbol turned into a date serial such as 42615, is deliberately not detected: it is indistinguishable from an Entrez gene ID, and a rule that flagged it would report real identifiers as mistakes.

D39 — A remote is pinned or recorded, and both are replayable

The first step of nearly every pipeline is a fetch, and until M10 it sat outside everything this engine promises — including in Ligase’s own demo, where a three-hundred-line shell script downloaded the reads and the engine took its word for what had arrived. That is a strange hole to leave in a tool whose whole claim is that a result can be accounted for.

Closing it means answering a question first: what, exactly, is being promised about a fetched input? There are two answers, and conflating them would be the usual mistake.

Two reproducibility claims, kept apart

Source reproducibility — that a third party will serve the same answer next Tuesday — is a real promise and some sources make it. An SRA run accession is an undertaking; HXB2 is K03455.1 and will be; a numbered database release exists in order to be cited. Where a source makes that promise, a digest is how you hold it to it.

No live query can make it, and pretending otherwise is how a paper becomes unreproducible without anyone noticing. Open Targets will answer differently next quarter, because it is supposed to.

So a recorded query promises something else, and something that turns out to be enough: record reproducibility, that anyone with the bundle can run the pipeline again and get the same result. That needs exactly one thing — the canonicalised response body goes into the content-addressed store, not merely its digest. The body is then an ordinary input with a content hash, and everything downstream is deterministic by construction.

This is the record/replay pattern of vcrpy and of a Nix fixed-output derivation. Ligase’s contribution is not the pattern: it is that the record is typed, content-addressed, part of the cache key, and enforced by the engine rather than by discipline.

Three modes, set per run

The default replays what exists and records what does not, and the run report always says which it did. In replay mode a query with no record is an error, never a silent live fetch — a half-replayed run is the plausible-but-wrong result Ligase exists to refuse. And a refresh never replaces a record: it writes a new dated one beside the old, so every earlier run stays replayable. A surveillance pipeline refreshing weekly is not reproducing one run; it is a series of runs, each with its own record, each replayable forever.

A record’s identity is the canonical request: method, URL with sorted parameters, request body, nothing volatile. A step making twenty paginated calls writes twenty records keyed by request rather than by sequence, so call order is irrelevant. The response is canonicalised like any other kind — JSON keys sorted, request IDs and server timestamps and rate-limit headers dropped — so two responses meaning the same thing hash the same, on the @PG-line precedent of D22.

Volatile fields go from the top level and nowhere else

Added in M10b, because building the canonicaliser raised a question this decision had not answered: where to look for a server’s bookkeeping.

The closed set is requestId, traceId, correlationId, timestamp, generatedAt, servedAt, retrievedAt, took, elapsed and queryTime, matched against a key lower-cased with - and _ removed, so four spellings of one name are one name. Closed and listed, on the same ground as the four schemes: a set extensible by configuration would be a promise the engine could not keep, and the failure would be a silently deleted field rather than an error.

Dropping them only from the top level of a JSON object is the part worth arguing. Recursing is the obvious implementation and the wrong one. A timestamp key at the top of a response envelope is the server describing its own work, which is D22’s case exactly; the same key inside a data record is what the record says, and a pipeline reading a time series would have it silently deleted. That is D26’s distinction — a field that is part of what the data is about cannot be dropped — and the top level is the one place the two cases can be told apart without guessing.

Even there the key might have been the user’s own data, so each drop is a note rather than a silence. The hash stands and the engine says what it saw.

Response headers need no rule at all: the type the transport returns has no field for them, so a rate-limit header never reaches anything that is stored.

A digest is over what the file means

Not over its bytes: the canonical form of what arrives, read as its declared kind, folded over its files in order. For a kind with no canonical form the fold is over the bytes, which needs no special case because D21 already says canonicalisation falls back to them.

This is D4 applied to an input, and the argument for it came from the shell script that M10 deleted. It pinned HXB2 by the checksum of its sequence rather than of the file, with a comment saying why: NCBI may reflow lines or reword a defline without the biology changing, and that must not break the build.

The comment is right, and the reason is about people rather than bytes. An alarm that fires on a cosmetic reformat teaches you to bump pins without reading — and then the one that matters gets bumped too. So a changed meaning is a loud failure naming both digests, and bytes that moved with the meaning intact are noticed rather than failed on.

Folding even a single file means a lone file and a paired run are described the same way, and that swapping a pair’s mates is a different digest — which it must be, since it reverses every read’s orientation.

Resolution happens before the graph exists

A remote resolves when the pipeline declares it. That is not a deferred optimisation: a remote’s dimensions cannot be known until its bytes are in hand, and the engine checks dimensions before any compute. You cannot type-check data you have not got.

Which is why “a wrong digest fails before any step runs” needed no arranging. There is no graph yet when it fails. The cost is that ligase check on a cold store reaches the network once; on a warm one it reaches nothing.

A closed set of four schemes

sra:, ncbi:, ensembl:, https:, and anything else is an error naming the four. A scheme is a promise that the engine knows how to turn an accession into bytes and how to recognise the answer — ncbi: asks eutils for FASTA, sra: knows ENA’s directory layout and that a run is a pair of files. An open set, or one extensible by configuration, would be a promise the engine could not keep, and the failure would be a fetch of the wrong thing rather than an error. Adding a fifth is a decision, not a setting.

https: is the escape hatch and deliberately the only one: it says “these exact bytes, from this exact URL”, which is a promise the user makes rather than one the engine makes for them. A loopback address may be written http://, which is the same scheme by another name — TLS protects a channel against something sitting in it, and between two ends of a loopback socket there is nothing to sit in.

A bare accession with no scheme is refused rather than guessed at. SRR8525904 looks like an SRA run and K03455.1 like an NCBI accession, so a guess would be right often enough to be trusted and then one day fetch from the wrong archive.

A 200 is not a promise about the body

An archive asked for an accession it does not have does not always say so. NCBI answers with an HTTP 200 and an HTML page beginning Error: ID list is empty. Nothing in the status says anything is wrong, and with no digest declared there is nothing to compare against.

So what arrives is read as the kind it was declared to be, and a body that will not parse as one is refused. That is a deliberate departure from D21, which lets a local file that will not parse fall back to its bytes: a local file is one the user chose, and a fetched one is whatever an archive returned.

A secret has nowhere to be stored

Auth headers are not stripped before storage. The type the transport returns has no field for request headers, and neither does anything derived from it, so there is nothing to strip. That is the only version of “a bundle never carries a secret” that a later field cannot quietly undo, and the test for it asserts over the whole record and the whole bundle rather than over a list of header names.

What this retires

python/ligase/provenance.py records the limitation that “a bundle cannot fetch its own data”. With recorded bodies in the bundle it does not need to: a reviewer replays the run with no network and no API keys, and the methods paragraph reads “retrieved from Open Targets on 22 October 2026; responses are included in the reproducibility bundle.”

A pinned remote’s bundle is different and deliberately so. It names the locator and the digest rather than carrying the bytes, because that is what pinned means — anyone can obtain it from the source and check it against the digest recorded there, and copying gigabytes of reads into every bundle would buy nothing.

The store became a trait in the same change

Records are the first thing written to the content-addressed store that a lab will want to share, so M10 introduced the small trait that M14 implements over object storage: has, get, put, with the local filesystem as the only backend for now. Two additions to the three operations were not conveniences. A key carries a suffix beside the hash, because a step’s output is handed to bcftools as a path and those tools dispatch on the extension — a trait keyed on the hash alone could not name the file it was asked for. And verification is a provided method on the trait, not a backend’s job, because “nothing read from a store is trusted” must not be something each backend author remembers separately. See remote inputs for the shape and what was given up to get it.

D40 — A stochastic step declares itself, or it is an error

A step whose tool draws on randomness — IQ-TREE, RAxML, BEAST, AutoDock Vina, seqtk sample, any model training — declares stochastic=True and names its seed parameter. An unseeded stochastic step is a StepDefinitionError at decoration time, like an untyped return.

@lg.step(tool="seqtk", stochastic=True, seed="seed")
def subsample(reads: Reads, fraction: float = 0.1, seed: int = 20260101) -> Reads:
    return lg.sh("seqtk sample -s{seed} {reads.r1} {fraction} > {out}")

The engine cannot look at a tool and know whether randomness is in it, and no amount of running a step twice can tell it either: two runs that agree agree for one of three reasons — the step is arithmetic, the step is seeded, or the step got lucky — and comparing outputs cannot distinguish them. Only the pipeline knows. So the pipeline says.

What the declaration is for

The audit vocabulary already distinguishes deterministic from reproducible — three verdicts, not two, because every real tool records its own command line and byte equality is therefore too strict a test. This is what gives that distinction teeth.

A verdict from ligase audit is a claim about the step, not about the pair of files in front of it. The question it answers is “if I ran this again, would I get the same answer”. A correctly seeded stochastic step is byte-identical across two runs — that is the success case — and under byte equality alone it was reported deterministic.

That word is false of it. It tells a reader there is no randomness in the step. Move the seed and the bytes move, so the reader has been told something untrue, by the engine, in the report they consulted to decide whether to trust the pipeline. An engine stating something false about data is the one thing this project exists not to do.

So deterministic requires byte equality and a step that declared nothing, and a step that declared itself stochastic saturates at reproducible. A declaration adds no verdict. It takes one out of reach.

“This pipeline is reproducible” is a far stronger claim once the engine refuses to let you write an irreproducible step by accident.

Why the verdict stays on the audit’s own scale

D39 put pinned, replayed and live in a separate RemoteVerdict rather than in Verdict, because the two are not on one scale: a remote’s verdict is about where bytes came from, not about whether a step repeats. The same instinct applied here would give a StochasticVerdict, and would be wrong.

The test is whether the new fact is on the audit’s own axis. “This step draws on randomness” is a statement about whether a step repeats — the very question a Verdict answers. It changes what two runs can tell you about that question rather than answering a different question beside it. A second enum would make a reader combine two words to learn one fact, and would leave Verdict::Deterministic free to go on being wrong, which is what this decision exists to stop.

The line is confirmed from the other side. prov::bundle::Match::Identical needs none of this and was not touched: Match compares one run against a record, a question about two artefacts, to which “identical” is a true answer however the step arrived at it. The two enums differ in subject, not in strictness.

The saturation is not a relabel

Reporting reproducible and leaving the wording alone would make the report lie by a different route. The ordinary text for reproducible is “the bytes differ; what they mean does not”, which is false of a byte-identical seeded step. So the report says what the equality rests on:

reproducible     `permute`
    the same bytes both times   6c636bdb27d9
    canonical form              8a1f52d0c4bb
    because `seed` was held at 20260101, not because the step cannot wobble

And the most useful case is the worst one. A step that names a seed and still disagrees with itself has a declaration that is false — the parameter is not what the tool draws on, because the flag was renamed or a library reads the global generator or a parallel reduction adds in whatever order it finishes. Filing that under ordinary irreproducibility would waste the one thing the declaration bought, so the report names the parameter you believed was controlling the tool and tells you it is not.

Both keywords, or neither

stochastic=True without seed= is the case this decision is named for. The other direction — seed="seed" without stochastic=True — is refused too, and the tempting reading is that it should not be: a user might reasonably mean “determinism by seeding, on a tool you have no opinion about”.

Rejected, because accepting it makes stochastic=True ceremony. Two spellings of one declaration is how a codebase acquires a second code path that nobody tests, and the content of this decision is that a stochastic step declares itself. A declaration with an optional half is not one. So there is exactly one spelling, and the wrong one gets an error that teaches it.

Below the decorator the pair is a single Option<String> — the seed parameter’s name, None for an ordinary step — so “stochastic with no seed” is unrepresentable in the engine rather than merely unreached.

Three further refusals follow from names its seed parameter, all at decoration: a seed= naming a parameter the step does not take, one naming a parameter not typed int, and one naming a data parameter (caught by the same annotation check). The int requirement is not about what can be hashed. A seed that is a float, a timestamp or None is a seed somebody will set differently by accident, and the type is the cheapest place to say so.

The seed itself needs nothing new

CLAUDE.md requires random seeds to be explicit, typed, hashed parameters, and a seed already was one: it is a plain parameter, so it is in the cache key like min_freq, and two seeds are two keys and two results. Nothing was built to arrange that, and this decision deliberately did not move it anywhere special — which is why seed= names a parameter rather than taking a value. The engine is not handed a seed to manage; it is told which of the step’s own parameters already is one. The declaration adds knowledge, not machinery.

The declaration is not in the key either: it says what a parameter is, not what it is set to, so it cannot change a result. Which is not the same as saying that adding it reruns nothing — a step’s source is hashed as written, decorator line included, so editing the decorator moves the key exactly as editing a comment does, and early cutoff stops the rerun travelling downstream. The same has always been true of cite=, whose field is out of the key while the line it is typed on is not.

What the write-up had to change

A methods section must state the seed, and a seed is a plain parameter, so the paragraph already printed it — with seed = 20260101, indistinguishable from min_freq = 0.01. That satisfies the clause’s letter and fails its purpose: a reader cannot tell from it that the step samples, or that this is the number to copy to repeat the run. The sentence therefore says seeded with, in a clause of its own, last, so it ends on that number.

One guard had to give way for it. methods.rs::restating drops a parameter whose value is merely the node’s own scope, so that sample_id = 'SRR8525904' does not appear beside a sentence already naming four samples. That rule was written when every parameter was a setting, and its catch-all is too wide once one parameter is obliged to appear: a seed caught by it would vanish from the paragraph. The declared seed is exempt, and the exemption is principled rather than defensive — a seed never merely restates the scope, which is the only thing that function is for.

No Tree kind

The example this decision is usually written with returns a phylogenetic tree, and no Tree kind was added, because D35 refuses one. A tree’s tip labels are identifiers, on the identifier_namespace axis an EntitySet already claims; its organisms are organism, set-valued since D37; a reference tree’s version is release. There is no axis a Tree brings that nothing has, and what is left of it is Newick with a schema — a parser, which D35 says no to however convenient it would be for an example in a specification.

Nothing here needs a new kind. A stochastic step is orthogonal to what it produces, which is the first evidence that the declaration is in the right place: it is a fact about the step, and it is recorded on the step.

What this does not promise

The engine cannot read a tool, so stochastic=True, seed="seed" on a tool that ignores the flag is accepted. The declaration is a claim, and ligase audit is what tests it. That division is the same one everywhere else here: the pipeline declares, the engine checks what it can, and what cannot be checked at least cannot be left unsaid.

See steps that draw on randomness, declaring one and how the verdict is decided.

D41 — In a ranked list, rank order is the result

Every canonical form in Ligase sorts its records, and the reason is stated once, in D14: the order records came off whatever produced them is not a fact about the biology. canon/report.rs sorts a resistance report’s rows because a report is a set of findings. Alignment records, variant records, intervals, reads and a gene list’s rows all sort for the same reason.

A ranked list is the one kind where the rule runs the other way, and it is the only one.

A ranking of differentially expressed genes, of predicted drug targets, of candidate compounds, is an answer to the question which of these matters most. Rank is simply the row position; there is nothing else for it to be. So the same twelve genes in two orders are two different answers, and a canonical form that sorted them would declare the two answers identical. Downstream, a cache keyed on that form would serve one when the pipeline asked for the other, and would do it in silence — which is the single failure mode semantic hashing exists to be incapable of.

So a RankedList’s canonical form preserves order, which makes it the exact inverse of its own unranked sibling over the very same bytes.

The rule the code has to carry

This decision is unusual in that it is mostly a decision about reading the code. Every other canonicaliser in the module sorts; this one does not; and the two calls that make the difference — Digest::record and Digest::ordered_record — take the same arguments and differ by one word. A later contributor tidying the module for consistency would make the change in one line, it would compile, and every test but one would still pass.

Two things are therefore required rather than merely encouraged. canon/ranked_list.rs says in a comment why it does not sort, at the top of the module where it cannot be missed. And the test that pins it asserts the contrast rather than the bare fact: the_order_of_a_ranked_list_is_its_result hashes the same two files as a ranking and as a set, and asserts that the ranking disagrees and the set agrees — because a reader finding only an assert_ne! between two orderings would reasonably take it for a bug and “fix” the code to match.

What the two kinds share, and what they must not

A RankedList and an EntitySet carry identical dimensions: organism, identifier_namespace, subject and release. They share their loader, because reading a TSV, finding the identifier column, inferring the space off the identifiers and noticing a symbol a spreadsheet has turned into a date is the same work with the same subtleties for both. They do not share their canonical form, because that is the one place they are not the same, and sharing it would mean expressing the difference as a flag somewhere else.

That the difference is confined to one function is itself asserted, by a test over every dimension. If the two kinds ever diverge on an axis, that is where it surfaces — and it would mean the distinction had leaked out of the one place it is meant to live.

The cost, accepted

Keeping order makes the cache stricter than a scientist would. A tool that re-emits a ranking having broken a tie the other way produces a file Ligase calls different and a reader would call the same, and everything below it reruns. There is no tolerance to loosen that with either: a score column is compared exactly, because no drift has been measured for one and a guessed tolerance would be baked into every cache key.

This is the project’s standing bias applied where it costs something: a cache that is too cold costs time, and a cache that is too warm is wrong. The measurement that would justify a tolerance is ligase audit’s, and until it has been taken the exact comparison stands.

Two consequences worth stating

A file cannot say whether its own order is meaningful — that is a fact about what the file is for, and it lives in the annotation of the step that consumes it. So ligase inspect, which has no pipeline to read an annotation from, reads a .tsv as the weaker claim: an EntitySet. Everything it reports is identical for both kinds, so nothing is lost; guessing the stronger one would have the command assert something about a user’s data that the data does not support.

And a ranking mapped across identifier spaces comes out shorter, so its ranks close up: if the top-ranked identifier had no mapping, the second becomes first. That follows from D38’s refusal to carry an unmapped name into a space it is not in, and it is visible rather than silent, because every identifier that did not cross is named and counted in the run report, in provenance, in a bundle and in the methods paragraph (D26).

D42 — A library step ships with its dimensions, its citation and its canonical form

ligase.steps is a curated library, not a convenience wrapper. A step is admitted only when it declares its tool’s version capture, its citation, the dimensions of its output and how that output canonicalises. A step that cannot say what it produces is not a library step.

Why an entrance exam and not a bag of wrappers

Every workflow framework grows a library of wrappers, and they are all the same thing: the tool’s command line with its arguments renamed. Ligase cannot use that. Its whole claim is that it understands what flows between steps — the organism, the build, the coordinate convention, the sample set — and that understanding does not come from the file. It comes from what the steps declare. A wrapper that renames -t to threads has declared nothing, and a pipeline of those is one where the checker has nothing to check and the cache nothing to key on.

So the four declarations are the price of being in the library, and each is checked rather than claimed. ligase.steps.LIBRARY holds them as data so that one parametrised test iterates them, and the half that needs no tool installed runs on every machine — because a criterion that only fails in CI is a criterion nobody is held to while they are writing the step.

What the last sentence has kept out

It is the clause that does the work, and its first two uses were both refusals. Neither is about an obscure tool.

samtools index writes a .bai. Declared Alignment → Alignment that is false twice over: the output is an index rather than an alignment, and the engine would not notice, because canon::hash falls back to hashing the bytes of a file that does not parse as its kind. An index’s bytes would stand in for an alignment’s meaning and everything downstream would accept it. Nor does a new kind help: an index brings no dimension, so under D35 it earns none.

Trimming paired reads — fastp, and every read trimmer — cannot be a step at all. A step’s output is exactly one file and paired reads are two, so Reads → Reads has no honest reading for a pair: one mate inherits pairing = paired-end from its input and is a false dimension the graph then propagates, and an interleaved FASTQ loads as Pairing::Single, which is the same falsehood with extra steps.

Both are one missing engine feature — a step that can produce more than one file — and naming it as a gap is worth more than a step with a false type on it. The architecture page says what closing it would take.

The version capture clause, and why it is first

Because writing it down found the engine breaking its own promise. exec::tool_version ran <tool> --version, read stdout, fell back to stderr, and ignored the exit status. bwa does not understand --version: it prints [main] unrecognized command '--version' and exits non-zero. So any step declaring tool="bwa" recorded that sentence as bwa’s version, in provenance, in the methods paragraph and in the cache key.

The cache consequence is the serious one. That sentence is identical for every bwa ever released, so upgrading the aligner did not move a single key, and every alignment computed by the old one would have been served for the new. Preventing exactly that is the only thing tool= promises.

A tool therefore declares how it is asked — arguments, whether a non-zero exit is expected, and which line of the output holds the version — and a probe that fails unexpectedly, or stops finding its line, is an error rather than a usage banner hashed as a version.

What the clause assumed, and no longer does, is that a tool is a program. D46 admits a third kind: a library, which has nothing to run and whose version the host supplies. The part of this clause that was load-bearing survives unchanged — a non-answer is never hashed as a version — and that is the part bwa taught.

The citation clause is curated, not declared

cite= is how a step you wrote says what it wants cited. A step the project ships does not get to take that liberty on the project’s behalf, so its tools resolve in the engine’s own table instead — which keeps D24’s bargain intact: one promise, kept in one place, and a tool with nothing to cite named as uncited rather than omitted.

The table’s doc comment now says it holds two sets of tools, the reference pipeline’s and the library’s, so that the next person to add a row knows which bar they are clearing.

Four steps is not a shortfall

A long list of half-checked declarations would be worth less than a short list of checked ones, for the same reason the citation table is deliberately small. Four is what the project is prepared to stand behind. @lg.step is not reserved, nothing in the library is privileged, and a step you write can declare everything a library step declares — the difference is who is answerable for it.

D43 — The store is a blob tier with pluggable backends; the index is local and rebuildable

The content-addressed store — step outputs, canonical forms, recorded response bodies — is defined by a small trait: has, get, put, plus the two a filesystem affords and an object store has to emulate (a path for a tool to be handed, and a move rather than a copy through memory). The local filesystem is the default and only backend. An optional one over Apache’s object_store serves S3, every S3-compatible store (MinIO, Cloudflare R2, Backblaze), GCS and Azure, behind a cargo feature so that the default install stays zero-config (D6).

This is the Nix binary-cache model: a local store, always the first tier, and a remote substituter behind it — consulted on a miss, written through on a put, never a source of truth. A lab points every machine at one bucket, one person aligns the cohort, and everybody else’s ligase run hits the cache.

Nothing read from a remote is trusted

Every object is hashed on arrival and refused if it does not match the name it was asked for. A store a lab shares is a store other people can write to, and a wrong answer served quickly from a cache is worse than no cache. Objects are written once and never overwritten, for the same reason: every cache key derived from an object rests on it still being what it was.

Verification lives on the trait rather than in each backend, so a backend author cannot forget it, and the remote is held behind a boxed trait object no caller outside the tiering module can reach — the rule is structural rather than remembered. Promotion into the local tier verifies before it installs, because the local tier is trusted and a corrupt object copied down unverified would be caught once and then never again.

An unreachable remote is a miss, not a failure

Not in the spec, and the design is worse without an answer. A remote holds nothing the engine cannot recompute, so falling back to computing is always correct; failing the run instead would make the engine worse than having no remote at all.

The cost is time, and the one way that cost becomes a bug is a user who is not told. So every fallback is counted and the run report says plainly that work a colleague had already done may have been done again, and that ligase store push sends this run’s objects up once the store is back. A hash mismatch is the other case and is not a miss: it is returned, reported in full with the store that served it, and the step reruns.

The index stays local — and “rebuildable” needed earning

SQLite over object storage is a known way to corrupt a database, so the metadata index stays on local disc.

The spec said it is rebuildable from the store “because everything in it derives from content hashes”. Checked against cache.rs, that is false. A step key is a hash of ingredients — code, parameters, input hashes, tool versions — that no object carries, and remotes is keyed by a locator, which is not a hash of anything. A clean machine pointed at a shared bucket would compute exactly the right key, find the output under exactly the right name, and rerun everything.

So the rows that make a hit possible are published into the store as well, each one an object named for the question it answers: the step entry, the pinned-remote resolution, and the provenance record. An index miss consults them and writes back what it learns, so a rebuild is lazy — one store read per step actually needed, not a scan of the bucket.

The provenance row is the one nothing asked for and the one that makes the feature usable. Without it the headline scenario gives every colleague a run that completes perfectly and a methods paragraph reading “is in the cache from a run that recorded no provenance” for every step: right results that nobody can write up. The hole predates the shared store — delete provenance.sqlite and keep the cache and you get the same message — but the two databases live side by side and are deleted together, so it was unreachable in practice until the cache could be shared.

What a shared store is as trustworthy as

Worth being exact about, because content addressing is easy to over-read.

Hashing an object proves the bytes are the bytes their name claims. It proves nothing about the claim “step K produced object O”, and nothing can, short of running the step — that is what a cache is. A published row is therefore the one object class whose name is not a hash of its contents, and it carries the verification it can: the body restates the table and key it was found under, on every read, and a row restating anything else is refused.

What remains is that a shared store is as trustworthy as write access to it. Someone who can write your bucket can make a step return some other intact object; they cannot serve corrupted data. This is a Nix binary cache’s trust boundary, Nix answers it with signatures, v0.2 has none, and saying so is better than implying a guarantee that is not there. Point Ligase at a bucket your lab writes.

Credentials are never Ligase’s

They come from each provider’s standard chain — environment, instance, ECS-task or EKS-pod role — never from Ligase’s own configuration. A [store] table in ligase.toml takes a URL and nothing else, and a key that looks like a credential is refused by name rather than ignored: ligase.toml is a file people commit, so the one place a user would try to put a secret is the one place that has to say no.

One correction to the claim above: object_store does not read ~/.aws/credentials, so a named profile has to be exported into the environment first. The guide says so rather than the decision implying otherwise.

What the feature costs, and why it is a feature

object_store’s aws, gcp and azure all expand to include aws-lc-rs, whose aws-lc-sys has a non-optional build-dependency on cmake; Cargo features being additive, the crate’s own ring feature cannot switch it off. So enabling the tier means cmake at build time, a second TLS stack beside ring, a second HTTP stack beside ureq, and a tokio runtime — 134 crates against a lock file holding 140.

That is precisely why it is behind a feature, and why the claim is tested: tests/default_build.rs reads cargo tree and fails if any of them reaches a default build. A sentence in a document would say the same thing and would not notice the day someone adds remote-store to default, which is how an install story is actually lost — not by a decision, by a convenience.

A Python extra cannot reach a cargo feature, so ligase-bio[remote-store] is a documented name rather than a mechanism: the build is selected with MATURIN_PEP517_ARGS, and ligase store status reports which kind of build is in front of you.

The local tier grows and does not shrink

No eviction in v0.2. An LRU whose source of truth is a remote turns a cache miss into a network failure mode, and there is no measurement yet to size one with. ligase store gc is a v0.3 item.

D44 — A graph’s sources are read from its edges, not declared

A knowledge graph is the first kind in Ligase that is assembled rather than measured, and the one dimension that follows from that is which evidence sources were folded into it. Add DisGeNET to a graph that had Open Targets and a literature-mined set, and edges appear, neighbourhoods widen and every propagated score moves. Two graphs that differ only in that are two different graphs, and nothing in either file says so — which is my target ranking changed and I cannot find the analysis change, and which is what admits the kind under D35.

The question this decision settles is where the value comes from, and the obvious answer — the user declares it, as they declare organism, subject and release — is wrong here. The sources are in the file. A graph that carries per-edge provenance says on every edge where that edge came from, and the dimension is the union over all of them. Asking the user for it would be the engine demanding something it is holding, and worse: a declaration the engine could have checked and did not is a declaration that drifts from the data it describes.

So the loader reads it: the union of every sources list on a node, on an edge, and at the graph’s top level, recorded as inferred and naming the file, so the reading is checkable. That is the same shape as a gene list’s namespace inference — read where the data settles it — with one difference worth stating: namespace inference is a guess from a pattern and declines whenever it cannot be certain, while this is a reading of a field that means exactly what it says. There is nothing to be uncertain about.

A declaration still wins where there is one. A graph’s own names are sometimes a mirror, a shard, or an internal pipeline stage rather than the source a reader would recognise, and a user correcting that is telling the engine something it cannot read. As with identifier_namespace, a declaration is recorded rather than checked against the file: Ligase cannot tell a user’s better name for a source from a wrong one.

An empty reading is a gap, not an empty set

A graph whose nodes and edges name no sources leaves the axis unknown, with a reason. It is not recorded as a known, empty set, because two known empty sets compare equal, and two graphs that each recorded nothing would then agree on the one axis the kind exists for. That is the false agreement this engine exists to refuse, and it is the same rule D37 already holds for organism=[]: a declaration naming nothing declares nothing.

An empty declaration is refused outright on the Python surface rather than falling back to the file, because somebody who passed source_set=[] has made a mistake worth naming where the message is read.

What a graph is not admitted for

There is no standard knowledge-graph file format, so this milestone defines the node/edge JSON Ligase reads. Defining a shape is explicitly not what earns a kind its place — a kind that is a file format with a schema is a parser, and D35 says Ligase does not need more parsers. The shape is the cheapest envelope that can carry per-edge provenance, and nothing more is claimed for it. Take source_set away and this kind should not exist.

The honest consequence is a limit: a graph from Open Targets’ own Parquet dumps, from Monarch, or from a Neo4j or RDF export is not readable today. That is named on the architecture page rather than implied away, and a reader for one of those formats is a parser that can be written when a pipeline needs it.

D45 — A claim binds to the computation, not to the run that fetched it

A claim is a number a paper states, declared beside the pipeline that produced it, and the engine binds it to the step key and the canonical hash of the output — never to the occasion the write-up happened on.

A key is already the identity of a computation: the same code, the same parameters, the same input contents, the same tool versions. That is the property D22 spent a milestone restoring to provenance, and it is why prov::methods reads rather than recomputes. So a step served from the cache backs its claim completely, and what the binding adds is when that computation happened: the record’s own date and platform, which may be months before the run that wrote the paper up. A bound claim therefore names a computation and the day it happened, not “this run did it”.

The alternative is worth stating because it looks more rigorous and is not. Binding only what the current run recomputed would make ligase run --force the only honest way to write up a paper. A content-addressed cache exists so that a result need not be recomputed to be trusted, and a claim mechanism that did not believe the cache would be arguing with the engine it lives in.

A claim is in no cache key

The second half, and it needs no balancing. A claim in a key would mean that editing a sentence in a manuscript reruns the aligner. Nothing about the computation changed, so nothing downstream may move — and a mechanism whose cost was a re-alignment per draft would not be kept current, which is the worst available outcome for a feature whose entire job is to be current.

The same reasoning is why a claim can never fail check(). check() is about dimensions, before any compute; a claim is a statement about a paper. A pipeline whose write-up names a renamed step is not a pipeline that must refuse to run. It is reported where it is read, and the enforcement is the exit code.

What is checked, and what is taken on trust

M13’s finding, recorded with the step library: a declaration nothing can check is a comment. A Claims section that reprinted the user’s own sentences under a heading would be worse than no section at all, because it would make an unchecked paragraph look checked — the exact failure D24 refuses in the reference list.

So every part of a claim except its text is checked: the step is one the write-up accounts for, the sample is one that step ran on, the parameter or tool it restates is one the record holds, and the value agrees. Five checks, and a claim that fails one is named rather than printed.

The text is taken on trust and cannot be otherwise, which is why the sentence and the value are two fields. A paper writes 1%; the pipeline holds 0.01. Comparing the sentence against the parameter would be a guess about arithmetic in English, wrong in both directions — and the direction that matters is the one where it calls a wrong paper right.

Three verdicts, and why contradicted is not a kind of unbacked

bound, unbacked, contradicted. The third is its own verdict because it is the stronger statement: the claim is bound, to a computation that disagrees with the paper. Both numbers are stated and the write-up does not say which is wrong, because it cannot know whether the manuscript or the pipeline is the thing that should move.

An unbacked or contradicted claim raises ligase methods’ exit code, where an uncited tool does not. That asymmetry is deliberate: an uncited tool is a stated absence with nothing wrong behind it, and an unbacked claim is a number in a paper the analysis does not support. CI is the only place a drifting manuscript is actually caught.

The engine does not read your number out of your file

It does not open a VCF and count its rows to check a reported total. It could do that for the formats it knows and would have to skip the rest, which is the lenient-canonicalisation failure in another costume: some numbers verified and the others silently waved through, with nothing on the page saying which was which.

A claim with a value is compared. A claim without one is bound — to a key, a hash and a date, so a reviewer holding the bundle can find the computation and rerun it. Neither is a verification of arithmetic, and the documentation says so rather than letting a reader assume the stronger thing.

The scope rule does not apply here, and that is the reckoning

D35 asks a kind to bring a dimension. Claim binding is not a kind and brings none, and stretching D35’s letter to cover it would be a category error that happens to be convenient — by that test ligase audit, the bundle, early cutoff and the citation table all fail too.

What it was held to instead is the reasoning behind D35: does it catch a documented silent failure that passes unremarked today, and is the engine already holding what it needs to catch it? It is. D24 names an uncited tool, D26 names a field that cannot be dropped, and a methods Gap names a step nothing is known about — three instances of one promise, that the record names what it cannot account for. A number in a paper was the one thing the record could not account for and never mentioned. This is the fourth instance of an existing promise rather than a new capability, which is why it cost no store, no dimension and no dependency.

D46 — A tool may be a library, and its version comes from the host

tool= has always meant two things at once: the name a citation resolves from (D24), and something the engine runs to ask its version (D42). Those coincide for a program and for nothing else.

M17 ran into the gap. RDKit is software that produced a result and a reviewer will want it cited, and there is nothing on PATH to interrogate — so it could not be a tool= at all. The step declared tool="ligase" and carried the toolkit’s version as an ordinary hashed parameter, which got the version into the cache key, which is the half that matters for correctness. The other half was simply lost: a dependency the analysis rested on, absent from the methods section, with no mechanism that could have put it there. Adding a row to the citation table would not have helped, because nothing would have looked the row up.

So a tool may be a library: software with no program to run, whose version only the host knows.

Why the host supplies it rather than the engine asking

Everywhere else the engine asks, and that is deliberate — a version a caller hands over is a version a caller can get wrong, and D42’s version clause exists because a wrong version is worse than a missing one.

A library is the case where asking is not available and not better. The library lives in the interpreter that is driving the pipeline, which is the exact copy of it the step will call. Reading rdkit.__version__ there is a more accurate answer than execing something of that name would be — there is nothing of that name to exec, and if there were, it would be a version of the wrong thing. The engine therefore refuses to run a library even where a program shares its name, which is asserted rather than assumed.

What is kept from D42 is the part that was load-bearing: a non-answer is never hashed as a version. A library nothing has registered is an error naming the library and what to install, not an empty string. That is bwa’s lesson in another costume — a non-answer is identical for every release, so recording one would mean upgrading the library moved no cache key at all.

Registering is refused on a conflict rather than resolved

A library’s version is in every cache key beneath it, so two answers inside one process would mean two keys for one computation. Reading a library’s own version twice cannot disagree, so a conflict is a bug rather than a situation — which is why it is refused with both values named, instead of the second quietly winning.

Where this does not reach

The registration has to happen before a key is computed, and a graph is built from decorated functions without executing them — so nothing inside a step body runs early enough. ligase.chem registers from the function that builds the step, which is a wrapper above it, and that is the one piece of ceremony this decision costs. A step whose library nobody registered fails with the message above rather than silently.

It also does not make the engine able to find a library. ligase.chem knows RDKit is there because it imported it; the engine knows only what it was told. A tool that is neither a program, nor carried by the engine, nor registered by a host is still a tool with no version, and still an error.

D47 — A molecule’s identity is its standardisation, not its structure

An InChIKey is a hash of a molecular structure. Standardisation changes the structure on purpose — that is what it is for — so it changes the key. Nothing about the key says which standardisation produced it.

That would be a curiosity if it were rare. It is not. Measured across six RDKit releases spanning four years, with thirty input structures in six standardisation states each, twelve of the thirty get a different InChIKey under a different state. The thirty were marketed drugs, salt forms and tautomer pairs, not cases chosen to make a point. Aspirin, arriving as its sodium salt the way a supplier ships it, has three identities:

The last two differ in the final character, which is the charge block, and the third is the key plain aspirin gets.

So representation is a dimension: an identity, and the operations applied before the names were computed. Two sets whose representations disagree do not join.

Why this is the strongest case for a dimension the project has had

D38’s failure is visible. TP53 is not ENSG00000141510, and anybody who looks at two columns sees it in a second; the dimension catches something a careful person would also have caught, faster and more reliably.

This one is invisible. Both files are columns of twenty-seven-character InChIKeys, in one format, from one piece of software. There is no glance that distinguishes them, and no amount of care substitutes for being told. A check that catches only what inspection would also catch is a convenience; a check that catches what inspection cannot is why the engine exists.

The overlap it produces is also the worst possible size. The unaffected compounds still match, so two sets that differ in standardisation overlap in roughly the fraction of their compounds that have no salt, no charge and no tautomer — tens of per cent, which is exactly the range people attribute to assay noise, library composition or a real finding. A zero would have been a gift.

The failure is documented, not conceivable

D35 requires a documented silent failure. According to PubMed:

• Fourches, Muratov & Tropsha, Trust, but verify: on the importance of chemical structure curation in cheminformatics and QSAR modeling research, J Chem Inf Model 50:1189–1204 (2010), DOI . Names the exact operations this axis records — removal of counterions, salts and mixtures, normalisation of chemotypes, curation of tautomeric forms, deletion of duplicates — as mandatory steps, with case studies where performing them is what made the modelling work.
• Fourches, Muratov & Tropsha, Trust, but Verify II: A Practical Guide to Chemogenomics Data Curation, J Chem Inf Model 56:1243–1252 (2016), DOI .
• Bento et al., An open source chemical structure curation pipeline using RDKit, J Cheminform 12:51 (2020), DOI . ChEMBL’s own standardiser, whose stated reason for existing is this failure: incoming compounds are not standardised consistently, and standardisation is what makes it possible to compare and integrate data on the same compound from different sources.

The third settles it. A database of nearly two million compounds built a three-stage standardiser because comparing compounds across sources does not otherwise work. That is not a conceivable mismatch; it is infrastructure somebody built to avoid one.

Why the identity is part of the value

Because the alternative moves. In the same six releases that left every InChIKey untouched, RDKit’s canonical SMILES for cisplatin changed:

2022.09.5-2024.09.6  Cl[Pt]Cl.N.N
2025.09.6-2026.09.1  N.N.[Cl][Pt][Cl]

Its InChIKey, LXZZYRPGZAFOLE-UHFFFAOYSA-L, did not move at all. A set named by structure strings therefore has an identity its own toolkit can change under it, and must not compare equal to one named by keys.

This is also, measured, why §4.4’s instruction to hash InChIKeys rather than SMILES was right. A canonical form over SMILES would have moved the content hash of every file holding that compound on an upgrade that changed no chemistry.

Why the operations are a closed set

CoordinateSystem’s grounds: a value the engine cannot render is a value it cannot explain. Four operations are recognised — salt-stripped, tautomer-normalised, charge-neutralised, stereo-discarded — and a file naming a fifth is a format error naming the four, rather than a state carried along unread. A canonicaliser or a loader that reads the parts it understands and skips the rest is the lenient kind canon’s own module comment calls dangerous.

They are also phrased as what was applied, so an absent operation means it was not done. That polarity is what makes stereo-discarded the name rather than stereo-kept: the dangerous state is the one that lost something, so it is the one with a name.

Stereochemistry is observed, and the check is asymmetric

§4.4 describes the standardisation state as “salt-stripped, tautomer-normalised, charge-neutralised, stereo kept”. Stereo is not kept. RDKit’s tautomer canonicaliser removes stereochemistry at an sp3 centre inside a tautomeric system, and a carboxylic acid’s alpha-carbon is inside one, so ibuprofen’s two enantiomers both arrive as HEFNNWSXXWATRW-UHFFFAOYSA-N. Two enantiomers are routinely one active compound and one inactive one.

A user asking for tautomer normalisation cannot be expected to predict that. So the standardiser checks its own work, and the two directions are not treated alike, because they are not equally dangerous:

• stereo discarded and not declared is an error, naming the count. Two sets would otherwise agree while one of them has lost the difference between an active enantiomer and an inactive one, which is the false agreement this engine exists to refuse.
• declared and nothing discarded is a note. It can only cause a refusal a user fixes by declaring one thing consistently.

That is the asymmetry M10a settled for case folding, applied to a second axis: a false agreement is unrecoverable, and a loud refusal is not.

And a declared representation is recorded, not verified

As a declared namespace is (D38), and for a sharper reason: the engine has no chemistry (D49), so it could not check the claim if it wanted to.

That sits beside M13’s finding, which D45 states as a rule: a declaration nothing can check is a comment. It is not a contradiction, and the difference is worth being precise about. A claim’s text is the whole of a claim, so a Claims section that only reprinted it would be a comment dressed as a check. A representation is one of two statements about a molecule table, and the other one — what the standardiser actually did — is checked, by the step that did it. Nor is the declaration the normal case: the normal case is a line the standardiser wrote, read off the file, which is D44’s habit on a second axis. A declaration is the fallback for a table somebody else prepared, and it is better than the alternative, which is an axis unusable for exactly the in-house sets that most need it. The step that does the standardising checks its own work, which is the half that can be checked; nothing checks a user’s claim about a table somebody else prepared. That limit is named on the architecture page rather than smoothed over.

D48 — A kind may also earn its place by its canonical form

D35 says a kind earns its place only by bringing a dimension whose mismatch is a documented silent failure. Written that way, the rule does not admit RankedList, which the project shipped in M12: a ranked list carries exactly the dimensions an EntitySet carries, and the catalogue says so in as many words.

So either M12 was outside the rule, or the rule has a second limb nobody wrote down. It has one, and this is where it is written down, because MoleculeRanking is the second kind to need it.

This is the second reading of D35’s limits to be recorded, and the two are about different things. D45 observes that D35 asks a kind to bring a dimension, so it does not apply to a feature that is not a kind — by its letter ligase audit, the bundle and the citation table would all fail. This decision is about the kinds the rule does govern, where the letter was too narrow for one the project had already shipped.

A kind also earns its place if it brings a canonical form whose absence makes the cache serve a wrong answer.

That is not a loosening of D35. It is the same standard — a silent failure, plausible and attributable to anything but bookkeeping — relocated from the data to the engine, and D41 already states it in full for the one case that existed: a ranking whose canonical form sorted its rows would declare two different answers identical, and a cache keyed on that form would serve one when the pipeline asked for the other, in silence. A ranking of ten thousand docked compounds is that failure with more at stake than a ranking of genes, because the order decides the length of the prefix anybody actually buys.

What the limb does not admit

It is narrower than it looks, and the narrowness is the point.

A kind qualifies only where the same bytes, read as an existing kind, would hash to something that means a different thing. That is why it admits a ranked sibling and nothing else so far: MoleculeRanking over a compound table and RankedList over a gene list are each the exact inverse of an existing kind’s canonical form over the very same file, and there is no way to express that as a flag without putting the difference somewhere it does not live.

It does not admit a kind that merely wants a different parser, a different set of columns, or a tidier name. A kind whose canonical form would be some other kind’s with extra fields hashed is that other kind, and the extra fields are columns. The question to ask is not “does this want its own canonical form?” but “would sharing one make the cache return the wrong answer?”, and for everything the project has declined so far the answer is no.

Which is also why BindingScores was renamed

§4.4 of the v0.2 design asked for a BindingScores kind. Examined axis by axis it brings nothing: its target is subject, its compounds carry representation, its collection’s version is release, its scoring function and software are provenance that already enter the cache key (D32, D42), and its score column is a column compared exactly with no tolerance (D41). It is admitted on this limb and on nothing else.

And it is named MoleculeRanking, because the same file shape holds docking scores, QSAR-predicted activities, predicted solubilities and synthetic-accessibility scores, and only the first is a binding score. A kind named for one of the things it holds is a kind whose dimension table and error messages describe something the user is not holding — the objection load::named_list already records against sharing wording between kinds.

D49 — The engine knows no chemistry

Computing an InChIKey needs a cheminformatics toolkit. The engine does not have one, will not link one, and does not need one.

What the engine does with chemistry is compare two representation values and refuse to connect two molecule tables that disagree. That is string work. It needs no structure parsing, no ring perception and no hashing of molecular graphs — so loading a compound table, reading its representation, hashing it, early cutoff, the cache and every refusal all work on an install with no toolkit at all.

What needs RDKit is producing a set whose standardisation is known, and that is one function in one Python module.

Why not in the core

The core would have to link an InChI library, which is C. D6’s zero-config install and D43’s keep-cmake-out rule exist to keep pip install ligase-bio free of a toolchain beyond Rust, and tests/default_build.rs is what defends that claim against a convenience. A chemistry dependency in the core would end it for every user, genomics ones included, to serve the subset doing chemistry.

Why the extra is not empty, unlike the other one

The remote-store extra adds no Python package, because there is none to add: its backend is behind a cargo feature and a Python extra cannot reach one. RDKit is a Python package, so chem = ["rdkit>=2024.3.1"] does the thing an extra is for. The two extras look alike and are not: one is a documented name for a build you have to make yourself, and this one installs something.

The floor is measured rather than guessed. InChIKeys were identical across 2022.09.5, 2023.09.6, 2024.09.6, 2025.09.6, 2026.03.6 and 2026.09.1, so any of those would serve on the chemistry; 2024.3 is the floor because it is the first series with NumPy 2 wheels, and the two series below it need numpy<2 pinned somewhere an extra has no way to say.

And the claim is proved rather than stated

A sentence in the docs would say all of the above and would not notice the day somebody moves import rdkit to the top of the module. So the division is asserted: a subprocess imports ligase and checks that no entry of sys.modules names RDKit, a default Rust build loads two compound tables and refuses the join between them, and the error for a missing toolkit is asserted to name pip install 'ligase-bio[chem]' and to say what works without it — the precedent being ligase store status, which tells a user which build they have rather than leaving them to the release notes.

The docs site builds with no toolkit either. A site that could not be built without the optional extra would have made it mandatory in everything but name.

D50 — Standardisation is a conversion, so its failures are named

lg.chem.standardize is registered in the conversion registry rather than being an ordinary step that happens to write molecules.

It fits the shape D38 built: it crosses exactly one axis, it reads its destination from the same to= every conversion reads, and — the clause that matters — it is an operation that can fail for some inputs. A structure that does not parse, a mixture with no single parent, a molecule the standardiser rejects. convert’s own module comment draws the line: “a conversion that could not fail would be a rename, and a rename across a dimension is what D38 forbids.”

So its failures follow the contract map_identifiers already keeps (D26): a molecule that does not standardise is left out of the output and named, twenty individually with the rest carried by a count, in the run report, in provenance, in a bundle and in the methods paragraph. The output being smaller than the input is a recorded fact rather than a discrepancy somebody notices six months later.

What registering it proved about the registry

D5 wanted the registry shaped so that liftover and rename_contigs would be rows rather than redesigns. This is the first evidence either way, and it is better evidence than another row of the same kind would have been: standardize’s body is Python, because the engine has no chemistry, where map_identifiers’ is Rust.

Nothing in the registry had to change. An entry is metadata about an axis — which axis, which kinds, what the user supplies — and who runs the crossing is the caller’s business. That the registry took a Python-bodied entry without a change of shape is worth knowing before the next two arrive.

The version capture, which could not be a tool=

D42 has a step declare tool= and the engine probe <tool> --version, so that an upgrade reruns the steps it could have changed. RDKit has no executable to probe: it is a library.

So the version is read before the step is built and passed as one of its parameters, which puts it in the cache key. Read inside the body it would not be, and the step would serve a cached answer across an upgrade that could have changed every key it computed — which is precisely the failure D42’s version clause was written for, where bwa --version printed a complaint identical for every release ever shipped and upgrading the aligner moved no key at all.

D51 — A worker claims work through the store; Ligase submits no jobs

Cluster execution is ligase run --worker[=NAME]. A worker builds the whole graph, as every run does, claims each step through the shared store, leaves alone whatever another worker holds, and takes the rest from the store. N of them, started however a group starts things — a Slurm array, a loop over ssh, a container per node — converge on one complete run and exit. Then one ordinary ligase run reports the whole graph and writes it up.

Ligase submits nothing. Three readings of “cluster execution” were available and two were rejected.

A local worker pool over the graph’s independent nodes is M3, and has been since M3.

Submission to a scheduler is the obvious reading and the wrong boundary. A site’s Slurm is its own: partitions, accounts, QoS, time limits, a module system, a scratch convention, a container story. A submitter is therefore either wrong at most sites or endlessly configurable at all of them, and either way Ligase would own job lifecycle — submission, polling, requeue, cancellation, log scraping — which is the scheduler’s job and is where workflow tools accumulate their weight.

It also mistakes what a research group is missing. A group that has a cluster already knows how to start twenty of something. What they do not have is twenty processes that can divide one pipeline without a coordinator — and that is the gap content addressing closes, which no tool that moves files can claim.

So the deployment is N workers plus one collector, and the collector is a command the user already knows. It serves the two things a research group actually has: a Slurm cluster, where the store is a bucket or a directory on Lustre; and a few workstations, where it is a bucket.

--jobs is for one machine and --worker is for several. They compose and are not alternatives: twenty workers with eight jobs each. Running eight workers on one host pays store round-trips for what a mutex settles for free.

D52 — Correctness needs no lock; the canonical form is what makes it safe

Two workers that compute the same step key agreed on the code, the parameters, the inputs’ canonical hashes and the tool versions (the key). There is no sense in which they are doing different work. Objects and published rows are write-once, so whoever publishes first has published an answer the other would have agreed with, and the loser adopts it.

A lock would prevent the duplicated computation; it is not what makes the result right. The result is right because the key says what the work was. So the lease (D53) is an efficiency device, and the engine is correct with it mis-tuned, racing or switched off. That is the property worth having on a machine nobody is watching at three in the morning: the thing that can break is the thing whose breakage costs CPU.

The canonical form is what carries the argument

Two workers almost never produce the same bytes. minimap2 writes its command line into @PG, that command line holds the output path, and the path is the staging file named after the step’s own key — so a step’s bytes move whenever its key does. It does not matter, because a downstream key is built on the hash of what an output means (D4).

The property M4 introduced to make early cutoff earnable is therefore the property that makes lock-free distribution safe. That is the same observation used twice rather than a coincidence: what a step produces is what its output means.

Where it breaks, and what the engine does there

The argument needs the two canonical forms to agree. Two cases break it:

• a kind whose canonical form does not fix its result, which is the residual ligase audit exists to measure;
• a seeded stochastic step whose tool is stochastic in more than its seed. D40 is necessary and not sufficient: IQ-TREE, RAxML and BEAST are reproducible given a seed and a thread count, and two cluster nodes with different core counts are the normal case rather than the exception.

A steps row therefore carries the output’s canonical hash, so the worker whose publish was dropped learns for free whether the row it lost to means what it computed. Unequal goes on the run report, naming the step and both hashes; the published answer is the one in force on every machine, so nothing below the step disagrees between workers.

Reported on the run report and not as a Note: a note is recorded with the cache entry and read back on every later hit, because a note is a fact about the output, and a divergence is a fact about this run on this worker. Persisting it would attach it to a key for good.

Pinning such a step to one worker was rejected. It sounds safer and is worse: the answer would depend on how many workers happened to be running, and a real property of the pipeline would be hidden instead of reported.

A format version belongs in the name of the thing it versions

Found while bumping ROW_VERSION for the canonical-hash field, and latent since M14. published::key_of hashed the domain, the table and the key — and not the version — so every format competed for one object address. Objects are write-once, so a bucket holding a version 1 row would have refused every version 2 row at that key, for as long as the bucket existed: the put is dropped, the read finds a version the build will not parse, the fetch answers nothing, and that step key becomes permanently unpublishable by anyone. Not one recomputation, which is what the format’s doc comment promises, but all of them for ever.

The version is in the address now. A format version that is not in the name of the thing it versions can only be discovered by the reader, which in a write-once store is already too late.

D53 — A lease is advisory, and expiry does not have to be right

Workers divide work with a lease: a published row saying which worker is computing which step key, held until a deadline.

Advisory, in those words. The store affords write-once blobs, no listing and no compare-and-swap. A lock built on that fails silently under exactly the conditions it exists for, so nothing pretends to be one. By D52 it does not need to be.

Which is what lets expiry be approximate, and that is the point. A worker that dies holding leases needs no recovery protocol, no heartbeat to miss, no fencing token and no coordinator to notice: its deadline passes and an idle worker computes the step. Taking a step somebody may still be working on is safe, so the decision about when to take it is allowed to be a guess.

Resumability falls out for nothing. Every answer that landed is in the store under a name derived from the work rather than from the run, so starting again is resuming.

One knob. --lease, default an hour, and the rule is to set it above your longest step. Both ways of getting it wrong are costs rather than wrong answers: too short and a worker duplicates a step whose holder is still working; too long and a worker that vanished keeps others waiting until its deadline. A worker always prefers unclaimed ready work, so “steal only when idle” is a consequence of the ready queue rather than a policy, and there is no second flag for it.

Generations, because a row cannot be deleted or rewritten. Rows live at <step key>/<generation> and the highest occupied one is the state of the key. Taking a free key publishes the next generation; giving it back publishes the next generation with a deadline already past, because a release is just a lease nobody holds — which is how a failed step hands its key back at once without a second object class. Extension would publish the next generation with a deadline further out; it is not built, and the format carries it, so adding it later is not a format change.

A lease is never cached locally, and this is the one place the store’s central premise does not hold. Everything else in store/ rests on a local copy being as good as the remote’s, because an object’s name fixes its contents. A lease is not a fact about a key: it is a fact about the other machines, and it changes without anything local being told. Written through the tiering, each worker’s claim landed in its own local tier and every read was served from it, so every worker saw only its own claims, found every key free, and computed the whole graph. The feature was silently absent and the runs were still correct — D52 being true — which is why this needed a test rather than a reading.

D54 — Two workers never share a cache directory

--worker=NAME puts its cache in <cache>/workers/NAME by default.

The index is SQLite, D43 keeps it local, and SQLite over a network filesystem is the failure D43 names. A Slurm array whose twenty tasks all default to one $HOME/.ligase walks into it without having chosen to — so the right thing is the default rather than merely the documented thing. An explicit --cache is still honoured, with a lock file refusing a second worker in one directory and naming the holder.

A stale lock after a crash blocks that directory, and the error says to delete it. A real wart, accepted because the alternatives are a dependency on flock — unreliable on exactly the network filesystems being guarded — or a liveness probe that is not portable, and because a stale lock costs a deletion rather than a wrong answer. It is only reachable with --cache set by hand.

A worker with no shared store is refused before anything runs. It has nobody to divide work with, and would compute the whole pipeline alone and look like a success.

And a worker’s name is supplied rather than discovered. There is no standard-library hostname call, and taking one would have cost a dependency for a string the user is better placed to choose: --worker=$SLURM_ARRAY_TASK_ID says something a hostname does not. An unnamed worker gets a short token from the clock, the process and a counter — enough to tell two apart, and honest about saying nothing else.

D55 — A methods paragraph names every platform and engine a run used

prov::methods took the engine version and the platform from the first group’s record and wrote “Analyses were run with Ligase X on Y.” Every record has always carried its own, and before cluster execution they were all equal because one machine ran everything.

A distributed run is the first time they are not: a mixed cluster, an ARM workstation beside x86 nodes, or one worker on an older wheel. The paragraph then named one platform for a run that spanned several — in a methods section, which is the one place in this project where a quiet half-truth costs the most. Methods carries every distinct engine and platform now, and the sentence names them all; the reproducibility bundle gives each one its own row, because environment.tsv is a table and a run that spanned two platforms has two facts to state rather than one sentence about both.

This is a public API change and it is the clause this milestone would have been least honest to skip. A tool whose argument is reproducibility cannot be vague about where a result was computed.

D56 — A shape with no marker is not a namespace

identifier_namespace is the one dimension the engine can usually read without being told, and D38 already lists three cases where it declines to guess. This is the fourth, and it was measured against the published 0.3.0 wheel by a consumer rather than found here.

Six of the seven shapes in the catalogue carry a marker: a prefix (ENS, CHEMBL), an underscore (NM_), a colon (HGNC:), digits throughout, or digits in fixed positions at a fixed length. HGNC symbol carries none. It is an upper-case letter followed by letters, digits and hyphens, which is what an approved symbol looks like — and equally what ChEMBL’s component synonyms, a consortium’s internal names and most in-house panels look like.

So the file below infers HGNC symbol and is not in that space at all:

id	score
DHFR-TS	0.90
PPPK-DHPS	0.82

DHFR-TS is P. falciparum’s bifunctional dihydrofolate reductase–thymidylate synthase, named in ChEMBL’s component-synonym space. PPPK-DHPS is the same story. MT-CYB, HLA-A and NAT8B-AS1 are real HGNC symbols of the same shape, so no matcher separates them — this is not a pattern that wants improving.

Why a wrong value here is worse than no value

A wrong guess would be bad enough on one side. The failure is that it is made identically on both: two files of symbol-shaped identifiers each inferred HGNC symbol, the values compared equal, and the join went ahead. ligase check printed no problems found and exited 0 for a pipeline scoring a ranking against a truth set across two identifier spaces — which is the exact defect this dimension was added to catch, in its purest form, produced by the engine rather than by the user.

The consumer who measured it put the cost at 0 of 7 entities recovered undeclared, against 2 of 7 after an explicit mapping step. Neither number was reported by the undeclared pipeline. It simply ran.

What the engine does instead

KnownNamespace::infer answers Inference::Indistinguishable wherever every identifier fits one shape and that shape carries no marker. The dimension is then unknown with a reason, ligase inspect says so in the language it already has for an unknown dimension, and a join of two such lists is refused by the demand required_pair already makes:

The six marker-bearing shapes infer exactly as they did. A declaration is still recorded and not verified, so identifier_namespace="HGNC symbol" is accepted on the user’s word — which is the whole remedy, and is one argument.

Why not keep the guess and mark it

Two weaker options were considered, both proposed in the report. Marking the value — printing HGNC symbol (guessed) — leaves a value in the field, and a value in the field is what check compares; the join would still pass unless equality were also taught to distinguish a guess from a fact, which is two mechanisms where one will do. Refusing only guess-against-guess keeps equality honest but leaves inspect printing a space the file never stated, and leaves a guess satisfying a comparison against a declared value — so a correct declaration on one side would be validated by a guess on the other.

Neither is wrong, and both are smaller changes. What decides it is that this engine’s answer to a question it cannot settle is to say so, and the identifiers genuinely do not settle this one. An unknown dimension is a state the whole system already handles: it has a reason, inspect prints it, check demands it where it matters, and the remedy is named.

The cost, which is real

A list of bare gene symbols no longer states its own space, so pipelines that joined two such lists without declaring now refuse until one argument is added to each load. That is a behaviour change in the direction this project always takes it — a refusal before anything runs, with the fix in the message — but it is a change that breaks working pipelines, and it is the reason D57 asks for it to be announced where a consumer choosing a pin will see it rather than in a changelog body. It is listed in MIGRATING.md .

The project’s own fixtures felt it: sixteen Python tests and nine Rust tests declared a space they had been getting for free, and the four generated captures that build a pipeline over a symbol list now declare one. That is the migration a consumer performs, performed here first.

D58 — The engine cites itself, because a gap has to be closable

D24 names a tool with no reference rather than omitting it, so that a short reference list cannot be mistaken for a complete one. ligase was one of those tools, and it was the one case where the mechanism turned against itself.

A consumer reported it: every ligase methods write-up ended

No reference is recorded for ligase. Add one with `@lg.step(cite=...)`, or
cite it by hand.

even with a reference for Ligase declared on their own step. The uncited tool was the ligase of map_ranked_identifiers — a built-in conversion, which is a step the engine owns and which takes no decorator. There was no @lg.step for the advice to be applied to.

Why that is worse than an omission

MethodsGap works because it is unignorable. A gap that cannot be closed trains a reader to ignore it, and that costs the mechanism for every gap that is real. In the reporting consumer’s pipeline it was the only remaining gap, so the last word of their methods output was a warning about something they had already done.

So the rule: a gap the engine reports must name a remedy the user can apply. Where it cannot, the engine owes the answer itself.

What it does now

cite::engine builds the engine’s own reference from the versions the run recorded, and Methods::citations puts it in the reference list of every write-up and never in the uncited list:

Njeri K. Ligase: dimension-checked, reproducible computational biology
    pipelines (version 0.3.0). https://github.com/Neoforge-Labs/ligase

Three details, each deliberate. It is not a row in cite::KNOWN, because every row there is a constant and this reference has to name the version that ran — which a distributed run may have more than one of (D55), and which the table cannot hold. It carries no DOI, because this project has none; the repository is what a reader is pointed at, in the form RDKit’s entry already set for software with no paper, and a DOI-shaped string invented to make the row look like its neighbours is the plausible wrongness D24 refuses. And it is added for the run rather than per step, so two steps running the engine do not make two references and a step declaring its own does not make the engine’s disappear.

Why it is in every write-up, not only in those with a built-in step

The engine always ran. Methods::paragraph opens by naming it and the version it was — “Analyses were run with Ligase 0.3.0 on macos-aarch64” — so a reference list that left it out was missing one of the pieces of software the sentence above it credits. A user who was citing it by hand can stop.

D59 — A claim is cited by its output digest, not by its step key

D45 settled what a claim binds to: the step key, because the key is the identity of a computation and a step served from the cache therefore backs its claim completely. What it did not settle is what a claim is cited by, and ligase methods printed the key — which is the one identifier that must not go in a manuscript.

A consumer measured it. One pipeline, one claim, cite=(...) added to the step and nothing else changed:

"malaria F1 = 0.333 …" rests on `validate`, at key 25c2762d4d8766aa, …
"malaria F1 = 0.333 …" rests on `validate`, at key b4884fc9df9aa706, …

Nothing reran — the run report says cached for both steps, which is D45’s other half working exactly as promised, since cite never joins a cache key. The artifact was byte-identical. Both provenance rows survived in the store under the two keys, carrying the same output digest and the same canonical digest. So the identifier the paper would have quoted stopped resolving for an edit that provably cannot change the number.

Why the key moves, and why that is correct

The key is a hash of the step’s code, its parameters, its input content and its tool versions. A citation is part of the step’s declaration, so it is part of the code that was hashed — and that is the right behaviour for a key, whose job is to identify a computation exactly. The defect is not that the key moved. It is that the key was offered as a citable identifier.

What is printed instead

The digest of the output’s canonical form, twelve characters, as the rest of the engine already prints a content hash:

"variants were called above 1% allele frequency" rests on `call`, where
    min_freq = 0.01 as this claim states, canonical 4a366b6cfeb0, recorded
    30 October 2026 on macos-aarch64.

The canonical digest rather than the raw bytes’, although the report asked for the bytes’. The canonical form is what D4 makes “scientifically identical” mean; it is what ligase audit compares two runs by, and what a bundle’s verify checks a rerun against. It is therefore the digest that still resolves when a rerun on another machine writes a different timestamp into a header — and a manuscript’s identifier wants every bit of that stability.

Why the key is no longer in the sentence

Printing both was the report’s first suggestion and the smaller change. It was declined for one reason: a reader who is offered two identifiers will sometimes quote the wrong one, and the whole finding here is that one of them is unsafe to quote. A write-up that names only the stable identifier cannot be misread.

The key is not lost. It is in provenance, in the bundle’s claims.tsv beside the canonical digest, and on bound.key for anything that wants to look a computation up — which is where an internal identifier belongs.

The multi-sample sentence changes with it

A claim naming no sample binds to one computation per sample, and the sentence listed every key under Keys: because naming beats counting (D15). It lists every digest under Outputs: now, for the same reason and with the same per-sample resolution: a reviewer checking one sample needs that sample’s identifier, and it has to be the one that still means something next year.

Both words were output when this decision shipped, which was wrong about the one thing the decision is for; they are canonical and Canonical digests: since D64.

D60 — A one-to-many crossing is declared, and a ranking does not survive one

D38 leaves an identifier the mapping table sends to several targets out of the output and names it, on the grounds that emitting a row for each would make the output larger than the input without saying so, and choosing one of them would be a guess about which the analysis meant. Both halves of that are still true. What it assumed is that the ambiguity is a data-quality problem, and for a real class of entity it is not.

A consumer put it precisely: the remedy “decide which target you mean” assumes the ambiguity is a data-quality problem. Often it is. But some of the time the identifier genuinely denotes more than one thing, and picking one is the wrong answer.

• Bifunctional enzymes. P. falciparum DHFR-TS is a single polypeptide with two catalytic domains, corresponding to human DHFR and TYMS. PPPK-DHPS is the same story. Both targets are correct; these are not curation errors, they are the organism.
• Protein complexes named as a unit, mapping to their subunit genes.
• Paralogue families where a non-human identifier has no one-to-one human orthologue but a defensible set of them.

And “keep the identifier in the space it is already in” does not resolve it either, because the reason to cross at all is to join against something in the other space: an identifier left unmapped is absent from the join, which is the same outcome as dropping it, arrived at deliberately. In the reporting pipeline the top-ranked entity was exactly this case, so the entity the result was most about was the one that could not cross.

What is added, and why it is declared rather than defaulted

map_identifiers takes ambiguous, which is drop — the behaviour above, unchanged — or fan-out, which emits a row per target and says so:

noticed   `map_identifiers`: the identifier `NKX2-1` maps to more than one
          target, and this step asked for a fan-out, so the output holds a
          row for each
noticed   `map_identifiers`: 1 identifier fanned out into 2 rows, so the
          output is 1 entity larger than the input

Declared at the call site, never inferred. The engine cannot tell an untidy mapping table from a bifunctional enzyme, and the whole value of the note is that it says you asked for this rather than the engine decided which biology this was. It is an ordinary step parameter, so it joins the cache key: changing the answer about the biology reruns the step, which it must, because the output is different.

Two rules are not suspended by a fan-out. A target that two identifiers arrive at is still left out, because the output is a set and one entity named twice with two sets of attributes is not one — a fanned identifier can collide exactly as a one-to-one one can, and the collision pass counts fanned targets for that reason. And an identifier the table has no row for still does not cross.

A ranking does not survive it

D41 says rank is the row position; there is nothing else for it to be. One row becoming two therefore has no defensible rank, and all three ways of pretending otherwise are worse than refusing:

• tie ranks invent an order the pipeline never produced;
• renumbering makes rank n of the output mean something no computation assigned;
• keeping the fanned entity at its rank and pushing the rest down does both at once.

So map_ranked_identifiers has no ambiguous argument and goes on dropping, and fan_out_ranked_identifiers is a step of its own whose return type is an EntitySet. The ranking did not survive the crossing, and saying so is more honest than handing back something shaped like a ranking.

That it is a separate function rather than an argument is forced as well as preferred: a step’s output kind is read off its return annotation once, when it is decorated, so one function cannot produce both kinds. The same constraint split map_ranked_identifiers from map_identifiers in M12, and the function’s name is now where the declaration lives.

The practical advice that follows is the one D41 already gives about cut-offs, pointed the other way: take a top ten before fanning. The top ten of a fanned set is not a thing; the fan-out of the top ten is.

D61 — A claim may name where its number lives

D45 binds a sentence to a computation, and M13’s finding — a declaration nothing can check is a comment — made the engine check every part of that declaration except the sentence itself. One part was never checkable: the number. lg.Claim could say that 0.333 was stated and that validate produced something; it could not say that 0.333 was what validate produced, and the guide said so plainly.

That is the gap that makes published numbers wrong in practice. They go wrong not because anyone falsified anything, but because the pipeline was rerun after the table was pasted in, or the table was copied from a run with different inputs — and nothing in the toolchain was in a position to notice.

So a claim may name where its number lives:

lg.Claim(
    "malaria F1 = 0.333 against seven curated targets",
    step="validate",
    value=0.333,
    at=lg.cell(column="f1", row=0),
    tolerance=5e-4,
)

And when the artifact moves and the manuscript does not:

ligase methods exits non-zero on that, because a refuted number is not a number the write-up stands behind — the same rule accounted_for already applied to an unbacked claim.

Four constraints, each a refusal

Only a declared location. The engine never goes looking for a number in output it was not pointed at, and never parses a format it was not told. One location kind exists — a cell of a TSV with a header, by column name and row position — and Located is an enum so that a second is a variant rather than a redesign. A claim that cannot say where its number is stays step-only, which is a legitimate state and not an error.

The tolerance is the consumer’s, declared. A claim to three decimals and a claim to machine precision are different claims, and a default would invent a precision nobody stated. at without tolerance is refused where it is written, as is tolerance without at.

Not a statistics engine. No recomputation, and no checking that an F1 is a well-formed F1. Just: the number you published is the number in the bytes.

An unreadable location is unbacked, not contradicted. A column that is not there says nothing about whether the sentence is true, so the write-up reports that it could not check rather than that the claim is wrong — and names what was there, because the columns the output does have is what tells a wrong column from a wrong row.

Why verified is a fourth verdict

bound says the sentence rests on a computation this write-up accounts for. verified says the number in the sentence is the number in the bytes. The second is strictly stronger, and collapsing them would throw away the whole of what a located claim buys — a reader could not tell a claim the engine checked from one it merely bound. is_bound is true for both, because verified is bound and more.

A disagreement reuses contradicted rather than adding a refuted, because the engine already has a word for something stands behind this claim and disagrees with it and two words for one state is how a vocabulary comes apart. What does differ is the sentence: for a parameter or a tool version the write-up says one of the two is wrong, and this write-up cannot tell you which, because the claim may be describing a run that happened and the pipeline may have been edited since. For a number read out of the artifact the claim is bound to, there is nothing left to be uncertain about, so it says which.

Why this belongs in the engine and not in a consumer’s tests

A consumer can and should assert this in their own tests. But a test asserts against whatever the test can see, and what it cannot see is the provenance chain — that the artifact it read is the one the claim is bound to, produced by that step, with those inputs, at those tool versions. Ligase is the only component holding both halves. It is the same argument that puts cite in the engine rather than in a consumer’s methods script (D24).

What it cost elsewhere

Produced and Bound now carry the output’s path, which is the only thing in a write-up that depends on an artifact still being on disk; everything else comes from the record. And Eq is gone from Claim, Restated, ClaimVerdict and ClaimCheck, because a claim can hold an f64 and f64 has no total equality. Nothing wanted it: these types are compared for sameness and never used as a key.

D57 — A change that alters results is named where a pin is chosen

Two consumer integrations lost a debugging cycle to the same thing, and neither was a defect in the behaviour that broke them. One of them put it as a process request rather than a bug report, which is how it is recorded here.

lg.reference() began requiring frame=. A reference that is not whole-genome length is refused without it, which is right on the merits: the silent coordinate shift it prevents had invented Intermediate-level PI resistance in a consumer’s own evaluation, and the error message names the exact fix. It broke 4 of 78 tests on 0.2.0 and the identical 4 on 0.3.0.

A gene_coverage default dropped twelve drugs from a well-formed report. A record covering the full p51 reverse transcriptase — every position any rule scores — came back with no RT drugs at all, eight codons short of a floor computed over the gene’s whole length.

Both shipped as minor bumps with the migration documented inside a long changelog body. A consumer pinned ligase-bio>=0.1.0 picks that up on their next lock with no signal at all.

The asymmetry worth fixing

The frame change fails at pipeline.check(), before anything runs. That is ideal: a refusal with the fix in the message costs a consumer an afternoon.

The coverage default did the opposite. The run succeeded, the report was well-formed, and twelve drugs were simply absent — recorded in a note, so nothing was concealed, but a consumer who does not promote notes to a visible log level ships a clinical resistance report with no NRTI or NNRTI calls and no error. (That consumer logged notes at DEBUG, which they have fixed. The point stands: the engine made it possible.)

“The run succeeds and the result is different” is the one category a consumer cannot defend against.

So the rule, in three parts.

1. A result-altering change is named where a pin is chosen. MIGRATING.md holds every change that can alter results or refuse what used to pass, newest first, with what breaks, the symptom and the exact edit. The changelog keeps everything; this file keeps only what a consumer must act on, which is what makes it short enough to read before upgrading.

2. The pair is enforced, not intended. An entry that points at MIGRATING.md with no section there fails tests/test_import.py, and again in the release workflow’s version job — the last thing between a tag and two append-only registries. A migration heading for a version the changelog does not know fails too, because MIGRATING.md is read from a release and an unreachable heading is no better than a missing one. M13’s finding applies to process as much as to code: a declaration nothing can check is a comment, and “we will remember to write release notes” is exactly such a declaration.

3. Where a result-altering change can refuse, it refuses. Preferred over every form of notice, because a refusal at check time cannot be missed and a note can. D56 is the first change made under this rule and it takes the loud option: inference stops answering, two undeclared lists are refused at check(), and the remedy is one argument. The alternative — keep inferring and mark the value as a guess — would have been quieter and would have left the join passing.

What this does not promise

Not a major bump per behaviour change. At 0.x that would spend version numbers on a stability nobody has been offered, and the project says plainly that it has no outside users yet. What is promised is the signal: the change is in MIGRATING.md, the changelog entry says what breaks in its first sentence rather than its fifth, and the engine refuses rather than notes wherever it can.

And not a guarantee that every such change is caught. A default that alters numbers is sometimes only visible to the person who measured it — both of these were. What the rule buys is that once it is known, there is one place it goes, and a release cannot quietly fail to mention it.

D62 — A build that is not a release says so

env!("CARGO_PKG_VERSION") is read from the manifest at compile time, and the manifest is bumped by a release commit. So for every commit between one release and the next, a source build reported the number of the release it followed — and that string is not only printed. It is the engine’s half of a hivdb step’s tool version (D20’s rule applied in architecture/hivdb), so it reaches the cache key.

A consumer measured what that costs. A source build of 550a70f — the commit carrying the scored-codon floor, after v0.3.0 and before the 0.4.0 bump — reported read by ligase 0.3.0 while answering differently from the published 0.3.0, which does not carry the floor. Their words for the consequence:

if CI builds from a branch and shares a store, two engines collide under one key.

That is the failure the cache is built to prevent, arriving through the version string rather than through the content. Two engines that disagree about a result share a key, so the store serves one engine’s answer to the other — and worse than a wrong number, it is a wrong number with a provenance record asserting the version that did not produce it. The same string goes into a methods paragraph, so a manuscript generated from a branch build cites a version whose released artifact would not reproduce it.

Two questions, so two answers

The mistake was having one number answer both of these:

• What release is this? — what a wheel publishes, what lg.__version__ reports, what ligase-bio>=0.4.0 resolves against. version().
• What software ran? — what the rule fingerprint, tool="ligase", the provenance record and the self-citation report. build().

Between releases those differ, and only the second may carry a marker: a wheel whose metadata said 0.4.0+13.g0c5980e would be a different distribution from the one a pin resolves to, and the workspace’s one-number rule exists so the Rust and Python halves cannot disagree about that number. So version() is untouched and build() is new beside it.

build() is the release version plus a local version part — 0.4.0, 0.4.0+13.g0c5980e, 0.4.0+dirty — composed from what git describe says about the source. After a single +, because the result is printed in a methods paragraph and has to stay a version a reader can resolve.

Why the git description, and not the two alternatives

The reporter offered three remedies. The other two were a bare dirty-or-branch marker, and refusing to share a store between builds whose version strings match but whose binary hashes do not.

A bare marker is strictly weaker: it separates a modified tree from a clean one and leaves every clean commit after a release claiming that release. The binary hash is stronger than what is implemented here and was not taken, for the reason the project usually takes the stronger option and here does not: a hash is not a version. It cannot be printed in a methods section, it differs between two builds of one commit that are the same software — a different toolchain, a different feature set — and it would put a value in the key that nobody can reconcile with anything published. The git description is the conventional answer because it is the one that stays legible: 0.4.0+13.g0c5980e names a commit a reader can check out.

What it does not catch is written down in build.rs rather than discovered later. A build with no repository to ask — an sdist, crates.io — falls back to the manifest version, which is the truth for a release and the old behaviour for a snapshot tarball; LIGASE_GIT_DESCRIBE is how a packager says which. Cargo reruns a build script only when a watched file changes, so HEAD, the current branch’s ref and packed-refs are watched and a marker can still sit a commit behind until the crate is rebuilt. And it describes the source, not the binary, which is the third remedy above and remains available if a consumer measures a collision it would have caught.

The intended cost

Development builds get new cache keys, and every hivdb step reruns once per build. That is the change, not a side effect of it: the keys were wrong, and a key that is right is a key that differs whenever the engine does.

D63 — One work is one reference, and the engine knows its own

D58 closed a gap nobody could close and opened a smaller one in its place. The consumer who reported the gap had followed 0.3.0’s advice:

No reference is recorded for ligase. Add one with `@lg.step(cite=...)`, or
cite it by hand.

So they cited it by hand. On the build that cites itself, that pipeline renders:

References

Njeri K. Ligase: dimension-checked, reproducible computational biology
    pipelines (version 0.3.0). https://github.com/Neoforge-Labs/ligase
Njeri, K. Ligase: type-safe, reproducible computational biology pipelines
    (version 0.3.0). https://github.com/Neoforge-Labs/ligase
Python Software Foundation. Python 3.15.0. https://www.python.org

Two entries for one work. Their framing of why that is not merely untidy is the part worth keeping: deleting the hand-added line fixes the new build and brings the unclosable gap straight back on the pinned one, so no spelling of the pipeline was correct on both versions. A consumer at the floor had to choose which version their source file was right for, and they kept a REMOVE this when the floor moves comment instead.

A fix that only documented the remedy would have been D57 applied to D58’s own change, and worth doing. It is not enough on its own: the advice would be delete a line you were told to write, acted on at a different time by every consumer, with a duplicated reference in a manuscript as the cost of forgetting.

Deduplicate, rather than assume nobody cited you

The engine knows its own citation. A declared reference pointing at the same repository is the same work, so one of the two goes.

The engine’s own is kept. It names the versions the run actually recorded — the build, since D62 — where a hand-written string names whichever version it was typed against. The consumer’s example says version 0.3.0 in a write-up produced by 0.4.0, which is the kind of wrongness this project refuses from itself and should not propagate because a user supplied it. The cost is that their formatting is dropped; the alternative is a reference list asserting a version that did not run.

The URL identifies the work. For software with no DOI that is how it is cited at all: a reference-list entry is free text, and the repository a reader is pointed at is the thing being named. cite::REPOSITORY is one constant, rendered by cite::engine and matched by cite::is_engine_reference, so the renderer and the recogniser cannot drift.

The match ends at the repository name. github.com/Neoforge-Labs/ligase followed by /, ., a space or nothing is this work; .../ligase-bio is a different one. A loose substring test would have silently eaten a consumer’s reference to a sibling repository, which is the same class of error in the opposite direction.

What this does not become

A general citation deduplicator. The engine compares nothing else: two cite= strings for the same paper, written differently, are both printed, because the engine has no way to know they are one work and guessing would be D24’s forbidden invention in a new place. The engine’s own reference is the one case where it holds the fact rather than inferring it — which is exactly what makes the deduplication checkable rather than clever.

D64 — Two commands naming one result name it the same way

D59 put the stable identifier in a claim’s sentence and called it the wrong thing. The consumer who had asked for it checked the fix and found the same irreconcilability in a new form:

$ ligase methods pipelines/malaria_validation.py
"malaria F1 = 0.333 against seven curated targets" rests on `validate`,
    output b78ad78abbb2, recorded 30 October 2026 on macos-aarch64.

$ ligase audit validate pipelines/malaria_validation.py
deterministic    `validate`
    the same bytes both times   98bf0c2be9e1

98bf0c2be9e1 is the output’s bytes — it is the artifact’s filename in the store. b78ad78abbb2 is the digest of its canonical form, and there is no file under store/b7. So a reader given the first identifier and shown the second could not tell whether they were looking at the same result, a different one, or two kinds of digest. The whole point of D59 was that a reader can take an identifier from a methods section and reconcile it with the artifact.

Two separate mistakes, with two separate fixes.

The word was wrong. A write-up says canonical b78ad78abbb2, because that is which of a step’s two digests it is. output was accurate about whose digest it is and silent about which, and silence is what let the two commands look like they disagreed. The multi-sample line reads Canonical digests: for the same reason.

The other command printed only one of the two. ligase audit now prints canonical form under the bytes for every verdict whose bytes matched. The verdicts that disagree about the bytes already printed it — what they mean does not is the canonical digest, and not reproducible prints both runs’ — so this is the case where the report had named only the filename.

Why the canonical digest is the one cited, which the report left open

The reporter was tentative about this and right. The canonical form is what D4 makes scientifically identical mean: it is stable across byte-level differences that do not change the result, which is exactly the stability a manuscript needs from an identifier it will carry for a year. The bytes digest moves when a tool writes the time of day into a header — a rerun that reproduces perfectly gets a different one — so citing it would have reintroduced D59’s defect through a different door.

So: cited by the canonical digest, labelled as the canonical digest, and the bytes digest stays out of the write-up altogether. It is a filename, and a filename in somebody else’s cache is not an identifier a reader can use.

The general rule it leaves behind

Two surfaces naming one thing name it the same way, and a word in front of an identifier says which identifier it is. This is D26’s rule — every surface says the same thing in the same words — applied to identifiers rather than to findings, and the failure mode is the same: a reader who cannot tell two accounts apart assumes they disagree.

D65 — A declaration is refused where the file itself refutes it

D38 says a declared dimension is recorded and not verified, and architecture/molecules gives the sharper reason for the representation axis: the engine has no chemistry (D49), so it could not check the claim if it wanted to.

A consumer took that seriously enough to test it, and found the sentence true of three operations and false of the fourth.

salt-stripped does not pin a molecule’s charge state, but an InChIKey’s final block does.

Their measurement, against the released 0.3.0: one generated set, and two reference sets differing only in charge state, each declared inchikey; salt-stripped.

reference keyed -N (salt-stripped then uncharged)
    generated=2 reference=2 novel=1 -> novelty=50%
reference keyed -M (salt-stripped only)
    generated=2 reference=2 novel=0 -> novelty=0%

ligase check printed no problems found and exited 0 for both, because both declarations read the same and they matched. Aspirin was in the reference set either way; in the first reading it came back novel.

They opened by saying lg.chem.standardize is correct and that they had gone looking for a bug in it and found none. That is right, and it is what makes the finding structural: the vocabulary term is coarser than the identifier it keys to. It is D56’s defect one level down — there, two inferred values compared equal and the join passed; here, two honest declarations do.

What a key can be made to say

An InChIKey’s third block is the protonation layer: N for a proton balance of zero, M, L, K for one, two and three protons removed, O, P, Q for as many added. So:

charge-neutralised declared over a key that is not neutral is refused at load. Not noted, not inferred around — refused, as a contig-naming mismatch is, and for the same reason: the file contains the evidence against the claim. The error names the key, its row, its block and the state those keys are in, so the honest declaration is in front of the reader.

All-neutral keys under salt-stripped alone are a note. This direction cannot be proved. A molecule with nothing to neutralise keys -N with nothing applied, so a table of such compounds is honest and looks identical to one that was neutralised by a step nobody declared. The note says what the shape usually means and leaves the judgement with the person who prepared the file.

That asymmetry is this project’s standing answer — the stereo check reached it from the other side, and M10a’s case folding before that: a false agreement is unrecoverable, a loud refusal is not.

Why this is not an exception to D38

D38 is about what the engine must not invent. Checking a declaration against evidence the file already contains invents nothing: the engine reads one character of a string it is already reading, and no structure is parsed, no ring perceived, no InChI computed. D49 is untouched.

The rule the two make together is the one worth carrying to the next axis: a declaration is taken on trust only where there is nothing to check it against. Where the data can refute it, the data wins.

Why salt-stripped stays expressible

The report’s third suggestion was to consider removing the one-term form as a destination for standardize, since every constructible use of it also wants charge-neutralised. It stays, for a reason that is about the axis rather than about the API: files really are in that state. A salt-stripped export is a thing a collaborator hands you, and a vocabulary that could not describe it would push that file into as given or into a gap — which is the dimension system failing at its own job to avoid a trap at a different surface.

What was wrong was the documentation, so that is what changed. The guide says plainly that public collections ship neutral parents, that the one-term form emits keys matching nothing in the wild, and that it is not a state to standardise to. A trap named where somebody is choosing is worth more than a removed option, and the removal was the reporter’s own least-preferred remedy.