Conversions

Why a registry, with one entry in it

v0.2 ships one conversion. Building a registry for one entry looks like over-design, and the reason it is not is D5: the dimensions are nodes, conversions are edges, and the automatic fixes planned for v0.3 are a path search over those edges. “Your HGNC list cannot join this Ensembl table, but map_identifiers would get you there” is the one-edge case of that search.

A search needs edges it can inspect. Three functions named map_identifiers, liftover and rename_contigs give it nothing to search: the engine would have to know, in code, which axis each one moves and what it needs. So each is a row instead.

What an entry declares

crates/ligase-core/src/convert.rs:

pub struct Conversion {
    /// The step name a user writes, and the name an error sends them to.
    pub name: &'static str,
    /// The one axis it crosses. Every other dimension is inherited.
    pub axis: Dimension,
    /// The kinds it converts. In and out are the same kind.
    pub kinds: &'static [DataKind],
    /// What the user has to supply, for a message that has to say so.
    pub supplies: &'static str,
}

The whole registry, today:

Two lookups, and they are what a path search will use: Conversion::by_name and Conversion::across(axis). The second is the interesting one — “is there a way across this axis?” — and its honest answer for nearly every axis is None. There is no way across organism.

The axis is a field, not an implication

Propagation has to know which single dimension of a step’s output to overwrite, and which to carry. graph::propagate inherits every applicable dimension from the inputs as usual, and then applies the conversion’s target:

if let Some(converted) = &spec.conversion {
    converted.target.apply(
        &mut dimensions,
        Source::Derived { step: step.to_owned() },
    );
}

Overriding after inheriting rather than instead of it is the load-bearing detail. A map_identifiers output that lost its organism, its subject and its release would be a new gene list rather than the same one renamed, and every check below it would have less to work with than the check above it. The axis field is what makes “one axis changes, the rest travel” expressible at all.

This mechanism is not new. coordinates already works this way for a step that writes Intervals: a tool producing BED from a VCF changes the convention, so the step declares what it writes rather than inheriting a lie. A conversion is the general case of that.

The target is read from one parameter

Conversion::TARGET is "to", for every conversion there will ever be.

The destination cannot live on the registry entry, because it is per call: one written map_identifiers step converts to whichever space its call named. It cannot live on the step decorator either, for the same reason. So it is an argument — and the Python surface has to find it among a step’s arguments without knowing which conversion it is looking at.

One spelling solves that, and it is checked when the step is decorated rather than when a pipeline runs:

@step(tool="ligase", conversion="map_identifiers")
def map_identifiers(
    entities: EntitySet, table: EntitySet, *, column: str, to: str
) -> EntitySet: ...

Step.__init__ asks the engine two questions — is this a registered conversion, and what is the target parameter called — and raises StepDefinitionError if the answer to either is wrong. Both answers are the engine’s, so the decorator and the graph cannot disagree about which conversions exist. Pipeline._place then reads arguments["to"] and hands (name, to) to add_step, which resolves it against the registry and refuses a target the axis cannot take — a blank one, in particular, because a blank label compares equal to another blank label and two outputs that agreed about being in no particular space would be the exact false agreement the dimension exists to break.

Why the mapping table is a typed input

table= takes an EntitySet rather than a path, and this was a deliberate choice with two consequences.

Content, not a path, joins the cache key. A plain parameter is hashed by its value, and a path’s value does not change when the file behind it does. A typed input is hashed by its canonical form, so editing the mapping table reruns the mapping and everything below it.

The table’s own space is checked. Its identifiers are the source side of the mapping, so they are in the same space as the list being converted. An Entrez table handed to a list of HGNC symbols is therefore an ordinary DimensionError found by the ordinary checker — no special case, no extra validation inside the step — rather than a mapping that silently matches nothing. This is the dimension system being used on the conversion’s own inputs, which is a good sign that the shape is right.

What a conversion does with what it cannot convert

Every entry in this registry is an operation that can fail for some inputs. That is what distinguishes a conversion from a rename, and a rename across a dimension is what D38 forbids.

convert::map_identifiers reads the table into a BTreeMap<String, Vec<String>> — a Vec per key, not one value, because the one-to-many case has to stay visible; a map keeping the last row would resolve it silently, and which row came last is not a fact about the genes. Then, per input row:

The last row was missed when this was first written, and it is the mirror image of the third. A table holding both a deprecated symbol and its replacement sends two of your identifiers to one gene, and the output would then name that gene twice with two sets of attributes — not a set, and a cardinality change of exactly the kind the one-to-many rule exists to refuse. Keeping one of the two would be a guess about whose attributes to carry. So both are left out and both are named.

Collision cannot be decided row by row: the second identifier to arrive is what makes the first one a problem, and by then the first has been written. So map_identifiers reads the list twice — once to find the contested targets, once to write — which is why collisions exists as a separate pass.

That pass depends on ambiguous, and the dependency is easy to miss. Under Drop an ambiguous identifier lands nowhere and therefore cannot collide, which is what the first implementation assumed everywhere. Under FanOut it lands at every one of its targets and can collide at each, so collisions takes the setting and counts fanned targets as reached. Forgetting that would have been a silent way to break the set property — two identifiers at one target, each believing itself alone — which is the one thing this conversion’s failure handling exists to prevent.

An identifier all of whose targets collide is reported as a collision rather than as a fan: it reached the output nowhere, and the note a user reads should say what happened to it rather than what was asked for. Only one note is ever recorded per identifier, because the dedup key is the identifier itself — so the arm that noticed it is the arm that decides what it was.

No failure is dropped and none is carried over unchanged. Carrying one over would put a name from the old space into a set that declares the new one, and the whole point of the output’s declaration is that it is true. So the output is smaller than the input by exactly the identifiers the notes name, and the notes reach the run report, provenance, the bundle and the methods paragraph by the ordinary route (D26).

The named ones are capped and the total is not. A twenty-thousand-row list converted through a table covering most of it leaves thousands behind, and naming every one puts a provenance row per identifier and interpolates all of them into a single methods sentence. So NAMED_AT_MOST is twenty — enough to recognise a pattern, which is what naming them was for — and Note::IdentifiersDidNotCross carries the count, which is the half of spec §5.1’s “noticed, counted, and listed” that the first implementation left out. Noticed keeps the fan counters apart from the drop counter for the same reason it keeps the cap: the two totals say opposite things about the output’s size, and one counter serving both would report a fan-out as a loss. Both totals can appear in one pass, since a table may send one identifier nowhere and another to two places. Deduplication is by a BTreeSet of note values rather than by scanning the notes, because the scan is quadratic in the number of notes and the number of notes is the size of the user’s data.

A fan-out demotes a ranking, and the type system is what says so

Ambiguous is a parameter of convert::map_identifiers and nothing in the registry changes for it: the fan-out is a property of the crossing, not a new edge. What does change is which Python step may ask for it.

map_identifiers (EntitySet → EntitySet) takes ambiguous and passes it through. map_ranked_identifiers (RankedList → RankedList) passes "drop" always, and has no argument for anything else, because a ranking that gained rows is not a ranking (D41). fan_out_ranked_identifiers takes a RankedList and returns an EntitySet, passing "fan-out".

That third step needed no registry change either, and the reason is worth recording. graph::propagate guards a conversion’s axis override on conversion.converts(output) — the output kind — and map_identifiers is registered for both named-list kinds, so an output typed EntitySet clears the guard and the target namespace is applied as it is for any other crossing. The input’s kind is the step signature’s business, which is where a demotion belongs.

A separate function rather than an argument is forced rather than preferred: a step’s output kind is read off its return annotation once, when it is decorated, so one function cannot produce both kinds. The same constraint split map_ranked_identifiers off in M12, and here the function’s name is the declaration — there is no way to write the fan-out of a ranking without also writing down that what comes back is a set.

One refusal is an Err rather than a note: column naming the table’s own identifier column. That would map every identifier to itself and then declare a space the identifiers are not in — the implicit rename, arrived at by accident — so it is an error before anything is written.

The engine is a tool

map_identifiers declares tool="ligase", and exec::BUILT_IN lists "ligase" beside "hivdb". Neither is an external program to probe with --version, and both change answers exactly as a tool does.

Without it, nothing about the engine would reach the cache key for this step. key.rs hashes tool versions, parameters and input content; it does not hash the crate. A step body that only calls into the engine hashes the same however convert.rs changes underneath it, so the declaration is what makes a new engine rerun a conversion. The version reported is the crate’s, which is weaker than hivdb’s content fingerprint of its own rules — an unreleased change to a conversion will not rerun a cached step — and that is the obvious thing to strengthen if it ever bites.

The next two entries

Both are out of scope for v0.2 (§7 of the feature spec), and the registry is shaped so that each is a row rather than a redesign.

What each needs that does not exist yet is a Target variant — Target::Reference(ReferenceId) and Target::ContigNaming(ContigNaming) — and the matching arm in Conversion::target, which parses a user’s to= into the value the dimension holds. Target is an enum rather than a string precisely so that a conversion’s declared value is the same type its dimension holds and cannot be put in the wrong field.

Two tests hold the shape to this. the_registry_is_the_shape_liftover_and_rename_contigs_will_use asserts that Conversion::across returns None for reference and contig_naming today, so adding either is a visible change rather than a quiet one. And the_registry_names_the_only_way_across_each_axis_it_covers asserts that every registered conversion is named in its axis’s leading_question — so a liftover added without updating what the reference axis tells users to do will fail the suite. That is the registry keeping the error messages honest, rather than the other way round.

Where the code is

What this does not touch

source_set was claimed by no kind when this was built. An EntitySet carries organism, identifier_namespace, subject and release and nothing else, so the question D36 reserved — whether the shared NameSet type’s ASCII-case deduplication is right for what source_set would hold — was untouched here. M10 settled that it must not fold ASCII case and M12 declined the axis, so the type split travelled to M15, where a KnowledgeGraph applies it and NameSet became generic over its comparison rule.

No conversion crosses a graph, and that is deliberate. map_identifiers renames a column. A graph’s identifiers are node ids and edge endpoints, and a conversion that rewrote one and not the other would produce a graph whose edges point at nothing. propagate guards a conversion’s axis override on converts(output), so the guard holds today — and would become a hiding place the moment somebody registered the existing conversion for the kind, in exactly the way M12 found for a RankedList. A conversion of a graph has to rewrite endpoints and merge the nodes that collide, and merging nodes is a decision about biology rather than a rename, so it is a different feature. The way across is a map_identifiers step on the list being joined to the graph.

What was discharged is the other half of that obligation. Until a kind applied an axis, check_step never reached its comparison arm and propagate never reached its inheritance arm, so neither could be tested — and the compiler could not help, because Release and Subject are one type. A gene list claims four axes, so each_axis_an_entity_set_claims_compares_its_own_field and a_step_over_a_gene_list_carries_every_axis_it_claims now pin each arm to the field it is supposed to read.