Crossing identifier spaces

You have a list of differentially expressed genes as HGNC symbols, and a published truth set as Ensembl IDs. This is how you compare them without quietly losing half of each.

Load each list

A gene list is an EntitySet: a TSV with a header, one column of identifiers, and as many attribute columns as you like. The identifier column is id unless you say otherwise.


import ligase as lg
 
mine = lg.entity_set(
    "results/degs.tsv",
    organism="Homo sapiens",
    identifier_namespace="HGNC symbol",
)
published = lg.entity_set(
    "truth/published_set.tsv",
    organism="Homo sapiens",
    subject="breast carcinoma",
)

One call declares an identifier space and the other does not, and the difference is the whole of what the loader can and cannot read. Ask what was loaded:


>>> mine.dimension("identifier_namespace").value
'HGNC symbol'
>>> mine.dimension("identifier_namespace").source
'declared'
>>> published.dimension("identifier_namespace").value
'Ensembl gene'
>>> published.dimension("identifier_namespace").source
"from the identifiers' own pattern: truth/published_set.tsv"

ENSG00000141510 carries a marker no other space has, so the published set states its own space and the loader says where it read it. TP53 carries none: it is the shape of an HGNC symbol, of a ChEMBL component synonym and of most in-house panels at once, so a symbol list has to say which it is (D56). Leave it off and the dimension is unknown with a reason, which the join then refuses rather than guesses past. See identifier spaces on all four cases where inference declines.

Watch the join be refused


lg.check_step("score", {"truth": published, "found": mine})


DimensionError in step `score`:
  truth.identifier_namespace = Ensembl gene (from the identifiers' own pattern: truth/published_set.tsv)
  found.identifier_namespace = HGNC symbol (from the identifiers' own pattern: results/degs.tsv)
  These must match. Add a `map_identifiers` step; identifiers are never renamed implicitly.

Nothing has run. check_step read the two identifier columns and compared dimensions. There is no flag that makes this join work, and that is the design: see D38.

Load the mapping table

The mapping is itself an EntitySet, not a path, and the reason is worth a moment. Its identifiers are the source side of the mapping, so they are in the same space as the list you are converting; the targets live in an attribute column.


hgnc_to_ensembl = lg.entity_set(
    "refs/hgnc_to_ensembl.tsv",
    organism="Homo sapiens",
    identifier_namespace="HGNC symbol",
    release="HGNC 2026-09",
)

A table like that looks like this, and the target column can be called anything:


id      ensembl_gene_id
TP53    ENSG00000141510
BRCA1   ENSG00000012048

Two things follow from the table being typed. Its contents join the cache key, so editing the table reruns the mapping — which a path would not do, because a plain parameter is hashed by its value and a path’s value does not change when the file behind it does. And its own space is checked against the list being mapped, so a table of Entrez IDs handed to a list of symbols is a DimensionError before anything runs, rather than a mapping that matches nothing.

Declaring the table’s release is optional and worth doing. It is compared by equality wherever both sides state it, and it is what a methods paragraph cites when a reviewer asks which HGNC download you used.

Convert


in_ensembl = lg.map_identifiers(
    mine,
    hgnc_to_ensembl,
    column="ensembl_gene_id",
    to="Ensembl gene",
)

column= names the table column holding the identifiers to convert to; to= declares the space the output is in. Both are explicit on purpose: what the output is must not depend on which identifiers happened to cross.

The output declares Ensembl gene and carries everything else unchanged — the organism, the subject, the release — because a conversion crosses one axis and is not a new list. So the join now passes:


@lg.step
def score(truth: lg.EntitySet, found: lg.EntitySet) -> lg.EntitySet:
    """Score a result set against a truth set."""
    return lg.sh("comm -12 <(cut -f1 {truth} | sort) <(cut -f1 {found} | sort) > {out}")
 
 
pipe = lg.Pipeline(score(published, in_ensembl))
pipe.check()   # passes; nothing has run
report = pipe.run()

Read what did not cross

This is the part that makes the step worth having over a merge.


for run in report.steps:
    for note in run.notes:
        print(note)


the identifier `SEPT7` is in no row of the mapping table, so it is not in the output
the identifier `NKX2-1` maps to more than one target, so it is not in the output


2 identifiers did not cross into the target space, so the output is 2 entities smaller than the input

Three kinds of identifier do not cross, and none is dropped in silence:

No row in the table. Usually a symbol renamed or retired since the table was downloaded. Check the table’s release.
Several rows in the table. One symbol legitimately covering more than one gene. It is left out rather than resolved, because emitting a row for each would make the output larger than the input without saying so, and choosing one would be a guess about which you meant. Decide, and add a one-to-one row — or, where the identifier really does denote two things, ask for a fan-out; see when an identifier means two things below.
A target another of your identifiers also reaches. Ordinary for a table holding both a deprecated symbol and its replacement, when your list holds both. Emitting both rows would make the output name one gene twice with two sets of attributes, which is not a set; keeping one would be a guess about whose attributes to carry. Both are left out and named.

So the output is smaller than the input by exactly the identifiers the notes name, and the last note says how many that is. The named ones stop at twenty — enough to recognise a pattern, and short of putting four thousand rows in your provenance — while the count covers the rest. Compare it against the size of your input before reading the result: a large share usually means the mapping table is from a different release than the list.

The notes travel into the run report, provenance, a reproducibility bundle and the methods paragraph. Nothing about the conversion is invisible later. See fixing a noticed line for what to do with one.

What a spreadsheet did to your file

If the list has been through Excel, some symbols are now dates:


>>> genes = lg.entity_set("results/degs_from_excel.tsv", organism="Homo sapiens")  # undeclared, on purpose
>>> for note in genes.notes:
...     print(note)
the identifier `2-Sep` is a date, not a name; a spreadsheet does this to SEPT7, MARCH1 and DEC1
the identifier `1-Mar` is a date, not a name; a spreadsheet does this to SEPT7, MARCH1 and DEC1
>>> genes.dimension("identifier_namespace")      # declined to infer
None
>>> genes.unknown_reason("identifier_namespace")
"not declared, and some identifiers are dates rather than names -- see this dataset's notes, and re-export the file with the identifier column typed as text"

The file loads — it is valid TSV, and refusing it would be turning a format problem into an engine failure. What the loader does instead is say what it saw and decline to infer a space for a file it cannot fully read.

The reason names the cause rather than giving the generic advice, and that is deliberate. “Declare identifier_namespace” is sound for a file whose identifiers are merely unrecognised, and it is the worst possible advice for this one: declare it and the file is accepted from then on with SEPT7 missing from every join, which is the corruption made permanent.

The fix is to re-export with the identifier column typed as text. There is no option to have Ligase repair it: 2-Sep could have been SEPT7 or SEPT2, and the engine will not guess which gene you meant.

When an identifier really means two things

“Decide which target you mean” assumes the ambiguity is a data-quality problem. Often it is. Sometimes the identifier genuinely denotes more than one entity, and then there is no one-to-one row to add:

P. falciparum DHFR-TS is one polypeptide with two catalytic domains, answering to human DHFR and TYMS. PPPK-DHPS is the same story. Both targets are right; this is the organism, not a curation error.
A protein complex named as a unit maps to its subunit genes.
A paralogue family has no one-to-one human orthologue but a defensible set of them.

ambiguous="fan-out" asks for a row at each target:


in_ensembl = lg.map_identifiers(
    mine,
    hgnc_to_ensembl,
    column="ensembl_gene_id",
    to="Ensembl gene",
    ambiguous="fan-out",
)


the identifier `NKX2-1` maps to more than one target, and this step asked for a fan-out, so the output holds a row for each
1 identifier fanned out into 2 rows, so the output is 1 entity larger than the input

The size change is on the record, in the one direction nothing else in this conversion moves — so anything counting entities before and after will disagree, and the second count is the right one for the space the output is in.

It is declared at the call site and never inferred, because the engine cannot tell an untidy mapping table from a bifunctional enzyme. It is an ordinary step parameter, so changing it reruns the step: the output is different. And a target two of your identifiers reach is still left out, because the output is a set (D60).

A ranking cannot fan out

Rank is the row position (D41), and one row becoming two has no rank: tie ranks would invent an order nothing produced, and renumbering would make rank n mean something no computation assigned. So map_ranked_identifiers has no ambiguous argument, and the step that fans a ranking returns an EntitySet:


targets = lg.fan_out_ranked_identifiers(
    ranking,
    hgnc_to_ensembl,
    column="ensembl_gene_id",
    to="Ensembl gene",
)   # -> EntitySet, because the ranking did not survive

Declaring a space of your own

An in-house panel, a consortium’s numbering, a lab’s accessions: declare the space and the engine will compare it like any other.


panel = lg.entity_set(
    "refs/resistance_panel.tsv",
    organism="HIV-1",
    identifier_namespace="NeoRx panel v3",
    subject="HIV-1 drug resistance",
)

A declared space is recorded rather than checked against the identifiers — the engine knows seven spaces and yours is not one of them, so it takes your word. It still refuses to join this panel to anything declaring a different space, which is the protection you wanted.

Identifier spaces — why this is a dimension, and how inference decides.
My overlap with the published gene set is empty — the failure, and what the engine says about it.
Conversions — the registry map_identifiers is the first entry in, and what the next two will be.