Dimension Catalogue

Every biological type carries dimensions. Each dimension is inferred from the file where possible; otherwise the user declares it once and it propagates.

The last five arrived after the others and sit at the bottom on purpose: a report of several disagreements is ordered by this list, so slotting a new axis in beside the ones it resembles would have reordered every report that already existed. source_set was applied by no kind until M15, and is now applied by exactly one: a knowledge graph is the only kind in Ligase that is assembled rather than measured, so it is the only one for which “which sources contributed” is a fact at all.

A gene list is the first kind here that holds no sequence, which is why the last three columns are mostly n/a: a list of gene names has no coordinate system, no contigs, no read pairing and no quality scores. Those are not gaps to be filled in but properties the data does not have, and demanding one is the category error this table exists to prevent.

The last column but one covers two kinds, and that is the point rather than a shortcut. A RankedList — a ranking of genes, of drug targets, of candidates — carries exactly the dimensions an EntitySet carries. What differs is what their canonical forms keep: a set’s rows are a set, and a ranking’s order is its result (D41). A dimension on one and not the other would mean that difference had spread past the one place it is meant to live.

The last column but one is the one kind that is assembled. A knowledge graph answers three of the named list’s questions identically — it names entities, it is about something, and whatever named its entities had a release — and brings a fourth that follows from being assembled rather than measured: which evidence sources were folded into it. That is the axis the kind was admitted for, and the failure it catches is my target ranking changed and I cannot find the analysis change.

Rules

• Reference identity is determined by contig names + lengths + MD5 where available, never by filename alone.
• A reference may also carry the coordinate frame it is numbered along. Declared, never inferred.
• Contig naming mismatches are a distinct error from reference mismatches.
• Sample sets must match by identity and order wherever steps join samples.
• A sample identifier read from a file is never corrected, and never dropped: it is what the file is about. An identifier that is a file name rather than a sample — which is what bcftools writes when an alignment carries no @RG SM — is reported instead (D26).
• An organism is a set, compared by set equality: order does not matter, a host–pathogen dataset declares both organisms, and a mismatch names the organism one side lists and the other does not (D37).
• A release is compared by equality, never by containment. That is why it is a dimension of its own rather than part of reference, which allows a contained subset (D36).
• release, subject and source_set are never demanded: each is compared wherever both sides state it, and a file that records none is still loadable. A FASTA records no release, so requiring one would make an ordinary reference unusable; a graph assembled by hand names no sources, and a demand a user cannot always meet is worse than a missed check.
• A source set is compared by set equality like organism, and its members are compared exactly where an organism’s fold ASCII case. An organism is a Latin binomial, a closed conventional vocabulary in which HIV-1 and hiv-1 are one organism; a source is usually an object key or a URL, and s3://bucket/A.json and s3://bucket/a.json are two objects by specification. Folding case would drop a member silently; not folding it can only produce a refusal you can read (D36).
• A knowledge graph’s source set is read off the graph, from the sources named on its nodes, its edges and its top level, and a graph naming none leaves the axis unknown rather than known and empty (D44).
• An identifier namespace is compared by equality and never coerced. Two gene lists in different spaces do not join, and the only way across is a map_identifiers step, because mapping between spaces is a lookup with failures rather than a rename (D38).
• Unlike release, subject and source_set, an identifier namespace is demanded where two lists of named entities are joined: a user can always meet the demand, by declaring it or by letting the loader read it off the file, and two lists that each say nothing would otherwise agree by silence. This holds in either combination, a ranking scored against a set included, which is the commonest reason to join the two kinds at all.
• A representation is compared by equality of its identity and set equality of its operations, and the operations are a closed set: a file naming one the engine has never heard of is refused rather than carried along unread. InChIKey and SMILES never compare equal, because they are not equally good identities — across six RDKit releases every InChIKey held still while one canonical SMILES moved (D47).
• A representation is recorded rather than verified, with one exception the keys themselves supply. An InChIKey’s third block is its protonation layer, so a table declared charge-neutralised whose keys end in anything but N is refused at load: the file refutes the claim, and the engine needs no chemistry to see it. The other direction is a note, because a molecule with nothing to neutralise keys -N with nothing applied (D65). Salt stripping and tautomer normalisation leave no such trace, which is why they are taken on your word.
• stereo-discarded is an operation that was applied, not a setting somebody chose. RDKit’s tautomer canonicaliser discards stereochemistry at an sp3 centre inside a tautomeric system, so a tautomer-normalised set cannot tell ibuprofen’s two enantiomers apart. The standardiser therefore checks what it did against what it was asked for, asymmetrically: stereo lost and not declared is an error, declared and not lost is a note.
• Like identifier_namespace and unlike release, subject and source_set, a representation is demanded wherever two molecule tables are joined. A user can always meet it — declare it, or standardise and let the step write the line — and two tables that each say nothing would otherwise overlap in whatever their two toolchains happened to agree on.
• A molecule table claims no identifier_namespace, though an InChIKey is an identifier space: representation already says which space the file is in, and two axes carrying one fact is the mirror image of what D36 forbids.
• Unknown dimensions are allowed only if no downstream step requires them; otherwise check() fails with a request to declare them.

A dataset may be about more than one organism

A host–pathogen sample, a metagenome and a xenograft are each about several organisms, and the reference for the first is usually a combined FASTA holding both genomes. So organism takes a list:

ref = lg.reference("refs/hxb2_plus_host.fasta", organism=["HIV-1", "Homo sapiens"])
aln = lg.alignment("aln/sample01.bam", organism=["HIV-1", "Homo sapiens"])

Two sets are equal when they name the same organisms in any order, each compared without regard to surrounding space or case, and a spelling repeated in one list counts once. A plain string is a set of one, so organism="HIV-1" means what it has always meant.

Where the sets differ, the error states both and then names the difference — ref also lists Homo sapiens — rather than leaving two lists to be compared by eye. There is no alias table: human and Homo sapiens are two different names and will be reported as a mismatch, because equating them means guessing. My pathogen results include host reads is the failure this exists to catch, with the message it produces.

A gene list lives in an identifier space

TP53, ENSG00000141510, 7157 and P04637 are four names for one gene, and no two of them are equal as strings. A list of gene names records nowhere which of those spaces it is in, so joining two lists written in different ones returns an empty or nearly empty result and no complaint. That is my gene-set overlap is empty.

So the space is a dimension, and it is the one dimension in this table the engine can usually read without being told:

genes = lg.entity_set("results/degs_ensembl.tsv", organism="Homo sapiens")
genes.dimension("identifier_namespace").value   # 'Ensembl gene'
genes.dimension("identifier_namespace").source  # "from the identifiers' own pattern: ..."

Seven spaces are recognised — Ensembl gene, UniProt accession, RefSeq, HGNC ID, ChEMBL target, Entrez gene and HGNC symbol — and the shapes are tried most specific first, because a symbol has no distinguishing marker and would otherwise claim a UniProt accession. Each admits what people actually have: ten-character TrEMBL accessions as well as six-character reviewed ones, versioned Ensembl IDs such as ENSG00000141510.18, mixed-case symbols such as C9orf72, and readthrough symbols such as TNFSF12-TNFSF13. A shape too narrow merely declines to infer; a shape too broad claims another space’s identifiers, which is the silent failure itself.

The space is inferred only when every identifier in the file resolves to the same one and that one carries a marker; a file that mixes two, a file with no rows, a file holding one name nothing claims, and a file of bare gene symbols all leave the dimension unknown rather than guessed.

The last of those is the one that catches people, and it is deliberate. The symbol shape is the only one with no marker, and ChEMBL’s component synonyms (DHFR-TS), a lab’s own panel and an approved readthrough symbol (TNFSF12-TNFSF13) are the same string shape, so a list of symbols has not stated its space and is not given one (D56). Declare it:

genes = lg.entity_set(
    "results/degs.tsv",
    organism="Homo sapiens",
    identifier_namespace="HGNC symbol",
)

A declaration is accepted as given and not checked against the identifiers, because a curated list’s own spelling is a space too and the engine has never heard of it. See identifier spaces.

Coordinates are semantics, not encoding

The coordinates dimension describes what a step sees, not how bytes sit on disk. Alignment records are 1-based closed in the SAM specification, and that is how Ligase presents them from either container, so a lossless samtools view -b does not change the dimension. The 0-based offsets inside the BAM binary encoding are a container detail; they are recorded in provenance, never as a dimension.

This matters because the dimension exists to catch a real class of bug — feeding 0-based half-open BED intervals to a tool expecting 1-based closed coordinates. Re-encoding the same alignments is not that bug, and reporting it as one would train users to ignore check().

A reference carries the frame it is numbered along

Reference identity answers “is this the same sequence?”. It does not answer “what are these positions counted along?”, and for a reference that is a slice of a genome those are different questions.

hxb2_pol.fasta holds HXB2 K03455 2085–5096. Every position in it is 2084 lower than the same base of K03455, and nothing about its bases, its defline or its length says so. A step whose own coordinates are fixed — lg.hivdb.interpret holds the HIV-1 gene windows as positions in K03455 — cannot tell that file from a 3,012-base genome of something else.

So a reference can say:

pol = lg.reference("hxb2_pol.fasta", organism="HIV-1", frame=("K03455", 2085))

read as base 1 of this file is base 2085 of K03455. A whole genome declares ("K03455", 1), and a reference whose single contig is 9,719 bases is taken to be whole-genome K03455 without declaring anything, because length is the fact the gene windows depend on.

It is declared and never inferred: no FASTA records where its first base sits in anything else. A step that needs a frame and cannot establish one asks for it by name, and a reference in the wrong frame is a ReferenceFrameError before any compute.

This is not the coordinates dimension and not a reading frame. coordinates says whether a position is counted from zero or from one; a frame says what it is counted along. See D33.

References relate by containment, not equality

Two references compared with each other must be identical: same contigs, in the same order, with the same lengths and MD5s.

But data checked against a reference need only be contained in it. Every contig the data mentions must appear in the reference with a matching length and MD5; extra contigs in the reference are fine. A VCF whose header lists only chr20 is correct against a whole-genome reference, and rejecting it would be a false alarm.

The errors this produces are therefore:

Contig order gets its own error rather than being ignored: htslib matches contigs by name, so bcftools does not care, but GATK requires sequence dictionaries in matching order. Silence there would hide a real failure.