Loading data

A loader reads what it can from the file and takes the rest as declarations. What it inferred, and from where, is recorded on the result: dataset.dimension("reference").source says declared or names the header it came from.

reference

reference(
    path: 'str',
    *,
    organism: 'str | Sequence[str] | None' = None,
    build: 'str | None' = None,
    frame: 'tuple[str,
    int] | None' = None,
    release: 'str | None' = None,
) -> 'Reference'

Load a reference FASTA, inferring its contigs.

Each contig’s MD5 is computed here, over the uppercased sequence, as the SAM specification defines @SQ M5. Computing it ourselves is what lets us check an aligner’s claim about which reference it used.

Parameters

Returns

The loaded reference.

Raises

reads

reads(
    r1: 'str',
    r2: 'str | None' = None,
    *,
    organism: 'str | Sequence[str] | None' = None,
    sample: 'str | None' = None,
) -> 'Reads'

Load sequencing reads, inferring what a FASTQ can tell us.

The quality encoding comes from the observed score range, and comes back unknown rather than guessed when the scores fall in the range Phred+33 and Phred+64 share. Guessing would shift every quality value by 31, which no downstream tool would notice.

Parameters

Returns

The loaded reads.

Raises

samples

samples(
    pattern: 'str',
    *,
    paired: 'bool' = False,
    organism: 'str | Sequence[str] | None' = None,
) -> 'Samples'

Find sequencing runs and group them into samples.

Parameters

Returns

The samples, in the order their files appeared. Use .map() to apply a chain of steps to every one of them.

Raises

alignment

alignment(
    path: 'str',
    *,
    organism: 'str | Sequence[str] | None' = None,
    build: 'str | None' = None,
    samples: 'list[str] | None' = None,
) -> 'Alignment'

Load a SAM or BAM, inferring dimensions from its header.

Both containers yield identical dimensions: a lossless samtools view -b changes nothing biological.

Parameters

Returns

The loaded alignment.

Raises

variants

variants(
    path: 'str',
    *,
    organism: 'str | Sequence[str] | None' = None,
    build: 'str | None' = None,
    samples: 'list[str] | None' = None,
) -> 'Variants'

Load a VCF, inferring dimensions from its header.

Identity comes from ##contig alone. ##reference is deliberately not trusted: it holds a path, and two runs pointing at /refs/genome.fa can be pointing at different genomes.

Parameters

Returns

The loaded variant calls.

Raises

intervals

intervals(
    path: 'str',
    *,
    format: 'str | None' = None,
    organism: 'str | Sequence[str] | None' = None,
    build: 'str | None' = None,
) -> 'Intervals'

Load genomic intervals from a BED or GFF, inferring what they say.

The coordinate convention comes from the format — BED is 0-based half-open, GFF and GTF are 1-based closed — and that is the point: a tool handed the wrong one produces results that are off by one base and look fine.

Contigs come from the records, so they carry no length and no checksum. They are still compared against a reference by name, because comparing what is present is what makes these files checkable at all.

Parameters

Returns

The loaded intervals.

Raises

entity_set

entity_set(
    path: 'str',
    *,
    id_column: 'str' = 'id',
    organism: 'str | Sequence[str] | None' = None,
    identifier_namespace: 'str | None' = None,
    subject: 'str | None' = None,
    release: 'str | None' = None,
) -> 'EntitySet'

Load a set of genes or proteins from a TSV with a header.

The first type here that holds no sequence. What it carries instead is the identifier space its names live in, which is the dimension it exists for: a list of HGNC symbols joined to a table of Ensembl IDs overlaps in nothing, and until now nothing said so.

Parameters

Returns

The loaded set. Anything the loader noticed about the identifiers — a symbol a spreadsheet has turned into a date, say — is on notes.

Raises

ranked_list

ranked_list(
    path: 'str',
    *,
    id_column: 'str' = 'id',
    organism: 'str | Sequence[str] | None' = None,
    identifier_namespace: 'str | None' = None,
    subject: 'str | None' = None,
    release: 'str | None' = None,
) -> 'RankedList'

Load a ranked list from a TSV with a header, highest rank first.

An entity_set and one more thing: the row order is the result. Rank is the row position, so a ranking reordered is a different ranking and has a different semantic hash — where every other type in Ligase, this one’s own unranked sibling included, treats record order as saying nothing. Ranked differentially-expressed genes, ranked drug targets and a leaderboard of candidates are all this type.

Parameters

Returns

The loaded ranking. Anything the loader noticed about the identifiers — a symbol a spreadsheet has turned into a date, say — is on notes.

Raises

knowledge_graph

knowledge_graph(
    path: 'str',
    *,
    organism: 'str | Sequence[str] | None' = None,
    identifier_namespace: 'str | None' = None,
    subject: 'str | None' = None,
    release: 'str | None' = None,
    source_set: 'Sequence[str] | None' = None,
) -> 'KnowledgeGraph'

Load a knowledge graph of typed nodes and edges from JSON.

The first type here that is assembled rather than measured, and what follows from that is the dimension it exists for. What a graph holds is a function of which evidence sources were folded into it: add DisGeNET to a graph that had Open Targets and a literature-mined set and edges appear, neighbourhoods widen and every propagated score moves. Nothing in the file says which sources went in, so two graphs that differ only in that diff and score against each other without complaint, and the difference reads as biology.

So the sources are a dimension, and this loader reads them off the graph’s own nodes and edges rather than asking you for them.

The JSON is one object with a nodes array and an edges array. A node needs a string id; an edge needs a string source, target and type. A sources list on a node, on an edge, or at the top level is the per-edge provenance the dimension is read from, and a node’s ids are its other names. Everything else either carries is kept without being read, exactly as a gene list’s attribute columns are.

Parameters

Returns

The loaded graph.

Raises

molecule_set

molecule_set(
    path: 'str',
    *,
    organism: 'str | Sequence[str] | None' = None,
    representation: 'str | None' = None,
    subject: 'str | None' = None,
    release: 'str | None' = None,
) -> 'MoleculeSet'

Load a set of molecules from a tab-separated table.

The type the representation dimension exists for. The same substance has different InChIKeys depending on what was done to it before the key was computed: of thirty ordinary drugs measured across six RDKit releases, twelve change key under a different standardisation state, and aspirin has three. So two compound sets prepared differently overlap in nothing for the substances affected while the rest still match, and what comes out is a hit rate that is small, non-zero and reads as poor enrichment rather than as a mismatch.

Nothing about two columns of twenty-seven-character keys says which standardisation produced them. That is what makes this the one dimension in Ligase no glance at the data can catch, and why the engine refuses to join two sets that disagree about it.

The table needs an inchikey column or a smiles column. Every other column is carried without being read, a score or an activity included. A #ligase representation: line above the header is where the dimension is read from, and lg.chem.standardize is what writes one.

Parameters

Returns

The loaded set.

Raises

molecule_ranking

molecule_ranking(
    path: 'str',
    *,
    organism: 'str | Sequence[str] | None' = None,
    representation: 'str | None' = None,
    subject: 'str | None' = None,
    release: 'str | None' = None,
) -> 'MoleculeRanking'

Load a ranking of molecules, whose row order is the answer.

Everything lg.molecule_set reads, with one difference that is the whole reason the type exists: the canonical form keeps the row order. A ranking of docked compounds answers which of them to order, rank is the row position, and the same ten thousand compounds in two orders are two different answers. A cache that sorted them would serve one for the other in silence, and for a virtual screen the order decides the length of the prefix anybody actually buys.

A score column is optional and compared exactly, with no tolerance: no drift has been measured for one, and a guessed tolerance would be baked into every cache key below it.

Parameters

Returns

The loaded ranking.

Raises

remote

remote(
    locator: 'str',
    *,
    kind: 'type[DatasetT]',
    digest: 'str | None' = None,
    organism: 'str | Sequence[str] | None' = None,
    build: 'str | None' = None,
    frame: 'tuple[str,
    int] | None' = None,
    release: 'str | None' = None,
    samples: 'list[str] | None' = None,
    identifier_namespace: 'str | None' = None,
    subject: 'str | None' = None,
    source_set: 'Sequence[str] | None' = None,
    representation: 'str | None' = None,
) -> 'DatasetT'

Fetch a pinned remote input, and load what arrives.

The first step of nearly every pipeline is a fetch, and until now it sat outside everything this engine promises: a shell script downloaded some reads and the engine took its word for what had arrived. A remote declared here is an input like any other, with dimensions read from its own bytes and a content hash in every cache key below it.

A pinned remote promises the same bytes forever, from the source. An SRA run, an NCBI accession, a numbered release: the archive undertakes not to change them, so anyone, anywhere, at any time can obtain this input and check it. That is what makes digest= worth writing, and why a digest that does not match is a loud failure rather than a quiet re-fetch. A source that promises nothing needs the other half of this design, a recorded query, and is not what this function is for.

When it happens, and what it costs

It resolves when you call it, while the pipeline is still being read. That is not an optimisation to be deferred: a remote’s dimensions cannot be known until its bytes are in hand, and the engine checks dimensions before any compute. So a wrong digest fails before any step runs, because the failure happens before the graph is even built.

On a second run nothing is fetched. The locator and its digest are in the store, the objects are there, and resolution is a lookup — which is why a pipeline built on remotes runs offline once it has run online.

What digest is over

What the file means, not what its bytes are: the canonical form of what arrives, read as kind, folded over its files in order. A source that reflows a FASTA or rewords a defline has not changed the biology, and a pin that broke on that would teach you to bump pins without reading — after which the one that matters gets bumped too. Such a rewrite is reported as a note instead. A change in what the data means is the failure.

Leave it out on the first run: the digest of what arrived is reported and recorded, and you paste it in to make every later run a checked one.

Parameters

Returns

The loaded data, of the class kind named — so the call is typed as a Reference or a Reads and not as a bare dataset — carrying the dimensions inferred from what arrived and a record of where it came from. That record is what puts the fetch in the methods paragraph and the reproducibility bundle.

Raises

query

query(
    locator: 'str',
    *,
    kind: 'type[DatasetT]',
    auth_env: 'str | None' = None,
    organism: 'str | Sequence[str] | None' = None,
    build: 'str | None' = None,
    frame: 'tuple[str,
    int] | None' = None,
    release: 'str | None' = None,
    samples: 'list[str] | None' = None,
    identifier_namespace: 'str | None' = None,
    subject: 'str | None' = None,
    source_set: 'Sequence[str] | None' = None,
    representation: 'str | None' = None,
) -> 'DatasetT'

Record a live query’s answer, or replay the answer already recorded.

The other half of remote, for a source that promises nothing. An Open Targets query will answer differently next quarter, because it is supposed to — new evidence arrives — and a pipeline that re-asked it on every run would quietly stop being the pipeline that produced the paper.

A recorded query promises the same bytes forever, from the record. Not from the source, which is the promise remote makes and this one cannot. What makes that enough is that the canonicalised response body goes into the content-addressed store rather than merely its digest: the body is then an ordinary input with a content hash, everything below it is deterministic by construction, and a reviewer replays the whole run with no network and no API key.

The three modes, which are a setting for the run

Set by ligase run --replay and --refresh, or by ligase.remotes.configure(mode=...), and set for the run rather than for the step — because a run that replayed some of its queries and fetched the rest would be reporting a result nobody can repeat. In replay a query with no record is refused rather than fetched, and the run report always says which of the three each query did.

A refresh never overwrites a record. It writes a new dated one beside the old, so a pipeline refreshing weekly is a series of runs, each replayable forever.

What identifies a record

The request, not the order it was asked in: the method, the URL with its parameters sorted, and the body. A step making twenty paginated calls therefore writes twenty records and finds them all again next time whatever order it asks in.

Parameters

Returns

The loaded data, of the class kind named, carrying the dimensions inferred from the recorded body and a record of where it came from — which is what puts the query in the methods paragraph and its response in the reproducibility bundle.

Raises