Declaring and running steps

A step is an ordinary annotated Python function. The decorator records its signature; the graph is built from those signatures without calling any of them.

step

step(
    function: 'Callable[P,
    R] | None' = None,
    /,
    *,
    tool: 'str | Sequence[str] | None' = None,
    coordinates: 'str | None' = None,
    cite: 'str | Sequence[str] | None' = None,
    conversion: 'str | None' = None,
    stochastic: 'bool' = False,
    seed: 'str | None' = None,
) -> 'Step[P, R] | Callable[[Callable[P, R]], Step[P, R]]'

Make a function a pipeline step, without ever running it.

The signature is read once, here. Each parameter annotated with a Ligase data type becomes an edge in the graph; the rest are plain values. The return annotation is what says the step produces an Alignment, since nothing will ask the body.

Parameters

Returns

A Step, which a type checker sees as the function it decorated.

Raises

sh

sh(command: 'str') -> 'R'

Produce the output by running a shell command.

The command is not run here and the string is not formatted here: the engine fills it in once it knows where the inputs are, and runs it at whatever width the machine allows.

Placeholders name the step’s own parameters:

• {ref} — the file bound to the parameter ref.
• {reads.r1}, {reads.r2} — the mates of a paired input.
• {min_freq} — a plain parameter’s value.
• {out} — the file the step must write. A step that writes anywhere else produces nothing, as far as the engine is concerned.

Every value is substituted as exactly one shell word, so a path with a space in it stays one path. For a literal brace — an awk '{{print $1}}' — double it.

Parameters

Returns

A recipe, typed as the data the step declares, so a body reads as return lg.sh(...) under a -> Alignment annotation.

write

write(function: 'Callable[[Path], Iterable[Noticed] | None]') -> 'R'

Produce the output in Python, by writing the file yourself.

function is called with the path to write to. It is the one thing in a run that takes the GIL, so steps written this way do not run in parallel with each other — which is the right trade for an interpretation step and the wrong one for anything that does real work. Prefer lg.sh.

A body may also return what it noticed about its own work, as Noticed pairs of a note’s code and what it is about. That is the only way such a thing can be said: an assumption the work rested on need leave no trace in the file it wrote — a resistance reading made without coverage writes the same report a fully covered one does — so the body says it and the engine carries it into the run report, provenance, a bundle and the methods section. A note never fails a step; the work was done and its result stands.

Parameters

Returns

A recipe, typed as the data the step declares.

handle

handle(output: 'object') -> 'StepOutput'

The handle behind what a step returned.

A type checker sees align(reads, ref) as returning an Alignment, which is what lets it follow a pipeline and catch a mis-wired one while you are still editing. At runtime the value is a StepOutput — data that does not exist yet — and this is how to reach it without writing a cast at every call site.

Parameters

Returns

The handle, with its step, its arguments and (once a Pipeline has been built) the dimensions the output will carry.

Raises

map_identifiers

map_identifiers(
    entities: 'EntitySet',
    table: 'EntitySet',
    *,
    column: 'str',
    to: 'str',
    id_column: 'str' = 'id',
    table_id_column: 'str' = 'id',
    ambiguous: 'str' = 'drop',
) -> 'EntitySet'

Rewrite a gene list’s identifiers through a mapping table.

The output declares the space named by to, and carries every other dimension of entities — its organism, its subject, its release — because a conversion crosses one axis and is not a new list.

Parameters

Returns

A recipe, typed as the gene list the step produces.

map_ranked_identifiers

map_ranked_identifiers(
    entities: 'RankedList',
    table: 'EntitySet',
    *,
    column: 'str',
    to: 'str',
    id_column: 'str' = 'id',
    table_id_column: 'str' = 'id',
) -> 'RankedList'

Rewrite a ranked list’s identifiers, and say that its ranks moved.

map_identifiers with one difference, and the difference is the reason this is a second step rather than an argument: the output is a RankedList, and a step’s output kind is read off its return annotation once, when it is decorated. One function cannot produce both kinds.

The mapping itself is the same work and keeps the same guarantees. Rows come out in the order they went in, so rank order survives; an identifier that does not cross is named rather than dropped quietly or carried into a space it is not in (D38).

What is different, and why it needs saying

An EntitySet that loses members is still an honest set of the ones that crossed. A ranking that loses rows is not simply a shorter ranking: rank is the row position (D41), so every row below a dropped one moves up. If the top-ranked identifier had no mapping, the second is now first.

The output is a perfectly well-formed ranking of the entities that crossed, with ranks 1..n and nothing in the file to show that its rank 3 was not the input’s rank 3. So this step says it, with a ranks-renumbered note carrying what rank the output’s first row held in the input — in the run report, in provenance, in a bundle and in the methods section, like every other note.

The practical consequence that note is there to stop: take a cut-off before mapping, not after. The top ten of a mapped ranking is the top ten of what crossed, which is not the mapping of the top ten.

There is no ambiguous= here

map_identifiers takes ambiguous="fan-out" for the identifiers that genuinely denote more than one entity. A ranking cannot: rank is the row position (D41), and one row becoming two has no defensible rank — tie ranks would be invented, and renumbering would make the output a ranking of something nobody computed. So an ambiguous identifier is always left out here, and fan_out_ranked_identifiers is the step that fans a ranking: it returns an EntitySet, because that is what the data has become.

Parameters

Returns

A recipe, typed as the ranking the step produces.

fan_out_ranked_identifiers

fan_out_ranked_identifiers(
    entities: 'RankedList',
    table: 'EntitySet',
    *,
    column: 'str',
    to: 'str',
    id_column: 'str' = 'id',
    table_id_column: 'str' = 'id',
) -> 'EntitySet'

Map a ranking’s identifiers, fanning the ambiguous ones, as a set.

The declared way across a one-to-many crossing for data that arrived ranked. map_ranked_identifiers leaves an ambiguous identifier out, because rank is the row position (D41) and one row becoming two has no rank; this step emits a row per target and returns an EntitySet, because a table that has gained rows is no longer the ranking that was computed.

Why the ranking cannot be kept, stated once

Three ways to keep it were available and all three are worse. Giving the fanned rows tie ranks invents an order the pipeline never produced. Renumbering them makes rank n of the output mean something no computation assigned. Keeping the fanned entity at its original rank and pushing the rest down does both at once. Rank is the result for this kind, so a step that cannot preserve it has to say the result is gone rather than hand back something that looks like one.

The function’s name is that declaration: there is no ambiguous= argument, because choosing it on a ranking would change the output’s type, and a step’s output kind is read off its return annotation once, when it is decorated.

What to do with the order you had

Take any cut-off — a top ten, a significance line — on the ranking before this step, and fan the members of that cut-off. The top ten of a fanned set is not a thing; the fan-out of the top ten is.

Parameters

Returns

A recipe, typed as the gene list the step produces — not a ranking.

Pipeline

class Pipeline(
    outputs: 'Dataset | Iterable[Dataset]',
    *,
    claims: 'Sequence[Claim]' = (),
) -> 'None'

A graph of steps, built from its outputs and checked before it runs.

Construction walks back from the outputs through the handles each step was given, so a file passed to two steps becomes one node rather than two. Nothing is executed and nothing is read: the graph is built from signatures and from the dimensions the inputs were loaded with.

Pipeline.audit

audit(
    self,
    step: 'str',
    *,
    sample: 'str | None' = None,
    jobs: 'int | None' = None,
    cache: 'str | Path | None' = None,
) -> 'AuditReport'

Run one step twice and report whether it is reproducible.

The pipeline is run first, normally and through the cache, so that the step being audited has its inputs. Then each node of that step is run twice more, into scratch files that are thrown away, and the two outputs are compared both as bytes and as meaning.

Three verdicts, because two would not do. A step whose bytes move while its result does not — which is every tool that records its own command line — is reproducible, not a failure. Only a step whose result moves is not reproducible.

Parameters

Returns

What two runs of each node showed.

Raises

Pipeline.check

check(self) -> 'None'

Type-check the whole pipeline. Nothing is computed.

Every problem is found before anything is raised, so the raised error’s errors attribute holds all of them.

Raises

Pipeline.claims

claims: tuple[Claim, ...]  # property

The claims this pipeline declares, as they were written.

Pipeline.errors

errors(self) -> 'list[CheckError]'

Return every problem in this pipeline, without raising.

Returns

One error per problem, in the order the steps run. Empty when the pipeline is consistent. Nothing is computed.

Pipeline.methods

methods(
    self,
    *,
    bundle: 'str | Path | None' = None,
    thin: 'bool' = False,
    verify: 'str | Path | None' = None,
    include: 'Sequence[str | Path]' = (),
    jobs: 'int | None' = None,
    cache: 'str | Path | None' = None,
) -> 'Methods'

Say how this pipeline’s results were produced.

The pipeline is run first, normally and through the cache, because a provenance record is filed under a step’s key and computing keys is what a run does. A pipeline already computed costs a cache lookup per step, not a recomputation.

What comes back names, for every step: the tools and the versions they ran at, the parameters as they were hashed, the command as written, and every reference by accession and checksum. A step whose output predates provenance is reported as a gap rather than left out, because nothing can reconstruct how it was produced — the command that would have said so is deliberately not in the hash.

Parameters

Returns

How every step was produced, with the bundle written and the verification done, where either was asked for.

Raises

Pipeline.report

report(self) -> 'Report'

Check every step and return everything found, without raising.

Pipeline.run

run(
    self,
    *,
    jobs: 'int | None' = None,
    force: 'bool' = False,
    cache: 'str | Path | None' = None,
    worker: 'str | bool' = False,
    lease: 'int | None' = None,
) -> 'RunReport'

Check the pipeline, then compute whatever is not already known.

The pipeline is checked first, every time: running an unchecked pipeline is how an overnight run ends in a dimension error.

A step runs when something that could change its result has changed — its code, a parameter, an input’s contents, or the version of a tool it calls — and not otherwise. Independent steps run at once, and a step whose rerun produced the same output as before leaves everything below it alone.

Parameters

Returns

What every step did, and where its output is.

Raises

Pipeline.steps

steps: tuple[str, ...]  # property

The steps this pipeline runs, in dependency order.

Step

class Step(
    function: 'Callable[P,
    R]',
    *,
    tool: 'str | Sequence[str] | None' = None,
    coordinates: 'str | None' = None,
    cite: 'str | Sequence[str] | None' = None,
    conversion: 'str | None' = None,
    stochastic: 'bool' = False,
    seed: 'str | None' = None,
) -> 'None'

A function that records a call instead of running.

Attributes

Step.dataset_parameters

dataset_parameters: tuple[Parameter, ...]  # property

The parameters that take data, in declaration order.

Order matters: a dimension is inherited from the first argument that states it.

Step.name

name: str  # property

The step’s name, as it appears in messages.

Step.plain_parameters

plain_parameters: tuple[Parameter, ...]  # property

The parameters that take plain values, in declaration order.

Shell

class Shell(command: 'str') -> None

A shell command, with the step’s inputs still to fill in.

Attributes