Provenance and auditing

What a run recorded about itself: the versions every tool actually ran at, every parameter as it was hashed, and the references by accession and checksum — and, the other way round, what a number you published rests on.

Methods

class Methods(
    steps: 'list[StepMethods]' = <factory>,
    gaps: 'list[MethodsGap]' = <factory>,
    references: 'list[PinnedReference]' = <factory>,
    citations: 'list[str]' = <factory>,
    uncited: 'list[str]' = <factory>,
    claims: 'list[ClaimCheck]' = <factory>,
    engines: 'list[str]' = <factory>,
    platforms: 'list[str]' = <factory>,
    summary: 'str' = '',
    text: 'str' = '',
    bundle: 'Bundle | None' = None,
    verification: 'Verification | None' = None,
) -> None

How a set of results was produced.

Attributes

Methods.accounted_for

accounted_for: bool  # property

Whether every step’s production is recorded and every claim bound.

False means the write-up has a hole in it and says so. A number the analysis does not support is such a hole: a property with this name that returned True for one would be making the engine’s own promise falsely.

Methods.tools

tools: list[tuple[str, str]]  # property

Every tool any step used, with the version it ran at.

Methods.unbacked

unbacked: list[ClaimCheck]  # property

The claims this write-up cannot stand behind.

Contradicted ones included: a number the run positively disagrees with is not a number the write-up stands behind.

StepMethods

class StepMethods(
    step: 'str',
    nodes: 'int',
    scopes: 'list[str]',
    scope_kind: 'str | None',
    tools: 'list[tuple[str,
    str]]',
    parameters: 'list[tuple[str,
    str]]',
    command: 'str | None',
) -> None

One way a step was run.

One written step becomes one node per sample, and all of them are one entry here. Where two nodes of a step genuinely were run differently they are two entries, because then the difference is the thing worth reporting.

Attributes

MethodsGap

class MethodsGap(step: 'str', scopes: 'list[str]', nodes: 'int') -> None

A step nothing is known about.

Its output is in the cache from a run that recorded no provenance, and nothing can reconstruct how it was produced: the command that would have said so is not in the hash, by design. ligase run --force runs it again and records it.

Attributes

Claim

class Claim(
    text: 'str',
    step: 'str',
    scope: 'str | None' = None,
    parameter: 'str | None' = None,
    tool: 'str | None' = None,
    value: 'str | float | None' = None,
    at: 'Cell | None' = None,
    tolerance: 'float | None' = None,
) -> None

A number or a phrase a paper states, and where it came from.

pipe = lg.Pipeline( outputs, claims=[ lg.Claim( “variants were called above 1% allele frequency”, step=“call”, parameter=“min_freq”, value=0.01, ), ], )

text and value are separate on purpose. A paper writes 1%; the pipeline holds 0.01. Comparing the sentence against the parameter would be a guess about arithmetic in English, so the sentence is what a reader greps the write-up for and value is what the engine compares — written the way the pipeline holds it.

Everything except text is checked against the run: the step is one the write-up accounts for, the sample is one it ran on, the parameter or tool is one the record holds, and the value agrees. A claim that survives all of that is bound to the step key and the output hash behind it; one that does not is reported rather than reprinted, because a declaration nothing can check is a comment.

A claim never enters a cache key, so editing one reruns nothing.

Raises

Attributes

Claim.located

located: tuple[float, str, int, float] | None  # property

The number, the cell and the tolerance, as the engine takes them.

None for a claim that names no cell, which is most claims. __post_init__ has already refused every incomplete combination, so this reads the three parts without checking them again.

Claim.rendered

rendered: str | None  # property

The value as the engine compares it: text, however it was given.

value=0.01 is what a person writes and 0.01 is how the parameter was hashed, so one str() is the whole conversion. A number in a paper is text.

None for a located claim, whose value is a number the engine compares numerically rather than a string the record holds.

ClaimCheck

class ClaimCheck(
    claim: 'Claim',
    verdict: 'str',
    reason: 'str',
    recorded: 'str | None',
    bound: 'list[BoundTo]' = <factory>,
) -> None

One claim, and what the run had to say about it.

Attributes

ClaimCheck.is_bound

is_bound: bool  # property

Whether the write-up can stand behind the claim.

True for a verified claim as well as a bound one: verified is bound and more, so a property that excluded it would make accounted_for false for the claims the engine is most sure of.

ClaimCheck.is_verified

is_verified: bool  # property

Whether the number was read out of the output and agreed.

Cell

class Cell(column: 'str', row: 'int' = 0) -> None

A cell of a step’s output: a column by name, a row by position.

lg.Claim( “malaria F1 = 0.333 against seven curated targets”, step=“validate”, value=0.333, at=lg.cell(column=“f1”, row=0), tolerance=5e-4, )

Row 0 is the first row after the header, because that is the first row of data and a claim is about data. A ranking’s top row is therefore row 0, which is the row its own rank 1 is.

The engine reads the place it is given and nowhere else. It will not go looking for a number in output it was not pointed at, and it parses one format — a TSV with a header — rather than guessing at others. A claim that cannot say where its number is stays step-only, which is a legitimate state and not an error.

Attributes

cell

cell(*, column: 'str', row: 'int' = 0) -> 'Cell'

Name the cell of a step’s output a claim’s number lives in.

Parameters

Returns

A location a Claim can carry in at=.

BoundTo

class BoundTo(
    scope: 'str | None',
    key: 'str',
    canonical: 'str',
    recorded_at: 'int',
    platform: 'str',
) -> None

One computation a claim rests on.

The step key, not the occasion. A key is the identity of a computation — the same code, parameters, inputs and tool versions — so a step served from the cache backs its claim completely, and what this adds is when that computation happened, which may be long before the run that wrote the claim up.

Attributes

Bundle

class Bundle(
    root: 'str',
    files: 'list[str]',
    steps: 'int',
    gaps: 'int',
) -> None

A reproducibility bundle that was written.

Attributes

Verification

class Verification(
    results: 'list[ResultCheck]' = <factory>,
    inputs: 'list[InputCheck]' = <factory>,
    reproduced: 'bool' = False,
    verdict: 'str' = 'missing',
    summary: 'str' = '',
    text: 'str' = '',
) -> None

What checking a bundle against this machine found.

Attributes

Verification.differing

differing: list[ResultCheck]  # property

The results that did not come out the same.

Verification.moved

moved: list[InputCheck]  # property

The inputs that are not the ones the bundle was built from.

Worth reading before any differs verdict: a changed input explains a changed output, and most of a bundle’s inputs are named rather than carried — a local file by content hash, a pinned remote by locator and digest — so they are the ones that can have moved. A recorded query cannot appear here, because its response body travels in the bundle and is replayed from it.

PinnedReference

class PinnedReference(
    param: 'str',
    path: 'str',
    build: 'str | None',
    accession: 'str | None',
    contigs: 'int',
    length: 'int',
    checksum: 'str | None',
    fingerprint: 'str',
) -> None

A reference sequence, pinned as firmly as its file allows.

Nothing here is declared or guessed. The accession is the sequence name where the name is one — K03455.1 carries its own version — and the checksum is the MD5 the SAM specification defines for @SQ M5, over the uppercased sequence with whitespace removed.

Attributes

AuditReport

class AuditReport(
    steps: 'list[StepAudit]' = <factory>,
    failures: 'list[StepFailure]' = <factory>,
    verdict: 'str' = 'deterministic',
    summary: 'str' = '',
    text: 'str' = '',
) -> None

What an audit found.

Attributes

AuditReport.reproducible

reproducible: bool  # property

Whether every node audited agreed with itself.

AuditReport.unreproducible

unreproducible: list[StepAudit]  # property

The nodes whose two runs disagreed about the result.

StepAudit

class StepAudit(
    step: 'str',
    sample: 'str | None',
    scope: 'str | None',
    verdict: 'str',
    first: 'str',
    second: 'str',
    first_canonical: 'str',
    second_canonical: 'str',
    first_duration_ms: 'int',
    second_duration_ms: 'int',
    node: 'int',
    seed: 'tuple[str,
    str] | None' = None,
) -> None

What two runs of one step showed.

Attributes

StepAudit.reproducible

reproducible: bool  # property

Whether the two runs agreed on the result.

True for a step whose bytes moved and whose meaning did not: that is what reproducible means here.

Report

class Report(
    errors: 'list[CheckError]',
    summary: 'str',
    text: 'str',
) -> None

What a check found.

Attributes

RunReport

class RunReport(
    steps: 'list[StepRun]' = <factory>,
    failures: 'list[StepFailure]' = <factory>,
    summary: 'str' = '',
    text: 'str' = '',
    hashed: 'int' = 0,
    prefiltered: 'int' = 0,
    canonicalised: 'int' = 0,
    worker: 'str | None' = None,
    divergences: 'list[Divergence]' = <factory>,
) -> None

Everything a run did.

Attributes

RunReport.cached

cached: list[StepRun]  # property

The steps that were served from the cache.

RunReport.outputs

outputs(self, step: 'str') -> 'list[str]'

Where one step’s outputs are, in graph order.

Parameters

Returns

One path per node of that name.

RunReport.ran

ran: list[StepRun]  # property

The steps that actually ran, in graph order.

RunReport.skipped

skipped: list[StepRun]  # property

The steps an upstream rerun turned out not to change.

StepRun

class StepRun(
    step: 'str',
    sample: 'str | None',
    outcome: 'str',
    output: 'str',
    key: 'str',
    hash: 'str',
    canonical: 'str',
    duration_ms: 'int',
    node: 'int',
    notes: 'tuple[str,
    ...]' = (),
) -> None

One step that produced an output.

Attributes

StepRun.ran

ran: bool  # property

Whether this step actually ran, rather than being served.