Remote inputs

crates/ligase-core/src/remote/, crates/ligase-core/src/store.rs, and the remotes table in the cache’s database. The decision is D39; this page is how it is built and what was given up to build it that way.

Six files

Nothing above pinned::resolve and recorded::resolve knows about HTTP, and nothing in fetch.rs knows about digests or kinds. That seam is what lets a test take the network away without stubbing anything the engine actually does.

The two resolvers sit side by side rather than one calling the other, because they answer different questions. pinned::resolve asks “are these the bytes that were promised?” and fails if not; recorded::resolve asks “do we already have an answer to this?” and records one if not. Folding them into a function with a flag would put the two reproducibility claims D39 exists to separate back into one code path.

Resolution, in order

locator ──▶ parse ──▶ index hit? ──▶ store hit? ──▶ digest satisfies? ──▶ done, no network
                             │              │                 │
                             └──────────────┴─────────────────┴──▶ fetch
                                                                     │
                                            stage ─▶ hash ─▶ canonicalise ─▶ compare
                                                                     │
                                                      mismatch ──────┴────── match
                                                          │                    │
                                                    forget the record    store and record

Two orderings matter.

The digest is checked before anything is stored. A fetch that was not what the pin promised leaves the cache exactly as it found it: no object under a name that claims to be verified, and no row a later run would serve without asking. FsStore::put_file_as is the half of that in the store — a staged file is refused if it is not what the key says, so an interrupted download cannot acquire a whole file’s name.

A record that does not satisfy the declaration is not the answer. If the pipeline pins D and the index holds D', the index is ignored and the source is asked. That is the case where you bumped a pin because the source legitimately changed, and serving the old bytes would be the opposite of what you asked for.

Why resolution is eager

A remote resolves when the pipeline declares it, during the import of the pipeline file — not in a phase between graph-building and execution.

The reason is the type system. A remote’s dimensions cannot be known until its bytes are in hand, and check() compares dimensions before any compute. A lazy handle would mean a Reference whose dimensions are unknown until later, which is a hole in the one invariant the engine has.

The cost is real and worth naming: ligase check on a cold store reaches the network. On a warm one it reaches nothing. The benefit is that “a wrong digest fails before any step runs” is not a feature that had to be built — there is no graph yet when it fails.

It also forces one piece of plumbing. The cache directory has to be known before the pipeline file is imported, so --cache is applied by main in cli.py rather than passed to pipeline.run(). That is what ligase.remotes exists for: a per-run setting, set once, read during the import.

The transport trait

pub trait Transport: Send + Sync {
    fn get(&self, url: &str, into: &Path) -> Result<Fetched, FetchError>;
}

Two implementations. Network wraps a ureq agent. Offline always fails, with a message saying what to do — and it is a transport rather than a flag threaded through the resolver precisely so that an offline run exercises the same resolution path as any other, minus the fetch.

What the client is for

Restraint, not speed:

• A 404 is retried zero times; a 503 or 429 is retried with backoff. Retrying something that will not change wastes time and, against a rate-limited archive, spends a budget the next real request needs.
• A redirect is followed, bounded, and the URL that answered is recorded separately from the locator. Two different facts: the locator is what you asked for and belongs in a methods paragraph; the final URL is what replied and belongs in a record of the fetch.
• Nothing is buffered. An SRA run is gigabytes; it streams to a staged file, and a body one byte over the limit is refused without being read to its end.

An auth header has nowhere to go

“A request made with an auth header stores no trace of it” is not implemented as a filter. Fetched has no field for request headers, and neither does anything derived from it — so there is nothing to strip. That is the only version of the guarantee a later field cannot quietly undo, and a test asserts over the whole record rather than over a list of header names.

Why ureq and not reqwest

The feature spec named reqwest. Measured against this workspace’s actual tree, reqwest with blocking and rustls adds 82 crates; ureq with rustls and gzip adds 16.

The deciding argument is the install story, and it is D11 again. reqwest’s rustls feature resolves to aws-lc-rs, which pulls aws-lc-sys and cmake — a new build-tool requirement, and a wheel that needs cmake does not build on a machine without one. ureq resolves to ring, which needs only a C compiler, and rusqlite’s bundled SQLite already requires one. So ureq adds no toolchain requirement at all.

Second: blocking is what the engine wants. The executor is a rayon thread pool that deliberately never touches an async runtime. reqwest::blocking starts a tokio runtime on a background thread to serve calls that are synchronous anyway.

The GIL

The fetch releases it. Holding it through a minutes-long SRA download would freeze every other Python thread, and when the tests first ran it deadlocked outright against a local server the test was fetching from: the server’s thread could not run while the fetch it was answering held the lock. Found by writing the test, which is the only way that one gets found.

What the cache remembers

A remotes table, keyed on (locator, kind) — because the same URL read as two kinds is two facts, since a digest is over what a file means and what it means depends on how it is read.

Keeping both digests is what lets a source that rewrote a file without changing its meaning be noticed rather than either ignored or failed on. It is detectable when the objects have gone from the store and the row has not — a cleared CI cache, say — which is the case that actually happens.

files is tab-separated and readable with sqlite3, because a cache whose whole purpose is explaining a result should be explicable itself.

A pinned remote’s row is replaced by a later fetch of the same locator and kind. That is right here — a later fetch is either the same answer or a failure — and it is emphatically not how a recorded query behaves, which is why they are two tables rather than one with a flag.

The records table

Keyed on (request, revision), where request is the hash of the canonical request.

Two things about that key are load-bearing.

revision and not recorded_at. A refresh must never overwrite what it refreshes, and keying on a timestamp would let two refreshes inside the same second collide — which is to say it would rest “never overwrite” on a clock. The revision is assigned by Cache::record_query rather than by the caller, so getting it right is not something each call site has to remember.

No supersedes column. Revision n supersedes n − 1 by construction, and a stored pointer saying the same thing is a second source of truth that can drift from the first.

Bodies are never collected, so every revision stays replayable: a record’s body lives in the write-once blob store under its own hash, and two revisions of identical bytes cost one file.

A refresh that finds the same bytes writes no new revision. “Never replaces” is kept by never writing over anything, not by writing something every time — and a row per week recording bytes already recorded would buy a growing table and no replay.

Which record is which

remote/request.rs holds two types that look redundant and are not.

pub struct Request { method, url, body, auth }       // what is sent
pub struct CanonicalRequest { method, url, body }    // what is hashed

Request::url is the URL as written. CanonicalRequest::url is that URL with its query parameters sorted and its fragment dropped.

They are two fields because a server may care about parameter order and the user wrote what they wrote: sorting decides which record this is and must never decide what to ask. Sorting is over whole name=value strings rather than by name, so ?id=a&id=b keeps both values — it asks for two things and is not ?id=a.

The fragment goes because it is never sent. The host’s case is left alone: two spellings of one host are two records, which costs a fetch and cannot produce a wrong answer, where folding them would be the engine deciding that two URLs written differently are the same.

method and body are hashed although the transport can only issue a GET with no body. That is not dead weight — hashing all three now means a later POST, which Open Targets’ GraphQL would need, cannot change the identity of a record written today. A POST handed to Network is refused rather than downgraded to a GET, which would ask a different question and then record the answer as though it were the one wanted.

auth is on Request and on nothing else, which is the whole of the secret guarantee: see below.

What a recorded response means

remote/response.rs, and it is thin on purpose. The body is parsed as JSON; serde_json’s map is a BTreeMap with preserve_order off, so re-serialising sorts the keys at every depth without a traversal of our own to get wrong. Arrays keep their order, because there order is the answer.

arbitrary_precision is on, so every number is kept as the text the server sent. Without it a twenty-digit identifier would be rounded through an f64 on its way into the store and the stored body would differ from the answer with nothing reporting it. The cost is that 1 and 1.0 hash differently, which is the conservative answer: the engine does not claim an equality it has not verified.

A body that is not JSON is returned as it arrived, which needs no rule because D21 already says canonicalisation falls back to the bytes.

Volatile fields are dropped from the top level of a JSON object and nowhere else. The closed set is requestId, traceId, correlationId, timestamp, generatedAt, servedAt, retrievedAt, took, elapsed and queryTime, matched against a key lower-cased with - and _ removed so that four spellings of one name are one name.

Dropping them at all is D22: a generatedAt says how the answer was served, which is a signature rather than part of the answer. Dropping them only at the top level is D26: inside a data record the same key is what the record says, and recursing would silently delete a time series. The top level is the one place the two cases can be told apart, and each drop emits a note in case the key was the user’s own data after all.

Response headers need no dropping. Fetched has no field for them, so a rate-limit header never reaches anything the engine stores. The guarantee is already structural, which is why there is no filter for it.

Note the layering. This produces the bytes that go into the store; the stored body is then read as its declared kind by the ordinary loader. A query declared EntitySet over a TSV endpoint canonicalises to its bytes here and is parsed as a TSV there. No kind in v0.2 read JSON, so the JSON branch was exercised by unit tests rather than end to end. M15 closed that: a KnowledgeGraph reads JSON, and four tests in tests/test_queries.py now take a graph through the recorder, into the store and back out through the loader. The one that matters most is test_a_servers_signature_does_not_make_a_graph_a_new_graph — the same graph served a minute later, with a different generatedAt, is the same graph, and without the drop above every run would see drift and everything below the query would rerun.

A secret has nowhere to be stored

Request carries its Auth, because the token has to be sent. Nothing else does: Fetched has no field for a request header, nor does QueryRecord, nor does records.tsv. An Authorization header is therefore not stripped before storage — it is never representable in anything the engine stores, which is the only version of the guarantee a field added later cannot quietly undo.

Two doors are closed deliberately beyond that.

Auth’s Debug is hand-written and prints Auth(<redacted>), because a derived one would print the token into any message that formatted a request.

And the Python surface takes auth_env=, the name of an environment variable, rather than a token. A bundle copies the pipeline file into pipeline/, so a literal token in the pipeline would travel in every bundle — a leak the structural argument above does not cover, because it is in the pipeline’s source rather than in the record.

The test for all of it asserts over every file in the store and every file in the bundle rather than over a list of header names, so a future field is caught rather than trusted.

How a bundle replays

A bundle carries records.tsv and a records/ directory of bodies named by their own hashes, so the table and the files need no agreement about naming. Each body is read with checked_get on the way out: a bundle that carried a corrupted body would reproduce the wrong answer and report that it reproduced.

bundle::seed puts them into a cache — the bodies into the blob store, the rows into records — and ligase methods --verify calls it before the pipeline is imported, because a query resolves while the pipeline file is being read. That ordering is the whole of why seeding is not done inside methods(): by then the query has already resolved or failed.

This is the precise sense in which provenance.py’s “a bundle cannot fetch its own data” is retired. For a recorded query the bundle is the source. A pinned remote is unchanged and deliberately so: it is named rather than carried, because anyone can obtain it and check it against its digest, and copying gigabytes of reads into every bundle would buy nothing.

The store trait, and the shape M14 needs

pub struct ObjectKey { pub hash: ContentHash, pub suffix: String }
 
pub trait BlobStore {
    fn has(&self, key: &ObjectKey) -> Result<bool, StoreError>;
    fn get(&self, key: &ObjectKey) -> Result<Option<Vec<u8>>, StoreError>;
    fn put(&self, key: &ObjectKey, bytes: &[u8]) -> Result<(), StoreError>;
    fn put_file(&self, staged: &Path, suffix: &str) -> Result<ObjectKey, StoreError>;
    fn put_file_as(&self, staged: &Path, key: &ObjectKey) -> Result<(), StoreError>;
    fn local_path(&self, key: &ObjectKey) -> Result<Option<PathBuf>, StoreError>;
    fn checked_get(&self, key: &ObjectKey) -> Result<Option<Vec<u8>>, StoreError> { … }
}

D39 names three operations. Three were not enough, and the two additions are not conveniences.

A key carries a suffix. The existing store already wrote objects as store/ab/cdef….vcf.gz, and the suffix is functional: a step’s output is handed to bcftools as a path, and those tools dispatch on the extension. A trait keyed on the hash alone could not name the file it was asked for. An object store handles a suffixed key as happily as a filesystem does, so nothing is given up. The alternative — a pure hash→bytes tier with the local backend hard-linking a suffixed name on demand — is cleaner on paper and costs a second name for every object.

local_path is the escape hatch a filesystem has and an object store does not. It returns None for a backend with no local copy, so M14’s two-tier store answers by fetching into its local tier first and returning that path — which is what keeps “a local store is always the first tier” expressible.

checked_get is a provided method, not a backend’s job. D39’s rule that nothing read from a store is trusted must not be something each backend author remembers separately: an object’s name is a claim about its contents, and a wrong answer served quickly from a shared cache is worse than no cache at all. So the backends implement the unchecked primitives and the trait does the verifying.

Trade-offs, stated

ligase check can reach the network. On a cold store, resolving a remote is a fetch. The alternative was a dimension-less handle, which is worse.

A fetched body must parse as its kind. A local file that will not parse falls back to its bytes (D21); a fetched one is refused. The asymmetry is deliberate: a local file is one you chose, and a fetched one is whatever an archive returned — and the thing it returns when it has nothing is an HTML error page with a 200 on it. Refusing it is the only guard that works when no digest was declared.

A remote is recorded against every step that uses it. One fetch handed to three steps is three rows, de-duplicated by the methods report rather than at write time. Consistent with how references already works: a provenance record is about one computation.

The kind check is only as strong as the kind’s parser. A fetched body must read as its declared kind, which catches an HTML error page served as a Reference. It does not catch one served as an EntitySet, because a single-column TSV is a valid gene list and an error page is a gene list of one implausible gene. That is uniform across local files, pinned remotes and recorded queries rather than something a query introduced, and it is pinned by a test (an_error_page_passes_the_kind_check_for_an_entity_set_and_this_is_why) so the day the entity-set loader learns to refuse an identifier containing a <, something says so.

A recorded body is held in memory while it is canonicalised. Up to 32 MiB, which is the cap’s second job: a streamed canonicalisation would let the cap go and buy nothing, since the JSON has to be parsed whole to sort its keys. A pinned remote still streams to disk and is never buffered, which is why the two caps are different numbers for different reasons.

The dedup rule for source_set was settled here and implemented in M15. See D36 and the M10 milestone. No kind in v0.2 applied that axis — M8 and M12 each declined it with reasons, and a remote is not a kind at all — so the split travelled to M15, where a KnowledgeGraph applies it and SourceSet became a type of its own with a case-sensitive equality. Shipping an unexercised value type would have been the condition M7’s standing requirement exists to prevent, which is why the type waited for the kind rather than landing with the dimension.