SAWA: an evaluation in progress

SAWA is a sibling project working on HIV drug-resistance surveillance. It is evaluating whether to use Ligase’s resistance interpretation in place of calling out to Stanford’s Sierra service. This is not a decision to adopt, and SAWA is not an independent third party — it shares an author with Ligase.

It is on this page for one reason: somebody tried to use Ligase for real work and immediately hit a way in which Ligase was silently wrong. That is worth more than a feature list, and burying it would be the wrong instinct. The defect below is fixed; the account of it stays, because what a project does when it is caught being wrong is the more useful thing to read.

What it found: pol-relative coordinates were read as whole-genome ones

Ligase’s lg.hivdb.interpret expects positions in HXB2 K03455 whole-genome coordinates, with the gene windows for protease, reverse transcriptase and integrase fixed in the engine.

SAWA’s sequences are pol-relative. Its reference spans K03455 2085–5096, so every position carries an offset of 2084.

Handed a pol-relative reference, Ligase did not complain. It read what is actually reverse transcriptase and integrase as though it were protease, and reported intermediate protease-inhibitor resistance — a clinically meaningful, entirely wrong result, with no error and no warning.

This is exactly the class of bug the rest of this site is about, in Ligase itself. A coordinate system that the data does not state, assumed by a consumer, with plausible output when the assumption is wrong.

What was done about it

A reference now carries the coordinate frame its positions are numbered along, as part of its identity, and the resistance step states the frame its gene windows are positions in. The two are compared before any compute, and a reference that is not that frame — or that cannot be shown to be — is a ReferenceFrameError rather than a report. D33 argues the design, including why the frame is declared rather than inferred.

If you are using lg.hivdb.interpret

A whole-genome HXB2 reference needs nothing: 9,719 bases identifies it. A pol-only reference is padded to its true offset and declared —


ref = lg.reference("hxb2_padded.fasta", organism="HIV-1", frame=("K03455", 1))

— where hxb2_padded.fasta is "N" * 2084 + hxb2_pol. The padding reads as unsequenced, so capsid comes back not assessed rather than susceptible, which is the honest answer for a pol amplicon.

Other findings so far

Mutation names are relative to the supplied reference, not to HIVdb consensus B. Scoring HXB2 against itself produces a name like N232N — a position called as itself — with a score that agrees with Stanford while the name does not. Sierra emits no such names.

A barely-covered gene read as susceptible. Where a sequence covered only part of a gene, Ligase scored the gene anyway. A gene read in part refutes nothing and shows nothing, so every drug landed at zero, which the algorithm’s own ranges read as susceptible: SAWA saw a sequence covering 32% of integrase reported as five susceptible integrase inhibitors, where Stanford’s own service declines to report the gene. This was the dangerous direction for a resistance report, and it is fixed — a gene read below a stated share of the codons the algorithm scores is now reported as not assessed (D34).

The input shape is the fit problem, not the scoring. lg.hivdb.interpret takes a VCF plus a reference FASTA, optionally with a BAM for coverage. SAWA holds consensus sequences. The scoring engine is not what is in question — it reproduces 3,348 Stanford expectations exactly, as the HIV-1 worked case describes.

What is not here

The evaluation has run a comparison across SAWA’s own cohort, and this page deliberately does not quote its agreement figures. That work belongs to SAWA, is not yet published by SAWA, and is theirs to report. When it is published, this page will link to it.

What can be said is that the evaluation’s recommendation, as it stands, is conditional rather than either an adoption or a rejection.