Equivalent DNA Shouldn't Change the Answer
Reverse complements and small window shifts preserve the local sequence event. A classifier can still react to orientation or absolute position instead of biology.
strandshift enumerates both strands and a range of offsets, evaluates one prediction function across all of them, and turns disagreement into a release gate.
The Demo
Nine offsets on two strands make 18 equivalent views. Their predictions span 0.3273 around a 0.8182 consensus, so the fixture correctly fails its invariance budget.
It audits a predictor you supply. It doesn't establish biological validity, calibration, or robustness to unseen sequence families.
The Reverse-Complement Check Compared the Wrong Motif
Reverse-complementing a sequence reverse-complements everything inside it, the motif occurrence included. The same biology shows up as the reverse complement of the motif on the flipped strand, not as the original spelling. The audit was scoring reverse-complement transforms against the unmodified motif string, which asks whether the literal forward-strand spelling happens to reappear after flipping. It structurally almost never does.
So every reverse-complement transform was close to a guaranteed failure, and the audit was reporting a problem it had invented. Scored against the correctly oriented motif, the flipped strand matches perfectly at zero shift, mirroring the forward strand's own baseline: the signal really is recognizable on both strands. The published instability halved, with prediction range going from 0.7309 to 0.3273 and max deviation from 0.4145 to 0.1818, and the audit still fails, for the real reason.
An Audit That Fails Everything Isn't an Audit
Auditing this page, I noticed the tool only ever ran against its own deliberately brittle demo model. Nothing showed it could pass anything, which is the same trap as a linter that only ever demos a clean file, in reverse. And the README said it audits a supplied predictor while analyze hardcoded the bundled one, so there was no way to audit your own model at all.
analyze now takes the predictor as an argument, and the suite checks both directions. A genuinely invariant predictor, best match anywhere in the window, passes with a prediction range of exactly 0.0000. The demo model with its position bias turned off also passes. A predictor that scores only the left edge fails, and the published demo still fails with its exact numbers.
The demo's remaining failure is its deliberate position sensitivity, favoring matches near the center of the window. That's a real and separate source of brittleness from the bug above, which is why fixing one didn't paper over the other. 11 tests pass across Python 3.10 through 3.13.
Research Basis
Nucleotide Transformer highlights the growing use of foundation models across diverse genomic prediction tasks, which makes invariance checks a practical deployment requirement. Read the Nucleotide Transformer paper.
Try It
No dependencies:
git clone https://github.com/ahmeddoghri/strandshift && cd strandshift
pip install -e .
strandshift demo # 18 equivalent views, range 0.3273, audit fails
strandshift serve # workbench on :8090from strandshift.core import DEMO, analyze
analyze(DEMO, model=my_model)["audit_passed"] # the motif arrives oriented for the strand