A Bio Demo That Knows Its Own Size
Generative biology is full of huge claims from weekend side projects, which should be met with polite skepticism every single time. I wanted the opposite: a safe toy benchmark that borrows the shape of masked sequence modeling without borrowing credibility real biology hasn't earned yet.
proteinmask builds synthetic protein-like families with conserved positions, trains a profile model, fills masks, and writes toy sequences to FASTA. The README says plainly, more than once, that these are not wet-lab candidates.
Recover The Motif Or Stop Talking
The benchmark hides conserved positions in held-out sequences and checks whether the learned profile can actually recover them. That's the smallest honest version of this task, and the only one worth trusting.
The generator also tracks novelty so it can't pass the test by quietly copying the training strings. It has to preserve the motif while producing sequences that are genuinely new.
The Number
Profile motif recovery is 0.915. The random baseline is 0.000. Generated toy sequence novelty is 1.000, which means it earned that recovery number instead of memorizing its way to it.
The Random Baseline Was Not Random
Here's the awkward part of building something whose entire pitch is staying honest: I went back and actually checked the comparison baseline, the thing meant to represent a clueless guesser scoring zero, and it wasn't a guesser at all. It was a formula, a fixed letter computed from the position index, dressed up in code as a random pick.
I checked it against all five conserved positions directly. Position two's true letter is G, the formula picks R. Position five is L, the formula picks S. Every single one of the five positions comes back a miss, by construction, which is exactly why the published number always reads 0.000. A real random guess over a twenty letter alphabet does not reliably land on zero. It lands on the true letter roughly one time in twenty, about five percent, purely by chance.
I swapped the formula for an actual random draw and ran it across twenty seeds, then built a separate set of fifteen holdout seeds and checked it exactly once. Both landed right on the theoretical five percent rate, 0.054 and 0.052. The profile model's own score needed no correction at all, it comes out to the identical 0.915 either way, and it isn't circular either. It genuinely tracks the true fraction of held-out sequences that carry the fixed motif letter at each position, which is the correct thing for a profile model to have learned.
So the actual gain from the profile model over honest chance is closer to 86 points than the 91.5 the original number implied, still a large, real result, just not padded by a baseline that was rigged to never win. The original design and family code are untouched, and the published numbers still reproduce exactly.
14 tests pass locally and on GitHub Actions across Python 3.9, 3.11, and 3.13, with `artifacts/designs.fasta` regenerated by the benchmark every run.