Add the Evo 2 1B and 20B checkpoints to the leaderboard - #22
Open
Leo-T-Zang wants to merge 2 commits into
Open
Conversation
The 40B checkpoint is 76.6 GiB and takes over a minute to load, so one process per assay spends most of its time loading. Add --row_ids, which takes the same range syntax as a slurm array (0-8,11-32) and scores each assay in turn after a single load. --row_id is unchanged, so score_evo2.sh and any other caller keep working exactly as before. Two guards come with it. --require_fp8 aborts unless the constructed model reports use_fp8_input_projections=True: Evo2 falls back to bf16 for 7B checkpoints when Transformer Engine is unavailable, and it does so silently, so without the check a run can be bf16 while its provenance records FP8. The bf16 fallback is only valid for the 7B in any case, since the package refuses to build the other checkpoints without FP8. Scored CSVs are now written to a temporary file and renamed into place. A run that dies mid-write would otherwise leave a truncated CSV that the resume logic reads as a finished assay. Token budgeting also had to learn what the model actually processes. --max_tokens_per_batch divided by the raw sequence length, but a BOS token adds one position and FP8 pads the sequence dimension to a multiple of 16 inside every input projection, so an assay of length 87 runs at 96. effective_length accounts for both.
The leaderboard carried the 7B and the 40B. Adding the 1B and the 20B completes the family, so the four sizes can be read together on the same 31 ncRNA assays. evo2_20b scores 0.2129, second behind aido_rna_650m and ahead of the 40B. evo2_1b_base scores 0.1962, which needs a caveat: its Ribozyme mean is 0.0087 and the macro weights the three categories equally, so a strong tRNA mean over only three assays carries it. It is also the only base-pretrained 8k-context checkpoint here, where the 7B, 20B and 40B are mid-trained at 1M, so it is not a size point comparable to the rest. Both were scored with the packaged FP8 configuration, which is what the Evo 2 authors require for every checkpoint except the 7B. The 7B and the 40B were rescored the same way as a check and are not published: they reproduce the existing columns, the 40B bitwise on all 31 assays and the 7B to the resolution of its own float32 serialisation, so publishing them would add rows that duplicate what is already there. That reproduction is what allows the existing 7B and 40B columns to be read alongside these two as one comparable family. The row count in test_every_leaderboard_row_is_a_registered_model goes from 16 to 18 with them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes the Evo 2 family on the ncRNA fitness leaderboard. The board carried
the 7B and the 40B; this adds the 1B and the 20B, so the four sizes can be read
together on the same 31 assays.
Stacked on #21, because that PR rescored the masked models and moved the
leaderboard numbers this one has to sort against. Retarget to
v0.2once #21merges. Only the two commits here belong to this PR.
Results
evo2_20bis second overall, ahead of the 40B.Read
evo2_1b_basewith care. Its 0.1962 looks like it beats the 7B, butits Ribozyme mean is 0.0087 and the macro weights the three categories equally,
so a strong tRNA mean over only three assays carries it. It is also the only
base-pretrained 8k-context checkpoint on the board, where the 7B, 20B and 40B
are mid-trained at 1M context, so it is not a size point comparable to the rest.
Verification
The 7B and the 40B were rescored through the same pipeline as a check, and are
deliberately not published: they reproduce the existing columns, so
publishing them would add rows duplicating what is already there.
evo2_40breproduces bitwise, rank correlation 1.000000 and max|diff|0.00e+00 on all 31 assays, macro identical to every digit. Andreasson needed
its blank-mutant wild-type row set aside first, since the released file drops
that row.
evo2(7B) reproduces to the resolution of its own float32 serialisation,rank correlation 1.000000 on 24 of 30 comparable assays, median max|diff|
5.96e-08, macro 0.19036 against 0.19035.
That reproduction is the point: it is what allows the existing 7B and 40B
columns to be read alongside these two as one comparable family rather than as
numbers from two unrelated pipelines.
The leaderboard rows here were produced by rebuilding the whole table through
this branch's own
merge_scoring_files.pyandperformance_fitness.py --type ncRNA. All 16 rows already published on #21 reproduce to better than 1e-9, sothe two new rows were computed the same way and not merely appended.
Scoring configuration
Both checkpoints use the packaged FP8 configuration, which the Evo 2 authors
require for every checkpoint except the 7B. Scores are the mean per-token
log-likelihood averaged over forward and reverse-complement strands, matching
the existing evo2 entries.
Scorer changes
score_evo2_single_dms.pygains--row_ids, which takes slurm array rangesyntax and scores each assay after a single checkpoint load. The 40B is 76.6 GiB
and takes over a minute to load, so one process per assay spends most of its
time loading.
--row_idis untouched andscore_evo2.shkeeps working.Also
--require_fp8, which aborts unless the constructed model reports FP8 on.Evo2 falls back to bf16 for 7B checkpoints when Transformer Engine is missing
and does so silently, so without it a run can be bf16 while its provenance
records FP8. Scored CSVs are now written to a temporary file and renamed, so an
interrupted run cannot leave a truncated CSV that the resume logic reads as
finished. Token budgeting accounts for the BOS token and for FP8 padding the
sequence dimension to a multiple of 16, so an assay of length 87 runs at 96.
Tests
tests/test_masked_lm.pypasses, with the leaderboard row count updated from 16to 18.
tests/test_fitness.pyhas 5 errors, which are pre-existing on #21 and notintroduced here. Verified by running the same suite against a clean checkout of
four-fill-scoring: identical 5 errors. The fixture ships 8 prediction folderswhile
ALL_MODELSlists 16, and the stricter missing-model check added in #21raises on that. Fixing it belongs with #21, most simply by passing
--allow_incompletein the test.