Skip to content

Add the Evo 2 1B and 20B checkpoints to the leaderboard - #22

Open
Leo-T-Zang wants to merge 2 commits into
four-fill-scoringfrom
evo2-family-scoring
Open

Add the Evo 2 1B and 20B checkpoints to the leaderboard#22
Leo-T-Zang wants to merge 2 commits into
four-fill-scoringfrom
evo2-family-scoring

Conversation

@Leo-T-Zang

Copy link
Copy Markdown
Collaborator

Completes the Evo 2 family on the ncRNA fitness leaderboard. The board carried
the 7B and the 40B; this adds the 1B and the 20B, so the four sizes can be read
together on the same 31 assays.

Stacked on #21, because that PR rescored the masked models and moved the
leaderboard numbers this one has to sort against. Retarget to v0.2 once #21
merges. Only the two commits here belong to this PR.

Results

checkpoint Ribozyme tRNA Aptamer macro
aido_rna_650m 0.0660 0.4894 0.0934 0.2163
evo2_20b 0.1105 0.4345 0.0937 0.2129
evo2_40b 0.1081 0.4310 0.0970 0.2120
evo2_1b_base 0.0087 0.4466 0.1334 0.1962
evo2 (7B) 0.0651 0.3867 0.1192 0.1904

evo2_20b is second overall, ahead of the 40B.

Read evo2_1b_base with care. Its 0.1962 looks like it beats the 7B, but
its Ribozyme mean is 0.0087 and the macro weights the three categories equally,
so a strong tRNA mean over only three assays carries it. It is also the only
base-pretrained 8k-context checkpoint on the board, where the 7B, 20B and 40B
are mid-trained at 1M context, so it is not a size point comparable to the rest.

Verification

The 7B and the 40B were rescored through the same pipeline as a check, and are
deliberately not published: they reproduce the existing columns, so
publishing them would add rows duplicating what is already there.

  • evo2_40b reproduces bitwise, rank correlation 1.000000 and max|diff|
    0.00e+00 on all 31 assays, macro identical to every digit. Andreasson needed
    its blank-mutant wild-type row set aside first, since the released file drops
    that row.
  • evo2 (7B) reproduces to the resolution of its own float32 serialisation,
    rank correlation 1.000000 on 24 of 30 comparable assays, median max|diff|
    5.96e-08, macro 0.19036 against 0.19035.

That reproduction is the point: it is what allows the existing 7B and 40B
columns to be read alongside these two as one comparable family rather than as
numbers from two unrelated pipelines.

The leaderboard rows here were produced by rebuilding the whole table through
this branch's own merge_scoring_files.py and performance_fitness.py --type ncRNA. All 16 rows already published on #21 reproduce to better than 1e-9, so
the two new rows were computed the same way and not merely appended.

Scoring configuration

Both checkpoints use the packaged FP8 configuration, which the Evo 2 authors
require for every checkpoint except the 7B. Scores are the mean per-token
log-likelihood averaged over forward and reverse-complement strands, matching
the existing evo2 entries.

Scorer changes

score_evo2_single_dms.py gains --row_ids, which takes slurm array range
syntax and scores each assay after a single checkpoint load. The 40B is 76.6 GiB
and takes over a minute to load, so one process per assay spends most of its
time loading. --row_id is untouched and score_evo2.sh keeps working.

Also --require_fp8, which aborts unless the constructed model reports FP8 on.
Evo2 falls back to bf16 for 7B checkpoints when Transformer Engine is missing
and does so silently, so without it a run can be bf16 while its provenance
records FP8. Scored CSVs are now written to a temporary file and renamed, so an
interrupted run cannot leave a truncated CSV that the resume logic reads as
finished. Token budgeting accounts for the BOS token and for FP8 padding the
sequence dimension to a multiple of 16, so an assay of length 87 runs at 96.

Tests

tests/test_masked_lm.py passes, with the leaderboard row count updated from 16
to 18.

tests/test_fitness.py has 5 errors, which are pre-existing on #21 and not
introduced here. Verified by running the same suite against a clean checkout of
four-fill-scoring: identical 5 errors. The fixture ships 8 prediction folders
while ALL_MODELS lists 16, and the stricter missing-model check added in #21
raises on that. Fixing it belongs with #21, most simply by passing
--allow_incomplete in the test.

The 40B checkpoint is 76.6 GiB and takes over a minute to load, so one process
per assay spends most of its time loading. Add --row_ids, which takes the same
range syntax as a slurm array (0-8,11-32) and scores each assay in turn after a
single load. --row_id is unchanged, so score_evo2.sh and any other caller keep
working exactly as before.

Two guards come with it. --require_fp8 aborts unless the constructed model
reports use_fp8_input_projections=True: Evo2 falls back to bf16 for 7B
checkpoints when Transformer Engine is unavailable, and it does so silently, so
without the check a run can be bf16 while its provenance records FP8. The bf16
fallback is only valid for the 7B in any case, since the package refuses to
build the other checkpoints without FP8.

Scored CSVs are now written to a temporary file and renamed into place. A run
that dies mid-write would otherwise leave a truncated CSV that the resume logic
reads as a finished assay.

Token budgeting also had to learn what the model actually processes.
--max_tokens_per_batch divided by the raw sequence length, but a BOS token adds
one position and FP8 pads the sequence dimension to a multiple of 16 inside
every input projection, so an assay of length 87 runs at 96. effective_length
accounts for both.
The leaderboard carried the 7B and the 40B. Adding the 1B and the 20B completes
the family, so the four sizes can be read together on the same 31 ncRNA assays.

evo2_20b scores 0.2129, second behind aido_rna_650m and ahead of the 40B.
evo2_1b_base scores 0.1962, which needs a caveat: its Ribozyme mean is 0.0087
and the macro weights the three categories equally, so a strong tRNA mean over
only three assays carries it. It is also the only base-pretrained 8k-context
checkpoint here, where the 7B, 20B and 40B are mid-trained at 1M, so it is not a
size point comparable to the rest.

Both were scored with the packaged FP8 configuration, which is what the Evo 2
authors require for every checkpoint except the 7B. The 7B and the 40B were
rescored the same way as a check and are not published: they reproduce the
existing columns, the 40B bitwise on all 31 assays and the 7B to the resolution
of its own float32 serialisation, so publishing them would add rows that
duplicate what is already there. That reproduction is what allows the existing
7B and 40B columns to be read alongside these two as one comparable family.

The row count in test_every_leaderboard_row_is_a_registered_model goes from 16
to 18 with them.
@Leo-T-Zang
Leo-T-Zang requested a review from murfalo August 24, 2026 20:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant