Repository navigation
Conversation
…d recurrent models Hybrid and recurrent models keep the recurrent state only for the latest position, so `llama_memory_seq_pos_min` always points at the latest token. The KV cache position check in `_eraseContextTokenRanges` treated every tail erase as unrecoverable, and the reevaluation fallback corrupted the sequence state when used during speculative token prediction, which changed the generated output and slowed generation down significantly. Set `n_rs_seq` when creating a context so the recurrent state can be rolled back natively, and let the native removal call decide whether an erase is possible on hybrid and recurrent models (it fails atomically, so the reevaluation fallback still covers the cases it rejects). Closes: withcatai#652
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #652.
The problem
On hybrid (e.g. Qwen3.5) and recurrent models, using a token predictor changes the generated output and slows generation down significantly.
The root cause is two cooperating issues in the rollback path of refuted token predictions:
n_rs_seq, so llama.cpp's recurrent memory keeps no rollback snapshots and a partial tail removal can never succeed for the recurrent state._eraseContextTokenRanges, the KV cache position check compares the erase start againstllama_memory_seq_pos_min. For hybrid and recurrent models that value always points at the latest token (the recurrent state is kept only for the current position), so every tail erase is treated as unrecoverable before the native removal is even attempted. The erase then falls back to a checkpoint restore or a full sequence reevaluation, and that fallback is not consistent with the speculative evaluation loop's token bookkeeping - the sequence state gets corrupted (committed tokens are replaced by fed-back continuation tokens), so the model faithfully continues from a corrupted token history.Dense attention-only models are unaffected because their KV cell removal succeeds natively.
Reproduction (greedy,
temperature: 0, completion of1, 2, 3,withQwen3.5-0.8B):4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...InputLookupTokenPredictor(before this fix):4, 4, 1, 2, 3, 4, 1, 2, 3, 1, 2, 3, 1...and ~6x slowerThe fix
n_rs_seqwhen creating a context, so the recurrent state can be rolled back natively. llama.cpp clamps it to 0 for models that don't support recurrent partial rollback, so dense models are unaffected. The value 16 matches llama.cpp's own default speculative draft depth. Note this multiplies the recurrent state memory by1 + n_rs_seqper sequence on hybrid/recurrent models; happy to make it configurable if you prefer.Testing
qwen3.5-0.8b: greedy completion with and withoutInputLookupTokenPredictormust produce identical output, and it asserts that predictions were both validated and refuted so the rollback path is exercised. The test fails without this fix and passes with it.Qwen3-0.6B) produces identical output with and without the predictor before and after the change.