Skip to content

fix: prevent token prediction from corrupting generation on hybrid and recurrent models - #692

Open
tomwangcn wants to merge 1 commit into
withcatai:masterfrom
tomwangcn:fix-hybrid-recurrent-token-predictor
Open

tomwangcn wants to merge 1 commit into
withcatai:masterfrom
tomwangcn:fix-hybrid-recurrent-token-predictor

Conversation

@tomwangcn

Copy link
Copy Markdown
Contributor

Closes #652.

The problem

On hybrid (e.g. Qwen3.5) and recurrent models, using a token predictor changes the generated output and slows generation down significantly.

The root cause is two cooperating issues in the rollback path of refuted token predictions:

  1. Contexts are created without n_rs_seq, so llama.cpp's recurrent memory keeps no rollback snapshots and a partial tail removal can never succeed for the recurrent state.
  2. In _eraseContextTokenRanges, the KV cache position check compares the erase start against llama_memory_seq_pos_min. For hybrid and recurrent models that value always points at the latest token (the recurrent state is kept only for the current position), so every tail erase is treated as unrecoverable before the native removal is even attempted. The erase then falls back to a checkpoint restore or a full sequence reevaluation, and that fallback is not consistent with the speculative evaluation loop's token bookkeeping - the sequence state gets corrupted (committed tokens are replaced by fed-back continuation tokens), so the model faithfully continues from a corrupted token history.

Dense attention-only models are unaffected because their KV cell removal succeeds natively.

Reproduction (greedy, temperature: 0, completion of 1, 2, 3, with Qwen3.5-0.8B):

  • without predictor: 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...
  • with InputLookupTokenPredictor (before this fix): 4, 4, 1, 2, 3, 4, 1, 2, 3, 1, 2, 3, 1... and ~6x slower

The fix

  • Set n_rs_seq when creating a context, so the recurrent state can be rolled back natively. llama.cpp clamps it to 0 for models that don't support recurrent partial rollback, so dense models are unaffected. The value 16 matches llama.cpp's own default speculative draft depth. Note this multiplies the recurrent state memory by 1 + n_rs_seq per sequence on hybrid/recurrent models; happy to make it configurable if you prefer.
  • On hybrid and recurrent models, let the native removal call decide whether an erase is possible instead of rejecting it up front based on the KV cache position. The removal rolls back the recurrent state via the snapshots and fails atomically when it can't, so the existing reevaluation fallback still covers the cases it rejects.

Testing

  • Added a model-dependent test for qwen3.5-0.8b: greedy completion with and without InputLookupTokenPredictor must produce identical output, and it asserts that predictions were both validated and refuted so the rollback path is exercised. The test fails without this fix and passes with it.
  • Verified a dense model (Qwen3-0.6B) produces identical output with and without the predictor before and after the change.

…d recurrent models

Hybrid and recurrent models keep the recurrent state only for the latest
position, so `llama_memory_seq_pos_min` always points at the latest token.
The KV cache position check in `_eraseContextTokenRanges` treated every
tail erase as unrecoverable, and the reevaluation fallback corrupted the
sequence state when used during speculative token prediction, which changed
the generated output and slowed generation down significantly.

Set `n_rs_seq` when creating a context so the recurrent state can be
rolled back natively, and let the native removal call decide whether an
erase is possible on hybrid and recurrent models (it fails atomically, so
the reevaluation fallback still covers the cases it rejects).

Closes: withcatai#652

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

InputLookupTokenPredictor changes greedy output and slows generation on hybrid (qwen35) models

1 participant