[dexhunter]

TL;DR

• This PR removes repeated per-query work in ranking_eval. NextItemEvaluation.evaluate uses it for every test session.
• The candidate item array does not change between queries. The code now builds it once and slices it per query.
• With trained TIGER and RPG models, evaluation is about 0.5% to 1.2% faster (measured by @hieuddo in this thread).
• With cheap fixed scores, the gain is larger: 217.5 ms -> 169.8 ms, 22% faster (about 2000 queries).
• Tests: pytest tests/cornac/eval_methods tests/cornac/models passes (107 passed, 1 skipped). Metric values are bit-for-bit identical.

What & Why

Before, each query built two full-length integer masks (one entry per item). Then it called np.nonzero three times to get the positive items, the negative items and all candidate items. The candidate items are the same for every query. The loop also looked up the rank method, the test set attributes and the user_based branch again for every query.

Now the code builds item_indices = np.arange(num_candidates) once. For the usual case of one positive item, it slices that array to get the positive and negative items. Other cases (for example several positives) still use the old mask code. The loop reads sessions directly from test_set.sessions, in the same order as si_iter. It also picks the rank method and the result container once.

The arrays passed to model.rank and to the metrics have the same values, order and dtype as before. So the scores, the warnings and the result dicts stay identical.

Results

Median of 7 runs, Recall@10 and NDCG@10, exclude_unknowns=True, plain NextItemRecommender with fixed scores (except the first row).

Situation	Before	After	Change
SID model, semantic_id_metrics off, 440 items, 1200 sessions (test_semantic_views_require_explicit_opt_in shape, scaled up)	217.5 ms	169.8 ms	22% faster
400 items, 1200 sessions, mode="last"	68.6 ms	59.5 ms	13% faster
5000 items, 1200 sessions, mode="last"	268.2 ms	230.7 ms	14% faster
400 items, 1200 sessions, mode="next"	174.0 ms	156.2 ms	10% faster
Slow model (scoring dominates), 400 items, 300 sessions	228.7 ms	226.6 ms	no change (within noise)

Why this is safe and easy to maintain

The change touches only ranking_eval and keeps the old mask code as the general fallback. The new fast path is a short helper with a comment. Outputs are unchanged: metric values, numpy types, result containers and warnings match exactly. tests/cornac/eval_methods covers the semantic_id_metrics=True, exclude_unknowns=False and mode="next" paths.

Validation

• Native tests on the patched tree:
  pytest tests/cornac/models/test_semantic_id_adapters.py tests/cornac/data/test_dataset.py tests/cornac/eval_methods/test_semantic_id_metrics.py tests/cornac/eval_methods/test_next_item_evaluation.py -> 96 passed, 1 skipped.
  pytest tests/cornac/eval_methods tests/cornac/models -> 107 passed, 1 skipped.
• Benchmark: 217.5 ms -> 169.8 ms (3 repeated measurements, ratios 0.779 / 0.777 / 0.781), single thread (OMP_NUM_THREADS=1).
• Reproduce: time NextItemEvaluation.from_splits(train, test, fmt="USIT").evaluate(model, [Recall(10), NDCG(10)], user_based=False) before and after the patch on a 1200-session test split.

Description

See the sections above. No public signature changes and no new dependencies.

Related Issues

None.

Checklist:

• I have added tests.
• I have updated the documentation accordingly.
• I have updated README.md (if you are adding a new model).
• I have updated examples/README.md (if you are adding a new example).
• I have updated datasets/README.md (if you are adding a new dataset).

AI assistance: Codex (gpt-6.1-sol) generated the candidate changes. Weco Observe tracked them, and an automated harness checked each candidate for identical outputs and measured its speed.

Weco share link: https://dashboard.weco.ai/share/vG4ePh8hQlA092gOCUDxcR2_Ce0ea6D1

[qtuantruong]

Thanks @dexhunter for the fix. I think the changes make sense to me. Just a small comment on the test case introduced.

[qtuantruong]

@hieuddo would you mind running some benchmark to see if the improvement is significant?

[dexhunter]

Thanks for the review. I updated the test in 5b5ae2f, see the inline reply.

For the benchmark: I ran the timing again on the current branch. The setup is two NextItemEvaluation.evaluate calls (session-based and user-based) with Recall@1 on a SID model, 1200 test sessions, single thread. The new code takes about 0.77x the time of master (repeated runs: 0.764 to 0.781). This matches the first row of the results table in the PR description (217.5 ms -> 169.8 ms). The other rows use different setups, which the table describes.

[hieuddo]

The 22% headline describes a ranking-loop improvement on a scaled SID-test-shaped workload with semantic metrics disabled. The referenced test uses predetermined scores rather than learned inference. To examine the improvement with neural inference included, we compared upstream against this PR using trained TIGER and RPG, with semantic metrics both disabled and enabled:

Model	Semantic metrics	Upstream	PR	Improvement¹
TIGER	Off	1,245.97 ms	1,240.75 ms	0.55%
TIGER	On	1,262.57 ms	1,253.31 ms	0.59%
RPG	Off	325.82 ms	321.40 ms	1.15%
RPG	On	332.24 ms	327.61 ms	1.16%

¹ Median across nine runs.

Models were fitted and prepared outside timing:

model.fit(evaluator.train_set)
model.transform(evaluator.test_set)

We then substituted each evaluator implementation and timed this call with sid-metrics enabled=False and True:

b.evaluation.ranking_eval = FUNCTIONS[label]  # upstream or PR
result = evaluator.eval(
    model, evaluator.train_set, evaluator.test_set,
    True,  # exclude_unknowns
    [b.Recall(10), b.NDCG(10)],
    user_based=False,
    mode="last",
    semantic_id_metrics=enabled,
)

Both versions used the same 440 candidates and 120 Yoochoose sessions. Outputs matched exactly.

So in summary, I think the PR does remove genuine redundant code, particularly repeated candidate-array construction and mask processing. That cleanup is useful. However, these results suggest the large percentage primarily reflects overhead reduction around fixed scoring, rather than a comparable improvement when inference is involved.

[dexhunter]

Thanks @hieuddo for the careful measurement with trained models. I agree with your reading.

The change only removes per-query work around model.rank: the candidate array build, the masks and the np.nonzero calls. It does not touch scoring. So the 22% applies only when scoring is cheap, as in the fixed-score benchmark. When TIGER or RPG inference takes most of the time, the gain drops to about 1%, as your table shows. The last row of the PR table (slow model, no change) shows the same effect.

The headline number is too strong for real models. If you prefer, I can change the title to describe this as a cleanup that cuts evaluation overhead, without the 22% figure. Outputs stay identical either way.

[qtuantruong]

@dexhunter @hieuddo thanks both for the discussion. I think we should revise the headline with softer claim on the perf improvement. This is a good work and we should merge it.

[dexhunter]

Thanks @qtuantruong. I changed the title and the TL;DR to use a softer claim.

• The title now says the PR cuts per-query overhead in ranking_eval. It gives about 1% with trained models.
• The TL;DR now puts the TIGER and RPG results from @hieuddo first (about 0.5% to 1.2% faster).
• The 22% figure stays only as the cheap fixed-score case, and the TL;DR labels it that way.

The code is the same as in 5b5ae2f. Outputs stay identical.