[hieuddo]

Semantic ID predictions can match the target ID while failing to identify the target item when IDs collide. This change adds optional SID and Collision-Corrected Evaluation (CCE) views to NextItemEvaluation while preserving ordinary item metrics, so Semantic ID and ordinary next-item models can share an experiment.

• Keep semantic_id_metrics=False by default; SID examples explicitly enable it. Supported finite-K Recall/NDCG metrics additionally produce labeled SID-* and CCE-* results.
• Add a lightweight SemanticIDMixin and adapt TIGER, LETTER, RPG and DiffGRM to return item rankings and raw SID predictions from one inference. Item resolution stays inside each model.
• Compact invalid/repeated SID outputs, respect candidate filtering and collision-group widths, and warn on short predictions without changing decoding budgets. Mixed-model tables show N/A for unavailable views.
• Update documentation and examples to demonstrate explicit opt-in and distinguish ordinary item, SID and CCE metric views.

[saghiles]

Maybe better to move this out of models. It can go to utils for instance.

[hieuddo]

Sounds good. That'd be a better fit than inside models

[saghiles]

What is the difference between this file "semantic_id_evaluation.py" and "sid_evaluation.py"?

• I would recommend improving the naming. For instance, can we say that semantic_id_evaluation.py implements a NextItemEvaluation for Semantic-ID based recommenders? If yes we can name it explicitly
• Add more documentation explain why and when to use this eval over the base NextItem evaluation
• Are there any references in the literature supporting this method? If yes, we can point to them
• Is this method dedicated to SIDs based models are can be used to evaluate list of models including both SID-based (e.g., Tiger) and non-SIDs-based (e.g., Sasrec) models?
• Also: semantic_id_evaluation can be confusing, what we are evaluating here is recommendation performance, not the quality of semantic IDs.

[saghiles]

• Add more docs with a concrete example to explain what code inside this scripts allows to do.
• /eval_methods may not be the right path for this script. Consider moving it to utils.

[hieuddo]

@saghiles Thanks for the suggestions. I’ve consolidated the implementation into:

• cornac/eval_methods/semantic_id_evaluation.py: contains SemanticIDEvaluation(NextItemEvaluation), prediction collection, and metric computation from saved predictions. Most of the Semantic ID methods have been using leave-last-out evaluation method (which is the same as our NextItemEvaluation), but with these new changes in item-level evaluation, I think it'd be better to create a separate interface for Semantic ID.
• cornac/utils/semantic_id.py: contains shared item-to-SID mappings, SID predictions, prediction traces, and item resolvers.
• cornac/models/recommender.py: SemanticIDRecommender extends NextItemRecommender with interfaces for accessing item-to-SID mappings and raw SID predictions.

The documentation now distinguishes two ways to convert SID predictions into item-level results:

• CCE (one SID to all associated items): expands each predicted SID into its complete item bucket and computes expected metrics under uniformly random ordering within each bucket, following the CCE/ZCR paper.
• Explicit resolution (one SID to one item): selects a concrete member of each bucket.
  • Popularity selects the most frequent item in the training data
  • cosine selects the item most similar to the user’s training-history feature profile

[saghiles]

@saghiles Thanks for the suggestions. I’ve consolidated the implementation into:

• cornac/eval_methods/semantic_id_evaluation.py: contains SemanticIDEvaluation(NextItemEvaluation), prediction collection, and metric computation from saved predictions. Most of the Semantic ID methods have been using leave-last-out evaluation method (which is the same as our NextItemEvaluation), but with these new changes in item-level evaluation, I think it'd be better to create a separate interface for Semantic ID.
• cornac/utils/semantic_id.py: contains shared item-to-SID mappings, SID predictions, prediction traces, and item resolvers.
• cornac/models/recommender.py: SemanticIDRecommender extends NextItemRecommender with interfaces for accessing item-to-SID mappings and raw SID predictions.

The documentation now distinguishes two ways to convert SID predictions into item-level results:

• CCE (one SID to all associated items): expands each predicted SID into its complete item bucket and computes expected metrics under uniformly random ordering within each bucket, following the CCE/ZCR paper.

• Explicit resolution (one SID to one item): selects a concrete member of each bucket.

  • Popularity selects the most frequent item in the training data
  • cosine selects the item most similar to the user’s training-history feature profile

Hi @hieuddo , thank you so much for the clarification. to follow up,

1. From the description, this still seems to be a Next Item evaluation method. Ideally, the SID-to-ID mapping should be handled at the model level, with optional parameters to support different mapping strategies if needed. Evaluation methods should remain as model-agnostic as possible to preserve Cornac’s ability to compare a broad range of models. Their responsibilities should primarily be to prepare, sample, or split the data, obtain predictions from models, and evaluate those predictions. They should not dictate how models generate predictions or transform model outputs as part of model-specific logic.
2. Could we achieve the current objective without extending the existing NextItemEval method and enabling semantic-ID-based models, such as TIGER, to optionally perform various SID-to-ID mapping before passing their final predictions to the evaluation method?
3. Based on my current understanding, I would not recommend introducing a separate SemanticIDEval method, as it seems to overlap with the responsibilities of NextItemEval and could introduce unnecessary model-specific logic into the evaluation layer. Maybe let's get @qtuantruong input as well.

[hieuddo]

@saghiles Thanks for the suggestions. I’ve consolidated the implementation into:

• cornac/eval_methods/semantic_id_evaluation.py: contains SemanticIDEvaluation(NextItemEvaluation), prediction collection, and metric computation from saved predictions. Most of the Semantic ID methods have been using leave-last-out evaluation method (which is the same as our NextItemEvaluation), but with these new changes in item-level evaluation, I think it'd be better to create a separate interface for Semantic ID.
• cornac/utils/semantic_id.py: contains shared item-to-SID mappings, SID predictions, prediction traces, and item resolvers.
• cornac/models/recommender.py: SemanticIDRecommender extends NextItemRecommender with interfaces for accessing item-to-SID mappings and raw SID predictions.

The documentation now distinguishes two ways to convert SID predictions into item-level results:

• CCE (one SID to all associated items): expands each predicted SID into its complete item bucket and computes expected metrics under uniformly random ordering within each bucket, following the CCE/ZCR paper.

• Explicit resolution (one SID to one item): selects a concrete member of each bucket.

  • Popularity selects the most frequent item in the training data
  • cosine selects the item most similar to the user’s training-history feature profile

Hi @hieuddo , thank you so much for the clarification. to follow up,

1. From the description, this still seems to be a Next Item evaluation method. Ideally, the SID-to-ID mapping should be handled at the model level, with optional parameters to support different mapping strategies if needed. Evaluation methods should remain as model-agnostic as possible to preserve Cornac’s ability to compare a broad range of models. Their responsibilities should primarily be to prepare, sample, or split the data, obtain predictions from models, and evaluate those predictions. They should not dictate how models generate predictions or transform model outputs as part of model-specific logic.
2. Could we achieve the current objective without extending the existing NextItemEval method and enabling semantic-ID-based models, such as TIGER, to optionally perform various SID-to-ID mapping before passing their final predictions to the evaluation method?
3. Based on my current understanding, I would not recommend introducing a separate SemanticIDEval method, as it seems to overlap with the responsibilities of NextItemEval and could introduce unnecessary model-specific logic into the evaluation layer. Maybe let's get @qtuantruong input as well.

@saghiles, your design actually makes more sense. I tried to take on too big a scope for this PR.

Let me reduce this PR to SID-level vs. item-level evaluation metrics for NextItemEvaluation first (using CCE).

I'm thinking we can extend some flag for NextItemEvaluation for SID models; then enforce both SID- and item-level metrics (where applicable).

I'll skip the item resolvers for now and think it through first.

[qtuantruong]

I largely agree with @saghiles on trying to enable these SID models to work with NextItemEvaluation because that's still the final goal of a recommender. My suggestions are as follows:

1. Use a SemanticIDMixin to expose SID predictions and collision-group information. Keep explicit SID-to-item resolution policies in the model, different models might do it differently. Something like:

class SemanticIDMixin:
    def semantic_id_catalog(self):
        raise NotImplementedError

    def predict_semantic_ids(self, history_items, **context):
        raise NotImplementedError


class TIGER(NextItemRecommender, SemanticIDMixin):
    ...

2. Reuse NextItemEvaluation so SID models can be compared with other models such as SASRec, etc. That's still the goal I believe.
3. Add CCE as opt-in metrics, computed when the model exposes the required collision-group information. Unsupported models should still run their standard metrics; CCE results can be marked unavailable.

There will be other details to get right, but I hope we can align on the above to more forward.

[hieuddo]

I largely agree with @saghiles on trying to enable these SID models to work with NextItemEvaluation because that's still the final goal of a recommender. My suggestions are as follows:

1. Use a SemanticIDMixin to expose SID predictions and collision-group information. Keep explicit SID-to-item resolution policies in the model, different models might do it differently. Something like:

class SemanticIDMixin:
    def semantic_id_catalog(self):
        raise NotImplementedError

    def predict_semantic_ids(self, history_items, **context):
        raise NotImplementedError


class TIGER(NextItemRecommender, SemanticIDMixin):
    ...

2. Reuse NextItemEvaluation so SID models can be compared with other models such as SASRec, etc. That's still the goal I believe.
3. Add CCE as opt-in metrics, computed when the model exposes the required collision-group information. Unsupported models should still run their standard metrics; CCE results can be marked unavailable.

There will be other details to get right, but I hope we can align on the above to more forward.

@qtuantruong Thanks for the feedback. The updated implementation is ready for another review. I updated #728 (comment) to reflect the revised implementation.

Could you take another look and see whether the changes address your earlier concerns?

[hieuddo]

With semantic_id_metrics=True, the output would be:

SASRec + TIGER

Model	NDCG@1	Recall@1	SID-NDCG@1	SID-Recall@1	CCE-NDCG@1	CCE-Recall@1
SASRec	0.2500	0.2500	N/A	N/A	N/A	N/A
TIGER	0.2500	0.2500	0.2500	0.2500	0.2500	0.2500

SASRec alone

Model	NDCG@1	Recall@1
SASRec	0.2500	0.2500

TIGER alone

Model	NDCG@1	Recall@1	SID-NDCG@1	SID-Recall@1	CCE-NDCG@1	CCE-Recall@1
TIGER	0.2500	0.2500	0.2500	0.2500	0.2500	0.2500

[qtuantruong]

do we need this?

[qtuantruong]

With semantic_id_metrics=True, the output would be:

SASRec + TIGER

Model NDCG@1 Recall@1 SID-NDCG@1 SID-Recall@1 CCE-NDCG@1 CCE-Recall@1
SASRec 0.2500 0.2500 N/A N/A N/A N/A
TIGER 0.2500 0.2500 0.2500 0.2500 0.2500 0.2500

SASRec alone

Model NDCG@1 Recall@1
SASRec 0.2500 0.2500

TIGER alone

Model NDCG@1 Recall@1 SID-NDCG@1 SID-Recall@1 CCE-NDCG@1 CCE-Recall@1
TIGER 0.2500 0.2500 0.2500 0.2500 0.2500 0.2500

why all metrics are 0.25? isn't that suspicious?

[hieuddo]

With semantic_id_metrics=True, the output would be:

SASRec + TIGER

Model NDCG@1 Recall@1 SID-NDCG@1 SID-Recall@1 CCE-NDCG@1 CCE-Recall@1
SASRec 0.2500 0.2500 N/A N/A N/A N/A
TIGER 0.2500 0.2500 0.2500 0.2500 0.2500 0.2500

SASRec alone

Model NDCG@1 Recall@1
SASRec 0.2500 0.2500

TIGER alone

Model NDCG@1 Recall@1 SID-NDCG@1 SID-Recall@1 CCE-NDCG@1 CCE-Recall@1
TIGER 0.2500 0.2500 0.2500 0.2500 0.2500 0.2500

why all metrics are 0.25? isn't that suspicious?

@qtuantruong Thanks for the feedback. I removed the redundant examples.

The previous table used synthetic data with only a few users/items/interactions.

I ran LETTER on a small Beauty subset (100 items), with 5 epochs (each, for tokenizer and recommender model):

Model	NDCG@5	Recall@5	SID-NDCG@5	SID-Recall@5	CCE-NDCG@5	CCE-Recall@5	Train (s)	Test (s)
LETTER	0.0134	0.0208	0.1841	0.3472	0.0114	0.0193	4.1108	1.6399

The three views measure different things:

• Ordinary item metrics: item ranking. Items sharing a predicted SID receive equal scores, but Cornac’s (numpy) sorting still places them in an arbitrary order.
• SID metrics: SID ranking, at SID level
• CCE metrics: expected performance in item positions, expanding predicted IDs into their collision groups and averaging over uniformly random ordering within each group (same as in CCE paper).

Consequently, ordinary Recall@5 (0.0208) and CCE-Recall@5 (0.0193) are close but need not match: one evaluates the concrete ordering, while the other evaluates the uniform-order expectation. I suppose for bigger data and more test samples, CCE and ordinary item score would come closer

[qtuantruong]

Thanks @hieuddo, this looks good to me.
@saghiles do you have further comments for this PR?