Microsoft.ML.TorchSharp NamedEntityRecognition: predicted tags come one word to the right (first slot is the start token's, the last word's tag is missing) — a model that tags every word correctly scores as if it had learned nothing #7889

Package: Microsoft.ML.TorchSharp 0.23.1 (Microsoft.ML 5.0.x), mlContext.MulticlassClassification.Trainers.NamedEntityRecognition(...) + MapKeyToValue("PredictedLabel").

Observed. Train on 300 short English sentences, one tag per word (6 tags: B-PER O O B-LOC for Alice lives in Paris, etc.), 10 epochs, then score 50 held-out sentences and compare the PredictedLabel vector with the expected tags word by word:

• same position: 0.28 of the words right (macro accuracy over tags 0.09, entity F1 0);
• predicted position = word position + 1: 0.82;
• append one extra word ( .) to every sentence, drop the first predicted entry, compare: 1.00 of the words, the last word included (the model never saw the extra word in training).

Concretely, The train from Lisbon arrives at 8 am (expected O O O B-LOC O O B-TIME I-TIME) came back as O O O O B-LOC O O B-TIME for the sentence as given — one O in front, the last tag (I-TIME) missing, the vector length equal to the word count.

Expected: the i-th predicted tag belongs to the i-th word, one tag per word.

Likely cause (read from NerTrainer.cs, not stepped through): training prepends a 0 to the label vector and an init token to the input; CondenseOutput writes prediction[0] first and then prediction[i] for each token that starts with a space, indexing the model output by the encoding index although the model input has the init token in front, so each word receives the prediction of the position before it. The first word has no space prefix, so the output has word-count entries and the final word's own position is never read.

Workaround on our side: score the sentence with one successor word appended and drop the first entry of the answer.

Request: write the per-word tags at their own positions (or document the offset), with a test that a model fitted on a trivially learnable tagging gets its own training sentences right.