fix: Raise a clear error when HFSystemFormat's tokenizer has no chat template

[wak327]

Summary

HFSystemFormat renders prompts with tokenizer.apply_chat_template. When the model's tokenizer has no chat_template (e.g. ibm-granite/granite-20b-code-base from #1144, or gpt2), every instance fails while the dataset is being generated with transformers' generic error:

ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! ...

That message doesn't name the model, and it doesn't mention the options unitxt already supports.

Changes

• HFSystemFormat.prepare() checks the tokenizer right after loading it. If there is no chat template and none was passed through chat_kwargs_dict, it raises a UnitxtError once, when the format is created:

  HFSystemFormat cannot be used with model 'openai-community/gpt2' because its tokenizer does not define a chat template (no 'chat_template' in its tokenizer_config.json). Either use a model whose tokenizer has a chat template, pass a Jinja chat template explicitly with chat_kwargs_dict={'chat_template': '<template>'}, or use a format that does not depend on the tokenizer, such as SystemFormat.
  

• The HFSystemFormat docstring now mentions passing a template through chat_kwargs_dict={"chat_template": ...}. That already works today, since chat_kwargs_dict is forwarded to apply_chat_template.

• UnitxtError isn't in the retry list of retry_connection_with_exponential_backoff, so the new check is not retried.

Fixes #1144

Test plan

• New test_hf_system_format_without_chat_template: creating the format with openai-community/gpt2 raises the UnitxtError, and passing an explicit chat_template in chat_kwargs_dict formats correctly. Fails on main (UnitxtError not raised), passes here.
• Existing test_hf_system_format (zephyr) still passes.
• pre-commit run on the changed files.

[wak327]

@elronbandel @yoavkatz could you take a look when you have a moment? Small change: HFSystemFormat now fails early with a clear message when the model's tokenizer has no chat template (#1144), instead of failing on every instance with the generic transformers error.

[wak327]

@yoavkatz thanks for the approval and for merging main in. I pushed one more commit to fix the unittests and eager failures. Could you re-approve, and approve the workflow runs for the new commit?

Fix (54c1f19, test-only): since #1989, fork runs skip installing llmevalkit. That made the 15 tests that build ReflectionToolCallingMetric / ReflectionToolCallingMetricSyntactic error with MissingRequirementsError. They were the only failures in both jobs: the other 767 tests passed, including the new HFSystemFormat test. They're now marked skipUnless(find_spec("llmevalkit")), so they still run wherever llmevalkit is installed, including trusted runs. The PR's library code is unchanged.

Still red, but not fixable from this branch:

• helm: HELM loads unitxt 1.26.10 from the Hub (unitxt/data). That version predates the as_dataset(**kwargs) fix from Security: Fix CodeQL alerts for ReDoS, path injection, and workflow permissions #1984, so helm has failed on every push to main since then. It should pass once the next release is published to the Hub.
• performance: gets a 429 on an unauthenticated download while profiling main (the main_results.json step), before this branch's code runs.
• preparation: on fork runs, shards time out on gated datasets without HF_TOKEN. They also time out on 0x22almostEvil/multilingual-wikihow-qa-16k, which no longer exists on the Hub, and that also times out shards 7 and 8 on main. The timeouts come from DatasetNotFoundError subclassing FileNotFoundError, which retry_connection_with_exponential_backoff retries. With 10 attempts and backoff_factor=2, that's about 17 minutes per call before the error shows up. I'm happy to open a separate PR so permanent "not found" and gated errors aren't retried, if that's useful.

[yoavkatz]

Hi @wak327 . Thank you for your contributions. Once the test that can pass complete, I'll merge. As you can see there is less active development over the last few months, and the library is more in a maintainance mode (security issues or fixes requested by existing user). Are you actively using it now?