fix(grammar): preserve distinct JSON Schema reference identities

[Metis-dot]

Summary

Preserve full JSON Schema reference identities instead of treating an existing grammar rule with the same last path token as an already resolved reference.

Two distinct references such as #/$defs/left/$defs/Item and #/$defs/right/$defs/Item currently reuse the first Item rule. A definition named string or integer can also reuse an unrelated built-in rule. This can silently drop a valid union alternative or remove a referenced property's constant constraint.

The fix caches each full resolved reference, reserves a unique sanitized name before descending into its target, and retains that name for recursive and repeated references. It also covers empty definition names and names that collide after sanitization. No changes to native code, schema fetching, or JSON Pointer token decoding are included.

Minimal reproduction

schema = {
    "anyOf": [
        {"$ref": "#/$defs/left/$defs/Item"},
        {"$ref": "#/$defs/right/$defs/Item"},
    ],
    "$defs": {
        "left": {"$defs": {"Item": {"const": "left"}}},
        "right": {"$defs": {"Item": {"const": "right"}}},
    },
}

Before the fix, both alternatives reference Item, and its only literal is "left"; "right" is absent from the generated grammar. After the fix, each alternative reaches its corresponding literal.

Validation

• Linux, Python 3.12.14; pytest 9.1.1 and Ruff 0.16.9 from PyPI
• Downloaded the official llama-cpp-python==0.3.36 PyPI source distribution and verified its published SHA-256. Its llama_grammar.py is byte-identical to current main at 1652066e0af45f2313b339670ef9555e8a54e545
• Ran the three existing grammar tests and eleven new regression cases against the actual grammar module, loaded directly with a minimal package shim so no native package startup was needed: baseline 9 failed, 5 passed; patched 14 passed
• Regression coverage includes different references with the same leaf, sanitized-name collisions, an empty definition key, built-in string/integer/space collisions, same-reference reuse, self-recursion, and mutual recursion with colliding leaf names
• An independent stdlib GBNF recognizer also checked accepted and rejected JSON strings for collision and recursion cases. This is auxiliary language-level validation, not native libllama grammar execution
• Ruff lint and formatting checks pass for both changed Python files; git diff --check passes

The full repository suite, native libllama grammar execution, inference, model-dependent tests, and GPU tests were not run. No model weights were downloaded. The separate existing root-$ref traversal and JSON Pointer escape issues are outside this patch's scope.

AI assistance

This contribution was prepared with AI assistance. The root cause, diff, and tests were independently reviewed, and the validation limits above are intentional and explicit.