diff --git a/docs/generators/json-schema.rst b/docs/generators/json-schema.rst index ea2ca9cad9..9b0a235aaf 100644 --- a/docs/generators/json-schema.rst +++ b/docs/generators/json-schema.rst @@ -322,6 +322,27 @@ This is what the underlying JSON-Schema looks like: +Optional slots and null +^^^^^^^^^^^^^^^^^^^^^^^ + +By default, optional (non-required) slots accept an explicit JSON ``null`` in +addition to their base type — the generator emits ``"type": ["string", "null"]`` +for an optional string slot. This is convenient for producers that serialise +missing values as ``null``. + +Some target schemas forbid explicit ``null`` on optional properties (a property +must either be present with a typed value or absent). Use +``--no-include-null`` to restrict optional slots to their base type: + +.. code:: bash + + gen-json-schema --no-include-null personinfo.yaml + +With ``--no-include-null`` the optional string slot above is emitted as +``"type": "string"``, so instance documents carrying ``"slot": null`` fail +validation. The default (``--include-null``) preserves the previous behaviour. + + Patterns ^^^^^^^^ @@ -355,6 +376,43 @@ will generate: LinkML also supports `Structured patterns `_, these are compiled down to patterns during JSON Schema generation. +Dictionary key constraints (propertyNames) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A multivalued, inlined slot whose range class has an identifier slot is +compiled to a JSON object keyed by that identifier (see *Inlining* above). +When the identifier slot carries string-applicable constraints, they are +emitted as a `propertyNames `_ +schema on the container object, so the *keys* of the dictionary are validated, +not just the values: + +.. code-block:: yaml + + slots: + tags: + range: Tag + multivalued: true + inlined: true + uid: + identifier: true + pattern: "^(0|[1-9][0-9]*)$" + +generates on the container: + +.. code-block:: json + + "tags": { + "additionalProperties": {"$ref": "#/$defs/Tag"}, + "propertyNames": {"pattern": "^(0|[1-9][0-9]*)$"}, + "type": "object" + } + +The constraints carried over from the key slot are those applicable to JSON +Schema strings: ``pattern``, ``minimum_value``/``maximum_value`` are not +key-applicable, while ``pattern`` and length bounds are. Keys of a plain +CURIE/identifier form without constraints are unaffected. + + Rules ^^^^^ diff --git a/docs/generators/owl.rst b/docs/generators/owl.rst index 4b6f076fe8..3ed387ceec 100644 --- a/docs/generators/owl.rst +++ b/docs/generators/owl.rst @@ -67,6 +67,26 @@ Mapping .. note:: The current default settings for ``metaclasses`` and ``type-objects`` may change in the future +Prefix normalization +^^^^^^^^^^^^^^^^^^^^ + +Schemas sometimes declare non-standard aliases for well-known namespaces +(e.g. ``sh1:`` for the SHACL namespace, or a versioned alias for ``skos:``). +By default these aliases are carried through into the generated artifact. + +Use ``--normalize-prefixes`` to remap declared prefixes whose namespace IRI +matches a well-known vocabulary to that vocabulary's conventional name in the +output (``owl``, ``rdf``, ``rdfs``, ``skos``, ``sh``, ``xsd``, ...): + +.. code:: bash + + gen-owl --normalize-prefixes schema.yaml + +The mapping is a static, version-independent table; namespace IRIs that are +not in the table are left untouched. The option is also available on +``gen-shacl`` and ``gen-jsonld-context``. + + Enums and PermissibleValues ^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -311,6 +331,40 @@ Other examples translation of Biolink schema to OWL +Deterministic output +^^^^^^^^^^^^^^^^^^^^ + +Generated Turtle can differ between runs — blank-node identifiers and +statement order depend on Python dict ordering and rdflib serialization +internals — which makes version-controlled artifacts show large spurious +diffs. Use ``--deterministic`` for byte-identical output across invocations: + +.. code:: bash + + gen-owl --deterministic schema.yaml + +The pipeline has three phases: + +1. `RDFC-1.0 `_ canonicalization (via + `pyoxigraph `_), so isomorphic + inputs produce identical triple sets; +2. Weisfeiler–Lehman structural hashing replaces the sequential ``_:c14nN`` + labels with content-derived ones, so adding or removing a triple only + renames the directly involved blank nodes (diff-stable output); +3. re-serialization with rdflib recovers idiomatic Turtle — inline blank + nodes (`Turtle §2.7 `_), collection + syntax (`§2.8 `_) — and only + declares prefixes actually used in the graph. + +All triples are preserved; only the syntactic form is normalised. Unordered +collections such as ``owl:oneOf`` items are additionally sorted. The option +is available on ``gen-owl``, ``gen-shacl``, ``gen-jsonld``, and +``gen-jsonld-context`` (for JSON output it deep-sorts objects instead). + +``pyoxigraph >= 0.4.0`` is required and imported lazily — it is only needed +when the flag is used and is deliberately not a core dependency. + + Docs ---- diff --git a/docs/generators/shacl.rst b/docs/generators/shacl.rst index 3e88f0090f..659e55f63c 100644 --- a/docs/generators/shacl.rst +++ b/docs/generators/shacl.rst @@ -84,6 +84,83 @@ Example Output: shacl:targetClass . +Rule constraints (SHACL-SPARQL) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +LinkML `rules `_ express +cross-parameter, conditional validation ("if slot A holds X, slot B must +..."). Plain per-slot SHACL property shapes cannot express these, so the +generator translates recognised rule shapes into +`SHACL-SPARQL constraints `_ +(``sh:sparql`` / ``sh:SPARQLConstraint``) on the class's ``sh:NodeShape``. +Generation is controlled by ``--emit-rules/--no-emit-rules`` (default: on). + +Three named patterns are recognised first: + +* **Boolean guard** — precondition ``value_presence: PRESENT`` on a value + slot, postcondition ``equals_string: "true"`` on a *boolean-range* flag + slot: if the value is present, the flag must be true. +* **Presence implies value** — precondition ``value_presence: PRESENT``, + postcondition ``equals_string`` / ``equals_string_in`` on a target slot: + if the guard is present, the target must hold one of the allowed values. + Enum values resolve to their ``meaning`` IRIs; values without ``meaning`` + compare as string literals. +* **Exclusive value** — precondition ``equals_string`` and postcondition + ``maximum_cardinality`` on the *same* multivalued slot: if the value is + present, the slot has at most N values. + +Combinations outside the named patterns are handled by a compositional +fallback that conjoins the preconditions and negates a single postcondition: +conditional-required (``required: true``), conditional-absent +(``value_presence: ABSENT``), numeric threshold preconditions +(``minimum_value`` / ``maximum_value``), a one-hop nested precondition into +an inlined child object (``range_expression.slot_conditions``), and +``has_member`` list membership. + +The translation contract is *skip, never mis-translate*: a rule whose +conditions set any operator outside the translated set (including +expression-level ``any_of``/``all_of``/``none_of``/``exactly_one_of``), or +whose slot keys resolve to no slot, is skipped and logged at ``DEBUG``. +``deactivated`` rules are skipped; ``bidirectional``, ``open_world``, and +``elseconditions`` warn (the forward direction is emitted). + +Example: + +.. code-block:: yaml + + classes: + Weather: + slots: [sun_altitude, daytime] + rules: + - description: If sun_altitude is present, daytime must be day or twilight. + preconditions: + slot_conditions: + sun_altitude: + value_presence: PRESENT + postconditions: + slot_conditions: + daytime: + equals_string_in: [day, twilight] + +generates (abridged): + +.. code-block:: turtle + + ex:Weather a sh:NodeShape ; + sh:sparql [ a sh:SPARQLConstraint ; + sh:message "If sun_altitude is present, daytime must be day or twilight." ; + sh:select """SELECT $this WHERE { + $this ?value . + OPTIONAL { $this ?target . } + FILTER ( !BOUND(?target) || ?target NOT IN (, ) ) + }""" ] . + +``$this`` is pre-bound to each focus node per +`SHACL §5.3.1 `_. +Note that SPARQL-based constraints require a SHACL processor with +SHACL-SPARQL support (e.g. ``pyshacl`` with ``advanced=True``). + + Command Line ^^^^^^^^^^^^ diff --git a/packages/linkml/pyproject.toml b/packages/linkml/pyproject.toml index 32b31e2a87..ec0ea2eaf6 100644 --- a/packages/linkml/pyproject.toml +++ b/packages/linkml/pyproject.toml @@ -50,7 +50,10 @@ dependencies = [ # Specifier syntax: https://peps.python.org/pep-0631/ "openpyxl", "parse", "prefixcommons >= 0.1.7", - "prefixmaps >= 0.2.2", + # TODO(prefixmaps-0.2.8): Replace git pin with "prefixmaps >= 0.2.8" once released, + # then remove [tool.hatch.metadata] allow-direct-references and regenerate uv.lock. + # Tracked in: https://github.com/linkml/prefixmaps/issues/82 + "prefixmaps @ git+https://github.com/linkml/prefixmaps@75435150a1b31760b9780af2b64a265943a9b263", "pydantic >= 2.0.0, < 3.0.0", "pyjsg >= 0.12.3", "pyshex >= 0.9.0", @@ -203,6 +206,10 @@ vcs = "git" style = "pep440" fallback-version = "0.0.0" +[tool.hatch.metadata] +# TODO(prefixmaps-0.2.8): Remove this section once the git pin is replaced with >= 0.2.8 +allow-direct-references = true + [tool.hatch.version] source = "uv-dynamic-versioning" diff --git a/packages/linkml/src/linkml/generators/jsonldcontextgen.py b/packages/linkml/src/linkml/generators/jsonldcontextgen.py index c30afc72a5..0c81a0edc4 100644 --- a/packages/linkml/src/linkml/generators/jsonldcontextgen.py +++ b/packages/linkml/src/linkml/generators/jsonldcontextgen.py @@ -15,7 +15,7 @@ from linkml._version import __version__ from linkml.utils.deprecation import deprecated_fields -from linkml.utils.generator import Generator, shared_arguments +from linkml.utils.generator import Generator, shared_arguments, well_known_prefix_map from linkml_runtime.linkml_model.meta import ClassDefinition, EnumDefinition, SlotDefinition from linkml_runtime.linkml_model.types import SHEX from linkml_runtime.utils.formatutils import camelcase, underscore @@ -90,6 +90,9 @@ class ContextGenerator(Generator): frame_root: str | None = None def __post_init__(self) -> None: + # Must be set before super().__post_init__() because the parent triggers + # the visitor pattern (visit_schema), which accesses _prefix_remap. + self._prefix_remap: dict[str, str] = {} super().__post_init__() if self.namespaces is None: raise TypeError("Schema text must be supplied to context generator. Preparsed schema will not work") @@ -127,8 +130,14 @@ def _collect_external_elements(sv: SchemaView) -> tuple[set[str], set[str]]: external_slots.update(schema_def.slots.keys()) return external_classes, external_slots + def add_prefix(self, ncname: str) -> None: + """Add a prefix, applying well-known prefix normalisation when enabled.""" + super().add_prefix(self._prefix_remap.get(ncname, ncname)) + def visit_schema(self, base: str | Namespace | None = None, output: str | None = None, **_): - # Add any explicitly declared prefixes + # Add any explicitly declared prefixes. + # Direct .add() is safe here: the normalisation block below explicitly + # rewrites emit_prefixes entries for any renamed prefixes (Cases 1-3). for prefix in self.schema.prefixes.values(): self.emit_prefixes.add(prefix.prefix_prefix) @@ -136,6 +145,68 @@ def visit_schema(self, base: str | Namespace | None = None, output: str | None = for pfx in self.schema.emit_prefixes: self.add_prefix(pfx) + # Normalise well-known prefix names when --normalize-prefixes is set. + # If the schema declares a non-standard alias for a namespace that has + # a well-known standard name (e.g. ``sdo`` for + # ``https://schema.org/``), replace the alias with the standard name + # so that generated JSON-LD contexts use the conventional prefix. + # + # Three cases are handled: + # 1. Standard prefix is not yet bound → just rebind from old to new. + # 2. Standard prefix is bound to a *different* URI: + # a. User-declared (in schema.prefixes) → collision, skip with warning. + # b. Runtime default (e.g. linkml-runtime's ``schema: http://…``) + # → remove stale binding, then rebind. + # 3. Standard prefix is already bound to the *same* URI (duplicate) + # → just drop the non-standard alias. + # + # A remap dict is stored for ``_build_element_id`` because + # ``prefix_suffix()`` splits CURIEs on ``:`` without looking up the + # namespace dict. + self._prefix_remap.clear() + if self.normalize_prefixes: + wk = well_known_prefix_map() + for old_pfx in list(self.namespaces): + url = str(self.namespaces[old_pfx]) + std_pfx = wk.get(url) + if not std_pfx or std_pfx == old_pfx: + continue + if std_pfx in self.namespaces: + if str(self.namespaces[std_pfx]) != url: + # Case 2: std_pfx is bound to a different URI. + # If the user explicitly declared std_pfx in the schema, + # it is intentional — skip to avoid data loss. + if std_pfx in self.schema.prefixes: + self.logger.warning( + "Prefix collision: cannot rename '%s' to '%s' because '%s' is " + "already declared for <%s>; skipping normalisation for <%s>", + old_pfx, + std_pfx, + std_pfx, + str(self.namespaces[std_pfx]), + url, + ) + continue + # Not user-declared (e.g. linkml-runtime default) — safe to remove + self.emit_prefixes.discard(std_pfx) + del self.namespaces[std_pfx] + else: + # Case 3: standard prefix already bound to same URI + # — just drop the non-standard alias + del self.namespaces[old_pfx] + if old_pfx in self.emit_prefixes: + self.emit_prefixes.discard(old_pfx) + self.emit_prefixes.add(std_pfx) + self._prefix_remap[old_pfx] = std_pfx + continue + # Case 1 (or Case 2 after stale removal): bind standard name + self.namespaces[std_pfx] = self.namespaces[old_pfx] + del self.namespaces[old_pfx] + if old_pfx in self.emit_prefixes: + self.emit_prefixes.discard(old_pfx) + self.emit_prefixes.add(std_pfx) + self._prefix_remap[old_pfx] = std_pfx + # Add the default prefix if self.schema.default_prefix: dflt = self.namespaces.prefix_for(self.schema.default_prefix) @@ -143,6 +214,8 @@ def visit_schema(self, base: str | Namespace | None = None, output: str | None = self.default_ns = dflt if self.default_ns: default_uri = self.namespaces[self.default_ns] + # Direct .add() is safe: default_ns is already resolved from + # the (possibly normalised) namespace bindings above. self.emit_prefixes.add(self.default_ns) else: default_uri = self.schema.default_prefix @@ -236,7 +309,61 @@ def end_schema( with open(frame_path, "w", encoding="UTF-8") as f: json.dump(frame, f, indent=2, ensure_ascii=False) - return str(as_json(context)) + "\n" + if self.deterministic: + return self._deterministic_context_json(json.loads(str(as_json(context))), indent=3) + return str(as_json(context)) + + @staticmethod + def _deterministic_context_json(data: dict, indent: int = 3) -> str: + """Serialize a JSON-LD context with deterministic key ordering. + + Preserves the conventional JSON-LD context structure: + 1. ``comments`` block first (metadata) + 2. ``@context`` block second, with: + a. ``@``-prefixed directives (``@vocab``, ``@base``) first + b. Prefix declarations (string values) second + c. Class/property term entries (object values) last + 3. Each group sorted alphabetically within itself + + Unlike :func:`deterministic_json`, this understands JSON-LD + conventions so that the output remains human-readable while + still being byte-identical across invocations. + """ + from linkml.utils.generator import deterministic_json + + ordered = {} + + # 1. "comments" first (if present) + if "comments" in data: + ordered["comments"] = data["comments"] + + # 2. "@context" with structured internal ordering + if "@context" in data: + ctx = data["@context"] + ordered_ctx = {} + + # 2a. @-prefixed directives (@vocab, @base, etc.) + for k in sorted(k for k in ctx if k.startswith("@")): + ordered_ctx[k] = ctx[k] + + # 2b. Prefix declarations (string values — short namespace URIs) + for k in sorted(k for k in ctx if not k.startswith("@") and isinstance(ctx[k], str)): + ordered_ctx[k] = ctx[k] + + # 2c. Term definitions (object values) — deep-sorted for determinism + term_entries = {k: v for k, v in ctx.items() if not k.startswith("@") and not isinstance(v, str)} + sorted_terms = json.loads(deterministic_json(term_entries)) + for k in sorted(sorted_terms): + ordered_ctx[k] = sorted_terms[k] + + ordered["@context"] = ordered_ctx + + # 3. Any remaining top-level keys + for k in sorted(data): + if k not in ordered: + ordered[k] = data[k] + + return json.dumps(ordered, indent=indent, ensure_ascii=False) def visit_class(self, cls: ClassDefinition) -> bool: if self.exclude_imports and cls.name not in self._local_classes: @@ -486,6 +613,11 @@ def _build_element_id(self, definition: Any, uri: str) -> None: @return: None """ uri_prefix, uri_suffix = self.namespaces.prefix_suffix(uri) + # Apply well-known prefix normalisation (e.g. sdo → schema). + # prefix_suffix() splits CURIEs on ':' without checking the + # namespace dict, so it may return a stale alias. + if uri_prefix and uri_prefix in self._prefix_remap: + uri_prefix = self._prefix_remap[uri_prefix] is_default_namespace = uri_prefix == self.context_body["@vocab"] or uri_prefix == self.namespaces.prefix_for( self.context_body["@vocab"] ) diff --git a/packages/linkml/src/linkml/generators/jsonldgen.py b/packages/linkml/src/linkml/generators/jsonldgen.py index 75d2068e16..c94c74d9dd 100644 --- a/packages/linkml/src/linkml/generators/jsonldgen.py +++ b/packages/linkml/src/linkml/generators/jsonldgen.py @@ -1,5 +1,6 @@ """Generate JSONld from a LinkML schema.""" +import json import os from collections.abc import Sequence from copy import deepcopy @@ -179,6 +180,8 @@ def end_schema(self, context: str | Sequence[str] | None = None, context_kwargs: # TODO: The _visit function above alters the schema in situ # force some context_kwargs context_kwargs["metadata"] = False + # Forward prefix normalisation into the inline @context. + context_kwargs.setdefault("normalize_prefixes", self.normalize_prefixes) add_prefixes = ContextGenerator(self.original_schema, **context_kwargs).serialize() add_prefixes_json = loads(add_prefixes) metamodel_ctx = self.metamodel_context or METAMODEL_CONTEXT_URI @@ -203,6 +206,10 @@ def end_schema(self, context: str | Sequence[str] | None = None, context_kwargs: self.schema["@context"].append({"@base": base_prefix}) # json_obj["@id"] = self.schema.id out = str(as_json(self.schema, indent=" ")) + "\n" + if self.deterministic: + from linkml.utils.generator import deterministic_json + + out = deterministic_json(json.loads(out), indent=2) + "\n" self.schema = self.original_schema return out diff --git a/packages/linkml/src/linkml/generators/jsonschemagen.py b/packages/linkml/src/linkml/generators/jsonschemagen.py index b9cfa6b883..0832424dc5 100644 --- a/packages/linkml/src/linkml/generators/jsonschemagen.py +++ b/packages/linkml/src/linkml/generators/jsonschemagen.py @@ -423,7 +423,18 @@ class JsonSchemaGenerator(Generator, LifecycleMixin): top_level_schema: JsonSchema = None include_null: bool = True - """Whether to include a "null" type in optional slots""" + """Whether optional (non-required) slots also accept an explicit JSON ``null``. + + When ``True`` (default) an optional slot is rendered with ``null`` added to its + type (e.g. ``["string", "null"]``), so an explicit ``null`` value validates. When + ``False`` the slot keeps its bare type and optionality is expressed solely by + absence from ``required``. + + JSON Schema treats presence (``required``, Validation 6.5.3) as separate from the + value type (``type``, Validation 6.1.1, where ``null`` is one of the value types), + and JSON ``null`` is a distinct value, not an absent member (RFC 8259 sec. 3). Set + this ``False`` for strict parity with reference schemas that declare a bare type + and forbid ``null``.""" preserve_names: bool = False """If true, preserve LinkML element names in JSON Schema output (e.g., for $defs, properties, $ref targets).""" @@ -828,6 +839,28 @@ def get_subschema_for_slot( else: typ = ["object", "null"] prop = JsonSchema({"type": typ, "additionalProperties": additionalProps}) + # In the inlined-dict form the mapping key *is* the value of the + # range's identifier/key slot, so constraints declared on that + # slot constrain the keys. Render them onto ``propertyNames`` + # (the JSON Schema key-constraint keyword, draft-06+) rather than + # dropping them. JSON object keys are always strings (JSON Schema + # Core 2019-09, 9.3.2.5), so only the string-applicable subset of + # the slot's constraints is emitted: ``pattern``, ``enum`` + # (``equals_string_in``) and a string ``const`` (``equals_string``). + # Numeric constraints (``minimum``/``maximum``, numeric ``const`` + # from ``equals_number``) and ``allOf`` are intentionally excluded + # -- they cannot match a string key (a numeric ``const`` would + # reject every key). Like value patterns, ``structured_pattern`` is + # honoured only when ``materialize_patterns`` is enabled. Backward + # compatible: emitted only when a key constraint applies. + slot_constraints = self.get_value_constraints_for_slot(range_id_slot) + key_constraints = JsonSchema( + {k: slot_constraints[k] for k in ("pattern", "enum") if k in slot_constraints} + ) + if isinstance(slot_constraints.get("const"), str): + key_constraints["const"] = slot_constraints["const"] + if key_constraints: + prop["propertyNames"] = key_constraints self.top_level_schema.add_lax_def(reference, self.aliased_slot_name(range_id_slot)) else: prop = JsonSchema.array_of(JsonSchema.ref_for(reference), include_null, required=slot.required) @@ -1064,6 +1097,13 @@ def serialize(self, **kwargs) -> str: show_default=True, help="If set, expand subproperty_of constraints to enum constraints.", ) +@click.option( + "--include-null/--no-include-null", + default=True, + show_default=True, + help="If set (default), optional slots also accept an explicit JSON null. " + "Use --no-include-null to forbid explicit null on optional slots.", +) @click.version_option(__version__, "-V", "--version") def cli(yamlfile, **kwargs): """Generate JSON Schema representation of a LinkML model""" diff --git a/packages/linkml/src/linkml/generators/owlgen.py b/packages/linkml/src/linkml/generators/owlgen.py index 4b06d1d7ea..bf84c96ac9 100644 --- a/packages/linkml/src/linkml/generators/owlgen.py +++ b/packages/linkml/src/linkml/generators/owlgen.py @@ -21,8 +21,9 @@ from linkml._version import __version__ from linkml.generators.common.subproperty import is_xsd_anyuri_range from linkml.utils.deprecation import deprecation_warning -from linkml.utils.generator import Generator, shared_arguments +from linkml.utils.generator import Generator, normalize_graph_prefixes, shared_arguments from linkml.utils.language_tags import LanguageTagResolver +from linkml.utils.rdf_canonicalize import canonicalize_rdf_graph from linkml_runtime import SchemaView from linkml_runtime.linkml_model.meta import ( AnonymousClassExpression, @@ -43,7 +44,7 @@ ) from linkml_runtime.utils.formatutils import camelcase, underscore from linkml_runtime.utils.introspection import package_schemaview -from linkml_runtime.utils.rdf_canonicalize import canonicalize_rdf_graph +from linkml_runtime.utils.yamlutils import YAMLRoot logger = logging.getLogger(__name__) @@ -56,6 +57,21 @@ SWRLB = rdflib.Namespace("http://www.w3.org/2003/11/swrlb#") +def _expression_sort_key(expr: YAMLRoot) -> str: + """Return a stable sort key for LinkML anonymous expressions. + + Used by ``--deterministic`` to order ``any_of``, ``all_of``, + ``none_of``, and ``exactly_one_of`` members reproducibly. + + This relies on ``YAMLRoot.__repr__()`` which formats objects using + their **field values** (not memory addresses). All anonymous + expression dataclasses in ``linkml_runtime.linkml_model.meta`` + use ``@dataclass(repr=False)`` and inherit this field-based repr, + so the output is deterministic across runs. + """ + return repr(expr) + + @unique class MetadataProfile(Enum): """ @@ -313,6 +329,10 @@ def as_graph(self) -> Graph: self.graph.bind(prefix, self.metamodel.namespaces[prefix]) for pfx in schema.prefixes.values(): self.graph.namespace_manager.bind(pfx.prefix_prefix, URIRef(pfx.prefix_reference)) + if self.normalize_prefixes: + normalize_graph_prefixes( + graph, {str(v.prefix_prefix): str(v.prefix_reference) for v in schema.prefixes.values()} + ) graph.add((base, RDF.type, OWL.Ontology)) # Add main schema elements @@ -348,6 +368,10 @@ def serialize(self, **kwargs: Any) -> str: """ self.as_graph() fmt = "turtle" if self.format in ["owl", "ttl"] else self.format + if self.deterministic and fmt == "turtle": + from linkml.utils.generator import deterministic_turtle + + return deterministic_turtle(self.graph) return canonicalize_rdf_graph(self.graph, output_format=fmt) def add_metadata(self, e: Definition | PermissibleValue, uri: URIRef) -> None: @@ -654,13 +678,17 @@ def transform_class_expression( own_slots = self.get_own_slots(cls) owl_exprs: list[OWL_EXPRESSION] = [] if cls.any_of: - any_of_expr = self._union_of([self.transform_class_expression(x) for x in cls.any_of]) + members = list(cls.any_of) + if self.deterministic: + members = sorted(members, key=_expression_sort_key) + any_of_expr = self._union_of([self.transform_class_expression(x) for x in members]) if any_of_expr: owl_exprs.append(any_of_expr) if cls.exactly_one_of: - sub_exprs: list[OWL_EXPRESSION] = self._present( - self.transform_class_expression(x) for x in cls.exactly_one_of - ) + members = list(cls.exactly_one_of) + if self.deterministic: + members = sorted(members, key=_expression_sort_key) + sub_exprs: list[OWL_EXPRESSION] = self._present(self.transform_class_expression(x) for x in members) if isinstance(cls, ClassDefinition): cls_uri = self._class_uri(cls.name) listnode = BNode() @@ -668,11 +696,11 @@ def transform_class_expression( graph.add((cls_uri, OWL.disjointUnionOf, listnode)) else: sub_sub_exprs: list[OWL_EXPRESSION] = [] - for i, x in enumerate(cls.exactly_one_of): + for i, x in enumerate(members): operand_expr = self.transform_class_expression(x) if not operand_expr: continue - rest = cls.exactly_one_of[0:i] + cls.exactly_one_of[i + 1 :] + rest = members[0:i] + members[i + 1 :] neg_expr = self._complement_of_union_of([self.transform_class_expression(nx) for nx in rest]) pos_expr = self._intersection_of([operand_expr, neg_expr]) if pos_expr: @@ -682,11 +710,17 @@ def transform_class_expression( owl_exprs.append(union_expr) # owl_exprs.extend(sub_exprs) if cls.all_of: - all_of_expr = self._intersection_of([self.transform_class_expression(x) for x in cls.all_of]) + members = list(cls.all_of) + if self.deterministic: + members = sorted(members, key=_expression_sort_key) + all_of_expr = self._intersection_of([self.transform_class_expression(x) for x in members]) if all_of_expr: owl_exprs.append(all_of_expr) if cls.none_of: - none_of_expr = self._complement_of_union_of([self.transform_class_expression(x) for x in cls.none_of]) + members = list(cls.none_of) + if self.deterministic: + members = sorted(members, key=_expression_sort_key) + none_of_expr = self._complement_of_union_of([self.transform_class_expression(x) for x in members]) if none_of_expr: owl_exprs.append(none_of_expr) for slot in own_slots: @@ -859,19 +893,29 @@ def _get_slot_nodes( ) return rdflib_nodes or None - if any_of_rdflib_nodes := _get_slot_nodes(slot.any_of): + def _maybe_sort_slots( + slot_definitions: Sequence[SlotDefinition | AnonymousSlotExpression] | None, + ) -> Sequence[SlotDefinition | AnonymousSlotExpression] | None: + if slot_definitions and self.deterministic: + return sorted(slot_definitions, key=_expression_sort_key) + return slot_definitions + + if any_of_rdflib_nodes := _get_slot_nodes(_maybe_sort_slots(slot.any_of)): owl_exprs.append(self._union_of(any_of_rdflib_nodes)) - if all_of_rdflib_nodes := _get_slot_nodes(slot.all_of): + if all_of_rdflib_nodes := _get_slot_nodes(_maybe_sort_slots(slot.all_of)): owl_exprs.append(self._intersection_of(all_of_rdflib_nodes)) - if none_of_rdflib_nodes := _get_slot_nodes(slot.none_of): + if none_of_rdflib_nodes := _get_slot_nodes(_maybe_sort_slots(slot.none_of)): owl_exprs.append(self._complement_of_union_of(none_of_rdflib_nodes)) if slot.exactly_one_of: + members = list(slot.exactly_one_of) + if self.deterministic: + members = sorted(members, key=_expression_sort_key) disj_exprs: list[OWL_EXPRESSION] = [] - for i, operand in enumerate(slot.exactly_one_of): + for i, operand in enumerate(members): operand_expr = self.transform_class_slot_expression(cls, operand, main_slot, owl_types) if not operand_expr: continue - rest = slot.exactly_one_of[0:i] + slot.exactly_one_of[i + 1 :] + rest = members[0:i] + members[i + 1 :] neg_expr = self._complement_of_union_of( [self.transform_class_slot_expression(cls, x, main_slot, owl_types) for x in rest], owl_types=owl_types, @@ -1154,7 +1198,10 @@ def add_enum(self, e: EnumDefinition) -> None: owl_types: list[URIRef | None] = [] enum_owl_type = self._get_metatype(e, self.default_permissible_value_type) - for pv in e.permissible_values.values(): + pvs = e.permissible_values.values() + if self.deterministic: + pvs = sorted(pvs, key=lambda x: x.text) + for pv in pvs: pv_owl_type = self._get_metatype(pv, enum_owl_type) owl_types.append(pv_owl_type) if pv_owl_type == RDFS.Literal: diff --git a/packages/linkml/src/linkml/generators/rdfgen.py b/packages/linkml/src/linkml/generators/rdfgen.py index a3fcf6a848..95d832f2b3 100644 --- a/packages/linkml/src/linkml/generators/rdfgen.py +++ b/packages/linkml/src/linkml/generators/rdfgen.py @@ -19,8 +19,8 @@ from linkml._version import __version__ from linkml.generators.jsonldgen import JSONLDGenerator from linkml.utils.generator import Generator, shared_arguments +from linkml.utils.rdf_canonicalize import canonicalize_rdf_graph from linkml_runtime.linkml_model import SchemaDefinition -from linkml_runtime.utils.rdf_canonicalize import canonicalize_rdf_graph @dataclass diff --git a/packages/linkml/src/linkml/generators/shaclgen.py b/packages/linkml/src/linkml/generators/shaclgen.py index afdd0cf953..7f29b3e668 100644 --- a/packages/linkml/src/linkml/generators/shaclgen.py +++ b/packages/linkml/src/linkml/generators/shaclgen.py @@ -1,4 +1,5 @@ import logging +import math import os import string from collections.abc import Callable @@ -14,11 +15,11 @@ from linkml.generators.common.subproperty import get_subproperty_values, is_uri_range from linkml.generators.shacl.shacl_data_type import ShaclDataType from linkml.generators.shacl.shacl_ifabsent_processor import ShaclIfAbsentProcessor -from linkml.utils.generator import Generator, shared_arguments +from linkml.utils.generator import Generator, normalize_graph_prefixes, shared_arguments from linkml.utils.language_tags import LanguageTagResolver -from linkml_runtime.linkml_model.meta import ClassDefinition, ElementName +from linkml.utils.rdf_canonicalize import canonicalize_rdf_graph +from linkml_runtime.linkml_model.meta import ClassDefinition, ElementName, PresenceEnum from linkml_runtime.utils.formatutils import underscore -from linkml_runtime.utils.rdf_canonicalize import canonicalize_rdf_graph from linkml_runtime.utils.yamlutils import TypedNode, extended_float, extended_int, extended_str logger = logging.getLogger(__name__) @@ -142,6 +143,26 @@ class ShaclGenerator(Generator): ignores any per-slot ``in_language``. """ + emit_rules: bool = True + """Emit ``sh:sparql`` constraints from LinkML ``rules:`` blocks. + + When ``True`` (default), recognised rule patterns are translated into + SHACL-SPARQL constraints (``sh:SPARQLConstraint``) on the corresponding + ``sh:NodeShape``. Currently three patterns are recognised: + + * *Boolean guard* — a precondition with ``value_presence: PRESENT`` on a + value slot and a postcondition with ``equals_string: "true"`` on a + boolean flag slot. + * *Presence implies value* — a precondition with ``value_presence: PRESENT`` + on a value slot and a postcondition with ``equals_string`` or + ``equals_string_in`` on a (typically enum-valued) target slot. This + generalises the boolean guard to arbitrary required values. + * *Exclusive value* — a precondition with ``equals_string`` on a slot and + a postcondition with ``maximum_cardinality`` on the *same* slot. + + See `W3C SHACL §5 `_ + and `linkml/linkml#2464 `_. + """ generatorname = os.path.basename(__file__) generatorversion = "0.0.1" valid_formats = ["ttl"] @@ -180,6 +201,10 @@ def generate_header(self) -> str: def serialize(self, **args) -> str: g = self.as_graph() fmt = "turtle" if self.format in ["owl", "ttl"] else self.format + if self.deterministic and fmt == "turtle": + from linkml.utils.generator import deterministic_turtle + + return deterministic_turtle(g) return canonicalize_rdf_graph(g, output_format=fmt) def as_graph(self) -> Graph: @@ -191,6 +216,10 @@ def as_graph(self) -> Graph: for pfx in self.schema.prefixes.values(): g.bind(str(pfx.prefix_prefix), pfx.prefix_reference) + if self.normalize_prefixes: + normalize_graph_prefixes( + g, {str(v.prefix_prefix): str(v.prefix_reference) for v in self.schema.prefixes.values()} + ) for c in sv.all_classes(imports=not self.exclude_imports).values(): @@ -275,9 +304,9 @@ def prop_pv_text(p, v): if msg_text: g.add((pnode, SH.message, Literal(msg_text, lang=self._resolve_language(None)))) # minCount - if s.minimum_cardinality: + if s.minimum_cardinality is not None: prop_pv_literal(SH.minCount, s.minimum_cardinality) - elif s.exact_cardinality: + elif s.exact_cardinality is not None: prop_pv_literal(SH.minCount, s.exact_cardinality) # Identifiers map to the node's IRI rather than a property triple, # so there's no arc to constrain with sh:minCount 1 — emitting it @@ -285,9 +314,9 @@ def prop_pv_text(p, v): elif s.required and not s.identifier: prop_pv_literal(SH.minCount, 1) # maxCount - if s.maximum_cardinality: + if s.maximum_cardinality is not None: prop_pv_literal(SH.maxCount, s.maximum_cardinality) - elif s.exact_cardinality: + elif s.exact_cardinality is not None: prop_pv_literal(SH.maxCount, s.exact_cardinality) elif not s.multivalued: prop_pv_literal(SH.maxCount, 1) @@ -343,6 +372,11 @@ def st_node_pv(p, v): add_simple_data_type(st_node_pv, r) range_list.append(st_node) + # Propagate pattern constraint to the branch node. + # A branch may combine range + pattern (e.g. range: string + # with pattern: "^...") or specify pattern alone (no range). + if any.pattern: + g.add((range_list[-1], SH.pattern, Literal(any.pattern))) Collection(g, or_node, range_list) else: prop_pv_literal(SH.hasValue, s.equals_number) @@ -389,10 +423,728 @@ def st_node_pv(p, v): if default_value: prop_pv(SH.defaultValue, default_value) + if self.emit_rules: + self._add_rules(g, class_uri_with_suffix, c) + return g LINKML_ANY_URI = "https://w3id.org/linkml/Any" + # ------------------------------------------------------------------- + # Rules → sh:sparql + # ------------------------------------------------------------------- + + def _add_rules(self, g: Graph, shape_uri: URIRef, cls: ClassDefinition) -> None: + """Emit ``sh:sparql`` constraints from LinkML ``rules:`` blocks. + + Each recognised rule is converted into an ``sh:SPARQLConstraint`` + attached to *shape_uri*. Unrecognised patterns are logged at + ``DEBUG`` level and silently skipped. + + Currently recognised patterns: + + * **Boolean guard** — a *precondition* with + ``value_presence: PRESENT`` on a value slot and a *postcondition* + with ``equals_string: "true"`` on a boolean flag slot. + + * **Presence implies value** — a *precondition* with + ``value_presence: PRESENT`` on a value slot and a *postcondition* + with ``equals_string`` or ``equals_string_in`` on a target slot. + Enforces that when the value slot is present, the target slot must + be present and hold one of the allowed values (generalises the + boolean guard to enum-valued targets). + + * **Exclusive value** — a *precondition* with ``equals_string`` on + a slot and a *postcondition* with ``maximum_cardinality`` on the + *same* slot. Enforces that when a specific value is present in a + multivalued slot, the total number of values must not exceed the + given cardinality (typically 1 for mutual exclusion). + + Operator combinations outside these named patterns are handled by a + small compositional fallback (:meth:`_compose_rule_sparql`) covering + conditional-required / conditional-absent postconditions, numeric + threshold and nested-object preconditions, and ``has_member`` list + membership. + + See `W3C SHACL §5 `_. + """ + if not cls.rules: + return + + sv = self.schemaview + for rule in cls.rules: + if getattr(rule, "deactivated", False): + continue + + if getattr(rule, "bidirectional", False): + logger.warning( + "Rule in class %r has bidirectional=true; " + "SHACL-SPARQL generation does not yet support bidirectional rules. " + "Only the forward direction is emitted.", + cls.name, + ) + + if getattr(rule, "open_world", False): + logger.warning( + "Rule in class %r has open_world=true; " + "SHACL operates under closed-world assumption. " + "The constraint is emitted but may not match open-world semantics.", + cls.name, + ) + + if getattr(rule, "elseconditions", None) is not None: + logger.warning( + "Rule in class %r has elseconditions; " + "SHACL-SPARQL generation emits the forward (if/then) direction only. " + "The else branch is not enforced.", + cls.name, + ) + + sparql_query = self._rule_to_sparql(sv, cls, rule) + if sparql_query is None: + logger.debug( + "Skipping unsupported rule pattern in class %r: %s", + cls.name, + getattr(rule, "description", "(no description)"), + ) + continue + + constraint = BNode() + g.add((shape_uri, SH.sparql, constraint)) + g.add((constraint, RDF.type, SH.SPARQLConstraint)) + + message = getattr(rule, "description", None) + if message: + g.add((constraint, SH.message, Literal(message))) + + g.add((constraint, SH.select, Literal(sparql_query))) + + # Fields on a slot condition / class expression that carry no constraint + # semantics: they never change which instances satisfy the condition, so + # they are ignored by the operator accounting below. Anything set on a + # condition that is neither here nor explicitly translated by a converter + # makes the rule untranslatable — the converters must SKIP such a rule + # rather than emit a query that silently drops a conjunct (which would + # widen the trigger or narrow the check: a mis-translation, not a skip). + _NON_OPERATOR_FIELDS = frozenset( + { + "name", + "description", + "title", + "deprecated", + "todos", + "notes", + "comments", + "examples", + "in_subset", + "from_schema", + "imported_from", + "source", + "in_language", + "see_also", + "deprecated_element_has_exact_replacement", + "deprecated_element_has_possible_replacement", + "aliases", + "structured_aliases", + "local_names", + "mappings", + "exact_mappings", + "close_mappings", + "related_mappings", + "narrow_mappings", + "broad_mappings", + "created_by", + "contributors", + "created_on", + "last_updated_on", + "modified_by", + "status", + "rank", + "categories", + "keywords", + "extensions", + "annotations", + "alt_descriptions", + "id_prefixes", + "id_prefixes_are_closed", + "definition_uri", + "conforms_to", + "implements", + "instantiates", + } + ) + + @classmethod + def _set_operator_fields(cls, cond) -> set[str]: + """Return the names of the constraint-bearing fields actually set on a + rule condition or class expression. + + A field counts as *set* when it is not ``None`` and not an empty + collection (SchemaView materialises unset multivalued fields as empty + lists / dicts). Scalars are never judged by truthiness, so legitimate + falsy constraints such as ``minimum_value: 0`` or + ``equals_string: ""`` still count as set. Metadata fields + (:data:`_NON_OPERATOR_FIELDS`) are excluded. + + The converters compare this set against the exact operator set they + translate and skip the rule on any mismatch, so an unrecognised (or + future-metamodel) operator can never be silently dropped. + """ + fields: set[str] = set() + for name, value in vars(cond).items(): + if name.startswith("_") or name in cls._NON_OPERATOR_FIELDS: + continue + if value is None: + continue + if isinstance(value, list | dict) and not value: + continue + if isinstance(value, JsonObj) and not as_dict(value): + continue + fields.add(name) + return fields + + def _rule_slot(self, sv, slot_name: str, cls: ClassDefinition): + """Resolve a rule condition's slot key to the slot it names, or ``None`` + when no such slot exists. + + Resolution order mirrors ``sh:path`` in the main slot loop: the induced + (class-specific) slot when the key names one of the class's slots, then + the underscored alias form (a rule key ``my_slot`` for a slot named + ``my slot`` — SchemaView normalises names the same way elsewhere), then + the base slot. Callers treat ``None`` as *unknown slot* and skip the + rule rather than fabricating a predicate no shape uses. + """ + class_slot_names = sv.class_slots(cls.name) + if slot_name in class_slot_names: + return sv.induced_slot(slot_name, cls.name) + canonical = next((s for s in class_slot_names if underscore(s) == underscore(slot_name)), None) + if canonical is not None: + return sv.induced_slot(canonical, cls.name) + return sv.get_slot(slot_name) + + def _rule_to_sparql(self, sv, cls: ClassDefinition, rule) -> str | None: + """Convert a ``ClassRule`` to a SPARQL SELECT query string. + + Returns ``None`` when the rule does not match any supported pattern. + Each pattern requires its conditions to set **exactly** the operators + it translates; a rule whose pre/postconditions carry anything more + (extra scalar operators, expression-level ``any_of``/``all_of``/ + ``none_of``/``exactly_one_of``, ...) is skipped rather than partially + translated. + """ + pre = getattr(rule, "preconditions", None) + post = getattr(rule, "postconditions", None) + if not pre or not post: + return None + + # Expression-level exactness: only a plain conjunction of slot + # conditions is translatable. An any_of/all_of/none_of/exactly_one_of + # branch cannot be honoured by any converter below; dropping it would + # widen the precondition (false positives) or weaken the postcondition + # (false negatives), so the whole rule is skipped. + if self._set_operator_fields(pre) != {"slot_conditions"}: + return None + if self._set_operator_fields(post) != {"slot_conditions"}: + return None + + pre_slots = pre.slot_conditions or {} + post_slots = post.slot_conditions or {} + + if len(pre_slots) == 1 and len(post_slots) == 1: + pre_slot_name = next(iter(pre_slots)) + post_slot_name = next(iter(post_slots)) + + pre_cond = pre_slots[pre_slot_name] + post_cond = post_slots[post_slot_name] + + pre_ops = self._set_operator_fields(pre_cond) + post_ops = self._set_operator_fields(post_cond) + + is_value_present = pre_ops == {"value_presence"} and pre_cond.value_presence == PresenceEnum( + PresenceEnum.PRESENT + ) + + # Pattern: boolean guard + # preconditions: exactly one slot with (only) value_presence PRESENT + # postconditions: exactly one boolean-range slot with (only) + # equals_string "true". The range gate matters: on a non-boolean + # slot the string "true" must be compared as a string, which is + # the presence-implies-value pattern below — without the gate the + # boolean comparison mistranslates and flags conforming data. + if ( + is_value_present + and post_ops == {"equals_string"} + and post_cond.equals_string == "true" + and getattr(self._rule_slot(sv, post_slot_name, cls), "range", None) == "boolean" + ): + return self._build_boolean_guard_sparql(sv, cls, post_slot_name, pre_slot_name) + + # Pattern: presence implies value (enum guard) + # preconditions: value slot with (only) value_presence PRESENT + # postconditions: target slot with (only) equals_string or (only) + # equals_string_in. + # Semantics: "If the value slot is present, the target slot must be + # present and hold one of the allowed values." Generalises the + # boolean guard (equals_string "true") to arbitrary enum values. + if is_value_present and post_ops in ({"equals_string"}, {"equals_string_in"}): + if post_ops == {"equals_string_in"}: + allowed = list(post_cond.equals_string_in) + else: + allowed = [post_cond.equals_string] + return self._build_presence_implies_value_sparql(sv, cls, pre_slot_name, post_slot_name, allowed) + + # Pattern: exclusive value + # preconditions: slot X with (only) equals_string (a specific enum value) + # postconditions: same slot X with (only) maximum_cardinality N + # Semantics: "If value V is present in slot X, then X has at most N values." + if pre_ops == {"equals_string"} and post_ops == {"maximum_cardinality"} and pre_slot_name == post_slot_name: + return self._build_exclusive_value_sparql( + sv, cls, pre_slot_name, pre_cond.equals_string, int(post_cond.maximum_cardinality) + ) + + # Fallback: a small compositional builder for operator combinations not + # covered by the three named patterns above (conditional-required, + # threshold preconditions, list membership, ...). Tried only after the + # named patterns, so their output is unchanged. + composed = self._compose_rule_sparql(sv, cls, rule) + if composed is not None: + return composed + + return None + + def _compose_rule_sparql(self, sv, cls: ClassDefinition, rule) -> str | None: + """Compose a SHACL-SPARQL violation query for rule shapes not covered + by the three named patterns. + + Translates a conjunction of *precondition* slot conditions and a single + *postcondition* slot condition into one ``SELECT $this`` query that + selects focus nodes which satisfy every precondition but violate the + postcondition. Supported operators grow incrementally in + :meth:`_precondition_patterns` and :meth:`_postcondition_violation`; + the method returns ``None`` (rule skipped, never mis-translated) as soon + as any operator is unsupported. + + A rule's ``postconditions`` are a conjunction, so violating a single + slot condition is sufficient; the single-postcondition case covers the + modeled cross-parameter rules. + + Conforms to `SHACL §5.3.1 + `_: ``$this`` + is pre-bound to each focus node. + """ + pre = getattr(rule, "preconditions", None) + post = getattr(rule, "postconditions", None) + if not pre or not post: + return None + + pre_slots = getattr(pre, "slot_conditions", None) or {} + post_slots = getattr(post, "slot_conditions", None) or {} + if not pre_slots or len(post_slots) != 1: + return None + + pre_lines = self._precondition_patterns(sv, cls, pre_slots) + if pre_lines is None: + return None + + post_slot_name, post_cond = next(iter(post_slots.items())) + violation = self._postcondition_violation(sv, cls, post_slot_name, post_cond) + if violation is None: + return None + + body = "\n".join(f" {line}" for line in (pre_lines + violation)) + return f"SELECT $this WHERE {{\n{body}\n}}" + + def _scalar_filters(self, var: str, cond, resolve: Callable[[str], str]) -> list[str] | None: + """Return the SPARQL ``FILTER`` lines for the scalar operators on *cond*. + + Unlike a first-match dispatch, **every** recognised operator contributes + a line, so a condition combining operators — e.g. a bounded range + ``{minimum_value: X, maximum_value: Y}`` — emits *both* bounds instead of + silently keeping only the first and under-constraining the query. + ``value_presence: PRESENT`` contributes no filter (the caller's triple + binding already enforces presence). + + *resolve* maps an ``equals_string`` value to a SPARQL term (an enum + ``meaning`` IRI or an escaped string literal). + + Returns ``None`` when *cond* sets no recognised scalar operator, sets + any operator *outside* the recognised set (per + :meth:`_set_operator_fields` — partial translation would drop a + conjunct), sets ``value_presence`` to anything but ``PRESENT``, or + carries a non-numeric threshold bound. In every such case the caller + skips the rule it cannot faithfully translate rather than emitting an + under-constrained (or vacuous) query. + """ + op_fields = self._set_operator_fields(cond) + if not op_fields or not op_fields <= {"value_presence", "equals_string", "minimum_value", "maximum_value"}: + return None # unsupported operator present (or none at all): skip + if "value_presence" in op_fields and cond.value_presence != PresenceEnum(PresenceEnum.PRESENT): + # ABSENT (or a future presence value) cannot be expressed as a + # triple binding + filter; translating the other operators anyway + # would invert the declared trigger. + return None + + filters: list[str] = [] + if "equals_string" in op_fields: + filters.append(f"FILTER ( {var} = {resolve(cond.equals_string)} )") + if "minimum_value" in op_fields: + minimum = self._sparql_number(cond.minimum_value) + if minimum is None: + return None + filters.append(f"FILTER ( {var} >= {minimum} )") + if "maximum_value" in op_fields: + maximum = self._sparql_number(cond.maximum_value) + if maximum is None: + return None + filters.append(f"FILTER ( {var} <= {maximum} )") + return filters + + def _precondition_patterns(self, sv, cls: ClassDefinition, pre_slots) -> list[str] | None: + """Translate a conjunction of precondition slot conditions into SPARQL + graph patterns (plus ``FILTER`` lines) that bind focus nodes satisfying + every condition. + + Returns ``None`` if any condition sets no recognised operator. + + Supported operators, which **combine** on a single condition (so a + bounded range ``{minimum_value: X, maximum_value: Y}`` emits both + bounds): ``value_presence: PRESENT``, ``equals_string``, and the numeric + thresholds ``minimum_value`` / ``maximum_value`` (inclusive, per the + LinkML metamodel). A ``range_expression`` with inner ``slot_conditions`` + reaches one hop into an inlined child object. + """ + lines: list[str] = [] + for i, (slot_name, cond) in enumerate(pre_slots.items()): + path = self._slot_uri(sv, slot_name, cls) + if path is None: + return None + var = f"?pre{i}" + if self._set_operator_fields(cond) == {"range_expression"}: + # One-hop into an inlined child object: bind the child node and + # apply the inner slot conditions to it. The nested expression + # must itself be a plain conjunction of slot conditions; a + # condition mixing range_expression with scalar operators (or a + # nested any_of/...) is skipped rather than partially + # translated. + range_expr = cond.range_expression + if self._set_operator_fields(range_expr) != {"slot_conditions"}: + return None + node = f"{var}_node" + lines.append(f"$this <{path}> {node} .") + inner = self._member_conditions(sv, cls, slot_name, node, range_expr.slot_conditions) + if inner is None: + return None + lines.extend(inner) + continue + filters = self._scalar_filters( + var, cond, lambda v, sn=slot_name: self._resolve_enum_value_ref(sv, sn, v, cls) + ) + if filters is None: + return None + lines.append(f"$this <{path}> {var} .") + lines.extend(filters) + return lines + + def _member_conditions( + self, sv, cls: ClassDefinition, container_slot_name: str, node_var: str, slot_conditions + ) -> list[str] | None: + """Constrain the object bound to *node_var* — an instance of the range + class of *container_slot_name* — by a set of inner slot conditions. + + Shared by the nested ``range_expression`` precondition (single inlined + child) and the ``has_member`` postcondition (a list member). Inner + slots live on the container slot's **range class**, so both their + property IRIs and their enum values are resolved in that class's + induced context (the container slot itself is induced against *cls*, + honouring a ``slot_usage`` range narrowing). Resolving against the + outer class instead would emit predicates the member nodes never carry + — the ``sh:path`` on the member shape and the SPARQL body would + diverge, making ``FILTER NOT EXISTS`` member checks vacuously true + (false positives) or preconditions never bind (false negatives). + + Like preconditions, combining operators on one condition emits all of + them. Returns ``None`` for unsupported inner operators or when the + container's range is not a class (inner conditions on a non-class + range cannot be resolved faithfully). + """ + container = self._rule_slot(sv, container_slot_name, cls) + range_name = getattr(container, "range", None) + if not range_name or range_name not in sv.all_classes(): + return None + range_cls = sv.get_class(range_name) + + lines: list[str] = [] + for j, (inner_name, icond) in enumerate(slot_conditions.items()): + ipath = self._slot_uri(sv, inner_name, range_cls) + if ipath is None: + return None + ivar = f"{node_var}_{j}" + filters = self._scalar_filters( + ivar, + icond, + lambda v, inm=inner_name: self._resolve_enum_value_ref(sv, inm, v, range_cls), + ) + if filters is None: + return None + lines.append(f"{node_var} <{ipath}> {ivar} .") + lines.extend(filters) + return lines + + @staticmethod + def _sparql_number(value) -> str | None: + """Render a numeric threshold bound as a SPARQL numeric literal, or + ``None`` when the value is not a finite number (callers then skip the + rule). + + The metamodel range of ``minimum_value`` / ``maximum_value`` is + ``Anything``, so YAML strings, dates, booleans, ``.nan`` / ``.inf`` + all pass through SchemaView unchanged. Interpolating them raw is + unsound: ``"abc"`` yields unparsable SPARQL that poisons the whole + shapes graph at validation time, and a date like ``2020-01-01`` parses + as the arithmetic expression ``2020-01-01 = 2018`` and silently never + fires. Only ``int`` / finite ``float`` (including the + ``extended_int`` / ``extended_float`` runtime subclasses) are + rendered; their ``str`` yields a plain numeric token (e.g. ``4000`` or + ``0.0``) that SPARQL compares with numeric promotion against + ``xsd:float`` / ``xsd:decimal`` data values. + """ + if isinstance(value, bool) or not isinstance(value, int | float): + return None + if isinstance(value, float) and not math.isfinite(value): + return None + return str(value) + + @staticmethod + def _sparql_string_literal(value: str) -> str: + """Render *value* as a double-quoted SPARQL string literal, escaping the + characters the grammar forbids raw. + + ``equals_string`` / permissible-value names are schema-controlled but + may legitimately contain a double quote, backslash, or newline; without + escaping these would break the ``sh:select`` query (or allow SPARQL + injection). See `SPARQL 1.1 §19.7 escape sequences + `_. + """ + escaped = ( + str(value) + .replace("\\", "\\\\") + .replace('"', '\\"') + .replace("\n", "\\n") + .replace("\r", "\\r") + .replace("\t", "\\t") + ) + return f'"{escaped}"' + + def _postcondition_violation(self, sv, cls: ClassDefinition, slot_name: str, cond) -> list[str] | None: + """Translate a single postcondition slot condition into SPARQL that + matches a *violation* of it. + + Returns ``None`` for operators not handled here, or when the condition + sets anything beyond the single operator a branch translates (dropping + a co-set operator would weaken the postcondition — the rule is skipped + instead). + + Supported operators: + + * ``required: true`` — violation = the target slot is absent on a focus + node that satisfies the preconditions. + * ``value_presence: ABSENT`` — violation = the target slot *is* present + (inapplicable-slot / conditional-absent). + * ``has_member`` with a nested ``range_expression`` — violation = *no* + member of the (multivalued) target slot matches the inner conditions + (list-membership; e.g. the light-group list must contain a + ``{group: Vehicle, type: front_fog_light}`` entry). + """ + path = self._slot_uri(sv, slot_name, cls) + if path is None: + return None + op_fields = self._set_operator_fields(cond) + if op_fields == {"required"} and cond.required is True: + return [f"FILTER NOT EXISTS {{ $this <{path}> ?post . }}"] + if op_fields == {"value_presence"} and cond.value_presence == PresenceEnum(PresenceEnum.ABSENT): + return [f"$this <{path}> ?post ."] + if op_fields == {"has_member"}: + has_member = cond.has_member + if self._set_operator_fields(has_member) != {"range_expression"}: + return None + range_expr = has_member.range_expression + if self._set_operator_fields(range_expr) != {"slot_conditions"}: + return None + member_lines = [f"$this <{path}> ?mem ."] + inner = self._member_conditions(sv, cls, slot_name, "?mem", range_expr.slot_conditions) + if inner is None: + return None + member_lines.extend(inner) + block = " ".join(member_lines) + return [f"FILTER NOT EXISTS {{ {block} }}"] + return None + + def _build_boolean_guard_sparql( + self, sv, cls: ClassDefinition, flag_slot_name: str, value_slot_name: str + ) -> str | None: + """Build a SPARQL SELECT query for the boolean-guard pattern. + + The query detects violations where the value property is present + but the boolean flag is absent or not ``true``. Returns ``None`` + (rule skipped) when either slot name resolves to no slot. + + Conforms to `SHACL §5.3.1 + `_: + ``$this`` is pre-bound to each focus node. + """ + flag_uri = self._slot_uri(sv, flag_slot_name, cls) + value_uri = self._slot_uri(sv, value_slot_name, cls) + if flag_uri is None or value_uri is None: + return None + + return ( + f"SELECT $this WHERE {{\n" + f" OPTIONAL {{ $this <{flag_uri}> ?flag . }}\n" + f" OPTIONAL {{ $this <{value_uri}> ?value . }}\n" + f" FILTER (\n" + f" ( !BOUND(?flag) || ?flag != true ) &&\n" + f" BOUND(?value)\n" + f" )\n" + f"}}" + ) + + def _build_presence_implies_value_sparql( + self, + sv, + cls: ClassDefinition, + value_slot_name: str, + target_slot_name: str, + allowed_values: list[str], + ) -> str | None: + """Build a SPARQL SELECT query for the presence-implies-value pattern. + + Detects violations where the *value slot* is present but the *target + slot* is absent or holds a value outside the allowed set. This + generalises the boolean-guard pattern to enum-valued targets: it + supports a single required value (``equals_string``) or a set of + acceptable values (``equals_string_in``). + + Each allowed value is resolved via the target slot's enum ``meaning`` + to a full IRI; values without a ``meaning`` (or non-enum targets) fall + back to a plain string literal. + + Conforms to `SHACL §5.3.1 + `_: + ``$this`` is pre-bound to each focus node. + """ + value_uri = self._slot_uri(sv, value_slot_name, cls) + target_uri = self._slot_uri(sv, target_slot_name, cls) + if value_uri is None or target_uri is None: + return None + refs = ", ".join(self._resolve_enum_value_ref(sv, target_slot_name, v, cls) for v in allowed_values) + + return ( + f"SELECT $this WHERE {{\n" + f" $this <{value_uri}> ?value .\n" + f" OPTIONAL {{ $this <{target_uri}> ?target . }}\n" + f" FILTER ( !BOUND(?target) || ?target NOT IN ({refs}) )\n" + f"}}" + ) + + def _build_exclusive_value_sparql( + self, + sv, + cls: ClassDefinition, + slot_name: str, + value_name: str, + max_card: int, + ) -> str | None: + """Build a SPARQL SELECT query for the exclusive-value pattern. + + Detects violations where a specific value is present in a multivalued + slot but the total number of values exceeds *max_card*. + + For the common case ``max_card == 1``, the query checks whether the + exclusive value coexists with any other value (simple existence test). + For ``max_card > 1``, a subquery counts all values and checks against + the limit. + + The exclusive value is resolved to its full IRI via the slot's enum + ``meaning`` field. If the slot is not an enum or the value has no + ``meaning``, the value is compared as a plain literal. + + Conforms to `SHACL §5.3.1 + `_: + ``$this`` is pre-bound to each focus node. + """ + slot_uri = self._slot_uri(sv, slot_name, cls) + if slot_uri is None: + return None + value_ref = self._resolve_enum_value_ref(sv, slot_name, value_name, cls) + + if max_card == 1: + return ( + f"SELECT $this WHERE {{\n" + f" $this <{slot_uri}> {value_ref} .\n" + f" $this <{slot_uri}> ?other .\n" + f" FILTER (?other != {value_ref})\n" + f"}}" + ) + + return ( + f"SELECT $this WHERE {{\n" + f" $this <{slot_uri}> {value_ref} .\n" + f" {{\n" + f" SELECT $this (COUNT(?val) AS ?count)\n" + f" WHERE {{ $this <{slot_uri}> ?val . }}\n" + f" GROUP BY $this\n" + f" HAVING (?count > {max_card})\n" + f" }}\n" + f"}}" + ) + + def _resolve_enum_value_ref(self, sv, slot_name: str, value_name: str, cls: ClassDefinition | None = None) -> str: + """Resolve an enum value name to a SPARQL term (IRI or literal). + + Looks up the slot's range as an enum, finds the permissible value + matching *value_name*, and returns its ``meaning`` as a full IRI + wrapped in angle brackets. Falls back to an escaped quoted literal if + the slot is not an enum or the value lacks a ``meaning``. + + When *cls* is given and the slot is declared on it, the slot is resolved + in the class's induced context, so a range narrowed via ``slot_usage`` + (a class-specific enum) selects the correct permissible values instead + of the base slot's enum. + """ + slot = self._rule_slot(sv, slot_name, cls) if cls is not None else sv.get_slot(slot_name) + if slot: + range_name = slot.range + if range_name and range_name in sv.all_enums(): + enum = sv.get_enum(range_name) + pv = enum.permissible_values.get(value_name) + if pv and pv.meaning: + iri = sv.expand_curie(pv.meaning) + return f"<{iri}>" + return self._sparql_string_literal(value_name) + + def _slot_uri(self, sv, slot_name: str, cls: ClassDefinition) -> str | None: + """Resolve a slot name to a full IRI string for use in SPARQL queries, + or ``None`` when the name resolves to no slot at all (callers then skip + the rule). + + Mirrors the resolution logic used for ``sh:path`` in the main slot loop, + including the **induced** (class-specific) slot: a ``slot_usage`` + override of ``slot_uri`` must yield the same IRI as ``sh:path``. + Otherwise the SPARQL body would query a property the data never uses and + the constraint would silently never fire (a false negative). A slot + that resolves but is not registered in the schema's element map falls + back to ``default_prefix:underscored_name``, again matching ``sh:path``; + an *unknown* name must NOT take that fallback — it would fabricate a + predicate no shape uses and emit a vacuous constraint. + """ + slot = self._rule_slot(sv, slot_name, cls) + if slot is None: + return None + if slot.name in sv.element_by_schema_map(): + return sv.get_uri(slot, expand=True) + pfx = sv.schema.default_prefix + return sv.expand_curie(f"{pfx}:{underscore(slot_name)}") + def _add_class(self, func: Callable, r: ElementName) -> None: """Add an sh:class constraint for range class *r*. @@ -418,13 +1170,13 @@ def _add_enum(self, g: Graph, func: Callable, r: ElementName) -> None: sv = self.schemaview enum = sv.get_enum(r) pv_node = BNode() + pv_items = list(enum.permissible_values.items()) + if self.deterministic: + pv_items = sorted(pv_items, key=lambda x: x[0]) Collection( g, pv_node, - [ - URIRef(sv.expand_curie(pv.meaning)) if pv.meaning else Literal(pv_name) - for pv_name, pv in enum.permissible_values.items() - ], + [URIRef(sv.expand_curie(pv.meaning)) if pv.meaning else Literal(pv_name) for pv_name, pv in pv_items], ) func(SH["in"], pv_node) @@ -660,6 +1412,18 @@ def add_simple_data_type(func: Callable, r: ElementName) -> None: 'Example: "{name} ({class}): {description} [{comments}]"' ), ) +@click.option( + "--emit-rules/--no-emit-rules", + default=True, + show_default=True, + help=( + "Emit sh:sparql constraints from LinkML rules: blocks. " + "When enabled (default), recognised rule patterns (boolean-guard, " + "presence-implies-value, exclusive-value) are translated into " + "SHACL-SPARQL constraints on the corresponding " + "sh:NodeShape. Use --no-emit-rules to suppress rule generation." + ), +) @click.version_option(__version__, "-V", "--version") def cli(yamlfile, **args): """Generate SHACL turtle from a LinkML model""" diff --git a/packages/linkml/src/linkml/generators/shexgen.py b/packages/linkml/src/linkml/generators/shexgen.py index 40a93ffbc9..704dd1ae61 100644 --- a/packages/linkml/src/linkml/generators/shexgen.py +++ b/packages/linkml/src/linkml/generators/shexgen.py @@ -15,6 +15,7 @@ from linkml._version import __version__ from linkml.generators.common.subproperty import get_subproperty_values from linkml.utils.generator import Generator, shared_arguments +from linkml.utils.rdf_canonicalize import canonicalize_rdf_graph from linkml_runtime.linkml_model.meta import ( ClassDefinition, ElementName, @@ -26,7 +27,6 @@ from linkml_runtime.linkml_model.types import SHEX from linkml_runtime.utils.formatutils import camelcase, sfx from linkml_runtime.utils.metamodelcore import URIorCURIE -from linkml_runtime.utils.rdf_canonicalize import canonicalize_rdf_graph @dataclass diff --git a/packages/linkml/src/linkml/utils/generator.py b/packages/linkml/src/linkml/utils/generator.py index 88fc485851..bca8ce30e7 100644 --- a/packages/linkml/src/linkml/utils/generator.py +++ b/packages/linkml/src/linkml/utils/generator.py @@ -20,11 +20,12 @@ import os import re import sys +import types from collections.abc import Callable, Mapping from dataclasses import dataclass, field from functools import lru_cache from pathlib import Path -from typing import ClassVar, TextIO, Union, cast +from typing import TYPE_CHECKING, ClassVar, TextIO, Union, cast import click from click import Argument, Command, Option @@ -37,6 +38,10 @@ from linkml.utils.schemaloader import SchemaLoader from linkml.utils.typereferences import References from linkml_runtime import SchemaView + +if TYPE_CHECKING: + from rdflib import Graph as RdfGraph + from linkml_runtime.linkml_model.meta import ( ClassDefinition, ClassDefinitionName, @@ -58,6 +63,9 @@ from linkml_runtime.utils.formatutils import camelcase, underscore from linkml_runtime.utils.namespaces import Namespaces +if TYPE_CHECKING: + from rdflib import Graph + logger = logging.getLogger(__name__) @@ -78,6 +86,428 @@ def _resolved_metamodel(mergeimports): return metamodel +def _wl_signatures( + quads: list, + iterations: int = 4, +) -> dict[str, str]: + """Compute Weisfeiler-Lehman structural signatures for blank nodes. + + Uses 1-dimensional WL colour refinement [1]_ to assign each blank + node a deterministic signature derived from its multi-hop + neighbourhood structure. The signature depends only on predicate + IRIs, literal values, and named-node IRIs — **not** on blank-node + identifiers — so it remains stable when unrelated triples are added + or removed. + + Parameters + ---------- + quads : list + Canonical quads from pyoxigraph (after RDFC-1.0). + iterations : int + Number of WL refinement rounds (default 4). + + Returns + ------- + dict[str, str] + Mapping from canonical blank-node ID (e.g. ``c14n42``) to a + truncated SHA-256 hash suitable for use as a stable blank-node + label. + + References + ---------- + .. [1] Weisfeiler, B. & Leman, A. (1968). "The reduction of a graph + to canonical form and the algebra which appears therein." + """ + import hashlib + + import pyoxigraph # guaranteed available — caller (deterministic_turtle) checks + + # Collect all blank node IDs and build adjacency index. + bnode_ids: set[str] = set() + # outgoing[b] = list of (predicate_str, object_str_or_bnode_id, is_bnode) + outgoing: dict[str, list[tuple[str, str, bool]]] = {} + # incoming[b] = list of (subject_str_or_bnode_id, predicate_str, is_bnode) + incoming: dict[str, list[tuple[str, str, bool]]] = {} + + for q in quads: + s, p, o = q.subject, q.predicate, q.object + s_is_bn = isinstance(s, pyoxigraph.BlankNode) + o_is_bn = isinstance(o, pyoxigraph.BlankNode) + p_str = str(p) + + if s_is_bn: + bnode_ids.add(s.value) + outgoing.setdefault(s.value, []).append((p_str, o.value if o_is_bn else str(o), o_is_bn)) + if o_is_bn: + bnode_ids.add(o.value) + incoming.setdefault(o.value, []).append((s.value if s_is_bn else str(s), p_str, s_is_bn)) + + # Initialise signatures: named-node edges only (no bnode IDs). + sig: dict[str, str] = {} + for bid in bnode_ids: + parts = [] + for p_str, o_str, o_is_bn in outgoing.get(bid, []): + if not o_is_bn: + parts.append(f"+{p_str}={o_str}") + for s_str, p_str, s_is_bn in incoming.get(bid, []): + if not s_is_bn: + parts.append(f"-{s_str}={p_str}") + sig[bid] = "|".join(sorted(parts)) + + # Iterative refinement: incorporate neighbour signatures. + for _ in range(iterations): + new_sig: dict[str, str] = {} + for bid in bnode_ids: + parts = [sig[bid]] + for p_str, o_str, o_is_bn in outgoing.get(bid, []): + if o_is_bn: + parts.append(f"+{p_str}={sig.get(o_str, '')}") + for s_str, p_str, s_is_bn in incoming.get(bid, []): + if s_is_bn: + parts.append(f"-{sig.get(s_str, '')}={p_str}") + new_sig[bid] = "|".join(sorted(parts)) + sig = new_sig + + # Convert signatures to truncated SHA-256 hashes. + # Use 12 hex chars (48 bits) — birthday-bound collision probability + # is ~n²/2^49: ~0.002% at 100k nodes. Collisions are handled by + # appending a counter (see below), so correctness is preserved. + hash_map: dict[str, str] = {} + seen_hashes: dict[str, int] = {} + for bid in sorted(bnode_ids): + digest = hashlib.sha256(sig[bid].encode("utf-8")).hexdigest()[:12] + # Handle collisions by appending a counter. + count = seen_hashes.get(digest, 0) + seen_hashes[digest] = count + 1 + label = f"b{digest}" if count == 0 else f"b{digest}_{count}" + hash_map[bid] = label + + return hash_map + + +def deterministic_turtle(graph: "RdfGraph") -> str: + """Serialize an RDF graph to Turtle with deterministic output ordering. + + Uses a three-phase hybrid pipeline for **correctness**, **diff + stability**, and **readability**: + + 1. **RDFC-1.0** [1]_ (via ``pyoxigraph``) canonicalizes the graph, + ensuring isomorphic inputs produce identical triple sets. + 2. **Weisfeiler-Lehman structural hashing** replaces the sequential + ``_:c14nN`` identifiers with content-based hashes derived from + each blank node's multi-hop neighbourhood. These hashes depend + only on predicate IRIs, literal values, and named-node IRIs — + not on blank-node numbering — so adding or removing a triple + only affects the identifiers of directly involved blank nodes. + 3. **Hybrid rdflib re-serialization** parses the canonicalized, + WL-hashed triples back into an rdflib ``Graph`` and serializes + with rdflib's native Turtle writer. This recovers idiomatic + Turtle features that pyoxigraph cannot emit: + + - **Inline blank nodes** (``[ … ]``) for singly-referenced + blank nodes (Turtle §2.7 [2]_), instead of verbose named + ``_:bHASH`` syntax. + - **Collection syntax** (``( … )``) for ``rdf:List`` chains + (Turtle §2.8 [2]_). + - **Prefix filtering**: only prefixes actually used in the + graph's IRIs are declared, following the practice of Apache + Jena, Eclipse RDF4J, and Raptor. + + All triples from the source graph are preserved — the hybrid step + only changes syntactic form, never semantic content. + + Parameters + ---------- + graph : rdflib.Graph + An rdflib Graph to serialize. + + Returns + ------- + str + Deterministic Turtle string with ``@prefix`` declarations. + + References + ---------- + .. [1] W3C (2024). "RDF Dataset Canonicalization (RDFC-1.0)." + W3C Recommendation. https://www.w3.org/TR/rdf-canon/ + .. [2] W3C (2014). "RDF 1.1 Turtle — Terse RDF Triple Language." + W3C Recommendation. https://www.w3.org/TR/turtle/ + """ + try: + import pyoxigraph + except ImportError as exc: + raise ImportError( + "pyoxigraph >= 0.4.0 is required for --deterministic output. " + "Install it with: pip install 'pyoxigraph>=0.4.0'" + ) from exc + + from rdflib import BNode, Graph, Literal, URIRef + + # ── Phase 1: RDFC-1.0 canonicalization ────────────────────────── + nt_data = graph.serialize(format="nt") + + dataset = pyoxigraph.Dataset(pyoxigraph.parse(nt_data, format=pyoxigraph.RdfFormat.N_TRIPLES)) + dataset.canonicalize(pyoxigraph.CanonicalizationAlgorithm.RDFC_1_0) + + canonical_quads = list(dataset) + + # ── Phase 2: WL structural hashing for diff-stable blank node IDs + wl_map = _wl_signatures(canonical_quads) + + def _remap(term): + if isinstance(term, pyoxigraph.BlankNode) and term.value in wl_map: + return pyoxigraph.BlankNode(wl_map[term.value]) + return term + + remapped = [pyoxigraph.Triple(_remap(q.subject), q.predicate, _remap(q.object)) for q in canonical_quads] + + # ── Phase 3: Hybrid rdflib re-serialization ───────────────────── + # Convert pyoxigraph terms to rdflib terms and populate a clean + # Graph that only carries explicitly-bound prefixes. + def _to_rdflib(term): + """Convert a pyoxigraph term to the equivalent rdflib term.""" + if isinstance(term, pyoxigraph.NamedNode): + return URIRef(term.value) + if isinstance(term, pyoxigraph.BlankNode): + return BNode(term.value) + if isinstance(term, pyoxigraph.Literal): + if term.language: + return Literal(term.value, lang=term.language) + if term.datatype: + dt_iri = term.datatype.value + # In RDF 1.1, simple literals are syntactic sugar for + # xsd:string (Turtle §2.5.1). Preserve the shorter form + # to match the original owlgen output and avoid spurious + # diffs on every string literal. + if dt_iri == "http://www.w3.org/2001/XMLSchema#string": + return Literal(term.value) + return Literal(term.value, datatype=URIRef(dt_iri)) + return Literal(term.value) + raise TypeError(f"Unexpected pyoxigraph term type: {type(term).__name__}: {term}") + + result_graph = Graph(bind_namespaces="none") + for triple in remapped: + result_graph.add( + ( + _to_rdflib(triple.subject), + _to_rdflib(triple.predicate), + _to_rdflib(triple.object), + ) + ) + + # Bind only prefixes whose namespace IRI is actually referenced + # by at least one subject, predicate, or object in the graph. + # This filters out rdflib's ~27 built-in default bindings + # (brick, csvw, doap, …) that leak through Graph() even when + # the schema never declared them. + used_iris: set[str] = set() + for s, p, o in result_graph: + for term in (s, p, o): + if isinstance(term, URIRef): + used_iris.add(str(term)) + + for pfx, ns in sorted(graph.namespaces()): + pfx_s, ns_s = str(pfx), str(ns) + if pfx_s and any(iri.startswith(ns_s) for iri in used_iris): + result_graph.bind(pfx_s, ns_s) + + # rdflib's Turtle serializer always emits a trailing double newline; + # normalize to a single newline for consistent file endings. + return result_graph.serialize(format="turtle").rstrip("\n") + "\n" + + +def deterministic_json(obj: object, indent: int = 3, preserve_list_order_keys: frozenset[str] | None = None) -> str: + """Serialize a JSON-compatible object with deterministic ordering. + + Recursively sorts all dict keys *and* list elements to produce + stable output across Python versions and process invocations. + + List elements are sorted by their canonical JSON representation + (``json.dumps(item, sort_keys=True)``), which handles lists of + dicts, strings, and mixed types. + + :param obj: A JSON-serializable object (typically parsed from ``as_json``). + :param indent: Number of spaces for indentation. + :param preserve_list_order_keys: Dict keys whose list values must NOT be + sorted (e.g. ``@context``, ``@list`` in JSON-LD where array order is + semantic). Defaults to ``_JSONLD_ORDERED_KEYS``. + :returns: Deterministic JSON string. + """ + import json + + skip = preserve_list_order_keys if preserve_list_order_keys is not None else _JSONLD_ORDERED_KEYS + + def _deep_sort(value: object, parent_key: str = "") -> object: + if isinstance(value, dict): + return {k: _deep_sort(v, parent_key=k) for k, v in sorted(value.items())} + if isinstance(value, list): + sorted_items = [_deep_sort(item) for item in value] + if parent_key in skip: + return sorted_items + try: + return sorted(sorted_items, key=lambda x: json.dumps(x, sort_keys=True, ensure_ascii=False)) + except TypeError: + return sorted_items + return value + + return json.dumps(_deep_sort(obj), indent=indent, ensure_ascii=False) + + +# JSON-LD keys whose array values carry ordering semantics and must not +# be sorted. @context arrays define an override cascade (JSON-LD 1.1 +# §4.1); @list containers are explicitly ordered; @graph and @set are +# included defensively. +_JSONLD_ORDERED_KEYS: frozenset[str] = frozenset({"@context", "@list", "@graph", "@set", "imports"}) + + +def well_known_prefix_map() -> dict[str, str]: + """Return a mapping from namespace URI to standard prefix name. + + Primary source: the ``linked_data`` context from `prefixmaps + `_ — the canonical curated + registry maintained by the LinkML team. This context provides + correct, community-consensus prefix names (e.g. ``sh`` not ``shacl``, + ``schema`` not ``sdo``). + + Secondary source: the ``merged`` context from prefixmaps, which + combines prefix.cc, bioregistry, and other sources for broad coverage. + + A small ``_PREFIX_OVERRIDES`` map corrects the few cases where the + merged context disagrees with rdflib/W3C canonical names. + + Both ``http`` and ``https`` variants of schema.org and wgs84 are + included because the linkml-runtime historically binds the HTTP form + while rdflib (and the W3C) prefer HTTPS. + + .. note:: + Requires ``prefixmaps >= 0.2.7``. For entries added in + linkml/prefixmaps#81 (W3C/OGC standard prefixes), pin to + ``prefixmaps @ git+https://github.com/linkml/prefixmaps@75435150`` + until v0.2.8 is released. + """ + return dict(_cached_well_known_prefix_map()) + + +@lru_cache(maxsize=1) +def _cached_well_known_prefix_map() -> dict[str, str]: + """Internal cached builder for well_known_prefix_map().""" + from prefixmaps import load_context + + # Layer 1: merged context (broad coverage, first-seen-wins for duplicates). + merged = load_context("merged") + ns_to_prefix: dict[str, str] = {} + for rec in merged.prefix_expansions: + if rec.namespace not in ns_to_prefix: + ns_to_prefix[rec.namespace] = rec.prefix + + # Layer 2: linked_data context (curated, correct names) overrides merged. + ld = load_context("linked_data") + for rec in ld.prefix_expansions: + ns_to_prefix[rec.namespace] = rec.prefix + + # Layer 3: overrides for the few cases where merged/linked_data disagrees + # with the rdflib/W3C canonical forms used by the RDF community. + for ns, pfx in _PREFIX_OVERRIDES.items(): + ns_to_prefix[ns] = pfx + + # Ensure both HTTP/HTTPS schema.org variants resolve to 'schema'. + ns_to_prefix.setdefault("https://schema.org/", "schema") + ns_to_prefix["http://schema.org/"] = "schema" + + # Ensure both HTTP/HTTPS wgs84 variants resolve to 'wgs'. + ns_to_prefix.setdefault("https://www.w3.org/2003/01/geo/wgs84_pos#", "wgs") + + return ns_to_prefix + + +# Overrides: corrections where prefixmaps merged context uses non-standard names +# that differ from rdflib 7.x / W3C canonical forms. +_PREFIX_OVERRIDES: types.MappingProxyType[str, str] = types.MappingProxyType( + { + # merged gives 'geosparql', rdflib/W3C uses 'geo' + "http://www.opengis.net/ont/geosparql#": "geo", + # merged gives 'sc', rdflib/W3C uses 'schema' + "https://schema.org/": "schema", + # merged gives 'WGS84', rdflib uses 'wgs' + "https://www.w3.org/2003/01/geo/wgs84_pos#": "wgs", + "http://www.w3.org/2003/01/geo/wgs84_pos#": "wgs", + } +) + + +def normalize_graph_prefixes(graph: "Graph", schema_prefixes: dict[str, str]) -> None: + """Normalise non-standard prefix aliases in an rdflib Graph. + + For each prefix bound in *schema_prefixes* (mapping prefix name → + namespace URI), check whether ``well_known_prefix_map()`` knows a + standard name for that URI. If the standard name differs from the + schema-declared name, rebind the namespace to the standard name. + + This is the **shared implementation** used by OWL, SHACL, and (via a + different code-path) JSON-LD context generators so that all serialisation + formats agree on prefix names when ``--normalize-prefixes`` is active. + + :param graph: rdflib Graph whose namespace bindings should be adjusted. + :param schema_prefixes: mapping of prefix name → namespace URI string, + typically from ``schema.prefixes``. + """ + from rdflib import Namespace + + wk = well_known_prefix_map() + + # Phase 1: normalise schema-declared prefixes. + for old_pfx, ns_uri in schema_prefixes.items(): + ns_str = str(ns_uri) + std_pfx = wk.get(ns_str) + if not std_pfx or std_pfx == old_pfx: + continue + # Collision: the user explicitly declared std_pfx for a different + # namespace — do not clobber their binding. + if std_pfx in schema_prefixes and schema_prefixes[std_pfx] != ns_str: + logger.warning( + "Prefix collision: cannot rename '%s' to '%s' because '%s' is already " + "declared for <%s>; skipping normalisation for <%s>", + old_pfx, + std_pfx, + std_pfx, + schema_prefixes[std_pfx], + ns_str, + ) + continue + # Rebind: remove old prefix, add standard prefix. + # ``replace=True`` forces the new prefix even if the prefix name + # is already bound to a different namespace. + graph.bind(std_pfx, Namespace(ns_str), override=True, replace=True) + + # Phase 2: normalise runtime-injected bindings (e.g. metamodel defaults). + # The linkml-runtime / rdflib may inject well-known namespaces under + # non-standard prefix names. After Phase 1 rebinds schema-declared + # prefixes, orphaned runtime bindings can appear as ``schema1``, ``dc0``, + # etc. Scan the graph's current bindings and fix any that map to a + # well-known namespace under a non-standard name, provided the standard + # name isn't already claimed by the user for a different namespace. + # + # Guard: if Phase 1 already bound std_pfx to a different URI (e.g. + # ``schema`` → ``https://schema.org/``), do not clobber it with the + # HTTP variant (``http://schema.org/``). Build a snapshot of the + # current bindings after Phase 1 to detect this. + current_bindings = {str(p): str(n) for p, n in graph.namespaces()} + for pfx, ns in list(graph.namespaces()): + pfx_str, ns_str = str(pfx), str(ns) + std_pfx = wk.get(ns_str) + if not std_pfx or std_pfx == pfx_str: + continue + # Same collision check as Phase 1: respect user-declared prefixes. + if std_pfx in schema_prefixes and schema_prefixes[std_pfx] != ns_str: + continue + # Guard: if std_pfx is already bound to a different (correct) URI + # by Phase 1, do not overwrite it. This prevents the HTTP variant + # of schema.org from clobbering the HTTPS binding. + if std_pfx in current_bindings and current_bindings[std_pfx] != ns_str: + continue + graph.bind(std_pfx, Namespace(ns_str), override=True, replace=True) + + @dataclass class Generator(metaclass=abc.ABCMeta): """ @@ -139,6 +569,9 @@ class Generator(metaclass=abc.ABCMeta): mergeimports: bool | None = True """True means merge non-linkml sources into importing package. False means separate packages""" + deterministic: bool = False + """True means produce stable, reproducible output with sorted keys and canonical blank-node ordering""" + source_file_date: str | None = None """Modification date of input source file""" @@ -180,6 +613,12 @@ class Generator(metaclass=abc.ABCMeta): stacktrace: bool = False """True means print stack trace, false just error message""" + normalize_prefixes: bool = False + """True means normalise non-standard prefix aliases to well-known names + from the ``prefixmaps`` package (linked_data + merged contexts, with + overrides for rdflib/W3C canonical forms). E.g. ``sdo`` → ``schema`` + for ``https://schema.org/``.""" + include: str | Path | SchemaDefinition | None = None """If set, include extra schema outside of the imports mechanism""" @@ -986,6 +1425,26 @@ def decorator(f: Command) -> Command: callback=stacktrace_callback, ) ) + f.params.append( + Option( + ("--deterministic/--no-deterministic",), + default=False, + show_default=True, + help="Generate stable, reproducible output with sorted keys and canonical blank-node ordering. " + "Supported by OWL, SHACL, JSON-LD, and JSON-LD Context generators. " + "Useful when generated artifacts are stored in version control.", + ) + ) + f.params.append( + Option( + ("--normalize-prefixes/--no-normalize-prefixes",), + default=False, + show_default=True, + help="Normalise non-standard prefix aliases to rdflib's curated default names " + "(e.g. sdo → schema for https://schema.org/). " + "Supported by OWL, SHACL, and JSON-LD Context generators.", + ) + ) return f diff --git a/packages/linkml/src/linkml/utils/rdf_canonicalize.py b/packages/linkml/src/linkml/utils/rdf_canonicalize.py new file mode 100644 index 0000000000..4b6f093b29 --- /dev/null +++ b/packages/linkml/src/linkml/utils/rdf_canonicalize.py @@ -0,0 +1,226 @@ +"""Deterministic RDF serialization via pyoxigraph RDFC-1.0 canonicalization. + +This module provides a function to canonicalize an rdflib Graph using +pyoxigraph's RDFC-1.0 implementation, producing deterministic output +with stable blank node labels and sorted triples. + +**Known limitations:** + +1. **xsd:string normalization**: pyoxigraph follows RDF 1.1, where plain + string literals and ``"text"^^xsd:string`` are identical. The output + will never contain explicit ``^^xsd:string`` annotations. Code that + re-parses the output with rdflib will see ``Literal("x")`` (datatype + ``None``) rather than ``Literal("x", datatype=XSD.string)``. + +2. **Non-standard RDF**: Graphs with literal predicates (e.g. SHACL + annotation mode) are rejected by pyoxigraph. This function falls + back to rdflib's serializer for such graphs. + +3. **Numeric short forms**: pyoxigraph uses Turtle short forms for + ``xsd:integer`` (``42``), ``xsd:boolean`` (``true``), and + ``xsd:decimal`` (``1.23``). rdflib parses these back with the + correct datatype, so this is lossless. + +4. **Base IRI / prefix collision**: When a graph has ``@base`` and a + prefix whose namespace equals the base IRI (e.g. rdflib's auto-bound + ``base:`` prefix), pyoxigraph emits CURIEs like ``base:label`` that + rdflib rejects. We skip such prefixes during serialization. + +5. **Trailing escaped dot in PN_LOCAL**: pyoxigraph emits CURIEs like + ``prefix:local\\.`` for IRIs whose local part ends with ``.``. This + is valid Turtle (PN_LOCAL_ESC), but rdflib's notation3 parser rejects + it because it conflicts with the statement-terminator dot. We + post-process the output to expand such CURIEs to full ```` form. +""" + +import io +import logging +import re + +import pyoxigraph as ox +import rdflib + +logger = logging.getLogger(__name__) + +# Mapping from rdflib/LinkML format strings to pyoxigraph RdfFormat objects. +_FORMAT_MAP: dict[str, ox.RdfFormat] = { + "turtle": ox.RdfFormat.TURTLE, + "ttl": ox.RdfFormat.TURTLE, + "nt": ox.RdfFormat.N_TRIPLES, + "ntriples": ox.RdfFormat.N_TRIPLES, + "n-triples": ox.RdfFormat.N_TRIPLES, + "nt11": ox.RdfFormat.N_TRIPLES, + "nquads": ox.RdfFormat.N_QUADS, + "n-quads": ox.RdfFormat.N_QUADS, + "xml": ox.RdfFormat.RDF_XML, + "rdf/xml": ox.RdfFormat.RDF_XML, + "trig": ox.RdfFormat.TRIG, + "n3": ox.RdfFormat.N3, +} + +# Formats that support prefix declarations. +_PREFIX_FORMATS = frozenset({ox.RdfFormat.TURTLE, ox.RdfFormat.TRIG, ox.RdfFormat.N3, ox.RdfFormat.RDF_XML}) + + +# Characters that may appear escaped in a Turtle PN_LOCAL via PN_LOCAL_ESC. +_PN_LOCAL_ESC_UNESCAPE = re.compile(r"\\([_~.\-!$&'()*+,;=/?#@%])") + + +def _expand_trailing_dot_curies(turtle_text: str, prefixes: dict[str, str]) -> str: + """Replace CURIEs whose local part ends in ``\\.`` with full ```` form. + + rdflib's notation3 parser rejects PN_LOCAL ending in an escaped dot + even though Turtle permits it (PN_LOCAL_ESC). pyoxigraph emits this + form for IRIs ending in ``.`` (e.g. ``biolink:StrandEnum#.``). We + rewrite each such CURIE to its expanded ```` form so the output + round-trips through rdflib. + """ + if not prefixes: + return turtle_text + + # Match: a prefix name, ':', a local part (no whitespace or token + # delimiters), ending in ``\.``, followed by whitespace. Use a + # negative lookbehind to avoid matching inside ``<...>`` or word + # characters that would make this a substring of something else. + pattern = re.compile( + r"(?\"'\[\]]*?\\\.)" + r"(?=\s)" + ) + + def replace(match: re.Match[str]) -> str: + prefix = match.group(1) + local_escaped = match.group(2) + namespace = prefixes.get(prefix) + if namespace is None: + return match.group(0) + local = _PN_LOCAL_ESC_UNESCAPE.sub(r"\1", local_escaped) + return f"<{namespace}{local}>" + + return pattern.sub(replace, turtle_text) + + +def _is_safe_prefix_iri(iri: str) -> bool: + """Check whether a namespace IRI is safe for prefix serialization. + + pyoxigraph rejects IRIs with invalid code-points (e.g. double ``#``), + and rdflib's Turtle parser cannot round-trip CURIEs whose namespace + contains query parameters or fragments in unexpected positions. This + function returns ``False`` for such IRIs so they can be skipped during + prefix collection. + """ + # A namespace IRI should end with '/' or '#'. If '#' appears + # *before* the final character, the IRI contains an embedded + # fragment which produces unusable CURIEs. + if "#" in iri[:-1]: + return False + # Query parameters in namespace IRIs produce CURIEs that rdflib + # cannot parse back. + if "?" in iri: + return False + return True + + +def canonicalize_rdf_graph( + graph: rdflib.Graph, + output_format: str = "turtle", +) -> str: + """Serialize an rdflib Graph deterministically using RDFC-1.0 canonicalization. + + The graph is transferred to pyoxigraph via N-Triples, canonicalized + with RDFC-1.0, sorted, and serialized back to the requested format. + Prefix bindings from the rdflib Graph are preserved in the output + for formats that support them (Turtle, TriG, N3, RDF/XML). + + Falls back to plain rdflib serialization for unsupported formats or + graphs containing non-standard RDF (e.g. literal predicates). + + :param graph: The rdflib Graph to serialize. + :param output_format: Target serialization format (e.g. ``"turtle"``, ``"nt"``). + :return: Deterministic string serialization of the graph. + """ + ox_format = _FORMAT_MAP.get(output_format.lower()) + if ox_format is None: + logger.warning( + "pyoxigraph does not support format %r; falling back to rdflib serializer", + output_format, + ) + # rdflib's Turtle serializer emits a trailing double newline; + # normalize to single newline for consistent file endings. + data = graph.serialize(format=output_format) + return data.rstrip("\n") + "\n" if data.endswith("\n") else data + + # 1. Transfer rdflib graph to pyoxigraph via N-Triples. + nt_data = graph.serialize(format="nt") + nt_bytes = nt_data.encode("utf-8") if isinstance(nt_data, str) else nt_data + + # 2. Parse into pyoxigraph and build a Dataset for canonicalization. + # Fall back to rdflib if the graph contains non-standard RDF + # (e.g. literal predicates from annotations) that pyoxigraph rejects. + try: + triples = list(ox.parse(io.BytesIO(nt_bytes), format=ox.RdfFormat.N_TRIPLES)) + except SyntaxError: + logger.warning( + "Graph contains non-standard RDF that pyoxigraph cannot parse; falling back to rdflib serializer" + ) + return graph.serialize(format=output_format) + + dataset = ox.Dataset() + for triple in triples: + dataset.add(ox.Quad(triple.subject, triple.predicate, triple.object, ox.DefaultGraph())) + + # 3. Canonicalize blank node labels with RDFC-1.0. + dataset.canonicalize(ox.CanonicalizationAlgorithm.RDFC_1_0) + + # 4. Sort triples for deterministic ordering. + quads = list(dataset) + sorted_triples = sorted( + (ox.Triple(q.subject, q.predicate, q.object) for q in quads), + key=lambda t: (str(t.subject), str(t.predicate), str(t.object)), + ) + + # 5. Collect prefixes for formats that support them. + base_iri = str(graph.base) if graph.base else None + prefixes: dict[str, str] | None = None + if ox_format in _PREFIX_FORMATS: + prefixes = {} + for prefix, namespace in graph.namespace_manager.namespaces(): + if not prefix: # skip empty prefix (base) + continue + ns_str = str(namespace) + # Skip prefixes whose namespace matches the base IRI to avoid + # pyoxigraph emitting CURIEs like `base:label` that conflict + # with the @base directive. + if base_iri and ns_str == base_iri: + continue + # Skip namespace IRIs that pyoxigraph rejects or that produce + # CURIEs rdflib cannot round-trip. Valid namespace IRIs for + # prefix use should end with '/' or '#' and contain no query + # parameters or fragment-like characters in the middle. + if not _is_safe_prefix_iri(ns_str): + continue + prefixes[str(prefix)] = ns_str + used_prefixes = prefixes + try: + result_bytes = ox.serialize( + sorted_triples, + format=ox_format, + prefixes=prefixes, + base_iri=base_iri, + ) + except ValueError: + # pyoxigraph rejects prefixes with invalid IRIs (e.g. containing + # fragment-like characters such as double '#'). Retry without + # the offending prefixes by falling back to no prefixes, which + # still produces valid (if verbose) Turtle. + logger.warning("pyoxigraph rejected one or more prefix IRIs; serializing without prefix declarations") + result_bytes = ox.serialize( + sorted_triples, + format=ox_format, + ) + used_prefixes = None + result = result_bytes.decode("utf-8") + if ox_format in _PREFIX_FORMATS and used_prefixes: + result = _expand_trailing_dot_curies(result, used_prefixes) + return result diff --git a/tests/linkml/test_generators/input/shaclgen/any_of_pattern.yaml b/tests/linkml/test_generators/input/shaclgen/any_of_pattern.yaml new file mode 100644 index 0000000000..5b247bb2a1 --- /dev/null +++ b/tests/linkml/test_generators/input/shaclgen/any_of_pattern.yaml @@ -0,0 +1,59 @@ +id: https://w3id.org/linkml/examples/any_of_pattern +name: test_any_of_pattern +description: >- + Test schema for pattern constraints inside any_of branches. + Exercises three cases: (1) pattern-only branch (no range), + (2) range + pattern on the same branch, (3) mixed branches + where some have pattern and some do not. +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://w3id.org/linkml/examples/any_of_pattern/ +imports: + - linkml:types +default_range: string +default_prefix: ex + +enums: + LicenseEnum: + permissible_values: + MIT: + Apache-2.0: + GPL-3.0-only: + +classes: + PatternOnlyBranch: + description: >- + A class where one any_of branch specifies only a pattern + (no range). The generated SHACL sh:or should contain a + node with sh:pattern but no sh:datatype or sh:class. + attributes: + license: + any_of: + - range: LicenseEnum + - range: uri + - pattern: "^LicenseRef-[a-zA-Z0-9\\-\\.]+$" + + RangeWithPattern: + description: >- + A class where an any_of branch combines range + pattern. + The generated SHACL sh:or node should have both sh:datatype + and sh:pattern. + attributes: + identifier: + any_of: + - range: string + pattern: "^[A-Z]{2}-[0-9]{4}$" + - range: integer + + MixedBranches: + description: >- + A class with three any_of branches: one with range only, + one with pattern only, one with range + pattern. Ensures + pattern is emitted only on branches that declare it. + attributes: + code: + any_of: + - range: integer + - pattern: "^CUSTOM-.*$" + - range: string + pattern: "^STD-[0-9]+$" diff --git a/tests/linkml/test_generators/input/shaclgen/boolean_guard_rules.yaml b/tests/linkml/test_generators/input/shaclgen/boolean_guard_rules.yaml new file mode 100644 index 0000000000..f56c2eca6a --- /dev/null +++ b/tests/linkml/test_generators/input/shaclgen/boolean_guard_rules.yaml @@ -0,0 +1,70 @@ +id: https://example.org/boolean-guards +name: boolean_guard_rules +description: >- + Test schema for SHACL generation of sh:sparql constraints from LinkML rules. + Models the boolean-guard pattern where a boolean flag must be true if a + corresponding value property is present. + +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/boolean-guards/ + +imports: + - linkml:types + +default_prefix: ex +default_range: string + +slots: + WeatherWind: + description: Whether wind conditions are present. + range: boolean + slot_uri: ex:WeatherWind + weatherWindValue: + description: Wind speed value. + range: decimal + slot_uri: ex:weatherWindValue + WeatherRain: + description: Whether rain conditions are present. + range: boolean + slot_uri: ex:WeatherRain + weatherRainValue: + description: Rain intensity value. + range: decimal + slot_uri: ex:weatherRainValue + Temperature: + description: Ambient temperature. + range: decimal + slot_uri: ex:Temperature + +classes: + Environment: + description: Environmental conditions. + class_uri: ex:Environment + slots: + - WeatherWind + - weatherWindValue + - WeatherRain + - weatherRainValue + - Temperature + rules: + - description: >- + If weatherWindValue is provided, WeatherWind must be true. + preconditions: + slot_conditions: + weatherWindValue: + value_presence: PRESENT + postconditions: + slot_conditions: + WeatherWind: + equals_string: "true" + - description: >- + If weatherRainValue is provided, WeatherRain must be true. + preconditions: + slot_conditions: + weatherRainValue: + value_presence: PRESENT + postconditions: + slot_conditions: + WeatherRain: + equals_string: "true" diff --git a/tests/linkml/test_generators/input/shaclgen/cardinality.yaml b/tests/linkml/test_generators/input/shaclgen/cardinality.yaml index 6bacffa680..d12b06df4c 100644 --- a/tests/linkml/test_generators/input/shaclgen/cardinality.yaml +++ b/tests/linkml/test_generators/input/shaclgen/cardinality.yaml @@ -17,6 +17,36 @@ classes: slots: - list_exact_size + ParentClass: + slots: + - inherited_slot + - restricted_slot + + ChildWithZeroMaxCard: + is_a: ParentClass + slot_usage: + restricted_slot: + maximum_cardinality: 0 + + ChildWithZeroExactCard: + is_a: ParentClass + slot_usage: + restricted_slot: + exact_cardinality: 0 + + ChildWithZeroMinCard: + is_a: ParentClass + slot_usage: + restricted_slot: + minimum_cardinality: 0 + + ChildWithRequiredAndZeroMinCard: + is_a: ParentClass + slot_usage: + restricted_slot: + required: true + minimum_cardinality: 0 + slots: list_min_max_size: range: integer @@ -28,3 +58,11 @@ slots: range: integer multivalued: true exact_cardinality: 3 + + inherited_slot: + range: string + multivalued: true + + restricted_slot: + range: string + multivalued: true diff --git a/tests/linkml/test_generators/test_deterministic_benchmark.py b/tests/linkml/test_generators/test_deterministic_benchmark.py new file mode 100644 index 0000000000..b7488a8dda --- /dev/null +++ b/tests/linkml/test_generators/test_deterministic_benchmark.py @@ -0,0 +1,356 @@ +"""Benchmark: deterministic Turtle serializer on real-world ontologies. + +Evaluates the ``--deterministic`` flag against schema.org (~16 000 triples, +~800 classes, ~1 400 properties) and the kitchen_sink LinkML schema to +demonstrate four properties: + +1. **Semantic equivalence** — ``rdflib.compare.isomorphic()`` confirms that + deterministic and non-deterministic outputs encode the same RDF graph. +2. **Byte-level stability** — SHA-256 identity across repeated runs proves + that deterministic output is truly reproducible. +3. **Diff quality** — controlled mutations show that small schema changes + produce small, focused diffs (high signal-to-noise ratio). +4. **Performance** — generation time stays within acceptable bounds even + on large real-world graphs. + +Schema.org tests exercise ``deterministic_turtle()`` directly on a +pre-existing OWL ontology. Kitchen_sink tests exercise the full +``OwlSchemaGenerator`` / ``ShaclGenerator`` pipeline with LinkML schemas. + +References +---------- +- W3C RDFC-1.0: https://www.w3.org/TR/rdf-canon/ +- W3C Turtle 1.1: https://www.w3.org/TR/turtle/ +- schema.org: https://schema.org/docs/developers.html +""" + +import difflib +import hashlib +import time +from pathlib import Path + +import pytest +import yaml +from rdflib import Graph +from rdflib.compare import isomorphic + +from linkml.generators.owlgen import OwlSchemaGenerator +from linkml.generators.shaclgen import ShaclGenerator +from linkml.utils.generator import deterministic_turtle + +_has_pyoxigraph = False +try: + import pyoxigraph + + _has_pyoxigraph = hasattr(pyoxigraph, "Dataset") +except ImportError: + pass + +pytestmark = pytest.mark.skipif( + not _has_pyoxigraph, + reason="pyoxigraph >= 0.4.0 required for deterministic benchmarks", +) + +KITCHEN_SINK = str(Path(__file__).parent / "input" / "kitchen_sink.yaml") +SCHEMA_ORG_URL = "https://schema.org/version/latest/schemaorg-current-https.ttl" + + +def _sha256(text: str) -> str: + return hashlib.sha256(text.encode()).hexdigest() + + +def _diff_line_count(a: str, b: str) -> int: + """Count lines present in *b* but not in *a* (unified-diff additions).""" + al = a.strip().splitlines() + bl = b.strip().splitlines() + return sum( + 1 for line in difflib.unified_diff(al, bl, lineterm="") if line.startswith("+") and not line.startswith("+++") + ) + + +# ── Schema.org: direct serializer benchmark ──────────────────────── + + +@pytest.fixture(scope="module") +def schema_org_graph(): + """Download and parse schema.org as an rdflib Graph. + + Cached for the module so the network fetch only happens once. + Skips all dependent tests if the download fails. + """ + try: + import urllib.request + + with urllib.request.urlopen(SCHEMA_ORG_URL, timeout=60) as resp: + data = resp.read().decode("utf-8") + except Exception as exc: + pytest.skip(f"Could not fetch schema.org: {exc}") + + g = Graph() + g.parse(data=data, format="turtle") + return g + + +@pytest.mark.network +class TestSchemaOrgDeterministicSerializer: + """Benchmark ``deterministic_turtle()`` on schema.org OWL ontology.""" + + def test_semantic_equivalence(self, schema_org_graph): + """Deterministic serialization must be isomorphic to the original graph.""" + det_ttl = deterministic_turtle(schema_org_graph) + + g_det = Graph() + g_det.parse(data=det_ttl, format="turtle") + + assert len(g_det) == len(schema_org_graph), ( + f"Triple count mismatch: original={len(schema_org_graph)}, deterministic={len(g_det)}" + ) + assert isomorphic(g_det, schema_org_graph), ( + "Deterministic output is NOT isomorphic to original schema.org graph" + ) + + def test_byte_stability(self, schema_org_graph): + """Two deterministic runs must produce byte-identical output.""" + run1 = deterministic_turtle(schema_org_graph) + run2 = deterministic_turtle(schema_org_graph) + assert _sha256(run1) == _sha256(run2), "Deterministic serializer produced different output across runs" + + def test_prefix_filtering(self, schema_org_graph): + """Only prefixes actually used in the graph should be declared.""" + det_ttl = deterministic_turtle(schema_org_graph) + + # Extract declared prefixes + declared = {} + for line in det_ttl.splitlines(): + if line.startswith("@prefix"): + parts = line.split() + pfx = parts[1].rstrip(":") + ns = parts[2].strip("<>") + declared[pfx] = ns + + # Collect all IRIs in the graph + from rdflib import URIRef + + used_iris = set() + for s, p, o in schema_org_graph: + for term in (s, p, o): + if isinstance(term, URIRef): + used_iris.add(str(term)) + + # Every declared prefix must have at least one IRI using it + for pfx, ns in declared.items(): + assert any(iri.startswith(ns) for iri in used_iris), f"Prefix '{pfx}:' <{ns}> declared but no IRI uses it" + + def test_performance(self, schema_org_graph): + """Serialization must complete within 60 seconds for ~16K triples.""" + start = time.time() + det_ttl = deterministic_turtle(schema_org_graph) + elapsed = time.time() - start + triple_count = len(schema_org_graph) + throughput = triple_count / elapsed if elapsed > 0 else float("inf") + + # Log for benchmark visibility (shows with pytest -v) + print(f"\n schema.org: {triple_count} triples in {elapsed:.1f}s ({throughput:.0f} triples/s)") + + assert elapsed < 60.0, f"Serialization took {elapsed:.1f}s (limit: 60s) for {triple_count} triples" + assert len(det_ttl) > 1000, "Output suspiciously short" + + +# ── Kitchen_sink: full pipeline benchmark ─────────────────────────── + + +def _mutate_kitchen_sink(description_suffix: str = "", add_slot: bool = False) -> str: + """Create a mutated copy of kitchen_sink.yaml **in the same directory** and return its path. + + The copy must live alongside the original so that LinkML relative imports + (``linkml:types``, ``core``, etc.) resolve correctly. + + Uses a unique filename (via ``os.getpid()``) to avoid race conditions + when tests run in parallel under pytest-xdist. + + Parameters + ---------- + description_suffix + Text appended to the first class description found. + add_slot + If True, adds a synthetic ``benchmark_notes`` slot to the first class. + """ + import os + + ks_path = Path(KITCHEN_SINK) + schema = yaml.safe_load(ks_path.read_text()) + + if description_suffix or add_slot: + # Find the first class with a description + for cls_name, cls_def in schema.get("classes", {}).items(): + if isinstance(cls_def, dict) and cls_def.get("description"): + if description_suffix: + cls_def["description"] += description_suffix + if add_slot: + slots = cls_def.get("slots", []) + slots.append("benchmark_notes") + cls_def["slots"] = slots + break + + # Define the synthetic slot if adding one + if add_slot: + slots_dict = schema.setdefault("slots", {}) + slots_dict["benchmark_notes"] = { + "description": "Synthetic benchmark slot for diff quality testing.", + "range": "string", + } + + # Write in the same directory so relative imports resolve. + # Use PID to avoid race conditions with pytest-xdist workers. + out_path = ks_path.parent / f"_benchmark_mutated_{os.getpid()}_kitchen_sink.yaml" + out_path.write_text( + yaml.dump(schema, default_flow_style=False, allow_unicode=True), + encoding="utf-8", + ) + return str(out_path) + + +@pytest.mark.parametrize( + "generator_cls", + [OwlSchemaGenerator, ShaclGenerator], + ids=["owl", "shacl"], +) +class TestKitchenSinkDiffQuality: + """Measure diff quality on the kitchen_sink schema with controlled mutations.""" + + def test_mutation_description_change(self, generator_cls): + """A single description change must produce a small, focused diff. + + Deterministic mode should change only the affected line(s) and their + immediate context (e.g. SHACL may repeat descriptions in sh:description). + Non-deterministic mode produces a much larger diff due to blank-node + and property-ordering instability. + """ + base = generator_cls(KITCHEN_SINK, deterministic=True).serialize() + mutated_path = _mutate_kitchen_sink(description_suffix=" (benchmark edit)") + try: + mutated = generator_cls(mutated_path, deterministic=True).serialize() + finally: + Path(mutated_path).unlink(missing_ok=True) + + det_diff = _diff_line_count(base, mutated) + + # Non-deterministic baseline for comparison + non_base = generator_cls(KITCHEN_SINK, deterministic=False).serialize() + non_mutated_path = _mutate_kitchen_sink(description_suffix=" (benchmark edit)") + try: + non_mutated = generator_cls(non_mutated_path, deterministic=False).serialize() + finally: + Path(non_mutated_path).unlink(missing_ok=True) + + non_diff = _diff_line_count(non_base, non_mutated) + + # The deterministic diff must be small (description + any SHACL mirrors) + assert det_diff <= 20, ( + f"Deterministic diff too large for a 1-description change: {det_diff} lines (expected ≤20)" + ) + # Signal-to-noise: deterministic must be at least 5× smaller + if non_diff > 0: + ratio = non_diff / max(det_diff, 1) + assert ratio >= 5, ( + f"Insufficient noise reduction: det={det_diff}, non-det={non_diff}, ratio={ratio:.1f}× (expected ≥5×)" + ) + + print( + f"\n {generator_cls.__name__} description mutation: " + f"det={det_diff} lines, non-det={non_diff} lines, " + f"noise reduction={non_diff / max(det_diff, 1):.0f}×" + ) + + def test_mutation_add_slot(self, generator_cls): + """Adding a new slot must produce a proportionally small diff. + + A new slot adds ~10-20 triples (label, range, domain, restrictions). + The diff should be roughly proportional to the new content, not a + full-file rewrite. + """ + base = generator_cls(KITCHEN_SINK, deterministic=True).serialize() + mutated_path = _mutate_kitchen_sink(add_slot=True) + try: + mutated = generator_cls(mutated_path, deterministic=True).serialize() + finally: + Path(mutated_path).unlink(missing_ok=True) + + det_diff = _diff_line_count(base, mutated) + + # Non-deterministic baseline for comparison + non_base = generator_cls(KITCHEN_SINK, deterministic=False).serialize() + non_mutated_path = _mutate_kitchen_sink(add_slot=True) + try: + non_mutated = generator_cls(non_mutated_path, deterministic=False).serialize() + finally: + Path(non_mutated_path).unlink(missing_ok=True) + + non_diff = _diff_line_count(non_base, non_mutated) + + g_base = Graph() + g_base.parse(data=base, format="turtle") + g_mut = Graph() + g_mut.parse(data=mutated, format="turtle") + new_triples = len(g_mut) - len(g_base) + + # Diff should be proportional to new triples (allow 5× margin) + assert det_diff <= max(new_triples * 5, 40), ( + f"Deterministic diff ({det_diff} lines) disproportionate to new triples ({new_triples})" + ) + # Signal-to-noise: deterministic must be at least 5× smaller + if non_diff > 0: + ratio = non_diff / max(det_diff, 1) + assert ratio >= 5, ( + f"Insufficient noise reduction: det={det_diff}, non-det={non_diff}, ratio={ratio:.1f}× (expected ≥5×)" + ) + + print( + f"\n {generator_cls.__name__} add-slot mutation: " + f"det_diff={det_diff} lines, non-det={non_diff} lines, " + f"new_triples={new_triples}, noise reduction={non_diff / max(det_diff, 1):.0f}×" + ) + + print(f"\n {generator_cls.__name__} add-slot mutation: det_diff={det_diff} lines, new_triples={new_triples}") + + +@pytest.mark.parametrize( + "generator_cls", + [OwlSchemaGenerator, ShaclGenerator], + ids=["owl", "shacl"], +) +class TestKitchenSinkEquivalence: + """Verify semantic equivalence between deterministic and non-deterministic modes.""" + + def test_triple_count_matches(self, generator_cls): + """Both modes must produce the same number of triples.""" + det = generator_cls(KITCHEN_SINK, deterministic=True).serialize() + nondet = generator_cls(KITCHEN_SINK, deterministic=False).serialize() + + g_det = Graph() + g_det.parse(data=det, format="turtle") + g_nondet = Graph() + g_nondet.parse(data=nondet, format="turtle") + + assert len(g_det) == len(g_nondet), ( + f"Triple count mismatch: deterministic={len(g_det)}, non-deterministic={len(g_nondet)}" + ) + + def test_byte_stability_across_runs(self, generator_cls): + """Three deterministic runs must produce identical output.""" + runs = [generator_cls(KITCHEN_SINK, deterministic=True).serialize() for _ in range(3)] + hashes = [_sha256(r) for r in runs] + assert hashes[0] == hashes[1] == hashes[2], f"Deterministic output varies across runs: {hashes}" + + def test_non_deterministic_instability(self, generator_cls): + """Non-deterministic output should vary across runs (documents the problem). + + This test is advisory — it passes regardless but logs the instability. + """ + runs = [generator_cls(KITCHEN_SINK, deterministic=False).serialize() for _ in range(3)] + hashes = [_sha256(r) for r in runs] + identical = hashes[0] == hashes[1] == hashes[2] + print( + f"\n {generator_cls.__name__} non-det stable: {identical} " + f"(expected: False for Turtle due to bnode/ordering instability)" + ) diff --git a/tests/linkml/test_generators/test_deterministic_output.py b/tests/linkml/test_generators/test_deterministic_output.py new file mode 100644 index 0000000000..6721c2ac93 --- /dev/null +++ b/tests/linkml/test_generators/test_deterministic_output.py @@ -0,0 +1,481 @@ +"""Tests for deterministic generator output. + +When ``deterministic=True``, generators must produce byte-identical output +across multiple invocations. This ensures version-controlled artifacts don't +show spurious diffs from blank-node relabeling or dict-ordering instability. + +Generators must also produce **isomorphic** output — the deterministic +serialization must encode the same RDF graph as non-deterministic mode. +""" + +import json +import time +from pathlib import Path + +import pytest +from rdflib import Graph +from rdflib.compare import isomorphic + +from linkml.generators.jsonldcontextgen import ContextGenerator +from linkml.generators.jsonldgen import JSONLDGenerator +from linkml.generators.owlgen import OwlSchemaGenerator +from linkml.generators.shaclgen import ShaclGenerator + +# Deterministic Turtle requires pyoxigraph >= 0.4.0 (for Dataset/canonicalize). +# When an older version is present (e.g. pulled in by morph-kgc), skip these tests. +_has_pyoxigraph = False +try: + import pyoxigraph + + _has_pyoxigraph = hasattr(pyoxigraph, "Dataset") +except ImportError: + pass + +pytestmark = pytest.mark.skipif(not _has_pyoxigraph, reason="pyoxigraph >= 0.4.0 required for deterministic tests") + +SCHEMA = str(Path(__file__).parent / "input" / "personinfo.yaml") + + +@pytest.mark.parametrize( + "generator_cls,kwargs", + [ + (OwlSchemaGenerator, {}), + (ShaclGenerator, {}), + (ContextGenerator, {}), + (JSONLDGenerator, {}), + ], + ids=["owl", "shacl", "context", "jsonld"], +) +def test_deterministic_output_is_identical_across_runs(generator_cls, kwargs): + """Generate output twice with deterministic=True and verify identity.""" + out1 = generator_cls(SCHEMA, deterministic=True, **kwargs).serialize() + out2 = generator_cls(SCHEMA, deterministic=True, **kwargs).serialize() + # JSONLDGenerator embeds a generation_date timestamp — normalize it + if generator_cls is JSONLDGenerator: + import re + + ts_re = re.compile(r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}") + out1 = ts_re.sub("TIMESTAMP", out1) + out2 = ts_re.sub("TIMESTAMP", out2) + assert out1 == out2, f"{generator_cls.__name__} produced different output across runs" + assert len(out1) > 100, "Output suspiciously short — generator may have failed silently" + + +@pytest.mark.parametrize( + "generator_cls", + [ContextGenerator, JSONLDGenerator], + ids=["context", "jsonld"], +) +def test_deterministic_json_has_sorted_keys(generator_cls): + """When deterministic=True, JSON dict keys should be sorted at all levels. + + For the ContextGenerator, @context keys use grouped ordering (prefixes + before term entries) — each group is sorted, but not globally. + """ + out = generator_cls(SCHEMA, deterministic=True).serialize() + parsed = json.loads(out) + + is_context_gen = generator_cls is ContextGenerator + + def _check_sorted_keys(obj, path="root"): + if isinstance(obj, dict): + keys = list(obj.keys()) + # Context generator groups @context keys: @-directives, prefixes, terms + if is_context_gen and path == "root.@context": + at_keys = [k for k in keys if k.startswith("@")] + prefix_keys = [k for k in keys if not k.startswith("@") and isinstance(obj[k], str)] + term_keys = [k for k in keys if not k.startswith("@") and not isinstance(obj[k], str)] + assert at_keys == sorted(at_keys), f"@-keys not sorted: {at_keys}" + assert prefix_keys == sorted(prefix_keys), f"Prefix keys not sorted: {prefix_keys}" + assert term_keys == sorted(term_keys), f"Term keys not sorted: {term_keys}" + else: + assert keys == sorted(keys), f"Keys not sorted at {path}: {keys}" + for k, v in obj.items(): + _check_sorted_keys(v, f"{path}.{k}") + elif isinstance(obj, list): + for i, item in enumerate(obj): + _check_sorted_keys(item, f"{path}[{i}]") + + _check_sorted_keys(parsed) + + +@pytest.mark.parametrize( + "generator_cls", + [ContextGenerator, JSONLDGenerator], + ids=["context", "jsonld"], +) +def test_deterministic_json_lists_are_sorted(generator_cls): + """When deterministic=True, JSON list elements should be sorted. + + Lists under JSON-LD structural keys (``@context``, ``@list``, ``imports``, + etc.) are exempt because their ordering carries semantic meaning. + """ + out = generator_cls(SCHEMA, deterministic=True).serialize() + parsed = json.loads(out) + + # JSON-LD keys whose array values carry ordering semantics. + _ORDERED_KEYS = {"@context", "@list", "@graph", "@set", "imports"} + + def _check_sorted_lists(obj, path="root", parent_key=""): + if isinstance(obj, dict): + for k, v in obj.items(): + _check_sorted_lists(v, f"{path}.{k}", parent_key=k) + elif isinstance(obj, list): + if parent_key not in _ORDERED_KEYS: + str_items = [json.dumps(item, sort_keys=True, ensure_ascii=False) for item in obj] + assert str_items == sorted(str_items), f"List not sorted at {path}" + for i, item in enumerate(obj): + _check_sorted_lists(item, f"{path}[{i}]") + + _check_sorted_lists(parsed) + + +@pytest.mark.parametrize( + "generator_cls", + [OwlSchemaGenerator, ShaclGenerator], + ids=["owl", "shacl"], +) +def test_deterministic_turtle_preserves_at_prefix(generator_cls): + """deterministic_turtle must produce standard @prefix, not SPARQL PREFIX.""" + out = generator_cls(SCHEMA, deterministic=True).serialize() + assert "@prefix" in out, "Output uses non-standard prefix syntax" + assert "PREFIX " not in out, "Output uses SPARQL PREFIX instead of Turtle @prefix" + + +def test_deterministic_turtle_performance(): + """Deterministic OWL generation must complete within 10 seconds for personinfo. + + The Weisfeiler-Lehman approach is O(n log n), so this should easily pass. + The previous canon=True approach was exponential and failed this test + for graphs above ~250 triples. + """ + start = time.time() + out = OwlSchemaGenerator(SCHEMA, deterministic=True).serialize() + elapsed = time.time() - start + assert elapsed < 10.0, f"Deterministic generation took {elapsed:.1f}s (limit: 10s)" + assert len(out) > 100, "Output suspiciously short" + + +def test_shacl_closed_ignored_properties_deterministic(): + """sh:ignoredProperties in closed shapes must be deterministic. + + ``_build_ignored_properties`` collects inherited slots into a set; without + explicit sorting this produces different ``rdf:first``/``rdf:rest`` chains + on each run. With ``deterministic=True`` (and sorted Collection inputs) + the output must be byte-identical. + """ + runs = [ShaclGenerator(SCHEMA, deterministic=True, closed=True).serialize() for _ in range(3)] + assert runs[0] == runs[1] == runs[2], "sh:ignoredProperties ordering differs across runs" + assert "sh:ignoredProperties" in runs[0], "Expected closed shapes with sh:ignoredProperties" + + +def test_shacl_enum_in_deterministic(): + """sh:in RDF lists for enums must be deterministic. + + ``_build_enum_constraint`` iterates ``enum.permissible_values.items()`` + (dict iteration order) into a ``Collection``. Without sorting, the + ``rdf:first``/``rdf:rest`` chain varies across runs. + """ + runs = [ShaclGenerator(SCHEMA, deterministic=True).serialize() for _ in range(3)] + assert runs[0] == runs[1] == runs[2], "sh:in enum list ordering differs across runs" + assert "sh:in" in runs[0], "Expected sh:in constraints for enums" + + +def test_owl_enum_one_of_deterministic(): + """owl:oneOf RDF lists for enums must be deterministic. + + ``_boolean_expression`` feeds ``pv_uris`` (from ``permissible_values``) + into a ``Collection``. Without sorting, ``owl:oneOf`` list ordering varies. + """ + runs = [OwlSchemaGenerator(SCHEMA, deterministic=True).serialize() for _ in range(3)] + assert runs[0] == runs[1] == runs[2], "owl:oneOf enum list ordering differs across runs" + + +KITCHEN_SINK = str(Path(__file__).parent / "input" / "kitchen_sink.yaml") + + +def test_deterministic_large_schema(): + """End-to-end idempotency on a complex schema (kitchen_sink). + + Exercises many code paths simultaneously: closed shapes, enums, imports, + class hierarchies, and mixed ranges. + """ + owl1 = OwlSchemaGenerator(KITCHEN_SINK, deterministic=True).serialize() + owl2 = OwlSchemaGenerator(KITCHEN_SINK, deterministic=True).serialize() + assert owl1 == owl2, "OWL output differs across runs for kitchen_sink" + assert len(owl1) > 500, "kitchen_sink output suspiciously short" + + shacl1 = ShaclGenerator(KITCHEN_SINK, deterministic=True).serialize() + shacl2 = ShaclGenerator(KITCHEN_SINK, deterministic=True).serialize() + assert shacl1 == shacl2, "SHACL output differs across runs for kitchen_sink" + assert len(shacl1) > 500, "kitchen_sink output suspiciously short" + + +def test_deterministic_context_preserves_jsonld_structure(): + """Deterministic JSON-LD context must preserve conventional structure. + + JSON-LD contexts have a conventional layout: + 1. ``comments`` block first (metadata) + 2. ``@context`` block second, with prefixes grouped before term entries + + ``deterministic_json()`` would scramble this by sorting all keys + uniformly. The context generator must use JSON-LD-aware ordering. + """ + out = ContextGenerator(SCHEMA, deterministic=True, metadata=True).serialize() + parsed = json.loads(out) + + # Top-level key order: "comments" before "@context" + top_keys = list(parsed.keys()) + assert "comments" in top_keys, "Expected 'comments' block with metadata=True" + assert top_keys.index("comments") < top_keys.index("@context"), ( + f"'comments' should precede '@context', got: {top_keys}" + ) + + # Inside @context: @-directives, then prefixes (str values), then terms (dict values) + ctx = parsed["@context"] + ctx_keys = list(ctx.keys()) + + at_keys = [k for k in ctx_keys if k.startswith("@")] + prefix_keys = [k for k in ctx_keys if not k.startswith("@") and isinstance(ctx[k], str)] + term_keys = [k for k in ctx_keys if not k.startswith("@") and not isinstance(ctx[k], str)] + + # Verify grouping: all @-keys before all prefix keys before all term keys + last_at = max(ctx_keys.index(k) for k in at_keys) if at_keys else -1 + first_prefix = min(ctx_keys.index(k) for k in prefix_keys) if prefix_keys else len(ctx_keys) + last_prefix = max(ctx_keys.index(k) for k in prefix_keys) if prefix_keys else -1 + first_term = min(ctx_keys.index(k) for k in term_keys) if term_keys else len(ctx_keys) + + assert last_at < first_prefix, "@-directives must come before prefixes" + assert last_prefix < first_term, "Prefixes must come before term entries" + + # Verify each group is sorted internally + assert at_keys == sorted(at_keys), f"@-directives not sorted: {at_keys}" + assert prefix_keys == sorted(prefix_keys), f"Prefixes not sorted: {prefix_keys}" + assert term_keys == sorted(term_keys), f"Term entries not sorted: {term_keys}" + + +def test_non_deterministic_is_default(): + """Verify that ``deterministic`` defaults to False.""" + gen = OwlSchemaGenerator(SCHEMA) + assert gen.deterministic is False + + +def test_wl_handles_structurally_similar_bnodes(): + """Blank nodes with identical local structure but different named neighbours + must receive different WL signatures and thus different stable labels. + + This tests the core WL property: two BNodes that differ only in their + connected named nodes (URIs/literals) must be distinguishable. + """ + from rdflib import BNode, Graph, Namespace, URIRef + + from linkml.utils.generator import deterministic_turtle + + RDF_TYPE = URIRef("http://www.w3.org/1999/02/22-rdf-syntax-ns#type") + OWL_RESTRICTION = URIRef("http://www.w3.org/2002/07/owl#Restriction") + OWL_ON_PROP = URIRef("http://www.w3.org/2002/07/owl#onProperty") + OWL_ALL_VALUES = URIRef("http://www.w3.org/2002/07/owl#allValuesFrom") + + EX = Namespace("http://example.org/") + g = Graph() + + # Two restrictions with same structure but different property URIs + r1 = BNode() + g.add((r1, RDF_TYPE, OWL_RESTRICTION)) + g.add((r1, OWL_ON_PROP, EX.alpha)) + g.add((r1, OWL_ALL_VALUES, EX.Target1)) + + r2 = BNode() + g.add((r2, RDF_TYPE, OWL_RESTRICTION)) + g.add((r2, OWL_ON_PROP, EX.beta)) + g.add((r2, OWL_ALL_VALUES, EX.Target2)) + + RDFS_SUBCLASS = URIRef("http://www.w3.org/2000/01/rdf-schema#subClassOf") + g.add((EX.MyClass, RDFS_SUBCLASS, r1)) + g.add((EX.MyClass, RDFS_SUBCLASS, r2)) + + # Must be deterministic across runs + out1 = deterministic_turtle(g) + out2 = deterministic_turtle(g) + assert out1 == out2, "WL-based serializer is not deterministic for similar BNodes" + + # Both restrictions must appear (not collapsed) + assert "alpha" in out1 + assert "beta" in out1 + + +def test_deterministic_turtle_no_bnodes(): + """Graphs with no blank nodes should still produce sorted, deterministic output.""" + from rdflib import Graph, Literal, Namespace + from rdflib.namespace import RDFS + + from linkml.utils.generator import deterministic_turtle + + EX = Namespace("http://example.org/") + g = Graph() + g.add((EX.B, RDFS.label, Literal("B"))) + g.add((EX.A, RDFS.label, Literal("A"))) + + out1 = deterministic_turtle(g) + out2 = deterministic_turtle(g) + assert out1 == out2 + + # A should appear before B (sorted) + a_pos = out1.find("example.org/A") + b_pos = out1.find("example.org/B") + assert a_pos < b_pos, "Triples should be sorted: A before B" + + +@pytest.mark.xfail( + reason=( + "Collection sorting (owl:oneOf, sh:in) in deterministic mode intentionally " + "reorders RDF list triples for canonical output. The resulting graph is " + "semantically equivalent (OWL/SHACL interpret these as unordered sets) but " + "not RDF-isomorphic because rdf:first/rdf:rest chains encode ordering." + ), + strict=True, +) +@pytest.mark.parametrize( + "generator_cls", + [OwlSchemaGenerator, ShaclGenerator], + ids=["owl", "shacl"], +) +def test_deterministic_turtle_is_isomorphic(generator_cls): + """Deterministic output is NOT RDF-isomorphic to non-deterministic output. + + This documents the trade-off identified in linkml/linkml#3295 review: + deterministic mode sorts Collection inputs (owl:oneOf, sh:in, + sh:ignoredProperties) to produce canonical RDF list ordering. Since RDF + Collections encode order via rdf:first/rdf:rest triples, the sorted graph + is structurally different from the insertion-order graph — even though the + OWL/SHACL semantics are identical (these Collections represent sets). + + The test is marked xfail(strict=True) so that it: + - Documents the known, intentional non-isomorphism + - Alerts maintainers if the behaviour changes (strict xfail fails on pass) + """ + out_det = generator_cls(SCHEMA, deterministic=True).serialize() + out_nondet = generator_cls(SCHEMA, deterministic=False).serialize() + + g_det = Graph() + g_det.parse(data=out_det, format="turtle") + + g_nondet = Graph() + g_nondet.parse(data=out_nondet, format="turtle") + + assert len(g_det) == len(g_nondet), ( + f"Triple count mismatch: deterministic={len(g_det)}, non-deterministic={len(g_nondet)}" + ) + assert isomorphic(g_det, g_nondet), ( + f"{generator_cls.__name__}: deterministic output is NOT isomorphic " + "to non-deterministic output — the serialization changed the graph" + ) + + +@pytest.mark.parametrize( + "generator_cls", + [OwlSchemaGenerator, ShaclGenerator], + ids=["owl", "shacl"], +) +def test_non_deterministic_output_unchanged(generator_cls): + """Non-deterministic output must still produce valid RDF. + + Ensures that changes for deterministic mode don't break default behavior. + """ + out = generator_cls(SCHEMA, deterministic=False).serialize() + assert len(out) > 100, "Output suspiciously short" + g = Graph() + g.parse(data=out, format="turtle") + assert len(g) > 50, f"Graph has too few triples ({len(g)})" + + +@pytest.mark.parametrize( + "generator_cls,kwargs", + [ + (OwlSchemaGenerator, {}), + (ShaclGenerator, {}), + (ContextGenerator, {}), + (JSONLDGenerator, {}), + ], + ids=["owl", "shacl", "context", "jsonld"], +) +def test_non_deterministic_produces_valid_output(generator_cls, kwargs): + """All generators must produce valid output in non-deterministic mode.""" + out = generator_cls(SCHEMA, deterministic=False, **kwargs).serialize() + assert len(out) > 100, f"{generator_cls.__name__} output suspiciously short" + + +@pytest.mark.xfail( + reason=( + "Collection sorting in deterministic mode produces non-isomorphic RDF " + "(different rdf:first/rdf:rest triples). See test_deterministic_turtle_is_isomorphic." + ), + strict=True, +) +@pytest.mark.parametrize( + "generator_cls", + [OwlSchemaGenerator, ShaclGenerator], + ids=["owl", "shacl"], +) +def test_deterministic_kitchen_sink_isomorphic(generator_cls): + """Isomorphism check on the complex kitchen_sink schema. + + Expected to fail for the same reason as test_deterministic_turtle_is_isomorphic: + Collection sorting changes the RDF structure while preserving OWL/SHACL semantics. + """ + out_det = generator_cls(KITCHEN_SINK, deterministic=True).serialize() + out_nondet = generator_cls(KITCHEN_SINK, deterministic=False).serialize() + + g_det = Graph() + g_det.parse(data=out_det, format="turtle") + + g_nondet = Graph() + g_nondet.parse(data=out_nondet, format="turtle") + + assert isomorphic(g_det, g_nondet), ( + f"{generator_cls.__name__}: kitchen_sink deterministic output is NOT isomorphic to non-deterministic output" + ) + + +@pytest.mark.skipif(False, reason="does not require pyoxigraph") +def test_expression_sort_key_is_stable(): + """``_expression_sort_key`` must produce stable, content-based keys. + + LinkML anonymous expressions inherit ``YAMLRoot.__repr__()``, which + formats objects using **field values** (not memory addresses). + The ``_expression_sort_key`` helper relies on this for deterministic + ordering of ``any_of`` / ``all_of`` / ``none_of`` members. + + This test verifies that: + 1. Two distinct objects with identical fields produce the same key. + 2. Objects with different fields produce different keys. + 3. Sorting is stable across repeated calls. + """ + from linkml.generators.owlgen import _expression_sort_key + from linkml_runtime.linkml_model.meta import AnonymousClassExpression, AnonymousSlotExpression + + # Two distinct objects with identical content → same key + a1 = AnonymousClassExpression(is_a="Parent") + a2 = AnonymousClassExpression(is_a="Parent") + assert a1 is not a2 + assert _expression_sort_key(a1) == _expression_sort_key(a2) + + # Different content → different keys + b = AnonymousClassExpression(is_a="Child") + assert _expression_sort_key(a1) != _expression_sort_key(b) + + # Sorting stability: same order every time + items = [b, a1, a2] + for _ in range(5): + result = sorted(items, key=_expression_sort_key) + # "Child" < "Parent" alphabetically, so b comes first + assert _expression_sort_key(result[0]) == _expression_sort_key(b) + assert _expression_sort_key(result[1]) == _expression_sort_key(result[2]) # a1, a2 together + + # Slot expressions work too + s1 = AnonymousSlotExpression(range="string") + s2 = AnonymousSlotExpression(range="integer") + assert _expression_sort_key(s1) != _expression_sort_key(s2) + order1 = sorted([s2, s1], key=_expression_sort_key) + order2 = sorted([s1, s2], key=_expression_sort_key) + assert [_expression_sort_key(x) for x in order1] == [_expression_sort_key(x) for x in order2] diff --git a/tests/linkml/test_generators/test_jsonldcontextgen.py b/tests/linkml/test_generators/test_jsonldcontextgen.py index 6e3170d5ac..3a1081ceeb 100644 --- a/tests/linkml/test_generators/test_jsonldcontextgen.py +++ b/tests/linkml/test_generators/test_jsonldcontextgen.py @@ -1637,3 +1637,118 @@ def test_kitchen_sink_employment_event_type_falls_back(kitchen_sink_path): slot_def = ctx["employed_at"] if isinstance(slot_def, dict) and "@context" in slot_def: assert "@vocab" not in slot_def.get("@context", {}) + + +def test_normalize_prefixes_renames_nonstandard_alias(tmp_path): + """When --normalize-prefixes is set, non-standard aliases are replaced by rdflib defaults. + + rdflib binds ``dc`` to ``http://purl.org/dc/elements/1.1/`` by default. + A schema that declares ``dce`` for the same URI should have it normalised + to ``dc`` when the flag is enabled. + + See: rdflib default namespace bindings. + """ + schema = tmp_path / "schema.yaml" + schema.write_text( + """\ +id: https://example.org/test +name: test_normalize +default_prefix: ex +prefixes: + ex: https://example.org/ + linkml: https://w3id.org/linkml/ + dce: http://purl.org/dc/elements/1.1/ +imports: + - linkml:types +classes: + Record: + class_uri: ex:Record + attributes: + title: + range: string + slot_uri: dce:title +""", + encoding="utf-8", + ) + + # Flag OFF (default): non-standard alias preserved + ctx_off = json.loads(ContextGenerator(str(schema), normalize_prefixes=False).serialize())["@context"] + assert "dce" in ctx_off, "With flag off, original prefix 'dce' must be preserved" + + # Flag ON: rdflib default name used + ctx_on = json.loads(ContextGenerator(str(schema), normalize_prefixes=True).serialize())["@context"] + assert "dc" in ctx_on, "With flag on, 'dce' should be normalised to 'dc'" + assert "dce" not in ctx_on, "With flag on, original alias 'dce' should be removed" + assert ctx_on["dc"] == "http://purl.org/dc/elements/1.1/" + + +def test_normalize_prefixes_default_is_off(tmp_path): + """The --normalize-prefixes flag defaults to False — no prefix renaming. + + Ensures backward compatibility: existing schemas produce identical output. + """ + schema = tmp_path / "schema.yaml" + schema.write_text( + """\ +id: https://example.org/test +name: test_default +default_prefix: ex +prefixes: + ex: https://example.org/ + linkml: https://w3id.org/linkml/ + sdo: https://schema.org/ +imports: + - linkml:types +classes: + Thing: + class_uri: sdo:Thing + attributes: + name: + range: string + slot_uri: sdo:name +""", + encoding="utf-8", + ) + + ctx = json.loads(ContextGenerator(str(schema)).serialize())["@context"] + # Without the flag, the schema's own prefix name must be preserved + assert "sdo" in ctx, "Default behavior must preserve schema-declared prefix 'sdo'" + + +def test_normalize_prefixes_curie_remapping(tmp_path): + """CURIEs in element @id values use the normalised prefix name. + + When ``sdo`` is normalised to ``schema``, slot URIs like ``sdo:name`` + must appear as ``schema:name`` in the generated context. + """ + schema = tmp_path / "schema.yaml" + schema.write_text( + """\ +id: https://example.org/test +name: test_curie +default_prefix: ex +prefixes: + ex: https://example.org/ + linkml: https://w3id.org/linkml/ + sdo: https://schema.org/ +imports: + - linkml:types +classes: + Person: + class_uri: sdo:Person + attributes: + full_name: + range: string + slot_uri: sdo:name +""", + encoding="utf-8", + ) + + ctx = json.loads(ContextGenerator(str(schema), normalize_prefixes=True).serialize())["@context"] + # The prefix declaration must use the standard name + assert "schema" in ctx, "Normalised prefix 'schema' must appear" + # Element @id must use the normalised prefix + person = ctx.get("Person", {}) + assert person.get("@id", "").startswith("schema:"), ( + f"Person @id should use normalised prefix 'schema:', got {person}" + ) diff --git a/tests/linkml/test_generators/test_jsonschemagen.py b/tests/linkml/test_generators/test_jsonschemagen.py index 04ee3dbfd9..37477667ac 100644 --- a/tests/linkml/test_generators/test_jsonschemagen.py +++ b/tests/linkml/test_generators/test_jsonschemagen.py @@ -1090,3 +1090,111 @@ def test_add_lax_def_missing_required(): schema["$defs"]["NormalClass"] = {"type": "object", "properties": {"id": {}}, "required": ["id", "name"]} schema.add_lax_def("NormalClass", "id") assert schema["$defs"]["NormalClass__identifier_optional"]["required"] == ["name"] + + +def _inlined_dict_schema(key_slot_yaml: str, key_decl: str = "identifier: true", key_range: str = "string") -> str: + """Build a schema with an inlined-as-dict slot whose key slot is configured by + ``key_decl`` (``identifier: true`` or ``key: true``), ``key_range`` (the key slot + range), and ``key_slot_yaml`` (extra YAML lines for the key slot).""" + return f""" +id: https://example.org/test-key-constraints +name: test-key-constraints +prefixes: + linkml: https://w3id.org/linkml/ +default_range: string +imports: + - linkml:types +classes: + Container: + tree_root: true + attributes: + entries: + range: Entry + multivalued: true + inlined: true + inlined_as_list: false + Entry: + attributes: + key: + {key_decl} + range: {key_range} +{key_slot_yaml} + val: + range: string +""" + + +@pytest.mark.parametrize("key_decl", ["identifier: true", "key: true"]) +def test_inlined_dict_key_pattern_emits_property_names(key_decl): + """A literal ``pattern`` on the inlined-dict key slot (identifier or key) must be + rendered onto ``propertyNames``.""" + schema = _inlined_dict_schema(' pattern: "^[0-9]+$"', key_decl=key_decl) + generated = json.loads(JsonSchemaGenerator(schema).serialize()) + assert generated["properties"]["entries"]["propertyNames"] == {"pattern": "^[0-9]+$"} + + +def test_inlined_dict_key_enum_emits_property_names(): + """``equals_string_in`` on the key slot becomes an ``enum`` constraint on keys.""" + schema = _inlined_dict_schema(" equals_string_in:\n - a\n - b") + generated = json.loads(JsonSchemaGenerator(schema).serialize()) + assert generated["properties"]["entries"]["propertyNames"] == {"enum": ["a", "b"]} + + +def test_inlined_dict_no_key_constraint_emits_no_property_names(): + """No constraint on the key slot -> no ``propertyNames`` (unchanged behavior).""" + schema = _inlined_dict_schema("") + generated = json.loads(JsonSchemaGenerator(schema).serialize()) + assert "propertyNames" not in generated["properties"]["entries"] + + +def test_inlined_dict_key_structured_pattern_requires_materialization(): + """``structured_pattern`` on the key slot is honored only when patterns are + materialized -- identical to how value patterns are handled.""" + schema = _inlined_dict_schema( + " structured_pattern:\n syntax: '^[0-9]+$'\n interpolated: true" + ) + without = json.loads(JsonSchemaGenerator(schema).serialize()) + assert "propertyNames" not in without["properties"]["entries"] + + with_materialized = json.loads(JsonSchemaGenerator(schema, materialize_patterns=True).serialize()) + assert with_materialized["properties"]["entries"]["propertyNames"] == {"pattern": "^[0-9]+$"} + + +def test_inlined_dict_property_names_rejects_nonmatching_keys(): + """Behavioral check: keys matching the pattern validate; non-matching keys fail.""" + schema = _inlined_dict_schema(' pattern: "^[0-9]+$"') + generated = json.loads(JsonSchemaGenerator(schema).serialize()) + + jsonschema.validate({"entries": {"0": {"val": "x"}}}, generated) + with pytest.raises(jsonschema.ValidationError): + jsonschema.validate({"entries": {"bad-key": {"val": "x"}}}, generated) + + +def test_inlined_dict_key_string_const_emits_property_names(): + """A string ``const`` (``equals_string``) on the key slot becomes a key const.""" + schema = _inlined_dict_schema(" equals_string: fixed") + generated = json.loads(JsonSchemaGenerator(schema).serialize()) + assert generated["properties"]["entries"]["propertyNames"] == {"const": "fixed"} + jsonschema.validate({"entries": {"fixed": {"val": "x"}}}, generated) + with pytest.raises(jsonschema.ValidationError): + jsonschema.validate({"entries": {"other": {"val": "x"}}}, generated) + + +def test_inlined_dict_key_numeric_const_is_not_emitted(): + """A numeric ``const`` (``equals_number``) must NOT be emitted onto propertyNames: + keys are always strings, so a numeric const would reject every key. The keys are + left unconstrained instead.""" + schema = _inlined_dict_schema(" equals_number: 5", key_range="integer") + generated = json.loads(JsonSchemaGenerator(schema).serialize()) + assert "propertyNames" not in generated["properties"]["entries"] + # numeric-looking string keys still validate (unconstrained) + jsonschema.validate({"entries": {"5": {"val": "x"}}}, generated) + jsonschema.validate({"entries": {"anything": {"val": "x"}}}, generated) + + +def test_inlined_dict_key_numeric_bounds_are_not_emitted(): + """Numeric ``minimum``/``maximum`` on the key slot are no-ops on string keys and + must not be emitted (they would be misleading clutter).""" + schema = _inlined_dict_schema(" minimum_value: 1\n maximum_value: 10", key_range="integer") + generated = json.loads(JsonSchemaGenerator(schema).serialize()) + assert "propertyNames" not in generated["properties"]["entries"] diff --git a/tests/linkml/test_generators/test_normalize_prefixes.py b/tests/linkml/test_generators/test_normalize_prefixes.py new file mode 100644 index 0000000000..0a832a5791 --- /dev/null +++ b/tests/linkml/test_generators/test_normalize_prefixes.py @@ -0,0 +1,545 @@ +"""Tests for the --normalize-prefixes flag across all generators. + +Verifies that non-standard prefix aliases (e.g. ``sdo`` for ``https://schema.org/``) +are normalised to well-known names (e.g. ``schema``) consistently in OWL, SHACL, +and JSON-LD context output. + +References: +- prefix.cc — community consensus RDF prefix registry +- rdflib 7.x curated default namespace bindings +- W3C Turtle §2.4 — prefix declarations are syntactic sugar +""" + +import json +import logging +import re +import textwrap + +import pytest + +# ── Shared test schema ────────────────────────────────────────────── + +SCHEMA_SDO = textwrap.dedent("""\ + id: https://example.org/test + name: test_normalize + default_prefix: ex + prefixes: + ex: https://example.org/ + linkml: https://w3id.org/linkml/ + sdo: https://schema.org/ + imports: + - linkml:types + classes: + Person: + class_uri: sdo:Person + attributes: + full_name: + range: string + slot_uri: sdo:name +""") + +SCHEMA_DCE = textwrap.dedent("""\ + id: https://example.org/test + name: test_normalize_dce + default_prefix: ex + prefixes: + ex: https://example.org/ + linkml: https://w3id.org/linkml/ + dce: http://purl.org/dc/elements/1.1/ + imports: + - linkml:types + classes: + Record: + class_uri: ex:Record + attributes: + title: + range: string + slot_uri: dce:title +""") + +# HTTP variant — linkml-runtime historically binds schema: http://schema.org/ +# while rdflib (and the W3C) prefer https://schema.org/. The normalize flag +# must handle both. +SCHEMA_HTTP_SDO = textwrap.dedent("""\ + id: https://example.org/test + name: test_http_schema + default_prefix: ex + prefixes: + ex: https://example.org/ + linkml: https://w3id.org/linkml/ + sdo: http://schema.org/ + imports: + - linkml:types + classes: + Place: + class_uri: sdo:Place + attributes: + geo: + range: string + slot_uri: sdo:geo +""") + +# Collision scenario: user declares 'foaf' for a custom namespace AND 'myfoaf' +# for http://xmlns.com/foaf/0.1/. Normalisation must NOT clobber the user's 'foaf'. +# Uses 'foaf' instead of 'schema' because 'schema' is declared in linkml:types, +# which causes a SchemaLoader merge conflict before normalisation even runs. +SCHEMA_COLLISION = textwrap.dedent("""\ + id: https://example.org/test + name: test_collision + default_prefix: ex + prefixes: + ex: https://example.org/ + linkml: https://w3id.org/linkml/ + foaf: https://something-else.org/ + myfoaf: http://xmlns.com/foaf/0.1/ + imports: + - linkml:types + classes: + Agent: + class_uri: myfoaf:Agent + attributes: + label: + range: string + slot_uri: myfoaf:name +""") + + +def _write_schema(tmp_path, content: str, name: str = "schema.yaml") -> str: + """Write schema content to a temporary file and return its path as string.""" + p = tmp_path / name + p.write_text(content, encoding="utf-8") + return str(p) + + +def _turtle_prefixes(ttl: str) -> dict[str, str]: + """Extract @prefix declarations from Turtle output → {prefix: namespace}.""" + result = {} + for m in re.finditer(r"@prefix\s+(\w+):\s+<([^>]+)>", ttl): + result[m.group(1)] = m.group(2) + return result + + +# ── OWL Generator Tests ───────────────────────────────────────────── + + +def test_owl_sdo_normalised_to_schema(tmp_path): + """sdo → schema when --normalize-prefixes is active.""" + from linkml.generators.owlgen import OwlSchemaGenerator + + schema_path = _write_schema(tmp_path, SCHEMA_SDO) + ttl = OwlSchemaGenerator(schema_path, normalize_prefixes=True).serialize() + pfx = _turtle_prefixes(ttl) + assert "schema" in pfx, f"Expected 'schema' prefix in OWL output, got: {sorted(pfx)}" + assert pfx["schema"] == "https://schema.org/" + assert "sdo" not in pfx, "Non-standard 'sdo' prefix should be removed" + + +def test_owl_flag_off_preserves_original(tmp_path): + """Without the flag, schema-declared prefix names are preserved.""" + from linkml.generators.owlgen import OwlSchemaGenerator + + schema_path = _write_schema(tmp_path, SCHEMA_SDO) + ttl = OwlSchemaGenerator(schema_path, normalize_prefixes=False).serialize() + pfx = _turtle_prefixes(ttl) + assert "sdo" in pfx, "With flag off, original prefix 'sdo' must be preserved" + + +def test_owl_dce_normalised_to_dc(tmp_path): + """dce → dc for http://purl.org/dc/elements/1.1/ in graph bindings. + + Note: rdflib's Turtle serializer only emits @prefix declarations for + namespaces actually used in triples. Since the OWL generator may not + produce triples using dc:elements URIs for simple attribute schemas, + we verify the graph's namespace bindings directly. + """ + from linkml.generators.owlgen import OwlSchemaGenerator + + schema_path = _write_schema(tmp_path, SCHEMA_DCE) + gen = OwlSchemaGenerator(schema_path, normalize_prefixes=True) + graph = gen.as_graph() + bound = {str(p): str(n) for p, n in graph.namespaces()} + assert "dc" in bound, f"Expected 'dc' in graph bindings, got: {sorted(bound)}" + assert bound["dc"] == "http://purl.org/dc/elements/1.1/" + + +def test_owl_custom_prefix_not_affected(tmp_path): + """Domain-specific prefixes (e.g. 'ex') are not touched by normalisation.""" + from linkml.generators.owlgen import OwlSchemaGenerator + + schema_path = _write_schema(tmp_path, SCHEMA_SDO) + ttl = OwlSchemaGenerator(schema_path, normalize_prefixes=True).serialize() + pfx = _turtle_prefixes(ttl) + assert "ex" in pfx, "Custom prefix 'ex' must survive normalisation" + assert pfx["ex"] == "https://example.org/" + + +def test_owl_http_schema_org_normalised(tmp_path): + """http://schema.org/ (HTTP variant) also normalises to 'schema'. + + The linkml-runtime historically binds ``schema: http://schema.org/`` + while the W3C and rdflib prefer ``https://schema.org/``. Both + variants must be recognised by the static well-known prefix map. + """ + from linkml.generators.owlgen import OwlSchemaGenerator + + schema_path = _write_schema(tmp_path, SCHEMA_HTTP_SDO) + ttl = OwlSchemaGenerator(schema_path, normalize_prefixes=True).serialize() + pfx = _turtle_prefixes(ttl) + assert "schema" in pfx, f"Expected 'schema' prefix for http://schema.org/, got: {sorted(pfx)}" + assert "sdo" not in pfx + + +def test_owl_no_schema1_from_runtime_http_binding(tmp_path): + """Runtime-injected ``schema: http://schema.org/`` must not create ``schema1``. + + The linkml metamodel (types.yaml) declares ``schema: http://schema.org/`` + (HTTP). When a user schema declares ``sdo: https://schema.org/`` (HTTPS), + normalisation must clean up *both* variants so the output never contains + auto-generated suffixed prefixes like ``schema1``. + """ + from linkml.generators.owlgen import OwlSchemaGenerator + + schema_path = _write_schema(tmp_path, SCHEMA_SDO) + ttl = OwlSchemaGenerator(schema_path, normalize_prefixes=True).serialize() + pfx = _turtle_prefixes(ttl) + suffixed = [p for p in pfx if re.match(r"schema\d+", p)] + assert not suffixed, ( + f"Auto-generated suffixed prefix(es) {suffixed} found — runtime http://schema.org/ binding was not cleaned up" + ) + + +# ── SHACL Generator Tests ─────────────────────────────────────────── + + +def test_shacl_sdo_normalised_to_schema(tmp_path): + """sdo → schema when --normalize-prefixes is active.""" + from linkml.generators.shaclgen import ShaclGenerator + + schema_path = _write_schema(tmp_path, SCHEMA_SDO) + ttl = ShaclGenerator(schema_path, normalize_prefixes=True).serialize() + pfx = _turtle_prefixes(ttl) + assert "schema" in pfx, f"Expected 'schema' prefix in SHACL output, got: {sorted(pfx)}" + assert pfx["schema"] == "https://schema.org/" + assert "sdo" not in pfx, "Non-standard 'sdo' prefix should be removed" + + +def test_shacl_flag_off_preserves_original(tmp_path): + """Without the flag, schema-declared prefix names are preserved.""" + from linkml.generators.shaclgen import ShaclGenerator + + schema_path = _write_schema(tmp_path, SCHEMA_SDO) + ttl = ShaclGenerator(schema_path, normalize_prefixes=False).serialize() + pfx = _turtle_prefixes(ttl) + assert "sdo" in pfx, "With flag off, original prefix 'sdo' must be preserved" + + +def test_shacl_dce_normalised_to_dc(tmp_path): + """dce → dc for http://purl.org/dc/elements/1.1/.""" + from linkml.generators.shaclgen import ShaclGenerator + + schema_path = _write_schema(tmp_path, SCHEMA_DCE) + ttl = ShaclGenerator(schema_path, normalize_prefixes=True).serialize() + pfx = _turtle_prefixes(ttl) + assert "dc" in pfx, f"Expected 'dc' prefix in SHACL output, got: {sorted(pfx)}" + assert pfx["dc"] == "http://purl.org/dc/elements/1.1/" + assert "dce" not in pfx, "Non-standard 'dce' prefix should be removed" + + +def test_shacl_custom_prefix_not_affected(tmp_path): + """Domain-specific prefixes (e.g. 'ex') are not touched by normalisation. + + Note: rdflib only emits @prefix for namespaces used in triples. + We verify graph bindings directly. + """ + from linkml.generators.shaclgen import ShaclGenerator + + schema_path = _write_schema(tmp_path, SCHEMA_SDO) + gen = ShaclGenerator(schema_path, normalize_prefixes=True) + graph = gen.as_graph() + bound = {str(p): str(n) for p, n in graph.namespaces()} + assert "ex" in bound, f"Custom prefix 'ex' must survive in graph bindings, got: {sorted(bound)}" + assert bound["ex"] == "https://example.org/" + + +def test_shacl_http_schema_org_normalised(tmp_path): + """http://schema.org/ (HTTP variant) also normalises to 'schema'.""" + from linkml.generators.shaclgen import ShaclGenerator + + schema_path = _write_schema(tmp_path, SCHEMA_HTTP_SDO) + ttl = ShaclGenerator(schema_path, normalize_prefixes=True).serialize() + pfx = _turtle_prefixes(ttl) + assert "schema" in pfx, f"Expected 'schema' prefix for http://schema.org/, got: {sorted(pfx)}" + assert "sdo" not in pfx + + +def test_shacl_no_schema1_from_runtime_http_binding(tmp_path): + """Runtime-injected ``schema: http://schema.org/`` must not create ``schema1``. + + Same scenario as the OWL test: linkml:types imports bring in + ``schema: http://schema.org/`` while the user schema has + ``sdo: https://schema.org/``. Phase 2 of normalisation must + clean up the orphaned HTTP binding. + """ + from linkml.generators.shaclgen import ShaclGenerator + + schema_path = _write_schema(tmp_path, SCHEMA_SDO) + ttl = ShaclGenerator(schema_path, normalize_prefixes=True).serialize() + pfx = _turtle_prefixes(ttl) + suffixed = [p for p in pfx if re.match(r"schema\d+", p)] + assert not suffixed, ( + f"Auto-generated suffixed prefix(es) {suffixed} found — runtime http://schema.org/ binding was not cleaned up" + ) + + +# ── JSON-LD Context Generator Tests ───────────────────────────────── + + +def test_context_http_schema_org_normalised(tmp_path): + """http://schema.org/ (HTTP variant) normalises to 'schema' in JSON-LD context. + + This covers the edge case where linkml-runtime's ``schema: http://schema.org/`` + conflicts with rdflib's ``schema: https://schema.org/``. The stale binding + must be removed and replaced with the correct one. + """ + from linkml.generators.jsonldcontextgen import ContextGenerator + + schema_path = _write_schema(tmp_path, SCHEMA_HTTP_SDO) + ctx = json.loads(ContextGenerator(schema_path, normalize_prefixes=True).serialize())["@context"] + assert "schema" in ctx, "HTTP schema.org should normalise to 'schema'" + assert "sdo" not in ctx, "Non-standard 'sdo' should be removed" + # The namespace URI must match the schema-declared one (http, not https) + schema_val = ctx["schema"] + if isinstance(schema_val, dict): + schema_val = schema_val.get("@id", "") + assert schema_val == "http://schema.org/", f"Namespace URI must be preserved: got {schema_val}" + + +# ── Static Prefix Map Tests ───────────────────────────────────────── + + +def test_well_known_prefix_map_returns_dict(): + from linkml.utils.generator import well_known_prefix_map + + wk = well_known_prefix_map() + assert isinstance(wk, dict) + assert len(wk) >= 29, f"Expected ≥29 entries, got {len(wk)}" + + +def test_well_known_prefix_map_schema_https(): + from linkml.utils.generator import well_known_prefix_map + + wk = well_known_prefix_map() + assert wk["https://schema.org/"] == "schema" + + +def test_well_known_prefix_map_schema_http_variant(): + """Both http and https schema.org must map to 'schema'.""" + from linkml.utils.generator import well_known_prefix_map + + wk = well_known_prefix_map() + assert wk["http://schema.org/"] == "schema" + + +def test_well_known_prefix_map_dc_elements(): + from linkml.utils.generator import well_known_prefix_map + + wk = well_known_prefix_map() + assert wk["http://purl.org/dc/elements/1.1/"] == "dc" + + +def test_well_known_prefix_map_returns_copy(): + """Callers should not be able to mutate the internal map.""" + from linkml.utils.generator import well_known_prefix_map + + wk1 = well_known_prefix_map() + wk1["http://never-in-any-real-prefix-map.test/"] = "test" + wk2 = well_known_prefix_map() + assert "http://never-in-any-real-prefix-map.test/" not in wk2 + + +def test_well_known_prefix_map_fully_resolved_from_prefixmaps(): + """All rdflib defaults must be resolved from prefixmaps (no residual map). + + This is the proof that pinning prefixmaps to the commit containing + linkml/prefixmaps#81 resolves all well-known prefixes without any + hardcoded fallback. If this test fails after a prefixmaps update, + add the missing prefix to the upstream linked_data.curated.yaml. + """ + from rdflib import Graph as RdfGraph + + from linkml.utils.generator import well_known_prefix_map + + wk = well_known_prefix_map() + rdflib_map = {str(ns): str(pfx) for pfx, ns in RdfGraph().namespaces() if str(pfx)} + missing = {ns: pfx for ns, pfx in rdflib_map.items() if ns not in wk} + assert not missing, f"Prefix map missing rdflib defaults (add to prefixmaps upstream): {missing}" + + +# ── Cross-Generator Consistency Tests ──────────────────────────────── + + +def test_all_generators_normalise_sdo_to_schema(tmp_path): + """OWL, SHACL, and JSON-LD context must all use 'schema' for schema.org.""" + from linkml.generators.jsonldcontextgen import ContextGenerator + from linkml.generators.owlgen import OwlSchemaGenerator + from linkml.generators.shaclgen import ShaclGenerator + + schema_path = _write_schema(tmp_path, SCHEMA_SDO) + + owl_ttl = OwlSchemaGenerator(schema_path, normalize_prefixes=True).serialize() + shacl_ttl = ShaclGenerator(schema_path, normalize_prefixes=True).serialize() + ctx = json.loads(ContextGenerator(schema_path, normalize_prefixes=True).serialize())["@context"] + + owl_pfx = _turtle_prefixes(owl_ttl) + shacl_pfx = _turtle_prefixes(shacl_ttl) + + assert "schema" in owl_pfx, "OWL must use 'schema'" + assert "schema" in shacl_pfx, "SHACL must use 'schema'" + assert "schema" in ctx, "JSON-LD context must use 'schema'" + + assert "sdo" not in owl_pfx, "OWL must not have 'sdo'" + assert "sdo" not in shacl_pfx, "SHACL must not have 'sdo'" + assert "sdo" not in ctx, "JSON-LD context must not have 'sdo'" + + +# ── Prefix Collision Tests ──────────────────────────────────────────── + + +@pytest.mark.parametrize( + "generator_cls,generator_module", + [ + ("OwlSchemaGenerator", "linkml.generators.owlgen"), + ("ShaclGenerator", "linkml.generators.shaclgen"), + ], + ids=["owl", "shacl"], +) +def test_graph_generator_collision_skips_rename(tmp_path, caplog, generator_cls, generator_module): + """Graph generators: myfoaf must NOT be renamed to 'foaf' when user claims that name.""" + import importlib + + mod = importlib.import_module(generator_module) + cls = getattr(mod, generator_cls) + + schema_path = _write_schema(tmp_path, SCHEMA_COLLISION) + with caplog.at_level(logging.WARNING): + gen = cls(schema_path, normalize_prefixes=True) + graph = gen.as_graph() + bound = {str(p): str(n) for p, n in graph.namespaces()} + assert "myfoaf" in bound, "Non-standard 'myfoaf' must remain when collision prevents renaming" + assert bound["myfoaf"] == "http://xmlns.com/foaf/0.1/" + assert "collision" in caplog.text.lower(), f"Expected collision warning, got: {caplog.text}" + + +def test_context_collision_preserves_user_prefix(tmp_path, caplog): + """JSON-LD: user's 'foaf: https://something-else.org/' must survive.""" + from linkml.generators.jsonldcontextgen import ContextGenerator + + schema_path = _write_schema(tmp_path, SCHEMA_COLLISION) + with caplog.at_level(logging.WARNING): + ctx = json.loads(ContextGenerator(schema_path, normalize_prefixes=True).serialize())["@context"] + # User's 'foaf' binding preserved + foaf_val = ctx.get("foaf") + if isinstance(foaf_val, dict): + foaf_val = foaf_val.get("@id", "") + assert foaf_val == "https://something-else.org/", f"User's 'foaf' binding must be preserved, got: {foaf_val}" + # myfoaf must remain (not renamed to foaf) + assert "myfoaf" in ctx, "Non-standard 'myfoaf' must remain when collision prevents renaming" + # Warning emitted + assert "collision" in caplog.text.lower(), f"Expected collision warning, got: {caplog.text}" + + +# ── JSONLDGenerator Flag Forwarding Tests ───────────────────────────── + + +def test_jsonld_generator_forwards_normalize_prefixes(tmp_path): + """JSONLDGenerator must pass normalize_prefixes to embedded ContextGenerator. + + Without forwarding, the inline @context in JSON-LD output would keep + non-standard prefix aliases even when --normalize-prefixes is set. + """ + from linkml.generators.jsonldgen import JSONLDGenerator + + schema_path = _write_schema(tmp_path, SCHEMA_SDO) + out = JSONLDGenerator(schema_path, normalize_prefixes=True).serialize() + parsed = json.loads(out) + # The @context may be a list; find the dict entry + ctx = parsed.get("@context", {}) + if isinstance(ctx, list): + for item in ctx: + if isinstance(item, dict): + ctx = item + break + assert "sdo" not in ctx, "normalize_prefixes not forwarded: 'sdo' still in embedded @context" + + +# ── Phase 2 HTTP/HTTPS Overwrite Bug Tests ──────────────────────────── + + +def test_phase2_does_not_overwrite_https_with_http(tmp_path): + """When Phase 1 binds schema → https://schema.org/, Phase 2 must not + overwrite it with http://schema.org/ from the runtime metamodel. + + Reproduction: linkml:types imports bring schema: http://schema.org/ + (HTTP) while the user schema has sdo: https://schema.org/ (HTTPS). + Phase 1 normalises sdo → schema (HTTPS). Phase 2 must not then + rebind schema → http://schema.org/ when it encounters the runtime + HTTP binding. + """ + from linkml.generators.owlgen import OwlSchemaGenerator + + schema_path = _write_schema(tmp_path, SCHEMA_SDO) + gen = OwlSchemaGenerator(schema_path, normalize_prefixes=True) + graph = gen.as_graph() + bound = {str(p): str(n) for p, n in graph.namespaces()} + assert "schema" in bound, f"Expected 'schema' in bindings, got: {sorted(bound)}" + # MUST be HTTPS (from the user's schema), not HTTP (from runtime) + assert bound["schema"] == "https://schema.org/", ( + f"Phase 2 overwrote HTTPS with HTTP: schema bound to {bound['schema']}" + ) + + +def test_normalize_graph_prefixes_phase2_guard(): + """Direct unit test for the Phase 2 guard in normalize_graph_prefixes. + + Simulates the exact scenario: Phase 1 binds schema → https://schema.org/, + then Phase 2 encounters schema1 → http://schema.org/ and must NOT rebind. + """ + from rdflib import Graph, Namespace, URIRef + + from linkml.utils.generator import normalize_graph_prefixes + + g = Graph(bind_namespaces="none") + # Simulate Phase 1 result + g.bind("schema", Namespace("https://schema.org/")) + # Simulate runtime-injected HTTP variant (would appear as schema1) + g.bind("schema1", Namespace("http://schema.org/")) + # Add a triple so the graph isn't empty + g.add((URIRef("https://example.org/s"), URIRef("https://schema.org/name"), URIRef("https://example.org/o"))) + + normalize_graph_prefixes(g, {"sdo": "https://schema.org/"}) + + bound = {str(p): str(n) for p, n in g.namespaces()} + assert bound.get("schema") == "https://schema.org/", f"Phase 2 guard failed: schema bound to {bound.get('schema')}" + + +def test_empty_schema_no_crash(tmp_path): + """A schema with no custom prefixes must not crash normalize_graph_prefixes.""" + from linkml.generators.owlgen import OwlSchemaGenerator + + (tmp_path / "empty.yaml").write_text( + textwrap.dedent("""\ + id: https://example.org/empty + name: empty + default_prefix: ex + prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/ + imports: + - linkml:types + """), + encoding="utf-8", + ) + # Should not raise + gen = OwlSchemaGenerator(str(tmp_path / "empty.yaml"), normalize_prefixes=True) + ttl = gen.serialize() + assert len(ttl) > 0 diff --git a/tests/linkml/test_generators/test_shaclgen.py b/tests/linkml/test_generators/test_shaclgen.py index 6b19cf24b1..9c765eb2c1 100644 --- a/tests/linkml/test_generators/test_shaclgen.py +++ b/tests/linkml/test_generators/test_shaclgen.py @@ -600,6 +600,151 @@ def test_multivalued_slot_exact_cardinality(input_path): ) in g +def test_zero_maximum_cardinality_emits_maxcount(input_path): + """Test that maximum_cardinality: 0 correctly emits sh:maxCount 0. + + Regression test for the bug where Python truthiness check + `if s.maximum_cardinality:` would skip the value 0 (falsy), + failing to emit sh:maxCount 0 in the generated SHACL shape. + The fix uses `if s.maximum_cardinality is not None:` instead. + + This is the primary mechanism for suppressing inherited slots on + subclasses via slot_usage (e.g., OWL maxCardinality 0 pattern). + """ + shacl = ShaclGenerator(input_path("shaclgen/cardinality.yaml"), mergeimports=True).serialize() + + g = rdflib.Graph() + g.parse(data=shacl) + + # Find the ChildWithZeroMaxCard shape + child_uri = URIRef("https://w3id.org/linkml/examples/cardinality/ChildWithZeroMaxCard") + restricted_slot_uri = URIRef("https://w3id.org/linkml/examples/cardinality/restricted_slot") + + # Get all property shapes for the child class + prop_nodes = list(g.objects(child_uri, SH.property)) + assert prop_nodes, "ChildWithZeroMaxCard should have property shapes" + + # Find the property shape for restricted_slot + restricted_prop_node = None + for pn in prop_nodes: + if (pn, SH.path, restricted_slot_uri) in g: + restricted_prop_node = pn + break + assert restricted_prop_node is not None, "Should have a property shape for restricted_slot" + + # The critical assertion: sh:maxCount 0 must be emitted + max_count_values = list(g.objects(restricted_prop_node, SH.maxCount)) + assert len(max_count_values) == 1, f"Expected exactly one sh:maxCount, got {max_count_values}" + assert max_count_values[0] == rdflib.term.Literal( + 0, datatype=rdflib.term.URIRef("http://www.w3.org/2001/XMLSchema#integer") + ), f"sh:maxCount should be 0, got {max_count_values[0]}" + + +def test_zero_exact_cardinality_emits_both_counts(input_path): + """Test that exact_cardinality: 0 emits both sh:minCount 0 and sh:maxCount 0. + + Same truthiness bug as maximum_cardinality: `if s.exact_cardinality:` + skips value 0 (falsy). The fix uses `is not None` instead. + """ + shacl = ShaclGenerator(input_path("shaclgen/cardinality.yaml"), mergeimports=True).serialize() + + g = rdflib.Graph() + g.parse(data=shacl) + + child_uri = URIRef("https://w3id.org/linkml/examples/cardinality/ChildWithZeroExactCard") + restricted_slot_uri = URIRef("https://w3id.org/linkml/examples/cardinality/restricted_slot") + + prop_nodes = list(g.objects(child_uri, SH.property)) + assert prop_nodes, "ChildWithZeroExactCard should have property shapes" + + restricted_prop_node = None + for pn in prop_nodes: + if (pn, SH.path, restricted_slot_uri) in g: + restricted_prop_node = pn + break + assert restricted_prop_node is not None, "Should have a property shape for restricted_slot" + + XSD_INT = rdflib.term.URIRef("http://www.w3.org/2001/XMLSchema#integer") + + min_count_values = list(g.objects(restricted_prop_node, SH.minCount)) + assert len(min_count_values) == 1, f"Expected exactly one sh:minCount, got {min_count_values}" + assert min_count_values[0] == rdflib.term.Literal(0, datatype=XSD_INT) + + max_count_values = list(g.objects(restricted_prop_node, SH.maxCount)) + assert len(max_count_values) == 1, f"Expected exactly one sh:maxCount, got {max_count_values}" + assert max_count_values[0] == rdflib.term.Literal(0, datatype=XSD_INT) + + +def test_zero_minimum_cardinality_emits_mincount(input_path): + """Test that minimum_cardinality: 0 emits sh:minCount 0. + + Same truthiness bug as maximum_cardinality: `if s.minimum_cardinality:` + skips value 0 (falsy). The fix uses `is not None` instead. sh:minCount 0 + is vacuously satisfied (W3C SHACL 4.2.2) but is emitted for consistency + with owlgen (owl:minCardinality 0) and to faithfully reflect the schema. + """ + shacl = ShaclGenerator(input_path("shaclgen/cardinality.yaml"), mergeimports=True).serialize() + + g = rdflib.Graph() + g.parse(data=shacl) + + child_uri = URIRef("https://w3id.org/linkml/examples/cardinality/ChildWithZeroMinCard") + restricted_slot_uri = URIRef("https://w3id.org/linkml/examples/cardinality/restricted_slot") + + prop_nodes = list(g.objects(child_uri, SH.property)) + assert prop_nodes, "ChildWithZeroMinCard should have property shapes" + + restricted_prop_node = None + for pn in prop_nodes: + if (pn, SH.path, restricted_slot_uri) in g: + restricted_prop_node = pn + break + assert restricted_prop_node is not None, "Should have a property shape for restricted_slot" + + XSD_INT = rdflib.term.URIRef("http://www.w3.org/2001/XMLSchema#integer") + + min_count_values = list(g.objects(restricted_prop_node, SH.minCount)) + assert len(min_count_values) == 1, f"Expected exactly one sh:minCount, got {min_count_values}" + assert min_count_values[0] == rdflib.term.Literal(0, datatype=XSD_INT) + + +def test_explicit_minimum_cardinality_overrides_required(input_path): + """An explicit minimum_cardinality: 0 takes precedence over required: true. + + The generator resolves min-count with an ``elif`` cascade in which an + explicit ``minimum_cardinality`` wins and ``required`` is only the fallback. + This mirrors owlgen.py (``if slot.minimum_cardinality is not None ... elif + slot.required``), so ``required: true`` + ``minimum_cardinality: 0`` yields + ``sh:minCount 0`` (not 1). The combination is a schema contradiction; the + explicit, more specific constraint is emitted. + """ + shacl = ShaclGenerator(input_path("shaclgen/cardinality.yaml"), mergeimports=True).serialize() + + g = rdflib.Graph() + g.parse(data=shacl) + + child_uri = URIRef("https://w3id.org/linkml/examples/cardinality/ChildWithRequiredAndZeroMinCard") + restricted_slot_uri = URIRef("https://w3id.org/linkml/examples/cardinality/restricted_slot") + + prop_nodes = list(g.objects(child_uri, SH.property)) + assert prop_nodes, "ChildWithRequiredAndZeroMinCard should have property shapes" + + restricted_prop_node = None + for pn in prop_nodes: + if (pn, SH.path, restricted_slot_uri) in g: + restricted_prop_node = pn + break + assert restricted_prop_node is not None, "Should have a property shape for restricted_slot" + + XSD_INT = rdflib.term.URIRef("http://www.w3.org/2001/XMLSchema#integer") + + min_count_values = list(g.objects(restricted_prop_node, SH.minCount)) + assert len(min_count_values) == 1, f"Expected exactly one sh:minCount, got {min_count_values}" + assert min_count_values[0] == rdflib.term.Literal(0, datatype=XSD_INT), ( + f"explicit minimum_cardinality: 0 should override required: true (minCount 0, not 1), got {min_count_values[0]}" + ) + + def test_exclude_imports(input_path): shacl = ShaclGenerator( input_path("shaclgen/exclude_imports.yaml"), mergeimports=True, exclude_imports=True @@ -1194,6 +1339,7 @@ def test_nodeidentifier_range_produces_blank_node_or_iri(): assert SH.IRI in uri_kinds, f"Expected sh:IRI for uri, got {uri_kinds}" +# --------------------------------------------------------------------------- # --------------------------------------------------------------------------- # --default-language tests # --------------------------------------------------------------------------- @@ -1250,6 +1396,11 @@ def _build_message_test_schema(): return sb.schema +# --------------------------------------------------------------------------- +# Helper functions +# --------------------------------------------------------------------------- + + def _parse_shacl(schema, **kwargs): shacl = ShaclGenerator(schema, mergeimports=False, **kwargs).serialize() g = rdflib.Graph() @@ -1744,3 +1895,2995 @@ def test_message_template_ignores_per_slot_in_language(): # Contrast: sh:name DOES follow the slot's in_language ("de"). names = _get_prop_objects(g, vehicle_shape, EX.vehicle_name, SH.name) assert Literal("Name", lang="de") in names + + +# --------------------------------------------------------------------------- +# --emit-rules / sh:sparql tests +# --------------------------------------------------------------------------- + +_RULES_SCHEMA_YAML = """ +id: https://example.org/boolean-guards +name: boolean_guard_rules +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/boolean-guards/ +imports: + - linkml:types +default_prefix: ex +default_range: string +slots: + WeatherWind: + range: boolean + slot_uri: ex:WeatherWind + weatherWindValue: + description: Wind speed value. + range: decimal + slot_uri: ex:weatherWindValue + WeatherRain: + range: boolean + slot_uri: ex:WeatherRain + weatherRainValue: + description: Rain intensity value. + range: decimal + slot_uri: ex:weatherRainValue + Temperature: + range: decimal + slot_uri: ex:Temperature +classes: + Environment: + class_uri: ex:Environment + slots: + - WeatherWind + - weatherWindValue + - WeatherRain + - weatherRainValue + - Temperature + rules: + - description: If weatherWindValue is provided, WeatherWind must be true. + preconditions: + slot_conditions: + weatherWindValue: + value_presence: PRESENT + postconditions: + slot_conditions: + WeatherWind: + equals_string: "true" + - description: If weatherRainValue is provided, WeatherRain must be true. + preconditions: + slot_conditions: + weatherRainValue: + value_presence: PRESENT + postconditions: + slot_conditions: + WeatherRain: + equals_string: "true" +""" + +EX_RULES = rdflib.Namespace("https://example.org/boolean-guards/") + + +def test_rule_boolean_guard_generates_sparql(): + """Boolean-guard rules produce sh:sparql constraints on the NodeShape.""" + g = _parse_shacl(_RULES_SCHEMA_YAML) + + shape = EX_RULES.Environment + sparql_nodes = list(g.objects(shape, SH.sparql)) + assert len(sparql_nodes) == 2, f"Expected 2 sh:sparql constraints, got {len(sparql_nodes)}" + + for node in sparql_nodes: + assert (node, RDF.type, SH.SPARQLConstraint) in g + selects = list(g.objects(node, SH.select)) + assert len(selects) == 1, "Each constraint must have exactly one sh:select" + query = str(selects[0]) + assert "$this" in query, "SPARQL must use $this pre-bound variable" + assert "OPTIONAL" in query, "SPARQL must use OPTIONAL for flag/value" + assert "FILTER" in query, "SPARQL must have a FILTER clause" + assert "BOUND" in query, "SPARQL must use BOUND()" + + +def test_rule_with_description_generates_message(): + """Rule description is emitted as sh:message on the SPARQLConstraint.""" + g = _parse_shacl(_RULES_SCHEMA_YAML) + + shape = EX_RULES.Environment + sparql_nodes = list(g.objects(shape, SH.sparql)) + + messages = set() + for node in sparql_nodes: + for msg in g.objects(node, SH.message): + messages.add(str(msg)) + + assert "If weatherWindValue is provided, WeatherWind must be true." in messages + assert "If weatherRainValue is provided, WeatherRain must be true." in messages + + +def test_rule_sparql_contains_correct_uris(): + """SPARQL queries reference the correct slot URIs.""" + g = _parse_shacl(_RULES_SCHEMA_YAML) + + shape = EX_RULES.Environment + sparql_nodes = list(g.objects(shape, SH.sparql)) + + queries = [str(list(g.objects(n, SH.select))[0]) for n in sparql_nodes] + all_sparql = "\n".join(queries) + + assert str(EX_RULES.WeatherWind) in all_sparql + assert str(EX_RULES.weatherWindValue) in all_sparql + assert str(EX_RULES.WeatherRain) in all_sparql + assert str(EX_RULES.weatherRainValue) in all_sparql + + +_DEACTIVATED_RULE_SCHEMA_YAML = """ +id: https://example.org/deactivated-test +name: deactivated_rule_test +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/deactivated-test/ +imports: + - linkml:types +default_prefix: ex +default_range: string +slots: + Flag: + range: boolean + slot_uri: ex:Flag + flagValue: + range: decimal + slot_uri: ex:flagValue +classes: + TestClass: + class_uri: ex:TestClass + slots: + - Flag + - flagValue + rules: + - description: This rule is deactivated. + deactivated: true + preconditions: + slot_conditions: + flagValue: + value_presence: PRESENT + postconditions: + slot_conditions: + Flag: + equals_string: "true" +""" + + +def test_rule_deactivated_skipped(): + """Deactivated rules do not produce sh:sparql constraints.""" + g = _parse_shacl(_DEACTIVATED_RULE_SCHEMA_YAML) + + shape = URIRef("https://example.org/deactivated-test/TestClass") + sparql_nodes = list(g.objects(shape, SH.sparql)) + assert len(sparql_nodes) == 0, f"Deactivated rule should not emit sh:sparql, got {len(sparql_nodes)}" + + +_UNSUPPORTED_RULE_SCHEMA_YAML = """ +id: https://example.org/unsupported-test +name: unsupported_rule_test +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/unsupported-test/ +imports: + - linkml:types +default_prefix: ex +default_range: string +slots: + slotA: + range: string + slot_uri: ex:slotA + slotB: + range: string + slot_uri: ex:slotB +classes: + TestClass: + class_uri: ex:TestClass + slots: + - slotA + - slotB + rules: + - description: Rule with no postconditions. + preconditions: + slot_conditions: + slotA: + value_presence: PRESENT +""" + + +def test_rule_unsupported_pattern_skipped(): + """Unrecognised rule patterns are silently skipped (no sh:sparql emitted).""" + g = _parse_shacl(_UNSUPPORTED_RULE_SCHEMA_YAML) + + shape = URIRef("https://example.org/unsupported-test/TestClass") + sparql_nodes = list(g.objects(shape, SH.sparql)) + assert len(sparql_nodes) == 0 + + +def test_rule_no_emit_rules_flag(): + """--no-emit-rules suppresses sh:sparql constraint generation.""" + g = _parse_shacl(_RULES_SCHEMA_YAML, emit_rules=False) + + shape = EX_RULES.Environment + sparql_nodes = list(g.objects(shape, SH.sparql)) + assert len(sparql_nodes) == 0, f"emit_rules=False should suppress rules, got {len(sparql_nodes)}" + + +_NO_RULES_SCHEMA_YAML = """ +id: https://example.org/no-rules +name: no_rules_test +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/no-rules/ +imports: + - linkml:types +default_prefix: ex +default_range: string +slots: + name: + range: string + slot_uri: ex:name +classes: + SimpleClass: + class_uri: ex:SimpleClass + slots: + - name +""" + + +def test_rule_no_rules_no_sparql(): + """Classes without rules: blocks produce no sh:sparql constraints.""" + g = _parse_shacl(_NO_RULES_SCHEMA_YAML) + + shape = URIRef("https://example.org/no-rules/SimpleClass") + sparql_nodes = list(g.objects(shape, SH.sparql)) + assert len(sparql_nodes) == 0 + + +def test_rule_multiple_rules_per_class(): + """Multiple boolean-guard rules on one class produce multiple sh:sparql constraints.""" + g = _parse_shacl(_RULES_SCHEMA_YAML) + + shape = EX_RULES.Environment + sparql_nodes = list(g.objects(shape, SH.sparql)) + assert len(sparql_nodes) == 2 + + # Each constraint should reference different slot pairs + queries = [str(list(g.objects(n, SH.select))[0]) for n in sparql_nodes] + wind_query = [q for q in queries if "weatherWindValue" in q] + rain_query = [q for q in queries if "weatherRainValue" in q] + assert len(wind_query) == 1, "Expected exactly one wind query" + assert len(rain_query) == 1, "Expected exactly one rain query" + + +# --------------------------------------------------------------------------- +# Tests for URI resolution without explicit slot_uri +# --------------------------------------------------------------------------- + +_NO_SLOT_URI_SCHEMA_YAML = """ +id: https://example.org/no-slot-uri +name: no_slot_uri_test +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/no-slot-uri/ +imports: + - linkml:types +default_prefix: ex +default_range: string +slots: + is_active: + range: boolean + measured_value: + range: decimal +classes: + Reading: + class_uri: ex:Reading + slots: + - is_active + - measured_value + rules: + - description: If measured_value is provided, is_active must be true. + preconditions: + slot_conditions: + measured_value: + value_presence: PRESENT + postconditions: + slot_conditions: + is_active: + equals_string: "true" +""" + + +def test_rule_no_explicit_slot_uri(): + """Slots without explicit slot_uri resolve via default_prefix + underscore(name).""" + g = _parse_shacl(_NO_SLOT_URI_SCHEMA_YAML) + + shape = URIRef("https://example.org/no-slot-uri/Reading") + sparql_nodes = list(g.objects(shape, SH.sparql)) + assert len(sparql_nodes) == 1 + + query = str(list(g.objects(sparql_nodes[0], SH.select))[0]) + # URIs should be default_prefix:underscore(name) + assert "https://example.org/no-slot-uri/is_active" in query + assert "https://example.org/no-slot-uri/measured_value" in query + + +# --------------------------------------------------------------------------- +# Tests for elseconditions rejection +# --------------------------------------------------------------------------- + +_ELSE_COND_SCHEMA_YAML = """ +id: https://example.org/else-test +name: else_cond_test +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/else-test/ +imports: + - linkml:types +default_prefix: ex +default_range: string +slots: + Flag: + range: boolean + slot_uri: ex:Flag + flagValue: + range: decimal + slot_uri: ex:flagValue + fallbackValue: + range: string + slot_uri: ex:fallbackValue +classes: + TestClass: + class_uri: ex:TestClass + slots: + - Flag + - flagValue + - fallbackValue + rules: + - description: Rule with elseconditions should be skipped. + preconditions: + slot_conditions: + flagValue: + value_presence: PRESENT + postconditions: + slot_conditions: + Flag: + equals_string: "true" + elseconditions: + slot_conditions: + fallbackValue: + value_presence: PRESENT +""" + + +def test_rule_with_elseconditions_emitted(): + """Rules with elseconditions now emit the forward (if/then) branch as sh:sparql.""" + g = _parse_shacl(_ELSE_COND_SCHEMA_YAML) + + shape = URIRef("https://example.org/else-test/TestClass") + sparql_nodes = list(g.objects(shape, SH.sparql)) + assert len(sparql_nodes) >= 1, "Rule with elseconditions should emit sh:sparql for the forward branch" + + +# --------------------------------------------------------------------------- +# SPARQL syntax validation +# --------------------------------------------------------------------------- + + +def test_rule_sparql_syntax_valid(): + """Generated SPARQL queries must be syntactically valid.""" + from rdflib.plugins.sparql import prepareQuery + + g = _parse_shacl(_RULES_SCHEMA_YAML) + + shape = EX_RULES.Environment + sparql_nodes = list(g.objects(shape, SH.sparql)) + assert len(sparql_nodes) >= 1 + + for node in sparql_nodes: + query_text = str(list(g.objects(node, SH.select))[0]) + # prepareQuery validates SPARQL syntax; $this is a valid variable name + prepareQuery(query_text) + + +# =========================================================================== +# Exclusive-value pattern tests (SHACL §5 SPARQL constraints) +# =========================================================================== +# +# The "exclusive value" pattern translates a LinkML rule where: +# - preconditions: slot X has equals_string (a specific enum value name) +# - postconditions: same slot X has maximum_cardinality N +# +# Semantics: "If value V is present in multivalued slot X, then X has at most +# N values total." For N=1 this means V must be the sole value (mutual +# exclusion with other enum members). +# +# Generated SHACL: sh:SPARQLConstraint per W3C SHACL §5.3.1, using $this +# pre-bound to each focus node. +# +# References: +# - W3C SHACL §5 +# - W3C SHACL §5.3.1 +# - ISO 34503:2023, 9.3.6 (motivating use case: EdgeNone exclusivity) +# =========================================================================== + +_EXCLUSIVE_VALUE_SCHEMA_YAML = """ +id: https://example.org/exclusive-value +name: exclusive_value_rules +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/exclusive-value/ +imports: + - linkml:types +default_prefix: ex +default_range: string + +enums: + EdgeTypeEnum: + permissible_values: + EdgeNone: + meaning: ex:EdgeNone + EdgeBarriers: + meaning: ex:EdgeBarriers + EdgeMarkers: + meaning: ex:EdgeMarkers + + PriorityEnum: + permissible_values: + High: + description: High priority (no meaning IRI). + Medium: + description: Medium priority (no meaning IRI). + Low: + description: Low priority (no meaning IRI). + +slots: + edgeType: + range: EdgeTypeEnum + multivalued: true + slot_uri: ex:edgeType + priority: + range: PriorityEnum + multivalued: true + slot_uri: ex:priority + otherSlot: + range: string + slot_uri: ex:otherSlot + +classes: + Road: + class_uri: ex:Road + slots: + - edgeType + - otherSlot + rules: + - description: >- + EdgeNone is mutually exclusive with other edge types. + preconditions: + slot_conditions: + edgeType: + equals_string: "EdgeNone" + postconditions: + slot_conditions: + edgeType: + maximum_cardinality: 1 + + Intersection: + class_uri: ex:Intersection + slots: + - edgeType + rules: + - description: >- + EdgeNone allows at most 2 total edge values. + preconditions: + slot_conditions: + edgeType: + equals_string: "EdgeNone" + postconditions: + slot_conditions: + edgeType: + maximum_cardinality: 2 + + Task: + class_uri: ex:Task + slots: + - priority + rules: + - description: >- + High priority is exclusive (literal fallback test). + preconditions: + slot_conditions: + priority: + equals_string: "High" + postconditions: + slot_conditions: + priority: + maximum_cardinality: 1 + + MismatchedSlots: + class_uri: ex:MismatchedSlots + slots: + - edgeType + - otherSlot + rules: + - description: >- + Different slots in pre/post — not an exclusive-value pattern. + preconditions: + slot_conditions: + edgeType: + equals_string: "EdgeNone" + postconditions: + slot_conditions: + otherSlot: + maximum_cardinality: 1 +""" + +EX_EXCL = rdflib.Namespace("https://example.org/exclusive-value/") + + +def test_exclusive_value_generates_sparql(): + """Exclusive-value rules produce sh:sparql constraints on the NodeShape.""" + g = _parse_shacl(_EXCLUSIVE_VALUE_SCHEMA_YAML) + + shape = EX_EXCL.Road + sparql_nodes = list(g.objects(shape, SH.sparql)) + assert len(sparql_nodes) == 1, f"Expected 1 sh:sparql constraint, got {len(sparql_nodes)}" + + node = sparql_nodes[0] + assert (node, RDF.type, SH.SPARQLConstraint) in g + selects = list(g.objects(node, SH.select)) + assert len(selects) == 1, "Constraint must have exactly one sh:select" + + +def test_exclusive_value_sparql_uses_enum_iri(): + """SPARQL references the enum value's meaning IRI, not a string literal. + + Per the enum definition, EdgeNone has meaning: ex:EdgeNone which expands + to . The generated SPARQL + must use this full IRI in angle brackets. + """ + g = _parse_shacl(_EXCLUSIVE_VALUE_SCHEMA_YAML) + + shape = EX_EXCL.Road + sparql_nodes = list(g.objects(shape, SH.sparql)) + query = str(list(g.objects(sparql_nodes[0], SH.select))[0]) + + edge_none_iri = str(EX_EXCL.EdgeNone) + assert f"<{edge_none_iri}>" in query, f"SPARQL must reference EdgeNone as full IRI <{edge_none_iri}>, got:\n{query}" + + +def test_exclusive_value_max_card_1_sparql_structure(): + """For maximum_cardinality: 1, SPARQL uses FILTER(?other != ). + + The query pattern for N=1 is: + SELECT $this WHERE { + $this . + $this ?other . + FILTER (?other != ) + } + + This is more efficient than the COUNT-based approach for the common + singleton exclusion case. + """ + g = _parse_shacl(_EXCLUSIVE_VALUE_SCHEMA_YAML) + + shape = EX_EXCL.Road + sparql_nodes = list(g.objects(shape, SH.sparql)) + query = str(list(g.objects(sparql_nodes[0], SH.select))[0]) + + assert "$this" in query, "SPARQL must use $this pre-bound variable (SHACL §5.3.1)" + assert "FILTER" in query, "N=1 pattern must use FILTER for exclusion check" + assert "?other" in query, "N=1 pattern must bind ?other for comparison" + # Must NOT use COUNT for the N=1 case (simpler pattern) + assert "COUNT" not in query, "N=1 pattern should use FILTER, not COUNT" + # The slot URI must appear (property path) + assert str(EX_EXCL.edgeType) in query, "SPARQL must reference the slot URI" + + +def test_exclusive_value_max_card_gt1_sparql_structure(): + """For maximum_cardinality > 1, SPARQL uses COUNT-based subquery. + + The query pattern for N>1 is: + SELECT $this WHERE { + $this . + { + SELECT $this (COUNT(?val) AS ?count) + WHERE { $this ?val . } + GROUP BY $this + HAVING (?count > N) + } + } + """ + g = _parse_shacl(_EXCLUSIVE_VALUE_SCHEMA_YAML) + + shape = EX_EXCL.Intersection + sparql_nodes = list(g.objects(shape, SH.sparql)) + assert len(sparql_nodes) == 1, f"Expected 1 sh:sparql constraint, got {len(sparql_nodes)}" + + query = str(list(g.objects(sparql_nodes[0], SH.select))[0]) + + assert "$this" in query, "SPARQL must use $this pre-bound variable" + assert "COUNT" in query, "N>1 pattern must use COUNT" + assert "GROUP BY" in query, "N>1 pattern must GROUP BY $this" + assert "HAVING" in query, "N>1 pattern must use HAVING for count check" + assert "> 2" in query, "HAVING must check count > maximum_cardinality (2)" + + +def test_exclusive_value_no_meaning_falls_back_to_literal(): + """When enum values lack a meaning IRI, the value is compared as a literal. + + PriorityEnum values have no meaning field, so 'High' is used as a + quoted string in the SPARQL rather than an IRI in angle brackets. + """ + g = _parse_shacl(_EXCLUSIVE_VALUE_SCHEMA_YAML) + + shape = EX_EXCL.Task + sparql_nodes = list(g.objects(shape, SH.sparql)) + assert len(sparql_nodes) == 1, f"Expected 1 sh:sparql constraint, got {len(sparql_nodes)}" + + query = str(list(g.objects(sparql_nodes[0], SH.select))[0]) + + # Should use quoted literal, not angle-bracket IRI + assert '"High"' in query, f"No-meaning enum should use literal '\"High\"', got:\n{query}" + assert "" not in query, "Should not emit as IRI when meaning is absent" + + +def test_exclusive_value_different_slots_not_recognised(): + """Rules where pre/post reference different slots are NOT exclusive-value. + + The pattern requires the SAME slot in both preconditions and + postconditions. When they differ, the rule is unrecognised and + silently skipped (no sh:sparql emitted). + """ + g = _parse_shacl(_EXCLUSIVE_VALUE_SCHEMA_YAML) + + shape = EX_EXCL.MismatchedSlots + sparql_nodes = list(g.objects(shape, SH.sparql)) + assert len(sparql_nodes) == 0, ( + f"Mismatched slots should not trigger exclusive-value pattern, got {len(sparql_nodes)}" + ) + + +def test_exclusive_value_message_from_description(): + """Rule description is emitted as sh:message on the SPARQLConstraint.""" + g = _parse_shacl(_EXCLUSIVE_VALUE_SCHEMA_YAML) + + shape = EX_EXCL.Road + sparql_nodes = list(g.objects(shape, SH.sparql)) + messages = [str(m) for node in sparql_nodes for m in g.objects(node, SH.message)] + + assert any("EdgeNone is mutually exclusive" in m for m in messages), ( + f"Expected message about EdgeNone exclusivity, got: {messages}" + ) + + +def test_exclusive_value_sparql_syntax_valid(): + """Generated SPARQL for exclusive-value rules must be syntactically valid. + + Uses rdflib's prepareQuery() which validates SPARQL syntax. + $this is a valid SPARQL variable name per the grammar. + """ + from rdflib.plugins.sparql import prepareQuery + + g = _parse_shacl(_EXCLUSIVE_VALUE_SCHEMA_YAML) + + for shape in (EX_EXCL.Road, EX_EXCL.Intersection, EX_EXCL.Task): + sparql_nodes = list(g.objects(shape, SH.sparql)) + for node in sparql_nodes: + query_text = str(list(g.objects(node, SH.select))[0]) + # prepareQuery validates SPARQL syntax + prepareQuery(query_text) + + +def test_exclusive_value_coexists_with_boolean_guard(): + """Exclusive-value and boolean-guard rules can coexist on the same class. + + When a class has both pattern types, both produce sh:sparql constraints. + """ + schema = """ +id: https://example.org/mixed-rules +name: mixed_rules +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/mixed-rules/ +imports: + - linkml:types +default_prefix: ex +default_range: string + +enums: + StatusEnum: + permissible_values: + None: + meaning: ex:None + Active: + meaning: ex:Active + +slots: + status: + range: StatusEnum + multivalued: true + slot_uri: ex:status + Flag: + range: boolean + slot_uri: ex:Flag + flagValue: + range: decimal + slot_uri: ex:flagValue + +classes: + Widget: + class_uri: ex:Widget + slots: + - status + - Flag + - flagValue + rules: + - description: None is exclusive. + preconditions: + slot_conditions: + status: + equals_string: "None" + postconditions: + slot_conditions: + status: + maximum_cardinality: 1 + - description: If flagValue present, Flag must be true. + preconditions: + slot_conditions: + flagValue: + value_presence: PRESENT + postconditions: + slot_conditions: + Flag: + equals_string: "true" +""" + g = _parse_shacl(schema) + + shape = URIRef("https://example.org/mixed-rules/Widget") + sparql_nodes = list(g.objects(shape, SH.sparql)) + assert len(sparql_nodes) == 2, ( + f"Expected 2 sh:sparql constraints (1 exclusive + 1 boolean guard), got {len(sparql_nodes)}" + ) + + queries = [str(list(g.objects(n, SH.select))[0]) for n in sparql_nodes] + # One should have FILTER(?other != ...) pattern, the other BOUND pattern + has_exclusive = any("?other" in q for q in queries) + has_boolean = any("BOUND" in q for q in queries) + assert has_exclusive, "Expected one exclusive-value SPARQL constraint" + assert has_boolean, "Expected one boolean-guard SPARQL constraint" + + +def test_any_of_with_pattern(input_path): + """Test that pattern constraints inside any_of branches emit sh:pattern. + + Exercises three cases: + 1. PatternOnlyBranch: any_of with a pattern-only branch (no range) + 2. RangeWithPattern: any_of with range + pattern on the same branch + 3. MixedBranches: combination of range-only, pattern-only, and range+pattern + """ + shacl = ShaclGenerator(input_path("shaclgen/any_of_pattern.yaml"), mergeimports=True).serialize() + g = rdflib.Graph() + g.parse(data=shacl) + + def get_or_branch_nodes(class_uri: str, slot_local: str) -> list[rdflib.BNode]: + """Return the list of BNodes inside sh:or for a given class property.""" + class_ref = URIRef(class_uri) + for prop_node in g.objects(class_ref, SH.property): + paths = list(g.objects(prop_node, SH.path)) + if any(slot_local in str(p) for p in paths): + for or_head in g.objects(prop_node, SH["or"]): + return list(Collection(g, or_head)) + return [] + + prefix = "https://w3id.org/linkml/examples/any_of_pattern/" + + # Case 1: PatternOnlyBranch — license slot has 3 branches: + # [enum sh:in], [sh:nodeKind sh:IRI], [sh:pattern "^LicenseRef-..."] + branches = get_or_branch_nodes(f"{prefix}PatternOnlyBranch", "license") + assert len(branches) == 3, f"Expected 3 branches, got {len(branches)}" + # Find the branch with sh:pattern + pattern_branches = [b for b in branches if list(g.objects(b, SH.pattern))] + assert len(pattern_branches) == 1, f"Expected 1 pattern branch, got {len(pattern_branches)}" + pattern_val = str(list(g.objects(pattern_branches[0], SH.pattern))[0]) + assert pattern_val == "^LicenseRef-[a-zA-Z0-9\\-\\.]+$" + # The pattern-only branch should NOT have sh:datatype or sh:class + assert list(g.objects(pattern_branches[0], SH.datatype)) == [] + assert list(g.objects(pattern_branches[0], SH["class"])) == [] + + # Case 2: RangeWithPattern — identifier slot has 2 branches: + # [sh:datatype xsd:string + sh:pattern "^[A-Z]{2}-[0-9]{4}$"], [sh:datatype xsd:integer] + branches = get_or_branch_nodes(f"{prefix}RangeWithPattern", "identifier") + assert len(branches) == 2, f"Expected 2 branches, got {len(branches)}" + # Find branch with both datatype and pattern + combo_branches = [b for b in branches if list(g.objects(b, SH.datatype)) and list(g.objects(b, SH.pattern))] + assert len(combo_branches) == 1, f"Expected 1 combo branch, got {len(combo_branches)}" + assert str(list(g.objects(combo_branches[0], SH.pattern))[0]) == "^[A-Z]{2}-[0-9]{4}$" + # The other branch (integer) should NOT have sh:pattern + int_branches = [b for b in branches if b not in combo_branches] + assert list(g.objects(int_branches[0], SH.pattern)) == [] + + # Case 3: MixedBranches — code slot has 3 branches: + # [sh:datatype xsd:integer], [sh:pattern "^CUSTOM-.*$"], [sh:datatype xsd:string + sh:pattern "^STD-[0-9]+$"] + branches = get_or_branch_nodes(f"{prefix}MixedBranches", "code") + assert len(branches) == 3, f"Expected 3 branches, got {len(branches)}" + # Exactly 2 branches should have sh:pattern + pattern_branches = [b for b in branches if list(g.objects(b, SH.pattern))] + assert len(pattern_branches) == 2, f"Expected 2 pattern branches, got {len(pattern_branches)}" + # Collect the patterns + patterns = sorted(str(list(g.objects(b, SH.pattern))[0]) for b in pattern_branches) + assert patterns == ["^CUSTOM-.*$", "^STD-[0-9]+$"] + # The integer-only branch should have no pattern + no_pattern = [b for b in branches if not list(g.objects(b, SH.pattern))] + assert len(no_pattern) == 1 + assert list(g.objects(no_pattern[0], SH.datatype)) == [URIRef("http://www.w3.org/2001/XMLSchema#integer")] + + +# =========================================================================== +# Presence-implies-value pattern tests (enum guard) +# =========================================================================== +# +# The "presence implies value" pattern generalises the boolean guard to +# enum-valued targets. It translates a LinkML rule where: +# - preconditions: a value slot has value_presence: PRESENT +# - postconditions: a target slot has equals_string (single required value) +# or equals_string_in (a set of acceptable values) +# +# Semantics: "If the value slot is present, the target slot must be present +# and hold one of the allowed values." The motivating use case is the aiSim +# environment model, e.g. "if texture_sky_color is set, sky_model must be +# TextureSky" and "if overcast_sky_illuminance is set, sky_model must be an +# overcast model". +# +# References: +# - W3C SHACL §5 +# - W3C SHACL §5.3.1 +# =========================================================================== + +_PRESENCE_IMPLIES_VALUE_SCHEMA_YAML = """ +id: https://example.org/presence-implies-value +name: presence_implies_value_rules +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/presence-implies-value/ +imports: + - linkml:types +default_prefix: ex +default_range: string + +enums: + SkyModelEnum: + permissible_values: + ClearSky: + meaning: ex:ClearSky + OvercastSky: + meaning: ex:OvercastSky + MeasuredOvercastSky: + meaning: ex:MeasuredOvercastSky + TextureSky: + meaning: ex:TextureSky + + ModeEnum: + permissible_values: + Auto: + description: Automatic mode (no meaning IRI). + Manual: + description: Manual mode (no meaning IRI). + +slots: + sky_model: + range: SkyModelEnum + slot_uri: ex:sky_model + texture_sky_color: + range: string + slot_uri: ex:texture_sky_color + overcast_sky_illuminance: + range: float + slot_uri: ex:overcast_sky_illuminance + mode: + range: ModeEnum + slot_uri: ex:mode + manual_value: + range: decimal + slot_uri: ex:manual_value + +classes: + Weather: + class_uri: ex:Weather + slots: + - sky_model + - texture_sky_color + - overcast_sky_illuminance + rules: + - description: If texture_sky_color is provided, sky_model must be TextureSky. + preconditions: + slot_conditions: + texture_sky_color: + value_presence: PRESENT + postconditions: + slot_conditions: + sky_model: + equals_string: "TextureSky" + - description: If overcast_sky_illuminance is provided, sky_model must be an overcast model. + preconditions: + slot_conditions: + overcast_sky_illuminance: + value_presence: PRESENT + postconditions: + slot_conditions: + sky_model: + equals_string_in: + - OvercastSky + - MeasuredOvercastSky + + Device: + class_uri: ex:Device + slots: + - mode + - manual_value + rules: + - description: If manual_value is provided, mode must be Manual (literal fallback). + preconditions: + slot_conditions: + manual_value: + value_presence: PRESENT + postconditions: + slot_conditions: + mode: + equals_string: "Manual" +""" + +EX_PIV = rdflib.Namespace("https://example.org/presence-implies-value/") + + +def test_presence_implies_value_generates_sparql(): + """Presence-implies-value rules produce sh:sparql constraints on the NodeShape.""" + g = _parse_shacl(_PRESENCE_IMPLIES_VALUE_SCHEMA_YAML) + + shape = EX_PIV.Weather + sparql_nodes = list(g.objects(shape, SH.sparql)) + assert len(sparql_nodes) == 2, f"Expected 2 sh:sparql constraints, got {len(sparql_nodes)}" + + for node in sparql_nodes: + assert (node, RDF.type, SH.SPARQLConstraint) in g + selects = list(g.objects(node, SH.select)) + assert len(selects) == 1, "Each constraint must have exactly one sh:select" + query = str(selects[0]) + assert "$this" in query, "SPARQL must use $this pre-bound variable" + assert "NOT IN" in query, "presence-implies-value SPARQL must use NOT IN membership test" + assert "FILTER" in query, "SPARQL must have a FILTER clause" + + +def test_presence_implies_value_single_uses_enum_iri(): + """A single equals_string target resolves to the enum meaning IRI.""" + g = _parse_shacl(_PRESENCE_IMPLIES_VALUE_SCHEMA_YAML) + + shape = EX_PIV.Weather + sparql_nodes = list(g.objects(shape, SH.sparql)) + queries = [str(list(g.objects(n, SH.select))[0]) for n in sparql_nodes] + + texture_query = [q for q in queries if "texture_sky_color" in q] + assert len(texture_query) == 1, "Expected exactly one texture_sky_color rule" + query = texture_query[0] + + # value slot and target slot URIs both present + assert str(EX_PIV.texture_sky_color) in query + assert str(EX_PIV.sky_model) in query + # target value resolves to the TextureSky meaning IRI in angle brackets + assert f"<{EX_PIV.TextureSky}>" in query, f"Expected TextureSky IRI, got:\n{query}" + + +def test_presence_implies_value_set_uses_all_iris(): + """equals_string_in resolves every allowed value to its enum meaning IRI.""" + g = _parse_shacl(_PRESENCE_IMPLIES_VALUE_SCHEMA_YAML) + + shape = EX_PIV.Weather + sparql_nodes = list(g.objects(shape, SH.sparql)) + queries = [str(list(g.objects(n, SH.select))[0]) for n in sparql_nodes] + + overcast_query = [q for q in queries if "overcast_sky_illuminance" in q] + assert len(overcast_query) == 1, "Expected exactly one overcast rule" + query = overcast_query[0] + + assert f"<{EX_PIV.OvercastSky}>" in query, f"Expected OvercastSky IRI, got:\n{query}" + assert f"<{EX_PIV.MeasuredOvercastSky}>" in query, f"Expected MeasuredOvercastSky IRI, got:\n{query}" + + +def test_presence_implies_value_no_meaning_falls_back_to_literal(): + """When the target enum value lacks a meaning IRI, it is compared as a literal.""" + g = _parse_shacl(_PRESENCE_IMPLIES_VALUE_SCHEMA_YAML) + + shape = EX_PIV.Device + sparql_nodes = list(g.objects(shape, SH.sparql)) + assert len(sparql_nodes) == 1 + + query = str(list(g.objects(sparql_nodes[0], SH.select))[0]) + assert '"Manual"' in query, f"No-meaning enum should use literal '\"Manual\"', got:\n{query}" + assert f"<{EX_PIV}Manual>" not in query, "Should not emit as IRI when meaning is absent" + + +def test_presence_implies_value_message_from_description(): + """Rule description is emitted as sh:message on the SPARQLConstraint.""" + g = _parse_shacl(_PRESENCE_IMPLIES_VALUE_SCHEMA_YAML) + + shape = EX_PIV.Weather + sparql_nodes = list(g.objects(shape, SH.sparql)) + messages = [str(m) for node in sparql_nodes for m in g.objects(node, SH.message)] + + assert any("sky_model must be TextureSky" in m for m in messages), ( + f"Expected message about TextureSky, got: {messages}" + ) + + +def test_presence_implies_value_sparql_syntax_valid(): + """Generated SPARQL for presence-implies-value rules must be syntactically valid.""" + from rdflib.plugins.sparql import prepareQuery + + g = _parse_shacl(_PRESENCE_IMPLIES_VALUE_SCHEMA_YAML) + + for shape in (EX_PIV.Weather, EX_PIV.Device): + sparql_nodes = list(g.objects(shape, SH.sparql)) + for node in sparql_nodes: + query_text = str(list(g.objects(node, SH.select))[0]) + prepareQuery(query_text) + + +def test_presence_implies_value_pyshacl_end_to_end(): + """End-to-end: pyshacl passes conforming instances and flags violations.""" + import pyshacl + + shacl_ttl = ShaclGenerator(_PRESENCE_IMPLIES_VALUE_SCHEMA_YAML, mergeimports=False, emit_rules=True).serialize() + + # Conforming: guarded slots paired with an allowed sky_model; and an + # unguarded instance (no texture/overcast) is unaffected by the rules. + conforming_data = """ + @prefix ex: . + @prefix xsd: . + + ex:wTexture a ex:Weather ; + ex:texture_sky_color "0,0,0" ; + ex:sky_model ex:TextureSky . + + ex:wOvercast a ex:Weather ; + ex:overcast_sky_illuminance "5000.0"^^xsd:float ; + ex:sky_model ex:OvercastSky . + + ex:wMeasured a ex:Weather ; + ex:overcast_sky_illuminance "4200.0"^^xsd:float ; + ex:sky_model ex:MeasuredOvercastSky . + + ex:wClear a ex:Weather ; + ex:sky_model ex:ClearSky . + """ + + conforms, _, results_text = pyshacl.validate( + data_graph=conforming_data, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert conforms, f"Conforming instances should pass SHACL validation:\n{results_text}" + + # Violating: texture_sky_color present but sky_model is ClearSky (not TextureSky). + violating_wrong_value = """ + @prefix ex: . + + ex:wBad a ex:Weather ; + ex:texture_sky_color "0,0,0" ; + ex:sky_model ex:ClearSky . + """ + conforms, _, results_text = pyshacl.validate( + data_graph=violating_wrong_value, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert not conforms, f"Wrong-value instance should fail SHACL validation:\n{results_text}" + + # Violating: overcast_sky_illuminance present but sky_model is TextureSky + # (not in the allowed overcast set). + violating_not_in_set = """ + @prefix ex: . + @prefix xsd: . + + ex:wBad2 a ex:Weather ; + ex:overcast_sky_illuminance "5000.0"^^xsd:float ; + ex:sky_model ex:TextureSky . + """ + conforms, _, results_text = pyshacl.validate( + data_graph=violating_not_in_set, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert not conforms, f"Not-in-set instance should fail SHACL validation:\n{results_text}" + + # Violating: texture_sky_color present but sky_model entirely absent. + violating_missing_target = """ + @prefix ex: . + + ex:wBad3 a ex:Weather ; + ex:texture_sky_color "0,0,0" . + """ + conforms, _, results_text = pyshacl.validate( + data_graph=violating_missing_target, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert not conforms, f"Missing-target instance should fail SHACL validation:\n{results_text}" + + +# =========================================================================== +# Compositional fallback: conditional-required pattern (M1) +# =========================================================================== +# +# Rule shape: +# - preconditions: slot X has equals_string V +# - postconditions: slot Y has required: true +# +# Semantics: "If X = V, then Y must be present." Emitted as an +# sh:SPARQLConstraint whose SELECT matches focus nodes where the precondition +# holds but the required slot is absent (FILTER NOT EXISTS). +# =========================================================================== + +_CONDITIONAL_REQUIRED_SCHEMA_YAML = """ +id: https://example.org/conditional-required +name: conditional_required_rules +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/conditional-required/ +imports: + - linkml:types +default_prefix: ex +default_range: string + +enums: + SkyModelEnum: + permissible_values: + ClearSky: + meaning: ex:ClearSky + OvercastSky: + meaning: ex:OvercastSky + MeasuredOvercastSky: + meaning: ex:MeasuredOvercastSky + +slots: + sky_model: + range: SkyModelEnum + slot_uri: ex:sky_model + overcast_sky_illuminance: + range: float + slot_uri: ex:overcast_sky_illuminance + +classes: + Weather: + class_uri: ex:Weather + slots: + - sky_model + - overcast_sky_illuminance + rules: + - description: The MeasuredOvercastSky model requires the sky illuminance. + preconditions: + slot_conditions: + sky_model: + equals_string: MeasuredOvercastSky + postconditions: + slot_conditions: + overcast_sky_illuminance: + required: true +""" + +EX_CR = rdflib.Namespace("https://example.org/conditional-required/") + + +def test_conditional_required_generates_sparql(): + """equals_string precondition + required postcondition → one sh:sparql constraint.""" + g = _parse_shacl(_CONDITIONAL_REQUIRED_SCHEMA_YAML) + + shape = EX_CR.Weather + sparql_nodes = list(g.objects(shape, SH.sparql)) + assert len(sparql_nodes) == 1, f"Expected 1 sh:sparql constraint, got {len(sparql_nodes)}" + + node = sparql_nodes[0] + assert (node, RDF.type, SH.SPARQLConstraint) in g + query = str(list(g.objects(node, SH.select))[0]) + + assert "$this" in query, "SPARQL must use $this pre-bound variable (SHACL §5.3.1)" + assert "FILTER NOT EXISTS" in query, "required violation must use FILTER NOT EXISTS" + # precondition references the enum meaning IRI and the trigger slot + assert f"<{EX_CR.MeasuredOvercastSky}>" in query, f"precondition must use the enum IRI, got:\n{query}" + assert str(EX_CR.sky_model) in query + assert str(EX_CR.overcast_sky_illuminance) in query + + +def test_conditional_required_message_from_description(): + """Rule description is emitted as sh:message.""" + g = _parse_shacl(_CONDITIONAL_REQUIRED_SCHEMA_YAML) + messages = [str(m) for node in g.objects(EX_CR.Weather, SH.sparql) for m in g.objects(node, SH.message)] + assert any("requires the sky illuminance" in m for m in messages), messages + + +def test_conditional_required_sparql_syntax_valid(): + """Generated SPARQL must be syntactically valid.""" + from rdflib.plugins.sparql import prepareQuery + + g = _parse_shacl(_CONDITIONAL_REQUIRED_SCHEMA_YAML) + for node in g.objects(EX_CR.Weather, SH.sparql): + prepareQuery(str(list(g.objects(node, SH.select))[0])) + + +def test_conditional_required_pyshacl_end_to_end(): + """End-to-end: pyshacl passes conforming instances and flags the violation.""" + import pyshacl + + shacl_ttl = ShaclGenerator(_CONDITIONAL_REQUIRED_SCHEMA_YAML, mergeimports=False, emit_rules=True).serialize() + + # Conforming: MeasuredOvercastSky WITH illuminance; ClearSky needs nothing. + conforming = """ + @prefix ex: . + @prefix xsd: . + + ex:wMeasured a ex:Weather ; + ex:sky_model ex:MeasuredOvercastSky ; + ex:overcast_sky_illuminance "4200.0"^^xsd:float . + + ex:wClear a ex:Weather ; + ex:sky_model ex:ClearSky . + """ + conforms, _, txt = pyshacl.validate( + data_graph=conforming, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert conforms, f"Conforming instances should pass:\n{txt}" + + # Violating: MeasuredOvercastSky WITHOUT the required illuminance. + violating = """ + @prefix ex: . + + ex:wBad a ex:Weather ; + ex:sky_model ex:MeasuredOvercastSky . + """ + conforms, _, txt = pyshacl.validate( + data_graph=violating, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert not conforms, f"MeasuredOvercastSky without illuminance should fail:\n{txt}" + + +# =========================================================================== +# Compositional fallback: conditional-absent pattern (M2) +# =========================================================================== +# +# Rule shape: +# - preconditions: slot X has equals_string V +# - postconditions: slot Y has value_presence: ABSENT +# +# Semantics: "If X = V, then Y must NOT be present" (inapplicable slot). +# Emitted as an sh:SPARQLConstraint whose SELECT matches focus nodes where the +# precondition holds and the forbidden slot is present. +# =========================================================================== + +_CONDITIONAL_ABSENT_SCHEMA_YAML = """ +id: https://example.org/conditional-absent +name: conditional_absent_rules +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/conditional-absent/ +imports: + - linkml:types +default_prefix: ex +default_range: string + +enums: + SkyModelEnum: + permissible_values: + ClearSky: + meaning: ex:ClearSky + OvercastSky: + meaning: ex:OvercastSky + +slots: + sky_model: + range: SkyModelEnum + slot_uri: ex:sky_model + overcast_sky_illuminance: + range: float + slot_uri: ex:overcast_sky_illuminance + +classes: + Weather: + class_uri: ex:Weather + slots: + - sky_model + - overcast_sky_illuminance + rules: + - description: ClearSky makes overcast_sky_illuminance inapplicable. + preconditions: + slot_conditions: + sky_model: + equals_string: ClearSky + postconditions: + slot_conditions: + overcast_sky_illuminance: + value_presence: ABSENT +""" + +EX_CA = rdflib.Namespace("https://example.org/conditional-absent/") + + +def test_conditional_absent_generates_sparql(): + """equals_string precondition + value_presence ABSENT → one sh:sparql constraint.""" + g = _parse_shacl(_CONDITIONAL_ABSENT_SCHEMA_YAML) + + sparql_nodes = list(g.objects(EX_CA.Weather, SH.sparql)) + assert len(sparql_nodes) == 1, f"Expected 1 sh:sparql constraint, got {len(sparql_nodes)}" + + query = str(list(g.objects(sparql_nodes[0], SH.select))[0]) + assert "$this" in query + # violation = precondition holds AND the forbidden slot is present; the + # forbidden-slot triple must NOT be wrapped in NOT EXISTS. + assert "FILTER NOT EXISTS" not in query, f"conditional-absent must not use NOT EXISTS, got:\n{query}" + assert f"<{EX_CA.ClearSky}>" in query + assert str(EX_CA.overcast_sky_illuminance) in query + + +def test_conditional_absent_sparql_syntax_valid(): + """Generated SPARQL must be syntactically valid.""" + from rdflib.plugins.sparql import prepareQuery + + g = _parse_shacl(_CONDITIONAL_ABSENT_SCHEMA_YAML) + for node in g.objects(EX_CA.Weather, SH.sparql): + prepareQuery(str(list(g.objects(node, SH.select))[0])) + + +def test_conditional_absent_pyshacl_end_to_end(): + """End-to-end: pyshacl passes conforming instances and flags the violation.""" + import pyshacl + + shacl_ttl = ShaclGenerator(_CONDITIONAL_ABSENT_SCHEMA_YAML, mergeimports=False, emit_rules=True).serialize() + + # Conforming: ClearSky without illuminance; OvercastSky may set illuminance. + conforming = """ + @prefix ex: . + @prefix xsd: . + + ex:wClear a ex:Weather ; + ex:sky_model ex:ClearSky . + + ex:wOvercast a ex:Weather ; + ex:sky_model ex:OvercastSky ; + ex:overcast_sky_illuminance "5000.0"^^xsd:float . + """ + conforms, _, txt = pyshacl.validate( + data_graph=conforming, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert conforms, f"Conforming instances should pass:\n{txt}" + + # Violating: ClearSky WITH the inapplicable illuminance. + violating = """ + @prefix ex: . + @prefix xsd: . + + ex:wBad a ex:Weather ; + ex:sky_model ex:ClearSky ; + ex:overcast_sky_illuminance "5000.0"^^xsd:float . + """ + conforms, _, txt = pyshacl.validate( + data_graph=violating, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert not conforms, f"ClearSky with illuminance should fail:\n{txt}" + + +# =========================================================================== +# Compositional fallback: numeric threshold precondition (M3) +# =========================================================================== +# +# Rule shape: +# - preconditions: slot X has maximum_value N (or minimum_value) +# - postconditions: slot Y has required: true +# +# Semantics: "If X <= N, then Y must be present." The threshold becomes a +# SPARQL FILTER; combined here with the M1 required violation. +# =========================================================================== + +_THRESHOLD_SCHEMA_YAML = """ +id: https://example.org/threshold +name: threshold_rules +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/threshold/ +imports: + - linkml:types +default_prefix: ex +default_range: string + +slots: + meteorological_optical_range: + range: float + slot_uri: ex:meteorological_optical_range + fog_note: + range: string + slot_uri: ex:fog_note + +classes: + Weather: + class_uri: ex:Weather + slots: + - meteorological_optical_range + - fog_note + rules: + - description: In fog (optical range at or below 4000) a fog note is required. + preconditions: + slot_conditions: + meteorological_optical_range: + maximum_value: 4000 + postconditions: + slot_conditions: + fog_note: + required: true +""" + +EX_THR = rdflib.Namespace("https://example.org/threshold/") + + +def test_threshold_precondition_generates_sparql(): + """maximum_value precondition emits a numeric FILTER on the trigger slot.""" + g = _parse_shacl(_THRESHOLD_SCHEMA_YAML) + + sparql_nodes = list(g.objects(EX_THR.Weather, SH.sparql)) + assert len(sparql_nodes) == 1, f"Expected 1 sh:sparql constraint, got {len(sparql_nodes)}" + + query = str(list(g.objects(sparql_nodes[0], SH.select))[0]) + assert "<= 4000" in query, f"threshold must emit '<= 4000', got:\n{query}" + assert "FILTER NOT EXISTS" in query, "required postcondition violation must use NOT EXISTS" + assert str(EX_THR.meteorological_optical_range) in query + assert str(EX_THR.fog_note) in query + + +def test_threshold_precondition_sparql_syntax_valid(): + """Generated SPARQL must be syntactically valid.""" + from rdflib.plugins.sparql import prepareQuery + + g = _parse_shacl(_THRESHOLD_SCHEMA_YAML) + for node in g.objects(EX_THR.Weather, SH.sparql): + prepareQuery(str(list(g.objects(node, SH.select))[0])) + + +def test_threshold_precondition_pyshacl_end_to_end(): + """End-to-end: below-threshold requires the note; above-threshold does not.""" + import pyshacl + + shacl_ttl = ShaclGenerator(_THRESHOLD_SCHEMA_YAML, mergeimports=False, emit_rules=True).serialize() + + # Conforming: foggy (400) with a note; clear (5000) needs nothing. + conforming = """ + @prefix ex: . + @prefix xsd: . + + ex:wFog a ex:Weather ; + ex:meteorological_optical_range "400.0"^^xsd:float ; + ex:fog_note "reduced visibility" . + + ex:wClear a ex:Weather ; + ex:meteorological_optical_range "5000.0"^^xsd:float . + """ + conforms, _, txt = pyshacl.validate( + data_graph=conforming, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert conforms, f"Conforming instances should pass:\n{txt}" + + # Violating: foggy (400) without the required note. + violating = """ + @prefix ex: . + @prefix xsd: . + + ex:wBad a ex:Weather ; + ex:meteorological_optical_range "400.0"^^xsd:float . + """ + conforms, _, txt = pyshacl.validate( + data_graph=violating, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert not conforms, f"Fog without the required note should fail:\n{txt}" + + +# =========================================================================== +# Compositional fallback: nested range_expression precondition (M4) +# =========================================================================== +# +# Rule shape: +# - preconditions: slot X (inlined child) has range_expression on an inner +# slot (e.g. sun_position.elevation <= 0) +# - postconditions: slot Y has required: true +# +# Semantics: "If the child's inner value satisfies the condition, then Y must +# be present." The SPARQL binds the child node with one extra hop. +# =========================================================================== + +_NESTED_SCHEMA_YAML = """ +id: https://example.org/nested +name: nested_rules +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/nested/ +imports: + - linkml:types +default_prefix: ex +default_range: string + +slots: + sun_position: + range: SunPosition + inlined: true + slot_uri: ex:sun_position + elevation: + range: float + slot_uri: ex:elevation + headlight_note: + range: string + slot_uri: ex:headlight_note + +classes: + SunPosition: + class_uri: ex:SunPosition + slots: + - elevation + Weather: + class_uri: ex:Weather + slots: + - sun_position + - headlight_note + rules: + - description: When the sun is at or below the horizon a headlight note is required. + preconditions: + slot_conditions: + sun_position: + range_expression: + slot_conditions: + elevation: + maximum_value: 0.0 + postconditions: + slot_conditions: + headlight_note: + required: true +""" + +EX_NEST = rdflib.Namespace("https://example.org/nested/") + + +def test_nested_precondition_generates_sparql(): + """A nested range_expression precondition emits a two-hop graph pattern.""" + g = _parse_shacl(_NESTED_SCHEMA_YAML) + + sparql_nodes = list(g.objects(EX_NEST.Weather, SH.sparql)) + assert len(sparql_nodes) == 1, f"Expected 1 sh:sparql constraint, got {len(sparql_nodes)}" + + query = str(list(g.objects(sparql_nodes[0], SH.select))[0]) + assert str(EX_NEST.sun_position) in query, "must traverse the container slot" + assert str(EX_NEST.elevation) in query, "must traverse the inner slot" + assert "<= 0.0" in query, f"inner threshold must appear, got:\n{query}" + assert "FILTER NOT EXISTS" in query + assert str(EX_NEST.headlight_note) in query + + +def test_nested_precondition_sparql_syntax_valid(): + """Generated SPARQL must be syntactically valid.""" + from rdflib.plugins.sparql import prepareQuery + + g = _parse_shacl(_NESTED_SCHEMA_YAML) + for node in g.objects(EX_NEST.Weather, SH.sparql): + prepareQuery(str(list(g.objects(node, SH.select))[0])) + + +def test_nested_precondition_pyshacl_end_to_end(): + """End-to-end: sun below horizon requires the note; above horizon does not.""" + import pyshacl + + shacl_ttl = ShaclGenerator(_NESTED_SCHEMA_YAML, mergeimports=False, emit_rules=True).serialize() + + # Conforming: night (elevation -90) with a note; day (45) needs nothing. + conforming = """ + @prefix ex: . + @prefix xsd: . + + ex:wNight a ex:Weather ; + ex:sun_position [ a ex:SunPosition ; ex:elevation "-90.0"^^xsd:float ] ; + ex:headlight_note "on" . + + ex:wDay a ex:Weather ; + ex:sun_position [ a ex:SunPosition ; ex:elevation "45.0"^^xsd:float ] . + """ + conforms, _, txt = pyshacl.validate( + data_graph=conforming, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert conforms, f"Conforming instances should pass:\n{txt}" + + # Violating: night (elevation -90) without the required note. + violating = """ + @prefix ex: . + @prefix xsd: . + + ex:wBad a ex:Weather ; + ex:sun_position [ a ex:SunPosition ; ex:elevation "-90.0"^^xsd:float ] . + """ + conforms, _, txt = pyshacl.validate( + data_graph=violating, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert not conforms, f"Night without a headlight note should fail:\n{txt}" + + +# =========================================================================== +# Compositional fallback: has_member list-membership postcondition (M5) +# =========================================================================== +# +# Rule shape: +# - preconditions: any supported precondition (here value_presence PRESENT) +# - postconditions: multivalued slot has_member with a nested +# range_expression constraining the member's inner slots +# +# Semantics: "If the precondition holds, the list must contain a member +# matching the inner conditions." Violation = no such member (FILTER NOT +# EXISTS over the members). Inner enum values resolve against the member +# class (LightControlGroup), which disambiguates the reused `type` slot. +# =========================================================================== + +_HAS_MEMBER_SCHEMA_YAML = """ +id: https://example.org/has-member +name: has_member_rules +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/has-member/ +imports: + - linkml:types +default_prefix: ex +default_range: string + +enums: + LightGroupEnum: + permissible_values: + Vehicle: + meaning: ex:Vehicle + StreetLight: + meaning: ex:StreetLight + LightTypeEnum: + permissible_values: + low_beam_headlight: + meaning: ex:low_beam_headlight + front_fog_light: + meaning: ex:front_fog_light + +slots: + fog_declared: + range: string + slot_uri: ex:fog_declared + enabled_light_control_groups: + range: LightControlGroup + multivalued: true + inlined: true + inlined_as_list: true + slot_uri: ex:enabled_light_control_groups + group: + range: LightGroupEnum + slot_uri: ex:group + type: + range: LightTypeEnum + slot_uri: ex:type + +classes: + LightControlGroup: + class_uri: ex:LightControlGroup + slots: + - group + - type + Weather: + class_uri: ex:Weather + slots: + - fog_declared + - enabled_light_control_groups + rules: + - description: When fog is declared, a front fog light group must be enabled. + preconditions: + slot_conditions: + fog_declared: + value_presence: PRESENT + postconditions: + slot_conditions: + enabled_light_control_groups: + has_member: + range_expression: + slot_conditions: + group: + equals_string: Vehicle + type: + equals_string: front_fog_light +""" + +EX_HM = rdflib.Namespace("https://example.org/has-member/") + + +def test_has_member_generates_sparql(): + """has_member postcondition emits a FILTER NOT EXISTS over list members.""" + g = _parse_shacl(_HAS_MEMBER_SCHEMA_YAML) + + sparql_nodes = list(g.objects(EX_HM.Weather, SH.sparql)) + assert len(sparql_nodes) == 1, f"Expected 1 sh:sparql constraint, got {len(sparql_nodes)}" + + query = str(list(g.objects(sparql_nodes[0], SH.select))[0]) + assert "FILTER NOT EXISTS" in query, "list-membership violation must use FILTER NOT EXISTS" + assert str(EX_HM.enabled_light_control_groups) in query + assert str(EX_HM.group) in query and str(EX_HM.type) in query + # inner enum values resolve against the member class (LightControlGroup), + # so the reused `type` slot picks LightTypeEnum, not another enum. + assert f"<{EX_HM.Vehicle}>" in query, f"group value must be the enum IRI, got:\n{query}" + assert f"<{EX_HM.front_fog_light}>" in query, f"type value must be the enum IRI, got:\n{query}" + + +def test_has_member_sparql_syntax_valid(): + """Generated SPARQL must be syntactically valid.""" + from rdflib.plugins.sparql import prepareQuery + + g = _parse_shacl(_HAS_MEMBER_SCHEMA_YAML) + for node in g.objects(EX_HM.Weather, SH.sparql): + prepareQuery(str(list(g.objects(node, SH.select))[0])) + + +def test_has_member_pyshacl_end_to_end(): + """End-to-end: fog requires a front-fog-light member; otherwise it fails.""" + import pyshacl + + shacl_ttl = ShaclGenerator(_HAS_MEMBER_SCHEMA_YAML, mergeimports=False, emit_rules=True).serialize() + + # Conforming: fog declared WITH a front-fog-light group; and no fog at all. + conforming = """ + @prefix ex: . + + ex:wFog a ex:Weather ; + ex:fog_declared "yes" ; + ex:enabled_light_control_groups + [ a ex:LightControlGroup ; ex:group ex:Vehicle ; ex:type ex:front_fog_light ] . + + ex:wNoFog a ex:Weather . + """ + conforms, _, txt = pyshacl.validate( + data_graph=conforming, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert conforms, f"Conforming instances should pass:\n{txt}" + + # Violating: fog declared but only a low-beam group (no front fog light). + violating = """ + @prefix ex: . + + ex:wBad a ex:Weather ; + ex:fog_declared "yes" ; + ex:enabled_light_control_groups + [ a ex:LightControlGroup ; ex:group ex:Vehicle ; ex:type ex:low_beam_headlight ] . + """ + conforms, _, txt = pyshacl.validate( + data_graph=violating, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert not conforms, f"Fog without a front-fog-light group should fail:\n{txt}" + + +# =========================================================================== +# Rule-converter robustness regressions (review hardening) +# +# These guard three defects found while reviewing the rule converters: +# 1. A single precondition combining minimum_value + maximum_value dropped +# all but the first bound (silent under-constraint / false positives). +# 2. A slot_usage `slot_uri` (or enum `range`) override made the SPARQL body +# query the *base* IRI while `sh:path` used the *induced* IRI, so the +# constraint silently never fired (false negative). +# 3. An `equals_string` value containing a quote/backslash produced invalid, +# unparsable SPARQL (broken artifact / injection). +# =========================================================================== + +_COMBINED_BOUNDS_SCHEMA_YAML = """ +id: https://example.org/combined-bounds +name: combined_bounds_rules +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/combined-bounds/ +imports: + - linkml:types +default_prefix: ex +default_range: string + +slots: + reading_value: + range: integer + slot_uri: ex:reading_value + reading_note: + range: string + slot_uri: ex:reading_note + +classes: + Reading: + class_uri: ex:Reading + slots: + - reading_value + - reading_note + rules: + - description: A mid-range reading requires an explanatory note. + preconditions: + slot_conditions: + reading_value: + minimum_value: 10 + maximum_value: 20 + postconditions: + slot_conditions: + reading_note: + required: true +""" + +EX_CB = rdflib.Namespace("https://example.org/combined-bounds/") + + +def test_rule_precondition_combines_min_and_max_bounds(): + """A precondition with both minimum_value and maximum_value must emit both + bounds; the pre-fix first-match dispatch kept only the maximum.""" + g = _parse_shacl(_COMBINED_BOUNDS_SCHEMA_YAML) + + nodes = list(g.objects(EX_CB.Reading, SH.sparql)) + assert len(nodes) == 1, f"Expected 1 sh:sparql constraint, got {len(nodes)}" + query = str(list(g.objects(nodes[0], SH.select))[0]) + assert ">= 10" in query, f"lower bound must be emitted, got:\n{query}" + assert "<= 20" in query, f"upper bound must be emitted, got:\n{query}" + + +def test_rule_combined_bounds_pyshacl_end_to_end(): + """End-to-end: only values inside [10, 20] trigger the required note. + + The below-threshold case is the key assertion — without the lower bound it + would be flagged as a violation.""" + import pyshacl + + shacl_ttl = ShaclGenerator(_COMBINED_BOUNDS_SCHEMA_YAML, mergeimports=False, emit_rules=True).serialize() + + conforming = """ + @prefix ex: . + @prefix xsd: . + + ex:mid a ex:Reading ; ex:reading_value 15 ; ex:reading_note "in range" . + ex:low a ex:Reading ; ex:reading_value 5 . + ex:high a ex:Reading ; ex:reading_value 25 . + """ + conforms, _, txt = pyshacl.validate( + data_graph=conforming, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert conforms, f"Out-of-range readings must not require a note:\n{txt}" + + violating = """ + @prefix ex: . + @prefix xsd: . + + ex:bad a ex:Reading ; ex:reading_value 15 . + """ + conforms, _, txt = pyshacl.validate( + data_graph=violating, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert not conforms, f"A mid-range reading without a note must fail:\n{txt}" + + +_SLOT_URI_OVERRIDE_SCHEMA_YAML = """ +id: https://example.org/slot-uri-override +name: slot_uri_override_rules +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/slot-uri-override/ +imports: + - linkml:types +default_prefix: ex +default_range: string + +slots: + trigger: + range: string + slot_uri: ex:GLOBAL_trigger + dependent: + range: string + slot_uri: ex:GLOBAL_dependent + +classes: + Scene: + class_uri: ex:Scene + slots: + - trigger + - dependent + slot_usage: + trigger: + slot_uri: ex:LOCAL_trigger + dependent: + slot_uri: ex:LOCAL_dependent + rules: + - description: If the trigger is present the dependent slot is required. + preconditions: + slot_conditions: + trigger: + value_presence: PRESENT + postconditions: + slot_conditions: + dependent: + required: true +""" + +EX_OVR = rdflib.Namespace("https://example.org/slot-uri-override/") + + +def test_rule_slot_uri_override_matches_sh_path(): + """The SPARQL body must use the same induced (class-local) IRIs as sh:path. + + A slot_usage slot_uri override changes sh:path; if the SPARQL keeps the base + IRI the query targets a property the data never uses and never fires.""" + g = _parse_shacl(_SLOT_URI_OVERRIDE_SCHEMA_YAML) + + paths = {str(o) for o in g.objects(None, SH.path)} + assert str(EX_OVR.LOCAL_trigger) in paths + assert str(EX_OVR.LOCAL_dependent) in paths + + nodes = list(g.objects(EX_OVR.Scene, SH.sparql)) + assert len(nodes) == 1 + query = str(list(g.objects(nodes[0], SH.select))[0]) + assert str(EX_OVR.LOCAL_trigger) in query, f"SPARQL must use the induced IRI, got:\n{query}" + assert str(EX_OVR.LOCAL_dependent) in query, f"SPARQL must use the induced IRI, got:\n{query}" + assert "GLOBAL_" not in query, f"SPARQL must not fall back to the base slot_uri, got:\n{query}" + + +def test_rule_slot_uri_override_pyshacl_end_to_end(): + """End-to-end: the constraint actually fires on data that uses the induced + (LOCAL) IRIs. Before the fix the SPARQL queried the base IRIs, so a missing + dependent slot slipped through as conforming.""" + import pyshacl + + shacl_ttl = ShaclGenerator(_SLOT_URI_OVERRIDE_SCHEMA_YAML, mergeimports=False, emit_rules=True).serialize() + + conforming = """ + @prefix ex: . + + ex:ok a ex:Scene ; ex:LOCAL_trigger "t" ; ex:LOCAL_dependent "d" . + ex:noTrigger a ex:Scene ; ex:LOCAL_dependent "d" . + """ + conforms, _, txt = pyshacl.validate( + data_graph=conforming, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert conforms, f"Trigger-with-dependent (and no-trigger) must pass:\n{txt}" + + violating = """ + @prefix ex: . + + ex:bad a ex:Scene ; ex:LOCAL_trigger "t" . + """ + conforms, _, txt = pyshacl.validate( + data_graph=violating, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert not conforms, f"Trigger present without the required dependent must fail:\n{txt}" + + +_ENUM_NARROWING_SCHEMA_YAML = """ +id: https://example.org/enum-narrowing +name: enum_narrowing_rules +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/enum-narrowing/ +imports: + - linkml:types +default_prefix: ex +default_range: string + +enums: + BaseMode: + permissible_values: + Active: + meaning: ex:GLOBAL_Active + SceneMode: + permissible_values: + Active: + meaning: ex:LOCAL_Active + +slots: + activator: + range: string + slot_uri: ex:activator + mode: + range: BaseMode + slot_uri: ex:mode + +classes: + Scene: + class_uri: ex:Scene + slots: + - activator + - mode + slot_usage: + mode: + range: SceneMode + rules: + - description: If an activator is present the mode must be Active. + preconditions: + slot_conditions: + activator: + value_presence: PRESENT + postconditions: + slot_conditions: + mode: + equals_string: Active +""" + +EX_EN = rdflib.Namespace("https://example.org/enum-narrowing/") + + +def test_rule_enum_range_narrowed_by_slot_usage(): + """A slot_usage range override to a class-specific enum must resolve the + value's meaning against the induced (narrowed) enum, not the base range.""" + g = _parse_shacl(_ENUM_NARROWING_SCHEMA_YAML) + + nodes = list(g.objects(EX_EN.Scene, SH.sparql)) + assert len(nodes) == 1 + query = str(list(g.objects(nodes[0], SH.select))[0]) + assert str(EX_EN.LOCAL_Active) in query, f"must resolve the narrowed enum meaning, got:\n{query}" + assert "GLOBAL_Active" not in query, f"must not resolve the base enum meaning, got:\n{query}" + + +_ESCAPING_SCHEMA_YAML = """ +id: https://example.org/escaping +name: escaping_rules +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/escaping/ +imports: + - linkml:types +default_prefix: ex +default_range: string + +slots: + trigger: + range: string + slot_uri: ex:trigger + label: + range: string + slot_uri: ex:label + +classes: + Item: + class_uri: ex:Item + slots: + - trigger + - label + rules: + - description: If a trigger is present the label must equal the quoted marker. + preconditions: + slot_conditions: + trigger: + value_presence: PRESENT + postconditions: + slot_conditions: + label: + equals_string: 'a"b\\\\c' +""" + +EX_ESC = rdflib.Namespace("https://example.org/escaping/") + + +def test_rule_equals_string_special_chars_escaped(): + """An equals_string value with a quote and backslash must be escaped so the + generated SPARQL stays syntactically valid (no injection / broken query).""" + from rdflib.plugins.sparql import prepareQuery + + g = _parse_shacl(_ESCAPING_SCHEMA_YAML) + nodes = list(g.objects(EX_ESC.Item, SH.sparql)) + assert len(nodes) == 1 + query = str(list(g.objects(nodes[0], SH.select))[0]) + + # Would raise ParseException on the unescaped `... = "a"b\c"` form. + prepareQuery(query) + assert '\\"' in query, f"double quote must be escaped, got:\n{query}" + assert "\\\\" in query, f"backslash must be escaped, got:\n{query}" + + +# =========================================================================== +# Audit-fix regression tests: operator exactness, nested-slot resolution, +# numeric bound gating, elseconditions warning +# =========================================================================== + +_PIV_EXTRA_PRE_SCHEMA_YAML = """ +id: https://example.org/piv-extra-pre +name: piv_extra_pre +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/piv-extra-pre/ +imports: + - linkml:types +default_prefix: ex +default_range: string +slots: + temp: + range: integer + slot_uri: ex:temp + mode: + range: string + slot_uri: ex:mode +classes: + Device: + class_uri: ex:Device + slots: [temp, mode] + rules: + - description: Above 100 the mode must be High (extra precondition operator). + preconditions: + slot_conditions: + temp: + value_presence: PRESENT + minimum_value: 100 + postconditions: + slot_conditions: + mode: + equals_string: "High" +""" + + +def test_rule_extra_precondition_operator_skipped(): + """A precondition combining PRESENT with a threshold must not dispatch to + presence-implies-value: dropping the threshold widens the trigger.""" + g = _parse_shacl(_PIV_EXTRA_PRE_SCHEMA_YAML) + shape = URIRef("https://example.org/piv-extra-pre/Device") + assert list(g.objects(shape, SH.sparql)) == [], "rule with an untranslated conjunct must be skipped" + + +def test_rule_extra_precondition_operator_pyshacl_end_to_end(): + """A device below the threshold satisfies the rule vacuously and must conform.""" + import pyshacl + + shacl_ttl = ShaclGenerator(_PIV_EXTRA_PRE_SCHEMA_YAML, mergeimports=False, emit_rules=True).serialize() + data = """ + @prefix ex: . + + ex:cool a ex:Device ; ex:temp 50 ; ex:mode "Low" . + """ + conforms, _, txt = pyshacl.validate( + data_graph=data, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert conforms, f"Below-threshold device must not be flagged:\n{txt}" + + +_POST_BOTH_EQUALS_SCHEMA_YAML = """ +id: https://example.org/post-both-equals +name: post_both_equals +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/post-both-equals/ +imports: + - linkml:types +default_prefix: ex +default_range: string +slots: + guard: + slot_uri: ex:guard + target: + slot_uri: ex:target +classes: + Thing: + class_uri: ex:Thing + slots: [guard, target] + rules: + - preconditions: + slot_conditions: + guard: + value_presence: PRESENT + postconditions: + slot_conditions: + target: + equals_string: "a" + equals_string_in: ["b", "c"] +""" + + +def test_rule_post_with_both_equals_forms_skipped(): + """equals_string and equals_string_in set together is ambiguous — skip, + do not let one form silently win.""" + g = _parse_shacl(_POST_BOTH_EQUALS_SCHEMA_YAML) + shape = URIRef("https://example.org/post-both-equals/Thing") + assert list(g.objects(shape, SH.sparql)) == [] + + +_MIXED_SCALAR_SCHEMA_YAML = """ +id: https://example.org/mixed-scalar +name: mixed_scalar +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/mixed-scalar/ +imports: + - linkml:types +default_prefix: ex +default_range: string +slots: + code: + slot_uri: ex:code + note: + slot_uri: ex:note +classes: + Obs: + class_uri: ex:Obs + slots: [code, note] + rules: + - preconditions: + slot_conditions: + code: + equals_string: fog + pattern: "^f" + postconditions: + slot_conditions: + note: + required: true +""" + + +def test_rule_recognized_plus_unrecognized_operator_skipped(): + """A condition mixing a supported operator (equals_string) with an + unsupported one (pattern) must skip — translating only the supported part + widens the trigger.""" + g = _parse_shacl(_MIXED_SCALAR_SCHEMA_YAML) + shape = URIRef("https://example.org/mixed-scalar/Obs") + assert list(g.objects(shape, SH.sparql)) == [] + + +_EXPR_ANY_OF_SCHEMA_YAML = """ +id: https://example.org/expr-any-of +name: expr_any_of +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/expr-any-of/ +imports: + - linkml:types +default_prefix: ex +default_range: string +slots: + code: + slot_uri: ex:code + other: + slot_uri: ex:other + note: + slot_uri: ex:note +classes: + Obs: + class_uri: ex:Obs + slots: [code, other, note] + rules: + - preconditions: + slot_conditions: + code: + equals_string: fog + any_of: + - slot_conditions: + other: + equals_string: x + - slot_conditions: + other: + equals_string: y + postconditions: + slot_conditions: + note: + required: true +""" + + +def test_rule_expression_level_any_of_skipped(): + """Expression-level any_of on the preconditions cannot be honoured by any + converter; dropping the branch widens the trigger, so the rule is skipped.""" + g = _parse_shacl(_EXPR_ANY_OF_SCHEMA_YAML) + shape = URIRef("https://example.org/expr-any-of/Obs") + assert list(g.objects(shape, SH.sparql)) == [] + + +def test_rule_expression_level_any_of_pyshacl_end_to_end(): + """An instance whose any_of branch is unmet satisfies the rule vacuously + and must conform.""" + import pyshacl + + shacl_ttl = ShaclGenerator(_EXPR_ANY_OF_SCHEMA_YAML, mergeimports=False, emit_rules=True).serialize() + data = """ + @prefix ex: . + + ex:o a ex:Obs ; ex:code "fog" ; ex:other "z" . + """ + conforms, _, txt = pyshacl.validate( + data_graph=data, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert conforms, f"Instance with unmet any_of branch must not be flagged:\n{txt}" + + +_POST_MIXED_SCHEMA_YAML = """ +id: https://example.org/post-mixed +name: post_mixed +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/post-mixed/ +imports: + - linkml:types +default_prefix: ex +default_range: string +slots: + guard: + slot_uri: ex:guard + target: + slot_uri: ex:target +classes: + Thing: + class_uri: ex:Thing + slots: [guard, target] + rules: + - preconditions: + slot_conditions: + guard: + equals_string: on + postconditions: + slot_conditions: + target: + required: true + pattern: "^x" +""" + + +def test_rule_post_mixed_operators_skipped(): + """A postcondition combining required with an untranslated operator must + skip — checking only required weakens the postcondition.""" + g = _parse_shacl(_POST_MIXED_SCHEMA_YAML) + shape = URIRef("https://example.org/post-mixed/Thing") + assert list(g.objects(shape, SH.sparql)) == [] + + +_ABSENT_COMBINED_SCHEMA_YAML = """ +id: https://example.org/absent-combined +name: absent_combined +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/absent-combined/ +imports: + - linkml:types +default_prefix: ex +default_range: string +slots: + count: + range: integer + slot_uri: ex:count + note: + slot_uri: ex:note +classes: + Obs: + class_uri: ex:Obs + slots: [count, note] + rules: + - preconditions: + slot_conditions: + count: + value_presence: ABSENT + minimum_value: 5 + postconditions: + slot_conditions: + note: + required: true +""" + + +def test_rule_absent_combined_with_bound_skipped(): + """value_presence ABSENT combined with another operator must skip: the + triple-binding translation would invert the declared trigger.""" + g = _parse_shacl(_ABSENT_COMBINED_SCHEMA_YAML) + shape = URIRef("https://example.org/absent-combined/Obs") + assert list(g.objects(shape, SH.sparql)) == [] + + +_STRING_TRUE_SCHEMA_YAML = """ +id: https://example.org/string-true +name: string_true +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/string-true/ +imports: + - linkml:types +default_prefix: ex +default_range: string +slots: + opt: + slot_uri: ex:opt + status: + range: string + slot_uri: ex:status +classes: + Conf: + class_uri: ex:Conf + slots: [opt, status] + rules: + - description: If opt is present, status must be the string "true". + preconditions: + slot_conditions: + opt: + value_presence: PRESENT + postconditions: + slot_conditions: + status: + equals_string: "true" +""" + + +def test_rule_equals_true_on_string_slot_uses_piv(): + """equals_string "true" on a NON-boolean slot must dispatch to + presence-implies-value (string comparison), not the boolean guard.""" + g = _parse_shacl(_STRING_TRUE_SCHEMA_YAML) + shape = URIRef("https://example.org/string-true/Conf") + sparql_nodes = list(g.objects(shape, SH.sparql)) + assert len(sparql_nodes) == 1 + query = str(list(g.objects(sparql_nodes[0], SH.select))[0]) + assert "NOT IN" in query, f"string-range 'true' must be a string comparison, got:\n{query}" + assert '"true"' in query, "the comparison term must be the string literal" + + +def test_rule_equals_true_on_string_slot_pyshacl_end_to_end(): + """status "true" (string) satisfies the rule; the boolean-guard hijack used + to flag it.""" + import pyshacl + + shacl_ttl = ShaclGenerator(_STRING_TRUE_SCHEMA_YAML, mergeimports=False, emit_rules=True).serialize() + conforming = """ + @prefix ex: . + + ex:ok a ex:Conf ; ex:opt "x" ; ex:status "true" . + """ + conforms, _, txt = pyshacl.validate( + data_graph=conforming, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert conforms, f"status 'true' satisfies the rule and must conform:\n{txt}" + + violating = """ + @prefix ex: . + + ex:bad a ex:Conf ; ex:opt "x" ; ex:status "other" . + """ + conforms, _, txt = pyshacl.validate( + data_graph=violating, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert not conforms, f"status 'other' violates the rule:\n{txt}" + + +_INNER_OVERRIDE_SCHEMA_YAML = """ +id: https://example.org/inner-override +name: inner_override +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/inner-override/ +imports: + - linkml:types +default_prefix: ex +default_range: string +slots: + sun_position: + range: SunPosition + inlined: true + slot_uri: ex:sun_position + elevation: + range: float + slot_uri: ex:elevation + headlight_note: + slot_uri: ex:headlight_note +classes: + SunPosition: + class_uri: ex:SunPosition + slots: [elevation] + slot_usage: + elevation: + slot_uri: ex:localElevation + Scene: + class_uri: ex:Scene + slots: [sun_position, headlight_note] + rules: + - description: Below the horizon a headlight note is required. + preconditions: + slot_conditions: + sun_position: + range_expression: + slot_conditions: + elevation: + maximum_value: 0 + postconditions: + slot_conditions: + headlight_note: + required: true +""" + + +def test_rule_nested_inner_slot_uri_resolved_on_range_class(): + """The inner slot of a nested precondition lives on the container's range + class; a slot_usage slot_uri override there must be honoured (sh:path / + SPARQL-body parity one hop down).""" + g = _parse_shacl(_INNER_OVERRIDE_SCHEMA_YAML) + shape = URIRef("https://example.org/inner-override/Scene") + sparql_nodes = list(g.objects(shape, SH.sparql)) + assert len(sparql_nodes) == 1 + query = str(list(g.objects(sparql_nodes[0], SH.select))[0]) + assert "https://example.org/inner-override/localElevation" in query, ( + f"inner slot must use the range class's induced slot_uri, got:\n{query}" + ) + assert "https://example.org/inner-override/elevation" not in query, ( + "the base slot_uri must not leak into the member pattern" + ) + + +def test_rule_nested_inner_slot_uri_override_pyshacl_end_to_end(): + """A night scene without the required note must be flagged — with the + base-URI mistranslation the constraint silently never fired.""" + import pyshacl + + shacl_ttl = ShaclGenerator(_INNER_OVERRIDE_SCHEMA_YAML, mergeimports=False, emit_rules=True).serialize() + # The float is typed explicitly so the sh:datatype property constraint is + # satisfied and the outcome discriminates on the rule constraint alone. + violating = """ + @prefix ex: . + @prefix xsd: . + + ex:night a ex:Scene ; ex:sun_position ex:sp . + ex:sp a ex:SunPosition ; ex:localElevation "-5.0"^^xsd:float . + """ + conforms, _, txt = pyshacl.validate( + data_graph=violating, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert not conforms, f"Night scene without headlight note must fail:\n{txt}" + + conforming = """ + @prefix ex: . + @prefix xsd: . + + ex:noon a ex:Scene ; ex:sun_position ex:sp2 . + ex:sp2 a ex:SunPosition ; ex:localElevation "45.0"^^xsd:float . + """ + conforms, _, txt = pyshacl.validate( + data_graph=conforming, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert conforms, f"Daytime scene needs no headlight note:\n{txt}" + + +_INNER_COLLISION_SCHEMA_YAML = """ +id: https://example.org/inner-collision +name: inner_collision +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/inner-collision/ +imports: + - linkml:types +default_prefix: ex +default_range: string +slots: + marker_flag: + slot_uri: ex:marker_flag + items: + range: Item + multivalued: true + inlined: true + inlined_as_list: true + slot_uri: ex:items + type: + slot_uri: ex:defaultType +classes: + Item: + class_uri: ex:Item + slots: [type] + slot_usage: + type: + slot_uri: ex:itemType + Box: + class_uri: ex:Box + slots: [marker_flag, items, type] + slot_usage: + type: + slot_uri: ex:boxType + rules: + - description: A flagged box must contain a marker item. + preconditions: + slot_conditions: + marker_flag: + value_presence: PRESENT + postconditions: + slot_conditions: + items: + has_member: + range_expression: + slot_conditions: + type: + equals_string: marker +""" + + +def test_rule_has_member_inner_slot_not_shadowed_by_outer_class(): + """An inner slot name that also exists on the OUTER class with a different + slot_usage URI must still resolve against the member class.""" + g = _parse_shacl(_INNER_COLLISION_SCHEMA_YAML) + shape = URIRef("https://example.org/inner-collision/Box") + sparql_nodes = list(g.objects(shape, SH.sparql)) + assert len(sparql_nodes) == 1 + query = str(list(g.objects(sparql_nodes[0], SH.select))[0]) + assert "https://example.org/inner-collision/itemType" in query, ( + f"member condition must use the member class's slot URI, got:\n{query}" + ) + assert "boxType" not in query, "the outer class's slot_usage URI must not shadow the member's" + + +def test_rule_has_member_inner_slot_collision_pyshacl_end_to_end(): + """A conforming box (marker item present via the member class's predicate) + must conform — the outer-class shadowing made FILTER NOT EXISTS vacuous.""" + import pyshacl + + shacl_ttl = ShaclGenerator(_INNER_COLLISION_SCHEMA_YAML, mergeimports=False, emit_rules=True).serialize() + conforming = """ + @prefix ex: . + + ex:b a ex:Box ; ex:marker_flag "y" ; ex:items ex:i1 . + ex:i1 a ex:Item ; ex:itemType "marker" . + """ + conforms, _, txt = pyshacl.validate( + data_graph=conforming, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert conforms, f"Box with a marker item must conform:\n{txt}" + + +_CONTAINER_NARROWED_SCHEMA_YAML = """ +id: https://example.org/container-narrowed +name: container_narrowed +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/container-narrowed/ +imports: + - linkml:types +default_prefix: ex +default_range: string +enums: + BaseKindEnum: + permissible_values: + special: + meaning: ex:BASE_special + SpecialKindEnum: + permissible_values: + special: + meaning: ex:SPECIAL_special +slots: + part: + range: BasePart + inlined: true + slot_uri: ex:part + kind: + range: BaseKindEnum + slot_uri: ex:kind + label_note: + slot_uri: ex:label_note +classes: + BasePart: + class_uri: ex:BasePart + slots: [kind] + SpecialPart: + class_uri: ex:SpecialPart + is_a: BasePart + slot_usage: + kind: + range: SpecialKindEnum + Assembly: + class_uri: ex:Assembly + slots: [part, label_note] + slot_usage: + part: + range: SpecialPart + rules: + - description: A special part requires a label note. + preconditions: + slot_conditions: + part: + range_expression: + slot_conditions: + kind: + equals_string: special + postconditions: + slot_conditions: + label_note: + required: true +""" + + +def test_rule_container_range_narrowing_resolves_inner_enum(): + """A slot_usage range-narrowing of the CONTAINER slot must resolve inner + enum values against the narrowed range class's enum.""" + g = _parse_shacl(_CONTAINER_NARROWED_SCHEMA_YAML) + shape = URIRef("https://example.org/container-narrowed/Assembly") + sparql_nodes = list(g.objects(shape, SH.sparql)) + assert len(sparql_nodes) == 1 + query = str(list(g.objects(sparql_nodes[0], SH.select))[0]) + assert "SPECIAL_special" in query, f"inner enum must resolve via the narrowed range, got:\n{query}" + assert "BASE_special" not in query, "the base range's enum must not be used" + + +_NON_NUMERIC_BOUNDS_SCHEMA_YAML = """ +id: https://example.org/non-numeric-bounds +name: non_numeric_bounds +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/non-numeric-bounds/ +imports: + - linkml:types +default_prefix: ex +default_range: string +slots: + v: + range: integer + slot_uri: ex:v + note: + slot_uri: ex:note +classes: + Obs: + class_uri: ex:Obs + slots: [v, note] + rules: + - preconditions: + slot_conditions: + v: + minimum_value: "abc" + postconditions: + slot_conditions: + note: + required: true + - preconditions: + slot_conditions: + v: + minimum_value: 2020-01-01 + postconditions: + slot_conditions: + note: + required: true + - preconditions: + slot_conditions: + v: + maximum_value: .nan + postconditions: + slot_conditions: + note: + required: true + - preconditions: + slot_conditions: + v: + minimum_value: true + postconditions: + slot_conditions: + note: + required: true +""" + + +def test_rule_non_numeric_bounds_skipped(): + """Non-numeric threshold bounds (string, date, NaN, boolean) must skip the + rule: raw interpolation produced unparsable SPARQL (poisoning the whole + shapes graph) or silently-wrong arithmetic (2020-01-01 == 2018).""" + g = _parse_shacl(_NON_NUMERIC_BOUNDS_SCHEMA_YAML) + shape = URIRef("https://example.org/non-numeric-bounds/Obs") + assert list(g.objects(shape, SH.sparql)) == [] + + +def test_rule_non_numeric_bounds_shapes_graph_still_validates(): + """The generated shapes graph must remain usable by pyshacl — one bad bound + used to raise a ParseException for every validation run.""" + import pyshacl + + shacl_ttl = ShaclGenerator(_NON_NUMERIC_BOUNDS_SCHEMA_YAML, mergeimports=False, emit_rules=True).serialize() + data = """ + @prefix ex: . + + ex:o a ex:Obs ; ex:v 1 . + """ + conforms, _, txt = pyshacl.validate( + data_graph=data, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert conforms, f"Shapes graph must stay parseable and the data conform:\n{txt}" + + +def test_rule_with_elseconditions_warns(caplog): + """The unenforced else branch must be signalled, consistent with the + bidirectional/open_world warnings.""" + import logging + + with caplog.at_level(logging.WARNING, logger="linkml.generators.shaclgen"): + _parse_shacl(_ELSE_COND_SCHEMA_YAML) + assert any("elseconditions" in rec.message for rec in caplog.records), ( + "dropping the else branch must produce a warning" + ) + + +def test_has_member_zero_members_pyshacl_end_to_end(): + """A node meeting the precondition with ZERO members violates has_member + ('must contain a matching member'); locks the semantics in.""" + import pyshacl + + shacl_ttl = ShaclGenerator(_HAS_MEMBER_SCHEMA_YAML, mergeimports=False, emit_rules=True).serialize() + violating = """ + @prefix ex: . + + ex:wZero a ex:Weather ; ex:fog_declared "fog" . + """ + conforms, _, txt = pyshacl.validate( + data_graph=violating, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + ) + assert not conforms, f"Zero members cannot contain the required member:\n{txt}" + + +_ALIAS_KEY_SCHEMA_YAML = """ +id: https://example.org/alias-key +name: alias_key +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/alias-key/ +imports: + - linkml:types +default_prefix: ex +default_range: string +slots: + my slot: + slot_uri: ex:customMySlot + note: + slot_uri: ex:note +classes: + Obs: + class_uri: ex:Obs + slots: ["my slot", note] + rules: + - description: Underscored alias key must resolve to the declared slot. + preconditions: + slot_conditions: + my_slot: + equals_string: trigger + postconditions: + slot_conditions: + note: + required: true +""" + + +def test_rule_alias_form_slot_key_resolves_override(): + """A rule key written `my_slot` for a slot named `my slot` must resolve to + that slot's URI (sh:path parity) instead of fabricating a default-prefix + predicate that makes the constraint vacuous.""" + g = _parse_shacl(_ALIAS_KEY_SCHEMA_YAML) + shape = URIRef("https://example.org/alias-key/Obs") + sparql_nodes = list(g.objects(shape, SH.sparql)) + assert len(sparql_nodes) == 1 + query = str(list(g.objects(sparql_nodes[0], SH.select))[0]) + assert "https://example.org/alias-key/customMySlot" in query, ( + f"alias-form key must resolve to the declared slot_uri, got:\n{query}" + ) + assert "https://example.org/alias-key/my_slot" not in query, ( + "the fabricated default-prefix predicate must not be emitted" + ) + + +_UNKNOWN_KEY_SCHEMA_YAML = """ +id: https://example.org/unknown-key +name: unknown_key +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/unknown-key/ +imports: + - linkml:types +default_prefix: ex +default_range: string +slots: + code: + slot_uri: ex:code + note: + slot_uri: ex:note +classes: + Obs: + class_uri: ex:Obs + slots: [code, note] + rules: + - description: A rule keyed on a nonexistent slot must be skipped. + preconditions: + slot_conditions: + no_such_slot: + equals_string: trigger + postconditions: + slot_conditions: + note: + required: true +""" + + +def test_rule_unknown_slot_key_skipped(): + """A rule whose condition keys a slot that does not exist must be skipped: + fabricating a default-prefix predicate would emit a constraint that can + never fire (or, for has_member, always fires).""" + g = _parse_shacl(_UNKNOWN_KEY_SCHEMA_YAML) + shape = URIRef("https://example.org/unknown-key/Obs") + assert list(g.objects(shape, SH.sparql)) == [] diff --git a/tests/linkml/test_scripts/test_gen_json_schema.py b/tests/linkml/test_scripts/test_gen_json_schema.py index 7c172e0085..471265518b 100644 --- a/tests/linkml/test_scripts/test_gen_json_schema.py +++ b/tests/linkml/test_scripts/test_gen_json_schema.py @@ -1,3 +1,4 @@ +import json import re import pytest @@ -70,3 +71,49 @@ def test_include_option(input_path): extra_import_path = str(input_path("deprecation.yaml")) result = runner.invoke(cli, ["--include", extra_import_path, schema]) assert "C4" in result.output + + +_INCLUDE_NULL_SCHEMA = """ +id: https://example.org/test-include-null-cli +name: test-include-null-cli +prefixes: + linkml: https://w3id.org/linkml/ +default_range: string +imports: + - linkml:types +classes: + C: + attributes: + opt: + range: string + opt_multi: + range: string + multivalued: true + req: + range: string + required: true +""" + + +@pytest.mark.parametrize( + "flag,expected_opt,expected_opt_multi", + [ + ([], ["string", "null"], ["array", "null"]), + (["--include-null"], ["string", "null"], ["array", "null"]), + (["--no-include-null"], "string", "array"), + ], +) +def test_include_null_cli_option(flag, expected_opt, expected_opt_multi, tmp_path): + """The --include-null/--no-include-null CLI option must control whether optional + slots accept an explicit JSON null, across scalar and multivalued ranges, while + leaving required slots unaffected.""" + schema = tmp_path / "schema.yaml" + schema.write_text(_INCLUDE_NULL_SCHEMA) + runner = CliRunner() + result = runner.invoke(cli, flag + [str(schema)]) + assert result.exit_code == 0, result.output + props = json.loads(result.output)["$defs"]["C"]["properties"] + assert props["opt"]["type"] == expected_opt + assert props["opt_multi"]["type"] == expected_opt_multi + # required scalar slot is unaffected by the flag + assert props["req"]["type"] == "string" diff --git a/uv.lock b/uv.lock index 6387537a8c..9605f09a41 100644 --- a/uv.lock +++ b/uv.lock @@ -2344,7 +2344,7 @@ requires-dist = [ { name = "openpyxl" }, { name = "parse" }, { name = "prefixcommons", specifier = ">=0.1.7" }, - { name = "prefixmaps", specifier = ">=0.2.2" }, + { name = "prefixmaps", git = "https://github.com/linkml/prefixmaps?rev=75435150a1b31760b9780af2b64a265943a9b263" }, { name = "pydantic", specifier = ">=2.0.0,<3.0.0" }, { name = "pyjsg", specifier = ">=0.12.3" }, { name = "pyshex", specifier = ">=0.9.0" }, @@ -3550,16 +3550,12 @@ wheels = [ [[package]] name = "prefixmaps" -version = "0.2.6" -source = { registry = "https://pypi.org/simple" } +version = "0.2.7.post2.dev0+7543515" +source = { git = "https://github.com/linkml/prefixmaps?rev=75435150a1b31760b9780af2b64a265943a9b263#75435150a1b31760b9780af2b64a265943a9b263" } dependencies = [ { name = "curies" }, { name = "pyyaml" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/4d/cf/f588bcdfd2c841839b9d59ce219a46695da56aa2805faff937bbafb9ee2b/prefixmaps-0.2.6.tar.gz", hash = "sha256:7421e1244eea610217fa1ba96c9aebd64e8162a930dc0626207cd8bf62ecf4b9", size = 709899, upload-time = "2024-10-17T16:30:57.738Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/89/b2/2b2153173f2819e3d7d1949918612981bc6bd895b75ffa392d63d115f327/prefixmaps-0.2.6-py3-none-any.whl", hash = "sha256:f6cef28a7320fc6337cf411be212948ce570333a0ce958940ef684c7fb192a62", size = 754732, upload-time = "2024-10-17T16:30:55.731Z" }, -] [[package]] name = "prettytable"