Skip to content

fix: apply the template extensions in the job parameter merge - #368

Merged
leongdl merged 1 commit into
OpenJobDescription:mainlinefrom
leongdl:fix/parameter-merge-feature-bundle-1-extensions
Sep 17, 2026
Merged

leongdl merged 1 commit into
OpenJobDescription:mainlinefrom
leongdl:fix/parameter-merge-feature-bundle-1-extensions

Conversation

@leongdl

@leongdl leongdl commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

What changed

preprocess_job_parameters re-validates each merged job parameter definition through parse_model, and it passed no supported_extensions. The parsing context that reached NameIdentifierLengthMixin therefore carried an empty extension set, so the mixin enforced the base 64-character limit on a parameter name that decode_job_template had already accepted at up to 512 under FEATURE_BUNDLE_1. create_job calls preprocess_job_parameters itself, so both entry points rejected the name.

merge_job_parameter_definitions now derives the union of the extensions declared by the job template and any environment templates, and threads it down to the parse_model call.

This is the sibling of the defect #362 fixed in instantiate_model. That PR closed three of the four FEATURE_BUNDLE_1 length ceilings; this one closes the fourth, which #362's tests did not reach because their templates declare no parameterDefinitions.

Where this sits in the parse flow

openjd-rs numbers the stages a template goes through, and the Python implementation performs the same work in the same order, so the numbering is a useful common vocabulary. Passes 1 to 4 are decode: text, to a generic mapping, to typed models. Passes 5 to 10 are template validation, enumerated in the module doc of validate_v2023_09/mod.rs — limits, structure, FEATURE_BUNDLE_1, format strings, TASK_CHUNKING, WRAP_ACTIONS. Job creation is the two stages after those: parameter preprocessing, then instantiation. Session execution follows.

The four raised length ceilings are a pass 5 rule. They are not a property of a value on its own; they are a property of a value together with the declared extension set. In openjd-rs that pairing is explicit — EffectiveLimits::from_context reads the extensions and yields max_identifier_len: if fb1 { 512 } else { 64 }, and limits::enforce_limits applies it once. In Python the same rule lives in the field validators, which read the extension set from the ModelParsingContext that parse_model builds per call:

# src/openjd/model/v2023_09/_model.py:1020
max_len = 512 if "FEATURE_BUNDLE_1" in context.extensions else 64

Where it broke

Python re-runs those validators after pass 5, because pydantic validates whenever a model is constructed, and both job-creation stages construct models. That is the whole defect class: a re-validation that does not carry the context silently downgrades to the base limit and rejects a value pass 5 accepted.

flowchart TD
    D["Passes 1 to 4: decode, decode_job_template"]
    V["Passes 5 to 10: template validation. Ceilings applied here, context present"]
    P["Parameter preprocessing: preprocess_job_parameters"]
    M["merge_job_parameter_definitions_for_one calls parse_model, which re-validates"]
    I["Instantiation: create_job calls instantiate_model, which re-validates"]
    J["Job"]
    E1["Context dropped, so the parameter name is rejected at 64. This PR."]
    E2["Context dropped, so job name, environment name and filename are rejected at their base limits. Fixed by PR 362."]
    D --> V
    V --> P
    P --> M
    M --> I
    I --> J
    M --> E1
    I --> E2
Loading

Two sites, one cause:

Stage Site Reconstruction call Status
Instantiation _internal/_create_job.py:295 target_model.model_validate(fields, context=context) fixed by #362, was target_model(**fields)
Parameter preprocessing _merge_job_parameter.py:256 parse_model(..., supported_extensions=...) this PR, was parse_model(model=..., obj=...)

The call chain to the second one, which is why both public entry points fail:

  • preprocess_job_parameters (_create_job.py:328) → merge_job_parameter_definitions (:406) → merge_job_parameter_definitions_for_oneparse_model
  • create_job (_create_job.py:569) → _create_job_and_symbol_table (:462) → preprocess_job_parameters (:496) → the same chain

Why openjd-rs does not have it

It validates once and never re-validates a constructed model; the later stages consume resolved values instead. create_job/parameters.rs never constructs EffectiveLimits at all, so there is no second computation of the ceiling to get wrong. The divergence is structural, not a missing check.

Why the fix belongs here

The rule is correct and pass 5 already enforces it. What was lost is the second half of the pair — the extension set — at the one site that dropped it. So the fix restores the context at that site and changes nothing else. The alternatives are each wrong in a way worth recording:

  • Guard the validators on context is None. Smaller diff, and it would accept a 512-character name on a template with no extensions block, because a context-free re-validation is indistinguishable from a caller that declared the extension. fix: validate instantiated job models with the template's extension context #362 rejected this same shape.
  • Raise the static type ceilings. Identifier already permits 512. The ceiling is not the constraint; the conditional falling back is.
  • Add a supported_extensions argument to the public merge_job_parameter_definitions. It already receives the templates that carry extensions, so this would ask callers to restate something derivable, and let them assert an extension the template never declared.
  • Fix it downstream in instantiation. Instantiation consumes preprocessing's output, so the value has to be accepted at the earlier stage regardless.

Measured before the change

Template declaring extensions: [FEATURE_BUNDLE_1] with a 512-character STRING parameter name:

Stage Result
decode_job_template OK
merge_job_parameter_definitions DecodeValidationError: name must be at most 64 characters long
preprocess_job_parameters same
create_job same

After: all four stages accept it, and job.parameters carries the 512-character key.

Why derive rather than add an argument

merge_job_parameter_definitions already receives the templates that carry extensions, so no public signature changes.

The union across templates is sound because a definition was decoded under its own template's extension set, and every definition of one job parameter carries the same name, so the union cannot admit a value decode did not already accept for the template that declared it. merge_job_parameter_definitions_for_one gains a keyword-only supported_extensions defaulting to None, which preserves its behaviour for any existing caller.

Tests, both lanes

All four FEATURE_BUNDLE_1 length ceilings, at job-creation time rather than only at decode.

v0 — group H in test/openjd/model_v0/v2023_09/test_feature_bundle_1.py:

  • the parameter name through the merge, preprocess_job_parameters and create_job, parametrized over STRING, PATH, INT and FLOAT (the four types the merge re-validates)
  • a genuine two-source merge, where constraints from an environment template and the job template are actually combined
  • the environment-template-only case: the env template enables FEATURE_BUNDLE_1 and the job template does not
  • the EXPR-typed model_copy branch, so unifying the two branches cannot silently reintroduce the base limit
  • controls: the base 64 limit still rejects at 65, the static Identifier ceiling rejects at 513, and driving the merge directly with no extensions still enforces 64 rather than skipping the check

v1 — a new TestCreateJobPreservesFeatureBundle1Lengths in test/openjd/model_v1/test_create_job.py. This lane had no coverage of these four ceilings at any level. Measured: v1 does not have the defect, for the structural reason above. These 22 tests are a ratchet, so a re-validation pass added upstream cannot land here unnoticed.

Verification

  • hatch run test: 6108 passed, 24 skipped, 3 xfailed. 6070 before this change.
  • hatch run lint, hatch run typing: clean.
  • Every v0 test mutation-checked. Seven mutants, all caught: dropping the threading at each of its three points, hardcoding FEATURE_BUNDLE_1 instead of deriving it, ignoring environment templates, applying the threading only to STRING, and applying it only to single-source merges.

Not verified

The v1 tests cannot be mutation-checked from this repository — the implementation they exercise is the Rust extension, so the mutants for them would have to be applied in openjd-rs. They are asserted to pass, not asserted to be falsifiable here.

preprocess_job_parameters re-validates each merged job parameter definition
through parse_model, and it passed no supported_extensions. The parsing context
that reached NameIdentifierLengthMixin therefore carried an empty extension set,
so the mixin enforced the base 64-character limit on a parameter name that decode
had already accepted at up to 512 under FEATURE_BUNDLE_1. create_job calls
preprocess_job_parameters itself, so both entry points rejected the name.

Measured before the change, for a template declaring FEATURE_BUNDLE_1 with a
512-character STRING parameter name:

    decode_job_template      OK
    preprocess_job_parameters DecodeValidationError: 1 validation errors for
                              JobStringParameterDefinition name: name must be
                              at most 64 characters long
    create_job                same

merge_job_parameter_definitions now derives the union of the extensions declared
by the job template and any environment templates, and threads it to
merge_job_parameter_definitions_for_one and on into parse_model. The union is
sound because a definition was decoded under its own template's extension set and
every definition of one job parameter carries the same name, so the union cannot
admit a value decode did not already accept for the template that declared it.

The public signature of merge_job_parameter_definitions is unchanged: it already
receives the templates that carry the extensions, so the set is derived rather
than passed. This follows the shape of OpenJobDescription#362, which fixed the sibling defect in
instantiate_model, and it keeps a caller from declaring an extension the template
did not. merge_job_parameter_definitions_for_one takes a new keyword-only
supported_extensions argument that defaults to None, preserving its behaviour for
any existing caller.

Tests cover all four FEATURE_BUNDLE_1 length ceilings at job-creation time on
both lanes. v0 gains group H in test_feature_bundle_1.py: the parameter name
through the merge, preprocess_job_parameters and create_job for each of STRING,
PATH, INT and FLOAT, a genuine two-source merge, the environment-template-only
case, the EXPR-typed model_copy branch, and controls for the base 64 limit and
the static 512 identifier ceiling. v1 gains an equivalent class in
model_v1/test_create_job.py, which had no coverage of these ceilings at all; the
Rust implementation validates in a single pass and already passes, so those tests
are a ratchet against a re-validation pass being added upstream unnoticed.

Every v0 test was mutation-checked. Seven mutants, each caught: dropping the
threading at any of its three points, hardcoding FEATURE_BUNDLE_1 instead of
deriving it, ignoring environment templates, applying the threading only to
STRING, and applying it only to single-source merges.

Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
@leongdl
leongdl requested a review from a team as a code owner September 17, 2026 21:36
Comment thread src/openjd/model/_merge_job_parameter.py
@leongdl
leongdl merged commit 8959e09 into OpenJobDescription:mainline Sep 17, 2026
32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants