Skip to content

Template every connection id accepted by provider operators - #73286

Open
shahar1 wants to merge 2 commits into
apache:mainfrom
shahar1:template-conn-ids
Open

shahar1 wants to merge 2 commits into
apache:mainfrom
shahar1:template-conn-ids

Conversation

@shahar1

@shahar1 shahar1 commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

closes: #35259

Human Summary

I've figured out that it would be easier and better to add all conn_id at once as template fields to all operators, as the original issue suggests, rather than managing a "burn-out" operation like we did in #70296. It is a very quick fix, and I don't want to spend days reviewing AI PRs that do it over and over again. To ensure that it is being enforced, also added a pre-commit for that.

AI Summary

Click here Connection ids are meant to be templatable (`conn_id="{{ params.conn_id }}"`), but coverage across providers was uneven: 331 of 1144 provider classes with `template_fields` listed their `*conn_id` argument, the rest did not, and many subclasses that redefine `template_fields` silently dropped the parent's connection id.

Commit 1 adds every *conn_id argument to template_fields where templating can take effect (523 entries across 237 files, 59 providers). Arguments read in __init__ or stored under a different attribute name (e.g. SparkSubmitOperator keeps _conn_id) are left alone since rendering would not reach them. Six Vertex AI operators gain an explicit gcp_conn_id parameter forwarded to the base class so validate-operators-init sees a plain assignment. Five tests asserting exact tuples are updated.

Commit 2 adds the check-conn-id-templated prek hook (providers config) so this does not erode. It parses the provider tree once, resolves template_fields across the inheritance chain the way Python does (literal tuples, *Parent.template_fields, tuple({...} | set(Parent.template_fields)), aws_template_fields(...)), and reports any *conn_id __init__ argument, own or inherited, that the resolved fields miss. Arguments templating cannot act on are skipped. Runs in ~6 s on the full tree, only parses files containing conn_id. Covered by 16 unit tests under scripts/tests/ci/prek/.

Verified with breeze run mypy on the touched files, the runtime superset check of every edited class, and the template_fields tests of all 59 providers.

related: #35259


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Fable 5.1)

Generated-by: Claude Code (Fable 5.1) following the guidelines

🤖 Generated with Claude Code

Users pass connection ids through Params and Variables to reuse one Dag
across environments, which only works when the operator lists the argument
in template_fields. Coverage was uneven: a third of provider operators
templated their connection id, the rest did not, and subclasses that
redefined template_fields silently dropped the parent's connection id.
Templating is added wherever it can take effect: arguments that are read
in __init__ or stored under a different attribute name are left alone.
Without a check the previous commit erodes: new operators forget the
argument, and subclasses that redefine template_fields drop the parent's
connection id. The hook resolves template_fields across the inheritance
chain the same way Python does, and skips arguments templating cannot act
on (read in __init__, or never stored under the argument name).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow jinja templating connection ids for all third party operators

1 participant