Skip to content

Add apache-airflow-providers-dbt-core provider with DbtKubernetesRunOperator - #73259

Open
Saketkr21 wants to merge 1 commit into
apache:mainfrom
Saketkr21:add-dbt-core-provider
Open

Saketkr21 wants to merge 1 commit into
apache:mainfrom
Saketkr21:add-dbt-core-provider

Conversation

@Saketkr21

Copy link
Copy Markdown

Add a new apache-airflow-providers-dbt-core provider package that ships a single
operator, DbtKubernetesRunOperator, which runs a dbt Core job inside a single
Kubernetes pod — mirroring a dbt Cloud job run without requiring dbt Cloud.

The operator subclasses KubernetesPodOperator and orchestrates the full dbt
lifecycle in one pod:

  1. Clone the dbt project from git (optional; repo may be baked into the image).
    A git-repo cache on S3/GCS lets runs recover if git is temporarily unreachable.
  2. Run dbt deps (optional).
  3. Execute steps — ordered shell commands (typically dbt CLI calls) sequentially;
    the first non-zero exit stops the run and fails the task.
  4. Upload target/ to S3 or GCS on both success and failure, so
    run_results.json and compiled SQL are always available for inspection.

Because the operator is a thin subclass, all KubernetesPodOperator arguments
(namespace, container_resources, deferrable, secrets, init_containers, …)
pass straight through.

Files added / changed

Path What
providers/dbt/core/pyproject.toml New provider package
providers/dbt/core/provider.yaml Provider metadata
providers/dbt/core/src/.../operators/dbt.py DbtKubernetesRunOperator
providers/dbt/core/src/.../operators/run.sh Pod entrypoint script
providers/dbt/core/docs/ Operator how-to, connections, security, changelog
providers/dbt/core/README.rst Package README
providers/dbt/core/tests/unit/.../test_dbt.py 26 unit tests (no pod created)
providers/dbt/core/tests/system/.../example_dbt_kubernetes.py System test / example Dag

Design choices

  • One entrypoint script (run.sh): the Python operator constructs a set of
    environment variables and delegates to a bundled bash script. This keeps the
    operator thin and makes the lifecycle trivially inspectable via pod logs.
  • steps not just dbt commands: the parameter accepts any shell command, so
    users can include setup steps (uv sync, pip install …) without needing a
    separate operator or custom image.
  • artifact_conn_id required with artifact_dest: credentials are resolved
    on the worker and injected as env vars; the pod never touches Airflow connections
    directly. This is documented in docs/security.rst.
  • git repo caching: slug-based S3/GCS key per DAG + repo + branch; the operator
    updates the cache on every successful clone and falls back to it on clone failure.
  • Optional cloud providers: amazon and google extras are guarded with
    AirflowOptionalProviderFeatureException; the base package only requires
    apache-airflow-providers-cncf-kubernetes.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code (claude-sonnet-4.6) following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg

boring-cyborg Bot commented Sep 17, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@Saketkr21
Saketkr21 force-pushed the add-dbt-core-provider branch from 98bdf50 to d305295 Compare September 17, 2026 00:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant