Skip to content

Trim old task logs in the Docker Compose quick-start - #73266

Open
Admaing wants to merge 1 commit into
apache:mainfrom
Admaing:add-log-groomer-to-docker-compose
Open

Admaing wants to merge 1 commit into
apache:mainfrom
Admaing:add-log-groomer-to-docker-compose

Conversation

@Admaing

@Admaing Admaing commented Sep 17, 2026

Copy link
Copy Markdown

The quick-start kept every task log forever, so the shared logs volume grows without bound until the
disk of the machine running it fills up.

The Helm chart has had a logGroomerSidecar for a long time, but the docker-compose.yaml that most
people start with had no equivalent, and the logging docs never said that Airflow neither rotates nor
deletes local task logs.

This adds an airflow-log-groomer service that reuses the same /clean-logs script as the chart:

  • task logs older than 15 days are deleted by default (the same default as the Helm chart), configurable
    with AIRFLOW__LOG_RETENTION_DAYS / AIRFLOW__LOG_RETENTION_MINUTES;
  • AIRFLOW__LOG_MAX_SIZE_BYTES / AIRFLOW__LOG_MAX_SIZE_PERCENT bound the size of the logs folder;
  • removing the airflow-log-groomer service keeps all logs, as before;
  • the behaviour is documented in the quick-start page, in "Logging for Tasks", and in the advanced
    logging configuration page, which previously only mentioned size-based rotation.

Note for reviewers: the AIRFLOW__LOG_* variables are read by the /clean-logs script, not by Airflow
itself. Setting them in .env therefore adds them to the environment of every service, where Airflow
ignores them (AIRFLOW__LOG_RETENTION_DAYS has no __ separator between a section and a key, so it is
not a configuration option), while the groomer picks them up through Compose interpolation. The names
are the ones the chart and the script already use.

Verified locally: docker compose config renders the service for both the default and an overridden
retention, and yamllint, the Compose JSON-schema hook, mypy for docker-tests and the remaining prek
hooks pass for the changed files. The new test_log_groomer_service_config asserts the service, its
command and the retention default/override against that same rendered config; it runs as part of the
docker-compose-tests suite, which was not run locally.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

DeepSeek Harness (deepseek-v4.1-flash) following the guidelines

The quick-start kept every task log forever, so its shared logs volume grows
unbounded until the disk of the machine running it fills up.

The Helm chart guards against this with its logGroomerSidecar, but the Docker
Compose file that most users start with had no equivalent, and the logging docs
never said that Airflow neither rotates nor deletes local task logs.

Reuse the same /clean-logs script as the chart, keeping 15 days by default, and
document the retention settings next to the task log configuration.
@boring-cyborg

boring-cyborg Bot commented Sep 17, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant