Conversation
The quick-start kept every task log forever, so its shared logs volume grows unbounded until the disk of the machine running it fills up. The Helm chart guards against this with its logGroomerSidecar, but the Docker Compose file that most users start with had no equivalent, and the logging docs never said that Airflow neither rotates nor deletes local task logs. Reuse the same /clean-logs script as the chart, keeping 15 days by default, and document the retention settings next to the task log configuration.
Admaing
requested review from
ashb,
gopidesupavan,
jason810496 and
potiuk
as code owners
September 17, 2026 03:52
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The quick-start kept every task log forever, so the shared
logsvolume grows without bound until thedisk of the machine running it fills up.
The Helm chart has had a
logGroomerSidecarfor a long time, but thedocker-compose.yamlthat mostpeople start with had no equivalent, and the logging docs never said that Airflow neither rotates nor
deletes local task logs.
This adds an
airflow-log-groomerservice that reuses the same/clean-logsscript as the chart:with
AIRFLOW__LOG_RETENTION_DAYS/AIRFLOW__LOG_RETENTION_MINUTES;AIRFLOW__LOG_MAX_SIZE_BYTES/AIRFLOW__LOG_MAX_SIZE_PERCENTbound the size of the logs folder;airflow-log-groomerservice keeps all logs, as before;logging configuration page, which previously only mentioned size-based rotation.
Note for reviewers: the
AIRFLOW__LOG_*variables are read by the/clean-logsscript, not by Airflowitself. Setting them in
.envtherefore adds them to the environment of every service, where Airflowignores them (
AIRFLOW__LOG_RETENTION_DAYShas no__separator between a section and a key, so it isnot a configuration option), while the groomer picks them up through Compose interpolation. The names
are the ones the chart and the script already use.
Verified locally:
docker compose configrenders the service for both the default and an overriddenretention, and
yamllint, the Compose JSON-schema hook, mypy fordocker-testsand the remaining prekhooks pass for the changed files. The new
test_log_groomer_service_configasserts the service, itscommand and the retention default/override against that same rendered config; it runs as part of the
docker-compose-testssuite, which was not run locally.Was generative AI tooling used to co-author this PR?
DeepSeek Harness (deepseek-v4.1-flash) following the guidelines