Skip to content

Drain local workers by working off remaining jobs on SIGTERM#5243

Open
svkrieger wants to merge 1 commit into
mainfrom
fix-local-worker-drain
Open

Drain local workers by working off remaining jobs on SIGTERM#5243
svkrieger wants to merge 1 commit into
mainfrom
fix-local-worker-drain

Conversation

@svkrieger

Copy link
Copy Markdown
Contributor

Thanks for contributing to cloud_controller_ng. To speed up the process of reviewing your pull request please provide us with:

A short explanation of the proposed change:

This adds LocalWorkerDrainPlugin, which replaces the SIGTERM handler for local workers only. Instead of stopping, the worker sets exit_on_complete so it finishes all remaining jobs in the queue before exiting, avoiding dangling PROCESSING_UPLOAD/PENDING states across restarts.

The plugin is a no-op for generic workers and named clock queues, which retain the default SIGTERM behaviour.

An explanation of the use cases your change solves

Local workers handle jobs that require local filesystem access (package, droplet, and buildpack uploads). When a local worker receives SIGTERM during drain, the default delayed_job handler calls stop(), which exits after the current job but leaves remaining queued jobs unprocessed until the worker restarts.

Links to any other associated PRs

  • I have reviewed the contributing guide

  • I have viewed, signed, and submitted the Contributor License Agreement

  • I have made this pull request to the main branch

  • I have run all the unit tests using bundle exec rake

  • I have run CF Acceptance Tests

Local workers handle jobs that require local filesystem access (package,
droplet, and buildpack uploads). When a local worker receives SIGTERM
during drain, the default delayed_job handler calls stop(), which exits
after the current job but leaves remaining queued jobs unprocessed until
the worker restarts.

This adds LocalWorkerDrainPlugin, which replaces the SIGTERM handler for
local workers only. Instead of stopping, the worker sets exit_on_complete
so it finishes all remaining jobs in the queue before exiting, avoiding
dangling PROCESSING_UPLOAD/PENDING states across restarts.

The plugin is a no-op for generic workers and named clock queues, which
retain the default SIGTERM behaviour.
@svkrieger svkrieger force-pushed the fix-local-worker-drain branch from 212debc to 0c6df7b Compare June 30, 2026 06:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant