Fix silent reconciler skips on lock contention#2332
Draft
ideaship wants to merge 2 commits into
Draft
Conversation
Member
|
@ideaship Please rebase. |
An explicit "osism sync inventory" dispatches reconciler.run and waits on its output stream. The task had two paths that left that client without an outcome: - When tasks are administratively locked, check_task_lock_and_exit() calls exit(1). The resulting SystemExit propagated out of the Celery task, churning the prefork worker and publishing no terminal marker. - When another run already held the execution lock, run() silently returned None, published nothing, and the waiting client blocked until its own output-inactivity timeout before failing. Convert both into a definite result. The admin-lock SystemExit is caught and turned into a terminal marker plus a non-zero return; lock contention now fails fast with a terminal marker instead of blocking or returning None. Contention is not retried: a retry could acquire a lease the in-flight run let expire and start a second concurrent /run.sh against shared inventory state. When no client is waiting (publish=False) the failure is logged instead of streamed. The periodic run_on_change path is left unchanged; it keeps coalescing on contention and is hardened separately. Upside: on both paths a waiting client now fails within seconds with a clear message instead of stalling until its 300s output-inactivity timeout. Tradeoffs and remaining limits: - The webhook (api.py) and watchdog (service.py) callers run with publish=True, so they now push the failure marker (three stream entries) to an output stream nobody reads and that has no TTL. This is marginal waste on paths that previously skipped silently, on top of the pre-existing unbounded leak of successful runs' output. - The admin-lock path now records the Celery task as SUCCESS with result 1 where a propagating SystemExit previously recorded FAILURE. Nothing in-repo consumes the AsyncResult -- the CLI watches the stream -- so this only changes external observability such as flower. - _publish_lock_failure and the success path still do unguarded Redis writes; an exception there escapes with no terminal marker. This narrows the no-outcome surface rather than closing it -- the same windows exist on main. - Scope is reconciler-only; other task sites keep the same SystemExit churn. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Roger Luethi <luethi@osism.tech>
The --task-timeout help described the value as a timeout for a scheduled task that has not been executed yet, implying a queueing deadline. The value is passed to fetch_task_output, which uses it as an output-inactivity timeout: it resets every time a line arrives and only fires after that many seconds with no further output. Reword the help so operators set it for the behaviour it actually controls. Spell out the queued-task case explicitly rather than only denying it is a queueing deadline: a queued task produces no output while it waits, so that wait is not excluded from the timeout -- it counts against it in full, which is the most likely way an operator hits it. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Roger Luethi <luethi@osism.tech>
ideaship
force-pushed
the
fix/silent-skip-on-lock-acquire-timeout
branch
from
July 17, 2026 07:50
97e2a8f to
b3e5a5a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
osism sync inventorydispatches an explicitreconciler.runtask and waitson that task's Redis output stream. Two paths left the waiting client with no
outcome, so the CLI blocked until its own output-inactivity timeout before
failing. This PR makes both paths fail fast with a terminal marker and a
non-zero result, and clarifies the
--task-timeouthelp text.Problem
reconciler.runattempted to acquire its execution lock for 20 seconds. If thelock stayed held, the task skipped its whole body, published no terminal
marker, returned
None, and was recorded by Celery as successful. The waitingCLI could not distinguish that silent skip from a still-running task, so it
stayed blocked until its configured task-output timeout.
Separately, when tasks were administratively locked,
check_task_lock_and_exit()calledexit(1); the resultingSystemExitpropagated out of the Celery task, churned the prefork worker, and again left
the waiting client with no terminal marker.
This surfaced during a deploy when a burst of reconciler tasks contended on the
lock: tasks skipped silently after ~20 seconds while a CLI waiting with a long
--task-timeoutblocked for the full timeout before aborting.Changes
SystemExitraised under an administrative lock, publish aterminal marker, and return a non-zero result instead of letting it propagate
and churn the prefork worker.
result instead of silently returning
Noneand blocking the waiting client.run let expire and start a second concurrent
/run.shagainst sharedinventory state.
publish=False), log the failure instead ofstreaming it.
run_on_changepath unchanged; it keeps coalescing oncontention.
--task-timeouthelp text: it is an output-inactivity timeout,and time a task spends queued (emitting no output) counts against it in full.
Tradeoffs and scope limits
is published. This fix narrows the no-outcome surface, it does not close it.
publish=True,so they now push the failure marker to an output stream nobody reads and that
has no TTL — marginal waste on paths that previously skipped silently.
SUCCESSwith result1where the propagating
SystemExitpreviously recordedFAILURE. Nothingin-repo consumes the
AsyncResult, so this only affects externalobservability (e.g. flower).
SystemExitchurn./run.shcanrun concurrently in one container after the 60s lease expires is not
addressed here. An earlier commit defaulting the worker to concurrency 1 was
dropped — its rationale misdescribed the locking and it reintroduced a
silent-queueing UX. A proper fix would pair a structural lock with burst
coalescing or dispatch expiry.
Testing
runfails fast on contention with a terminal marker, and logs + returnsnon-zero on contention without a waiting client.
runconverts the admin-lockSystemExitinto a terminal marker, andreturns non-zero quietly when no client is waiting.
runpublishes success and releases the lock on the happy path (with andwithout publishing).
run_on_changestill runs and releases when it acquires the lock, and skipssilently when contended.
--task-timeouthelp text describes an output-inactivity timeout.