Skip to content

Fix silent reconciler skips on lock contention#2332

Draft
ideaship wants to merge 2 commits into
mainfrom
fix/silent-skip-on-lock-acquire-timeout
Draft

Fix silent reconciler skips on lock contention#2332
ideaship wants to merge 2 commits into
mainfrom
fix/silent-skip-on-lock-acquire-timeout

Conversation

@ideaship

@ideaship ideaship commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Summary

osism sync inventory dispatches an explicit reconciler.run task and waits
on that task's Redis output stream. Two paths left the waiting client with no
outcome, so the CLI blocked until its own output-inactivity timeout before
failing. This PR makes both paths fail fast with a terminal marker and a
non-zero result, and clarifies the --task-timeout help text.

Problem

reconciler.run attempted to acquire its execution lock for 20 seconds. If the
lock stayed held, the task skipped its whole body, published no terminal
marker, returned None, and was recorded by Celery as successful. The waiting
CLI could not distinguish that silent skip from a still-running task, so it
stayed blocked until its configured task-output timeout.

Separately, when tasks were administratively locked,
check_task_lock_and_exit() called exit(1); the resulting SystemExit
propagated out of the Celery task, churned the prefork worker, and again left
the waiting client with no terminal marker.

This surfaced during a deploy when a burst of reconciler tasks contended on the
lock: tasks skipped silently after ~20 seconds while a CLI waiting with a long
--task-timeout blocked for the full timeout before aborting.

Changes

  • Catch the SystemExit raised under an administrative lock, publish a
    terminal marker, and return a non-zero result instead of letting it propagate
    and churn the prefork worker.
  • On execution-lock contention, fail fast with a terminal marker and a non-zero
    result instead of silently returning None and blocking the waiting client.
  • Do not retry on contention: a retry could acquire a lease the in-flight
    run let expire and start a second concurrent /run.sh against shared
    inventory state.
  • When no client is waiting (publish=False), log the failure instead of
    streaming it.
  • Leave the periodic run_on_change path unchanged; it keeps coalescing on
    contention.
  • Clarify the --task-timeout help text: it is an output-inactivity timeout,
    and time a task spends queued (emitting no output) counts against it in full.

Tradeoffs and scope limits

  • Terminal-marker publication is best-effort; if Redis is unavailable no marker
    is published. This fix narrows the no-outcome surface, it does not close it.
  • Fire-and-forget callers (NetBox webhook, watchdog) run with publish=True,
    so they now push the failure marker to an output stream nobody reads and that
    has no TTL — marginal waste on paths that previously skipped silently.
  • The admin-lock path now records the Celery task as SUCCESS with result 1
    where the propagating SystemExit previously recorded FAILURE. Nothing
    in-repo consumes the AsyncResult, so this only affects external
    observability (e.g. flower).
  • Scope is reconciler-only; other task sites keep the same SystemExit churn.
  • The concurrency question is deliberately left open: whether two /run.sh can
    run concurrently in one container after the 60s lease expires is not
    addressed here. An earlier commit defaulting the worker to concurrency 1 was
    dropped — its rationale misdescribed the locking and it reintroduced a
    silent-queueing UX. A proper fix would pair a structural lock with burst
    coalescing or dispatch expiry.

Testing

  • run fails fast on contention with a terminal marker, and logs + returns
    non-zero on contention without a waiting client.
  • run converts the admin-lock SystemExit into a terminal marker, and
    returns non-zero quietly when no client is waiting.
  • run publishes success and releases the lock on the happy path (with and
    without publishing).
  • run_on_change still runs and releases when it acquires the lock, and skips
    silently when contended.
  • The --task-timeout help text describes an output-inactivity timeout.
  • Full test suite, Black, and flake8 pass.

@berendt berendt moved this from New to In progress in Human Board Jul 1, 2026
@berendt

berendt commented Jul 16, 2026

Copy link
Copy Markdown
Member

@ideaship Please rebase.

ideaship added 2 commits July 17, 2026 09:44
An explicit "osism sync inventory" dispatches reconciler.run and waits on
its output stream. The task had two paths that left that client without an
outcome:

- When tasks are administratively locked, check_task_lock_and_exit() calls
  exit(1). The resulting SystemExit propagated out of the Celery task,
  churning the prefork worker and publishing no terminal marker.
- When another run already held the execution lock, run() silently returned
  None, published nothing, and the waiting client blocked until its own
  output-inactivity timeout before failing.

Convert both into a definite result. The admin-lock SystemExit is caught and
turned into a terminal marker plus a non-zero return; lock contention now
fails fast with a terminal marker instead of blocking or returning None.
Contention is not retried: a retry could acquire a lease the in-flight run
let expire and start a second concurrent /run.sh against shared inventory
state. When no client is waiting (publish=False) the failure is logged
instead of streamed.

The periodic run_on_change path is left unchanged; it keeps coalescing on
contention and is hardened separately.

Upside: on both paths a waiting client now fails within seconds with a clear
message instead of stalling until its 300s output-inactivity timeout.

Tradeoffs and remaining limits:

- The webhook (api.py) and watchdog (service.py) callers run with
  publish=True, so they now push the failure marker (three stream entries)
  to an output stream nobody reads and that has no TTL. This is marginal
  waste on paths that previously skipped silently, on top of the pre-existing
  unbounded leak of successful runs' output.
- The admin-lock path now records the Celery task as SUCCESS with result 1
  where a propagating SystemExit previously recorded FAILURE. Nothing in-repo
  consumes the AsyncResult -- the CLI watches the stream -- so this only
  changes external observability such as flower.
- _publish_lock_failure and the success path still do unguarded Redis writes;
  an exception there escapes with no terminal marker. This narrows the
  no-outcome surface rather than closing it -- the same windows exist on main.
- Scope is reconciler-only; other task sites keep the same SystemExit churn.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Roger Luethi <luethi@osism.tech>
The --task-timeout help described the value as a timeout for a scheduled
task that has not been executed yet, implying a queueing deadline. The value
is passed to fetch_task_output, which uses it as an output-inactivity
timeout: it resets every time a line arrives and only fires after that many
seconds with no further output. Reword the help so operators set it for the
behaviour it actually controls.

Spell out the queued-task case explicitly rather than only denying it is a
queueing deadline: a queued task produces no output while it waits, so that
wait is not excluded from the timeout -- it counts against it in full, which
is the most likely way an operator hits it.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
ideaship force-pushed the fix/silent-skip-on-lock-acquire-timeout branch from 97e2a8f to b3e5a5a Compare July 17, 2026 07:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In progress

Development

Successfully merging this pull request may close these issues.

3 participants