ob status and execution list do not show an in-flight scheduled job, making an exclusive lock look orphaned

[vishr]

What happens

An operation was refused with:

✗ ob: application scheduling rendezvous remained busy — wait for the current
     scheduled job or application operation to finish

while, at the same moment, onebox reported that nothing was running:

$ ob execution list
no durable executions recorded

$ ob status
schedule <weekly-job>  last run failed: timeout (exit 143), and nothing has run since ⚠
schedule <5min-job>    skipped 20 firings in a row: an application operation is taking its lock ⚠

Host inspection showed the truth: a scheduled job declaring deploy_lock: exclusive had been running for nearly eight hours and legitimately held the rendezvous.

PID 365044  started 10:00:11  elapsed 07:57:46  PPID 1
            /bin/sh /etc/systemd/system/ob-<app>-<job>.run
PID 982137  started 15:47:12  docker compose run --rm --no-deps ... <job>

both holding fd 8 on <base>/<app>/schedule.lock. The application lock file itself did not exist, so nothing was stuck — the lock was doing exactly its job.

The bug

An in-flight scheduled job is invisible to ob status and ob execution list.

• ob execution list returns no durable executions recorded while a job has been running for hours.
• ob status reports the previous run's outcome and adds "and nothing has run since", which is actively wrong and points away from the truth.

ob schedule list does show the current run's start time as LAST TRIGGER, so the information exists and is simply not joined up.

Why it matters

The refusal message is accurate but gives nothing to act on. Every diagnostic onebox offers then says the system is idle, so the natural conclusion is an orphaned lock from the earlier timeout — that is what I concluded, and it is wrong. The real answer, "a job with an exclusive lock has been running since 10:00 and will hold it until it finishes", is not reachable through any ob command. It took /proc/locks and an fd scan on the host to find it.

The confusion is compounded by skipped being the same word used for a healthy skip, so a schedule that has done nothing for hours looks unremarkable.

Suggested

• Show in-flight scheduled jobs in ob status and ob execution list. A running exclusive job is the single most useful thing to know when an operation is refused.
• Name the holder in the rendezvous remained busy error — job name and start time. The data is discoverable from /proc/locks plus an fd scan; surfacing it turns a dead end into a wait.
• Do not report "nothing has run since" when something is running.
• Consider whether repeated skipped outcomes deserve a distinct signal from ordinary ones.

Environment

• Runner: ob v2026.9.3 (ffe61080c66b)
• Host: Ubuntu, docker 29.1.3, compose 2.40.3
• Jobs: one weekly with deploy_lock: exclusive and an 8h timeout, two with deploy_lock: pinned

[vishr]

Additional finding while investigating: the recovery path I reached for is itself gated behind the same rendezvous.

ob schedule apply regenerates the timer units, and is the documented way to reconcile them after a runner or configuration change. It acquires the rendezvous first:

$ ob schedule apply
✗ ob: application scheduling rendezvous remained busy — wait for the current
     scheduled job or application operation to finish

That is defensible on its own — reconciling units while a job runs from them is not obviously safe. But combined with the reporting gap above it leaves an operator with no way forward and no way to find out why: ob exec refused, ob schedule apply refused, and every status command reporting an idle system.

Worth noting an asymmetry in the locking while this is being looked at. AcquireLock has a take-over path for the application lock — age > lockTTL → "expired Ns ago — taking over", default 10 minutes. The rendezvous has no equivalent. That is correct while a real holder exists, which was the case here, but it means a genuinely leaked flock would have no recovery path at all.

[vishr]

Correcting my own analysis: the lock was not orphaned.

My original report concluded that timer units generated before 3e7fe57 never release the lock. That was wrong, and I have rewritten the issue body to describe the actual defect.

What was happening: a scheduled job with deploy_lock: exclusive was running normally and holding the rendezvous for its whole run, exactly as documented. <base>/<app>/lock did not exist, so no application lock was stuck, and every skipped firing behind it was correct behaviour.

The units on that host are older than 3e7fe57 — the last deploy predates the fix — but that is not what caused this, and the older code path would have skipped correctly here too. Apologies for the noise.

What led me to the wrong conclusion is the part still worth fixing: with that job running for eight hours, ob execution list reported no durable executions recorded and ob status reported the previous run's timeout with "and nothing has run since". I trusted both, and they described an idle system while an exclusive job held the lock.

The remaining separate observation — that a runner upgrade does not regenerate timer units, so a fix to the generated scripts stays inert until someone runs ob schedule apply — still seems worth its own issue, but it did not cause this one.

[vishr]

One more command gated behind the same rendezvous: ob schedule pause.

$ ob schedule pause <job> --reason '...'
✗ ob: application scheduling rendezvous remained busy — wait for the current
     scheduled job or application operation to finish

So while a scheduled job is running, it cannot be paused through ob. Combined with the earlier findings, an operator facing a misbehaving long-running job has:

• ob exec — refused
• ob schedule apply — refused
• ob schedule pause — refused
• ob status / ob execution list — report an idle system

and no supported way to stop the job or find out what it is. The only route left is systemctl stop on the host, outside onebox entirely, which also means it happens without the journalling ob would otherwise provide.

pause in particular seems worth exempting. It does not mutate the release or the units — it stops a timer — and its value is highest exactly when a job is misbehaving, which is when it is currently unavailable. The error message even makes the case: pausing is how you stop the next occurrence of the run that is holding the lock.

[vishr]

Reproduced again, this time with an operator-initiated sealed-plan job rather than a timer-driven one, which rules out the timer path being the cause.

While the job is running — container up, actively logging progress, exclusive lock held:

$ ob execution list
no durable executions recorded

$ ob status
schedule <job> last run failed: failure (exit 143) ⚠

ob status is describing the previous run. There is no indication anywhere in ob's output that a job is running right now, despite ob job plan / ob job run having just sealed, approved and started it seconds earlier, and despite the audit recording it:

$ ob audit
<release>-job_run-<id>   job run <job>   <operator>   <sha>   INCOMPLETE   2026-09-14T19:20:12Z

So the state is known to the audit log and absent from the two commands an operator would actually consult.

Two adjacent observations from the same session, offered as context rather than separate reports:

ob job run is synchronous and attached. For a job whose declared timeout is measured in hours, that commits a terminal for the duration. The container is detached and survives a dropped connection, but ob stops tracking and the run is recorded INCOMPLETE — the same marker an abandoned approval leaves, so a completed-but-disconnected run and a cancelled one are indistinguishable afterwards.

INCOMPLETE is also what a cancelled approval leaves behind. In this session a run cancelled at the typed-confirmation prompt and a run that started and detached both produced INCOMPLETE rows, which makes the audit ambiguous about whether anything actually executed.

None of this is data-unsafe. It is that the tool's own reporting cannot answer "is something running, and did it do anything", which is the question an operator has when an exclusive lock is refusing their work.