Skip to content

fix: join contending daemon startups before reuse - #3130

Closed
thymikee wants to merge 3 commits into
fix/daemon-client-startupfrom
fix/daemon-startup-contention
Closed

thymikee wants to merge 3 commits into
fix/daemon-client-startupfrom
fix/daemon-startup-contention

Conversation

@thymikee

@thymikee thymikee commented Oct 3, 2026 •

Copy link
Copy Markdown
Member

Summary

Daemon startup now joins a child that explicitly reports lock contention before admitting another daemon. It waits for daemon or client holders, reapplies version/code/policy checks to a published winner, and relaunches only after release or proven abandonment. All attempts share the original 15-second admission deadline. Late takeovers preserve the winner; cleanup has its own bounded exit/join waits.

Fatal probing of a private startup retires and joins it before returning the original error. Stale metadata cannot bypass a different publishing owner. Legacy lock state is retained with upgrade guidance. This removes client-side stale-lock deletion.

8 files; 987 changed lines. Part of #3116, based on #3129. Timeout retirement and the session migration follow separately.

flowchart TD
    L[Launch] --> E{Monitored child result}
    E -->|still running| O[Probe this launch only]
    E -->|busy and joined| I[Inspect registration]
    E -->|other exit| F[Retire captured attempt]
    I -->|healthy winner| A[Apply reuse policy]
    I -->|held or publishing| I
    I -->|released or proven dead| L
    I -->|unproven| R[Retain and report]
Loading

Validation

Commit 180ee97381: 95 focused tests passed without skips; quick and Fallow checks passed. Mutation controls failed for adopting before joining, treating generic exits as contention, restarting the deadline, and skipping cleanup after a probe error. Real children and temporary filesystem claims exercised the changed paths; every child was joined.

Exact-head affected checks passed, including 955 tests across 139 files and unchanged RPC compatibility. Provider, coverage and native CI remain authoritative. This slice makes no device-performance claim.

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-10-03 21:14 UTC

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
Installed (including dependencies) 4.95 MB 4.96 MB +1.1 kB
Package (unpacked) 4.95 MB 4.95 MB +1.1 kB
Package (download) 1.49 MB 1.49 MB +456 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 18.9 ms 20.1 ms +1.2 ms
CLI --help 55.4 ms 55.6 ms +0.3 ms

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 8 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread src/daemon-client/daemon-client-lifecycle.ts
Comment thread src/daemon-client/daemon-client-metadata.ts Outdated
Comment thread src/daemon-client/daemon-client-lifecycle.ts
Comment thread src/daemon-client/daemon-client-lifecycle.ts Outdated
Comment thread src/daemon-client/daemon-client-lifecycle.ts
Comment thread src/daemon-client/daemon-client-transport.ts
Comment thread website/docs/docs/installation.md Outdated
Comment thread src/daemon-client/daemon-client-transport.ts
Comment thread website/docs/docs/installation.md Outdated
Comment thread src/daemon-client/__tests__/daemon-client-lifecycle.test.ts
@thymikee
thymikee force-pushed the fix/daemon-client-startup branch from 98cc766 to 359e453 Compare October 3, 2026 01:27
@thymikee
thymikee force-pushed the fix/daemon-startup-contention branch from 27db824 to 180ee97 Compare October 3, 2026 01:27

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 5 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread src/daemon-client/daemon-client-lifecycle.ts
Comment thread src/daemon-client/__tests__/daemon-client-startup-race.test.ts
@thymikee

thymikee commented Oct 3, 2026

Copy link
Copy Markdown
Member Author

I found one defect in 180ee97 that should be fixed before merge. When a launched daemon publishes its registration just as the launcher's deadline passes, the launcher retires it while a joined client is already reusing it. A second client can then lose its command mid-request, and the first client fails with daemon_startup_failed although its daemon was healthy.

The case is in daemon-client-lifecycle.ts. Client A launches daemon D, and D becomes reachable right as A's deadline passes. readReadyLaunchedDaemon sees canConnect === true but returns null because Date.now() < deadline is false. waitForDaemonStartup returns 'timeout', and attemptLocalDaemonStartup calls retireStartupAttempt, which sends SIGTERM to D. Client B, whose own child exited busy, joined in observeContendingDaemon with a later deadline. B saw D as the lock owner, probed it as reachable and returned it for reuse. B's request now goes to a daemon that A is stopping. The probe-throw branch near line 806 retires a published launch the same way. This is likely on slow hosts, such as cold CI, where #3129's preflight already reports daemon_startup_failed. I traced this from the code and did not reproduce it.

The rule the code must satisfy is this: once a launch has published a registration that any client can observe and reuse, it is shared. Neither its launcher's deadline nor its launcher's probe result may retire it. Only the takeover policy that every client applies may do that. Please enforce this in one place, the launch-retirement step retireStartupAttempt, which owns both sites. Before it stops anything, it should read the registration with readRegisteredDaemonOwnership(infoPath, launch identity). On 'match', it should report the timeout or error without retiring. It should retire only launches that never published. Please add a regression test where the launched fixture publishes after the deadline and a second client adopts it, and assert that the winner pid stays alive.

Could the join be smaller? It adds a client-side rule for who may reuse or retire a published registration, in registrationAllowsDaemonObservation, isRegistrationAvailable and classify. Would one owner question on the shared registration module from #3116 (src/daemon-registration*) work instead? It would return published-by-launch, held-by-other, released or unproven. Both the waitForDaemonStartup branches and the readReusableLocalDaemon gate would use it, and the fix above becomes "retire only an unpublished launch" in that one function. For this, the stopAndRetireDaemon boundary from #3129 would first need to expose a published-versus-unpublished observation, or accept an "only if unpublished" mode.

Not blocking, and you can take or leave these: in the 'held' case of daemon-client-startup-race.test.ts the virtual clock moves only 100ms per mocked sleep, so the loop runs about 150 real iterations with synchronous ps reads, and it passed the 5000ms timeout under coverage, so a larger step (for example max(ms, 1000) capped at the deadline) or an explicit timeout would fix it while keeping the now-started === 15000 assertion; and registrationAllowsDaemonObservation at line 217 rebuilds the pid and start-time match that readRegisteredDaemonOwnership already owns, and it treats two null start times as equal where the owner refuses that, so calling the owner and admitting only 'match' (or an absent lock) would keep one rule.

The Coverage job failed on this PR's own new test 'startup uses one deadline when the claim is held' with the 5000ms timeout, so that failure is related. In Smoke iOS, daemon startup succeeded in 134ms and then the open request timed out at 91s. That matches #3125 and sits in the iOS open and runner path, which this diff does not touch. I did not inspect the second Smoke failure or the cancelled jobs.

I did not run the mutation controls or the startup-race suite, so I judged the tests by reading them. I also did not trace whether a graceful SIGTERM lets B's in-flight request finish. I took the retirement internals from #3129 and #3127 as reviewed in those PRs.

Before merge, the retire-only-unpublished fix and its regression test need to land. After that, the Coverage timeout needs a fix, and the two open inline threads still stand.

@thymikee
thymikee added this pull request to stack #3146 October 3, 2026 08:46
@thymikee
thymikee force-pushed the fix/daemon-client-startup branch from 359e453 to ac2217c Compare October 3, 2026 14:40
@thymikee
thymikee force-pushed the fix/daemon-startup-contention branch from 180ee97 to 6e44617 Compare October 3, 2026 14:40
@thymikee
thymikee force-pushed the fix/daemon-client-startup branch from ac2217c to c4f6ab8 Compare October 3, 2026 16:41
@thymikee
thymikee force-pushed the fix/daemon-startup-contention branch 2 times, most recently from 83de7a2 to 745589d Compare October 3, 2026 17:49
@thymikee
thymikee force-pushed the fix/daemon-client-startup branch from c4f6ab8 to 34c1f64 Compare October 3, 2026 17:49
@thymikee
thymikee removed this pull request from stack #3146 October 3, 2026 19:38
@thymikee
thymikee force-pushed the fix/daemon-startup-contention branch from 745589d to d2c7a19 Compare October 3, 2026 19:39
@thymikee
thymikee force-pushed the fix/daemon-client-startup branch from 34c1f64 to 3e5be9a Compare October 3, 2026 19:39
@thymikee
thymikee added this pull request to stack #3187 October 3, 2026 19:45
@thymikee
thymikee removed this pull request from stack #3187 October 3, 2026 20:58
@thymikee

thymikee commented Oct 3, 2026

Copy link
Copy Markdown
Member Author

Consolidated into #3127 as part of reducing #3116 to seven PRs. The composition preserves the complete pre-consolidation source tree, including tests and later review corrections. This PR is superseded; its review discussion and native evidence remain available. Outstanding findings transfer to the owning keeper in the implementation record.

@thymikee thymikee closed this Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant