Bound vmcp health and session registry growth - #6450
Open
lorenzozanee wants to merge 1 commit into
Open
Conversation
The health monitor's removedBackends tombstones grew without expiry and the server's healthResyncRegistry retained workers for expired sessions between health-change events. Both maps scaled with backend churn and session churn, violating the invariant that registry size tracks live sessions/backends. Bound the tombstones with a TTL of 2*checkInterval (pruned lazily on read and periodically in UpdateBackends/ performHealthCheck) and filter dead sessions synchronously at fan-out before triggering resync workers. This keeps memory bounded regardless of churn and avoids unnecessary resync work. Fixes stacklok#5860 Signed-off-by: lorenzozanee <wyz0707@proton.me>
lorenzozanee
requested review from
ChrisJBurns,
JAORMX,
amirejaz,
jerm-dro,
jhrozek and
tgrunnagle
as code owners
August 28, 2026 08:09
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
All VirtualMCPServer instances grow 2-3x RSS over ~24h, scaling with backend count (12/13/14/19 backends grew 2.0-3.1x vs flat control proxies). Investigation shows two unbounded in-memory maps:
health.statusTracker.removedBackendskept a tombstone per removed backend forever to avoid in-flight health-check races. Under ID churn it grew linearly without expiry.server.healthResyncRegistrykept a per-sessionlistChangedResyncWorker(holding ClientSession + identity) for sessions that ended via TTL expiry. Pruning only happened on the next async resync, so the registry stayed larger than the live session set between health-change events.What changed:
removedBackendswith a TTL of2*checkInterval(default 60s, 5m fallback). Entries expire lazily onisRemovedand are pruned periodically inUpdateBackendsandperformHealthCheck, and proactively atRemoveBackendinsertion.snapshotnow returns a map,resyncSessionsOnBackendHealthChangechecksGetMultiSessionfor each entry and removes dead sessions before triggering, keeping|workers| == |liveSessions|.Fixes #5860
Type of change
Test plan
task test)task test-e2e)task lint-fix)New regression tests:
pkg/vmcp/server/regression_5860_health_resync_test.go: verifies fan-out prunes dead sessions and only triggers live sessions (fails on base with 4 workers vs 2 live).pkg/vmcp/health/regression_5860_tombstone_test.go: churns 50 distinct backends and asserts tombstones are bounded after TTL (fails on base with 50 entries).Existing
pkg/vmcp/healthandpkg/vmcp/serversuites pass (task testwith-race).API Compatibility
v1beta1API, OR theapi-break-allowedlabel is applied and the migration guidance is described above.Changes
pkg/vmcp/health/status.gomap[string]bool->map[string]time.Timewith TTL, lazy and periodic prunepkg/vmcp/health/monitor.gocheckInterval, prune inUpdateBackends/performHealthCheckpkg/vmcp/server/serve_health_resync.gosnapshotmap, synchronousGetMultiSessionfilter at fan-outpkg/vmcp/server/serve_health_resync_test.gopkg/vmcp/health/regression_5860_tombstone_test.gopkg/vmcp/server/regression_5860_health_resync_test.go