Skip to content

Bound vmcp health and session registry growth - #6450

Open
lorenzozanee wants to merge 1 commit into
stacklok:mainfrom
lorenzozanee:fix/vmcp-per-backend-memory-leak-5860
Open

Bound vmcp health and session registry growth#6450
lorenzozanee wants to merge 1 commit into
stacklok:mainfrom
lorenzozanee:fix/vmcp-per-backend-memory-leak-5860

Conversation

@lorenzozanee

Copy link
Copy Markdown
Contributor

Summary

All VirtualMCPServer instances grow 2-3x RSS over ~24h, scaling with backend count (12/13/14/19 backends grew 2.0-3.1x vs flat control proxies). Investigation shows two unbounded in-memory maps:

  • health.statusTracker.removedBackends kept a tombstone per removed backend forever to avoid in-flight health-check races. Under ID churn it grew linearly without expiry.
  • server.healthResyncRegistry kept a per-session listChangedResyncWorker (holding ClientSession + identity) for sessions that ended via TTL expiry. Pruning only happened on the next async resync, so the registry stayed larger than the live session set between health-change events.

What changed:

  • Bound removedBackends with a TTL of 2*checkInterval (default 60s, 5m fallback). Entries expire lazily on isRemoved and are pruned periodically in UpdateBackends and performHealthCheck, and proactively at RemoveBackend insertion.
  • Filter dead sessions synchronously at health-change fan-out: snapshot now returns a map, resyncSessionsOnBackendHealthChange checks GetMultiSession for each entry and removes dead sessions before triggering, keeping |workers| == |liveSessions|.

Fixes #5860

Type of change

  • Bug fix
  • New feature
  • Refactoring (no behavior change)
  • Dependency update
  • Documentation
  • Other (describe):

Test plan

  • Unit tests (task test)
  • E2E tests (task test-e2e)
  • Linting (task lint-fix)
  • Manual testing (describe below)

New regression tests:

  • pkg/vmcp/server/regression_5860_health_resync_test.go: verifies fan-out prunes dead sessions and only triggers live sessions (fails on base with 4 workers vs 2 live).
  • pkg/vmcp/health/regression_5860_tombstone_test.go: churns 50 distinct backends and asserts tombstones are bounded after TTL (fails on base with 50 entries).

Existing pkg/vmcp/health and pkg/vmcp/server suites pass (task test with -race).

API Compatibility

  • This PR does not break the v1beta1 API, OR the api-break-allowed label is applied and the migration guidance is described above.

Changes

File Change
pkg/vmcp/health/status.go map[string]bool -> map[string]time.Time with TTL, lazy and periodic prune
pkg/vmcp/health/monitor.go Inject checkInterval, prune in UpdateBackends/performHealthCheck
pkg/vmcp/server/serve_health_resync.go snapshot map, synchronous GetMultiSession filter at fan-out
pkg/vmcp/server/serve_health_resync_test.go Adapt coalescing test to new filter path
pkg/vmcp/health/regression_5860_tombstone_test.go New bounded-tombstone regression
pkg/vmcp/server/regression_5860_health_resync_test.go New bounded-registry regression

The health monitor's removedBackends tombstones grew without
expiry and the server's healthResyncRegistry retained workers
for expired sessions between health-change events. Both maps
scaled with backend churn and session churn, violating the
invariant that registry size tracks live sessions/backends.

Bound the tombstones with a TTL of 2*checkInterval (pruned
lazily on read and periodically in UpdateBackends/
performHealthCheck) and filter dead sessions synchronously at
fan-out before triggering resync workers. This keeps memory
bounded regardless of churn and avoids unnecessary resync work.

Fixes stacklok#5860

Signed-off-by: lorenzozanee <wyz0707@proton.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vmcp: all VirtualMCPServer instances grow 2-3x in memory over ~24h regardless of client activity

1 participant