Skip to content

fix: guard eBPF map iteration until tracer.Run loads the collection#294

Merged
blue4209211 merged 2 commits into
mainfrom
fix/startup-race-ebpf-stats-ticker
Jul 15, 2026
Merged

fix: guard eBPF map iteration until tracer.Run loads the collection#294
blue4209211 merged 2 commits into
mainfrom
fix/startup-race-ebpf-stats-ticker

Conversation

@mayankpande88

Copy link
Copy Markdown
Contributor

Problem

Registry.handleEvents is started before tracer.Run() (it must be — init() replays an event per existing pid into the channel, which can exceed the 2000 buffer on busy nodes). But its 5s ebpfStatsTicker can fire before Run has loaded the eBPF collection, and updateEbpfStatsAndActiveConns then dereferences a nil map:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x4d0102d]
github.com/coroot/coroot-node-agent/ebpftracer.(*Tracer).ActiveConnectionsIterator(...)
github.com/coroot/coroot-node-agent/containers.(*Registry).updateEbpfStatsAndActiveConns

Observed today on prod (0.1.2 rollout): a pod on a burstable spot node (t3.2xlarge) crashed 11 times with exit code 2 — losing the race on every start until the node warmed up — then ran stable. Pre-existing since the ticker was introduced in #219; affects 0.1.0+.

Fix

  • Tracer gets an atomic.Bool ready flag, stored after ebpf() loads the collection (the atomic store/load also gives the cross-goroutine happens-before for the collection pointer)
  • updateEbpfStatsAndActiveConns returns early until tracer.Ready() — it runs again 5s later, so nothing is lost

Other iterator callers were audited: only updateEbpfStatsAndActiveConns touches collection maps from outside the tracer's own lifecycle.

handleEvents starts before tracer.Run so it can drain the events channel
while init() replays existing pids. But its 5s ebpfStatsTicker can fire
before the eBPF collection is loaded on slow nodes (cold verifier,
burstable instances), and updateEbpfStatsAndActiveConns then panics:

  panic: runtime error: invalid memory address or nil pointer dereference
  ebpftracer.(*Tracer).ActiveConnectionsIterator(...)
  containers.(*Registry).updateEbpfStatsAndActiveConns

Seen in prod as a burst of restarts (exit code 2) on a spot t3.2xlarge
until the node warmed up. Present since the ticker was introduced (#219).

Add an atomic ready flag set once the collection is loaded, and skip the
stats pass until it is.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a readiness check using an atomic boolean in the eBPF tracer to prevent accessing and iterating eBPF maps before they are fully loaded, which avoids a nil dereference panic in the container registry. The reviewer suggests a safer approach where the tracer is marked as ready only after its initialization successfully completes, preventing potential race conditions and incorrect state on initialization failure.

Comment thread ebpftracer/tracer.go Outdated
@blue4209211 blue4209211 merged commit 5e1f9fc into main Jul 15, 2026
7 checks passed
@blue4209211 blue4209211 deleted the fix/startup-race-ebpf-stats-ticker branch July 15, 2026 09:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants