fix(iac): raise neighbour table thresholds on worker nodes - #3508
fix(iac): raise neighbour table thresholds on worker nodes#3508tomassrnka wants to merge 1 commit into
Conversation
PR SummaryLow Risk Overview Reviewed by Cursor Bugbot for commit 88417a9. Bugbot is set up for automated code reviews on this repo. Configure here. |
Worker nodes log "neighbour: arp_cache: neighbor table overflow!" and new neighbour entries fail with ENOBUFS. No neigh sysctl was set anywhere in the repo, so every host ran the kernel defaults of 128/512/1024. Each running sandbox puts one veth end in the host root netns with an on-link /31 and a /32 route, i.e. one ARP entry plus one IPv6 ND entry per slot. At max-sandboxes-per-node=200 plus the 131 pre-warmed pool slots that is ~330 entries steady state, before infra peers and not-yet-collected stale entries, which crosses gc_thresh2=512 under churn and reaches gc_thresh3=1024. Raise both the ARP and NDP tables to 4096/8192/16384 in the existing sysctl block on the GCP and AWS client node bootstrap scripts.
fea0e5a to
88417a9
Compare
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
LGTM — straightforward sysctl tuning to raise ARP/NDP neighbour table thresholds on worker node boot.
Extended reasoning...
Overview
This PR appends six net.ipv4/ipv6.neigh.default.gc_thresh{1,2,3} sysctl keys (4096/8192/16384) to the existing sysctl block in both the AWS and GCP worker-node bootstrap scripts (start-client.sh), applied via sysctl -p at boot. No application code, Terraform resources, or Nomad job specs are touched — it's purely additive lines within an existing heredoc block that already sets several similar generous kernel limits (somaxconn, tcp_max_syn_backlog, etc.).
Security risks
None. This does not touch auth, crypto, permissions, or any request-handling code path. Raising neighbour table capacity is a resource-limit change, not a trust-boundary change.
Level of scrutiny
Low. This is a bootstrap-only sysctl change, mechanical and consistent with the pattern already established in the same file for other kernel tunables. The PR description includes a clear root-cause analysis (host-root-netns veth entries from sandboxes/pool slots crossing the kernel's default gc_thresh2/3) and reports concrete reproduction/verification (fill tests before and after the change, confirming sysctl -p applies cleanly and returns 0). The bug-hunting system found no issues, and the one candidate concern raised (whether an unrecognized key could abort sysctl -p under set -e) was investigated and ruled out.
Other factors
Change is symmetric across both cloud providers, follows the existing formatting/comment style in the file, and is scoped only to newly-created nodes (no live rollout risk to existing infrastructure). No outstanding reviewer comments to address.
1 participant
Linear: https://linear.app/e2b/issue/EN-966/neighbor-arp-cache-neighbor-table-overflow
Broken: worker nodes log
neighbour: arp_cache: neighbor table overflow!and neighbour inserts fail with ENOBUFS.Root cause: no ARP/neigh sysctl set anywhere in the repo, so hosts run kernel defaults 128/512/1024. Each sandbox slot costs one ARP + one IPv6 ND entry in the host root netns (
CreateNetworkveth + /31 + /32 route); 200max-sandboxes-per-node(LaunchDarkly-raisable in prod with no code change) + 131 pre-warmed pool slots ≈ 330 steady-state entries before infra peers and stale entries — pastgc_thresh2=512under churn, reachinggc_thresh3=1024.Repro (Linux 6.8, privileged container): at defaults, fresh NUD_STALE adds fail at 1017–1019 with
RTNETLINK answers: No buffer space availableand the exact ticket line in dmesg; IPv6 (ndisc_cache) fails identically at 1024. Forced GC can't reclaim entries younger thangc_interval(30 s), so a creation burst overflows. Two flags: the counter is global, not per-netns (filling from a child netns broke root-netns adds — onearp_tbl/nd_tblper kernel), andneigh/defaultonly exists in the init netns, so the root-netns setting is the only lever and governs every namespace. IPv6 included because nothing setsdisable_ipv6and every veth gets a link-local address.Fix: raise thresholds to 4096/8192/16384 on both GCP and AWS (the comparable
vm.dirty_*change bb920f4 only landed on GCP). Sizing: ~330 steady state needs real headroom since the cap is runtime-raisable;gc_thresh1=4096keeps GC off the hot path; 16384 ≈ 50x steady state, ~6 MB per table worst case — consistent with existing generous limits (somaxconn 65535,nf_conntrack_max 2097152).Verification: appending the exact block to
/etc/sysctl.conf+sysctl -p(what the scripts do) returns 0, applies all six keys, and the previously failing 3000-entry IPv4 and IPv6 fills pass with no overflow logged.Rollout: bootstrap sysctl-only — only reaches nodes created after this lands. #2786 (NUD_PERMANENT entries in
Slot.CreateNetwork) was closed unmerged after smoke-test failures; this is the orthogonal, lower-risk fix the ticket itself suggests.