Skip to content

Recover inconsistent BMP peer state with opt-in periodic renewal - #14

Draft
nuclearcat wants to merge 7 commits into
mainfrom
fix/bmp-automatic-reconciliation
Draft

nuclearcat wants to merge 7 commits into
mainfrom
fix/bmp-automatic-reconciliation

Conversation

@nuclearcat

@nuclearcat nuclearcat commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #11.

A missing or misaddressed BMP Peer Down can leave an old peer and its routes connected indefinitely. Renew the affected speaker's BMP connection when an unmatched Peer Down, possible unnumbered-peer replacement, or peer-state capacity limit indicates inconsistent state. Rebuild from the exporter's current snapshot without guessing which same-router-ID peer to delete.

Healthy feeds have no periodic renewal timer by default (reconciliation.interval_secs = 0). Operators can opt into periodic snapshots by setting a nonzero interval, such as 21600 for six hours plus up to 10% jitter. Anomaly/capacity recovery remains enabled. Startup grace protects legitimate parallel peers during replay, early anomalies coalesce until the minimum session lifetime, and the capacity guard takes precedence.

The underlying exporter defect should be fixed by preserving the established session's identity through Peer Down. Correctly addressed notifications use ordinary peer cleanup without a BMP reconnect. This PR provides recovery, not a fix for FRR. With periodic renewal disabled, a silently missing Peer Down with no subsequent anomaly may remain undetected until a later reconnect. Per-peer BMP statistics can supply activity evidence, but they are optional; this PR does not infer expiry from missing statistics.

Recovery withdraws that speaker's monitored routes until replay restores them, producing a monitoring gap and full-table replay cost. It does not reset router BGP sessions. The exporter must reconnect and replay current state. The peer-state cap is not a total memory limit; GC and downstream backpressure affect reclamation. See docs/bmp-tcp-in.md for configuration and RFC 7854 references.

Validation:

  • BMP Rust tests: 65 passed, 16 ignored, including default idle/partial transports surviving eight simulated days and opt-in periodic renewal/redial.
  • Daemon build passed; existing deprecation/dead-code warnings remain.
  • Eight live simulations passed, with 20 replacements per churn case: healthy, matching Peer Down, misaddressed Peer Down, missing Peer Down with replacement, parallel peers, opt-in silent/partial recovery, and capacity recovery. Matching Peer Down required no additional snapshot replay.
  • Python syntax, Rust formatting, and git diff --check passed.

Assisted-by: OpenAI Codex

Copilot AI balanced review requested due to automatic review settings October 2, 2026 18:31
Comment thread scripts/bmp-peer-churn.py Fixed
Comment thread scripts/bmp-peer-churn.py Fixed

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

It introduces an always-on, non-disableable behavior change to core BMP processing that periodically forces full-table replay for every exporter, with broad operational impact that warrants human judgment despite the clean, well-tested implementation.

Review effort: Balanced
Findings: None

What changed in this PR

This PR addresses issue #11, where a misaddressed or missing BMP Peer Down (e.g. FRR reporting :: after an unnumbered peer loses its link-local address) leaves a stale peer and its Adj-RIB-In connected indefinitely. Rather than guessing which same-router-ID peer to delete, it introduces an always-on "reconciliation" mechanism: when an anomaly is detected (unmatched Peer Down, possible unnumbered-peer replacement, peer-state capacity reached, or a periodic/idle timer), the speaker's BMP transport is closed so the exporter replays an authoritative snapshot on reconnect. The logic lives in a new reconciliation module wired into the per-connection RouterHandler read loop and the BMP state machine.

Changes:

  • New reconciliation module: validated Config (bounds enforced via TryFrom, cannot be disabled), jittered periodic deadline, and anomaly/capacity deadline coalescing.
  • State-machine reconciliation_reason classifies Peer Up/Down against logical peer identity (type/distinguisher/ASN/BGP-id, excluding policy flags) and exposes peer_state_count; the read loop times out on the deadline and closes the transport for a fresh snapshot.
  • Config plumbing through BmpTcpIn/BmpTcpInRunner/RouterHandler (new connections only), plus extensive Rust tests, a Python churn simulator, and documentation.
File Description
src/​units/​bmp_tcp_in/​reconciliation.rs New module: config validation, jitter, deadline/observe logic, unit tests.
src/​units/​bmp_tcp_in/​state_machine/​machine.rs Adds peer_states/peer_state_count/reconciliation_reason for anomaly detection.
src/​units/​bmp_tcp_in/​router_handler.rs Read loop enforces reconciliation deadline/timeout, observes per-message, drops transport.
src/​units/​bmp_tcp_in/​unit.rs Threads reconciliation config through construction/reconfigure; adds active redial test.
src/​units/​bmp_tcp_in/​mod.rs Exposes the new reconciliation module.
src/​units/​bmp_tcp_in/​state_machine/​tests.rs Tests for identity scoping and non-destructive detection of parallel peers.
scripts/​bmp-peer-churn.py New stdlib-only simulator exercising seven reconciliation scenarios.
docs/​bmp-tcp-in.md Documents triggers, defaults, RFC 7854 basis, and the monitoring-gap/replay caveat.
TESTING.md Describes the simulator and Rust regression suite.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@nuclearcat
nuclearcat marked this pull request as draft October 4, 2026 11:16
Copilot AI balanced review requested due to automatic review settings October 20, 2026 16:39

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It is a large change that alters critical BMP transport/connection lifecycle behavior in the production daemon and adds a ~3,500-line decoder, warranting human review despite strong test coverage.

1 open issue

Review effort: Balanced

Comment thread Changelog.md Outdated
Comment on lines +30 to +33
* Added `netom-bmpdump`, an offline BMP/BGP/EVPN decoder for PCAP and PCAPNG
captures, with TCP reassembly, readable/colorized output, filters, and
JSON export. The standalone workspace package is included in the
DEB/RPM packages and runtime images.
Copilot AI balanced review requested due to automatic review settings October 20, 2026 17:29
@nuclearcat
nuclearcat force-pushed the fix/bmp-automatic-reconciliation branch from 52e4e2c to f7c1830 Compare October 20, 2026 17:29

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It introduces an always-on, non-disableable behavior change to core BMP handling that periodically tears down every speaker's connection and withdraws/replays its full table, so the operational monitoring-gap and load implications warrant human review despite the thorough tests.

2 open issues

Review effort: Balanced

Comment thread docs/bmp-tcp-in.md Outdated
Comment on lines +65 to +67
No opt-in is needed. The former experimental `recover_unnumbered_peers` heuristic
has been replaced: Netom does not infer that a shared BGP identifier authorizes
it to delete another peer.
Copilot AI balanced review requested due to automatic review settings October 20, 2026 17:50
@nuclearcat nuclearcat changed the title Fix stale BMP peer retention with automatic snapshot reconciliation Recover inconsistent BMP peer state with opt-in periodic renewal Oct 20, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It introduces a substantial new recovery mechanism in the core BMP ingestion path that intentionally drops transports and withdraws/replays full routing tables, carrying operational impact that warrants human review.

3 open issues

Review effort: Balanced

Comment thread docs/bmp-tcp-in.md Outdated
Comment on lines +65 to +68
and `connect`. Periodic renewal is disabled by default. The former experimental
`recover_unnumbered_peers` heuristic
has been replaced: Netom does not infer that a shared BGP identifier authorizes
it to delete another peer.
Copilot AI balanced review requested due to automatic review settings October 20, 2026 18:02

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread docs/bmp-tcp-in.md Outdated
Comment on lines +65 to +68
and `connect`. Periodic renewal is disabled by default. The former experimental
`recover_unnumbered_peers` heuristic
has been replaced: Netom does not infer that a shared BGP identifier authorizes
it to delete another peer.
Copilot AI balanced review requested due to automatic review settings October 20, 2026 18:10

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes core BMP state handling to force transport renewal and withdraw/replay a speaker's full route table, so its operational impact warrants final human review despite strong test coverage.

4 open issues

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unnumbered peer replacement leaves the old link-local Adj-RIB-In connected - implicit Peer Down feature?

2 participants