Recover inconsistent BMP peer state with opt-in periodic renewal

Changelog.md

@@ -27,6 +27,17 @@ Released yyyy-mm-dd.
 
 ### New
 
+* `bmp-tcp-in` recovers inconsistent peer state after an unmatched Peer Down,
+  a possible unnumbered-peer replacement, or the peer-state capacity limit by
+  renewing the affected speaker's BMP connection and replaying its snapshot.
+  This addresses stale peer retention described in issue #11 without guessing
+  which same-router-ID peer to delete. The `reconciliation` configuration
+  controls cooldown, startup grace, capacity, and optional periodic renewal.
+  Periodic renewal is disabled by default (`interval_secs = 0`).
+  Recovery temporarily withdraws the speaker's monitored routes and incurs a
+  full-table replay; it does not reset the router's BGP sessions. Exporters
+  must reconnect and provide current state. See [BMP input](docs/bmp-tcp-in.md).
+
 * `netom-cli show ip bgp detail` (and `show ipv6 bgp detail`) prints every
   attribute of each path: peer and AS, how it was learned (ingress, BMP,
   pre/post-policy, ADD-PATH path id), status and RPKI, origin, AS path,

TESTING.md

@@ -80,3 +80,84 @@ validity states.
 
 The `e2e-flowspec` GitHub workflow runs both scripts on demand
 (workflow_dispatch).
+
+## BMP reconciliation simulations
+
+`python3 scripts/bmp-peer-churn.py` builds and runs isolated real daemons, then
+acts as a BMP exporter over loopback TCP. No FRR, containers, external data, or
+third-party Python packages are required. It exercises the automatic recovery
+for [issue #11](https://github.com/FastNetMon/netom/issues/11).
+
+```sh
+python3 scripts/bmp-peer-churn.py --binary target/debug/netom
+python3 scripts/bmp-peer-churn.py --binary target/debug/netom --cycles 20
+```
+
+The exporter keeps an independent list of actually established peers. After a
+collector-initiated disconnect, it reconnects and replays only those peers and
+their routes, including End-of-RIB. Retired identities are not replayed.
+
+Eight scenarios run by default; select one with `--scenario NAME`:
+
+- `healthy`: an idle established peer stays connected with periodic renewal off.
+- `matching`: correctly addressed Peer Down, ordinary cleanup control.
+- `mismatched`: Peer Down with `::`, then a new link-local peer address.
+- `missing`: a new peer address with no preceding Peer Down.
+- `parallel`: legitimate parallel peers coexist in startup; adding another
+  later causes one reconciliation, and all three survive the new snapshot
+  beyond its startup window without another anomaly reset.
+- `silent`: a peer vanishes without any further BMP message; opt-in periodic renewal
+  removes it after the authoritative empty replay.
+- `partial`: the same disappearance with a stalled, incomplete BMP frame.
+- `limit`: startup churn reaches the peer-state cap; reconnect restores the
+  sole actually established peer.
+
+Only `silent` and `partial` enable a periodic timer. Other scenarios leave
+`interval_secs` unspecified to exercise the production default of no timer.
+The script shortens reconciliation timings in the test configuration and GC
+intervals in the test daemon's environment. It verifies current peer addresses,
+retained peer entries and route records after GC, then checks complete cleanup
+when the exporter closes. The table is constant-size during churn. A timeout,
+missing metric, failed replay, unexpected parallel-session reset, or retained
+state makes the script exit 1; success exits 0. These are object-count checks,
+not RSS thresholds: allocator caching can retain pages after objects are freed.
+
+Rust regressions are part of the normal suite:
+
+```sh
+cargo test --lib units::bmp_tcp_in
+```
+
+They cover default transports staying open past eight simulated days,
+deadline coalescing, startup replay, opt-in periodic jitter, capacity,
+configuration validation, peer identity scopes, policy/ADD-PATH teardown,
+idle/partial transport cancellation, and active-connector redial. Live network
+tests require permission to bind loopback sockets.
+
+See [automatic reconciliation](docs/bmp-tcp-in.md#automatic-reconciliation-and-stale-peer-retention)
+for the RFC 7854 references, defaults, and monitoring-gap/replay caveat.
+
+Validation on 2026-10-20 (periodic renewal opt-in):
+
+- `cargo test --locked --lib units::bmp_tcp_in`: 65 passed, 16 ignored.
+- `cargo build --locked --bin netom`: passed with existing deprecation and
+  unused `set_rib` warnings.
+- `python3 scripts/bmp-peer-churn.py --binary target/debug/netom --cycles 20`:
+  all eight scenarios passed. The healthy and matching-Peer-Down scenarios
+  required only their initial snapshot; matching cleanup reclaimed all retired
+  identities without a BMP reconnect. Anomaly and capacity cases recovered with
+  periodic renewal off. Silent and partial cases recovered with it opted in.
+- Changed Rust files passed rustfmt checks; Python syntax and
+  `git diff --check` passed.
+
+Validation on 2026-10-02 (implementation `0ec9bd6`, regressions `a816a12`):
+
+- `cargo test --lib --offline`: 354 passed, 31 ignored, no failures.
+- `cargo build --bin netom --offline`: passed (existing unused `set_rib` warning).
+- `python3 scripts/bmp-peer-churn.py --binary target/debug/netom --cycles 20`:
+  all seven scenarios passed. Each churn scenario ended with one peer and one
+  record after GC; the parallel replay retained all three peers/records; silent
+  and partial-frame recovery reclaimed all obsolete peers/records; capacity
+  recovery restored the one current peer. Every final disconnect cleaned up.
+- Changed Rust files passed rustfmt checks; the Python driver passed syntax
+  compilation; `git diff --check` passed.

docs/bmp-tcp-in.md

@@ -56,9 +56,113 @@ connect, handshake, idle reads and retry delays. Downstream backpressure can
 still delay cleanup; the daemon's shutdown/drain rules apply.
 
 The existing filtering and tracing settings apply to active input as well.
-`ignore_post_policy_routes` and `forward_raw_updates` changes apply to the next
+`ignore_post_policy_routes`, `forward_raw_updates`, and `reconciliation` changes apply to the next
 connection. Listening address changes preserve existing accepted connections.
 
+## Automatic reconciliation and stale-peer retention
+
+Anomaly and capacity recovery are enabled for every BMP input, for both `listen`
+and `connect`. Periodic renewal is disabled by default. Netom does not delete
+an existing peer solely because a new peer shares its BGP identifier; legitimate
+parallel sessions can share that identifier.
+
+An exporter can omit Peer Down or send it with an address that no longer matches
+Peer Up. [Issue #11](https://github.com/FastNetMon/netom/issues/11) describes FRR
+reporting `::` after losing an unnumbered peer's old link-local address. The
+collector otherwise retains that peer as connected, including all of its routes.
+GC cannot reclaim state that still appears live.
+
+The exporter should preserve the established session's identity through teardown
+and send Peer Down with the same peer address used for Peer Up. Correctly addressed
+Peer Down notifications use ordinary per-peer cleanup without restarting BMP.
+Reconciliation is a fallback for inconsistent observations, not an exporter fix.
+
+Netom repairs the observation by closing **that speaker's BMP transport** and
+rebuilding from a fresh snapshot on reconnection. It uses normal disconnect
+processing to withdraw every peer, policy view and ADD-PATH child owned by the
+connection. Their disconnected register entries and records are reclaimed by
+normal RIB GC if they are not rebound. Other speakers' state is not withdrawn.
+
+This follows [RFC 7854 §3.2](https://www.rfc-editor.org/rfc/rfc7854.html#section-3.2):
+a collector restarts monitoring by dropping the TCP connection. Under
+[§3.3](https://www.rfc-editor.org/rfc/rfc7854.html#section-3.3), reconnection starts
+with Initiation, Peer Up for each established monitored peer, and its current
+Adj-RIB-In. [§5](https://www.rfc-editor.org/rfc/rfc7854.html#section-5) describes
+that snapshot and per-peer End-of-RIB markers. BMP does not provide a collector
+request to refresh one peer, or a global marker proving all peers have been
+listed.
+
+Refresh triggers and default bounds:
+
+| Condition | Behavior |
+| --- | --- |
+| Accepted Peer Down cannot match a logical peer (ignoring policy flags) | Schedule reconciliation, no earlier than 60 seconds after connection start. Repeated anomalies do not postpone it. |
+| A new link-local address has the same ASN, BGP identifier, peer type and distinguisher as an existing link-local peer | After the startup window, reconcile the whole speaker; do not guess which session is obsolete. |
+| Peer Ups arrive within the first 300 seconds | Accept parallel identities as part of startup/replay. Possible-replacement detection is suppressed; unmatched Peer Down and capacity protection still apply. |
+| No detectable anomaly, including a completely silent connection | Keep the connection open by default. If `interval_secs` is nonzero, renew after that interval plus independently selected 0–10% jitter. |
+| The connection reaches 65,536 FSM peer states, including policy views | Close immediately, even within the minimum session lifetime. This is a per-connection peer-state limit, **not a total process-memory or route-count limit**. |
+
+The first anomaly after the minimum lifetime can close the connection immediately.
+Anomalies early in a new connection are coalesced until that minimum. Replay
+collisions within the startup window do not cause a new reset, so legitimate
+parallel sessions in an ordinary fresh snapshot survive. A timer also interrupts
+an idle transport or a partial BMP frame; a cancelled partial read is discarded
+with its transport and is never resumed.
+
+Default settings (anomaly/capacity recovery without periodic renewal):
+
+```toml
+[units.bmp-in]
+type = "bmp-tcp-in"
+listen = "0.0.0.0:11019"
+reconciliation = { interval_secs = 0, min_session_secs = 60, replay_grace_secs = 300, max_peer_states = 65536 }
+```
+
+To opt into a periodic snapshot every six hours plus jitter, set
+`interval_secs = 21600`. Setting it to `0` disables only periodic renewal.
+Validation requires `0 < min_session_secs <= replay_grace_secs < 604800` and
+`max_peer_states > 0`. A nonzero interval must satisfy
+`replay_grace_secs < interval_secs <= 604800` (seven days before jitter).
+Changes take effect on the next BMP connection. Unmatched messages rejected by
+a Roto filter do not trigger anomaly reconciliation; capacity protection and
+any configured periodic timer remain active.
+
+### Operational caveat
+
+This recovery has a cost: it withdraws the speaker's routes from Netom and its
+downstream consumers until replay restores them. It does **not** reset the
+router's BGP sessions, but it causes a monitoring gap and a full-table replay,
+which consumes exporter CPU, bandwidth, and collector work. Consumers may see
+withdrawal/reannouncement events. This is not an atomic or gap-free snapshot
+replacement, and downstream backpressure can delay withdrawal/GC processing.
+
+For passive inputs, the exporter must reconnect. For active inputs, Netom's
+normal retry/backoff reconnects. Replay depends on the exporter supplying a
+current snapshot; an incremental-only exporter cannot restore missing routes,
+and an exporter that replays obsolete peers cannot be corrected by reconciliation.
+Without an independently configured liveness expectation, silence cannot
+distinguish an idle live peer from a dead peer whose Peer Down was never sent.
+Netom never deletes a peer merely for having no updates. BMP statistics reports
+can provide per-peer activity evidence, but are optional and have no negotiated
+heartbeat interval. This implementation does not infer peer expiry from missing
+statistics. A silent disappearance, including one missed during startup, may
+remain visible until a later anomaly, disconnect, or explicitly enabled periodic
+renewal obtains a new snapshot.
+
+The startup window is a conservative time allowance, not proof that replay has
+finished. Size it for exporters with slow or very large initial dumps; parallel
+Peer Ups arriving after it can cause an additional refresh. A legitimate feed
+above the peer-state limit will repeatedly hit that limit until capacity is
+raised or the feed is split. The hard limit overrides cooldown; the active
+connector/exporter's reconnect backoff must still be respected. Neither the
+peer limit nor periodic renewal promises a fixed byte limit for all collector
+memory. Physical reclamation also waits for the normal RIB GC sweeps.
+
+Each refresh logs its reason and speaker address. `scripts/bmp-peer-churn.py`
+exercises current-state replay, disappearance, parallel peers, partial frames,
+and capacity protection; see
+[TESTING.md](https://github.com/FastNetMon/netom/blob/main/TESTING.md#bmp-reconciliation-simulations).
+
 ## History and initial data
 
 TCP and TLS use the same BMP parser, peer identities, ADD-PATH handling and