feat: stamp node_version and node_commit on every json log line - #225
Conversation
Every document in production Elasticsearch carries `tag.node_version = "unknown"`, so the build a node is running cannot be read from telemetry at all. That made the 0.18.1 staged rollout unreadable: fleet-wide `relay recv stream closed` errors rose 6x across the window, which looked like a regression in the build that fixed them, and only splitting by the build each service was actually running (inferred from process start time) showed the new build was ~4x better. The tag is a static Telegraf `[global_tags]` value set at provisioning time and never passed by any caller. Even if it were passed, a provisioning-time string can never be right across an auto-upgrade, so the process itself has to report its build. `WithBuildInfo` wraps the JSON event formatter: it renders the inner event into a thread-local buffer and replaces the closing brace with a compile-time `,"node_version":"<CARGO_PKG_VERSION>","node_commit":"<ANT_GIT_COMMIT>"}` tail. Cost is one memcpy per line on top of the serialisation already done. tracing-subscriber's JSON formatter has no hook for constant fields and span fields do not survive `tokio::spawn`, hence the wrapper. Both JSON sinks (stdout and rolling file) use it; the text format and the startup line's `version`/`commit` fields are unchanged, so the beta forwarder in ant-client keeps working. After an auto-upgrade the new process stamps its own constants, so the value is correct on both sides of the restart by construction. Telegraf promotes the fields to tags via `tag_keys` in saorsa-testnet-registry; until that config is re-provisioned the values still land as top-level `node_version` / `node_commit` fields. Test evidence: - 3 new unit tests (fields kept, stamp after a nested `span` object, version is semver) - cargo test: 1088 lib + 16 integration passed, 0 failed - smoke run in development mode: 295/295 stdout lines and 81/81 file-sink lines carried `node_version=0.18.1 node_commit=31fcbae`, 0 invalid JSON; text format byte-identical - clippy clean under both the CLAUDE.md deny set and CI's `--all-targets -D warnings`; `--no-default-features` still builds Closes V2-1153 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MYvuVpFYXXNrY9YFVBswvp
dirvine
left a comment
There was a problem hiding this comment.
REQUEST_CHANGES — Documentation CI job is red due to a doc lint error introduced by this PR.
cargo doc fails under -D warnings:
error: public documentation for `WithBuildInfo` links to private item `BUILD_INFO_TAIL`
--> src/logging.rs:142:7
142 | /// [`BUILD_INFO_TAIL`], and the result is copied to the real writer. Cost is
| ^^^^^^^^^^^^^^^^ this item is private
note: `-D rustdoc::private-intra-doc-links` implied by `-D warnings`
The new public WithBuildInfo struct's doc comment intra-doc-links the private BUILD_INFO_TAIL const (src/logging.rs:142). stamp_build_info (line 193) is fine because it's private. Your local clippy runs don't catch this — the cargo doc build is a separate CI job.
Fix (one line): change the link to plain backticks at line 142:
/// `BUILD_INFO_TAIL`, and the result is copied to the real writer. Cost is
The logic itself (WithBuildInfo wrapper, re-entrancy fallback, empty-object/nested-span splice handling, both sinks via json_event_format, text format unchanged) all looks correct and the tests cover the edge cases well. Just need the doc link fixed so CI is green — then this is good to go.
The public `WithBuildInfo` docs linked the private const, which `cargo doc` rejects under `-D warnings` (`rustdoc::private-intra-doc-links`) in the documentation CI job. Plain backticks carry the same meaning without the link. Verified locally with `RUSTDOCFLAGS="-D warnings" cargo doc --all-features --no-deps`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MYvuVpFYXXNrY9YFVBswvp
|
Fixed in 1c892f9 — the |
dirvine
left a comment
There was a problem hiding this comment.
APPROVE — re-review complete.
The doc-lint fix is confirmed in place: line 142 now uses plain backticks ( BUILD_INFO_TAIL ) instead of the intra-doc link to the private const, and the Documentation CI job is green.
Remaining state:
- Format, Clippy, Documentation, Test (no logging), linear-link, pr-template, self-test — all SUCCESS.
- Security Audit is red, but it's a Cargo.lock dependency advisory (RUSTSEC-2026-0285), not something this PR introduced — the PR only touches
src/bin/ant-node/main.rsandsrc/logging.rswith no dependency changes. Pre-existing and independent of this change; not a blocker here.
The logging change itself is sound (verified in the prior pass): WithBuildInfo stamps node_version/node_commit on every JSON line via both sinks, with correct handling of re-entrancy, empty objects, and nested span objects, and text format untouched. Good to go.
Linear issue
Closes V2-1153
Risk tier
Proposed T1: the only behavioural change is two extra fields on each JSON log line. No wire, storage, payment, routing or upgrade-mechanism change.
Compatibility
node_versionandnode_commitfields; the text format and the startup line's existingversion/commitfields (parsed by the ant-client beta forwarder) are unchanged.Semver impact
Summary
Every document in production Elasticsearch carries
tag.node_version = "unknown", so the build a node is running cannot be read from telemetry. That made the 0.18.1 staged rollout unreadable: fleet-widerelay recv stream closederrors rose 6x across the window, which looked like a regression in the build that fixed them; only splitting by the build each service was actually running (inferred from process start time) showed the new build was ~4x better (V2-986).The tag is a static Telegraf
[global_tags]value set at provisioning time insaorsa-testnet-registryand never passed by any caller. Even if it were passed, a provisioning-time string can never be right across an auto-upgrade, so the process itself has to report its build.ant_node::logging::WithBuildInfowraps the JSON event formatter: it renders the inner event into a thread-local buffer and replaces the closing brace with a compile-time,"node_version":"<CARGO_PKG_VERSION>","node_commit":"<ANT_GIT_COMMIT>"}tail. One memcpy per line on top of the serialisation already done; re-entrancy falls back to a freshStringrather than panicking; non-object output passes through untouched.tokio::spawn, so a root span would miss most lines.json_event_format()inmain.rs.execs the new binary or exits for the service manager to restart it, so the new process stamps its own constants — correct on both sides of the restart by construction.Companion change: jacderida/saorsa-testnet-registry adds
node_version,node_committo Telegraftag_keysso the fields land astag.node_version.keyword/tag.node_commit.keywordand beat the static global tag. Until that config is re-provisioned on ant-prod-01, the values still land as top-levelnode_version/node_commitfields, so they are queryable either way. Thebootstrap-upgrade-check/track-upgrades-setupskill notes sayingtag.node_versionis static remain true until both have rolled out.Test evidence
src/logging.rs: every line is valid JSON and keepsmessage/level/target/custom fields; the stamp lands after a nested"span":{...}object;node_versionequalsCARGO_PKG_VERSIONand parses as semver.cargo test: 1088 lib + 16 integration passed, 0 failed.node_version=0.18.1 node_commit=31fcbae, 0 invalid JSON (jq-checked); text format byte-identical.cargo clippy --all-features -- -D clippy::panic -D clippy::unwrap_used -D clippy::expect_usedclean; CI'scargo clippy --all-targets --all-features -- -D warningsclean;cargo build --no-default-featurescompiles.Sample line:
{"timestamp":"2026-09-11T23:58:07.002929Z","level":"INFO","message":"Bootstrap peers provided via CLI","count":1,"target":"ant_node","node_version":"0.18.1","node_commit":"31fcbae"}New dependency
none
ADR
n/a
Mitigation / rollback
Revert this commit; the two fields disappear and Telegraf's static global tag takes over again. No stored data or peers are affected.
🤖 Generated with Claude Code
https://claude.ai/code/session_01MYvuVpFYXXNrY9YFVBswvp