netutils/ptpd: hardware TX timestamps via MSG_ERRQUEUE and egress latency compensation - #3791
Draft
daniel-p-carvalho wants to merge 9 commits into
Draft
daniel-p-carvalho wants to merge 9 commits into
daniel-p-carvalho wants to merge 9 commits into
Conversation
ptp_sendmsg() called a blocking recvmsg(state->tx_socket, ...) right after sending a Delay_Req whenever hardware_ts was set, assuming a Linux-style MSG_ERRQUEUE/loopback semantics NuttX does not have. Since tx_socket and event_socket share the same underlying connection, this call instead blocked on and consumed whatever PTP packet arrived next on the wire — almost always the Delay_Resp, which typically arrives within milliseconds of the request. Its payload was read into a local buffer that went out of scope on return, so the packet never reached ptp_process_rx_packet() and path_delay_ns stayed at 0 in -H mode. t3 is now captured locally via ptp_gettime(), the same way -S mode already did, until hardware TX timestamping is supported. Also replaces the path delay heuristic in ptp_process_delay_resp() (which derived an approximation of (t2-t1) from path_delay_ns and last_delta_ns, only valid once the clock had already converged) with the canonical IEEE 1588-2008 §11.3 formula: store (t2-t1) directly from Sync/Follow_Up as sync_diff_ns, then average it with (t4-t3) from the Delay_Req/Delay_Resp exchange. Relaxes the path delay ceiling to 10ms unconditionally, since Delay_Req's t3 is software- timestamped in both modes until hardware TX timestamping is supported. Assisted-by: Claude:claude-sonnet-5 Assisted-by: Gemini:gemini-3.8-flash-medium Signed-off-by: Daniel P. Carvalho <danieloak@gmail.com>
Implements the Peer-to-Peer (P2P) transparent clock delay measurement mechanism (IEEE 1588-2008 §11.4 / IEEE 802.1AS / IEC/IEEE 61850-9-3) in apps/netutils/ptpd: - Add PTP_MSGTYPE_PDELAY_REQ, PTP_MSGTYPE_PDELAY_RESP, and PTP_MSGTYPE_PDELAY_RESP_FOLLOW_UP definitions and structs in ptpv2.h. - Define IEEE 1588-2008 Annex F peer delay multicast MAC address 01:80:c2:00:00:0e and Annex D peer delay IP address 224.0.0.107. - Replace bool delay_e2e with enum ptp_delay_mechanism_e (PTP_DELAY_NONE, PTP_DELAY_E2E, PTP_DELAY_P2P) in include/netutils/ptpd.h. - Add -P CLI option in system/ptpd/ptpd_main.c with mutual exclusion check against -E, and display last_transmitted_pdelayreq in status. - Implement responder logic in ptp_process_pdelay_req() sending Pdelay_Resp (t2) and Pdelay_Resp_Follow_Up (t3) regardless of master or slave state. - Implement requester logic in ptp_send_pdelay_req() gated on the physical link without requiring prior BMCA master selection. - Implement ptp_process_pdelay_resp() and ptp_process_pdelay_resp_followup() using canonical mean path delay formula ((t4 - t1) - (t3 - t2)) / 2. - Refactor path delay bounds checking and moving average filter into ptp_record_path_delay() shared across E2E and P2P mechanisms. - Set PTP version 2.0 and controlField 0x05 in Pdelay_Req, Pdelay_Resp and Pdelay_Resp_Follow_Up, and in the own-identity header, so that peers such as linuxptp accept the messages. - Clear pdelay_waiting_followup when a new Pdelay_Req is sent, so an orphaned Pdelay_Resp_Follow_Up from an abandoned cycle is not paired with stale timestamps. - Warn at startup when P2P is selected without CONFIG_SCHED_TICKLESS, since a tick-driven clock cannot resolve the peer delay. - Skip IP multicast join/leave handling for AF_PACKET. Assisted-by: Claude:claude-sonnet-5 Assisted-by: Gemini:gemini-3.8-flash-medium Signed-off-by: Daniel P. Carvalho <danieloak@gmail.com>
A single drift-rate sample computed between two consecutive sync updates was clamped against CLOCK_ADJTIME_SLEWLIMIT_PPM - the hardware's slew-rate safety limit, not a bound on how large a real crystal-oscillator drift measurement can plausibly be. An abnormally short or long measurement interval (e.g. right after a clock source outage/reconnect, or a burst of closely spaced sync packets following packet loss) could therefore produce a wildly implausible sample that still passed the check and corrupted the long-term drift_ppb average. Add CONFIG_NETUTILS_PTPD_MAX_DRIFT_PPB (default 500000, well above any real crystal's few-hundred-ppm drift) as a dedicated plausibility bound, intentionally much tighter than CLOCK_ADJTIME_SLEWLIMIT_PPM. A sample outside this bound is discarded and the previous averaged drift_ppb is kept unchanged instead of being corrupted. Assisted-by: Claude:claude-sonnet-5 Signed-off-by: Daniel P. Carvalho <danieloak@gmail.com>
When state->clockid is configured to a hardware PTP clock device (e.g., /dev/ptp0) instead of CLOCK_REALTIME, ptp_adjtime() previously passed only the measured frequency drift (-ppb) to clock_adjtime(), ignoring the residual phase offset (delta_ns / adjustment_ns). As a result, while the hardware counter tracked frequency, its phase was never pulled into alignment with the master clock. Convert delta_ns (which combines frequency drift and current phase error clamped to max_adjust_ns) to ppb over CONFIG_CLOCK_ADJTIME_PERIOD_MS, acting as a proportional-integral (PI) phase servo. This drives the hardware clock to phase lock with the master via POSIX clock_adjtime() using ADJ_FREQUENCY without requiring proprietary ioctl calls. Assisted-by: Claude:claude-sonnet-5 Assisted-by: Gemini:gemini-3.8-flash-medium Signed-off-by: Daniel P. Carvalho <danieloak@gmail.com>
The MAC latches a hardware receive timestamp later than the frame reaches the wire, because of the PHY and the clock domain crossing. This fixed delay is the ingressLatency port parameter of IEEE 1588 and shows up as a constant phase error between the local and the master clock. Subtract the configured latency from every hardware receive timestamp in ptp_getrxtime(), the single place where they enter the daemon, so Sync, Delay_Resp and the peer delay messages are all corrected. - Add CONFIG_NETUTILS_PTPD_INGRESS_LATENCY_NS (default 0, which applies no compensation). - Add the -I option to override it at run time. - Add ingress_latency_ns to struct ptpd_config_s. Software timestamps are not affected. Assisted-by: Claude:claude-sonnet-5 Signed-off-by: Daniel P. Carvalho <danieloak@gmail.com>
…tus() PR apache#3789 replaced the in-memory sigqueue + shared memory IPC in ptpd_status() with file-based IPC to support Protected and Kernel modes across address spaces. However, on microcontrollers running CONFIG_BUILD_FLAT, a filesystem or /tmp (TMPFS) is rarely mounted or available, causing ptpd_status() to fail with -ETIMEDOUT (errno 110) because the status file cannot be created. Retain the file-based IPC for !CONFIG_BUILD_FLAT (Protected and Kernel modes) while restoring the zero-overhead in-memory sigqueue + semaphore IPC for CONFIG_BUILD_FLAT. Both modes share the status serialization logic via ptp_populate_status() and support all fields including P2P. Assisted-by: Gemini:gemini-3.8-flash-medium Signed-off-by: Daniel P. Carvalho <danieloak@gmail.com>
A single Sync sample whose receive timestamp was taken late, for example because the task was scheduled late with software timestamping, was fed straight into the phase correction and the drift estimate, and could pull the clock away from the master. - Add CONFIG_NETUTILS_PTPD_OUTLIER_THRESHOLD_NS (default 0, which disables the check). A phase error that differs by more than this many nanoseconds from the median of the last five accepted samples is discarded, with a warning. - Accept the sample after eight consecutive rejections and restart the history from it, so that a real step of the master is still followed while a short burst of disturbed samples is ridden out. - Restart the history whenever the clock is stepped, since the old samples no longer describe the new time base. - With the default of 0 the behaviour is unchanged. Signed-off-by: Daniel P. Carvalho <danieloak@gmail.com> Assisted-by: Claude:claude-sonnet-5
Timestamp transmitted event messages with the hardware clock instead of reading the clock after sendmsg() returns. When hardware timestamping is selected on an AF_PACKET socket, request SOF_TIMESTAMPING_TX_HARDWARE for each event message, wait for the looped-back packet on the error queue with MSG_ERRQUEUE, and take the timestamp from its SO_TIMESTAMPING control message, as on Linux. Sync, Delay_Req and Pdelay_Req get their real departure time. - Use a transmit socket of its own, separate from the event socket, so the error queue is not shared with received packets. - Handle POLLERR separately from POLLIN in the main loop and drain all pending packets on each wakeup. - If the timestamp does not arrive, fall back to a software timestamp taken before the frame is sent. After three consecutive failures the driver is assumed not to provide hardware transmit timestamps, a warning is printed once and only software timestamps are used, so a driver without support does not stall the daemon. - Take the software timestamp before sending in every mode. It used to be taken after sendmsg() returned, so a fast peer's reply could appear to arrive before the request had left and give a negative delay. - Accept a measured path delay down to -100 microseconds and clamp it to zero, since hardware timestamps on both ends can make a short link measure slightly negative. Assisted-by: Claude:claude-sonnet-5 Signed-off-by: Daniel P. Carvalho <danieloak@gmail.com>
The frame leaves the MAC later than the moment its hardware transmit timestamp is latched, because of the clock domain crossing and the PHY. This fixed delay is the egressLatency port parameter of IEEE 1588. Add the configured latency to every hardware transmit timestamp obtained through MSG_ERRQUEUE, the counterpart of the ingress compensation. - Add CONFIG_NETUTILS_PTPD_EGRESS_LATENCY_NS (default 0, which applies no compensation). - Add the -O option to override it at run time. - Add egress_latency_ns to struct ptpd_config_s. Software timestamps are not affected. Assisted-by: Claude:claude-sonnet-5 Signed-off-by: Daniel P. Carvalho <danieloak@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
With hardware timestamping,
ptpdtook the transmit time of its event messages (Sync,Delay_Req,Pdelay_Req) from the clock aftersendmsg()returned. That is later than the frame really left, so a fast peer's reply could appear to arrive before the request had gone, and the peer delay measured with the peer-to-peer mechanism came out negative (about -18 µs on the board I tested with) and was rejected. #3782 says so: on this hardware P2P needs accurate transmit timestamps.This PR reads the transmit timestamp from the hardware the way Linux does. When hardware timestamping is selected on an
AF_PACKETsocket,ptpdrequestsSOF_TIMESTAMPING_TX_HARDWAREfor each event message, waits for the looped-back packet on the error queue withMSG_ERRQUEUEand takes the timestamp from itsSO_TIMESTAMPINGcontrol message. The driver side of that is apache/nuttx#20148.Commits
The first seven are the commits of #3782, unchanged.
Retrieve hardware TX timestamps via MSG_ERRQUEUE.Timestamps transmitted event messages with the hardware clock instead of reading the clock aftersendmsg()returns.POLLERRis handled separately fromPOLLINin the main loop, and all pending packets are drained on each wakeup.Add egress latency compensation for TX timestamps.The frame leaves the MAC later than the moment its hardware transmit timestamp is latched, because of the clock domain crossing and the PHY. That fixed delay is theegressLatencyport parameter of IEEE 1588. AddsCONFIG_NETUTILS_PTPD_EGRESS_LATENCY_NS(default 0, no compensation), the-Ooption to override it at run time andegress_latency_nsinstruct ptpd_config_s. It is the counterpart of the ingress compensation of netutils/ptpd: IEEE 1588 P2P delay mechanism, hardware clock phase-lock and outlier rejection #3782, and software timestamps are not affected.Impact
-OandCONFIG_NETUTILS_PTPD_EGRESS_LATENCY_NS, default 0.struct ptpd_config_sgetsegress_latency_ns; the only user in the tree issystem/ptpd, which is updated here.-H). A driver that does not return transmit timestamps makes the daemon switch to software timestamps after three failures.Testing
Built for
stm32f4discovery:ethrawwithCONFIG_STM32_ETH_TIMESTAMP_TX=y, without errors or warnings fromptpd../tools/checkpatch.sh -g <base>..HEADpasses.On hardware: a custom STM32F407 board with the STM32 Ethernet MAC and a DP83848 PHY, over
AF_PACKETand through a Fast Ethernet switch (ptpd -2 -s -p /dev/ptp0 -B -H -P -i eth0), against a GNSS-referencedptp4lGrandmaster using the IEC/IEEE 61850-9-3 profile, with the driver support of apache/nuttx#20148:path_delay9265.3 ns with σ = 2.5 ns over a 20 minute run (159 samples, no failed queries), phase offset about -2.2 µs, drift about -79.95 ppm.path_delaybetween 9.26 and 9.33 µs.path_delay_nsstayed at 0 and the phase offset was worse (about -6.7 µs against -2.4 µs).The end-to-end mechanism (
-E), master mode, the best master clock algorithm, a 24 hour run and a calibrated-Ovalue were not tested.