fix(clustering): stop SystemEventsJob dropping the majority of system events

[danielsolis-dotcms]

Fixes #36827

Proposed Changes

SystemEventsJob loses 50–63% of system events in a cluster. Its poll cursor is an in-memory wall-clock high-water mark, advanced to now() after processing, while rows are selected with created >= mark — but created is stamped when the SystemEvent is constructed, not when its transaction commits. Any poll landing in that gap sees nothing and still moves the mark past the event, which can then never satisfy created >= mark again.

This fixes four distinct causes of permanent event loss on that path:

• The commit-timing race (the reported defect). Each poll now reads from a bounded overlap window behind its cursor, so a late-committing row is re-read instead of skipped. Duplicates the window produces are suppressed by a bounded in-memory dedupe set.

• A cursor that lies about what it read. The mark was a static AtomicLong advanced from the clock rather than from the data. It is now a durable per-node value in a new additive system_event_cursor table, advanced only to the start time of a read that actually completed, and not advanced when a read fails — so a failed poll retries its range instead of skipping it.

  Restart recovery does not work yet, and this PR does not claim it. A durable cursor is only useful if the node keeps its identity, and today server_id is a fresh UUID on every JVM start (ServerAPI.SERVER_ID = Lazy.of(UUIDUtil::uuid)). A restarted node finds no cursor row, seeds at "now", and still loses events from its downtime. Filed as Node server_id is regenerated on every restart (persistence removed in the BSL relicensing commit?) #37291 — persistence for it was removed in the BSL relicensing commit and may not have been intentional. Once that is resolved, restart recovery follows from this change with no further work here.

• UserSessionBean could never be deserialized, so SWITCH_SITE events — emitted during ordinary admin activity — could not cross nodes. Fixed with an explicit @JsonCreator. (SystemTableUpdatedKeyEvent had the identical defect and was fixed upstream in fix(clustering): propagate system table set() cluster wide #37286; this branch defers to main for it.)

• One unreadable payload destroyed its whole batch. getEventsSince converted a polled batch as a unit, so a single bad payload discarded every event in that window — silently, at DEBUG. Rows are now converted individually and an unreadable one is logged and skipped. Found by running a real two-node cluster, where it caused 39 consecutive failed polls on each node. Tracked systemically in One undeserialisable payload destroys delivery of every event in its poll window #37249.

Loss is also no longer silent: warnings fire when commit lag approaches the window, when the poller stalls, and when a stale cursor is clamped, plus an authored-vs-observed reconciliation on an interval.

Delivery is at-least-once and always was — the >= boundary already permitted repeat delivery. It is now documented in docs/backend/SYSTEM_EVENTS.md rather than accidental, with an audit of all ten existing consumers.

Checklist

• Tests — 19 unit, 25 integration, registered in MainSuite3a so they run in CI
• Translations — n/a
• Security Implications Contemplated — see comment below

Additional Info

Verification detail, measurements, rollback classification and known gaps are in the first comment, to keep this description readable.

[semgrep-dotcms]

Semgrep found 1 CUSTOM_INJECTION-2 finding:

• dotCMS/src/main/java/com/dotcms/api/system/event/SystemEventsFactory.java
  • L169-171 - Triage

The method identified is susceptible to injection. The input should be validated and properly
escaped.

If this is a critical or high severity finding, please also link this issue in the #security channel in Slack.

[danielsolis-dotcms]

Verification detail

Moved out of the PR description to keep it readable.

---

Reproduction evidence

The overlap window was temporarily reverted to main's behaviour to capture the defect against this PR's own tests:

Tests run: 3, Failures: 2, Errors: 1

AssertionError: Reading from the overlap window floor must DELIVER the event - this is the fix
AssertionError: Every event from the long transaction must be delivered, not just the most recent one:
                07c23231-e18f-4d42-9ff3-d43cfe1504c6

Reproduced on all 4 forks, a different lost event id each time. Within the same test, the assertion "Reading from the bare cursor must MISS the event — this is the defect" passed: the event is in the database, the poller is running, and it is never returned. Restoring the window turns both green.

The loss is not probabilistic. With a bare cursor an event whose created predates the cursor is never returned again, however many polls run — which is what makes the reported 50–63% possible. Those events are not delayed; they are gone.

Measurements were captured against a local two-node cluster; the figures quoted here are from those runs.

---

Security

/security-review: no HIGH or MEDIUM findings.

The substantive question is that this fix increases the volume of events actually delivered — previously dropped events now reach consumers — so exposure genuinely changes and was verified rather than assumed. SystemEventsWebSocketEndPoint.sendSystemEvent applies apply(event, session) → validPayload → PayloadVerifier inside the per-session loop, so higher volume produces more authorized sends, not unauthorized ones.

All new SQL is parameterized via DotConnect.addParam (serverId comes from ServerAPI.readServerId(), never user input). The @JsonCreator makes one single-String class constructible and introduces no polymorphic typing. New logs carry only UUIDs, durations and counts; Jackson redacts the document ([Source: UNKNOWN]), so no payload content reaches a log.

---

Rollback safety

Rollback-safe — additive only. postgres.sql +10/−0, the migration contains 0 ALTER statements, and SystemEventsFactory is +63/−0. Nothing existing was modified, so system_event's row shape and INSERT are byte-identical to main:

• an old node parses new rows normally and keeps its in-memory mark (no worse than today);
• a new node writes rows an old node can read;
• a rollback simply never queries system_event_cursor.

The @JsonCreator affects deserialization only — Jackson serializes from getters, so the written form is unchanged.

---

Performance

Index scan on idx_system_event at every size and window tested; the planner never flipped to a sequential scan.

Rows	Window	Returned	Buffers	Execution
5,000	120s (default)	1	3	0.029 ms
250,000	120s	12	6	0.020 ms
1,000,000	120s	44	27	0.035 ms
1,000,000	3600s (30×, abusive)	1,342	720	0.383 ms

New write volume is one single-row upsert per node per poll — O(nodes), never O(events × nodes). The publish path gained no work, so no added latency inside caller transactions.

---

Additional Info

Delivery is at-least-once, and that is now written down. It always was — the >= boundary already permitted repeat delivery — but nobody had documented it, so no consumer was knowingly built for it. All 10 consumers were audited: every cache-invalidation consumer is set(resolve…()) and naturally idempotent. Contract documented on SystemEventsAPI and in docs/backend/SYSTEM_EVENTS.md.

Known residual limitation. A transaction held open longer than the overlap window can still lose its events. This is bounded, configurable, and now warned about — as opposed to the unbounded silent loss it replaces. It is covered by a test that pins it as a known boundary rather than a surprise.

Follow-up filed: #37249. getEventsSince converts a batch as a whole, so one undeserialisable payload destroys delivery of every event in its window. Two instances confirmed: SystemTableUpdatedKeyEvent (fixed here) and JobCompletedEvent (not fixed — it needs the batch-level change). Note for whoever takes it: BulkRefreshCompletionListener sends a user notification per event and is currently unable to fire twice only because JobCompletedEvent never deserializes — fixing #37249 makes that duplicate live, so it should be guarded there.

AC-003 verified on a real two-node cluster, across three deployments of the same cluster:

	cursor fix only	+ batch fix	+ UserSessionBean
Delegate-killing errors per node	39	0	0
Skip warnings	n/a	3	0
SWITCH_SITE handling	killed every poll	skipped + logged	delivered
Propagation node 1 → node 2	never (36 s of polling)	6 s	3 s

AC-008 measured over a 24-hour window: 0.00% loss on both nodes — 1,158 authored / 1,158 observed per node, confirmed independently by the issue's Step B method and by the in-product reconciliation, which reported 0.00% on every hourly sample. Zero delegate errors and zero undeserialisable payloads across the window.

Read that with its caveat. The wall-clock window was 24 hours, but the host (a laptop) entered idle sleep repeatedly overnight, so the cluster was actively running for roughly 9.7 hours of it. Three suspensions exceeded the 60-minute backlog bound and tripped the clamp on both nodes — the observability behaving correctly, not a delivery defect. Loss was nil across those stalls because the traffic generator froze alongside the nodes, so nothing was authored during the skipped spans. Consequently the run does not exercise the clamp's real loss scenario — one node down while other nodes keep authoring — which stays untested on real hardware, along with the 31-day purge boundary and bursty load.

An earlier 1-hour run under continuous load gave the same result — 354/354 per node, 0.00% — and covers the continuous-operation case this one does not.

(Superseded detail from the 1-hour run:) AC-008 measured over 1 hour: 0.00% loss on both nodes — 354 authored / 354 observed each, confirmed independently by the issue's Step B method and by the new in-product reconciliation (authored=328, observed=329, loss=0.00%). Zero delegate errors, zero skipped payloads, zero lag warnings, with SWITCH_SITE traffic running throughout.

Taken together the two runs cover ~10.7 hours of active operation across a 25-hour span with zero measured loss, against a ≤1% bar and a 50–63% reported baseline. AC-008 is reported as met for the conditions tested, with the untested scenarios named above rather than implied.

Known limitation found by that run: the reconciliation's observed count can exceed authored by ±1 at window boundaries (the two are counted over slightly different windows). Harmless at a few hundred events — 0.3% here — but a node authoring ~20 events/hour would read a boundary miss as 5% loss and raise a false WARN. A minimum-volume floor (~50 authored) would fix it; not applied here because it is a judgement call worth a reviewer's opinion.

Note for anyone reproducing this: the example compose files in docker/docker-compose-examples/cluster-mode do not run as-is — docker compose config rejects both because volumes: is followed only by commented-out lines. Removing that block fixes it; no license mount is needed now that dotCMS is BSL 1.1, which also makes that README's "license pack must contain at least two licenses" stale. Not fixed in this PR.

Two ADRs proposed, not written (Spec-Kit must never author ADRs): the at-least-once delivery contract, and the fact that dotCMS has two cluster-messaging mechanisms — DotPubSub and this queue — with durability that ought to be selectable per topic. Both are proposals only; neither is authored here.

Shared-state note for review. SystemEventsRetentionIntegrationTest calls
systemEventsAPI.deleteEvents(...), which removes rows older than the retention window from the
shared system_event table. Nothing else in MainSuite3a creates events that old, so the blast
radius is nil today — but it is a global mutation rather than a purely local one, and worth knowing
about rather than discovering later.

New configuration (all defaulted, none required):

Property	Default	Purpose
SYSTEM_EVENTS_OVERLAP_WINDOW_SECONDS	120	How far back each poll re-reads
SYSTEM_EVENTS_MAX_BACKLOG_MINUTES	60	Bounds recovery after downtime
SYSTEM_EVENTS_LAG_WARN_THRESHOLD_PERCENT	50	Warn before the window is exceeded
SYSTEM_EVENTS_RECONCILE_INTERVAL_MINUTES	60	Reconciliation cadence

[danielsolis-dotcms]

Correction: restart recovery does not work yet, and why

An earlier version of this description claimed the durable cursor means "events committed while a node was down are delivered when it returns." That is not true today, and the description has been corrected.

While trying to verify it on a real two-node cluster, the test kept failing for a reason that turned out not to be in this code:

// com/dotcms/cluster/business/ServerAPI.java:15
final static Lazy<String> SERVER_ID = Lazy.of(UUIDUtil::uuid);

A node's server_id is a fresh UUID on every JVM start — not persisted, not configurable, no environment override. So a restarted node finds no cursor row for its new identity, seeds at "now", and the events from its downtime are lost exactly as before.

Observed directly:

before restart: Server id : aabb2356
after  restart: Server id : 3d1ab6ad

/data/shared/assets/server/
  3d1ab6ad-.../   <- this boot
  aabb2356-.../   <- previous boot

Persistence for this used to exist and was removed in 7189abfb60 (PR #31261), the BSL relicensing change. Filed as #37291, framed as a suspected regression pending confirmation from that PR's author — if the removal was deliberate, dotCMS needs a new approach to durable node identity rather than a restoration of the old file-based one.

What this does and does not change here

Unaffected: the commit-timing race — the defect #36827 actually reports — is fixed by the overlap window, which does not depend on node identity at all. Same for the batch-conversion fix and the payload creator. The cursor still advances only from a read that completed, and still does not advance when a read fails.

Deferred: restart recovery. Once #37291 is resolved it follows from this change with no further work here, because the cursor is already durable and keyed on server_id.

Also corrected: data-model.md described orphaned cursor rows as the rare decommissioned-node case. With a new id per boot it is actually one orphan row per restart — small, but unbounded. That is a consequence of #37291, not of this design.

On the untested scenario

The "one node down while other nodes keep authoring" case listed under known gaps cannot be tested until #37291 is fixed — a restarted node is a different node, so the scenario cannot be constructed. Two attempts on a live cluster produced results that looked plausible and proved nothing; both were discarded rather than reported.