fix(ReindexThread): shutdown hot-loop and dead-runnable silent queue failure

[freddyDOTCMS]

Problem Statement

Two bugs in ReindexThread.java that together caused push-published content to remain invisible on Production for ~2 days (observed on 26.07.06-3 Evergreen in a dotCMS Cloud Production environment).

Since PR #33885, push-published content is indexed asynchronously via the reindex queue on the receiver. Both bugs prevent that queue from draining reliably.

Bug 1 — Shutdown hot-loop

During JVM shutdown, ShutdownCoordinator.isRequestDraining() becomes true before stopThread() sets state to STOPPED. The inner loop in runReindexLoop() detects shutdown and breaks — but this only exits the inner loop and returns control to the outer loop in ReindexThreadRunnable:

// Outer loop — ReindexThreadRunnable (line 155)
while (state.get() != ThreadState.STOPPED) {
    try {
        runReindexLoop();   // breaks due to shutdown, returns here
    } catch (Exception e) { ... }
    // state is still RUNNING → outer loop calls runReindexLoop() again → immediate log + break → repeat
}

Observed impact: ~1.2 million "Shutdown detected, stopping reindex operations" log lines in ~2 minutes — putting severe pressure on I/O and contributing to JVM instability/crash.

Bug 2 — Dead runnable not detected in unpauseImpl()

unpauseImpl() assumes that if state == PAUSED, the ReindexThreadRunnable is alive in its sleep() loop:

if (state == ThreadState.PAUSED) {
    Logger.info(ReindexThread.class, "--- Unpausing reindex thread ");
    cache.get().remove(REINDEX_THREAD_PAUSED);
    getInstance().state(ThreadState.RUNNING);  // flips flag — but nobody is listening
}

After a crash, the runnable may exit (uncaught Throwable, executor shutdown) while leaving state at PAUSED. When content is push-published, unpauseImpl() logs "Unpausing reindex thread", sets state to RUNNING — but there is no live thread to act on it. The queue never drains. No error is logged.

The STOPPED branch correctly re-submits the runnable, but it is never reached.

---

Steps to Reproduce

Bug 1:

1. Start dotCMS
2. Trigger a JVM shutdown while ReindexThread is in RUNNING state
3. Observe millions of "Shutdown detected, stopping reindex operations" log lines before the process exits

The hot-loop only runs during the window between ShutdownCoordinator.isRequestDraining() becoming true and stopThread() being called. In production this window was long due to a JVM stall. To reproduce locally, use one of these approaches:

• Debugger breakpoint (no code change): Set a breakpoint on the first line of stopThread() (line 299 of ReindexThread.java). Trigger a server shutdown — the breakpoint pauses stopThread() while ReindexThread hot-loops on its own thread. Watch the log accumulate lines rapidly, then resume to complete shutdown.

• Temporary sleep in ReindexThreadShutdownTask: Add a Thread.sleep(5000) before the stopThread() call to create a 5-second window. Start dotCMS, wait for ReindexThread to reach RUNNING state, shut down, and observe the dense burst of log lines.

• Unit test: Mock ShutdownCoordinator.isRequestDraining() to return true, start ReindexThreadRunnable in a thread without calling stopThread(), let it run for ~200 ms, then assert "Shutdown detected" was logged more than once — proving the hot-loop. Use mockStatic if isRequestDraining() is a static method.

Bug 2:

1. Start dotCMS — ReindexThread starts, queue is empty, thread enters PAUSED state
2. Pod/JVM becomes unstable (thread stall, executor shutdown) — runnable exits but state remains PAUSED
3. Push-publish content from a remote environment
4. Observe: "Unpausing reindex thread" is logged (transaction committed), state is set to RUNNING
5. Observe: content does NOT appear on the target site — queue is never drained
6. Manual reindex makes content visible immediately (bypasses the queue)

---

Acceptance Criteria

• When ShutdownCoordinator.isRequestDraining() returns true inside runReindexLoop(), state is set to STOPPED before breaking, so the outer loop in ReindexThreadRunnable exits without re-entering runReindexLoop()
• "Shutdown detected, stopping reindex operations" is logged at most once per shutdown event — not repeated in a hot-loop
• ReindexThreadRunnable sets a liveness flag (AtomicBoolean) to true on start and clears it in a finally block on exit, including when an uncaught Throwable causes the runnable to exit unexpectedly
• When unpauseImpl() is called and state == PAUSED but the liveness flag is false (runnable has exited), the method re-submits the ReindexThreadRunnable rather than only flipping state to RUNNING
• After a pod restart that leaves ReindexThread in a dead-but-PAUSED state, push-published content becomes visible on Production without requiring a manual reindex
• The catch (Exception e) in ReindexThreadRunnable is widened to catch (Throwable e) so JVM-level errors (e.g., OutOfMemoryError) do not bypass the finally liveness-clear
• Normal pause/unpause cycle (empty queue → pause → content pushed → unpause → indexed) continues to work correctly after both fixes
• Full reindex from admin UI completes and performs index switchover successfully after both fixes

---

dotCMS Version

26.07.06-3 Evergreen. Likely affects all Evergreen builds containing PR #33885 (async receiver reindex).

---

Severity

High - Major functionality broken

---

Links

• Freshdesk ticket #38675
• Related: PR task(pp): only reindex parents of any relationship when push publishing #33885 (async push-publish receiver reindex — changed sync refresh to queue insert)

[github-actions]

PRs linked to this issue

• fix(reindex): end the ReindexThread shutdown hot-loop and dead-worker queue stall by @danielsolis-dotcms