Skip to content

ThreadX SMP discards the execute list rebalance that tx_thread_relinquish asks for, starving a ready thread #756

Description

@fdesbiens

_tx_thread_relinquish in common_smp/src walks the ready list at the relinquishing thread's priority looking for a thread to hand its core to. When the walk reaches a thread it cannot place — one carrying preemption-threshold, or one excluded from the relinquishing thread's mapped core — it sets its local rebalance flag and breaks out, with the comment "We need to rebalance the list when we detect preemption-threshold."

That exit falls into the block whose comment reads "No other thread is ready at this priority... simply return". The walk established the opposite: it found a thread it could not place rather than running out of threads. That block restores interrupts and sets finished = TX_TRUE, and the rebalance at the end of the function is guarded by if (finished == TX_FALSE), so _tx_thread_smp_rebalance_execute_list never runs. The function asks for a rebalance and then discards the request.

The suppression is structural rather than occasional: every rebalance requested from inside the walk passes through that block first. Instrumenting a passing run of threadx_thread_relinquish_test counted 12 rebalance requests from the walk and 4 rebalances performed, and the four performed are reachable only from the three request sites that do not enter the walk.

The consequence on SMP is that a ready thread sitting behind the obstacle in the ready list never reaches a core. With every core held by threads of that priority relinquishing to each other, nothing else moves it, and the system keeps running without ever scheduling it. It is a livelock rather than a deadlock: the tick keeps arriving and the threads that hold the cores keep making calls, so the process looks alive.

This was found while running the SMP regression suite with ctest retries disabled, where it shows up as an intermittent timeout in threadx_thread_relinquish_test. A hung process captured untraced shows every port-level invariant holding — the Linux mutex free, the SMP protection not held, the scheduler idling in its own timed wait because every core's _tx_thread_current_ptr already equals its _tx_thread_execute_ptr entry — while three threads of the same priority sit TX_READY with their core-control flag set and parked on their run semaphores. No run count changes across a ten second sample. The port's own debug ring covers fifteen milliseconds of that state and holds 190 protect/unprotect pairs from the four spinning threads and zero system returns.

The trigger in that test is core exclusion rather than preemption-threshold: one thread is restricted to core 0 and permanently ready, so a relinquish from any other core meets it in the walk and takes the break.

The fix is to apply the early return only when no rebalance was requested, which lets control reach the function's own existing rebalance path — the one every other request site already reaches. No new code path is needed.

A regression test for it fills the four cores with threads of one priority relinquishing in a loop, places a thread excluded from every core behind them, and a runnable thread behind that. tx_thread_smp_core_exclude supports an all-core exclusion explicitly, leaving the thread ready and permanently unmapped, which makes it a stable obstacle rather than one that eventually gets a turn. The runnable thread never running within a bounded number of ticks is the failure. It fails deterministically before the change and passes after.

dev and master carry the same code, so this affects the current release line and not only older baselines.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions