Skip to content

Commit 8782ef0

Browse files
Workaround bmalloc hang with RT thread priorities.
To enable the usleep you need to set environmental variable WEBKIT_WPE_BMALLOC_MICROSECONDS_SLEEP (microseconds of sleep). When real time (RT) thread priorities are used for some of the gstreamer pipeline elements, we may run into a situation where several RT threads start spinning during a mutex acquisition process, leading to a system hang as most other threads won't be able to run. Sequence of events leading up to the hang: 1. A web process thread acquires the mutex lock for the heap and is then involuntary descheduled, and does not run again 2. vqueue:src (RT priority) enters the lockSlowCase and starts spinning in the while loop 3. multiqueue0:src (instance 1, RT priority) enters the lockSlowCase and starts spinning in the while loop 4. aqueue:src (RT priority) enters the lockSlowCase and starts spinning in the while loop 5. multiqueue0:src (instance 2, RT priority) enters the lockSlowCase and starts spinning in the while loop Once stage 5 is hit, the box is hung as the only thing that can run on a CPU core is: 1. one of the above RT threads (aqueue, vqueue, or multiqueue) 2. any other RT thread with a priority equal or greater than the above RT threads 3. any h/w irq The use of the usleep() will allow the low priority process to run and release the mutex lock, avoiding the hang Author of issue analysis and fix proposal: Steven Webster. * Source/bmalloc/bmalloc/Mutex.cpp: (bmalloc::yield):
1 parent a6a3fac commit 8782ef0

1 file changed

Lines changed: 18 additions & 1 deletion

File tree

Source/bmalloc/bmalloc/Mutex.cpp

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@
3232
#include <mach/thread_switch.h>
3333
#endif
3434
#include <thread>
35+
#include <unistd.h>
3536

3637
namespace bmalloc {
3738

@@ -41,7 +42,23 @@ static inline void yield()
4142
constexpr mach_msg_timeout_t timeoutInMS = 1;
4243
thread_switch(MACH_PORT_NULL, SWITCH_OPTION_DEPRESS, timeoutInMS);
4344
#else
44-
sched_yield();
45+
static size_t bmallocMicrosecondsSleep;
46+
static std::once_flag onceFlag;
47+
std::call_once(onceFlag, [] {
48+
const char* env = getenv("WEBKIT_WPE_BMALLOC_MICROSECONDS_SLEEP");
49+
if (env) {
50+
int value;
51+
if (sscanf(env, "%d", &value) == 1 && value > 0)
52+
bmallocMicrosecondsSleep = value;
53+
}
54+
});
55+
if (bmallocMicrosecondsSleep) {
56+
// The use of sched_yield() can lead to a system hang when real time
57+
// thread priorities are used, so use sleep in the absence of a better
58+
// alternative.
59+
usleep(bmallocMicrosecondsSleep);
60+
} else
61+
sched_yield();
4562
#endif
4663
}
4764

0 commit comments

Comments
 (0)