Repository navigation
Flaky tests when CPU is busy #4669
Description
Activity
Found the first source of flakiness. Looks like
SET_TERM_COND termCond=2, tolerance=0.001should normally be shown, except you manage to be fast enough to not process this command. The following patch would fix it.Before I fix many tests which have this part copy-paste: Is this the right way to wait for linuxcnc to be up and running? There are 57 tests having
test-ui.pyand at least 4 have this issue.diff --git a/tests/motion-logger/basic/expected.builtin-startup.in b/tests/motion-logger/basic/expected.builtin-startup.in index 2296018b31..fce76f310a 100644 --- a/tests/motion-logger/basic/expected.builtin-startup.in +++ b/tests/motion-logger/basic/expected.builtin-startup.in @@ -176,6 +176,7 @@ SPINDLE_OFF DISABLE JOINT_UNHOME joint=-2 FREE +SET_TERM_COND termCond=2, tolerance=0.001 JOG_ABORT joint=0 JOG_ABORT joint=1 JOG_ABORT joint=2 diff --git a/tests/motion-logger/basic/test-ui.py b/tests/motion-logger/basic/test-ui.py index 21da4c2bd4..8d85df15bc 100755 --- a/tests/motion-logger/basic/test-ui.py +++ b/tests/motion-logger/basic/test-ui.py @@ -45,6 +45,15 @@ c = linuxcnc.command() s = linuxcnc.stat() e = linuxcnc.error_channel() +# +# Wait for interpreter to be ready +# + +while True: + s.poll() + time.sleep(0.1) + if s.interp_state == linuxcnc.INTERP_IDLE: + break # # Come out of E-stop, turn the machine on, and switch to Auto mode.
An other sometimes failing tests, does not reproduce well except sometimes with parallel tests:
tests/ui-smoke/gmoccapy ... WARN: homing did not complete on attempt 1, retrying WARN: STATE_ESTOP_RESET not reached on attempt 1, retrying WARN: STATE_ESTOP_RESET not reached on attempt 2, retrying WARN: STATE_ESTOP_RESET not reached on attempt 3, retrying WARN: STATE_ESTOP_RESET not reached on attempt 4, retrying WARN: STATE_ESTOP_RESET not reached on attempt 5, retrying WARN: STATE_ESTOP_RESET not reached on attempt 6, retrying UI_SMOKE_FAIL: STATE_ESTOP_RESET did not hold stable across 6 attempts ...This is not a big issue, you can just run the parallel tests with less jobs. But these races might hit also somewhere else.
All these are races that need fixing. Either the base code has a race or the test has a race. Both are equally bad.
While working on #4665, I discovered that several tests start failing randomly when running with to many jobs. And I also remember having rare CI fails which just went away on the next push.
This is not a big issue, you can just run the parallel tests with less jobs. But these races might hit also somewhere else.
This can be reproduced sometimes by running on a slow PC or in a VM with only 2 CPU's mapped. When the CPU is kept busy with
stress-ng --cpu ncpu, the tests fail also in single threaded mode.touchy UI tests, 100% reproducible with ncpu=16:
touchy UI tests, somewhat reproducible, better with ncpu=128. Looks like loading touchy_postgui.hal has a race.
SET_TERM_COND / SET_LINE appearing, somewhat reproducible, better with ncpu=128:
hal-show changing order, does not reproduce well, once in a while, this happens only in parallel mode: