Repository navigation
Fix the 127 test failures and 8 collection errors that CI never sees #114
Description
Part of #108 · Phase 1 · Depends on #110
Problem
When the suite is actually executed (in a purpose-built venv, since the documented install cannot start pytest — see #110), the verbatim result is:
127 failed, 1738 passed, 26 skipped, 229 warnings, 8 errors in 134.78s (0:02:14)
from 1898/2280 tests collected (382 deselected).
None of these are visible in CI, which runs 15 tests.
Failures grouped by root cause
| N | Error | Interpretation |
|---|---|---|
| 9 | clustrix/executor_core.py:210: KeyError: 'manager' |
executor refactor changed the job-dict shape; callers not updated |
| 7 | ValueError: Unsupported cluster type: local |
config/executor drift — local is accepted by config but has no executor branch (executor_core.py:101) |
| 6 | TypeError: 'NoneType' object is not subscriptable |
|
| 6 | TypeError: __init__() missing 1 required keyword-only argument: 'response' |
googleapiclient API changed under us — unpinned dep |
| 5 | ValueError: Unknown configuration parameter: partition / namespace / network_timeout / cleanup_on_failure |
tests pass parameters ClusterConfig rejects |
| 5 | AttributeError: 'ClusterExecutor' object has no attribute '_get_k8s_result' / _get_k8s_error_log |
tests reference deleted methods (test_kubernetes_integration.py:466,491,558) |
| 4 | AttributeError: 'ValidationCredentials' object has no attribute 'cred_manager' |
|
| 3 | AttributeError: 'EnhancedClusterConfigWidget' object has no attribute 'aws_region' |
|
| 2 | ModuleNotFoundError: No module named 'sklearn' |
undeclared dep (see #110) |
| 2 | Failed: Timeout (>60.0s) at clustrix/kubernetes/aws_provisioner.py:763,771 |
real AWS retry loop (see #109) |
Plus stale mock patch targets — the mocks no longer match the code:
unittest/mock.py:1378: AttributeError: <module 'clustrix.executor'> does not have the attribute 'setup_remote_environment'
unittest/mock.py:1378: AttributeError: <module 'clustrix.executor'> does not have the attribute 'cloudpickle'
Top failing files: test_notebook_magic.py (9), test_loop_analysis_advanced.py (6), test_kubernetes_integration.py (6), test_cloud_providers_huggingface_spaces.py (6), comprehensive/test_edge_cases_real.py (6), test_auth_fallbacks_real.py (5).
Important: triage before fixing
A large share of these are tests asserting against a code shape that no longer exists (deleted methods, renamed config params, changed dict keys). For each failure decide:
- Code is wrong -> fix the code.
- Test is wrong -> fix the test.
- Test is worthless (asserts a mock returns what the test told it to return) -> delete it; it is covered by the de-mock issue.
Per project policy, do not "simplify" a test to make it pass. Either the code is fixed so the existing test passes, or the test is deleted outright with a stated reason.
Acceptance criteria
-
pytest tests/ -m "not real_world"-> 0 failed, 0 errors - Every deletion is justified in the PR description
-
googleapiclientand other volatile deps are version-pinned or adapted -
cluster_type: "local"either works end to end or is rejected at config time with a clear message — not accepted-then-crashed
Verification
pytest tests/ -m "not real_world" -o addopts="" -q --tb=no | tail -3
Activity
jeremymanning commentedon Aug 17, 2026
Partial update from #130 (merged as #134): the collection errors are gone; the test failures are not.
Measured on
masterat760432cin a clean[dev]-only venv:The 6 errors that were reachable were:
tests/real_world/test_container_registry_comprehensive.py:571— a backslash inside an f-string expression, aSyntaxErroron every Python before 3.12 whilerequires-pythonis>=3.8. An AST parse across all 295 files undertests/andclustrix/confirms it was the only one.numpy/pandasat module scope with neither declared in any extra. Both added to[dev].Worth noting for this issue's framing: that syntax error is why CI "never saw" much of anything. Collection aborted with
Interrupted: 1 error during collectionbefore the run began, so a barepytestexecuted zero tests rather than the suite anyone assumed.Still open here. With collection now clean, the failures are visible and countable for the first time. A partial run of the newly-collectible tree showed 13 failures in the first 59 tests, so the "127 failures" figure needs re-deriving against
masterrather than carried forward — it was measured when 6 modules could not even be imported.Suggested next step for this issue: re-run and re-baseline the failure count now that collection succeeds, then triage. Related: #133 (flake8 gate that can never pass, plus a real f-string bug in
test_direct_gpu_detection.py) and #135 (fast_ci.ymlis invalid YAML and has never run a job).Fix three fabricated-result bugs, close the RCE, and overhaul the doc…
jeremymanning commentedon Aug 19, 2026
Resolved
0 failed, 0 errors. Collection: 1258 of 1275 (17 deselected by marker), no import errors.
Every root cause this issue named is verifiably gone:
_get_k8s_result/_get_k8s_error_logmissinggrep→ 0 hits; the Kubernetes backend is deleted (#142)googleapiclientimport errorsgrep→ 0 hitsaws_regionattribute errorsgrep→ 0 hitspartition,namespace,network_timeout,cleanup_on_failureabsent fromClusterConfigcluster_type="local"raisedUnsupported cluster typeexecutor_core.py:65The 8 collection errors are gone with the modules that caused them.
Worth noting what actually made the last 13 failures stick, since it was not any of the above: test pollution. Every one passed in isolation. Three nested leaks —
reset_configrestored 8 of a hundred-odd fields; it restored fields but not the module binding; and a class-localreset_configfixture shadowed the autouse one. After that file ran, the live config still carried 20 drifted fields, and the widget reads the live config.