Skip to content

fix(tests): deflake custom-nodes uninstall test (global shutil.rmtree patch)#9369

Merged
Pfannkuchensack merged 2 commits into
invoke-ai:mainfrom
lstein:fix/flaky-custom-nodes-rmtree-test
Jul 23, 2026
Merged

fix(tests): deflake custom-nodes uninstall test (global shutil.rmtree patch)#9369
Pfannkuchensack merged 2 commits into
invoke-ai:mainfrom
lstein:fix/flaky-custom-nodes-rmtree-test

Conversation

@lstein

@lstein lstein commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

TestUninstallPackNameValidation::test_rejects_invalid_pack_names_before_filesystem_side_effects (added in #9256) fails intermittently in CI:

FAILED tests/app/routers/test_custom_nodes.py::TestUninstallPackNameValidation::test_rejects_invalid_pack_names_before_filesystem_side_effects
  AssertionError: Expected 'rmtree' to not have been called. Called 1 times.
  Calls: [call('/var/folders/g3/.../T/tmp7n0m81lc', onerror=<function TemporaryDirectory._rmtree.<locals>.onerror at 0x164d5e480>)]

Seen on py3.11 macos-default in this run on PR #9367 (which doesn't touch this code); the same run passed on re-run.

Root cause

The test patches invokeai.app.api.routers.custom_nodes.shutil.rmtree. Since custom_nodes.py does a plain import shutil, that target resolves to the global shutil module — the patch affects every caller in the process, not just the router.

Several tests in the suite leak TemporaryDirectory objects inside reference cycles (the CI log shows many ResourceWarning: Implicitly cleaning up <TemporaryDirectory ...>). Whenever the cyclic GC happens to run while the patch is active, tempfile's finalizer calls shutil.rmtree — i.e. the mock — and assert_not_called() fails. The recorded call's onerror=<TemporaryDirectory._rmtree...> argument shows it is tempdir cleanup, not the router. Whether it fires is pure GC-timing luck, hence one platform failing while five pass.

Reproducible deterministically:

import gc, tempfile
from unittest.mock import patch
gc.disable()

td = tempfile.TemporaryDirectory()
class H: pass
h = H(); h.td = td; h.self = h   # cycle: only the cyclic GC can free it
del td, h

with patch("invokeai.app.api.routers.custom_nodes.shutil.rmtree") as m:
    gc.collect()                  # stands in for CI's unlucky GC pass
    m.assert_not_called()         # AssertionError: Called 1 times

Fix

Patch the router module's shutil attribute (patch("...custom_nodes.shutil")) instead of the global shutil.rmtree, and assert on mock_shutil.rmtree. Only lookups through custom_nodes.shutil hit the mock; tempdir finalizers keep the real shutil.

Verified both properties:

  • the repro above no longer trips the narrowed mock;
  • a valid pack name still drives the router's real shutil.rmtree(target_dir) call into the mock, so the test still catches a broken validator.

QA Instructions

uv run pytest tests/app/routers/test_custom_nodes.py — 39 passed.

Merge Plan

Squash-merge whenever; test-only change.

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • Documentation added / updated (if applicable)
  • Updated What's New copy (if doing a release after this PR)

🤖 Generated with Claude Code

…stall test

test_rejects_invalid_pack_names_before_filesystem_side_effects patched
invokeai.app.api.routers.custom_nodes.shutil.rmtree, which mutates the
global shutil module. If Python's cyclic GC finalizes a TemporaryDirectory
leaked by any earlier test while the patch is active, tempfile's cleanup
lands on the mock and assert_not_called() fails — seen intermittently in CI
(py3.11 macos-default) as:

  AssertionError: Expected 'rmtree' to not have been called. Called 1 times.
  Calls: [call('/var/folders/.../tmp7n0m81lc',
          onerror=<function TemporaryDirectory._rmtree.<locals>.onerror ...>)]

Patch the router module's shutil attribute instead, so only calls made
through custom_nodes.shutil are intercepted and tempdir finalizers keep
using the real shutil.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions github-actions Bot added the python-tests PRs that change python tests label Jul 21, 2026
@lstein lstein assigned dunkeroni and Pfannkuchensack and unassigned dunkeroni Jul 21, 2026
@lstein lstein added the 6.14.0 label Jul 21, 2026
@lstein lstein moved this to 6.14.x Theme: USER EXPERIENCE in Invoke - Community Roadmap Jul 21, 2026
@Pfannkuchensack
Pfannkuchensack enabled auto-merge (squash) July 23, 2026 01:12
@Pfannkuchensack
Pfannkuchensack self-requested a review July 23, 2026 01:13

@Pfannkuchensack Pfannkuchensack left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good fix.

@Pfannkuchensack
Pfannkuchensack merged commit 679224d into invoke-ai:main Jul 23, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.14.0 python-tests PRs that change python tests

Projects

Status: 6.14.x Theme: USER EXPERIENCE

Development

Successfully merging this pull request may close these issues.

4 participants