pytest.ini is dead config that silently shadows pyproject.toml (--strict-markers, testpaths, markers all inert) #130

Activity

1. jeremymanning

   added

   bug

   P1-highRequired for production readinessRequired for production readiness

   testingTest suite, CI, coverageTest suite, CI, coverage

   on Aug 17, 2026
2. jeremymanning

   added a parent issue on Aug 17, 2026

   • Master Plan: Get Clustrix Production-Ready #108
3. jeremymanning

   added a commit that references this issue on Aug 17, 2026

   Issue #130: Add handoff plan for the pytest config-shadowing fix

   ...

   de09f1a
4. jeremymanning commented on Aug 17, 2026

   

   jeremymanning

   on Aug 17, 2026

   MemberAuthor

   More actions

   Handoff plan written — notes/handoff_130_pytest_config.md (commit de09f1a)

   A fresh session can pick this up cold. Everything in it was verified on master at a9393b7; every fact carries the command that established it.

   ⚠️ Landmine discovered while writing the plan

   Naively fixing this breaks bare pytest outright. Verified:

   $ mv pytest.ini /tmp && pytest --co -q
   ERROR: Refusing to run 'tests/integration': tests/integration provisions real,
   billable cloud resources (AWS EKS/EC2). Set CLUSTRIX_ALLOW_BILLABLE=1 ...
   

   Cause: master's pyproject sets testpaths = ["tests/unit", "tests/integration"], and pytest populates config.args from testpaths when no paths are given on the command line. The #109 guard inspects config.args, so the moment pyproject becomes live, every bare run aborts.

   This is a trap I created in #109 and it must be defused first. The clean fix — verified — is to key the guard off config.invocation_params.args (what the user actually typed) instead:

   invocation	config.args	config.invocation_params.args
   pytest (testpaths live)	['tests/unit', 'tests/integration']	(no path entries)
   pytest tests/unit/	['tests/unit/']	['tests/unit/', ...flags]

   Corrections to this issue's original body

   • pyproject declares 4 markers on master (real_world, slow, unit, integration), not 13. The longer list I cited was from the closed epic branch. pytest.ini declares 14.
   • master's pyproject addopts is just -v --tb=short — no -n 4. So activating pyproject does not require xdist here. The xdist problem is specific to the epic branch, which deletes pytest.ini. pip install -e ".[dev]" produces an environment where pytest cannot start #110 needs the same correction.

   Measured marker gap (the thing that gates --strict-markers)

   marker	uses	files	declared in pyproject?
   real_world	224	—	yes
   dartmouth_network	11	4	no
   slow	6	—	yes
   expensive	5	5	no
   performance	4	4	no
   integration	2	—	yes
   cleanup	1	1	declared nowhere — test_kubernetes_performance_benchmarks.py:1038, likely a typo

   Plan shape

   6 steps, one commit each, verification commands after every one:

   0. Baseline the numbers
   1. Defuse the landmine (guard → invocation_params.args) — must be first
   2. Delete pytest.ini, set testpaths = ["tests"]
   3. Declare the 3 missing markers; decide cleanup
   4. Enable --strict-markers last
   5. Add a regression test so a stray config can never shadow again
   6. Full scripts/pre_push_check.py, then correct pip install -e ".[dev]" produces an environment where pytest cannot start #110, MIGRATION.md:80, .claude/commands/testing/prime.md:95

   The regression test uses pytestconfig.inipath and getini("markers"); both were verified against pytest 8.4.2 in the broken and fixed states, so the snippet is known-working rather than assumed. It fails today and passes after Step 2 — write it first and watch it fail.
5. jeremymanning commented on Aug 17, 2026

   

   jeremymanning

   on Aug 17, 2026

   MemberAuthor

   More actions

   Fixed in #134 (branch fix/130-pytest-config). Before/after evidence below; the issue closes when that PR merges.

   Before — master @ a9393b7

   $ head -1 pytest.ini
   [tool:pytest]
   
   $ pytest tests/unit/ --co 2>&1 | grep configfile
   configfile: pytest.ini
   
   $ pytest --markers | grep -c '^@pytest.mark.real_world'
   0
   
   $ printf 'import pytest\n@pytest.mark.bogus_xyz\ndef test_x(): assert True\n' > tests/unit/test_probe_tmp.py
   $ pytest tests/unit/test_probe_tmp.py --co -q
   1 test collected            # --strict-markers inert
   
   $ pytest tests/ -m "not real_world" --co -q -o addopts= | tail -1
   1214/1592 tests collected (378 deselected), 6 errors
   

   After

   $ ls pytest.ini tox.ini setup.cfg 2>/dev/null | wc -l
   0
   
   $ pytest --co 2>&1 | grep configfile
   configfile: pyproject.toml
   
   $ pytest --markers | grep -cE '^@pytest.mark.(real_world|slow|unit|integration|expensive|dartmouth_network|performance):'
   7
   
   $ pytest tests/unit/test_probe_tmp.py --co -q
   ERROR ... Failed: 'bogus_xyz' not found in `markers` configuration option
   
   $ pytest --co -q | tail -1
   1670 tests collected        # bare pytest, 0 errors, no abort
   
   $ pytest tests/unit/ -q | tail -1
   75 passed
   

   Checklist from this issue

   • Decide on ONE config source — deleted pytest.ini, kept pyproject.toml
   • Inventory every mark in use before activating — 7 in use, 4 were declared; added expensive, dartmouth_network, performance; removed the single dead cleanup
   • Confirm pytest-xdist is installed wherever addopts applies — not needed: master's addopts has no -n. A comment above addopts now records that adding one requires moving xdist from [test] into [dev] first (pip install -e ".[dev]" produces an environment where pytest cannot start #110)
   • Verify testpaths activation does not pull tests/real_world/ into default runs — it does collect them (testpaths = ["tests"]), which is correct and matches the previous no-config behaviour. Nothing runs them by default: CI uses pytest tests/unit/ -m "not real_world", and scripts/pre_push_check.py now does the same, which it previously did not
   • Land --strict-markers last — Step 4, after the marker inventory was clean
   • Add a test asserting the config file is the expected one — tests/unit/test_pytest_config.py, four assertions, all failing on master

   Two corrections to this issue's own text

   "13 markers" / "1,532 marker issues." pytest.ini declared 10 markers and pyproject.toml 4. The suite uses 7 distinct non-builtin markers across 253 applications. The 1,532 figure came from scripts/test_discovery.py on the closed epic branch and does not correspond to anything measurable on master.

   The landmine was not in the issue. Deleting pytest.ini alone breaks pytest entirely, because testpaths then feeds config.args and the #109 guard refuses the run. That had to be fixed first, in its own commit. Details in #134 and in notes/handoff_130_pytest_config.md.

   Found along the way

   • The Step 0 baseline had 6 collection errors, not zero: an f-string backslash SyntaxError (fatal on Python <3.12 while requires-python is >=3.8) plus undeclared numpy/pandas. Fixed here, since Step 4's gate is "0 errors".
   • scripts/pre_push_check.py cannot exit 0 for unrelated reasons, and never could — filed as pre_push_check.py can never pass: 92 flake8 findings, incl. a real f-string bug in test_direct_gpu_detection.py #133, which also documents a real f-string bug in tests/integration/test_direct_gpu_detection.py.
   • pip install -e ".[dev]" produces an environment where pytest cannot start #110's addopts premise and its sklearn claim are both wrong; corrected there.
6. jeremymanning

   added 3 commits that reference this issue on Aug 17, 2026

   Issue #130: Close the guard bypasses red-teaming found, and fix my ow…

   ...

   a5b665f

   Issue #130: Revert the fast_ci.yml change; the diagnosis was wrong

   ...

   6c36bf3

   Issue #130: pytest.ini was dead config shadowing pyproject.toml (#134)

   ...

   760432c
7. jeremymanning

   closed this as completedin #134on Aug 17, 2026
8. jeremymanning

   added a commit that references this issue on Aug 17, 2026

   Update notes for handoff to a fresh session (#136)

   ...

   ed42375