feat(ecosystem): test all 19 repos, and stop reporting passes that never ran

[srpatcha]

eosim ecosystem is the command for validating the whole organisation. It was reaching 2 of the 19 repos, and could not report a failure for the ones it did reach.

It could not fail

tests_passed = max(tests_passed, tests_run if build_ok else 0)
tests_failed = max(0, tests_run - tests_passed)
passed       = build_ok and tests_failed == 0

tests_run was a count of executables on disk. Any repo that compiled reported every test passing. Reproduced directly:

ctest reported passing : 0
harness reports passed : 22
harness reports failed : 0
harness verdict        : PASS

It found 2 of 19 repos

find_repos held a hardcoded list of lowercase names — eai, eni, eipc, eboot, ebuild-tool. None match a real directory on a case-sensitive filesystem, and run_ecosystem_tests silently continued past anything not on the list.

Discovery is now by inspection: a .git makes it a repo, the files present decide the build system. A new product becomes testable by being cloned.

Honest statuses

A repo that cannot be tested reports SKIP, never PASS. DEPS is narrower — the suite exists but the repo's declared dependencies are absent here, which is not a broken test. Repos with two build systems get both exercised, which is how ebuild's broken CMake configure surfaced from behind its green Python suite.

Two runner bugs found by running it

• pytest was invoked with -q; a repo whose own addopts sets -q reached -q -q, which suppresses the summary line — a fully green EoStudio read as "no summary produced".
• PYTHONPATH omitted the repo, so anything not pip-installed died at conftest import. Root and src/ are both offered now (eDB uses a src layout).
• CLI: --simulate was accepted and never passed through, so --no-simulate did nothing. Added --only and --list.

Result

  Repos:  19 discovered | 7 passed | 1 failed | 11 skipped
  Tests:  2420 run      | 2404 passed | 16 failed

  [FAIL ] ebuild       cmake   cmake configure failed        ← real, see eBoot#60
  [PASS ] EoSim        python  tests:1702/1702
  [PASS ] EoStudio     python  tests:475/475
  [PASS ] eAI          cmake   tests:24/24
  [PASS ] eBoot        cmake   tests:17/17
  [PASS ] eFirmware    cmake   tests:3/3
  [PASS ] eNI          cmake   tests:40/40
  [PASS ] ebuild       python  tests:98/98
  [PASS ] eos          cmake   tests:22/22
  [DEPS ] eDB          python  tests:23/39  needs fastapi

Verification: 1702 pass (1666 before, +36 new tests here). Full ecosystem run against the 19-repo workspace reproduces that table.

🤖 Generated with Claude Code

[srpatcha]

Review — EoSim#16 "feat(ecosystem): test all 19 repos, and stop reporting passes that never ran"

head: 37fd2f1 author: srpatcha ci: fail (two red, both pre-existing on master — measured, see CI) · mergeable: CONFLICTING

Verdict: The runner rewrite is right and the reasoning behind it is the best kind — tests_run was a count of executables on disk, so any repo that compiled reported every test passing, and the replacement takes ctest's and pytest's own summary lines or reports SKIP. Separating SKIP, DEPS and FAIL is exactly the distinction the old code destroyed. One thing carries the old shape across into the new code: at the only boundary CI can read — the process exit status — a skipped repo is still indistinguishable from a passing one.

Separately, and more important than anything in this review: this PR contains the fix for the defect that is currently failing 22 CI jobs on eos#129. See below — it deserves to be said out loud, because the PR body does not mention it and the branch has been conflicting since 2026-08-29.

Findings

#	Severity	File:line	Finding	Recommended fix
1	High	eosim/cli/main.py:675-676; eosim/integrations/ecosystem.py:558-566	eosim ecosystem exits 0 when repos were never tested. The command's only failure path is if report.repos_failed > 0: sys.exit(1), and run_ecosystem_tests puts SKIP and DEPS into repos_skipped, never into repos_failed — correctly, since those are different facts. But that means the run the body reports — 19 discovered, 7 passed, 1 failed, 11 skipped — becomes exit 0 the moment ebuild's CMake configure is fixed, with eleven of nineteen repos untested. summary() is honest about it in prose (VERDICT: PASSED (11 repo(s) skipped — see above)), and prose is not what a CI gate reads. This is the same shape as tests_passed = max(tests_passed, tests_run if build_ok else 0) moved up one level: the report distinguishes "not tested" from "passed", and the exit code does not. The headline word being PASSED when 58% of the ecosystem was not exercised makes it worse, because that is the line someone pastes into a status update.	Two parts, and the first is free. Change the verdict word: INCOMPLETE (%d of %d repos not tested) whenever repos_skipped is non-zero — PASSED should require that everything discovered actually ran. Then add --strict (or --allow-skips, defaulting to allow) so the exit code can be made to reflect it: local runs legitimately skip on go not installed or an absent node_modules, and should stay exit 0, while a CI invocation passes --strict and goes red on any skip. A --require <repo>... naming the repos that must not skip would be even tighter and is the form that survives a new repo being added.
2	Medium	eosim/integrations/ecosystem.py:227-230 and :6-9	Discovery is by inspection; test enablement is still a hardcoded list, and the docstring says otherwise. test_c_repo configures every CMake repo with a fixed set of options — -DEOS_BUILD_TESTS=ON -DEBLDR_BUILD_TESTS=ON -DEAI_BUILD_TESTS=ON -DENI_BUILD_TESTS=ON — while the module docstring states that "a new product becomes testable by being cloned into the workspace, with no change here". That is true of find_repos and false here. A CMake repo whose option is spelled anything else (eDB, eIPC, eOffice, eFirmware, eBrowser, eos-health, eos-aero — none of their option names appear in that list) configures with tests off, registers none, and reports [SKIP ] … built OK; no tests registered. It fails safe, which is right, and the reason it prints misattributes the cause: the tests exist, they were not asked for. This is the identical failure mode as the eai/eni/eboot list the PR replaces — a hardcoded list that goes stale — surviving one function further down.	Move the per-repo configure options out of the runner into something each repo owns: a .eosim-ecosystem.yml at its root, read if present, with the current four as fallbacks. Failing that, keep the table but stop describing it as inspection: name it _CMAKE_TEST_OPTIONS, say in the docstring that it must be extended for a new CMake repo, and make the skip reason distinguish "no tests registered" from "tests may exist but no known option enables them".
3	Medium	.coverage, eosim.egg-info/PKG-INFO, eosim.egg-info/SOURCES.txt, out/logs/stm32f4.log	Four generated files are committed, and one of them grows sevenfold in this diff. .coverage is a binary SQLite database going from 77,824 to 548,864 bytes — it will change on every local test run, conflict on every branch, and is almost certainly part of why this PR is CONFLICTING. eosim.egg-info/ is setuptools build metadata; committing SOURCES.txt (+203 lines) means a stale file manifest ships in the tree and disagrees with pyproject.toml the moment either changes. out/logs/stm32f4.log is the output of a simulation run. None of the four is source.	git rm --cached all four and add .coverage, *.egg-info/, out/ to .gitignore. Doing this first will also shrink the conflict in finding 4.
4	Medium	PR state	The branch cannot merge. mergeable: CONFLICTING, mergeStateStatus: DIRTY, unchanged since 2026-08-29 — six days. For a 216-file change that renames the entire platform registry, the conflict surface only grows, and finding 6 means something else is waiting on it.	Rebase onto origin/master and resolve. Land finding 3 in the same push; a committed .coverage binary conflicts with every other branch that ran the tests.
5	Low	eosim/integrations/ecosystem.py:93-97	Dead branch: elif r.reason: detail = r.reason[:44] appears twice in immediate succession with an identical condition, so the second is unreachable. Copy-paste artifact in a file this PR largely rewrites.	Delete lines 96-97.
6	Low	eosim/integrations/ecosystem.py:376-386, :365-373	_parse_pytest and _missing_modules scan the last 4000 characters of stdout and stderr combined, not pytest's summary line. A test that prints 3 failed in its own output, or any log line containing No module named 'x', is counted as a result or as a missing dependency — and the DEPS classification at :350-354 turns on exactly that. Narrow, but this module's whole purpose is that reported numbers are the ones the tool produced.	Anchor to the summary: match on the final line matching ^=+ .* (passed|failed|error).* =+$, or run pytest with a machine-readable report. Same for _parse_ctest, which searches all of stdout for ctest's summary pattern — less exposed, because that pattern is distinctive.
7	Low	eosim/cli/main.py:40, :65	_find_platform calls PLATFORMS_DIR.iterdir() with no existence check, so a layout where the directory is absent — a wheel built before this PR's package-data change, or a partial install — raises FileNotFoundError and prints a raw traceback as the user's first experience of eosim run. .ai/tooling.md: raw tracebacks and internal exception strings do not surface as the primary message. Related and worth fixing in the same place: _load_registry hands PlatformRegistry a missing directory and eosim list then prints Available platforms (0): and exits 0 — I ran that directly against the pinned tree. An inventory command reporting success over an empty inventory is what let eos#129's nine green Install & Validate legs coexist with twenty-two red simulate jobs.	Guard _resolve_platforms_dir: if neither candidate exists, exit with a message naming both paths and saying the package is installed without its platform data. And make eosim list exit non-zero on an empty registry — it is the only cheap check that an install is usable.

The part that matters outside this repo

This PR fixes the defect that is currently failing 22 CI jobs on eos#129, and neither PR mentions the other.

eos#129 pins EoSim at commit 7dec3460 and installs it with pip install <path> instead of pip install -e. Every eosim run then dies with FileNotFoundError: .../site-packages/platforms, because at that commit platforms/ is a top-level directory of 199 files outside the eosim package, pyproject.toml declares only include = ["eosim*"], and eosim/cli/main.py resolves EOSIM_ROOT = Path(__file__).parent.parent.parent. Seven Nested Simulation legs, three Guest OS Install, eleven Simulate and both gates.

This branch is the fix, and it is a complete one:

• platforms/ → eosim/platforms/ (199 pure renames in the diffstat).
• pyproject.toml:69-73 adds [tool.setuptools.package-data] for "eosim.platforms" with the comment "Without this the wheel installs but eosim run cannot find any platform".
• eosim/cli/main.py:17-32, _resolve_platforms_dir(), prefers the packaged location and falls back to the repo-root path, with the failure mode written down: "An editable install hid this, because there EOSIM_ROOT is the checkout."
• .github/workflows/ci.yml:140 follows the move in the platform-count check.

It also closes the second open EoSim item listed in eos#129's 2026-09-03 comment: @click.version_option(version=__version__, prog_name="eosim") replaces the hardcoded version="2.0.0", so eosim --version stops printing a constant that could not distinguish a right install from a wrong one.

Practical consequence: eos#129 cannot be fixed properly until this merges. Its only options meanwhile are to keep pip install -e — which is what I recommended there — or to pin a commit from this branch, which it should not do. Worth a line in both PR bodies, and worth prioritising the rebase in finding 4 over anything else here.

What is right, and checked

• The core fix is real. test_c_repo:275-284 takes ctest's own summary via _parse_ctest and returns SKIP built OK; no tests registered when there is no summary line, rather than inferring a pass — which is the correct handling of ctest's exit-0-on-no-tests, and reaches the same place .ai/security.md's --no-tests=error rule aims at while keeping SKIP distinct from FAIL. test_python_repo:334-343 treats pytest exit 5 as "no tests collected" → SKIP and any other summary-less exit as FAIL. Both derive PASS from returncode == 0 **and** tests_failed == 0, never from one alone.
• DEPS is a genuine distinction, not a softening. :349-354 reaches DEPS only when failed == 0 and errors and missing — nothing asserted wrongly and the errors are import failures for named absent modules. run_ecosystem_tests:562-566 then counts those tests as total_blocked rather than folding them into total_failed, and summary() prints them under their own label. A repo in DEPS lands in repos_skipped, so it cannot be mistaken for a pass.
• test_repo_all exercising every detected build system is the right call and the body's evidence for it is concrete — ebuild's broken CMake configure was invisible behind its green Python suite.
• _build_dir_for puts CMake trees under ~/.cache/eosim/ecosystem, with the reasoning recorded: building into the checkout left an untracked directory in every repo the runner touched. That is the kind of thing this job is supposed to notice, and it is already fixed.
• The pytest -q and PYTHONPATH bugs are both real classes: -q -q suppresses the summary line entirely, which under the old code was indistinguishable from a failure, and offering both the repo root and src/ covers eDB's layout.
• pyproject.toml:8,19 are a §28 correction and deserve saying so. "World's most powerful universal simulation platform — supersedes 250+ tools across 20 domains" becomes "Multi-architecture embedded simulation platform for EoS — native engine plus Renode/QEMU backends", and Development Status :: 5 - Production/Stable becomes 4 - Beta. §28 defines the evidence each status claim requires and §27 warns against declaring maturity from vanity metrics; this is one of the few changes in this batch that walks a claim back rather than forward.

Architecture conformance

§21 Tier 1 — conforms, and this is Tier-1 work. EoSim sits in Tier 1 Foundation with eos, eBoot and ebuild, and §17 makes it "essential to developer onboarding and CI". Nothing here creates a dependency that points up a tier: the runner shells out to cmake, ctest, pytest, go, npm and make in other repos' checkouts and imports nothing from them.

§17 — the packaging half of this PR is the section's Install step. §17's chain is "Discover → Install → ebuild run --sim → EoS running" and §39 requires an independent developer to install the SDK and run EoS in simulation unaided. Moving the platform registry into the package is what makes an installed EoSim a working one. I have raised the remaining half — that EoSim still publishes no wheel or sdist at any of its thirteen releases — as an architecture proposal against §17 (.ai/autoreview/proposals/2026-09.md, "§17 makes Install an adoption step and nothing requires a tool to be installable"), triggered by eos#129. This PR satisfies the part of that proposal about runtime data being located through the packaging system rather than a source-tree path.

§28 Status, evidence and claims — the governing section for the runner itself, and the PR is a direct answer to it. §28 makes Implemented mean "code and functional tests" and Validated mean "hardware/CI/test reports"; a runner that reported every test passing whenever a repo compiled made both states unfalsifiable across the whole organisation. Finding 1 is the residue of that at the exit-code boundary.

Weakened checks — none. Everything here tightens. Two things that could look like weakening are not: SKIP replacing a false PASS is strictly more information, and DEPS is a narrowing of SKIP rather than of FAIL.

CI

Two red — Lint & Format (failing step: Ruff check) and Quick Checks (failing step: Lint) — and both fail on master too. Measured, rather than assumed: I extracted eosim/ and pyproject.toml at origin/master and at this head and ran ruff check eosim/ against each with the repo's own [tool.ruff] config (select = ["E","F","W","I","N","UP"], ignore = ["E501"]):

origin/master   Found 510 errors
this head       Found 507 errors

Per file, for the four this PR rewrites: eosim/cli/main.py 85 → 85, eosim/integrations/ecosystem.py 38 → 35, eosim/engine/native/__init__.py 8 → 8, eosim/engine/native/cpu/__init__.py 5 → 5. So this PR introduces no new violations and removes three. Caveat: my ruff is 0.16.5 and CI installs its own version, so the absolute counts will differ; the direction will not.

Five further jobs report skipping — Build Documentation, Build Package, Code Coverage, Simulator Smoke Test, Validate Platforms, and the Test (…) matrix — which is needs: on the failed lint, not a result. Green: Analyze (python), Analyze (javascript-typescript), Security Scan, Coverage Gate, Simulator Smoke, CI Summary.

CI Summary is green while two jobs in its own workflow are red and six are skipped. Same fail-open gate as noted on EoSim#15, and it is worth naming here specifically: this PR's entire subject is a runner that reported passes it had not observed, and the workflow reviewing it has a summary job doing the same thing.

Not checked

• Nothing in this PR was executed. No eosim ecosystem run, no pytest — the EoSim checkout has 179 modified .pyc files and the brief leaves dirty repos alone, so I read the branch through git show against a fetched ref and extracted copies, and built nothing. Findings 1, 2, 5, 6 and 7 are from reading the source; finding 3 is from the diffstat; the CI section is measured.
• The eosim list behaviour in finding 7 was measured against the tree at 7dec3460, the commit eos#129 pins, not at this head. This PR does not change _load_registry or the list command's exit path, so the conclusion carries — but I did not re-run it here.
• The reported table was not reproduced. 19 discovered | 7 passed | 1 failed | 11 skipped, 2420 tests run, and [PASS] eos cmake tests:22/22 are the author's, from 2026-08-28. One number is worth the author confirming rather than my guessing at: eos registers 38 tests on today's master — I configured and counted it, with and without -DEOS_PRODUCT=vbox_test, and got 38 both ways. The gap may simply be six days of new tests in eos. It is also the kind of gap this report should be able to answer for itself, which argues for the summary recording the configure flags each repo was built with.
• test_go_repo, test_node_repo and test_make_repo were read, not exercised. No Go, Node or Makefile-only repo in the workspace reaches them, so their counting logic is unverified — test_go_repo in particular derives counts from --- PASS / --- FAIL string occurrences in go test -v output, which is the pattern finding 6 is about.
• The 199 platform renames were verified as renames (0+ 0- in the diffstat, {platforms => eosim/platforms} in --stat), not read. I did not confirm that no platform.yml content changed beyond the path.
• eosim/cli/main.py is 1967 changed lines and I read roughly the first hundred plus the ecosystem command. The CLI rewrite is far larger than the PR title implies and I have not reviewed most of it; the same is true of the 1981 changed lines in tests/unit/test_gui.py and 1379 in test_core.py. If any of that is unrelated to the ecosystem runner, it belongs in its own PR — and if it is related, the body should say how.

---

Automated architecture review of 37fd2f1717f9 — scheduled, model claude-opus-5, checked against the EmbeddedOS Master Design v2.0. Advisory only: this reviewer never approves, requests changes, or merges. Reply here to discuss or push back — a wrong finding is a bug worth reporting.