ci: add one job branch protection can require - #90
Conversation
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
The gate was run against a real failure, not just reasoned aboutPushed a deliberately failing test to a branch on my fork so the whole workflow Job results in that run: And the gate's own log: Three things that confirms, which the workflow file alone does not:
The self-test branch has been deleted; nothing from it is in this PR. |
srpatcha
left a comment
There was a problem hiding this comment.
Review — eBoot#90 "ci: add one job branch protection can require"
head: 8896d6a author: Kartikey1306 ci: pass
Verdict: The gate itself is correct and the evidence behind it is the strongest in this batch — a real run with a deliberately failing job, showing if: always() firing, skipped counted as a failure, and the gate reporting red, plus a three-way mutation sweep on the test. CI Gate reports pass in this PR's own checks.txt, so the job works. Two things need saying: the gate covers one of sixteen workflow files, not the repository, and the fail-open shape it fixes still exists in a sibling gate that is not tracked anywhere.
Findings
| # | Severity | File:line | Finding | Recommended fix |
|---|---|---|---|---|
| 1 | Medium | .github/workflows/ci.yml:180-217; tests/unit/test_ci_gate.py:82 |
CI Gate gates needs: [test, build-arm, static-analysis] — three jobs in ci.yml. The repository has 16 workflow files, and this PR's own checks.txt reports 26 checks across at least five separate runs: run …142934 is ci.yml (the three gated jobs plus release and the gate), but …142914 contributes Analyze (C/C++), …142986 contributes Cross-Platform ×3, EoSim ×11 and Simulation Gate, …143025 contributes Host Build & Tests, Cross-compile STM32F4 and a second Static Analysis, and CodeQL runs on its own. So the maintainer action the PR closes with — "Settings → Branches → master → Require status checks → add CI Gate" — would leave roughly 19 checks unrequired, including CodeQL, every EoSim platform leg, all three Cross-Platform legs and a second host build. master could still go red from any of them. The test has the same boundary: WORKFLOW = … / ".github" / "workflows" / "ci.yml" (tests/unit/test_ci_gate.py:82), so test_gate_covers_every_job_that_runs_on_a_pull_request protects ci.yml and nothing else — a job added to build.yml, codeql.yml or simulation-test.yml is not wired in and not noticed, which is precisely the silent-drift failure the PR is built to prevent. |
Say what the gate covers, and cover the rest. Concretely: (a) reword the body and the maintainer action to "require CI Gate and CodeQL, Simulation Gate, Host Build & Tests, Analyze (C/C++)" — or whichever set the maintainers judge required — so nobody reads one name as sufficient; (b) generalise the test to for wf in (Path(...) / ".github" / "workflows").glob("*.yml"), asserting that each workflow producing pull-request checks has a gate job whose needs covers every non-tag-only job in it. That turns a one-file guard into the repository-wide invariant the PR is arguing for, and it is the version that keeps working. |
| 2 | Medium | .github/workflows/simulation-test.yml:93-109 |
The exact fail-open shape this PR removes from ci.yml is live in a sibling gate that the PR does not mention. Simulation Gate declares needs: [simulate, cross-platform] and then gates on one of them: it prints both results, tests only if [ "${{ needs.simulate.result }}" != "success" ], and otherwise prints ✅ All simulation checks passed. A red or skipped cross-platform — three OS legs — passes the gate. Simulation Gate reports pass in this PR's checks.txt and is a name a maintainer would plausibly require, so it is a required-check candidate that cannot fail for two thirds of what it claims to summarise. This PR's own words apply verbatim: "a job that did not run did not verify anything, and treating that as a pass is the fail-open shape #38, #59 and #82 removed from the boot path." The same pattern is in eosim-sanity.yml:135-156 (EoSim Sanity Gate, gating 1 of 5 dependencies) — that one is already recorded in .ai/autoreview/proposals/2026-09.md against #81, so I only note it; Simulation Gate appears in no proposal or issue I can find. |
Replace both gates' bodies with the toJSON(needs) + jq form this PR introduces, which is dependency-list-agnostic and cannot fall out of step with needs. It is a copy of the block at ci.yml:196-217. Doing it here would make this PR the single change that establishes one correct gate idiom repository-wide instead of adding a third idiom alongside two broken ones. |
| 3 | Low | .github/workflows/ci.yml:39 |
This PR and #88 both rewrite the same pip3 install line — #90 adds pyyaml, #88 adds cryptography — so whichever merges second conflicts. Both additions are necessary and neither is a substitute. Related and worth noting for the maintainers: the 1 skipped in this PR's own verification table (pytest tests/ → "25 passed, 1 skipped") is the silently-skipped signing suite #88 diagnoses, so the two PRs are independent evidence of the same weakness in this installer line. |
Trivial to resolve; land as pip3 install pytest pytest-cov pyyaml cryptography. Flagging so it is not resolved by dropping one. |
Verified clean, since a gate that fails open would be the serious defect here: the step cannot silently pass. GitHub's default shell for run: on Linux is bash -e {0}, so a jq failure inside bad=$(printf '%s' "$RESULTS" | jq -r …) fails the assignment and therefore the step, rather than leaving bad empty and falling through to echo "All jobs succeeded.". The jq filter selects on .value.result != "success", which catches failure, cancelled and skipped alike — matching the author's observed log, where build-arm: skipped was named as a reason for failure. test_ci_gate.py does a plain import yaml at module scope rather than pytest.importorskip, so a missing pyyaml is a collection error and a hard failure — the right choice, and notably stricter than the importorskip pattern used by the signing tests.
Architecture conformance
Conforms. §21 Infrastructure — ".github, website, docs, CI templates — Governance, release automation and documentation"; a workflow gate and its test belong exactly here. tests/unit/ is the right home for the test per .ai/architect.md's eBoot layout ("tests/ unit, functional, fuzz, performance, simulation"). Nothing in stage0/, stage1/, core/, hal/ or boards/ is touched, so §5.1's minimal-and-auditable TCB requirement is unaffected and no dependency points up a tier. §23's release model is respected: release is excluded from the gate because it is tag-gated, and test_jobs_left_out_of_the_gate_are_genuinely_tag_only enforces that the exclusion cannot be widened casually — that test is the best part of the PR, because it makes the exclusion a checked rule instead of a comment.
This is the enforcement side of a design gap already recorded. .ai/autoreview/proposals/2026-09.md carries "The evidence policy is silent on checks that verify nothing" (§28, triggered by eAI#39, eAI#41, eBoot#81), whose proposed §28.2 states: "An aggregating gate job must fail on any non-success among its dependencies, and must not print a summary asserting more than it checked." That is precisely what ci-gate implements and precisely what finding 2 shows Simulation Gate violates. No new proposal appended — this PR is evidence for the existing one, and I have added nothing to it because the text already covers the case.
Proposed changes
- Reword the body and the maintainer action to name the full required set rather than one name (finding 1a). Documentation only, and it is the difference between a maintainer closing the gap and believing they have.
- Generalise
test_ci_gate.pyto every workflow file (finding 1b). - Port the
toJSON(needs)gate body intosimulation-test.yml:98-109andeosim-sanity.yml:141-156(finding 2), coordinating the latter with #81. - Merge the
pip3 installline with #88's rather than choosing between them (finding 3).
Items 1 and 4 are trivial. Item 2 is the one that makes the guarantee durable; item 3 is the one that makes it true today. None of them argues against merging the gate as it stands — it is a strict improvement on required_status_checks: null, and it should not wait on any of the above.
Not checked
- Nothing was executed. No
pytest, noctest, noyaml.safe_load, no mutation sweep. Reproducing needs the PR head checked out, which the run brief forbids. The verification table —25 passed, 1 skipped,6 passed,ctest 20/20, and the three mutations each caught — is the author's and unverified by me. The gate's live behaviour is better evidenced than anything else in this batch: the 2026-09-01 comment shows a real run with a real failure and quotes the gate's own error output, andchecks.txthere independently showsCI Gate pass 2s. I did not see that run and cannot confirm the log, but the two agree with each other and with the workflow source. required_status_checks: nullwas not confirmed. That is the premise of the whole PR and needsgh api repos/embeddedos-org/eBoot/branches/master/protection, which I did not call. It rests on the author's statement and the quoted maintainer comment.- The check-to-workflow mapping in finding 1 is inferred from run IDs in
checks.txt, not from reading all 16 workflow files. I readci.yml,simulation-test.ymlandeosim-sanity.yml. Which workflow producesAnalyze (C/C++),Host Build & Tests,Cross-compile STM32F4and the secondStatic AnalysisI did not establish, only that their run IDs differ fromci.yml's. The count of ~19 uncovered checks follows from that grouping and should be confirmed before being quoted at a maintainer. - Finding 2 is from reading
simulation-test.yml, not from a failing run. I did not construct a case wherecross-platformfails andsimulatesucceeds to observeSimulation Gatepass anyway. The conditional is unambiguous on inspection, but the empirical form is what the author did forci-gateand is what would settle it. - The other thirteen workflow files were not examined for gate jobs, fail-open conditionals, or jobs that ought to be required.
nightly.yml,weekly.yml,scorecard.yml,build.yml,codeql.yml,book-build.yml,video-build.yml,deploy-pages.yml,cross-repo-dispatch.yml,release.yml,sync-release-branch.yml,auto-assign.ymlandclaude-code-review.ymlare all unread. Finding 1b exists partly because I could not rule out more instances of finding 2 among them. - 24 of 26 checks pass;
Create GitHub Releaseandassignreportskipping, both expected on a PR and both correctly outside the gate. No required check failing — though as finding 1 notes, at this commit there are no required checks at all. mergeStateStatus: BLOCKED,mergeable: MERGEABLE. No merge attempted; expect theci.ymlconflict with #88 described in finding 3.
Automated architecture review of 8896d6ade7eb — scheduled, model claude-opus-5, checked against the EmbeddedOS Master Design v2.0. Advisory only: this reviewer never approves, requests changes, or merges. Reply here to discuss or push back — a wrong finding is a bug worth reporting.
…rged broken master (22d8f8b) does not compile. Two independent double-merges, both the same shape: two PRs fixing adjacent things landed on stale bases, each was green on its own branch, and the result was never rebuilt. 1. include/eos_image.h — embeddedos-org#93 replaced reserved[30] with tlv_len (2) + tlv_hash[28], preserving every offset. embeddedos-org#87 merged afterwards carrying asserts written against the older struct: error: no member named 'reserved' in 'eos_image_header_t' (x2) embeddedos-org#93 already asserts tlv_len at 62 and tlv_hash at 64, so the offset assert was a duplicate; the width assert had no replacement and is restored as two asserts covering both halves of the same 30-byte span. No offset moves and the wire format is unchanged. 2. core/ed25519_verify.c — embeddedos-org#86 and embeddedos-org#57 both landed a subgroup guard, so the file carried two byte-identical point_is_identity() definitions: error: redefinition of 'point_is_identity' Only embeddedos-org#57's public_key_is_valid_subgroup() is wired to the call site, so embeddedos-org#86's key_has_prime_order() was dead. Kept the live function, folded embeddedos-org#86's fuller rationale onto it, deleted the duplicate. 3. tests/unit/test_ed25519.c — collateral from the same merge. Two copies of test_ed25519_identity_key_forgery_rejected, main() calling it twice and two tests not at all, and test_ed25519_low_order_R_with_a_valid_key_is_not_a_forgery referencing k_low_order[] and messages[] that the merge had dropped. While restoring the corpus, corrected it (review finding on embeddedos-org#86): the array claimed to hold "the eight low-order point encodings" and held five. Every order here was computed rather than copied — decode y, recover x, add the point to itself until it reaches the identity — giving 1, 2, 4, 4, 8, 8, 8, 8. Missing before: y=0 with the sign bit set, and both sign-flipped order-8 encodings. D9FF..FF was in the array and is not a low-order point at all — no x satisfies the curve equation for that y — so it moves to a separate k_non_canonical[], with EDFF..FF7F (y=p) and EEFF..FF7F (y=p+1). tests_run was assigned a literal (11) in main() and never incremented, which is how the duplicate call and the two unregistered tests went unnoticed. The TEST macro now increments it, so the total cannot drift. Verified: cmake -DEBLDR_BUILD_TESTS=ON on master FAILS to build, 3 errors same with this commit builds clean ctest 21/21 PASS ctest -DEBLDR_SANITIZE=ON (ASan+UBSan) 21/21 PASS pytest tests/ 24 passed, 1 skipped test_ed25519 14/14 PASS (was 11 claimed, 12 run) discrimination, with `public_key_is_valid_subgroup` disabled: test_ed25519_low_order_keys_rejected FAILS, as it must test_ed25519_non_canonical_... still PASSES — those are refused by unpackneg() on canonicality, a different mechanism, which is the reason they are held in a separate array rather than counted among the eight.
…gate that failed open Answers the review on embeddedos-org#90. Finding 1 (Medium) -- `CI Gate` gates three jobs in ci.yml. This repository has 16 workflow files and a PR head reports 26 checks across five runs, so the maintainer action in the body -- require one name -- would have left CodeQL, every EoSim platform leg, all three Cross-Platform legs and a second host build unrequired, and `master` could still go red from any of them. The rot-guard had the same boundary: WORKFLOW was hardcoded to ci.yml, so a job added to build.yml or codeql.yml was neither covered nor noticed. Adds two tests over every workflow with a `pull_request` trigger: REQUIRED_CHECKS names what a maintainer must actually require, NO_GATE excuses the rest with a reason about the workflow itself, and a third test asserts each gated workflow really has a job displaying under the name given. Finding 2 (Medium) -- `Simulation Gate` had the exact fail-open shape this PR removes from ci.yml: `needs: [simulate, cross-platform]`, then it printed both results and branched on `simulate` alone before printing "All simulation checks passed". A red or skipped `cross-platform` -- three OS legs -- passed it. Ported the `toJSON(needs)` + `jq` body, which is dependency-list-agnostic and cannot fall out of step with `needs:`. And made it a rule rather than a one-off: test_no_aggregating_gate_ignores_part_of_its_needs walks every gate in every pull-request workflow and fails if a declared dependency is never compared. Two refinements were needed to make it mean something: - printing a result is not testing it. The first version grepped for `needs.X.result` anywhere in the script, which Simulation Gate satisfied with its echo line. Only a line that compares counts. - book-build.yml's `summary` writes a step summary and claims no verdict. It is reporting, not gating, so the check applies only to jobs that either `exit 1` or assert that everything passed. Finding 3 (Low) -- this branch had replaced master's `pip3 install -r requirements.txt pytest-cov` with a hand-maintained list to add pyyaml, which regressed the guard in tests/unit/test_requirements.py: these jobs run pytest over tests/ but never install from requirements.txt ... ['ci.yml:test'] Restored the requirements.txt install; pyyaml was already declared there, so the hand list was not needed at all. That also removes the collision with embeddedos-org#88, which edits the same line. Verified: pytest tests/ 47 passed ctest 21/21 PASS yaml.safe_load of ci.yml, simulation-test.yml, book-build.yml all parse discrimination, both ways: - reverting Simulation Gate to its fail-open body gives simulation-test.yml:sanity-gate declares needs ['simulate', 'cross-platform'] but never tests ['cross-platform'] - with the toJSON(needs) body in place, 9 passed Refs embeddedos-org#90
8896d6a to
1bd6a22
Compare
… check Answers the review on embeddedos-org#81. The install fix was right; the workflow it revives could not have failed for most of what it says it verifies. Finding 1 (Medium) -- the step named "Validate all platform configs" ran `eosim list && eosim doctor` with no platforms/ copy in that job. `eosim list` prints "Available platforms (0)" without it and exits 0, so the step passed having validated nothing -- and the same shape was in windows-sanity and macos-sanity. All three now copy platforms/ from the pinned checkout and assert a non-zero count via discover_platforms(), because the exit code is exactly what cannot be trusted here. Finding 2 (Medium) -- nested-guest-install cloned EoSim a second time with no --branch for its platforms/ copy, so the package came from v${EOSIM_VERSION} while the platform data came from whatever the default branch pointed at that morning. Now copies from the same pinned checkout; the second clone is gone. Finding 3 (Medium) -- `eosim --version` was printed and never asserted, while the tag and the package's declared version disagree upstream (v1.5.0 ships "eosim, version 2.0.0"). Pinning to a tag therefore does not pin what the name suggests, and nothing would have noticed if the tag moved. Now asserted. The mismatch itself is EoSim's bug and is raised there rather than worked around here. Finding 4 (Medium) -- sanity-gate failed only on install-validate and then printed "All EoSim sanity checks passed", which it would do with the other four jobs red. Replaced with the toJSON(needs) + jq body from ci.yml, which cannot fall out of step with `needs:`. embeddedos-org#90 adds a test that enforces this across every gate in the repository; this gate passes it. Verified: yaml.safe_load of eosim-sanity.yml parses, 6 jobs the gate now iterates toJSON(needs), no longer branches on install-validate alone, and no longer prints an "all passed" claim embeddedos-org#90's test_no_aggregating_gate_ignores_part_of_its_needs, run against this workflow: eosim-sanity.yml is not among its offenders pytest tests/ 47 passed ctest 21/21 PASS NOT RUN: the workflow itself. It is `on: schedule` + `workflow_dispatch` only, so none of this PR's checks execute it -- which is finding 5, and it is the one piece of evidence this PR cannot produce from a fork branch without a maintainer dispatching it. The install sequence was verified locally end to end (clone -> pip install -> eosim --version 2.0.0 -> doctor -> run am62x --headless PASSED); the assertions added here are not covered by that and remain unexecuted. Refs embeddedos-org#81
… check Answers the review on embeddedos-org#81. The install fix was right; the workflow it revives could not have failed for most of what it says it verifies. Finding 1 (Medium) -- the step named "Validate all platform configs" ran `eosim list && eosim doctor` with no platforms/ copy in that job. `eosim list` prints "Available platforms (0)" without it and exits 0, so the step passed having validated nothing -- and the same shape was in windows-sanity and macos-sanity. All three now copy platforms/ from the pinned checkout and assert a non-zero count via discover_platforms(), because the exit code is exactly what cannot be trusted here. Finding 2 (Medium) -- nested-guest-install cloned EoSim a second time with no --branch for its platforms/ copy, so the package came from v${EOSIM_VERSION} while the platform data came from whatever the default branch pointed at that morning. Now copies from the same pinned checkout; the second clone is gone. Finding 3 (Medium) -- `eosim --version` was printed and never asserted, while the tag and the package's declared version disagree upstream (v1.5.0 ships "eosim, version 2.0.0"). Pinning to a tag therefore does not pin what the name suggests, and nothing would have noticed if the tag moved. Now asserted. The mismatch itself is EoSim's bug and is raised there rather than worked around here. Finding 4 (Medium) -- sanity-gate failed only on install-validate and then printed "All EoSim sanity checks passed", which it would do with the other four jobs red. Replaced with the toJSON(needs) + jq body from ci.yml, which cannot fall out of step with `needs:`. embeddedos-org#90 adds a test that enforces this across every gate in the repository; this gate passes it. Verified: yaml.safe_load of eosim-sanity.yml parses, 6 jobs the gate now iterates toJSON(needs), no longer branches on install-validate alone, and no longer prints an "all passed" claim embeddedos-org#90's test_no_aggregating_gate_ignores_part_of_its_needs, run against this workflow: eosim-sanity.yml is not among its offenders pytest tests/ 38 passed ctest 21/21 PASS NOT RUN: the workflow itself. It is `on: schedule` + `workflow_dispatch` only, so none of this PR's checks execute it -- which is finding 5, and it is the one piece of evidence this PR cannot produce from a fork branch without a maintainer dispatching it. The install sequence was verified locally end to end (clone -> pip install -> eosim --version 2.0.0 -> doctor -> run am62x --headless PASSED); the assertions added here are not covered by that and remain unexecuted. Refs embeddedos-org#81
Finishes finding 3 from the review on embeddedos-org#90. Findings 1 and 2 were already addressed in 1bd6a22; this is the one left. test_gate_fails_on_any_non_success_result searched the gate's `run:` text for `!= "success"` and `exit 1`. That is a string match on an implementation, not a check of behaviour: it passes for those tokens sitting in a comment or an unreachable branch, and fails for a correct rewrite expressing the same rule differently. The only real evidence that the rule works was the fork experiment, which is a one-off nothing re-runs. The rule now lives in .github/scripts/ci-gate-check.sh, and pytest runs it against real inputs: success, failure, skipped, cancelled, a mixed set, and an empty context. Both gates -- ci.yml's `CI Gate` and simulation-test.yml's `Simulation Gate` -- call that one script, and a further test asserts they do, so the behavioural cases cover every gate rather than one. The script also refuses an empty or null `needs` context. No results is not the same as no failures, and a gate that passes when it was handed nothing is the same fail-open shape in a different place. Verified by mutation: weakening the rule to `== "failure"` fails 3 of 18, and a gate that stops calling the script fails 1 of 18. Restored, 18 pass. 42 Python tests pass (1 skipped), ctest 20/20, all 16 workflow files parse.
|
Finding 3 addressed at The rule is now executed, not pattern-matched. It lives in
Both gates — The last row is an addition beyond the finding: the script refuses an empty or Verified by mutation: weakening the rule to On finding 2, and a sibling in the other repoWorth reporting back because your finding generalised. The same shape existed needs: [build-kernel, simulate-qemu, test-middleware, test-cad-pipeline]
if: always()prints PASS/FAIL for all four — and its only So that is three instances of one pattern across two repos: Finding 3's conflict noteAgreed, and it resolves as you said — |
srpatcha
left a comment
There was a problem hiding this comment.
Review — eBoot#90 "ci: add one job branch protection can require"
head: ffb5ff9 author: Kartikey1306 ci: pass
Verdict: The gate is correct — if: always(), non-success of any kind fails, release
correctly outside it, and the rule in an executable script rather than grepped out of YAML.
I ran the script against every result shape and it behaves as documented, except on an empty
dependency set. The bigger problem is that the PR body and tests/unit/test_ci_gate.py
disagree about the deliverable, and the tests are right: CI Gate covers ci.yml and
nothing else, so the maintainer action in the body leaves most of eBoot's pull-request
surface unrequired.
Findings
| # | Severity | File:line | Finding | Recommended fix |
|---|---|---|---|---|
| 1 | Medium | pr body ("What is still a maintainer action"); tests/unit/test_ci_gate.py:26-35, 159-176 |
The body says ci-gate is "the single name to require" and the maintainer action is "add CI Gate". The PR's own test file says otherwise, in as many words: "CI Gate covers ci.yml and nothing else -- cross-workflow needs is not something GitHub offers -- so the required set is this mapping, not one name", and test_every_pull_request_workflow_is_accounted_for documents that requiring one name "leaves every other workflow's checks unrequired". Enumerated the workflows: on pull_request eBoot also runs build.yml (Host Build & Tests, Cross-compile STM32F4, Static Analysis), codeql.yml (Analyze (C/C++)), simulation-test.yml (Simulation Gate) and book-build.yml. A maintainer who follows the body requires one check and believes the gap is closed. |
Replace the maintainer action with the actual list: CI Gate, CodeQL, Simulation Gate (requirable once this PR lands — see finding 2), and either build.yml's three names or a gate for it (finding 3). The test file's REQUIRED_CHECKS mapping is the right shape; make the body quote it instead of contradicting it. |
| 2 | Medium | tests/unit/test_ci_gate.py:51-57 |
simulation-test.yml sits in NO_GATE because its gate "tests only needs.simulate.result and then prints 'All simulation checks passed'… Requiring it today would assert more than it checks -- fixed below rather than excused". This PR is the fix: simulation-test.yml:95-109 now pipes toJSON(needs) through the shared script. So the one workflow this PR made trustworthy is still the one it excuses, and Simulation Gate stays unrequired. |
Move "simulation-test.yml": "Simulation Gate" into REQUIRED_CHECKS and delete the NO_GATE entry. test_gated_workflows_really_have_their_gate will then check it, and the entry becomes part of the maintainer action in finding 1. |
| 3 | Medium | tests/unit/test_ci_gate.py:47-50 |
build.yml is excused because its jobs are "under stable names that can be required directly; wrapping them adds a layer without adding coverage." That is precisely the argument the PR body rejects two paragraphs earlier for ci.yml: "Requiring test, build-arm and static-analysis individually works today and stops working the next time a job is added, silently." build.yml has three such jobs on every push and pull request — a second host build and a second static analysis — and no gate and no rot-guard. The reasoning cannot be sound in one file and unsound in the other. |
Give build.yml the same three-line gate reusing .github/scripts/ci-gate-check.sh, and list it in REQUIRED_CHECKS. If the real reason to leave it alone is that build.yml and ci.yml overlap and one should go, say that instead — it is a better argument, and it is a maintainer decision worth an issue. |
| 4 | Medium | .github/scripts/ci-gate-check.sh:19-36 |
The gate passes on an empty dependency set. Verified: printf '{}' | .github/scripts/ci-gate-check.sh → rc=0, "All jobs succeeded." The guard at :19 catches "" and "null" only; jq 'to_entries[]' over {} yields nothing, bad is empty, and the script reports success. The header says "a job that did not run did not verify anything" — zero jobs is the limit case of that, and it is treated as a pass. A needs: key that is deleted, mistyped, or narrowed to [] turns the required check green having verified nothing: the fail-open shape the script exists to remove. test_gate_script_refuses_an_empty_context parametrizes "" and null but not {}. |
count=$(printf '%s' "$results" | jq 'length'); [ "$count" -gt 0 ] || { echo "::error::CI Gate had no dependencies; refusing to pass."; exit 1; } before the bad computation, and add "{}" to that test's parameters. |
| 5 | Medium | .github/workflows/ci.yml:30-40 |
Seven lines of comment are deleted and the pip3 install line they explain is unchanged, byte for byte. The deleted text explained why the job installs -r requirements.txt rather than a hand-maintained list, and why pytest-cov is kept separate — that is the eBoot#88 silent-skip story, still true and still the reason the line looks the way it does. The body justifies the edit as "pyyaml is added to the pytest installer, which had only pytest pytest-cov"; nothing was added — requirements.txt:1 already declares pyyaml>=6.0 and master already installs -r requirements.txt pytest-cov. .ai/reviewer.md asks whether anything was deleted or overwritten without explanation. |
Restore the deleted comment and append the pyyaml line to it rather than in place of it. The install line needs no change. |
| 6 | Low | pr body ("Verification") | "Branches from master directly; unlike the other two repos, master here is green." origin/master@22d8f8b does not compile — duplicate point_is_identity in core/ed25519_verify.c:281,338 and include/eos_image.h:135,142 asserting on a reserved member that #93 removed. gh run list --branch master shows CI — eBoot, eBoot Build & Test and CodeQL all failure on that commit. This PR's own diff carries the fix for both, which is why it builds. The claim is the strongest argument for the PR, stated backwards: master is red right now, and a required check is what would have stopped it. |
Say that. "master is red today for the same reason #70 made it red — nothing built the merge result" is a better opening than the incident from 08-31. |
| 7 | Low | pr body ("The test is the part that keeps working", "Verification") | "tests/unit/test_ci_gate.py (6 tests)" and "PASS — 6 passed". The file defines 12 test functions, two of them parametrized. "ctest PASS — 20/20" does not match either: the head's suite is 21 tests (measured), and it cannot have been 20 on master, which does not build. |
Re-measure and quote the current numbers. The file is better than the body claims, which is an odd way to undersell it. |
| 8 | Low | tests/unit/test_ci_gate.py:99-103 |
_only_runs_on_tags returns true for any if containing the substring refs/tags, so a negated or unrelated condition mentioning tags would count as tag-only and let a job out of the gate. It is the check that decides which jobs may be excluded, so it is worth more than a substring. |
Match the shape the workflow actually uses: re.search(r"startsWith\(\s*github\.ref\s*,\s*'refs/tags", condition), and reject conditions containing !. |
Architecture conformance
Master design §28 (status/evidence policy — a claim needs the evidence its state requires),
§23 (release model: master is the line of development, every PR merges there), §21
Infrastructure tier (".github, website, docs, CI templates — governance, release automation");
.github/STANDARDS.md "Release model" (master is written by maintainers via PRs).
Conforms. This is Infrastructure-tier work on Infrastructure-tier files and introduces no
dependency in either direction. Nothing in the diff imports, includes or links anything.
The design mandates the evidence model but never says what enforces it at merge time. §23.1
and §28 describe channels and claim states; .github/STANDARDS.md describes branch roles.
Neither requires a status check on master, which is why required_status_checks: null is
not a policy violation today — it is a gap. That is worth a design proposal rather than a
finding against this PR, and I have filed one.
Verified by running, on master + this patch:
.github/scripts/ci-gate-check.sh, stdin -> rc:
{"test":{"result":"success"},"build-arm":{"result":"success"}} -> 0 "All jobs succeeded."
{"test":{"result":"success"},"build-arm":{"result":"skipped"}} -> 1 " build-arm: skipped"
{"test":{"result":"failure"}} -> 1 " test: failure"
{} -> 0 finding 4
null -> 1
(empty) -> 1
yaml.safe_load of both edited workflows -> OK
ci.yml jobs: test, build-arm, static-analysis, release, ci-gate
ci-gate name='CI Gate' if='always()' needs=[test, build-arm, static-analysis]
release if="startsWith(github.ref, 'refs/tags/v')" -> genuinely tag-only
-> the gate covers every non-tag job in ci.yml. Complete for this workflow.
cmake --build build/host && ctest --no-tests=error -j4 -> 21/21 passed
The design decisions are right and worth keeping on the record: if: always() so a failure
is a red X rather than a pull request waiting on a status that never arrives; skipped
treated as failure; release outside the gate because a required check that never reports
blocks merges forever; and the rule in a script so test_gate_script_accepts_only_all_success
executes it instead of pattern-matching the YAML. test_no_aggregating_gate_ignores_part_of_ its_needs is the strongest thing here — it catches the "prints one result, branches on
another" shape generically, across every pull-request workflow, and it is what found the
Simulation Gate defect this PR also fixes.
Proposed changes
- Rewrite the maintainer action as the real required set; promote
simulation-test.ymlinto
REQUIRED_CHECKS(findings 1, 2). - Gate
build.yml, or replace itsNO_GATEreason with the honest one (finding 3). - Reject an empty dependency set in the script, and parametrize
{}(finding 4). - Restore the deleted
ci.ymlcomment and append to it (finding 5). - Correct the three measurements in the body (findings 6, 7).
- Tighten
_only_runs_on_tags(finding 8).
Not checked
pytestis not installed in this environment, so none oftest_ci_gate.pywas run.
Every claim I make about that file is from reading it; the script behaviour in the table
above was executed directly, not through pytest. The body's mutation sweep ("drop
build-armfromneeds→ 2 failed", etc.) was not reproduced.- Branch protection itself was not inspected.
required_status_checks: nullonmasteris
taken from the PR body and the cross-references to eos#92 / ebuild#87; I did not query the
branch-protection API, and this run has no authority to change it. WhetherCI Gateis
actually addable under the current ruleset is unverified. - The gate was not observed running on GitHub.
checks.txtfor this head listsCI Gate
among the passing checks, which is evidence the job runs and succeeds on a green branch —
not evidence it fails correctly on a red one. book-build.yml's exclusion ("a docs failure should not block a code merge") is a
maintainer policy call, not a technical one, and I have not second-guessed it.mergeStateStatus: BLOCKED,reviewDecision: REVIEW_REQUIRED— not investigated.- This PR also carries the
core/ed25519_verify.c,include/eos_image.hand
tests/unit/test_ed25519.cchanges that repairmaster's broken host build, identical to
eBoot#81, #88 and #89. Reviewed under #81; not re-litigated here.
Automated architecture review of ffb5ff90bf14 — scheduled, model claude-opus-5, checked against the EmbeddedOS Master Design v2.0. Advisory only: this reviewer never approves, requests changes, or merges. Reply here to discuss or push back — a wrong finding is a bug worth reporting.
…harnesses Answers the second review on embeddedos-org#84. Finding 2 (Medium) -- eos_fdt_validate() and eos_fdt_get_prop() kept an out-of-bounds read by design. Both dereferenced hdr->totalsize before any length was known and handed that attacker-controlled value to the _sized form as its bound, so calling either on a short buffer read past it before a single check had run. The header called that a caller warrant; a warrant is not a check, and it is the exact bug this PR exists to fix. Neither had an in-tree caller. Removed rather than documented. There is now one form of each entry point and it always takes the length: int eos_fdt_validate(const void *fdt_blob, uint32_t avail); int eos_fdt_get_prop(const void *fdt, uint32_t fdt_len, ...); An exported unsafe twin in a TCB header is a future boot-path caller reintroducing the bug with no compiler complaint, which is worth more than the convenience of a one-argument call. eos_fdt_load() already passed max_size. The test that pinned the wrapper's behaviour is gone with the wrapper, and get_prop_sized_exact() collapsed into get_prop_exact() since they became the same function. Finding 3 (Medium) -- tests/fuzz/ was built by nothing. EBLDR_BUILD_FUZZ defaults OFF and no job set it, so the harness added here joined five others that no CI job compiles. A harness that is never built cannot fail to build, which is how fuzz_devicetree came to declare a function that did not exist and sit there unnoticed (eos#50). Adds a `fuzz-build` job: configure with clang, build every harness, and run each for five seconds over its own generated inputs. That is not a campaign -- it is enough to catch a harness that no longer compiles or crashes at once, which is the failure this repo has actually had. Note it needs EBLDR_BUILD_TESTS=ON as well: tests/fuzz/ is added from tests/CMakeLists.txt, so EBLDR_BUILD_FUZZ alone configures cleanly and builds no harness at all -- the job would have passed having compiled nothing. Found that locally before writing the job, not after. NOT RUN, and this is the honest limit: this host has no libFuzzer runtime (libclang_rt.fuzzer_osx.a is absent from the Xcode toolchain), so the link step cannot be reproduced here for any harness, old or new. What I verified is that CMake configures with both flags and reports all six targets -- "Fuzz targets: fuzz_image_verify, fuzz_recovery_protocol, fuzz_fw_update, fuzz_crypto, fuzz_bootctl, fuzz_fdt" -- and that fuzz_fdt.c passes `cc -fsyntax-only`. The link and the smoke run are CI's to show, and this job is what makes them visible. Coordination note: embeddedos-org#90 adds a `CI Gate` whose needs list is [test, build-arm, static-analysis]. Whichever of embeddedos-org#84 and embeddedos-org#90 lands second must add fuzz-build to that list -- and embeddedos-org#90's own test_gate_covers_every_job_that_runs_on_a_pull_request fails loudly if it is not, which is the guard working rather than a trap. Finding 1 (High) is a PR-body correction, made there: the diff carries embeddedos-org#94's master repair because this branch is stacked on it, and the body described only the FDT parser. Verified: ctest 22/22 PASS pytest tests/ 38 passed test_fdt_loader 13 tests PASS grep for a length-free entry point none remains in the header Refs embeddedos-org#84
required_status_checksisnullon this repository'smaster— the same gapeos#92 and ebuild#87 track. @srpatcha noted it applies here too:
This repo has paid for it. #70 landed a
TEST()macro that had lost a\linecontinuation;
masterwas red from 08-31 08:07 until #77, and the 24 resultingerrors all pointed away from the cause. Nothing built the merge result before it
became
master.Why the existing checks cannot be required as they are
releaseis skipped on every pull request (if: startsWith(github.ref, 'refs/tags/v')), and a required check that is skipped never reports — thepull request waits for a status that never arrives, which presents as
infrastructure flake rather than a policy.
Requiring
test,build-armandstatic-analysisindividually works today andstops working the next time a job is added, silently.
What this adds
One job,
ci-gate, displayed asCI Gate— the single name to require:if: always()so a real failure shows as a red X rather than a hang.skippedincluded — a job that did notrun did not verify anything, which is the fail-open shape fix(image_verify): fail closed when the CRC32 integrity path cannot read flash #38, fix(keystore): fail closed when the OTP trust anchor cannot be read #59 and fix(secure-boot): a debug lock that failed must not report a successful boot #82
removed from the boot path.
releaseis excluded deliberately, and the test enforces that the onlyjobs allowed outside the gate are tag-only.
The test is the part that keeps working
tests/unit/test_ci_gate.py(6 tests) parsesci.ymland fails if a job isadded without being wired into the gate.
Verified by mutation:
build-armfromneedsif: always()pyyamlis added to the pytest installer, which had onlypytest pytest-cov.Verification
pytest tests/pytest tests/unit/test_ci_gate.pyctestyaml.safe_loadof the edited workflowBranches from
masterdirectly; unlike the other two repos,masterhere isgreen.
What is still a maintainer action
Settings → Branches → master → Require status checks → add
CI Gate. Thesame PR is open on ebuild (#103) and eos (#121), so the name is identical in all
three.