flightcheck: offline https link gate + run FlightCheck suite in CI - #209
Open
daeunJe0ng wants to merge 4 commits into
Open
flightcheck: offline https link gate + run FlightCheck suite in CI#209daeunJe0ng wants to merge 4 commits into
daeunJe0ng wants to merge 4 commits into
Conversation
…ated link hosts before merge) FlightCheck emits portal, doc, and API URLs but nothing verifies the hosts are real and intended. A typo'd or hallucinated host (learn.microsft.com) or an http portal link sends the operator nowhere, and today that ships silently. Add a deterministic, offline test that scans FlightCheck source for every http(s) URL and asserts each static host is registered in a curated allowlist and that fetchable hosts use https. Wire it into ci.yml as a fast no-network job so it gates every PR. This is a host-level gate only; it does not verify a path is live (stale-path/404 detection is a separate networked concern that can reuse url_registry.py). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8fb19ec0-cad9-41c2-8064-2208b3844ede
…self-referential host allowlist) The prior gate required every host in FlightCheck source to be registered in a curated allowlist. That was self-referential (the author who writes a typo also edits the allowlist), high-maintenance (each new legit host, even docstring-only ones, was a red build), and it flagged legitimate hosts such as schema.management.azure.com and www.microsoft.com that are not clickable-typo risks. Replace it with one durable, fail-closed rule: every fetchable URL must be https, unless its host is a known namespace/identifier URI (SOAP/SAML namespaces) that is legitimately http. New hosts are covered by default, no registry upkeep. Rename url_registry.py -> url_hygiene_rules.py to reflect the rule-based purpose; add a structural host-shape check. Broaden CI: replace the single-file url-hygiene job with a job that runs the whole offline FlightCheck pytest suite (1003 tests), which previously ran in CI at all. Link clickability is handled by the report renderer (PR microsoft#208); live-path/redirect checking is a separate networked concern kept out of this deterministic job. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 54c58357-b127-4686-a331-df0344225da8
…(fix setuptools flat-layout discovery error) The new flightcheck-tests CI job installs the suite via `pip install -e .[test]`, which is also the command documented in pyproject.toml. On a clean runner this failed: setuptools flat-layout auto-discovery errors with "Multiple top-level packages discovered in a flat-layout: ['setup', 'samples', 'solutions']". This project is a test harness with no importable package of its own (code under test is put on sys.path via [tool.pytest.ini_options].pythonpath), so declare an explicit build backend and an empty module list. The editable install becomes a deps-only no-op, fixing the documented install command for CI and local developers alike. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 54c58357-b127-4686-a331-df0344225da8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a deterministic, offline CI gate on FlightCheck report links, and wires the previously-unrun FlightCheck test suite into CI.
FlightCheck emits URLs into the HTML report (
doc_linkbuttons and remediation deep-links). This PR adds one durable, offline-checkable guarantee about them: a URL an operator is meant to fetch useshttps. It also makes the whole FlightCheck pytest suite run on every PR, which it did not before.What this does
tests/flightcheck/url_hygiene_rules.py: the rule set. One fail-closed rule: every fetchable URL must behttps, unless its host is a known namespace / identifier URI (SOAP / SAML namespaces) that is legitimatelyhttp. A small structural host-shape helper catches malformed captures.tests/flightcheck/test_url_hygiene.py: scans FlightCheck source (stdlib only, no imports of check modules, no network) for everyhttp(s)URL and asserts each fetchable host useshttpsand looks like a real domain..github/workflows/ci.yml: replaces the old single-file job withflightcheck-tests, which installs runtime + test deps and runs the entire offline FlightCheck suite (1003 tests). None of these tests ran in CI before, so this is where the real enforcement now lives.What changed vs. the first version
The first version required every host appearing in FlightCheck source to be registered in a curated allowlist (
url_registry.py). That was dropped because:schema.management.azure.com(an ARM namespace URI) andwww.microsoft.com(the Azure service-tags download page).The fail-closed https rule keeps the one property worth enforcing offline, covers new hosts by default with no upkeep, and stops flagging legitimate hosts.
Scope and honest limits
https://learn.microsoft.com/<moved-article>that now 404s or redirects to the docs home still passes. That "stale path" problem needs a live, networked checker, intentionally kept out of this deterministic suite.Testing
python -m pytest tests/flightcheck/test_url_hygiene.py -q-> 7 passed.python -m pytest tests/flightcheck -q(the full suite the CI job runs) -> 1003 passed.ruff checkon both new files -> clean.ci.ymlparses as valid YAML.Relationship to #208
Complementary, not overlapping. #208 makes report URLs clickable; this PR makes sure a fetchable link uses https and runs the FlightCheck suite in CI.