Skip to content

flightcheck: offline https link gate + run FlightCheck suite in CI - #209

Open
daeunJe0ng wants to merge 4 commits into
microsoft:mainfrom
daeunJe0ng:dawnjeong/flightcheck-doc-link-host-gate
Open

flightcheck: offline https link gate + run FlightCheck suite in CI#209
daeunJe0ng wants to merge 4 commits into
microsoft:mainfrom
daeunJe0ng:dawnjeong/flightcheck-doc-link-host-gate

Conversation

@daeunJe0ng

@daeunJe0ng daeunJe0ng commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a deterministic, offline CI gate on FlightCheck report links, and wires the previously-unrun FlightCheck test suite into CI.

FlightCheck emits URLs into the HTML report (doc_link buttons and remediation deep-links). This PR adds one durable, offline-checkable guarantee about them: a URL an operator is meant to fetch uses https. It also makes the whole FlightCheck pytest suite run on every PR, which it did not before.

Reworked from the original host-allowlist approach. See "What changed vs. the first version" below.

What this does

  • tests/flightcheck/url_hygiene_rules.py: the rule set. One fail-closed rule: every fetchable URL must be https, unless its host is a known namespace / identifier URI (SOAP / SAML namespaces) that is legitimately http. A small structural host-shape helper catches malformed captures.
  • tests/flightcheck/test_url_hygiene.py: scans FlightCheck source (stdlib only, no imports of check modules, no network) for every http(s) URL and asserts each fetchable host uses https and looks like a real domain.
  • .github/workflows/ci.yml: replaces the old single-file job with flightcheck-tests, which installs runtime + test deps and runs the entire offline FlightCheck suite (1003 tests). None of these tests ran in CI before, so this is where the real enforcement now lives.

What changed vs. the first version

The first version required every host appearing in FlightCheck source to be registered in a curated allowlist (url_registry.py). That was dropped because:

  • Self-referential. The same author who writes a URL also edits the allowlist, so a typo copied into the allowlist passes. It only caught "typo the URL but forget the registry".
  • High-maintenance and false-positive-prone. Every new legitimate host, including hosts that only appear in docstrings and never reach a report, was a red build until hand-registered. It flagged legitimate hosts like schema.management.azure.com (an ARM namespace URI) and www.microsoft.com (the Azure service-tags download page).
  • The bug it targeted is rare and already covered by human review plus the repo's "No fabricated URLs" rule.

The fail-closed https rule keeps the one property worth enforcing offline, covers new hosts by default with no upkeep, and stops flagging legitimate hosts.

Scope and honest limits

  • This gate checks link scheme (https) and basic host shape. It is offline and deterministic, so it is safe to gate every PR on.
  • It does not make links clickable — the report renderer does that (flightcheck: autolink bare URLs in report remediation #208).
  • It does not verify a path is live. A well-formed https://learn.microsoft.com/<moved-article> that now 404s or redirects to the docs home still passes. That "stale path" problem needs a live, networked checker, intentionally kept out of this deterministic suite.

Testing

  • python -m pytest tests/flightcheck/test_url_hygiene.py -q -> 7 passed.
  • python -m pytest tests/flightcheck -q (the full suite the CI job runs) -> 1003 passed.
  • ruff check on both new files -> clean.
  • ci.yml parses as valid YAML.

Relationship to #208

Complementary, not overlapping. #208 makes report URLs clickable; this PR makes sure a fetchable link uses https and runs the FlightCheck suite in CI.

…ated link hosts before merge)

FlightCheck emits portal, doc, and API URLs but nothing verifies the hosts are real and intended. A typo'd or hallucinated host (learn.microsft.com) or an http portal link sends the operator nowhere, and today that ships silently. Add a deterministic, offline test that scans FlightCheck source for every http(s) URL and asserts each static host is registered in a curated allowlist and that fetchable hosts use https. Wire it into ci.yml as a fast no-network job so it gates every PR. This is a host-level gate only; it does not verify a path is live (stale-path/404 detection is a separate networked concern that can reuse url_registry.py).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 8fb19ec0-cad9-41c2-8064-2208b3844ede
daeunJe0ng and others added 2 commits July 28, 2026 13:54
…self-referential host allowlist)

The prior gate required every host in FlightCheck source to be registered
in a curated allowlist. That was self-referential (the author who writes a
typo also edits the allowlist), high-maintenance (each new legit host, even
docstring-only ones, was a red build), and it flagged legitimate hosts such
as schema.management.azure.com and www.microsoft.com that are not
clickable-typo risks.

Replace it with one durable, fail-closed rule: every fetchable URL must be
https, unless its host is a known namespace/identifier URI (SOAP/SAML
namespaces) that is legitimately http. New hosts are covered by default, no
registry upkeep. Rename url_registry.py -> url_hygiene_rules.py to reflect
the rule-based purpose; add a structural host-shape check.

Broaden CI: replace the single-file url-hygiene job with a job that runs the
whole offline FlightCheck pytest suite (1003 tests), which previously ran in
CI at all. Link clickability is handled by the report renderer (PR microsoft#208);
live-path/redirect checking is a separate networked concern kept out of this
deterministic job.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 54c58357-b127-4686-a331-df0344225da8
@daeunJe0ng daeunJe0ng changed the title flightcheck: offline URL host hygiene gate (catch typo'd/hallucinated link hosts) flightcheck: offline https link gate + run FlightCheck suite in CI Jul 28, 2026
…(fix setuptools flat-layout discovery error)

The new flightcheck-tests CI job installs the suite via `pip install -e .[test]`,
which is also the command documented in pyproject.toml. On a clean runner this
failed: setuptools flat-layout auto-discovery errors with "Multiple top-level
packages discovered in a flat-layout: ['setup', 'samples', 'solutions']".

This project is a test harness with no importable package of its own (code
under test is put on sys.path via [tool.pytest.ini_options].pythonpath), so
declare an explicit build backend and an empty module list. The editable
install becomes a deps-only no-op, fixing the documented install command for
CI and local developers alike.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 54c58357-b127-4686-a331-df0344225da8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant