Skip to content

test: cover the data/metrics.json render contract (tests/metrics-data.test.mjs) - #287

Open
kubestellar-hive[bot] wants to merge 1 commit into
mainfrom
quality/test-metrics-data-contract
Open

kubestellar-hive[bot] wants to merge 1 commit into
mainfrom
quality/test-metrics-data-contract

Conversation

@kubestellar-hive

Copy link
Copy Markdown
Contributor

Test Improvement

Adds one new test-only file, tests/metrics-data.test.mjs (11 tests),
asserting the real data/metrics.json against the render contract that
src/components/MetricsDashboard/index.js depends on.

Files claimed by this PR: tests/metrics-data.test.mjs (new). Nothing
else — no production code, no workflow, no package.json, no lockfile, no
dependency. It uses only node:test, node:assert/strict and node:fs.

The gap

scripts/validate-metrics.mjs is tolerant exactly where MetricsDashboard is
not, so a data/metrics.json that passes npm run validate:metrics can still
crash npm run build:

  • the validator reads data.omitted || []; the component calls
    metricsData.omitted.map(...) with no fallback
  • the validator only checks Array.isArray(chart.values); the component reads
    chart.values[0].value as the bar-width divisor
  • the validator rejects only undefined/null/'' for a metric value; the
    component calls metric.value.toLocaleString() expecting a number
  • the validator checks referenceArchitectureLifecycle.cards and .omitted
    but never inspects .trends, which Sparkline maps over

Bar width is item.value / chart.values[0].value * 100, so each breakdown must
also be sorted descending with a divisor greater than zero. A zero divisor is
finite, so it validates clean and makes every bar width NaN.

Verification

Local clone of cncf/endusers at 00b44df, node v26.8.2.

  • npm run test:unit: 66 pass, 0 fail (55 before; the 11 new tests are all
    that changed).
  • npx prettier --check tests/metrics-data.test.mjs: clean.
  • npm run validate:metrics on restored data: Validated 2 metrics.

The validator is blind to all of it. Seven mutations applied one at a time,
each passing the existing validator, each caught by this file:

mutation validate:metrics this file
delete omitted PASSES 2 fail
breakdowns.projectMaturity.values = [] PASSES 2 fail
breakdowns.projectMaturity.values[0].value = 0 PASSES 2 fail
breakdowns.projectCategories.values.reverse() PASSES 1 fail
metrics[0].value = '1,234' PASSES 1 fail
trends.submissions.values = 'x' PASSES 1 fail
delete trends.publications.values[0].date PASSES 1 fail

Four further mutations — duplicate breakdown name, reversed series, a
non-finite lifecycle card value, and an unparseable generatedAt — each fail
an assertion too, so all 11 tests are mutation-checked and none can silently
pass.

The first mutation was carried through a full build to confirm the consequence
is real rather than theoretical:

$ npm run validate:metrics
Validated 2 metrics
$ npm run build
build exit=1
  [cause]: TypeError: Cannot read properties of undefined (reading 'map')

data/metrics.json was restored byte-identical afterwards; git status shows
only the new test file.

Coverage evidence

  • Unit: npm run test:unit and node --test --experimental-test-coverage,
    local clone at 00b44df, 2026-09-18, node v26.8.2. These invariants are
    covered by no unit test. data/metrics.json is data rather than an
    executable module, so it appears in no coverage report at any percentage —
    this gap is invisible to --experimental-test-coverage rather than shown as
    a low number. The only existing test that reads the real file is the smoke
    test, which asserts just that the validator exits zero.
  • End-to-end: unavailable — this repository has no end-to-end or browser
    suite and publishes no coverage artifact from any suite (tracked in [quality] CI publishes no coverage evidence, so coverage findings cannot be verified #186).
    No claim is made that these paths lack end-to-end coverage.

Disjointness

One new file, touched by no other open PR. Distinct from #263, which covers
scripts/validate-metrics.mjs branch behaviour with synthetic fixtures and
asserts nothing about the real data/metrics.json — the mutation table above
is precisely the set of shapes #263 proves the validator accepts. Distinct from
the other data-file contract tests (#235, #239, #241, #253, #257, #260, #270),
the navigation contract (#275) and the script wiring contract (#278), none of
which read data/metrics.json. Unlike #225 and #229 it does not modify
package.json, and unlike #229/#268 it adds no JSX or hook import
infrastructure, so it cannot conflict with either.

Related Issue

Closes #286 — merging this leaves nothing for that issue to track: it asks for
exactly this file, and its three completion criteria (the file exists and runs
under npm run test:unit; every assertion mutation-checked; no production
code, workflow, package.json or dependency modified) are all met above.


Filed by quality agent (hold-gated mode). Human review required.

— hive: agent=quality backend=copilot model=claude-opus-5

scripts/validate-metrics.mjs is tolerant exactly where
src/components/MetricsDashboard/index.js is not, so a metrics.json that
passes npm run validate:metrics can still crash npm run build.

Assert the real data/metrics.json against the invariants the dashboard
depends on and the validator does not enforce: omitted present and an
array, non-empty breakdown values with a positive descending-sorted
divisor, finite numeric metric and lifecycle card values, and the
referenceArchitectureLifecycle.trends shape Sparkline consumes, which
the validator never inspects.

Test-only; no production code, workflow or dependency is changed.

Signed-off-by: kubestellar-hive[bot] <kubestellar-hive@hive.kubestellar.io>
@kubestellar-hive

Copy link
Copy Markdown
Contributor Author

Important

Held for human review by the hive's ACMM level gate.

This PR was opened by the "quality" agent while Hive policy required a human checkpoint for that agent. Non-outreach agents are held at ACMM L3–L5; the outreach agent is always held because it publishes project-facing communication.

Hive will automatically remove the hold label once current policy no longer requires a level hold for "quality". If this is an outreach PR, a human must review it and remove the label.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[quality] data/metrics.json render contract is unenforced: MetricsDashboard crashes the build on input validate-metrics accepts

0 participants