Skip to content

test(ecosystem): dogfood devtools-category modules with smoke tests #1046

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
antfu merged 2 commits into from
Jul 28, 2026
+6,006 −1,913
Merged

test(ecosystem): dogfood devtools-category modules with smoke tests #1046

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ddeea0d
test(ecosystem): dogfood devtools-category modules with smoke tests
antfubot Jul 27, 2026
67afbe9
ci(ecosystem): run the devtools smoke suite as a separate task
antfubot Jul 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 47 additions & 3 deletions .github/workflows/ecosystem-playground.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,12 @@
name: ecosystem-playground

# Manual-trigger only smoke check for playgrounds-ecosystem/modules/ — NOT
# part of the default push/pull_request CI path. See
# playgrounds-ecosystem/README.md and playgrounds-ecosystem/REPORTS.md for
# Manual-trigger only checks for playgrounds-ecosystem/modules/ — NOT part of
# the default push/pull_request CI path. Two independent tasks:
# * `smoke` — cheap "does the module combo still build" signal
# (stub devtools, `nuxt build`).
# * `devtools-smoke` — Playwright suite that drives the real embedded DevTools
# client and asserts each module's tab renders.
# See playgrounds-ecosystem/README.md and playgrounds-ecosystem/REPORTS.md for
# context (nuxt/devtools#1022).
on:
workflow_dispatch:
Expand Down Expand Up @@ -34,3 +38,43 @@ jobs:
- run: pnpm -C playgrounds-ecosystem/modules install
- name: Build the module combo
run: pnpm -C playgrounds-ecosystem/modules run build

# Separate, heavier task: actually drive the embedded Nuxt DevTools client and
# assert each ecosystem module's custom tab renders (see
# playgrounds-ecosystem/tests/). Kept apart from the `smoke` build job above
# because it needs the FULL static devtools client (`pnpm build`, not the stub
# `prepare`) plus a Playwright browser — and apart from the default e2e
# workflow because this sealed, opt-in workspace isn't in the normal CI path.
devtools-smoke:
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- uses: pnpm/action-setup@v6
- uses: actions/setup-node@v7
with:
node-version: lts/*
cache: pnpm
- run: npm install -g corepack@latest
- run: corepack enable
- run: pnpm install
# The smoke test drives the real embedded DevTools client, so — unlike the
# `smoke` job's stub `prepare` — it needs the full static client build.
- run: pnpm build
- name: Cache Playwright browsers
uses: actions/cache@v5
with:
path: ~/.cache/ms-playwright
key: playwright-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml') }}
- run: pnpm exec playwright install --with-deps chromium
- run: pnpm -C playgrounds-ecosystem/modules install
- run: pnpm run test:e2e:ecosystem
- uses: actions/upload-artifact@v7
if: failure()
with:
name: ecosystem-playwright-report
path: playwright-report/
retention-days: 7
if-no-files-found: ignore
1 change: 1 addition & 0 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
"test:e2e:built": "pnpm test:e2e:prebuild && PW_PROJECT='*:built' playwright test --config tests/e2e/playwright.config.ts",
"test:e2e:prebuild": "pnpm -C playgrounds/empty exec nuxt build && pnpm -C playgrounds/spa exec nuxt build && pnpm -C playgrounds/tab-pinia exec nuxt build && pnpm -C playgrounds/tab-seo exec nuxt build",
"test:e2e:ui": "playwright test --config tests/e2e/playwright.config.ts --ui",
"test:e2e:ecosystem": "playwright test --config playgrounds-ecosystem/tests/playwright.config.ts",
"docs": "pnpm -C docs install && nuxi dev docs",
"docs:build": "pnpm -C docs install && CI=true nuxi generate docs",
"typecheck": "vue-tsc --noEmit",
Expand Down
123 changes: 87 additions & 36 deletions playgrounds-ecosystem/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,28 +1,46 @@
# Ecosystem dogfooding playground

Dogfoods the Nuxt DevTools integrations of three popular Nuxt modules — all in
one combined app — against the **local** `@nuxt/devtools` (this repo's
Dogfoods the Nuxt DevTools integrations of popular Nuxt modules — all in one
combined app — against the **local** `@nuxt/devtools` (this repo's
`packages/devtools`) for integration verification.

Modules covered, in [`modules/`](./modules/): `nuxt-og-image`, `@nuxt/scripts`,
`@nuxt/fonts`. See [`REPORTS.md`](./REPORTS.md) for what was actually found
running each of them — including `@nuxt/content` and `@nuxt/image`, which
were tried and then removed: neither registers a DevTools tab in the versions
tested, so there was nothing to dogfood against.

> Why one combined playground instead of one per module, and why only three of
> the originally-considered seven modules? Grouping keeps the review surface
> small (this repo's own `docs/` app already proves `@nuxt/content` +
> `@nuxt/fonts` + `@nuxt/image` + `nuxt-og-image` coexist safely). Two were
> dropped before implementation: `@nuxthub/core` (out of scope) and
> `@nuxtjs/tailwindcss` (a real version conflict — it hard-depends on
> Tailwind v3, while `nuxt-og-image` lists Tailwind v4 as an optional peer).
> Two more, `@nuxt/content` and `@nuxt/image`, were built in and dogfooded
> first, then dropped once that run showed neither has a DevTools tab to test
> — see [`REPORTS.md`](./REPORTS.md#modules-removed-after-testing) for the
> evidence. The former standalone "plan 04" doc this all comes from has since
> been retired in favor of this directory being the living implementation —
> see nuxt/devtools#1022 for the full history.
Modules covered, in [`modules/`](./modules/), in two groups:

- **Original trio** (kept from the first dogfooding pass; not in nuxt.com's
"Devtools" category): `nuxt-og-image`, `@nuxt/scripts`, `@nuxt/fonts`.
- **Devtools-category set**, drawn from
<https://nuxt.com/modules?category=Devtools> and curated down to the ones
that actually register a DevTools tab: `@nuxt/eslint`, `@nuxt/hints`,
`@nuxt/a11y`, `@compodium/nuxt`, `@scalar/nuxt`.

Each module has a small fixture in `modules/` so its DevTools surface has real
data to show: `eslint.config.mjs` (ESLint config inspector),
`components/DemoButton.vue` (Compodium's component playground),
`server/api/widgets.get.ts` + `nitro.experimental.openAPI` (Scalar's API
reference), and a deliberate no-`alt` `<img>` in `pages/index.vue` (an axe
finding for `@nuxt/a11y`). See [`REPORTS.md`](./REPORTS.md) for what was
actually found running each of them.

> **Which "Devtools" category modules were left out, and why?** The category
> lists ~18 modules; most don't register a DevTools tab, so there's nothing to
> dogfood against. Skipped as no-surface: `@nuxt/test-utils`,
> `nuxt-typed-router`, `nuxt-prepare`, `nuxt-safe-runtime-config`,
> `nuxt-ssr-api-logger`, `nuxt-email-renderer`. Skipped as secret-gated (need
> auth tokens to do anything): `nuxt-doppler`, `@nuxtjs/ngrok`. Skipped as
> heavy or deprecated: `@nuxtjs/storybook`, `workflow`, `sonda`,
> `@nuxtjs/eslint-module` (superseded by `@nuxt/eslint`).

> Why one combined playground instead of one per module? Grouping keeps the
> review surface small (this repo's own `docs/` app already proves
> `@nuxt/content` + `@nuxt/fonts` + `@nuxt/image` + `nuxt-og-image` coexist
> safely). `@nuxt/content` and `@nuxt/image` were built in and dogfooded in the
> first pass, then dropped once that run showed neither has a DevTools tab to
> test — see [`REPORTS.md`](./REPORTS.md#modules-removed-after-testing) for the
> evidence. `@nuxthub/core` (out of scope) and `@nuxtjs/tailwindcss` (a real
> Tailwind v3-vs-v4 version conflict with `nuxt-og-image`) were dropped before
> implementation. The former standalone "plan 04" doc this all comes from has
> since been retired in favor of this directory being the living
> implementation — see nuxt/devtools#1022 for the full history.

## Opt-in — not part of the main install or CI

Expand Down Expand Up @@ -80,11 +98,12 @@ Click the Nuxt-logo entry to open the embedded Nuxt DevTools client. From
there:

- **Overview → `N modules`** lists every installed module with its setup
time — confirms all three ecosystem modules loaded.
time — confirms all ecosystem modules loaded.
- Each module's own tab is **not** in the SideNav's visible icon strip — check
the **"⋯" overflow menu** at the bottom of the SideNav. As of this report,
that's where `nuxt-og-image`, `@nuxt/scripts`, and `@nuxt/fonts`'s tabs
live.
the **"⋯" overflow menu** at the bottom of the SideNav. That's where the
module tabs live: `custom-nuxt-seo-og-image`, `custom-nuxt-scripts`,
`custom-fonts`, `custom-eslint-config`, `custom-hints`, `custom-nuxt-a11y`,
`custom-compodium`, and `custom-scalar`.

Don't navigate directly to `http://localhost:3000/__nuxt_devtools__/client/`
in a plain tab expecting the same result — that bypasses the RPC handshake the
Expand All @@ -97,14 +116,46 @@ appear, does it load without console errors, does it show any Plan 00
deprecation diagnostics, and an overall verdict. That report is the raw
material for upstream issues/PRs to each module.

## Automated smoke check (optional, manual trigger only)

`.github/workflows/ecosystem-playground.yml` is `workflow_dispatch`-only — it
installs the root workspace, stubs `packages/devtools` (`pnpm run prepare`),
installs this workspace, and runs `nuxt build` as a cheap "did the module
combo break" signal. It deliberately doesn't run the full `pnpm build` or set
`NUXT_DEVTOOLS_LOCAL` — DevTools no-ops outside `dev` mode, so build-mode
can't exercise anything devtools-specific anyway, and the cheap stub is
enough for the module to resolve. It is **not** part of the default CI path;
trigger it manually from the Actions tab when you want a sanity check without
dogfooding by hand.
## Automated checks (optional, manual trigger only)

`.github/workflows/ecosystem-playground.yml` is `workflow_dispatch`-only and
holds two independent tasks (jobs). Neither is part of the default push /
pull_request CI path; trigger them from the Actions tab.

- **`smoke`** — installs the root workspace, stubs `packages/devtools`
(`pnpm run prepare`), installs this workspace, and runs `nuxt build` as a
cheap "did the module combo break" signal. It deliberately doesn't run the
full `pnpm build` or set `NUXT_DEVTOOLS_LOCAL` — DevTools no-ops outside
`dev` mode, so build-mode can't exercise anything devtools-specific anyway,
and the cheap stub is enough for the modules to resolve.
- **`devtools-smoke`** — the heavier task that actually drives the embedded
DevTools client: it runs the full `pnpm build` (real static client),
installs Playwright's Chromium, then runs the smoke suite
(`pnpm run test:e2e:ecosystem`, see [Playwright smoke tests](#playwright-smoke-tests-opt-in)).
On failure it uploads the Playwright HTML report as an artifact.

## Playwright smoke tests (opt-in)

[`tests/`](./tests/) holds a Playwright suite that boots the combined app and
asserts each Devtools-category module registers its custom tab and that tab
renders against this repo's built devtools client — so "do all the modules'
DevTools still function?" is a one-command check instead of a manual pass.

Unlike the hand-dogfooding runbook above (which uses `NUXT_DEVTOOLS_LOCAL`),
this suite runs against the **built** `@nuxt/devtools` static client — the same
way the repo's main `tests/e2e` suite and CI do — so you must build first.
It's kept **out** of the default `pnpm test:e2e` because it depends on this
sealed, opt-in workspace being installed. Run it explicitly from the repo root:

```sh
pnpm install # repo root
pnpm run build # real static devtools client
pnpm -C playgrounds-ecosystem/modules install # this sealed workspace
pnpm run test:e2e:ecosystem # boots the app + drives devtools
```

The suite spawns its own dev server (on port 13200 by default; override with
`PW_ECOSYSTEM_PORT`) with `VITE_DEVTOOLS_DISABLE_CLIENT_AUTH=true`, so no manual
authorization step is needed. See
[`tests/ecosystem-modules.spec.ts`](./tests/ecosystem-modules.spec.ts) for the
per-module tab assertions.
95 changes: 94 additions & 1 deletion playgrounds-ecosystem/REPORTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,16 +67,57 @@ rebased tip:
all three tabs are still reachable (still only via the SideNav overflow
menu), and `@nuxt/scripts` still shows "No scripts loaded".

**Addendum 3 — Devtools-category modules added:** the playground was expanded

@coderabbitai coderabbitai Bot Jul 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Make “Addendum 3” a real Markdown heading.

The text at Line 70 is bold paragraph text, so it does not generate the #addendum-3--devtools-category-modules-added anchor used at Lines 115, 120, 227, and 322. Change it to a heading such as ## Addendum 3 — ..., or update all references to an existing anchor.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@playgrounds-ecosystem/REPORTS.md` at line 70, Convert the “Addendum 3 —
Devtools-category modules added” text in REPORTS.md from bold paragraph
formatting to a Markdown heading, preferably using the same heading hierarchy as
the surrounding addenda, so it generates the existing
`#addendum-3--devtools-category-modules-added` anchor referenced elsewhere.

Source: Linters/SAST tools

beyond the original trio to cover modules from
<https://nuxt.com/modules?category=Devtools>. That category lists ~18 modules;
most register no DevTools tab, so — per the same methodology that dropped
`@nuxt/content`/`@nuxt/image` — only the five that expose a real tab were kept:
`@nuxt/eslint`, `@nuxt/hints`, `@nuxt/a11y`, `@compodium/nuxt`, `@scalar/nuxt`
(see [`README.md`](./README.md) for the full skip-list and rationale). Verified
by booting the combined app against this repo's **built** `@nuxt/devtools` and
driving the devtools client with Playwright (the new opt-in suite in
[`tests/`](./tests/) — `pnpm run test:e2e:ecosystem`). Findings:

- **All five register their custom tab and it renders** — `custom-eslint-config`
(launch), `custom-hints`, `custom-nuxt-a11y`, `custom-compodium`, and
`custom-scalar` (all iframe). Per-module notes in
[Per-module findings](#per-module-findings) below.
- **Environment: the sealed workspace had to be aligned to Vite 8.** Nuxt's
default (`nuxt@4.4.8` → Vite 7.3.6) made the linked `@nuxt/devtools`'s
`@vitejs/devtools@0.4.9` **crash the dev server at startup** — it eagerly
fails to resolve `@vitejs/devtools-rolldown`. Bumping the workspace to
`nuxt@^4.5.0` and adding a `vite: ~8.0.16` override (mirroring the root
`pnpm-workspace.yaml`, which pins Vite 8 for exactly this reason) fixed it.
This is a packaging/version-alignment issue, not a fault in any of the five
modules — but worth noting that `@vitejs/devtools@0.4.9` hard-requires Vite 8.
- **`@scalar/nuxt`'s `/docs` page (its tab's iframe target) 500s under SSR** on
this Nuxt 4.5 / Vite 8 stack: *"Cannot destructure property 'mod' of
'threads.workerData' as it is undefined."* The API reference is a
client-rendered app, so `routeRules: { '/docs/**': { ssr: false } }`
sidesteps the crash with no loss of functionality. Good candidate for an
upstream issue to `@scalar/nuxt` about SSR under Vite 8.
- **Console noise** is the same pre-existing/environmental set as before
(`DTK0008` auth-disabled warning in the e2e config, the OG-image
auto-signing-secret warning, and the `NDT_DEP_0003` deprecation diagnostics —
now also fired by more of these modules' `extendServerRpc`/legacy kit usage).

## Summary

| Module | Version | Devtools surface? | Verdict |
|---|---|---|---|
| `nuxt-og-image` | 6.7.2 | Custom tab (`custom-nuxt-seo-og-image`) | Works — lazy-installs a companion panel |
| `@nuxt/scripts` | 1.3.1 | Custom tab (`custom-nuxt-scripts`) | Works |
| `@nuxt/fonts` | 0.14.0 | Custom tab (`custom-fonts`) | Works |
| `@nuxt/eslint` | 1.16.0 | Custom tab (`custom-eslint-config`, launch) | Works — launches the ESLint config inspector on demand |
| `@nuxt/hints` | 1.1.4 | Custom tab (`custom-hints`, iframe) | Works |
| `@nuxt/a11y` | 1.0.0-alpha.1 | Custom tab (`custom-nuxt-a11y`, iframe) | Works |
| `@compodium/nuxt` | 0.1.0-beta.13 | Custom tab (`custom-compodium`, iframe) | Works |
| `@scalar/nuxt` | 0.6.59 | Custom tab (`custom-scalar`, iframe) | Works — but its `/docs` page needs `ssr: false` on this stack (see [Addendum 3](#addendum-3--devtools-category-modules-added)) |

No console errors were caused by any of the three modules themselves. The only
No console errors were caused by the first three modules themselves. The only
console noise was pre-existing / environmental (see [Other observations](#other-observations)).
The five Devtools-category modules were added and verified later — see
[Addendum 3](#addendum-3--devtools-category-modules-added).

### A UX trap during verification, worth flagging on its own

Expand Down Expand Up @@ -144,6 +185,49 @@ cross-referencing this report when implementing Plan 03.
the panel renders a live "Aa" preview of it.
- No console errors.

### `@nuxt/eslint` 1.16.0 — works (lazy launcher)

- Registers a custom tab `eslint-config` ("ESLint Config"), reachable via the
SideNav overflow menu (route `/modules/custom-eslint-config`).
- Its `config.devtools.enabled` defaults to `"lazy"`, so the tab renders a
**launch panel** ("Start ESLint config inspector to analyze the local ESLint
configs") rather than eagerly booting `@eslint/config-inspector`. Clicking
Launch starts the inspector and swaps the tab to an iframe. The fixture
`eslint.config.mjs` re-exports the generated `.nuxt/eslint.config.mjs` so the
inspector has a real config to introspect.

### `@nuxt/hints` 1.1.4 — works

- Registers a custom tab `hints` ("Hints", `category: analyze`, iframe →
`/__nuxt-hints`). Embeds its performance/security/hydration hints UI; no
per-page fixture needed since it inspects the app itself.

### `@nuxt/a11y` 1.0.0-alpha.1 — works

- Registers a custom tab `nuxt-a11y` ("Nuxt a11y", iframe →
`/__nuxt-a11y-client`) with its real-time axe-core accessibility panel.
- `pages/index.vue` carries a deliberate no-`alt` `<img>` (a classic
`image-alt` violation) so the panel has a real finding to surface. Alpha
release; worked without incident in this run.

### `@compodium/nuxt` 0.1.0-beta.13 — works

- Registers a custom tab `compodium` ("Compodium", iframe →
`/__compodium__/devtools`) — a component playground. Scans the app's
`components/` dir, so the fixture `components/DemoButton.vue` gives it a
prop-driven component to preview. Beta release; worked without incident.

### `@scalar/nuxt` 0.6.59 — works, with an SSR caveat

- Registers a custom tab `scalar` ("Scalar", `category: server`, iframe →
`/docs`) rendering its API reference from Nitro's OpenAPI document
(`nitro.experimental.openAPI: true` + the `defineRouteMeta({ openAPI })` on
`server/api/widgets.get.ts`; the spec is served at `/_openapi.json`).
- **Caveat:** `/docs` throws a 500 under SSR on the Nuxt 4.5 / Vite 8 stack
(see [Addendum 3](#addendum-3--devtools-category-modules-added)); the fixture
sets `routeRules: { '/docs/**': { ssr: false } }` to render it client-side
instead. Flagged as an upstream candidate.

## Modules removed after testing

Both of these were part of the playground during testing (see the git history
Expand Down Expand Up @@ -232,3 +316,12 @@ playground anymore.
6. Re-run after Plan 01 (Messages) lands to see whether `@nuxt/scripts`'s
"No scripts loaded" / "Not connected to the client app" gap resolves once
more of the client bridge is exercised.
7. File an upstream issue to `@scalar/nuxt` about its `/docs` page 500ing under
SSR on Vite 8 (*"Cannot destructure property 'mod' of 'threads.workerData'"*);
the playground works around it with `routeRules: { '/docs/**': { ssr: false } }`
(see [Addendum 3](#addendum-3--devtools-category-modules-added)).
8. `@vitejs/devtools@0.4.9` (pulled in by the linked `@nuxt/devtools`)
hard-requires Vite 8 — on Vite 7 it crashes the dev server trying to resolve
`@vitejs/devtools-rolldown`. Not a bug in any ecosystem module, but a sharp
edge for anyone consuming this devtools alpha on a Vite-7 app; worth a
clearer error or a documented peer requirement.
41 changes: 41 additions & 0 deletions playgrounds-ecosystem/modules/components/DemoButton.vue
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
<script setup lang="ts">
// @compodium/nuxt — a component playground. Compodium scans the app's
// `components/` directory, so this single component is enough to give its
// DevTools "Compodium" tab a real, prop-driven component to preview and
// tweak in isolation.
withDefaults(defineProps<{
/** Text shown inside the button. */
label?: string
/** Visual emphasis. */
variant?: 'primary' | 'secondary'
}>(), {
label: 'Click me',
variant: 'primary',
})
</script>

<template>
<button type="button" :class="['demo-button', `demo-button--${variant}`]">
{{ label }}
</button>
</template>

<style scoped>
.demo-button {
padding: 0.5rem 1rem;
border: none;
border-radius: 0.375rem;
font: inherit;
cursor: pointer;
}

.demo-button--primary {
background: #00dc82;
color: #0b1e13;
}

.demo-button--secondary {
background: #1a1a2e;
color: #fff;
}
</style>
Loading
Loading