Skip to content

test(boost): validate AI Catalog performance at scale - #4927

Merged
gabemontero merged 1 commit into
redhat-developer:mainfrom
rohitkrai03:codex/rhidp-14581-ai-catalog-performance
Sep 22, 2026
Merged

gabemontero merged 1 commit into
redhat-developer:mainfrom
rohitkrai03:codex/rhidp-14581-ai-catalog-performance

Conversation

@rohitkrai03

Copy link
Copy Markdown
Contributor

Summary

  • Add an opt-in Playwright performance benchmark with 500 synthetic AI asset entities.
  • Measure initial grid rendering and client-side search/filter response.
  • Document the baseline results and emitted frontend bundle size.

Results

  • Initial rendering of the default 20-card page: 1,636 ms (target <2,000 ms).
  • Search/filter response: 385 ms (target <500 ms).
  • Emitted JavaScript: 116,977 bytes raw / 23,098 bytes gzipped together.

Validation

  • yarn test:e2e:performance
  • yarn workspace @red-hat-developer-hub/backstage-plugin-ai-catalog build
  • yarn workspace @red-hat-developer-hub/backstage-plugin-ai-catalog test --runInBand
  • yarn tsc
  • yarn lint

Relates to RHIDP-14581.

@rhdh-qodo-merge

Copy link
Copy Markdown

PR Summary by Qodo

Benchmark AI Catalog performance with 500 entities

🧪 Tests 📝 Documentation ⚙️ Configuration changes 🕐 10-20 Minutes

Grey Divider

AI Description

• Adds an opt-in Playwright benchmark using 500 synthetic AI Catalog entities.
• Enforces initial-load and client-side filter response thresholds.
• Records measured latency and JavaScript bundle-size baselines.
Diagram

graph TD
  A["Performance Script"] --> B["Synthetic Entities"] --> C["Catalog API Mock"] --> D["AI Catalog Page"]
  D --> E["Twenty Card Grid"] --> G["Threshold Checks"] --> H["JSON Artifact"]
  D --> F["Client Search"] --> G
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Browser-native performance instrumentation
  • ➕ Measures timing directly within the browser execution context
  • ➕ Can isolate rendering and interaction phases more precisely
  • ➖ Requires application instrumentation or additional page evaluation logic
  • ➖ May exclude navigation and guest sign-in overhead captured by the benchmark
2. Real backend load benchmark
  • ➕ Measures full frontend-to-catalog latency under realistic service conditions
  • ➕ Can expose backend scaling and network bottlenecks
  • ➖ Introduces infrastructure and dataset variability
  • ➖ Does not isolate the client-side rendering and filtering behavior targeted here

Recommendation: Keep the opt-in, API-mocked Playwright benchmark because it provides a repeatable frontend regression baseline without destabilizing normal E2E runs. Browser-native marks could be added later if finer rendering attribution becomes necessary; backend load testing should remain a separate benchmark with different targets.

Files changed (3) +187 / -0

Tests (1) +155 / -0
boost.AiCatalogPerformance.test.tsAdd a 500-entity AI Catalog performance benchmark +155/-0

Add a 500-entity AI Catalog performance benchmark

• Adds an opt-in Playwright test that mocks 500 representative AI assets and measures initial 20-card rendering plus client-side search response. It enforces 2,000 ms and 500 ms thresholds and attaches structured JSON results.

workspaces/boost/e2e-tests/boost.AiCatalogPerformance.test.ts

Documentation (1) +31 / -0
ai-catalog.mdDocument the AI Catalog performance baseline +31/-0

Document the AI Catalog performance baseline

• Records the benchmark environment, workload, passing latency measurements, and emitted JavaScript sizes. It also documents reproduction commands and why timing checks remain opt-in.

workspaces/boost/performance/ai-catalog.md

Other (1) +1 / -0
package.jsonExpose an opt-in performance test command +1/-0

Expose an opt-in performance test command

• Adds 'test:e2e:performance', which enables the benchmark environment flag and runs only the English AI Catalog performance test.

workspaces/boost/package.json

@sonarqubecloud

Copy link
Copy Markdown

@rhdh-qodo-merge

rhdh-qodo-merge Bot commented Sep 22, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (4) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

Grey Divider


Remediation recommended

1. English copy changes break the benchmark 📘 Rule violation ⚙ Maintainability
Description
signInAsGuest and the benchmark assertions select controls and results through literal labels such
as Enter, AI Catalog, Search, and All (500) instead of the existing getTranslations
utility. Any update to the English message catalog can make these selectors time out even when
rendering and filtering still work correctly.
Code

workspaces/boost/e2e-tests/boost.AiCatalogPerformance.test.ts[R93-94]

+  const enter = page.getByRole('button', { name: 'Enter' });
+  const heading = page.getByRole('heading', { name: 'AI Catalog' });
Relevance

●●● Strong

Recent translation-suite feedback confirms hardcoded locale-sensitive selectors are accepted for
correction.

PR-#4841

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Rule 2596 prohibits hardcoded user-facing strings in E2E selectors and assertions. The new test
directly embeds several English labels despite an existing translation-loading utility for the AI
Catalog tests.

Rule 2596: E2E tests must use translation keys instead of hardcoded UI strings
workspaces/boost/e2e-tests/boost.AiCatalogPerformance.test.ts[90-131]
workspaces/boost/e2e-tests/utils/translations.ts[46-62]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The performance test uses hardcoded user-facing English strings in selectors and assertions instead of messages loaded from the plugin translation modules.

## Fix Focus Areas
- workspaces/boost/e2e-tests/boost.AiCatalogPerformance.test.ts[90-131]
- workspaces/boost/e2e-tests/utils/translations.ts[46-62]

## Recommended Fix
Import `getTranslations`, load the English AI Catalog messages during test setup, and replace literal UI labels in role and text queries with the corresponding translated values.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. Asset fixtures use unprefixed types 📘 Rule violation ⚙ Maintainability
Description
entityKinds defines canonical fixture types as skill, agent, rule, mcp-server, and
ai-tool, none of which starts with boost-. Every entity produced by createPerformanceEntities
copies one of those values into spec.type, so all 500 generated resources exercise identifiers
outside the required namespace.
Code

workspaces/boost/e2e-tests/boost.AiCatalogPerformance.test.ts[R25-28]

+  { kind: 'AiResource', type: 'skill' },
+  { kind: 'AiResource', type: 'agent' },
+  { kind: 'AiResource', type: 'rule' },
+  { kind: 'API', type: 'mcp-server' },
Relevance

●●● Strong

Fixture types violate the explicitly cited canonical boost- prefix rule.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Rule 2417 requires canonical resource type identifiers in changed files to begin with boost-. The
newly added fixture declares five unprefixed values and assigns them to generated entity
specifications.

Rule 2417: Prefix resource type identifiers with boost-
workspaces/boost/e2e-tests/boost.AiCatalogPerformance.test.ts[24-30]
workspaces/boost/e2e-tests/boost.AiCatalogPerformance.test.ts[52-56]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The synthetic catalog entities use resource type identifiers that do not carry the required `boost-` prefix.

## Fix Focus Areas
- workspaces/boost/e2e-tests/boost.AiCatalogPerformance.test.ts[24-30]

## Recommended Fix
Rename each canonical type value in `entityKinds` to begin with `boost-`, and update any fixture expectations needed to keep the benchmark representative.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. Accessibility regressions go unseen 📘 Rule violation ☑ Accessibility
Description
The new performance test completes its populated catalog flow without invoking the existing
runAccessibilityTests helper or otherwise running an axe-core scan. Accessibility defects exposed
only by the 500-asset grid and filtered result state therefore never produce WCAG-tagged results or
a failure attachment.
Code

workspaces/boost/e2e-tests/boost.AiCatalogPerformance.test.ts[R103-105]

+  test('measures 500-entity initial load and client-side filtering', async ({
+    page,
+  }, testInfo) => {
Relevance

●●● Strong

Recent repository precedent accepted adding runAccessibilityTests to E2E coverage.

PR-#4841

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Rule 2654 requires E2E accessibility checks using axe-core, WCAG tags, TestInfo attachments, and
false-positive filtering. The benchmark has no accessibility invocation, while the existing shared
helper already implements all required mechanics.

Rule 2654: E2E accessibility checks must use axe-core with WCAG tags and attach results to TestInfo
workspaces/boost/e2e-tests/boost.AiCatalogPerformance.test.ts[103-153]
workspaces/boost/e2e-tests/utils/accessibility.ts[30-63]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new E2E benchmark does not run the repository's axe-core accessibility helper on its populated AI Catalog state.

## Fix Focus Areas
- workspaces/boost/e2e-tests/boost.AiCatalogPerformance.test.ts[103-153]
- workspaces/boost/e2e-tests/utils/accessibility.ts[30-63]

## Recommended Fix
Import `runAccessibilityTests` and invoke it after the populated or filtered grid stabilizes, passing `page` and `testInfo` so WCAG-tagged results are attached and real violations fail the test.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


View medium (2)
4. Retries hide performance regressions 🐞 Bug ≡ Correctness
Description
The timing thresholds are ordinary Playwright assertions, so the workspace configuration retries
either failure twice when CI is set. A benchmark that exceeds a target on its first attempt can
pass on a later attempt, allowing an unstable or regressed result to produce a successful CI exit.
Code

workspaces/boost/e2e-tests/boost.AiCatalogPerformance.test.ts[R152-153]

+    expect(initialLoadMs).toBeLessThan(LOAD_TARGET_MS);
+    expect(filterResponseMs).toBeLessThan(FILTER_TARGET_MS);
Relevance

●● Moderate

Retry interaction is plausible, but no closely matching accepted or rejected precedent was found.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The new test enforces both performance targets using normal assertions, while the shared Playwright
configuration retries failed tests twice whenever CI is set. Playwright can therefore report the
benchmark as successful after a faster retry rather than requiring every measured run to satisfy the
targets.

workspaces/boost/e2e-tests/boost.AiCatalogPerformance.test.ts[134-153]
workspaces/boost/playwright.config.ts[23-25]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The performance benchmark inherits Playwright's two CI retries, allowing failed timing thresholds to pass on a later attempt.

## Fix Focus Areas
- workspaces/boost/e2e-tests/boost.AiCatalogPerformance.test.ts[102-153]
- workspaces/boost/playwright.config.ts[23-25]

## Recommended Fix
Configure the performance test suite with zero retries, for example using `test.describe.configure({ retries: 0 })`, so every threshold violation fails the benchmark run.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


5. Guest sign-in steps can drift 📘 Rule violation ⚙ Maintainability
Description
signInAsGuest is added inline even though the same navigation and guest-entry helper already
exists in boost.AiCatalogPage.test.ts. Future sign-in flow changes must therefore update both test
files, and inconsistent copies can make only one AI Catalog suite fail.
Code

workspaces/boost/e2e-tests/boost.AiCatalogPerformance.test.ts[R90-93]

+async function signInAsGuest(page: Page) {
+  page.on('dialog', dialog => dialog.accept());
+  await page.goto('/ai-catalog');
+  const enter = page.getByRole('button', { name: 'Enter' });
Relevance

●● Moderate

Recent E2E history accepts helper improvements, but lacks direct centralization precedent.

PR-#4841

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Rule 2622 requires reusable E2E logic to reside under the E2E utils/ directory. The new helper
duplicates the existing AI Catalog helper rather than centralizing it there.

Rule 2622: Reusable e2e test logic must be placed in a utils/ directory under the e2e test directory
workspaces/boost/e2e-tests/boost.AiCatalogPerformance.test.ts[90-100]
workspaces/boost/e2e-tests/boost.AiCatalogPage.test.ts[109-119]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new performance test duplicates reusable AI Catalog guest sign-in logic already defined in another E2E test file.

## Fix Focus Areas
- workspaces/boost/e2e-tests/boost.AiCatalogPerformance.test.ts[90-100]
- workspaces/boost/e2e-tests/boost.AiCatalogPage.test.ts[109-119]

## Recommended Fix
Move the shared guest sign-in routine into a module under `workspaces/boost/e2e-tests/utils/`, then import and use that helper from both test files.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context sources
✅ Compliance rules (platform): 37 rules
✅ Cross-repo context — repo relationships
Review mode: ⚖️ Balanced: This adds a new opt-in Playwright benchmark and package script with behavioral test logic, API interception, authentication flow, timing assertions, and documentation, warranting a careful single-pass review.

Grey Divider

Tip of the day
💡 Did you know, you can route each action level your way: inline, summary, both, or drop

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

@rhdh-qodo-merge rhdh-qodo-merge Bot added documentation Improvements or additions to documentation Tests labels Sep 22, 2026
@rhdh-qodo-merge

Copy link
Copy Markdown

Important

The /generate_labels command by Qodo is sunsetting on the 1st of October 2026 and will no longer be available. We recommend switching to the latest Qodo review capabilities. Learn more

@codecov

codecov Bot commented Sep 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.30%. Comparing base (495a084) to head (a4f2018).
⚠️ Report is 4 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4927   +/-   ##
=======================================
  Coverage   62.30%   62.30%           
=======================================
  Files        2699     2699           
  Lines      106428   106428           
  Branches    29984    29984           
=======================================
  Hits        66308    66308           
  Misses      39548    39548           
  Partials      572      572           
Flag Coverage Δ *Carryforward flag
adoption-insights 84.77% <ø> (ø) Carriedforward from 495a084
ai-integrations 87.10% <ø> (ø) Carriedforward from 495a084
app-defaults 67.23% <ø> (ø) Carriedforward from 495a084
augment 46.67% <ø> (ø) Carriedforward from 495a084
boost 92.92% <ø> (ø)
bulk-import 73.12% <ø> (ø) Carriedforward from 495a084
cost-management 13.53% <ø> (ø) Carriedforward from 495a084
dcm 73.47% <ø> (ø) Carriedforward from 495a084
e2e-adoption-insights 60.00% <ø> (ø) Carriedforward from 495a084
e2e-extensions 62.31% <ø> (ø) Carriedforward from 495a084
e2e-global-header 52.40% <ø> (ø) Carriedforward from 495a084
e2e-homepage 61.11% <ø> (ø) Carriedforward from 495a084
e2e-intelligent-assistant 45.57% <ø> (ø) Carriedforward from 495a084
e2e-orchestrator 49.49% <ø> (ø) Carriedforward from 495a084
e2e-orchestrator-plugin 49.48% <ø> (ø) Carriedforward from 495a084
e2e-quickstart 55.21% <ø> (ø) Carriedforward from 495a084
e2e-scorecard 49.77% <ø> (ø) Carriedforward from 495a084
e2e-theme 16.36% <ø> (ø) Carriedforward from 495a084
extensions 58.30% <ø> (ø) Carriedforward from 495a084
global-floating-action-button 71.18% <ø> (ø) Carriedforward from 495a084
global-header 69.10% <ø> (ø) Carriedforward from 495a084
homepage 55.05% <ø> (ø) Carriedforward from 495a084
install-dynamic-plugins 73.52% <ø> (ø) Carriedforward from 495a084
intelligent-assistant 78.72% <ø> (ø) Carriedforward from 495a084
konflux 91.98% <ø> (ø) Carriedforward from 495a084
lightspeed 69.02% <ø> (ø) Carriedforward from 495a084
mcp-integrations 84.46% <ø> (ø) Carriedforward from 495a084
orchestrator 77.69% <ø> (ø) Carriedforward from 495a084
quickstart 63.74% <ø> (ø) Carriedforward from 495a084
sandbox 79.56% <ø> (ø) Carriedforward from 495a084
scorecard 88.99% <ø> (ø) Carriedforward from 495a084
theme 87.94% <ø> (ø) Carriedforward from 495a084
translations 5.12% <ø> (ø) Carriedforward from 495a084
x2a 53.78% <ø> (ø) Carriedforward from 495a084

*This pull request uses carry forward flags. Click here to find out more.


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 495a084...a4f2018. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@gabemontero

Copy link
Copy Markdown
Contributor

/fs-review

@fullsend-ai-review

fullsend-ai-review Bot commented Sep 22, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 1:11 PM UTC · Completed 1:29 PM UTC

Commit: a4f2018 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $4.17

@fullsend-ai-review fullsend-ai-review Bot added the risk/moderate PR risk: moderate label Sep 22, 2026
@fullsend-ai-review

Copy link
Copy Markdown

Risk Assessment: moderate (2/5)

Details

Small PR adding an opt-in performance benchmark test, documentation, and a single package.json bump with no security-sensitive paths, low churn, and decent test ratio yields a moderate composite risk score of 2.

@fullsend-ai-review

Copy link
Copy Markdown

Review

Findings

Medium

  • [code organization] workspaces/boost/e2e-tests/boost.AiCatalogPerformance.test.ts:61isCatalogEntitiesPath, mockCatalogEntities, and signInAsGuest are verbatim copies of functions already defined in boost.AiCatalogPage.test.ts (and the first two also in boost.translations.test.ts). The codebase has an established convention of extracting shared test helpers into e2e-tests/utils/ — the utils/ directory already contains accessibility.ts, localeSkip.ts, and translations.ts for exactly this purpose. This PR adds a third copy of isCatalogEntitiesPath and mockCatalogEntities, and a second copy of signInAsGuest.
    Remediation: Move isCatalogEntitiesPath, mockCatalogEntities, and signInAsGuest into e2e-tests/utils/catalogMocks.ts (or an equivalent utils module), export them, and import them in all three test files.

Low

  • [code organization] workspaces/boost/e2e-tests/boost.AiCatalogPerformance.test.ts:145console.log is used to emit performance results to stdout. Neither boost.AiCatalogPage.test.ts nor boost.translations.test.ts uses console.log; structured output is produced via testInfo.attach() instead. The console.log call is redundant given that the same data is attached as an artifact on line 147.
    Remediation: Remove the console.log call. The testInfo.attach() already provides structured, persisted output that is the established pattern.

  • [intent-implementation-mismatch] workspaces/boost/e2e-tests/boost.AiCatalogPerformance.test.ts:152 — The PR body describes the purpose as measuring rendering and filter response, while the documentation file’s Target/Status columns make clear that targets are enforced. The code uses hard-failing assertions (expect(...).toBeLessThan(...)), which is consistent with the documentation but not with the PR body’s "measure" framing.
    Remediation: Update the PR description to clarify this is a performance gate with hard thresholds, not just a measurement tool.

  • [missing-doc] workspaces/boost/README.md:70 — The Development section documents yarn test:e2e for Playwright runs but does not mention the new yarn test:e2e:performance script added in this PR.
    Remediation: Add a line beneath yarn test:e2e mentioning the performance benchmark command with a pointer to performance/ai-catalog.md.

  • [missing-doc] workspaces/boost/AGENTS.md:371 — The "Build & verify" command table lists yarn test:e2e for Playwright e2e but omits the new yarn test:e2e:performance command.
    Remediation: Add a row to the table for yarn test:e2e:performance.


Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR

@fullsend-ai-review fullsend-ai-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the review comment for full details.

Comment thread workspaces/boost/e2e-tests/boost.AiCatalogPerformance.test.ts
Comment thread workspaces/boost/e2e-tests/boost.AiCatalogPerformance.test.ts
Comment thread workspaces/boost/e2e-tests/boost.AiCatalogPerformance.test.ts
@gabemontero

Copy link
Copy Markdown
Contributor

/fs-fix [missing-doc] workspaces/boost/README.md:70 — The Development section documents yarn test:e2e for Playwright runs but does not mention the new yarn test:e2e:performance script added in this PR. Remediation: Add a line beneath yarn test:e2e mentioning the performance benchmark command with a pointer to performance/ai-catalog.md. [missing-doc] workspaces/boost/AGENTS.md:371 — The "Build & verify" command table lists yarn test:e2e for Playwright e2e but omits the new yarn test:e2e:performance command. Remediation: Add a row to the table for yarn test:e2e:performance.

@gabemontero
gabemontero merged commit e5b2eb7 into redhat-developer:main Sep 22, 2026
46 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation risk/moderate PR risk: moderate Tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants