Skip to content

feat(intelligent-assistant): screen context chip, kebab opt-in, and gated attachments ( - #4862

Open
its-mitesh-kumar wants to merge 8 commits into
redhat-developer:mainfrom
its-mitesh-kumar:feat/screen-context-chip
Open

its-mitesh-kumar wants to merge 8 commits into
redhat-developer:mainfrom
its-mitesh-kumar:feat/screen-context-chip

Conversation

@its-mitesh-kumar

@its-mitesh-kumar its-mitesh-kumar commented Sep 17, 2026

Copy link
Copy Markdown
Member

Summary

Implements the screen context UX for the Intelligent Assistant message bar: users opt in via the header kebab, see a context chip that reflects recording/paused/unavailable states, and send messages with DOM and/or screenshot context only when admin config, user settings, model capabilities, and display mode allow it.

Changes

  • User settingsuseScreenContextSettings persists screenContextSharingEnabled and screenContextPaused in the lightspeed StorageApi bucket (sharing defaults to off).
  • Context chipScreenContextChip with dynamic labels/tooltips (search, software templates, settings breadcrumbs, catalog routes) and label refresh on SPA navigation via history/MutationObserver subscription.
  • Send gatingscreen-context-utils centralizes shouldAttachScreenContext / buildScreenContextAttachments (admin screen-context.enabled, user sharing, pause, non-fullscreen; DOM via dom-extraction.enabled; screenshots via screenshots.enabled + model supportsVision).
  • UI integration — Chip in message bar; kebab enable/disable screen context; vision/screenshot indicator on model selector when screenshots apply; drawer/modal excluded from capture via data-screen-capture-exclude.
  • i18n — New strings in ref.ts and locale files; alpha API report updated.
  • Tests — Hooks, utils, chip, model selector, label subscription; LightspeedChat / RBAC gating tests register storageApiRef for the new hook.

Config

intelligent-assistant:
  screen-context:
    enabled: false            # gates the entire feature (default: false)
    screenshots:
      enabled: true           # admin can disable screenshots org-wide
    dom-extraction
      enabled: true.           # admin can disable screenshots org-wide

UI after changes

S_.2026-09-18.at.12.32.07.AM.mov
S_.2026-09-18.at.12.36.01.AM.mov
S_.2026-09-18.at.8.37.11.AM.mov
S_.2026-09-18.at.8.mp4

Test plan

  • Enable screen context in IA kebab; confirm chip appears when admin screen-context.enabled is true.
  • Pause/resume from chip; confirm send does not attach context while paused.
  • Navigate catalog, search, /create/templates, and template detail; confirm chip label and tooltip update.
  • Send with a vision-capable model: verify DOM and/or screenshot attachments per config.
  • Fullscreen mode: chip/state behaves as designed; capture exclusions on chat drawer.
  • yarn test in workspaces/intelligent-assistant/plugins/intelligent-assistant

Related

  • A changeset describing the change and affected packages. (more info)
  • Added or Updated documentation
  • Tests for new functionality and regression tests for bug fixes
  • Screenshots attached (for UI changes)

Signed-off-by: its-mitesh-kumar <itsmiteshkumar98@gmail.com>
@its-mitesh-kumar

Copy link
Copy Markdown
Member Author

/fs-review

@rhdh-gh-app

rhdh-gh-app Bot commented Sep 17, 2026

Copy link
Copy Markdown

Important

This PR includes changes that affect public-facing API. Please ensure you are adding/updating documentation for new features or behavior.

Changed Packages

Package Name Package Path Changeset Bump Current Version
@red-hat-developer-hub/backstage-plugin-intelligent-assistant workspaces/intelligent-assistant/plugins/intelligent-assistant minor v5.2.0

@fullsend-ai-review

fullsend-ai-review Bot commented Sep 17, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 7:45 PM UTC · Completed 8:10 PM UTC

Commit: 56c1bff · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $9.61

@codecov

codecov Bot commented Sep 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.93204% with 57 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.29%. Comparing base (ce6f5f5) to head (c9776e4).
⚠️ Report is 40 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4862      +/-   ##
==========================================
+ Coverage   60.14%   60.29%   +0.14%     
==========================================
  Files        2673     2679       +6     
  Lines      106162   106660     +498     
  Branches    29807    29943     +136     
==========================================
+ Hits        63856    64310     +454     
- Misses      41789    41832      +43     
- Partials      517      518       +1     
Flag Coverage Δ *Carryforward flag
adoption-insights 84.77% <ø> (ø) Carriedforward from 2b71de1
ai-integrations 79.83% <ø> (ø) Carriedforward from 2b71de1
app-defaults 54.82% <ø> (ø) Carriedforward from 2b71de1
augment 46.67% <ø> (ø) Carriedforward from 2b71de1
boost 84.97% <ø> (ø) Carriedforward from 2b71de1
bulk-import 73.12% <ø> (ø) Carriedforward from 2b71de1
cost-management 13.53% <ø> (ø) Carriedforward from 2b71de1
dcm 73.47% <ø> (ø) Carriedforward from 2b71de1
e2e-adoption-insights 60.00% <ø> (ø) Carriedforward from 2b71de1
e2e-extensions 62.31% <ø> (ø) Carriedforward from 2b71de1
e2e-global-header 51.82% <ø> (ø) Carriedforward from 2b71de1
e2e-homepage 61.11% <ø> (ø) Carriedforward from 2b71de1
e2e-intelligent-assistant 45.96% <ø> (-0.05%) ⬇️ Carriedforward from 2b71de1
e2e-orchestrator 49.49% <ø> (ø) Carriedforward from 2b71de1
e2e-orchestrator-plugin 49.48% <ø> (ø) Carriedforward from 2b71de1
e2e-quickstart 55.21% <ø> (ø) Carriedforward from 2b71de1
e2e-scorecard 50.05% <ø> (ø) Carriedforward from 2b71de1
e2e-theme 16.36% <ø> (ø) Carriedforward from 2b71de1
extensions 58.30% <ø> (ø) Carriedforward from 2b71de1
global-floating-action-button 71.18% <ø> (ø) Carriedforward from 2b71de1
global-header 67.76% <ø> (ø) Carriedforward from 2b71de1
homepage 48.39% <ø> (ø) Carriedforward from 2b71de1
install-dynamic-plugins 71.77% <ø> (ø) Carriedforward from 2b71de1
intelligent-assistant 78.75% <88.93%> (+0.76%) ⬆️
konflux 91.98% <ø> (ø) Carriedforward from 2b71de1
lightspeed 69.02% <ø> (ø) Carriedforward from 2b71de1
mcp-integrations 84.46% <ø> (ø) Carriedforward from 2b71de1
orchestrator 77.32% <ø> (ø) Carriedforward from 2b71de1
quickstart 63.74% <ø> (ø) Carriedforward from 2b71de1
sandbox 79.56% <ø> (ø) Carriedforward from 2b71de1
scorecard 88.44% <ø> (ø) Carriedforward from 2b71de1
theme 87.91% <ø> (ø) Carriedforward from 2b71de1
translations 5.12% <ø> (ø) Carriedforward from 2b71de1
x2a 13.86% <ø> (ø) Carriedforward from 2b71de1

*This pull request uses carry forward flags. Click here to find out more.


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ce6f5f5...c9776e4. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@its-mitesh-kumar

Copy link
Copy Markdown
Member Author

/cc @JslYoon @HusneShabbir

@fullsend-ai-review fullsend-ai-review Bot added the risk/moderate PR risk: moderate label Sep 17, 2026
@fullsend-ai-review

Copy link
Copy Markdown

Risk Assessment: moderate (2/5)

Details

Large-footprint feature addition (2500 lines, 31 files) from a regular contributor with good test coverage and no protected-path, security, CI, or dependency concerns; elevated regression history on LightSpeedChat.tsx offset by majority new files.

@fullsend-ai-review

Copy link
Copy Markdown

Review

Findings

High

  • [stale-doc] README.md:164 — The README states the assistant "automatically captures structured page context" and "attaches it to each message." This PR adds a user-level opt-in gate via the kebab menu (isScreenContextSharingEnabled defaults to false), making the "automatically" claim incorrect. Administrators relying on the README will misunderstand the feature's behavior.
    Remediation: Update the Screen Context section to explain two-level enablement: (1) admin enables screen-context.enabled, (2) user opts in via the kebab menu. Document the ScreenContextChip states (recording, paused, unavailable).

Medium

  • [logic-error] screenContextLabelSubscription.ts:41isInsideScreenCaptureExclude returns false for Text nodes because they fail the instanceof Element check. The MutationObserver is configured with characterData: true (line 101), so text mutations inside [data-screen-capture-exclude] areas (e.g., streaming chatbot responses) bypass the filter, triggering unnecessary snapshot recalculations on every token update.
    Remediation: Walk up to the nearest Element parent: const el = node instanceof Element ? node : node.parentElement;

  • [logic-error] screen-context-utils.ts:241getScreenContextRouteKind uses search.includes(\${key}=`)for search query detection. This produces false positives:?freq=5matchesq=, ?longterm=1matchesterm=. The companion getSearchQueryFromLocationcorrectly usesURLSearchParams. Remediation: Use new URLSearchParams(search).has(key)` instead of substring matching.

  • [architectural-coherence] screenContextLabelSubscription.ts:128 — The module patches window.History.prototype.pushState/replaceState at load time, affecting all JavaScript on the page. The instance-own property on window.history (lines 118–126) already shadows the prototype, making the prototype patch dead code for window.history calls. This is an architectural departure from Backstage's plugin isolation model.
    Remediation: Remove the prototype patch (lines 128–138). Use React Router hooks or popstate events instead.

  • [code-organization] de.ts:362 — All ~30 new translation keys across de.ts, es.ts, fr.ts, it.ts, and ja.ts use English-only text. The established pattern is proper localization (e.g., 'Gespeicherte Prompts aktivieren' in de.ts).
    Remediation: Translate the new strings or add // TODO: translate comments.

  • [missing-doc] README.md:162 — The Screen Context section does not mention the new ScreenContextChip indicator or the user-controlled pause/resume flow (recording, paused, unavailable states).
    Remediation: Add a sub-section documenting chip states and user interactions.

  • [missing-doc] README.md:173 — The README documents screenshots.enabled but does not explain that screenshots require a vision-capable model (supportsVision).
    Remediation: Add a note that screenshot capture requires a vision-capable model.

Low

  • [logic-error] screen-context-utils.ts:232pathname.includes('/create/') broadly matches any path containing /create/. In Backstage /create/ is the scaffolder route, limiting false-positive risk; a misclassification only affects chip label text.

  • [logic-error] screen-context-utils.ts:480getScreenContextTooltipLine2Key falls through to adminLimited for {domEnabled: false, screenshotsEnabled: true, supportsVision: false}. The tooltip is misleading since the limitation is the model's lack of vision support, not admin settings.

  • [dead-code] useScreenContextSettings.ts:129handleScreenContextSharingToggle has identical if/else branches that both execute setIsScreenContextPaused(false); persistPaused(false);.

  • [test-adequacy] screen-context-utils.test.ts:275getScreenContextTooltipLine2Key tests cover only 3 of 8 boolean input combinations, missing the misleading adminLimited fallback path.

  • [fail-open] useScreenContextSettings.ts:131 — The toggle callback guards against undefined user but not guest users. A guest can toggle in-memory isScreenContextSharingEnabled after mount, though persistence is blocked and practical risk is limited.

  • [naming-conventions] LightspeedChatBoxHeader.tsx:300 — The disable screen context DropdownItem uses truncated value="disa..." and key="disa..." while its sibling uses "enableScreenContext".

  • [incomplete-doc] README.md:179 — The prose bullet list omits screenshots.enabled though the YAML example includes it.

  • [architectural-coherence] useWindowLocation.ts:29 — Module-level patchBrowserHistoryForScreenContext() call is typically preempted by the guard since screenContextLabelSubscription runs first via index.tsx import; serves as defensive initialization.

  • [architectural-coherence] useWindowLocation.ts:48useWindowLocation is a general-sounding hook name that internally delegates to subscribeToScreenContextLabel, obscuring the coupling to the screen context system.


Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR

@fullsend-ai-review fullsend-ai-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the review comment for full details.

Signed-off-by: its-mitesh-kumar <itsmiteshkumar98@gmail.com>
…IDP-14319

Fill gating-matrix and kebab RTL gaps, and add mocked Playwright journeys for kebab/chip/pause/fullscreen so QE layers under RHIDP-14768 can gate the Deep Context UX.

Co-authored-by: Cursor <cursoragent@cursor.com>
Complete CaptureResult mocks for tsc, and assert paused-send attachments via the same query route pattern used by MCP specs (no try/catch).

Co-authored-by: Cursor <cursoragent@cursor.com>

@HusneShabbir HusneShabbir left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

HusneShabbir and others added 2 commits September 18, 2026 18:25
Avoid Escape between kebab verify/enable (it closes the chatbot), re-enable sharing after fullscreen remount for guest e2e, and wait on the Options control in the chat header.

Assisted-by: Composer
Co-authored-by: Cursor <cursoragent@cursor.com>
… strict mode

getByText on the paused label matched both the Label button and inner span; use the same state CSS class pattern as recording/unavailable.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread workspaces/intelligent-assistant/plugins/intelligent-assistant/src/index.tsx Outdated
Signed-off-by: its-mitesh-kumar <itsmiteshkumar98@gmail.com>
@openshift-ci openshift-ci Bot removed the lgtm label Sep 19, 2026
@openshift-ci

openshift-ci Bot commented Sep 19, 2026

Copy link
Copy Markdown

New changes are detected. LGTM label has been removed.

@its-mitesh-kumar

Copy link
Copy Markdown
Member Author

@debsmita1 I have addressed your comment and verify it, its working fine. If no other comment then we can get this PR in.

…ext-coverage

test(intelligent-assistant): screen context L1/L3/L4a coverage (RHIDP-14319)
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants