feat: add Vector Search plugin

[adamgurary]

Summary

• Adds @databricks/appkit-vector-search — a plugin that gives Databricks Apps built with AppKit instant vector search query capabilities
• Ships backend (Express routes, VS REST API client, service principal + OBO auth) and frontend (React hook, styled components with Radix UI)
• Developer experience target: ~45 lines for a full search page with search box, results, filters, and keyword highlighting
• 82 tests included; validated against real VS index on dogfood

What's included

Backend plugin (src/plugin/):

• VectorSearchPlugin.ts — plugin class with lifecycle, config, route injection
• VectorSearchClient.ts — REST API client for VS endpoints
• routes.ts — Express route handlers
• auth.ts — service principal default, OBO opt-in per index

Frontend UI (src/ui/):

• useVectorSearch React hook
• SearchBox, SearchResults, SearchResultCard, SearchLoadMore components

Design decisions

Decision	Choice	Rationale
Package structure	Single package (backend + UI)	Shared types, single dependency, matches Lakebase plugin pattern
Default search mode	Hybrid (ANN + keyword)	Best out-of-the-box quality
Reranking	Off by default, opt-in per index	Adds latency; too slow for interactive search by default
Auth	Service principal default, OBO opt-in	Simple default, secure option when needed

Test plan

• Review plugin structure against existing AppKit plugin patterns (Lakebase)
• Run test suite (vitest run in packages/vector-search/)
• Validate against live VS index on dogfood
• Review API surface and types

[calvarjorge]

These comments from the AI review make sense to me:

  #: 1
  Severity: critical
  File: plugins/vector-search/vector-search.ts
  Line: 175-189
  Issue: Missing execute() interceptor
  Description: Every other plugin wraps API calls in this.execute() for cache/retry/timeout/telemetry. This plugin calls
  the
    connector directly, bypassing the entire interceptor chain.
  ────────────────────────────────────────
  #: 2
  Severity: high
  File: plugins/vector-search/defaults.ts
  Line: 1-7
  Issue: Defaults defined but never used
  Description: vectorSearchDefaults is exported but never imported anywhere — dead code.
  ────────────────────────────────────────
  #: 3
  Severity: high
  File: connectors/vector-search/client.ts
  Line: 24-26
  Issue: timeout config stored but never applied
  Description: Constructor stores this.config.timeout but no AbortSignal or timeout mechanism is ever created from it.
  ────────────────────────────────────────
  #: 4
  Severity: high
  File: plugins/vector-search/vector-search.ts
  Line: 140
  Issue: No request body validation
  Description: req.body is cast directly to SearchRequest without Zod validation. The repo design philosophy states
    "Type-safe — Heavy TypeScript usage with runtime validation (Zod)". Other plugins validate at system boundaries.
  ────────────────────────────────────────
  #: 5
  Severity: medium
  File: plugins/vector-search/vector-search.ts
  Line: 349
  Issue: fromCache: false hardcoded
  Description: Since execute() isn't used, caching never happens — this field is always false, making it misleading in the
    response type.
  ────────────────────────────────────────
  #: 6
  Severity: medium
  File: plugins/vector-search/vector-search.ts
  Line: 296-298
  Issue: shutdown() calls streamManager.abortAll() but plugin never uses streaming
  Description: The plugin has no SSE streams. This won't error (base class provides streamManager), but it's misleading
    about the plugin's capabilities.
  ────────────────────────────────────────
  #: 7
  Severity: medium
  File: plugins/vector-search/vector-search.ts
  Line: 234
  Issue: Non-null assertion on endpointName!
  Description: indexConfig.endpointName! uses ! operator. Validated at setup time, but if config changes dynamically or
    setup is skipped, this will throw an untyped error.

[atilafassina]

Great work!

I left a few comments, some are minor but I think there are 3 critical ones

1. not wrapping the plugin connector calls in the executor
2. returning plain objects as error
3. not wrapping the embeddingFn in a try/catch

The AbortControl in the connector would be nice to have too, but that's not a big issue if we have the plugin calls wrapped in the executor.

The "hybrid" hardcode in the response is mostly a question from me, maybe it's a non-issue.

---

Once we land in the right API, we need to add docs and a route in the dev-playground. Btu I'd defer that for after a the reviews to avoid re-work. Ofc, you do what you prefer :)

[pkosiec]

Thanks a lot for your contribution! I think most have been already said by Atila and Jorge (e.g. about the dev playground addition), just a few small comments from my side.

IMO the most important point is to figure out the apps init flow (configure required resource(s) for the plugin, to be selectable interactively as a part of the init flow), and also make sure that the API is designed around the resource. See the Files, Genie or Lakebase plugins as an example. Thanks!

[adamgurary]

Addressed all review feedback from @atilafassina, @calvarjorge, and @pkosiec. Rebased onto current main.

Blockers (Atila)

• this.execute() wrapping — All connector calls now go through the interceptor chain (retry/cache/timeout/telemetry). vectorSearchDefaults feeds into execute(), no longer dead code.
• Error instances — Programmatic query() throws new Error(...) instead of plain objects.
• embeddingFn try/catch — User-provided embedding function wrapped in try/catch with clear error message.

Pattern alignment

• Error responses now use { error, plugin } shape matching Files/Genie
• Extracted _prepareQuery() helper to DRY shared logic between query() and route handlers
• Removed streamManager.abortAll() from shutdown() (plugin has no streams)
• Removed fromCache from response type (execute handles caching transparently)
• Replaced endpointName! non-null assertion with runtime guard
• Next-page response uses indexConfig.queryType instead of hardcoded "hybrid"
• Removed unused Context import from connector; AbortSignal passed from execute() to connector

New content

• manifest.json — Added vector_search_index required resource with env fields for apps init flow
• Dev playground — Server config, nav link, search route with form + results display
• Docs — docs/docs/plugins/vector-search.md with usage, config options, API reference
• Template — App.tsx routes, VectorSearchPage.tsx, appkit.plugins.json entry
• Package export — vectorSearch added to top-level index.ts

Skipped

• Zod request body validation (Jorge's AI review feat: arrow stream #4) — No existing plugin uses Zod for req.body validation. All use simple if (!field) guards. Matched existing patterns.

Note on CI

CI doesn't appear to run on fork PRs (protected runner group). Could not run lint/typecheck/tests locally either — the Databricks npm proxy (npm-proxy.dev.databricks.com) is returning empty replies / ECONNRESET. If a maintainer can trigger CI or approve the workflow run, that would help verify. Happy to fix anything CI catches.

[atilafassina]

🏆 Amazing work, @adamgurary
thank you very much

---

What's next:
I'll QA and update the docs, then we'll prepare a new release with the plugin available.

I'll ping you for review when writing the skills for it