Skip to content

feat(shared): bake execution capabilities into synced plugin manifests - #623

Open
MarioCadenas wants to merge 14 commits into
resource-and-execution-base-prfrom
manifest-scope-capabilities
Open

MarioCadenas wants to merge 14 commits into
resource-and-execution-base-prfrom
manifest-scope-capabilities

Conversation

@MarioCadenas

@MarioCadenas MarioCadenas commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Front 1, appkit side (F1.1, F1.2, F1.5). appkit plugin sync now bakes the type-level execution facts into appkit.plugins.json, so the Go CLI can pick user_api_scopes and force SP binds from plain data, with no scope map of its own. Stacked on #592 (needs S2 only) and orthogonal to the Front 2 runtime stack (#594 onward). Do not merge independently of the base.

Contract (what the CLI reads)

All fields are optional and additive, so an older CLI ignores them.

  • Per resource entry
    • scope: string, present only when the type can run on behalf of the user. Value from SCOPE_BY_TYPE.
    • appOnly: true, present only when the type is in APP_ONLY_RESOURCE_TYPES (secret, database, postgres). Omitted otherwise.
  • Per plugin entry
    • scopes: string[], the scopes the plugin always needs (unconditional on-behalf-of calls, or capabilities with no resource ID), copied from the authoring manifest. Omitted when empty.

Example, the analytics sql_warehouse entry in template/appkit.plugins.json:

{
  "type": "sql_warehouse",
  "alias": "SQL Warehouse",
  "resourceKey": "sql-warehouse",
  "description": "SQL Warehouse for executing analytics queries",
  "permission": "CAN_USE",
  "fields": {
    "id": {
      "env": "DATABRICKS_WAREHOUSE_ID",
      "description": "SQL Warehouse ID",
      "discovery": { "type": "kind", "resourceKind": "warehouse" },
      "origin": "user"
    }
  },
  "scope": "sql"
}

Changes

  • templateResourceRequirementSchema gains scope and appOnly; templatePluginSchema gains scopes. JSON schema regenerated, template manifest re-synced.
  • Both sync discovery paths (plugins dir via loadPluginEntry, and the node_modules scan in scanForPlugins) now build entries through one toTemplatePlugin helper, so they can't drift apart again.
  • template/databricks.yml.tmpl: when the CLI sets .bundle.userApiScopes (pre-rendered list items, generator owns indentation, same as .bundle.resources), it renders under user_api_scopes:. Otherwise the existing plugin-presence block renders unchanged.
  • Plugins that always call on behalf of the user declare that scope in their manifest scopes, which the CLI unions whenever any resource is OBO: genie ["genie"], serving ["model-serving"]. Without this, a mixed app (for example --auth-mode obo with the genie space bound as SP) dropped the genie scope and every genie request failed closed. analytics, files, ai-search, and agents only act as the user when a resource or config opts in, so they declare nothing. The plugin scopes enum widens from the seven capability-only scopes to every user_api_scope.
  • appkit add drops its separate SCOPE_BY_RESOURCE_TYPE map (deprecated long names, three types) and reads SCOPE_BY_TYPE (separate commit).

Scope names are the short ones from the authoritative list in design-docs/execution-identity-e2e.md section 8. The long forms are deprecated aliases. SP stays the default: nothing here changes what a CLI that doesn't read these fields generates.

Verification

  • Sync tests exercise both discovery paths against real manifests on disk and assert scope, appOnly, scopes, and that they are omitted when they don't apply.
  • Template fallback: rendered with Go text/template. With no userApiScopes, output is byte-identical to the previous template, with and without a genie plugin. With it set, the provided list renders. A guard test pins the fallback block.
  • pnpm --filter=shared typecheck, shared suite 662 passed, pnpm check, pnpm build, pnpm docs:build (no generated-doc drift).

This pull request and its description were written by Isaac.

MarioCadenas and others added 3 commits September 30, 2026 12:25
`appkit plugin sync` now resolves the type-level execution facts into
appkit.plugins.json so the Go CLI can select user_api_scopes from plain
data, with no scope map of its own:

- each resource entry gets `scope` (from SCOPE_BY_TYPE) when the type can
  run on behalf of the user, and `appOnly: true` when it is in
  APP_ONLY_RESOURCE_TYPES
- each plugin entry gets `scopes`, the capability-only scopes from the
  authoring manifest, omitted when empty

All fields are optional and additive, so older CLIs ignore them. Both
discovery paths (plugins dir and node_modules scan) now build entries
through one `toTemplatePlugin` helper so they cannot drift apart again.
Regenerates the template-plugins JSON schema and template manifest.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
databricks.yml.tmpl now renders `.bundle.userApiScopes` under
user_api_scopes when the generator provides it (pre-rendered list items,
generator owns indentation, same convention as `.bundle.resources`).

When the value is unset, the existing plugin-presence block renders
unchanged, so an older CLI produces exactly today's output. A guard test
pins that fallback block.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
`appkit add` kept its own SCOPE_BY_RESOURCE_TYPE map with the deprecated
long scope names (dashboards.genie, files.files, serving.serving-endpoints)
and only three types. It now reads SCOPE_BY_TYPE, so there is one map and
the deploy hint names the same scope the synced manifest carries, including
sql, vector-search, and catalog.connections.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
@MarioCadenas
MarioCadenas requested a review from a team as a code owner September 30, 2026 10:41
@MarioCadenas
MarioCadenas requested review from atilafassina and removed request for a team September 30, 2026 10:41
genie and serving run every route on behalf of the user, whatever the
resource is bound as. In a mixed app where the CLI emits user_api_scopes
from OBO resources only, their scope was missing and every request failed
closed.

Plugins now declare such scopes as data in their manifest `scopes`, which
the CLI already unions whenever any resource is OBO:

- genie: ["genie"]
- serving: ["model-serving"]

analytics (.obo.sql lanes), files and ai-search (per-volume or per-index
`auth` config, default service principal), and agents (tools carry their
own scopes; the model call runs as the service principal) only act as the
user when something opts in, so they declare nothing.

The plugin `scopes` enum widens from the seven capability-only scopes to
every user_api_scope. Widening accepts more manifests and rejects none that
were valid before. The all-SP default is unchanged: the CLI emits no
user_api_scopes there and the template keeps its plugin-presence block.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
@MarioCadenas
MarioCadenas changed the base branch from resource-and-execution-base-pr to main September 30, 2026 12:28
@MarioCadenas MarioCadenas reopened this Sep 30, 2026
@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

📦 Bundle size report

Compared against bundle-size-baseline.json (main).

@databricks/appkit

npm tarball (packed): 1.2 MB (+9.6 KB) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist raw gzip
JS (runtime) 1.2 MB (+9.1 KB) 431 KB (+3.4 KB)
Type declarations 449 KB (+3.3 KB) 163 KB (+1.4 KB)
Source maps 2.4 MB (+19 KB) 809 KB (+6.6 KB)
Other 11 KB 3.7 KB
Total 4.0 MB (+31 KB) 1.4 MB (+11 KB)
Per-entry composition (own code — deps external (as shipped))
Entry Initial (gz) Lazy (gz) Total (gz) node_modules (min) Own code (min)
. 98 KB (+1.1 KB) 2.5 KB 100 KB (+1.1 KB) external 319 KB (+3.5 KB)
./beta 94 KB (+992 B) 479 B (+1 B) 95 KB (+993 B) external 285 KB (+2.7 KB)
./testing 40 KB (+1.0 KB) 31 KB (+1 B) 70 KB (+1.0 KB) external 205 KB (+3.2 KB)
./tsdown 520 B 0 B 520 B external 813 B
./type-generator 23 KB 0 B 23 KB external 65 KB

Chunks:

Entry Chunk Load Size (gz)
. index.js initial 93 KB
. utils.js initial 4.6 KB
. remote-tunnel-manager.js lazy 2.5 KB
./beta beta.js initial 77 KB
./beta stream-manager.js initial 5.8 KB
./beta databricks.js initial 3.3 KB
./beta wide-event-emitter.js initial 3.1 KB
./beta configuration.js initial 2.3 KB
./beta service-context.js initial 1.9 KB
./beta client.js initial 593 B
./beta client-options.js initial 219 B
./beta supervisor-api.js lazy 191 B
./beta databricks.js lazy 165 B
./beta index.js lazy 123 B
./testing manifest.js initial 27 KB
./testing index.js initial 10 KB
./testing wide-event-emitter.js initial 2.9 KB
./testing index.js lazy 27 KB
./testing remote-tunnel-manager.js lazy 2.5 KB
./testing utils.js lazy 1.8 KB
./tsdown index.js initial 520 B
./type-generator index.js initial 23 KB

@databricks/appkit-ui

npm tarball (packed): 350 KB — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist raw gzip
JS (runtime) 395 KB 132 KB
Type declarations 229 KB 84 KB
Source maps 766 KB 253 KB
CSS 16 KB 3.2 KB
Total 1.4 MB 472 KB
Per-entry composition (consumer bundle — deps bundled, peerDeps external)
Entry Initial (gz) Lazy (gz) Total (gz) node_modules (min) Own code (min)
./js 5.3 KB 49 KB 55 KB 208 KB 14 KB
./js/beta 20 B 0 B 20 B 0 B 0 B
./react 432 KB 49 KB 481 KB 1.3 MB 177 KB
./react/beta 1.0 KB 0 B 1.0 KB 0 B 1.9 KB

Chunks:

Entry Chunk Load Size (gz)
./js index.js initial 5.2 KB
./js chunk initial 120 B
./js apache-arrow lazy 49 KB
./js/beta beta.js initial 20 B
./react index.js initial 430 KB
./react tslib initial 2.1 KB
./react apache-arrow lazy 49 KB
./react/beta beta.js initial 1.0 KB

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

🤖 AppKit PR bot

🔬 Run evals

Start an eval for this PR from the evals-monitor app: Go to Evals Monitor →

📦 Try this PR's app template

Scaffolds a new app from this PR's SDK build. Run it in any folder (requires the GitHub CLI — gh auth login — and the Databricks CLI):

gh run download 36865245264 -R databricks/appkit -n appkit-template-0.81.0-pr.33fc7c9-manifest-scope-capabilities-623 -D appkit-pr-623 \
  && unzip -o "appkit-pr-623/appkit-template-0.81.0-pr.33fc7c9-manifest-scope-capabilities-623.zip" -d "appkit-pr-623" \
  && databricks apps init --template "appkit-pr-623"

The template pins @databricks/appkit and @databricks/appkit-ui to tarballs built from this branch, so the scaffolded app runs against this PR's code.

@MarioCadenas
MarioCadenas changed the base branch from main to resource-and-execution-base-pr September 30, 2026 12:46
…ilities

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
@MarioCadenas
MarioCadenas changed the base branch from resource-and-execution-base-pr to main October 1, 2026 08:55
@MarioCadenas MarioCadenas reopened this Oct 1, 2026
@MarioCadenas
MarioCadenas changed the base branch from main to resource-and-execution-base-pr October 1, 2026 09:19
MarioCadenas and others added 2 commits October 1, 2026 14:10
After the plugin `scopes` fields moved to userApiScopeSchema, the exported
CapabilityScope type has no consumer. The capabilityScopeSchema value
stays, since its options feed userApiScopeSchema.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
…ilities

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
@MarioCadenas
MarioCadenas changed the base branch from resource-and-execution-base-pr to main October 1, 2026 12:32
@MarioCadenas MarioCadenas reopened this Oct 1, 2026
…ilities

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
@MarioCadenas
MarioCadenas changed the base branch from main to resource-and-execution-base-pr October 1, 2026 13:07
…ames

The commented example in databricks.yml.tmpl listed the deprecated long
scope names. Update it to the current short names (sql, genie, files,
model-serving). Comment only; the active plugin-presence fallback block is
unchanged.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
@MarioCadenas
MarioCadenas changed the base branch from resource-and-execution-base-pr to main October 1, 2026 14:54
@MarioCadenas MarioCadenas reopened this Oct 1, 2026
@MarioCadenas
MarioCadenas changed the base branch from main to resource-and-execution-base-pr October 1, 2026 15:03
MarioCadenas and others added 5 commits October 1, 2026 18:23
…ment

The template comment was updated to short scope names, but this test still
pinned the old long-name comment, so the fallback-block assertion failed.
Update the expected block.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Move the CLI's per-type databricks.yml binding spec into the manifest so
AppKit owns it and the CLI consumes it as data instead of a hardcoded map.

- Add DABS_BINDING_BY_TYPE next to SCOPE_BY_TYPE: a faithful port of the
  CLI's appResourceSpecs, keyed by resource type, with { yamlKey,
  varFields: [[manifestField, dabsField]], staticFields?: [[dabsField,
  value]] }. App-only types still bind (as the service principal). The app
  type is absent, matching the CLI's commented-out entry.
- Add an optional per-resource `binding` object to the template resource
  schema. Additive and optional, so older CLIs ignore it.
- sync resolves binding from DABS_BINDING_BY_TYPE and bakes it onto each
  resource, materializing mutable tuples so nothing shares a reference with
  the source table.
- Re-sync template/appkit.plugins.json: every resource now carries binding.
- Regenerate the template-plugins JSON schema.
- Tests: both sync paths bake the expected binding for sql_warehouse,
  secret, and job, and the real core manifests bake genie_space and the
  volume uc_securable static field.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Add a compile-time drift anchor for DABS_BINDING_BY_TYPE yamlKeys, isolated
to one SDK-version-dependent seam so finishing it after the SDK migration is
a three-line change in a single place.

- The seam holds a local AppResource shim (the installed sdk-experimental
  0.17 does not export the apps AppResource types) and the single skew
  exception union AppResourceKind = keyof AppResource | "postgres" |
  "experiment" | "app", with a TODO(sdk-migration) listing the finish steps:
  repoint the import to the SDK apps AppResource, drop the exceptions, and
  tighten to a bare keyof AppResource.
- A type-only AssertAllTrue check fails the build if any binding yamlKey is
  not a known AppResource kind.

Types-only: DABS_BINDING_BY_TYPE values, sync output, and the baked template
manifest are unchanged.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
…rate TODO

The CLI side owns the real Apps SDK anchoring (Go reflection of
apps.AppResource, available now), and the JS SDK apps types are not
importable until the modular migration. So no TS shim should linger.

Remove the local AppResource shim, the AppResourceKind union, and the
AssertAllTrue drift check. DABS_BINDING_BY_TYPE stays a plain table, now
carrying a TODO(sdk-migration) grounded in the real target shape
(@databricks/sdk-apps AppResource $case union) with the exact finish steps.

Types and comment only: the baked template manifest and runtime output are
unchanged, and every resource still carries its binding.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
The binding specs referenced manifest field `id` for two types whose
resources declare a different field, so apps init emitted
${var.<res>_id} against an unset variable:
- volume: now [["path", "securable_full_name"]] (declares `path`)
- experiment: now [["experimentId", "experiment_id"]] (declares `experimentId`)
yamlKey and staticFields are unchanged. This corrects the baked binding for
files, agents-skills (volumes) and agents-mlflow-experiment.

Add a sync-time guard: when baking binding, assert every varFields
manifestField is one of the resource's declared fields, and throw a clear
error naming plugin/resource/field otherwise. The error is a distinct
BindingFieldError that discovery re-throws instead of swallowing, so a
misconfig fails sync. A unit test covers a resource whose binding
references an undeclared field.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant