From 9a95b6ebd864a39c5c426d63a114fe5a6e3fcdb7 Mon Sep 17 00:00:00 2001 From: Preetam Dwivedi Date: Wed, 30 Sep 2026 13:58:57 -0700 Subject: [PATCH 1/2] docs: define scoped sequential resource IDs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary ### Why? Slash-delimited resource IDs repeat queue and resource-kind context that stores, APIs, and messages already carry separately, while making a single ID span multiple browser path segments. ### What? Define counter-generated resource IDs as positive numeric values scoped by owner domain, queue, and resource kind. Persist durable counter high-water marks, store numeric IDs directly under queue-leading keys, keep queue and kind out of the ID, and reserve prefixes such as `request.42` for presentation only. ## Test Plan ✅ `make fmt` ✅ `make lint-binary lint-license lint-message-id lint-queue-shard` ✅ `git diff --check` --- doc/rfc/index.md | 1 + doc/rfc/scoped-resource-ids.md | 73 ++++++++++++++++++++++++++++++++++ 2 files changed, 74 insertions(+) create mode 100644 doc/rfc/scoped-resource-ids.md diff --git a/doc/rfc/index.md b/doc/rfc/index.md index d3697190..12981bcb 100644 --- a/doc/rfc/index.md +++ b/doc/rfc/index.md @@ -10,6 +10,7 @@ Design documents and technical proposals, grouped by scope. Shared/cross-cutting - [Consumer Gate](consumer-gate.md) - Stopping and starting individual queue controllers at runtime via a consumer-side check: blocked deliveries are recorded as parked and postponed back to the queue (re-checked on redelivery), gate state as a separate extension with a file-based first implementation shared by tests and operators - [Consumer Hold](consumer-hold.md) - Fourth delivery outcome letting a controller postpone its delivery: the message becomes a partition barrier that pauses consumption for a chosen delay, redelivers in order, and does not count as a failure toward dead-lettering - [Change URIs](change-uri.md) - Identity of a code change: `scheme://{host[:port]}/{path}` per provider (GitHub PR, Phabricator Diff, git ref/commit) and canonical-form rules +- [Scoped Sequential Resource IDs](scoped-resource-ids.md) - Queue-scoped positive numeric IDs allocated by durable per-domain, per-kind counters, stored without queue/kind prefixes, and rendered directly in resource URL segments - [Hooks Framework](hook-framework.md) - Implemented fire-and-forget side effects: one shared `HookEvent` contract (`api/base/hook/`) on a durable per-domain hook topic, dispatched by `platform/hook` to `platform/extension/hook`. Stovepipe `process` and `record` publish repository events; the SubmitQueue orchestrator registers the stage and does not publish events yet - [Service-Scoped Extensions](service-scoped-extensions.md) - Implemented for SubmitQueue storage: gateway and orchestrator aggregates, schemas, and the core packages that serve one service have moved, while store contracts stay at `submitqueue/extension/storage`. Domain-level `buildrunner`, `conflict`, and `speculation` have not moved, and `changeset` still declares its own store slice diff --git a/doc/rfc/scoped-resource-ids.md b/doc/rfc/scoped-resource-ids.md new file mode 100644 index 00000000..af745253 --- /dev/null +++ b/doc/rfc/scoped-resource-ids.md @@ -0,0 +1,73 @@ +# Scoped Sequential Resource IDs + +## Status + +Proposed. + +## Decision + +A generated resource ID is the positive `int64` returned by a durable counter scoped to `(owner domain, queue, resource kind)`. + +| Resource | Current ID | Proposed ID | Complete identity | +|---|---:|---:|---| +| SubmitQueue request | `demo-queue/42` | `42` | `(submitqueue, demo-queue, request, 42)` | +| SubmitQueue batch | `demo-queue/batch/7` | `7` | `(submitqueue, demo-queue, batch, 7)` | +| Stovepipe request | `request/monorepo/main/42` | `42` | `(stovepipe, monorepo/main, request, 42)` | + +The numeric ID is unique only within its scope. The same value may appear in another queue, resource kind, or domain. APIs and messages therefore carry the queue separately; their typed field or message type supplies the resource kind. + +Do not embed scope into the ID. Forms such as `demo-queue/42`, `demo-queue/batch/7`, `request.42`, and ARN-like resource names are not stored or accepted as IDs. + +## Counter + +The counter backend persists one high-water mark per `(owner domain, queue, resource kind)`. For example: + +```text +(submitqueue, demo-queue, request) -> 42 +(submitqueue, demo-queue, batch) -> 7 +(stovepipe, demo-queue, request) -> 11 +``` + +Controllers allocate an ID before creating the resource; stores accept the caller-supplied ID and never generate one. + +- The first ID is `1`; `0` is the unset value. +- Allocation is atomic across replicas and durable across restarts. +- Allocated values are never reused. Failed writes may leave gaps. +- Overflow fails instead of wrapping. +- Numeric order is allocation order only within the same scope. + +The counter contract requires an atomic durable increment, not MySQL specifically. MySQL remains the initial implementation. + +## Storage and contracts + +Resource tables store the numeric value directly. Queue remains the leading key: + +```text +request(queue, id BIGINT, ...) PRIMARY KEY (queue, id) +batch(queue, id BIGINT, ...) PRIMARY KEY (queue, id) +``` + +Reference columns use the same numeric type. Domain entities use distinct named types such as `RequestID` and `BatchID`, and protobuf resource fields use `int64`. + +This proposal applies only to counter-generated resources. Provider build IDs, message and hook IDs, change URIs, and content hashes keep their existing contracts. + +## URLs and display + +The decimal ID is used directly as one path segment: + +```text +/requests/42 +/batches/7 +``` + +The route supplies the resource kind; the request context supplies the queue. Queue URL design is separate. + +A UI may display `request.42`, `batch.7`, or `#42`, but those are derived labels, not identities. + +## Rejected alternatives + +- **Queue or kind prefixes:** duplicate explicit context, lengthen keys, require parsing, and introduce URL separators. +- **ARN-like names:** solve global lookup, which current APIs neither provide nor require. +- **UUIDs or a global counter:** provide global uniqueness at the cost of unnecessary encoding or coordination. +- **SQL auto-increment or `MAX(id) + 1`:** move allocation into one storage implementation or fail under concurrency. +- **Process-local counters:** reuse IDs after restart and collide across replicas. From d16ebc4323fe4f46e653b31767b15913de7debbf Mon Sep 17 00:00:00 2001 From: Preetam Dwivedi Date: Fri, 2 Oct 2026 12:58:05 -0700 Subject: [PATCH 2/2] docs: keep scoped resource IDs string-valued ## Summary ### Why? Resource identity should stay flexible at storage and API boundaries even when the current allocator produces sequential numbers. ### What? Define generated IDs as canonical decimal strings, keep resource and reference columns as VARCHAR, and limit integer storage to counter high-water marks. --- doc/rfc/scoped-resource-ids.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/doc/rfc/scoped-resource-ids.md b/doc/rfc/scoped-resource-ids.md index af745253..99122361 100644 --- a/doc/rfc/scoped-resource-ids.md +++ b/doc/rfc/scoped-resource-ids.md @@ -6,15 +6,15 @@ Proposed. ## Decision -A generated resource ID is the positive `int64` returned by a durable counter scoped to `(owner domain, queue, resource kind)`. +A generated resource ID is the canonical decimal string for a positive value returned by a durable counter scoped to `(owner domain, queue, resource kind)`. | Resource | Current ID | Proposed ID | Complete identity | |---|---:|---:|---| -| SubmitQueue request | `demo-queue/42` | `42` | `(submitqueue, demo-queue, request, 42)` | -| SubmitQueue batch | `demo-queue/batch/7` | `7` | `(submitqueue, demo-queue, batch, 7)` | -| Stovepipe request | `request/monorepo/main/42` | `42` | `(stovepipe, monorepo/main, request, 42)` | +| SubmitQueue request | `demo-queue/42` | `"42"` | `(submitqueue, demo-queue, request, "42")` | +| SubmitQueue batch | `demo-queue/batch/7` | `"7"` | `(submitqueue, demo-queue, batch, "7")` | +| Stovepipe request | `request/monorepo/main/42` | `"42"` | `(stovepipe, monorepo/main, request, "42")` | -The numeric ID is unique only within its scope. The same value may appear in another queue, resource kind, or domain. APIs and messages therefore carry the queue separately; their typed field or message type supplies the resource kind. +The decimal ID is unique only within its scope. The same value may appear in another queue, resource kind, or domain. APIs and messages therefore carry the queue separately; their typed field or message type supplies the resource kind. Do not embed scope into the ID. Forms such as `demo-queue/42`, `demo-queue/batch/7`, `request.42`, and ARN-like resource names are not stored or accepted as IDs. @@ -40,14 +40,14 @@ The counter contract requires an atomic durable increment, not MySQL specificall ## Storage and contracts -Resource tables store the numeric value directly. Queue remains the leading key: +Resource tables keep IDs as strings. Queue remains the leading key: ```text -request(queue, id BIGINT, ...) PRIMARY KEY (queue, id) -batch(queue, id BIGINT, ...) PRIMARY KEY (queue, id) +request(queue, id VARCHAR(...), ...) PRIMARY KEY (queue, id) +batch(queue, id VARCHAR(...), ...) PRIMARY KEY (queue, id) ``` -Reference columns use the same numeric type. Domain entities use distinct named types such as `RequestID` and `BatchID`, and protobuf resource fields use `int64`. +Reference columns use the same string type. Domain entities may use distinct named string types such as `RequestID` and `BatchID`; protobuf resource fields remain `string`. The counter backend may store its high-water marks as integers, and controllers convert allocated values to canonical decimal strings before creating resources. This proposal applies only to counter-generated resources. Provider build IDs, message and hook IDs, change URIs, and content hashes keep their existing contracts. @@ -69,5 +69,6 @@ A UI may display `request.42`, `batch.7`, or `#42`, but those are derived labels - **Queue or kind prefixes:** duplicate explicit context, lengthen keys, require parsing, and introduce URL separators. - **ARN-like names:** solve global lookup, which current APIs neither provide nor require. - **UUIDs or a global counter:** provide global uniqueness at the cost of unnecessary encoding or coordination. +- **Integer resource fields:** couple the persisted and wire contracts to the current counter representation without adding identity semantics. - **SQL auto-increment or `MAX(id) + 1`:** move allocation into one storage implementation or fail under concurrency. - **Process-local counters:** reuse IDs after restart and collide across replicas.