Skip to content

feat: Added separate Database application. - #1230

Open
ShogunPanda wants to merge 22 commits into
supabase:masterfrom
ShogunPanda:watt-db-application
Open

feat: Added separate Database application.#1230
ShogunPanda wants to merge 22 commits into
supabase:masterfrom
ShogunPanda:watt-db-application

Conversation

@ShogunPanda

Copy link
Copy Markdown

This PR adds a separate Watt database application and routes PostgreSQL operations through Platformatic messaging when Watt messaging is available.

Main areas:

  1. Database Watt app
  2. Storage-to-database messaging integration
  3. Watt runtime config and acceptance coverage

Review Order

  1. watt.json
  2. watt.database.json
  3. src/database/index.ts
  4. src/database/pools.ts
  5. src/database/locks.ts
  6. src/internal/database/client.ts
  7. src/internal/database/watt-connection.ts
  8. acceptance/specs/database-watt.test.ts

High-Risk Areas

  1. WattPgTenantConnection and WattPgTransaction are cast to PgTenantConnection/PgTransaction, not actual instances. Review any instanceof PgTransaction usage.
  2. toPgQueryResult() returns empty command, fields, and oid: src/internal/database/watt-connection.ts:372. Any caller relying on those fields may regress.
  3. Pool global budget checks may race under concurrent acquisition: src/database/pools.ts:138.
  4. Acquire timeout races pool.connect(): src/database/pools.ts:57. Verify no delayed client leak.
  5. beginTransaction validation does not appear to validate isolationLevel/readOnly: src/database/validation.ts:5.
  6. Error responses carry stack traces across messaging: src/internal/database/watt-connection.ts:339. Confirm this is acceptable.
  7. watt.json uses schema/runtime 3.56.0, while Platformatic dependencies resolve from ^3.59.0.

Manual Review Focus

Check these behaviors carefully:

  1. Single-tenant Watt DB routing.
  2. Multitenant tenant resolution.
  3. Transaction commit, rollback, and savepoint behavior.
  4. Query cancellation during long-running SQL.
  5. Duplicate-key/Postgres error SQLSTATE mapping.
  6. Pool exhaustion and acquire timeout behavior.
  7. Vector operations, because transaction compatibility is a likely risk area.
  8. Shutdown while a transaction/query is active.

Assisted-By: OpenAI:GPT-5.6-Sol <openai/gpt-5.6-sol>

Signed-off-by: Paolo Insogna <paolo@cowtech.it>
Signed-off-by: Paolo Insogna <paolo@cowtech.it>
Signed-off-by: Paolo Insogna <paolo@cowtech.it>
Copilot AI review requested due to automatic review settings July 14, 2026 15:34
@ShogunPanda
ShogunPanda requested a review from a team as a code owner July 14, 2026 15:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a dedicated Watt database background application and routes PostgreSQL operations from the storage app through Platformatic messaging when Watt messaging is available, with tests and acceptance coverage to validate the messaging-based DB path.

Changes:

  • Adds Watt runtime configuration for running storage and database as separate applications.
  • Implements a Database Watt worker (pooling, destination resolution, locks, cancellation, limits, metrics) and a storage-side messaging adapter.
  • Adds unit tests plus a new Watt-managed acceptance runner and acceptance spec for Database Watt behavior.

Reviewed changes

Copilot reviewed 38 out of 40 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
watt.storage.json Defines the storage Watt app entrypoint for runtime-managed execution.
watt.json Converts to a Platformatic runtime config and registers storage + database applications.
watt.database.json Defines the database Watt app entrypoint as a background (no server) worker.
src/internal/database/watt-connection.ts Adds storage-side messaging adapter for tenant connections and transactions.
src/internal/database/watt-connection.test.ts Unit tests for messaging adapter behavior (queries, tx lifecycle, errors, cancellation).
src/internal/database/pg-connection.ts Introduces PgTransactionLike / PgTenantConnectionLike interfaces and aligns classes to them.
src/internal/database/multitenant-pg.ts Routes master executor work through Database Watt when messaging is available.
src/internal/database/multitenant-pg.test.ts Tests Watt routing selection for multitenant master queries/transactions.
src/internal/database/migrations/migrate.ts Documents that migrations intentionally remain direct PG (out of Database Watt scope).
src/internal/database/client.ts Routes getPostgresConnection() via Database Watt when messaging is available.
src/internal/database/client.test.ts Tests Watt routing and multitenant forwarded-host validation behavior.
src/database/index.ts Implements Database Watt app handlers and lifecycle (query/tx/locks/cancel/stats/shutdown).
src/database/config.ts Adds Database Watt configuration parsing from environment.
src/database/types.ts Defines Database Watt wire request/response types.
src/database/validation.ts Adds validation helpers for request envelopes, metadata bounds, and payload size.
src/database/errors.ts Defines Database Watt error contract and error serialization to wire responses.
src/database/destinations.ts Implements destination resolution for single-tenant and multitenant/master routing.
src/database/pools.ts Implements pool registry, acquisition limits, eviction, and query execution.
src/database/locks.ts Implements lock registry to pin connections for tx/locked queries and serialize per-lock work.
src/database/cancellation.ts Implements cancellation tracking and PG backend cancel signaling for in-flight work.
src/database/result-limits.ts Enforces row/byte result limits for messaging responses.
src/database/ssl.ts Implements sslmode-derived TLS settings for database connections.
src/database/metrics.ts Registers observable gauges for pool/connection stats via OpenTelemetry.
src/database/pg-lib-connection.d.ts Declares types for pg/lib/connection used by cancellation support.
src/database/tests/index.test.ts Tests Database Watt handlers end-to-end with loopback messaging and mocked pg.
src/database/tests/validation.test.ts Tests request validation bounds and error behavior.
src/database/tests/ssl.test.ts Tests SSL settings behavior for common sslmode scenarios.
src/database/tests/result-limits.test.ts Tests result size/row enforcement behavior.
src/database/tests/locks.test.ts Tests lock registry query/tx behaviors and serialization.
src/database/tests/errors.test.ts Tests error serialization/mapping contract.
src/database/tests/destinations.test.ts Tests single-tenant destination/external pool resolution.
src/database/tests/cancellation.test.ts Tests cancellation registry behavior and PG cancel wiring.
src/admin-app.ts Switches Watt-detection to hasField('applicationId').
src/admin-app.test.ts Updates admin app tests to use globals update/remove patterns instead of getGlobal mocking.
docs/database-watt-postgresql-scope.md Documents what DB access is routed through Database Watt vs intentionally left direct.
acceptance/specs/database-watt.test.ts Adds acceptance tests validating Database Watt query/tx/cancel/error and multitenant behaviors.
acceptance/scripts/run-managed-watt-local.ts Adds a Watt-managed local acceptance runner that boots both apps and runs acceptance.
acceptance/scripts/run-managed-local.ts Explicitly disables Database Watt acceptance when running the non-Watt managed script.
package.json Adds acceptance:watt script and @platformatic/runtime devDependency.
package-lock.json Locks @platformatic/runtime addition.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/database/pools.ts Outdated
Comment on lines +61 to +79
async acquire(destination: DestinationConfig): Promise<PoolClient> {
const entry = this.getOrCreatePool(destination)
this.assertCanAcquire(entry)
this.pendingGlobalAcquisitions++

const timeout = createTimeout(this.config.acquireTimeoutMs)
try {
return await Promise.race([
entry.pool.connect(),
timeout.promise.then(() => {
throw new DatabaseWattError('ACQUIRE_TIMEOUT', 'Timed out acquiring database connection')
}),
])
} finally {
timeout.clear()
this.pendingGlobalAcquisitions--
entry.lastUsedAt = Date.now()
}
}
Comment thread src/database/index.ts
Comment on lines +289 to +291
if (request.readOnly) {
modes.push('READ ONLY')
}
Comment on lines +38 to +42
export async function getWattPostgresConnection(
options: TenantConnectionOptions
): Promise<PgTenantConnection> {
return new WattPgTenantConnection(options) as unknown as PgTenantConnection
}
Comment on lines +156 to +165
async beginTransaction(options?: TransactionOptions): Promise<PgTransaction> {
const response = await sendDatabaseMessage<{ lockId: string }>('database.beginTransaction', {
destination: this.destination,
isolationLevel: options?.isolation,
operationName: this.operation?.(),
readOnly: options?.readOnly,
})

return new WattPgTransaction(response.lockId, this.operation) as unknown as PgTransaction
}
Comment on lines +54 to +66
export function toErrorResponse(error: unknown, context: ErrorContext = {}): DatabaseErrorResponse {
if (error instanceof DatabaseWattError) {
return {
code: error.code,
message: error.message,
requestId: context.requestId,
operationName: context.operationName,
destination: context.destination,
lockId: context.lockId,
sqlState: error.sqlState,
stack: error.stack,
connectionDiscarded: error.connectionDiscarded,
}
Signed-off-by: ferhat elmas <elmas.ferhat@gmail.com>
Signed-off-by: ferhat elmas <elmas.ferhat@gmail.com>
Signed-off-by: ferhat elmas <elmas.ferhat@gmail.com>
@coveralls

coveralls commented Jul 15, 2026

Copy link
Copy Markdown

Coverage Report for CI Build 30076416950

Coverage decreased (-0.1%) to 80.268%

Details

  • Coverage decreased (-0.1%) from the base build.
  • Patch coverage: 107 uncovered changes across 14 files (527 of 634 lines covered, 83.12%).
  • No coverage regressions found.

Uncovered Changes

Top 10 Files by Coverage Impact Changed Covered %
src/applications/database/locks.ts 85 52 61.18%
src/applications/database/pools.ts 92 71 77.17%
src/internal/database/watt/connection.ts 72 58 80.56%
src/applications/database/application.ts 132 121 91.67%
src/internal/database/watt/client.ts 54 46 85.19%
src/applications/database/metrics.ts 15 9 60.0%
src/applications/database/validation.ts 30 26 86.67%
src/internal/database/postgres/cancellation.ts 33 30 90.91%
src/applications/database/errors.ts 15 13 86.67%
src/http/plugins/vector.ts 2 1 50.0%
Total (22 files) 634 527 83.12%

Coverage Regressions

No coverage regressions found.


Coverage Stats

Coverage Status
Relevant Lines: 13430
Covered Lines: 11235
Line Coverage: 83.66%
Relevant Branches: 7870
Covered Branches: 5862
Branch Coverage: 74.49%
Branches in Coverage %: Yes
Coverage Strength: 420.13 hits per line

💛 - Coveralls

@ferhatelmas ferhatelmas self-assigned this Jul 15, 2026
# Conflicts:
#	acceptance/specs/admin.test.ts
#	src/internal/database/pg-connection.ts
Signed-off-by: ferhat elmas <elmas.ferhat@gmail.com>
Signed-off-by: ferhat elmas <elmas.ferhat@gmail.com>
Signed-off-by: ferhat elmas <elmas.ferhat@gmail.com>
Signed-off-by: ferhat elmas <elmas.ferhat@gmail.com>
Signed-off-by: ferhat elmas <elmas.ferhat@gmail.com>
Signed-off-by: ferhat elmas <elmas.ferhat@gmail.com>
Signed-off-by: ferhat elmas <elmas.ferhat@gmail.com>
Signed-off-by: ferhat elmas <elmas.ferhat@gmail.com>
# Conflicts:
#	Dockerfile
#	src/admin-app.test.ts
#	src/admin-app.ts
#	src/http/plugins/db.ts
#	src/internal/database/multitenant-pg.ts
#	src/internal/database/pg-connection.ts
#	src/storage/database/adapter.ts
#	src/storage/database/pg.ts
#	watt.json
#	watt.storage.json
Signed-off-by: ferhat elmas <elmas.ferhat@gmail.com>
Signed-off-by: ferhat elmas <elmas.ferhat@gmail.com>
Signed-off-by: ferhat elmas <elmas.ferhat@gmail.com>
Signed-off-by: ferhat elmas <elmas.ferhat@gmail.com>
Signed-off-by: ferhat elmas <elmas.ferhat@gmail.com>
this.role = options.user.payload.role || 'anon'
this.executor = new WattPgExecutor(
{
connectionString: options.dbUrl,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Severity: MEDIUM

The full database connection URL—including embedded credentials (username and password)—is placed into the DatabasePoolTarget.connectionString field and transmitted with every database.query, database.acquire, database.beginTransaction, and database.lockedQuery message over the inter-process messaging bus. In multi-tenant deployments each tenant's individual DB URL is serialized per operation. If the Platformatic messaging layer emits debug-level traces or if monitoring captures message payloads, database credentials will be logged for every DB operation.
Helpful? Add 👍 / 👎

💡 Fix Suggestion

Suggestion: Replace the raw connectionString (full database URL with embedded credentials) in DatabasePoolTarget with an opaque pool identifier so that database credentials never travel over the inter-process messaging bus.

The recommended approach is:

  1. Add a pool-registration message (e.g., database.registerPool) to protocol.ts that accepts the connectionString once (during application startup or first use) and returns a stable, opaque poolId (e.g., a UUID or a hash). The database application side stores the mapping of poolId → connectionString in memory.

  2. Update DatabasePoolTarget in protocol.ts to replace connectionString: string with poolId: string.

  3. Update WattPgTenantConnection (around line 147–152 in connection.ts) to call the new database.registerPool message on first use and cache the returned poolId keyed by the connection URL. Pass the poolId instead of the raw connectionString into DatabasePoolTarget.

  4. Update PoolRegistry in pools.ts and the application handlers in application.ts to resolve a poolId back to the actual connectionString using the in-memory registry before creating or reusing a pool.

This ensures credentials are transmitted exactly once (during pool registration, over an already-trusted IPC channel) and all subsequent database.query, database.acquire, and database.beginTransaction messages carry only the opaque poolId, eliminating credential exposure in every operation's message payload.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants