Repository navigation
fix(marketplace): ambiguous AWS errors and crashes between create and persist can leave a duplicate or a stuck pending row #525
Description
Activity
Plan for #525 (checked against current main e4c5cfc; no open PR or branch claims this; #554 and #582 touched only credential resolution and 422 term validation, not these paths).
Failure modes,
internal/api/handler_marketplace.go:- Random per-call ClientToken (
reserveAndCreateListing,uuid.New()at :291): STILL PRESENT. An ambiguous AWS error releases the claim (:297) so a retry creates a second live listing. - Crash between CreateMarketplaceListing and UpdatePurchaseHistoryListing (:306): STILL PRESENT for a hard process crash. The DB-failure case is ALREADY FIXED by fix(api): keep marketplace listing tracked when rollback cancel fails #506 (compensating cancel :308-315,
keepUncanceledListing:323). marketplaceCancelonly acceptsactive(:404): STILL PRESENT, pending rows cannot be recovered via the API.- Cancel succeeds on AWS, DB write fails (:424): STILL PRESENT, retry calls AWS cancel on a canceled listing.
- Empty AWS Status recorded as '' : STILL PRESENT (
UpdatePurchaseHistoryListingwrites result.State verbatim; claim treats '' as free). - Release after a successful cancel uses the request ctx (:314): PARTLY FIXED (the cancel and keepUncanceledListing use a detached ctx; the release at :314 still uses ctx).
AWS semantics (CreateReservedInstancesListing docs): ClientToken is a required idempotency token; a repeat with the same token returns the existing listing.
Design: derive the token deterministically from (purchase_id, previously recorded listing_id, count, price schedule) so a retry of the same attempt is idempotent on AWS's side while a deliberate re-list after cancel (listing_id changed) gets a fresh token. This needs no migration. The full fix (persist the token in a column, write the pending row before AWS, reconcile via DescribeReservedInstancesListings) needs a migration: next free number is 000026 (highest on main is 000025_admin_role_unique; no open PR adds a migration), e.g.
purchase_history.listing_client_token text NULL(no backfill; NULL for legacy rows; down drops the column).Split:
- PR 1 (~60 lines + tests, no migration): deterministic ClientToken. Fixes build: publish the self-hosted platform as a standalone repository #1 and makes the ambiguous-timeout retry idempotent.
- PR 2 (~250 lines, migration 000026): persist token before the AWS call; reconcile pending rows through DescribeReservedInstancesListings by token; fixes sec(iac): rename-proof OIDC deploy trust + branch-restricted environments for the split platform repo #2.
- PR 3 (~200 lines): allow cancel/recover for pending rows (sec(deps): clear Trivy alerts inherited from the monorepo (docker, otel/sdk, x/crypto openpgp) #3), cancel-then-DB-failure idempotence (sec(iac/aws): drop unconditioned account-wide KMS data-plane grant from deploy role #4), reject empty Status (sec(iac/azure): pull and push ACR with Entra identities, disable the admin account #5), detached ctx for release (ux(meta): residual UX backlog from 2026-04-22 audit (~75 MEDIUM/LOW items across 8 page areas) #6).
Open question for the owner: whether PR 2's reconcile runs inline on the next list/cancel request or in a background poller (the #292 status poller is referenced in comments). I will do PR 1 only now.
- Random per-call ClientToken (
Notes for PR2 (from review of #604): (1) the default price schedule is clock-derived, so a retry across a month boundary changes the derived token and AWS creates a duplicate; persisting the token before the AWS call removes the dependency. (2) row.ListingID is read before ClaimMarketplaceListingSlot, so a stale row in a concurrent request can reuse a canceled listing's token; PR2 should re-read the row after the claim or persist the token in the claim UPDATE.
Summary
Marketplace listing create and cancel can leave a duplicate listing or a row stuck in
pending, and operators cannot recover a stuck row through the API.Location (origin/main 496d9d7,
internal/api/handler_marketplace.go)reserveAndCreateListing(:221):ClientTokenisuuid.New()per call (:251), so it is not persisted and AWS cannot dedup a retry. An ambiguous AWS error (for example a timeout after AWS accepted the request) hitsreleaseMarketplaceClaim(:257), so the slot is free and a second listing is possible.CreateMarketplaceListingandUpdatePurchaseHistoryListing(:266) leaves the rowpendingwith no listing id.marketplaceCancel(:336) accepts onlyactive(:360), so apendingrow cannot be recovered through the API.marketplaceCancel(:371-381): AWS cancel succeeds, the DB write fails, the row staysactive, and a retry calls AWS on an already-cancelled listing.Statusas'', which the claim treats as free while a listing is live.releaseMarketplaceClaim(:329) after a successful cancel still uses the request context.Suggested direction
Persist the
ClientTokenbefore the AWS call and reconcile throughDescribeReservedInstancesListings.Context
Found in the #506 review (merged). Related: #335 (closed), #267, #261, #449.
Acceptance
A simulated timeout after AWS accepts the listing, and a simulated crash before persist, both recover to a single correct listing state without a duplicate, covered by tests.