diff --git a/CHANGELOG.md b/CHANGELOG.md index a0118484..c13ef3c9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -227,9 +227,46 @@ under the pre-1.0 conventions in [VERSIONING.md](docs/governance/VERSIONING.md): `usageHistory.negativeEvents` and `usageHistory.operatingConditions[].note`, and textile `repairHistoryUrl`. This node never reads what a field says. - Not yet done: the erasable store for the records a `held: outside` statement - names, and the route that erases one. Until they exist an operator keeps - those records elsewhere and names them. +- **The records a `held: outside` statement names have a store of their own, and + an admin credential can erase one.** A published passport is signed, frozen, + archived and copied to the back-up provider, so personal data it is not + required to carry is held in a record that can be erased instead, and a + withdrawn consent (GDPR Art. 7(3)) obliges erasure (Art. 17(1)(b)). Core's + `PersonalDataPort` is now implemented on PostgreSQL in its own table, + `odal.personal_data_record` (migration 0043). It is not a column of the + passport, and the archive of past versions, the audit chain, the snapshots, the + outboxes and the back-up copy never see it. No route reads a record. + + `DELETE /vault/api/v1/personal-data/{record}` erases one and answers the + receipt: the record, the passport and field it belonged to, and when it was + erased. Erasing a record already erased answers the first receipt again, so a + caller that lost the answer asks again. A record this node never held is `404`. + It needs an admin credential, like every other route that removes something. + It does not touch the passport or the statement in it, which keeps naming the + record, so a reader learns that the record is gone rather than that it never + existed. The store lists every record a passport has, erased ones included, so a + record whose identifier was lost on the way back from `store` can still be + found through the port. + + **What a tombstone keeps, and why it needs a basis.** Erasure drops the + content and keeps the row: the identifier, the passport and field, and the + instant. None of the content, but a passport identifier can relate to whoever + owns the item, and information that can be linked to a person is personal data + (GDPR Recital 26). So the tombstone is protected like the passport's + individual-item data, and the operator needs a basis for keeping it. Showing + that it erased (Art. 5(2)) is one. Nothing here deletes it, and the database + role cannot. Erasure overwrites the content in place; the database's own + write-ahead log, replicas and backups hold earlier row versions, and how long + those live is the operator's retention. + + Not yet done: no route stores a record, so nothing in this node calls `store` + or lists a passport's records. An operator who keeps records in this store + writes them through the port. + + `POST /dpp/validate` now runs the same personal-data check create runs, on the + draft create would store, and so refuses a field with a value and no statement + where it used to pass. Because it now builds that draft, it also refuses a + malformed `commodityCode`, which create refused and the preview passed. - **The GS1 resolver accepts a consumer product variant (AI 22) and a third-party extension (AI 235).** A label printed elsewhere may carry either diff --git a/CLAUDE.md b/CLAUDE.md index 3b06f983..c938c42c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -471,6 +471,7 @@ unprotected one. Read the table in the code. | POST | `/vault/api/v1/unsold-goods` | Bearer **(write)** | Record an ESPR Art. 24 disclosure line for unsold consumer products discarded in a financial year. Not a passport — Art. 24's subject is an operator over a year. `exemptDestruction` requires a justification (Art. 25 bans the destruction outright from 19 July 2026); any other destination refuses one | | GET | `/vault/api/v1/unsold-goods` | Bearer (admin) | List those lines, newest first, optionally filtered by financial year | | POST | `/vault/api/v1/credentials` | Bearer (admin) | Issue a DPP access credential signed with this node's key. **Legitimate-interest roles only** — an authority's standing is conferred by a member state, so the three authority roles are refused. No revocation list is published, so the lifetime is capped | +| DELETE | `/vault/api/v1/personal-data/{record}` | Bearer (admin) | Erase a record a passport's `personalData` statement names, keeping a tombstone, and answer the receipt. Idempotent: a retry gets the first receipt; `404` for a record this node never held. Touches neither the passport nor its statement. **No route reads a record** — they are in `odal.personal_data_record` (migration 0043) only, never in the passport, archive or back-up copy | | POST | `/vault/api/v1/plugins` | Bearer (admin) | Install a **signed** product group plugin and hot-swap it | | POST | `/vault/api/v1/ruleset/reload` | Bearer (admin) | Re-read the **signed** compliance-ruleset channel and hot-swap a verified bundle | | GET | `/vault/api/v1/webhooks` | Bearer (admin) | List webhook subscriptions | diff --git a/api/components/schemas/passport-reports/PersonalDataErasureReceipt.yaml b/api/components/schemas/passport-reports/PersonalDataErasureReceipt.yaml new file mode 100644 index 00000000..6c895622 --- /dev/null +++ b/api/components/schemas/passport-reports/PersonalDataErasureReceipt.yaml @@ -0,0 +1,33 @@ +type: object +description: |- + What is left of an erased record of personal data: which record it was, which + passport and field it related to, and when it was erased. None of the erased + content. + + The tombstone is not free of personal data. A passport identifier can relate to + whoever owns the item (GDPR Recital 26), so the receipt is as sensitive as the + passport's individual-item data and is kept only as long as the operator has a + basis to keep it. +required: + - record + - passportId + - field + - erasedAt +properties: + record: + type: string + description: The erased record's identifier, as the passport's statement names it. + passportId: + type: string + format: uuid + description: The passport the record related to. + field: + type: string + description: >- + The marked field the record related to, as a dotted path inside + `productGroupData`. + example: usageHistory.negativeEvents + erasedAt: + type: string + format: date-time + description: When the record's content was erased. diff --git a/api/openapi.bundled.json b/api/openapi.bundled.json index 01995bf5..195f4355 100644 --- a/api/openapi.bundled.json +++ b/api/openapi.bundled.json @@ -70,6 +70,10 @@ "name": "Webhooks", "description": "Signed outbound event delivery — subscribe, list, remove, test." }, + { + "name": "Personal Data", + "description": "Erasing the records a passport's `personalData` statements name. A record is held outside the passport so it can be erased, and no route reads one." + }, { "name": "Plugins", "description": "Signed product-group plugin hot-install — verify, persist, hot-swap (admin-only)." @@ -215,7 +219,7 @@ "post": { "operationId": "validateDpp", "summary": "Dry-run a passport body without creating it", - "description": "Runs exactly the validation `POST /vault/api/v1/dpp` runs, and persists\nnothing. The same code path serves both, so a preview and the real create\ncannot disagree.\n\n**Two verdicts, because create and publish deliberately differ.** Create is\nlenient about a product group with no resolvable JSON Schema — a draft is allowed\nto be incomplete — while publish fails closed on it, since a signed\npassport must have passed a real schema check. A body can therefore be\ncreatable but not yet clear the publish-time schema gate, and that gap is\nreported rather than hidden until publish is attempted.\n\nA body that create **would reject** gets back the identical rejection create\nwould have returned, not a paraphrase of it: a `422`, or a `409` for a\n`carrierSerial` another passport already holds. A body create would accept\nreturns `200` with the two verdicts.\n\n**`productGroupDataValid` is not a publish verdict.** It reports one of publish's\npreconditions — the product group-data gates — and publish applies others this\nroute does not run:\n\n- the **registry identity** requirement (a default facility and a primary\n operator identifier), which needs operator state this route never reads;\n- the **category-mandatory content** gate, which is reachable only by\n attempting the lifecycle transition, so it cannot be previewed;\n- the **compliance** gate, which needs a `placedOnMarketDate` and a stored\n passport.\n\nA `productGroupDataValid: true` therefore means \"this body clears the schema\ngate\", never \"publish will succeed\". The field was called `publishValid`\nand was renamed because that name promised the latter.\n", + "description": "Runs exactly the validation `POST /vault/api/v1/dpp` runs, and persists\nnothing. The same code path serves both, so a preview and the real create\ncannot disagree.\n\n**Two verdicts, because create and publish deliberately differ.** Create is\nlenient about a product group with no resolvable JSON Schema — a draft is allowed\nto be incomplete — while publish fails closed on it, since a signed\npassport must have passed a real schema check. A body can therefore be\ncreatable but not yet clear the publish-time schema gate, and that gap is\nreported rather than hidden until publish is attempted.\n\nA body that create **would reject** gets back the identical rejection create\nwould have returned, not a paraphrase of it: a `422`, or a `409` for a\n`carrierSerial` another passport already holds. A body create would accept\nreturns `200` with the two verdicts.\n\nThe `422` includes the personal-data check. A field the product group's\nschema marks as able to hold personal data that has a value and no\n`personalData` statement, a statement about a field that is not marked, an\n`outside` statement naming no record, and a basis a governing act does not\nadmit are all refused here as they are on create, each naming its field. The\ncheck reads no store, so it is part of the preview. It never reads what a\nfield says.\n\n**`productGroupDataValid` is not a publish verdict.** It reports one of publish's\npreconditions — the product group-data gates — and publish applies others this\nroute does not run:\n\n- the **registry identity** requirement (a default facility and a primary\n operator identifier), which needs operator state this route never reads;\n- the **category-mandatory content** gate, which is reachable only by\n attempting the lifecycle transition, so it cannot be previewed;\n- the **compliance** gate, which needs a `placedOnMarketDate` and a stored\n passport.\n\nA `productGroupDataValid: true` therefore means \"this body clears the schema\ngate\", never \"publish will succeed\". The field was called `publishValid`\nand was renamed because that name promised the latter.\n", "tags": [ "DPP Management" ], @@ -2815,6 +2819,64 @@ } } }, + "/vault/api/v1/personal-data/{record}": { + "delete": { + "operationId": "erasePersonalDataRecord", + "summary": "Erase a record of personal data", + "description": "Erases a record that a passport's `personalData` statement names\n(`{\"held\": \"outside\", \"record\": \"\"}`), keeping a tombstone. This is how\na withdrawn consent (GDPR Art. 7(3)) is honoured: a published passport is\nsigned, frozen, archived and copied to the back-up provider, so the personal\ndata is held in a record that can be erased instead.\n\n**Erasing touches nothing else.** Not the passport, which is signed and\nfrozen. Not the statement in it, which keeps naming the record, so a reader\nfollowing the statement learns that the record is gone rather than that it\nnever existed. Not any other record.\n\n**Idempotent.** Erasing a record that is already erased returns the receipt\nit was given the first time, so a caller that lost the answer asks again. It\ntakes no `Idempotency-Key`.\n\n**The tombstone is not free of personal data.** It keeps the identifier, the\npassport and field the record belonged to, and when it was erased — none of\nthe content. A passport identifier can relate to whoever owns the item, and\ninformation that can be linked to a person is personal data (GDPR Recital\n26). The receipt is therefore as sensitive as the passport's individual-item\ndata, and the operator needs a basis for keeping the tombstone; to show that\nit erased (Art. 5(2)) is one. Nothing in this node deletes it.\n\n**No route reads a record.** A record is served by no resolver, public,\ncredentialed or authenticated view of a passport, and is never part of the\narchive of past versions or of the back-up copy. Serving one is a disclosure\nthat needs a basis of its own, which only the controller can establish for\neach purpose.\n\nAdmin-scoped, like every other route that removes something: the write scope\nproduces and moves passports, and none of that destroys anything.\n", + "tags": [ + "Personal Data" + ], + "security": [ + { + "BearerApiKey": [] + }, + { + "BasicAuth": [] + } + ], + "parameters": [ + { + "name": "record", + "in": "path", + "required": true, + "description": "The record's identifier, as the passport's statement names it (`personalData..record`).", + "schema": { + "type": "string", + "minLength": 1 + } + } + ], + "responses": { + "200": { + "description": "The record is erased. The same receipt on every call after the first.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/PersonalDataErasureReceipt" + } + } + } + }, + "401": { + "$ref": "#/components/responses/Unauthorized" + }, + "403": { + "$ref": "#/components/responses/Forbidden" + }, + "404": { + "description": "This node never held a record with that identifier, which includes a string that could not have been one. A record that was held and is erased is not a `404`: it answers `200` with its receipt.", + "content": { + "application/problem+json": { + "schema": { + "$ref": "#/components/schemas/Problem" + } + } + } + } + } + } + }, "/vault/api/v1/unsold-goods": { "get": { "operationId": "listUnsoldGoods", @@ -6639,6 +6701,37 @@ } } }, + "PersonalDataErasureReceipt": { + "type": "object", + "description": "What is left of an erased record of personal data: which record it was, which\npassport and field it related to, and when it was erased. None of the erased\ncontent.\n\nThe tombstone is not free of personal data. A passport identifier can relate to\nwhoever owns the item (GDPR Recital 26), so the receipt is as sensitive as the\npassport's individual-item data and is kept only as long as the operator has a\nbasis to keep it.", + "required": [ + "record", + "passportId", + "field", + "erasedAt" + ], + "properties": { + "record": { + "type": "string", + "description": "The erased record's identifier, as the passport's statement names it." + }, + "passportId": { + "type": "string", + "format": "uuid", + "description": "The passport the record related to." + }, + "field": { + "type": "string", + "description": "The marked field the record related to, as a dotted path inside `productGroupData`.", + "example": "usageHistory.negativeEvents" + }, + "erasedAt": { + "type": "string", + "format": "date-time", + "description": "When the record's content was erased." + } + } + }, "ValidateResponse": { "type": "object", "description": "The dry-run verdict. Two booleans rather than one, because create and publish deliberately differ: a body can be creatable as a draft and not yet publishable, and collapsing that into a single flag would hide the gap until the caller tried to publish.", diff --git a/api/openapi.bundled.yaml b/api/openapi.bundled.yaml index 70a6d9ff..9f8e2f4b 100644 --- a/api/openapi.bundled.yaml +++ b/api/openapi.bundled.yaml @@ -60,6 +60,8 @@ tags: description: API key management — create, list, revoke. - name: Webhooks description: Signed outbound event delivery — subscribe, list, remove, test. + - name: Personal Data + description: Erasing the records a passport's `personalData` statements name. A record is held outside the passport so it can be erased, and no route reads one. - name: Plugins description: Signed product-group plugin hot-install — verify, persist, hot-swap (admin-only). - name: Ruleset @@ -185,6 +187,14 @@ paths: `carrierSerial` another passport already holds. A body create would accept returns `200` with the two verdicts. + The `422` includes the personal-data check. A field the product group's + schema marks as able to hold personal data that has a value and no + `personalData` statement, a statement about a field that is not marked, an + `outside` statement naming no record, and a basis a governing act does not + admit are all refused here as they are on create, each naming its field. The + check reads no store, so it is part of the preview. It never reads what a + field says. + **`productGroupDataValid` is not a publish verdict.** It reports one of publish's preconditions — the product group-data gates — and publish applies others this route does not run: @@ -2246,6 +2256,72 @@ paths: application/problem+json: schema: $ref: '#/components/schemas/Problem' + /vault/api/v1/personal-data/{record}: + delete: + operationId: erasePersonalDataRecord + summary: Erase a record of personal data + description: | + Erases a record that a passport's `personalData` statement names + (`{"held": "outside", "record": ""}`), keeping a tombstone. This is how + a withdrawn consent (GDPR Art. 7(3)) is honoured: a published passport is + signed, frozen, archived and copied to the back-up provider, so the personal + data is held in a record that can be erased instead. + + **Erasing touches nothing else.** Not the passport, which is signed and + frozen. Not the statement in it, which keeps naming the record, so a reader + following the statement learns that the record is gone rather than that it + never existed. Not any other record. + + **Idempotent.** Erasing a record that is already erased returns the receipt + it was given the first time, so a caller that lost the answer asks again. It + takes no `Idempotency-Key`. + + **The tombstone is not free of personal data.** It keeps the identifier, the + passport and field the record belonged to, and when it was erased — none of + the content. A passport identifier can relate to whoever owns the item, and + information that can be linked to a person is personal data (GDPR Recital + 26). The receipt is therefore as sensitive as the passport's individual-item + data, and the operator needs a basis for keeping the tombstone; to show that + it erased (Art. 5(2)) is one. Nothing in this node deletes it. + + **No route reads a record.** A record is served by no resolver, public, + credentialed or authenticated view of a passport, and is never part of the + archive of past versions or of the back-up copy. Serving one is a disclosure + that needs a basis of its own, which only the controller can establish for + each purpose. + + Admin-scoped, like every other route that removes something: the write scope + produces and moves passports, and none of that destroys anything. + tags: + - Personal Data + security: + - BearerApiKey: [] + - BasicAuth: [] + parameters: + - name: record + in: path + required: true + description: The record's identifier, as the passport's statement names it (`personalData..record`). + schema: + type: string + minLength: 1 + responses: + '200': + description: The record is erased. The same receipt on every call after the first. + content: + application/json: + schema: + $ref: '#/components/schemas/PersonalDataErasureReceipt' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '404': + description: 'This node never held a record with that identifier, which includes a string that could not have been one. A record that was held and is erased is not a `404`: it answers `200` with its receipt.' + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' /vault/api/v1/unsold-goods: get: operationId: listUnsoldGoods @@ -5586,6 +5662,38 @@ components: skip: type: integer description: The offset this page starts at. + PersonalDataErasureReceipt: + type: object + description: |- + What is left of an erased record of personal data: which record it was, which + passport and field it related to, and when it was erased. None of the erased + content. + + The tombstone is not free of personal data. A passport identifier can relate to + whoever owns the item (GDPR Recital 26), so the receipt is as sensitive as the + passport's individual-item data and is kept only as long as the operator has a + basis to keep it. + required: + - record + - passportId + - field + - erasedAt + properties: + record: + type: string + description: The erased record's identifier, as the passport's statement names it. + passportId: + type: string + format: uuid + description: The passport the record related to. + field: + type: string + description: The marked field the record related to, as a dotted path inside `productGroupData`. + example: usageHistory.negativeEvents + erasedAt: + type: string + format: date-time + description: When the record's content was erased. ValidateResponse: type: object description: 'The dry-run verdict. Two booleans rather than one, because create and publish deliberately differ: a body can be creatable as a draft and not yet publishable, and collapsing that into a single flag would hide the gap until the caller tried to publish.' diff --git a/api/openapi.yaml b/api/openapi.yaml index fed76284..dec86476 100644 --- a/api/openapi.yaml +++ b/api/openapi.yaml @@ -154,6 +154,8 @@ components: $ref: components/schemas/passport-requests/TransferReason.yaml PassportListResponse: $ref: components/schemas/passport-reports/PassportListResponse.yaml + PersonalDataErasureReceipt: + $ref: components/schemas/passport-reports/PersonalDataErasureReceipt.yaml ValidateResponse: $ref: components/schemas/passport-reports/ValidateResponse.yaml PassportAuditEntry: @@ -415,6 +417,8 @@ paths: $ref: paths/vault/vault_api_v1_api-keys_{id}.yaml /vault/api/v1/credentials: $ref: paths/vault/vault_api_v1_credentials.yaml + /vault/api/v1/personal-data/{record}: + $ref: paths/vault/vault_api_v1_personal-data_{record}.yaml /vault/api/v1/unsold-goods: $ref: paths/vault/vault_api_v1_unsold-goods.yaml /vault/api/v1/webhooks: @@ -532,6 +536,10 @@ tags: description: API key management — create, list, revoke. - name: Webhooks description: Signed outbound event delivery — subscribe, list, remove, test. + - name: Personal Data + description: >- + Erasing the records a passport's `personalData` statements name. A record + is held outside the passport so it can be erased, and no route reads one. - name: Plugins description: >- Signed product-group plugin hot-install — verify, persist, hot-swap diff --git a/api/paths/vault/vault_api_v1_dpp_validate.yaml b/api/paths/vault/vault_api_v1_dpp_validate.yaml index 6e46053e..c8cffbf0 100644 --- a/api/paths/vault/vault_api_v1_dpp_validate.yaml +++ b/api/paths/vault/vault_api_v1_dpp_validate.yaml @@ -18,6 +18,14 @@ post: `carrierSerial` another passport already holds. A body create would accept returns `200` with the two verdicts. + The `422` includes the personal-data check. A field the product group's + schema marks as able to hold personal data that has a value and no + `personalData` statement, a statement about a field that is not marked, an + `outside` statement naming no record, and a basis a governing act does not + admit are all refused here as they are on create, each naming its field. The + check reads no store, so it is part of the preview. It never reads what a + field says. + **`productGroupDataValid` is not a publish verdict.** It reports one of publish's preconditions — the product group-data gates — and publish applies others this route does not run: diff --git a/api/paths/vault/vault_api_v1_personal-data_{record}.yaml b/api/paths/vault/vault_api_v1_personal-data_{record}.yaml new file mode 100644 index 00000000..fde8df8d --- /dev/null +++ b/api/paths/vault/vault_api_v1_personal-data_{record}.yaml @@ -0,0 +1,71 @@ +delete: + operationId: erasePersonalDataRecord + summary: Erase a record of personal data + description: | + Erases a record that a passport's `personalData` statement names + (`{"held": "outside", "record": ""}`), keeping a tombstone. This is how + a withdrawn consent (GDPR Art. 7(3)) is honoured: a published passport is + signed, frozen, archived and copied to the back-up provider, so the personal + data is held in a record that can be erased instead. + + **Erasing touches nothing else.** Not the passport, which is signed and + frozen. Not the statement in it, which keeps naming the record, so a reader + following the statement learns that the record is gone rather than that it + never existed. Not any other record. + + **Idempotent.** Erasing a record that is already erased returns the receipt + it was given the first time, so a caller that lost the answer asks again. It + takes no `Idempotency-Key`. + + **The tombstone is not free of personal data.** It keeps the identifier, the + passport and field the record belonged to, and when it was erased — none of + the content. A passport identifier can relate to whoever owns the item, and + information that can be linked to a person is personal data (GDPR Recital + 26). The receipt is therefore as sensitive as the passport's individual-item + data, and the operator needs a basis for keeping the tombstone; to show that + it erased (Art. 5(2)) is one. Nothing in this node deletes it. + + **No route reads a record.** A record is served by no resolver, public, + credentialed or authenticated view of a passport, and is never part of the + archive of past versions or of the back-up copy. Serving one is a disclosure + that needs a basis of its own, which only the controller can establish for + each purpose. + + Admin-scoped, like every other route that removes something: the write scope + produces and moves passports, and none of that destroys anything. + tags: + - Personal Data + security: + - BearerApiKey: [] + - BasicAuth: [] + parameters: + - name: record + in: path + required: true + description: >- + The record's identifier, as the passport's statement names it + (`personalData..record`). + schema: + type: string + minLength: 1 + responses: + '200': + description: >- + The record is erased. The same receipt on every call after the first. + content: + application/json: + schema: + $ref: ../../components/schemas/passport-reports/PersonalDataErasureReceipt.yaml + '401': + $ref: ../../components/responses/Unauthorized.yaml + '403': + $ref: ../../components/responses/Forbidden.yaml + '404': + description: >- + This node never held a record with that identifier, which includes a + string that could not have been one. A record that was held and is erased + is not a `404`: it answers `200` with its receipt. + content: + application/problem+json: + schema: + $ref: ../../components/schemas/errors/Problem.yaml diff --git a/crates/dpp-dal/src/pg/mod.rs b/crates/dpp-dal/src/pg/mod.rs index 60c2bcfb..3fba9431 100644 --- a/crates/dpp-dal/src/pg/mod.rs +++ b/crates/dpp-dal/src/pg/mod.rs @@ -18,6 +18,7 @@ pub mod repo_idempotency; pub mod repo_operator_config; pub mod repo_passport; pub mod repo_passport_version; +pub mod repo_personal_data; pub mod repo_registry_identity; pub mod repo_registry_sync; pub mod repo_registry_transfer; @@ -40,6 +41,7 @@ pub use repo_idempotency::PgIdempotencyRepo; pub use repo_operator_config::PgOperatorConfigRepo; pub use repo_passport::PgPassportRepo; pub use repo_passport_version::PgPassportVersionRepo; +pub use repo_personal_data::PgPersonalDataRepo; pub use repo_registry_identity::PgRegistryIdentityRepo; pub use repo_registry_sync::PgRegistrySyncRepo; pub use repo_registry_transfer::PgRegistryTransferRepo; diff --git a/crates/dpp-dal/src/pg/repo_personal_data.rs b/crates/dpp-dal/src/pg/repo_personal_data.rs new file mode 100644 index 00000000..dcd16151 --- /dev/null +++ b/crates/dpp-dal/src/pg/repo_personal_data.rs @@ -0,0 +1,206 @@ +//! `PersonalDataPort` on PostgreSQL (`ops/pg/0043`). +//! +//! The holder of the records a passport's `personalData` statement names. Core +//! states the contract (`dpp_domain::ports::personal_data`); this is the part +//! that is about where the records live. +//! +//! # Its own table, and nothing else's neighbour +//! +//! `odal.personal_data_record` is not a column of `odal.passport`, and no query +//! that builds a passport view, an archived version, a snapshot, an audit entry +//! or a back-up copy reads it. A record lodged with any of those could not be +//! erased with this one, which is the reason the store exists at all. +//! +//! # Erasure keeps the row +//! +//! Erasing is an `UPDATE` that drops `content` and stamps `erased_at`, not a +//! `DELETE`: the row is the tombstone, and the app role has no `DELETE` on this +//! table. The tombstone is not free of personal data — the passport identifier +//! can relate to whoever owns the item — so it is kept only as long as the +//! controller has a basis to keep it, and this table has no sweep that decides +//! when that is. +//! +//! The retry rule needs no lock beyond the row's own. Two concurrent erasures of +//! one record both run the `UPDATE`; the second waits for the first's commit, +//! re-reads the row, finds it erased, and matches nothing. It then reads the +//! receipt the first one wrote, so both callers are handed the same one. + +use async_trait::async_trait; +use chrono::{DateTime, Utc}; +use sqlx::Row; +use uuid::Uuid; + +use dpp_domain::DppError; +use dpp_domain::passport::PassportId; +use dpp_domain::personal_data::PersonalDataRecordId; +use dpp_domain::ports::personal_data::{ + ErasureReceipt, HeldRecord, PersonalDataPort, PersonalDataRecord, +}; + +use super::{PgDal, db_err}; + +/// PostgreSQL implementation of core's [`PersonalDataPort`]. +pub struct PgPersonalDataRepo { + dal: PgDal, +} + +impl PgPersonalDataRepo { + /// Construct a repo sharing the given pool handle. + pub fn new(dal: PgDal) -> Self { + Self { dal } + } +} + +/// An identifier this store minted is a UUID, so anything else was never minted +/// here: it names no record, and is answered the way an unknown UUID is. +fn parse(record: &PersonalDataRecordId) -> Option { + Uuid::parse_str(record.as_str()).ok() +} + +fn unknown(record: &PersonalDataRecordId) -> DppError { + DppError::NotFound(format!("no personal data record {record} is held here")) +} + +/// Every column a lookup needs, in one place so the three reads cannot select +/// different sets. +macro_rules! columns { + () => { + "id, passport_id, field, content, stored_at, erased_at" + }; +} + +fn held_from(row: &sqlx::postgres::PgRow) -> Result { + let id = PersonalDataRecordId::new(row.get::("id").to_string()); + let passport_id = PassportId(row.get::("passport_id")); + let field = row.get::("field"); + + if let Some(erased_at) = row.get::>, _>("erased_at") { + return Ok(HeldRecord::Erased(ErasureReceipt { + record: id, + passport_id, + field, + erased_at, + })); + } + // Present and without content is a state the table's CHECK refuses; saying + // so beats reading it as an empty record. + let Some(content) = row.get::, _>("content") else { + return Err(DppError::Internal(format!( + "personal data record {id} is neither erased nor holds content" + ))); + }; + Ok(HeldRecord::Present(PersonalDataRecord { + id, + passport_id, + field, + content, + stored_at: row.get::, _>("stored_at"), + })) +} + +fn receipt_from(row: &sqlx::postgres::PgRow) -> ErasureReceipt { + ErasureReceipt { + record: PersonalDataRecordId::new(row.get::("id").to_string()), + passport_id: PassportId(row.get::("passport_id")), + field: row.get::("field"), + erased_at: row.get::, _>("erased_at"), + } +} + +#[async_trait] +impl PersonalDataPort for PgPersonalDataRepo { + async fn store( + &self, + passport_id: PassportId, + field: &str, + content: &serde_json::Value, + ) -> Result { + let id = Uuid::now_v7(); + sqlx::query( + "INSERT INTO odal.personal_data_record (id, passport_id, field, content) + VALUES ($1, $2, $3, $4)", + ) + .bind(id) + .bind(passport_id.0) + .bind(field) + .bind(content) + .execute(self.dal.pool()) + .await + .map_err(|e| match &e { + // The passport is the thing a record hangs off. A node that has no + // row for it has nothing to hold the record for. + sqlx::Error::Database(db) if db.is_foreign_key_violation() => { + DppError::NotFound(format!("no passport {} is held here", passport_id.0)) + } + _ => db_err(e), + })?; + Ok(PersonalDataRecordId::new(id.to_string())) + } + + async fn fetch(&self, record: &PersonalDataRecordId) -> Result, DppError> { + let Some(id) = parse(record) else { + return Ok(None); + }; + let row = sqlx::query(concat!( + "SELECT ", + columns!(), + " FROM odal.personal_data_record WHERE id = $1" + )) + .bind(id) + .fetch_optional(self.dal.pool()) + .await + .map_err(db_err)?; + row.as_ref().map(held_from).transpose() + } + + async fn records_for(&self, passport_id: PassportId) -> Result, DppError> { + // `id` breaks a tie between two records stored in the same instant; it + // is a UUID v7, so it orders as they were minted. + let rows = sqlx::query(concat!( + "SELECT ", + columns!(), + " FROM odal.personal_data_record + WHERE passport_id = $1 + ORDER BY stored_at, id" + )) + .bind(passport_id.0) + .fetch_all(self.dal.pool()) + .await + .map_err(db_err)?; + rows.iter().map(held_from).collect() + } + + async fn erase(&self, record: &PersonalDataRecordId) -> Result { + let id = parse(record).ok_or_else(|| unknown(record))?; + + let erased = sqlx::query( + "UPDATE odal.personal_data_record + SET content = NULL, erased_at = now() + WHERE id = $1 AND erased_at IS NULL + RETURNING id, passport_id, field, erased_at", + ) + .bind(id) + .fetch_optional(self.dal.pool()) + .await + .map_err(db_err)?; + if let Some(row) = erased { + return Ok(receipt_from(&row)); + } + + // Nothing was erased by this call: the record is either unknown or was + // erased already, by an earlier call or by one that committed while this + // one waited. The receipt is the first one's either way. + let row = sqlx::query( + "SELECT id, passport_id, field, erased_at + FROM odal.personal_data_record + WHERE id = $1 AND erased_at IS NOT NULL", + ) + .bind(id) + .fetch_optional(self.dal.pool()) + .await + .map_err(db_err)?; + row.as_ref() + .map(receipt_from) + .ok_or_else(|| unknown(record)) + } +} diff --git a/crates/dpp-dal/tests/pg_integration.rs b/crates/dpp-dal/tests/pg_integration.rs index 4eb8fb82..fc0a1cb4 100644 --- a/crates/dpp-dal/tests/pg_integration.rs +++ b/crates/dpp-dal/tests/pg_integration.rs @@ -643,6 +643,7 @@ async fn t8_app_role_can_read_every_table() { "odal.unsold_goods_report", "odal.registry_identity_audit", "odal.passport_transfer", + "odal.personal_data_record", "identity.did_document", "identity.key_pair", ]; diff --git a/crates/dpp-dal/tests/pg_personal_data.rs b/crates/dpp-dal/tests/pg_personal_data.rs new file mode 100644 index 00000000..c8733fd4 --- /dev/null +++ b/crates/dpp-dal/tests/pg_personal_data.rs @@ -0,0 +1,545 @@ +//! The personal-data store, against a real PostgreSQL (`ops/pg/0043`). +//! +//! Core states the contract in `dpp_domain::ports::personal_data` and tests it +//! against an in-memory holder. What these cover that a holder in memory cannot: +//! +//! 1. **The contract holds on the real table** — the same cases, through the +//! SQL: a tombstone with no content, a retried erasure handing back the first +//! receipt, a listing per passport that includes what was erased. +//! 2. **A lost answer does not lose the record.** A `store` whose answer never +//! arrived still committed a row; it is found through `records_for`. +//! 3. **Two erasures racing are handed one receipt.** The retry rule is a +//! property of the `UPDATE ... WHERE erased_at IS NULL` and the row lock, and +//! only concurrent connections can show it. +//! 4. **The database refuses what the Rust never asks for.** The app role cannot +//! `DELETE` a tombstone, rewrite a record, or bring an erased one back, and +//! neither can the trigger's other callers: a superuser's `UPDATE` hits it +//! too. +//! 5. **Erasing touches nothing else** — not the passport row, not another +//! record. + +#![cfg(feature = "integration-tests")] + +use dpp_dal::pg::{PgPersonalDataRepo, sqlx}; +use dpp_dal::test_harness::{TestPg, start_pg}; +use dpp_domain::DppError; +use dpp_domain::passport::PassportId; +use dpp_domain::personal_data::PersonalDataRecordId; +use dpp_domain::ports::personal_data::{HeldRecord, PersonalDataPort}; +use serde_json::json; +use sqlx::Row as _; +use uuid::Uuid; + +const FIELD: &str = "usageHistory.negativeEvents"; + +/// A row in `odal.passport`, which is all a record's foreign key asks of a +/// passport. Written as the superuser with a minimal `doc`: this suite is about +/// the record store, and the typed `Passport` builders the other suites carry +/// would add a sixth copy to say nothing more. +async fn a_passport(pg: &TestPg) -> PassportId { + let id = Uuid::now_v7(); + let admin = sqlx::PgPool::connect(&pg.admin_url).await.expect("admin"); + sqlx::query( + "INSERT INTO odal.passport (id, product_group, schema_version, doc) + VALUES ($1, 'battery', '2.0.0', $2)", + ) + .bind(id) + .bind(json!({ "id": id, "productName": "Personal Data Test Battery" })) + .execute(&admin) + .await + .expect("insert passport"); + admin.close().await; + PassportId(id) +} + +async fn admin(pg: &TestPg) -> sqlx::PgPool { + sqlx::PgPool::connect(&pg.admin_url).await.expect("admin") +} + +fn present(held: Option) -> dpp_domain::ports::personal_data::PersonalDataRecord { + match held { + Some(HeldRecord::Present(record)) => record, + other => panic!("expected a present record, got {other:?}"), + } +} + +#[tokio::test] +async fn a_stored_record_is_held_under_the_identifier_it_was_given() { + let pg = start_pg().await; + let store = PgPersonalDataRepo::new(pg.dal.clone()); + let passport = a_passport(&pg).await; + let content = + json!({ "driver": "named in the accident report", "events": [1, 2, { "n": null }] }); + + let id = store.store(passport, FIELD, &content).await.expect("store"); + + let record = present(store.fetch(&id).await.expect("fetch")); + assert_eq!(record.id, id); + assert_eq!(record.passport_id, passport); + assert_eq!(record.field, FIELD); + assert_eq!(record.content, content); +} + +/// JSON `null` is a value, and the controller keeps data "in whatever shape" it +/// likes. A column that read it back as SQL `NULL` would call a held record +/// erased. +#[tokio::test] +async fn a_record_whose_content_is_json_null_is_still_present() { + let pg = start_pg().await; + let store = PgPersonalDataRepo::new(pg.dal.clone()); + let passport = a_passport(&pg).await; + + let id = store + .store(passport, FIELD, &serde_json::Value::Null) + .await + .expect("store"); + + let record = present(store.fetch(&id).await.expect("fetch")); + assert_eq!(record.content, serde_json::Value::Null); +} + +/// The identifier goes into a signed passport that outlives the record. +#[tokio::test] +async fn every_store_mints_a_new_identifier_that_carries_nothing_of_the_content() { + let pg = start_pg().await; + let store = PgPersonalDataRepo::new(pg.dal.clone()); + let passport = a_passport(&pg).await; + let content = json!("Jane Example"); + + let first = store.store(passport, FIELD, &content).await.expect("store"); + let second = store.store(passport, FIELD, &content).await.expect("store"); + + assert_ne!(first, second); + assert!(!first.as_str().contains("Jane")); + Uuid::parse_str(first.as_str()).expect("a UUID, not anything derived from the data"); +} + +/// Storing does not check that the field is marked: that is the passport's check, +/// made when the statement naming the record is written. +#[tokio::test] +async fn storing_does_not_ask_whether_the_field_is_marked() { + let pg = start_pg().await; + let store = PgPersonalDataRepo::new(pg.dal.clone()); + let passport = a_passport(&pg).await; + + store + .store(passport, "productName", &json!("anything")) + .await + .expect("held"); +} + +#[tokio::test] +async fn erasure_keeps_a_tombstone_without_the_data() { + let pg = start_pg().await; + let store = PgPersonalDataRepo::new(pg.dal.clone()); + let passport = a_passport(&pg).await; + let id = store + .store(passport, FIELD, &json!("Jane Example")) + .await + .expect("store"); + + let receipt = store.erase(&id).await.expect("erase"); + assert_eq!(receipt.record, id); + assert_eq!(receipt.passport_id, passport); + assert_eq!(receipt.field, FIELD); + + let held = store.fetch(&id).await.expect("fetch"); + assert_eq!(held, Some(HeldRecord::Erased(receipt))); + + // Not only absent from the Rust type: absent from the row. + let admin = admin(&pg).await; + let row = sqlx::query("SELECT content IS NULL AS no_content, to_jsonb(r)::text AS whole FROM odal.personal_data_record r") + .fetch_one(&admin) + .await + .expect("the tombstone is a row"); + assert!(row.get::("no_content")); + assert!( + !row.get::("whole").contains("Jane"), + "nothing of the content survives anywhere in the row" + ); +} + +/// 🚨 The retry rule. A caller that lost the answer asks again and is handed the +/// receipt it was always going to be given — not a second one with a later +/// instant, which would misstate when the data was erased. +#[tokio::test] +async fn a_retried_erasure_returns_the_first_receipt() { + let pg = start_pg().await; + let store = PgPersonalDataRepo::new(pg.dal.clone()); + let passport = a_passport(&pg).await; + let id = store + .store(passport, FIELD, &json!("x")) + .await + .expect("store"); + + let first = store.erase(&id).await.expect("erase"); + // Long enough that a receipt stamped again would show a later instant. + tokio::time::sleep(std::time::Duration::from_millis(50)).await; + let again = store.erase(&id).await.expect("a retry is safe"); + + assert_eq!(first, again); + assert_eq!(first.erased_at, again.erased_at); +} + +/// Two callers racing to erase one record are both answered with the same +/// receipt: the second waits on the row, finds it erased, and reads the first's. +#[tokio::test(flavor = "multi_thread")] +async fn concurrent_erasures_are_handed_one_receipt() { + let pg = start_pg().await; + let store = std::sync::Arc::new(PgPersonalDataRepo::new(pg.dal.clone())); + let passport = a_passport(&pg).await; + let id = store + .store(passport, FIELD, &json!("x")) + .await + .expect("store"); + + let mut racers = Vec::new(); + for _ in 0..8 { + let store = store.clone(); + let id = id.clone(); + racers.push(tokio::spawn(async move { store.erase(&id).await })); + } + let mut receipts = Vec::new(); + for racer in racers { + receipts.push( + racer + .await + .expect("joined") + .expect("every caller is answered"), + ); + } + + assert!( + receipts.windows(2).all(|pair| pair[0] == pair[1]), + "one receipt, handed to all eight: {receipts:?}" + ); +} + +#[tokio::test] +async fn erasing_one_record_leaves_the_others_and_the_passport() { + let pg = start_pg().await; + let store = PgPersonalDataRepo::new(pg.dal.clone()); + let passport = a_passport(&pg).await; + let erased = store + .store(passport, FIELD, &json!("a")) + .await + .expect("store"); + let kept = store + .store(passport, FIELD, &json!("b")) + .await + .expect("store"); + + let admin = admin(&pg).await; + let doc_before: serde_json::Value = + sqlx::query_scalar("SELECT doc FROM odal.passport WHERE id = $1") + .bind(passport.0) + .fetch_one(&admin) + .await + .expect("doc"); + + store.erase(&erased).await.expect("erase"); + + present(store.fetch(&kept).await.expect("fetch")); + let doc_after: serde_json::Value = + sqlx::query_scalar("SELECT doc FROM odal.passport WHERE id = $1") + .bind(passport.0) + .fetch_one(&admin) + .await + .expect("doc"); + assert_eq!( + doc_before, doc_after, + "erasing a record does not touch the passport" + ); +} + +#[tokio::test] +async fn an_identifier_never_minted_is_unknown() { + let pg = start_pg().await; + let store = PgPersonalDataRepo::new(pg.dal.clone()); + + // One that could have been minted here and was not, and ones that never + // could be: neither names a record, and both are answered alike. + for stranger in [ + Uuid::now_v7().to_string(), + "never-minted".to_owned(), + String::new(), + ] { + let stranger = PersonalDataRecordId::new(stranger); + assert_eq!( + store.fetch(&stranger).await.expect("fetch"), + None, + "{stranger}" + ); + assert!( + matches!(store.erase(&stranger).await, Err(DppError::NotFound(_))), + "{stranger}" + ); + } +} + +/// A `store` whose answer never arrived still left a row, and the caller must +/// be able to find it to erase it. +#[tokio::test] +async fn a_record_whose_identifier_was_lost_is_found_by_its_passport_and_erased() { + let pg = start_pg().await; + let store = PgPersonalDataRepo::new(pg.dal.clone()); + let passport = a_passport(&pg).await; + let elsewhere = a_passport(&pg).await; + + // The caller's `store` committed and its answer was lost: the id is dropped + // on the floor, which is exactly what losing it means. + let _ = store + .store(passport, FIELD, &json!("a")) + .await + .expect("store"); + let erased = store + .store(passport, FIELD, &json!("b")) + .await + .expect("store"); + store + .store(elsewhere, FIELD, &json!("c")) + .await + .expect("store"); + let receipt = store.erase(&erased).await.expect("erase"); + + let held = store.records_for(passport).await.expect("records_for"); + assert_eq!( + held.len(), + 2, + "this passport's records only, erased ones too" + ); + let HeldRecord::Present(lost) = &held[0] else { + panic!("the first record stored is listed first, with its data: {held:?}"); + }; + assert_eq!(lost.content, json!("a")); + assert_eq!(held[1], HeldRecord::Erased(receipt)); + + store + .erase(&lost.id) + .await + .expect("the lost record is erased by the id the listing gave"); + assert!( + store + .records_for(passport) + .await + .expect("records_for") + .iter() + .all(|h| matches!(h, HeldRecord::Erased(_))), + "nothing is held for the passport any more" + ); + assert!( + matches!( + store.records_for(elsewhere).await.expect("records_for")[0], + HeldRecord::Present(_) + ), + "the other passport's record is untouched" + ); +} + +/// Listed in the order stored, not the order the keys happen to sort. +#[tokio::test] +async fn records_for_lists_in_the_order_they_were_stored() { + let pg = start_pg().await; + let store = PgPersonalDataRepo::new(pg.dal.clone()); + let passport = a_passport(&pg).await; + + let mut stored = Vec::new(); + for n in 0..5 { + stored.push( + store + .store(passport, FIELD, &json!(n)) + .await + .expect("store"), + ); + } + // Erased out of order: a tombstone keeps its place. + store.erase(&stored[3]).await.expect("erase"); + store.erase(&stored[0]).await.expect("erase"); + + let listed: Vec = store + .records_for(passport) + .await + .expect("records_for") + .into_iter() + .map(|h| match h { + HeldRecord::Present(r) => r.id.to_string(), + HeldRecord::Erased(r) => r.record.to_string(), + }) + .collect(); + let expected: Vec = stored.iter().map(ToString::to_string).collect(); + assert_eq!(listed, expected); +} + +#[tokio::test] +async fn a_passport_with_nothing_held_lists_nothing() { + let pg = start_pg().await; + let store = PgPersonalDataRepo::new(pg.dal.clone()); + let passport = a_passport(&pg).await; + + assert!( + store + .records_for(passport) + .await + .expect("records_for") + .is_empty() + ); +} + +/// The record hangs off a passport this node holds. Refused as the passport +/// being unknown, and nothing is written. +#[tokio::test] +async fn a_record_for_a_passport_this_node_does_not_hold_is_refused() { + let pg = start_pg().await; + let store = PgPersonalDataRepo::new(pg.dal.clone()); + + let refused = store.store(PassportId::new(), FIELD, &json!("x")).await; + assert!(matches!(refused, Err(DppError::NotFound(_))), "{refused:?}"); + + let admin = admin(&pg).await; + let held: i64 = sqlx::query_scalar("SELECT count(*) FROM odal.personal_data_record") + .fetch_one(&admin) + .await + .expect("count"); + assert_eq!(held, 0); +} + +/// What the app role is not allowed to do, tried as the app role: the grants are +/// the first line, and a tombstone is the controller's proof that it erased. +#[tokio::test] +async fn the_app_role_can_neither_delete_a_record_nor_rewrite_it() { + let pg = start_pg().await; + let store = PgPersonalDataRepo::new(pg.dal.clone()); + let passport = a_passport(&pg).await; + let id = store + .store(passport, FIELD, &json!("x")) + .await + .expect("store"); + let other = a_passport(&pg).await; + store.erase(&id).await.expect("erase"); + let id = Uuid::parse_str(id.as_str()).expect("uuid"); + + let app = sqlx::PgPool::connect(&pg.app_url).await.expect("app"); + + let delete = sqlx::query("DELETE FROM odal.personal_data_record WHERE id = $1") + .bind(id) + .execute(&app) + .await; + assert!( + delete.is_err(), + "a tombstone is not the app role's to delete" + ); + + // Moving a record to another passport would hand its data to someone else. + let repoint = + sqlx::query("UPDATE odal.personal_data_record SET passport_id = $2 WHERE id = $1") + .bind(id) + .bind(other.0) + .execute(&app) + .await; + assert!( + repoint.is_err(), + "UPDATE is granted on the two erasure columns only" + ); + + let still: i64 = sqlx::query_scalar( + "SELECT count(*) FROM odal.personal_data_record WHERE id = $1 AND passport_id <> $2", + ) + .bind(id) + .bind(other.0) + .fetch_one(&app) + .await + .expect("count"); + assert_eq!(still, 1); +} + +/// The trigger is the second line, and holds for the superuser too: an erased +/// record does not come back, and a present one is erased or left alone. +#[tokio::test] +async fn an_erased_record_cannot_be_brought_back_and_a_present_one_cannot_be_rewritten() { + let pg = start_pg().await; + let store = PgPersonalDataRepo::new(pg.dal.clone()); + let passport = a_passport(&pg).await; + let erased = store + .store(passport, FIELD, &json!("gone")) + .await + .expect("store"); + let present = store + .store(passport, FIELD, &json!("held")) + .await + .expect("store"); + let receipt = store.erase(&erased).await.expect("erase"); + let erased = Uuid::parse_str(erased.as_str()).expect("uuid"); + let present = Uuid::parse_str(present.as_str()).expect("uuid"); + + let admin = admin(&pg).await; + + let resurrect = sqlx::query( + "UPDATE odal.personal_data_record SET content = '\"back\"', erased_at = NULL WHERE id = $1", + ) + .bind(erased) + .execute(&admin) + .await + .expect_err("an erased record stays erased"); + assert!( + resurrect.to_string().contains("ODAL_PERSONAL_DATA"), + "{resurrect}" + ); + + let restamp = + sqlx::query("UPDATE odal.personal_data_record SET erased_at = now() WHERE id = $1") + .bind(erased) + .execute(&admin) + .await + .expect_err("the receipt is the first one"); + assert!( + restamp.to_string().contains("ODAL_PERSONAL_DATA"), + "{restamp}" + ); + + let rewrite = + sqlx::query("UPDATE odal.personal_data_record SET content = '\"other\"' WHERE id = $1") + .bind(present) + .execute(&admin) + .await + .expect_err("a record is erased, not edited"); + assert!( + rewrite.to_string().contains("ODAL_PERSONAL_DATA"), + "{rewrite}" + ); + + assert_eq!( + store + .fetch(&PersonalDataRecordId::new(erased.to_string())) + .await + .expect("fetch"), + Some(HeldRecord::Erased(receipt)), + "the tombstone is as it was" + ); +} + +/// Neither state can be half-written: the migration's CHECK. +#[tokio::test] +async fn a_row_is_present_or_erased_and_never_both_or_neither() { + let pg = start_pg().await; + let passport = a_passport(&pg).await; + let admin = admin(&pg).await; + + for (content, erased_at) in [("'\"x\"'::jsonb", "now()"), ("NULL", "NULL")] { + let sql = format!( + "INSERT INTO odal.personal_data_record (id, passport_id, field, content, erased_at) + VALUES ($1, $2, 'f', {content}, {erased_at})" + ); + let refused = sqlx::query(sqlx::AssertSqlSafe(sql)) + .bind(Uuid::now_v7()) + .bind(passport.0) + .execute(&admin) + .await + .expect_err("a row in neither state is refused"); + assert!( + refused + .to_string() + .contains("personal_data_record_one_state"), + "{refused}" + ); + } +} diff --git a/crates/dpp-node/src/boot/db.rs b/crates/dpp-node/src/boot/db.rs index 80e3ca18..1c9aa27b 100644 --- a/crates/dpp-node/src/boot/db.rs +++ b/crates/dpp-node/src/boot/db.rs @@ -55,6 +55,7 @@ pub struct DbComponents { pub seal_audit: Arc, pub scan_repo: Arc, pub unsold_goods_repo: Arc, + pub personal_data: Arc, pub job_store: Arc, pub db_ping: Arc, pub idempotency: Arc, @@ -151,6 +152,7 @@ pub async fn init_db(cfg: &NodeConfig) -> anyhow::Result { seal_audit: Arc::new(dpp_dal::pg::PgSealAuditRepo::new(dal.clone())), scan_repo: Arc::new(PgScanTelemetryRepo::new(dal.clone())), unsold_goods_repo: Arc::new(dpp_dal::pg::PgUnsoldGoodsRepo::new(dal.clone())), + personal_data: Arc::new(dpp_dal::pg::PgPersonalDataRepo::new(dal.clone())), job_store: Arc::new(PgJobStore::new(dal.clone())), idempotency: Arc::new(PgIdempotencyRepo::new(dal.clone())), db_ping: Arc::new(PgPing(dal)), diff --git a/crates/dpp-node/src/main.rs b/crates/dpp-node/src/main.rs index 8aa69620..3dbf85db 100644 --- a/crates/dpp-node/src/main.rs +++ b/crates/dpp-node/src/main.rs @@ -398,6 +398,7 @@ async fn main() -> anyhow::Result<()> { cors_allowed_origins: cfg.cors_allowed_origins.clone(), scan_repo: db.scan_repo.clone(), unsold_goods_repo: db.unsold_goods_repo.clone(), + personal_data: db.personal_data.clone(), plugin_admin: Some(plugin_admin), // Reported on the authenticated `/vault/api/v1/node/state`, not on the // public `/health` — see `dpp_node::router::node_health`. diff --git a/crates/dpp-node/tests/openapi_contract.rs b/crates/dpp-node/tests/openapi_contract.rs index b389708c..dbf9e3db 100644 --- a/crates/dpp-node/tests/openapi_contract.rs +++ b/crates/dpp-node/tests/openapi_contract.rs @@ -522,6 +522,10 @@ fn object_cases() -> Vec { // ── dpp-vault: request and response bodies ──────────────────────────── case!("CreatePassportRequest", fixtures::create_request()); case!("ValidateResponse", fixtures::validate_response()); + case!( + "PersonalDataErasureReceipt", + fixtures::personal_data_erasure_receipt() + ); case!("PassportListResponse", fixtures::passport_list_response()); case!("WhoamiResponse", fixtures::whoami_response()); case!("PassportScanStats", fixtures::passport_scan_stats()); @@ -2363,6 +2367,7 @@ mod handler_sources { include_str!("../../dpp-vault/src/handlers/list.rs"), include_str!("../../dpp-vault/src/handlers/node_state.rs"), include_str!("../../dpp-vault/src/handlers/operator.rs"), + include_str!("../../dpp-vault/src/handlers/personal_data.rs"), include_str!("../../dpp-vault/src/handlers/plugins.rs"), include_str!("../../dpp-vault/src/handlers/public_read.rs"), include_str!("../../dpp-vault/src/handlers/public_read_by_gtin.rs"), @@ -2858,6 +2863,7 @@ mod fixtures { info::VaultInfo, list::PassportListResponse, node_state::NodeState, + personal_data::PersonalDataErasureReceipt, registry_status::{ CurrentOperatorView, PassportRegistryView, RegistrationCounts, RegistrationView, RegistryRollupView, RegistryVerificationView, TransferNotificationCounts, @@ -3858,6 +3864,15 @@ mod fixtures { } } + pub fn personal_data_erasure_receipt() -> PersonalDataErasureReceipt { + PersonalDataErasureReceipt { + record: "0198f2c4-7a3e-7c1d-9b52-3e8f6a1d4c70".into(), + passport_id: uuid(), + field: "usageHistory.negativeEvents".into(), + erased_at: ts(), + } + } + pub fn validate_response() -> ValidateResponse { ValidateResponse { create_valid: true, diff --git a/crates/dpp-node/tests/smoke.rs b/crates/dpp-node/tests/smoke.rs index cc639f9f..f108a770 100644 --- a/crates/dpp-node/tests/smoke.rs +++ b/crates/dpp-node/tests/smoke.rs @@ -209,6 +209,7 @@ async fn start_node_with_ruleset( let auth_provider: Arc = Arc::new(TestAuthProvider); let scan_repo = Arc::new(PgScanTelemetryRepo::new(dal.clone())); let unsold_goods_repo = Arc::new(dpp_dal::pg::PgUnsoldGoodsRepo::new(dal.clone())); + let personal_data = Arc::new(dpp_dal::pg::PgPersonalDataRepo::new(dal.clone())); // Declared before the vault state, which now carries the trust posture so // the authenticated node-state route can report it. let trust = std::sync::Arc::new(dpp_types::trust::NodeTrustReport::new( @@ -253,6 +254,7 @@ async fn start_node_with_ruleset( cors_allowed_origins: Vec::new(), scan_repo, unsold_goods_repo, + personal_data, plugin_admin: None, // The trust posture is asserted through the authenticated node-state // route, not the public probe. @@ -2777,6 +2779,242 @@ async fn a_draft_holding_personal_data_without_a_statement_is_refused_on_every_w assert_eq!(resp.status(), 200, "personalData is patchable on a draft"); } +/// 🚨 The reason the records have a store of their own: a published passport is +/// signed, frozen, archived, sealed and copied to the back-up provider, so +/// nothing that reaches any of those can be erased. The content of a record is +/// found in none of them, however the passport is read and whatever it went on +/// to feed. +/// +/// Two halves, because each alone can pass by not looking. Every read route and +/// audience is asked for the passport and none returns the content. And every +/// table other than the record's own is searched for it, which covers the +/// archive of past versions, the audit chain, the outboxes and everything a +/// later change adds — a route nobody thought to list cannot hide it. +/// +/// A record the passport's statement names, so the statement is in every view +/// that carries it. The *identifier* is meant to appear there; the *content* is +/// not. +#[tokio::test(flavor = "multi_thread")] +async fn a_personal_data_record_is_in_no_view_of_the_passport_and_no_other_table() { + use dpp_domain::ports::personal_data::PersonalDataPort as _; + use sqlx::Row as _; + + const MARKER: &str = "accident-report-of-jane-example-7f3a91"; + + let (base, pg) = start_db_and_node().await; + let token = make_jwt("00000000-0000-0000-0000-000000000085"); + let client = reqwest::Client::new(); + + // A draft that says nothing is held, to mint the passport; then the record, + // then the statement naming it. + let created: serde_json::Value = client + .post(format!("{base}/vault/api/v1/dpp")) + .bearer_auth(&token) + .json(&textile_with_a_repair_history(Some( + serde_json::json!({ "repairHistoryUrl": { "held": "nothing" } }), + ))) + .send() + .await + .expect("create request failed") + .json() + .await + .unwrap(); + let id = created["id"].as_str().expect("an id").to_owned(); + + let store = dpp_dal::pg::PgPersonalDataRepo::new(pg.dal.clone()); + let record = store + .store( + dpp_domain::passport::PassportId(id.parse().expect("uuid")), + "repairHistoryUrl", + &serde_json::json!({ "report": MARKER }), + ) + .await + .expect("the record is held"); + let statement = serde_json::json!({ + "repairHistoryUrl": { + "held": "outside", "lawfulBasis": "consent", "record": record.as_str() + } + }); + let resp = client + .put(format!("{base}/vault/api/v1/dpp/{id}")) + .bearer_auth(&token) + .json(&serde_json::json!({ "personalData": statement })) + .send() + .await + .expect("update request failed"); + assert_eq!(resp.status(), 200); + + let resp = client + .post(format!("{base}/vault/api/v1/dpp/{id}/publish")) + .bearer_auth(&token) + .json(&serde_json::json!({})) + .send() + .await + .expect("publish request failed"); + let status = resp.status(); + let body = resp.text().await.unwrap_or_default(); + assert_eq!(status, 200, "the passport publishes: {body}"); + + // Every way of reading the passport, and the audiences above public. + let jws = issue_repairer_credential(&base, &token, &client).await; + let evidence = client + .post(format!("{base}/vault/api/v1/dpp/{id}/evidence")) + .bearer_auth(&token) + .json(&serde_json::json!({})) + .send() + .await + .expect("evidence request failed"); + assert_eq!(evidence.status(), 201); + + let reads = vec![ + ( + "public", + client.get(format!("{base}/vault/public/dpp/{id}")), + ), + ( + "authenticated", + client + .get(format!("{base}/vault/api/v1/dpp/{id}")) + .bearer_auth(&token), + ), + ( + "credentialed", + client + .get(format!("{base}/vault/credential/dpp/{id}")) + .header("X-DPP-Credential", &jws), + ), + ( + "history", + client + .get(format!("{base}/vault/api/v1/dpp/{id}/history")) + .bearer_auth(&token), + ), + ( + "versions", + client + .get(format!("{base}/vault/api/v1/dpp/{id}/versions")) + .bearer_auth(&token), + ), + ( + "evidence", + client + .get(format!("{base}/vault/api/v1/dpp/{id}/evidence")) + .bearer_auth(&token), + ), + ( + "list", + client + .get(format!("{base}/vault/api/v1/dpps")) + .bearer_auth(&token), + ), + ]; + let mut served = Vec::new(); + for (name, request) in reads { + let resp = request.send().await.expect("read request failed"); + assert!(resp.status().is_success(), "{name}: {}", resp.status()); + let text = resp.text().await.unwrap_or_default(); + assert!( + !text.contains(MARKER), + "the {name} view carries the content of a record: {text}" + ); + served.push((name, text)); + } + assert!( + served + .iter() + .any(|(name, text)| *name == "authenticated" && text.contains(record.as_str())), + "the statement names the record in the view that serves statements, so the \ + absence above is not an empty read" + ); + + // And no other table holds it. Searched as whole rows, so a column nobody + // thought of is covered; the record's own table is the control that the + // search can find it at all. + let admin = sqlx::PgPool::connect(&pg.admin_url).await.expect("admin"); + let tables = sqlx::query( + "SELECT table_schema, table_name FROM information_schema.tables + WHERE table_schema IN ('odal', 'identity') AND table_type = 'BASE TABLE'", + ) + .fetch_all(&admin) + .await + .expect("tables"); + assert!( + tables.len() > 10, + "the search covers the schema, not an empty list" + ); + let mut holding = Vec::new(); + for row in &tables { + let table = format!( + "{}.{}", + row.get::("table_schema"), + row.get::("table_name") + ); + // Names come from the catalogue just read, not from a caller. + let found: i64 = sqlx::query_scalar(sqlx::AssertSqlSafe(format!( + "SELECT count(*) FROM {table} t WHERE to_jsonb(t.*)::text LIKE '%' || $1 || '%'" + ))) + .bind(MARKER) + .fetch_one(&admin) + .await + .unwrap_or_else(|e| panic!("search {table}: {e}")); + if found > 0 { + holding.push(table); + } + } + assert_eq!( + holding, + ["odal.personal_data_record"], + "the content is held in its own table and in no other" + ); + + // Erasing it changes none of what the passport serves. + let erased = client + .delete(format!( + "{base}/vault/api/v1/personal-data/{}", + record.as_str() + )) + .bearer_auth(&token) + .send() + .await + .expect("erase request failed"); + assert_eq!(erased.status(), 200); + let public_after = client + .get(format!("{base}/vault/public/dpp/{id}")) + .send() + .await + .expect("read request failed") + .text() + .await + .unwrap_or_default(); + let authenticated_after = client + .get(format!("{base}/vault/api/v1/dpp/{id}")) + .bearer_auth(&token) + .send() + .await + .expect("read request failed") + .text() + .await + .unwrap_or_default(); + for (name, after) in [ + ("public", public_after), + ("authenticated", authenticated_after), + ] { + let before = &served + .iter() + .find(|(n, _)| *n == name) + .expect("read above") + .1; + assert_eq!(*before, after, "the {name} view is as it was after erasure"); + } + let remaining: i64 = sqlx::query_scalar( + "SELECT count(*) FROM odal.personal_data_record WHERE content IS NOT NULL", + ) + .fetch_one(&admin) + .await + .expect("count"); + assert_eq!(remaining, 0, "and nothing of the content is left"); +} + /// The vocabulary belongs to one regulation, so it belongs to one product group. #[tokio::test] async fn a_non_battery_passport_may_not_claim_a_battery_life_status() { diff --git a/crates/dpp-vault/src/domain/service/create.rs b/crates/dpp-vault/src/domain/service/create.rs index da98d127..cf7d2011 100644 --- a/crates/dpp-vault/src/domain/service/create.rs +++ b/crates/dpp-vault/src/domain/service/create.rs @@ -538,7 +538,7 @@ pub(super) fn apply_patch( /// Core's [`dpp_domain::check_personal_data`] makes the decision and never /// reads what a field says; this maps its findings to the `422` the other /// field-level refusals use, one entry per field. -pub(super) fn guard_personal_data(passport: &Passport) -> Result<(), DppError> { +pub(crate) fn guard_personal_data(passport: &Passport) -> Result<(), DppError> { let errors = dpp_domain::check_personal_data(passport); if errors.is_empty() { Ok(()) diff --git a/crates/dpp-vault/src/domain/service/mod.rs b/crates/dpp-vault/src/domain/service/mod.rs index aa07ad4a..a58fc2c6 100644 --- a/crates/dpp-vault/src/domain/service/mod.rs +++ b/crates/dpp-vault/src/domain/service/mod.rs @@ -21,6 +21,9 @@ mod amend; mod create; +/// The personal-data check every write that stores a passport runs, shared with +/// the dry-run route so a preview cannot pass what a create refuses. +pub(crate) use create::guard_personal_data; /// The component-reference rule, shared with the create handler so the two /// entry points cannot drift apart on what a valid BOM entry is. pub(crate) use create::validate_component_ref; diff --git a/crates/dpp-vault/src/handlers/create.rs b/crates/dpp-vault/src/handlers/create.rs index 0a0724a4..8ce9446d 100644 --- a/crates/dpp-vault/src/handlers/create.rs +++ b/crates/dpp-vault/src/handlers/create.rs @@ -54,6 +54,41 @@ pub async fn create_handler( return resp; } + let passport = match assemble_passport(body) { + Ok(passport) => passport, + Err(resp) => return resp, + }; + + match state.service.create(passport, &auth).await { + Ok(p) => ( + StatusCode::CREATED, + Json(crate::api::PassportResponse::from(&p)), + ) + .into_response(), + // What the service refuses on the assembled record rather than on the + // request: a field able to hold personal data that has a value and no + // statement, which names the field. + Err(dpp_domain::DppError::Validation(errs)) => field_validation_error(&errs), + Err(e) => internal_error(e), + } +} + +/// The draft `POST /api/v1/dpp` would store for `body`, or the response that +/// refuses it. +/// +/// Apart from [`create_handler`] so the dry-run route can build the same record +/// and ask of it what the service asks, rather than guess at it from the +/// request: a rule judged on the assembled passport (the product group a body +/// implies, the acts recorded for it, the statements it carries) cannot be +/// previewed from the body alone without restating how the passport is put +/// together. A second copy of that would diverge the first time one of them +/// learned something. +/// +/// Reads no store and writes none. +#[allow(clippy::result_large_err)] +pub fn assemble_passport( + body: CreatePassportRequest, +) -> Result { // ProductGroup is the dispatch key: explicit if supplied, else derived from the // typed product group data, else Other. let product_group = body @@ -103,11 +138,11 @@ pub async fn create_handler( Some(raw) if !raw.is_empty() => match dpp_domain::CommodityCode::parse(raw) { Ok(code) => Some(code), Err(e) => { - return api_error( + return Err(api_error( StatusCode::UNPROCESSABLE_ENTITY, "VALIDATION_ERROR", &e.to_string(), - ); + )); } }, _ => None, @@ -193,18 +228,7 @@ pub async fn create_handler( carrier_serial: body.carrier_serial, }; - match state.service.create(passport, &auth).await { - Ok(p) => ( - StatusCode::CREATED, - Json(crate::api::PassportResponse::from(&p)), - ) - .into_response(), - // What the service refuses on the assembled record rather than on the - // request: a field able to hold personal data that has a value and no - // statement, which names the field. - Err(dpp_domain::DppError::Validation(errs)) => field_validation_error(&errs), - Err(e) => internal_error(e), - } + Ok(passport) } /// `409` when the serial `body` attributes is already held by another passport diff --git a/crates/dpp-vault/src/handlers/mod.rs b/crates/dpp-vault/src/handlers/mod.rs index 2d8518aa..dcbeddc8 100644 --- a/crates/dpp-vault/src/handlers/mod.rs +++ b/crates/dpp-vault/src/handlers/mod.rs @@ -16,6 +16,7 @@ pub mod lint; pub mod list; pub mod node_state; pub mod operator; +pub mod personal_data; pub mod plugins; pub mod public_read; pub mod public_read_by_gtin; diff --git a/crates/dpp-vault/src/handlers/personal_data.rs b/crates/dpp-vault/src/handlers/personal_data.rs new file mode 100644 index 00000000..7f1d99b1 --- /dev/null +++ b/crates/dpp-vault/src/handlers/personal_data.rs @@ -0,0 +1,106 @@ +//! `/api/v1/personal-data/{record}` — erase a record of personal data. +//! +//! # Why the route exists +//! +//! A passport's `personalData` statement may name a record held outside the +//! passport, on a stated lawful basis. When the basis is consent, withdrawing it +//! (GDPR Art. 7(3)) obliges erasure (Art. 17(1)(b)), and a published passport +//! cannot be erased from — so the record is held where it can be, and this is +//! how the operator honours the withdrawal. +//! +//! # What erasing does and does not touch +//! +//! It drops the record's content and keeps a tombstone: the identifier, the +//! passport and field the record belonged to, and when it was erased. It never +//! reaches the passport, which is signed and frozen, nor the statement in it, +//! which keeps naming the record — so a reader following the statement learns +//! that the record is gone rather than that it never existed. +//! +//! **The tombstone is not free of personal data.** A passport identifier can +//! relate to whoever owns the item, and information that can be linked to a +//! person is personal data (GDPR Recital 26). So the receipt is as sensitive as +//! the passport's individual-item data, and the operator needs a basis for +//! keeping the tombstone — to show that it erased (Art. 5(2)) is one. Nothing +//! here deletes it. +//! +//! # Scope +//! +//! Admin, like every other route that removes something (`DELETE` on API keys, +//! webhooks, facilities and operator identifiers). The write scope produces and +//! moves passports; none of that destroys anything, and this cannot be undone. +//! Loosening the gate later breaks nobody; tightening it would. +//! +//! # No `Idempotency-Key` +//! +//! The route is idempotent without one: erasing a record already erased returns +//! the receipt it was given the first time, so a caller that lost the answer +//! asks again. Keying it would store a second copy of the receipt. +//! +//! # Why there is no read route +//! +//! A record is served by no route of this node. Serving one to a reader is a +//! disclosure that needs a basis of its own, which only the controller can +//! establish for each purpose; GDPR Art. 25(2) makes withholding it the default. + +use axum::extract::{Path, State}; +use axum::response::{IntoResponse as _, Response}; +use chrono::{DateTime, Utc}; +use dpp_domain::{DppError, ErasureReceipt, PersonalDataRecordId}; +use serde::Serialize; +use uuid::Uuid; + +use crate::extract::Json; +use crate::handlers::error::{internal_error, not_found_error}; +use crate::middleware::scope::RequireAdmin; +use crate::state::AppState; + +/// What is left of an erased record: which record it was, which passport and +/// field it related to, and when it was erased. None of the erased content. +/// +/// The same receipt on every call after the first. +#[derive(Debug, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct PersonalDataErasureReceipt { + /// The erased record's identifier, as the passport's statement names it. + pub record: String, + /// The passport the record related to. Still related to whoever owns the + /// item, so this receipt is kept like the passport's individual-item data. + pub passport_id: Uuid, + /// The marked field the record related to, as a dotted path inside + /// `productGroupData`. + pub field: String, + /// When the record's content was erased. + pub erased_at: DateTime, +} + +impl From for PersonalDataErasureReceipt { + fn from(receipt: ErasureReceipt) -> Self { + Self { + record: receipt.record.to_string(), + passport_id: receipt.passport_id.0, + field: receipt.field, + erased_at: receipt.erased_at, + } + } +} + +/// `DELETE /api/v1/personal-data/{record}` — erase one record, keeping its +/// tombstone, and answer with the receipt. +/// +/// `404` for a `record` this node never minted — including a string that could +/// not have been, which is not told apart from a well-formed identifier that +/// names nothing. Erasing a record already erased is not an error. +pub async fn personal_data_erase_handler( + State(state): State, + RequireAdmin(_auth): RequireAdmin, + Path(record): Path, +) -> Response { + let record = PersonalDataRecordId::new(record); + match state.personal_data.erase(&record).await { + Ok(receipt) => Json(PersonalDataErasureReceipt::from(receipt)).into_response(), + Err(DppError::NotFound(_)) => { + not_found_error(&format!("No personal data record {record} is held here.")) + } + Err(e) => internal_error(e), + } +} diff --git a/crates/dpp-vault/src/handlers/validate.rs b/crates/dpp-vault/src/handlers/validate.rs index 28ca25ef..fbdf1de0 100644 --- a/crates/dpp-vault/src/handlers/validate.rs +++ b/crates/dpp-vault/src/handlers/validate.rs @@ -3,9 +3,12 @@ use axum::{extract::State, http::StatusCode, response::IntoResponse}; use serde::Serialize; +use dpp_domain::DppError; + use crate::state::AppState; use super::create::CreatePassportRequest; +use super::error::field_validation_error; use crate::extract::Json; use crate::middleware::scope::RequireWrite; @@ -64,10 +67,25 @@ pub async fn validate_handler( if let Some(rejection) = super::create::carrier_serial_conflict(&state, &body).await { return rejection; } + // The draft create would store, and what the service asks of it before it + // does. The personal-data check is judged on the assembled passport, not the + // body, so the preview builds the same one rather than restating how it is + // put together — and it reads no store, so it belongs in a dry run. Without + // it the preview would pass a draft whose marked field has a value and no + // statement, which create refuses. + let passport = match super::create::assemble_passport(body) { + Ok(passport) => passport, + Err(rejection) => return rejection, + }; + if let Err(DppError::Validation(errors)) = + crate::domain::service::guard_personal_data(&passport) + { + return field_validation_error(&errors); + } // Create would accept it. Now the stricter publish gate, so the caller // learns now rather than at publish time. - let publish_blocker = match body.product_group_data.as_ref() { + let publish_blocker = match passport.product_group_data.as_ref() { Some(product_group_data) => state .service .publish_readiness(product_group_data) diff --git a/crates/dpp-vault/src/main.rs b/crates/dpp-vault/src/main.rs index 8b7266c9..233518ee 100644 --- a/crates/dpp-vault/src/main.rs +++ b/crates/dpp-vault/src/main.rs @@ -75,6 +75,7 @@ async fn main() -> anyhow::Result<()> { let webhook_repo = Arc::new(PgWebhookRepo::new(dal.clone())); let scan_repo = Arc::new(PgScanTelemetryRepo::new(dal.clone())); let unsold_goods_repo = Arc::new(dpp_dal::pg::PgUnsoldGoodsRepo::new(dal.clone())); + let personal_data = Arc::new(dpp_dal::pg::PgPersonalDataRepo::new(dal.clone())); let identity = Arc::new(IdentityHttpClient::new(cfg.identity_service_url.clone())); let compliance = Arc::new(PassthroughRegistry::new()); @@ -179,6 +180,7 @@ async fn main() -> anyhow::Result<()> { cors_allowed_origins: cfg.cors_allowed_origins.clone(), scan_repo, unsold_goods_repo, + personal_data, // The standalone vault binary hosts no Wasm plugin engine; runtime // plugin install is available only on the fused node. plugin_admin: None, diff --git a/crates/dpp-vault/src/router.rs b/crates/dpp-vault/src/router.rs index 2c951458..ed592b91 100644 --- a/crates/dpp-vault/src/router.rs +++ b/crates/dpp-vault/src/router.rs @@ -37,6 +37,7 @@ use crate::{ list::list_handler, node_state::node_state_handler, operator::{operator_get_handler, operator_patch_handler}, + personal_data::personal_data_erase_handler, plugins::install_plugin_handler, public_read::public_read_handler, public_read_by_gtin::public_read_by_gtin_handler, @@ -168,6 +169,13 @@ pub fn build(state: AppState) -> Router { ) .route("/api-keys/{id}", delete(api_keys_delete_handler)) .route("/credentials", post(issue_credential_handler)) + // Erases a record a passport's `personalData` statement names. The only + // route that touches the records: there is deliberately none that reads + // one, and nothing under `/public` or `/credential` reaches the store. + .route( + "/personal-data/{record}", + delete(personal_data_erase_handler), + ) .route( "/unsold-goods", get(unsold_goods_list_handler).post(unsold_goods_create_handler), diff --git a/crates/dpp-vault/src/state.rs b/crates/dpp-vault/src/state.rs index 78424f51..aee4b154 100644 --- a/crates/dpp-vault/src/state.rs +++ b/crates/dpp-vault/src/state.rs @@ -107,6 +107,17 @@ pub struct AppState { /// ESPR Art. 24 unsold-goods disclosure lines. A bare port like `scan_repo` /// — a row is written once and read back, with no lifecycle to orchestrate. pub unsold_goods_repo: Arc, + /// The records a passport's `personalData` statements name, held where they + /// can be erased (migration 0043). Not optional, unlike the ports that are + /// absent on a deployment without a composition root: a node that cannot + /// erase a record it was given cannot honour a withdrawn consent, and an + /// erasure route that answered `501` would say so only when someone needed it. + /// + /// Used by the erasure route and by nothing that builds a passport view. A + /// record is never served by a read route, never archived and never part of + /// the back-up copy, which is why this is a port of its own and not a method + /// of the passport repository. + pub personal_data: Arc, /// Origins allowed for CORS requests (empty = CORS disabled). pub cors_allowed_origins: Vec, /// Runtime plugin administration (the Wasm plugin host). `None` on diff --git a/crates/dpp-vault/tests/helpers/mod.rs b/crates/dpp-vault/tests/helpers/mod.rs index 8841223e..e1e10090 100644 --- a/crates/dpp-vault/tests/helpers/mod.rs +++ b/crates/dpp-vault/tests/helpers/mod.rs @@ -472,6 +472,7 @@ async fn start_vault_with_identity( let auth_provider: Arc = Arc::new(TestAuthProvider); let scan_repo = Arc::new(PgScanTelemetryRepo::new(dal.clone())); let unsold_goods_repo = Arc::new(dpp_dal::pg::PgUnsoldGoodsRepo::new(dal.clone())); + let personal_data = Arc::new(dpp_dal::pg::PgPersonalDataRepo::new(dal.clone())); let state = AppState { service, @@ -488,6 +489,7 @@ async fn start_vault_with_identity( cors_allowed_origins: Vec::new(), scan_repo, unsold_goods_repo, + personal_data, plugin_admin: None, // These tests exercise the vault in isolation, with no composition // root resolving trust ports or a ruleset. diff --git a/crates/dpp-vault/tests/personal_data_route.rs b/crates/dpp-vault/tests/personal_data_route.rs new file mode 100644 index 00000000..4cfc86e8 --- /dev/null +++ b/crates/dpp-vault/tests/personal_data_route.rs @@ -0,0 +1,359 @@ +//! The personal-data erasure route and the dry run's personal-data findings, +//! against a real PostgreSQL. +//! +//! What is here, and why each one has to be an HTTP test: +//! +//! 1. **The receipt, and the retry.** `DELETE /api/v1/personal-data/{record}` +//! answers the receipt the first time and the same one after. +//! 2. **Erasing touches nothing else.** The passport read back after an erasure +//! is the one read back before it, statement and all. +//! 3. **The scope gate runs before the erasure.** A refused caller leaves the +//! record present — not merely a `403`. +//! 4. **`404` and `200` are told apart.** The pair CLAUDE.md names as worth a +//! test: a record this node never held, against one it held and erased. +//! 5. **The dry run says what create says.** A preview that passed a body create +//! refuses would send an operator into a draft that cannot be stored. + +#![cfg(feature = "integration-tests")] + +mod helpers; + +use dpp_dal::pg::PgPersonalDataRepo; +use dpp_domain::passport::PassportId; +use dpp_domain::ports::personal_data::{HeldRecord, PersonalDataPort}; +use helpers::{TestClient, make_jwt, make_jwt_scoped, start_postgres, start_vault}; +use serde_json::{Value, json}; + +const FIELD: &str = "repairHistoryUrl"; + +/// A textile draft whose `repairHistoryUrl` — a log of repairs to one specific +/// item, marked as able to hold personal data — has a value. +fn textile_with_a_repair_history(personal_data: Option) -> Value { + let mut body = json!({ + "productName": "Organic Cotton Tee", + "productGroup": "textile", + "manufacturer": {"name": "RouteTestCorp", "address": "Berlin, DE"}, + "materials": [], + "productGroupData": { + "carbonFootprintKgCo2e": 4.2, + "careInstructions": "Machine wash cold, line dry", + "chemicalComplianceStandard": "OEKO-TEX 100", + "countryOfOrigin": "IT", + "countryOfRawMaterialOrigin": "IN", + "disassemblyInstructions": "Remove label and neck tape before recycling", + "endOfLifeInstructions": "Return to a textile collection point", + "expectedWashCycles": 50, + "fibreComposition": [{"fibre": "cotton", "pct": 95}, {"fibre": "elastane", "pct": 5}], + "microplasticSheddingMgPerWash": 12, + "productGroup": "textile", + "productIdentifier": {"gtin": "09506000134369", "scheme": "gs1"}, + "productWeightGrams": 220, + "recyclabilityClass": "B", + "recycledContentPct": 30, + "sparePartsAvailable": false, + "waterUseLitres": 1800, + "repairHistoryUrl": "https://repairs.example.com/item/42", + }, + }); + if let Some(statements) = personal_data { + body["personalData"] = statements; + } + body +} + +fn admin(url: &str) -> TestClient { + TestClient::new(url, make_jwt("00000000-0000-0000-0000-0000000000a1")) +} + +/// A draft whose marked field is stated as held in `record`, with the record +/// held for it. +async fn a_draft_naming_a_record( + client: &TestClient, + store: &PgPersonalDataRepo, +) -> (String, String, Value) { + let created: Value = client + .post_json( + "/api/v1/dpp", + textile_with_a_repair_history(Some(json!({ FIELD: { "held": "nothing" } }))), + ) + .await + .json() + .await + .expect("created"); + let id = created["id"].as_str().expect("an id").to_owned(); + + let record = store + .store( + PassportId(id.parse().expect("uuid")), + FIELD, + &json!({ "owner": "named in the repair log" }), + ) + .await + .expect("held"); + + let statement = json!({ + FIELD: { "held": "outside", "lawfulBasis": "consent", "record": record.as_str() } + }); + let updated = client + .put_json( + &format!("/api/v1/dpp/{id}"), + json!({ "personalData": statement }), + ) + .await; + assert_eq!(updated.status(), 200, "a draft may name its record"); + (id, record.to_string(), statement) +} + +/// 🚨 The route that honours a withdrawn consent. The first call erases and +/// answers the receipt; the second — a caller that lost the first answer — +/// is handed the same one, with the same instant. +#[tokio::test(flavor = "multi_thread")] +async fn erasing_answers_the_receipt_and_a_retry_answers_the_same_one() { + let pg = start_postgres().await; + let url = start_vault(pg.dal.clone()).await; + let client = admin(&url); + let store = PgPersonalDataRepo::new(pg.dal.clone()); + let (id, record, _) = a_draft_naming_a_record(&client, &store).await; + + let first = client + .delete(&format!("/api/v1/personal-data/{record}")) + .await; + assert_eq!(first.status(), 200); + let first: Value = first.json().await.expect("receipt"); + assert_eq!(first["record"], record); + assert_eq!(first["passportId"], id); + assert_eq!(first["field"], FIELD); + assert!(first["erasedAt"].is_string(), "{first}"); + assert!( + !first.to_string().contains("named in the repair log"), + "nothing of the content is in the receipt: {first}" + ); + + tokio::time::sleep(std::time::Duration::from_millis(50)).await; + let again = client + .delete(&format!("/api/v1/personal-data/{record}")) + .await; + assert_eq!(again.status(), 200, "a retry is not an error"); + assert_eq!(again.json::().await.expect("receipt"), first); + + let held = store + .fetch(&dpp_domain::PersonalDataRecordId::new(record)) + .await + .expect("fetch"); + assert!(matches!(held, Some(HeldRecord::Erased(_))), "{held:?}"); +} + +/// Erasing must not touch the passport or its statement: the statement keeps +/// naming a record that is gone, which is how a reader learns it was erased +/// rather than never there. +#[tokio::test(flavor = "multi_thread")] +async fn erasing_leaves_the_passport_and_the_statement_naming_the_record() { + let pg = start_postgres().await; + let url = start_vault(pg.dal.clone()).await; + let client = admin(&url); + let store = PgPersonalDataRepo::new(pg.dal.clone()); + let (id, record, statement) = a_draft_naming_a_record(&client, &store).await; + + let before: Value = client + .get(&format!("/api/v1/dpp/{id}")) + .await + .json() + .await + .expect("passport"); + assert_eq!(before["personalData"], statement); + + let erased = client + .delete(&format!("/api/v1/personal-data/{record}")) + .await; + assert_eq!(erased.status(), 200); + + let after: Value = client + .get(&format!("/api/v1/dpp/{id}")) + .await + .json() + .await + .expect("passport"); + assert_eq!(before, after, "the passport is as it was, byte for byte"); + assert_eq!(after["personalData"], statement); +} + +/// A caller that cannot administer is refused before anything is erased: the +/// record is still present afterwards, so the refusal is not merely a status. +#[tokio::test(flavor = "multi_thread")] +async fn a_caller_without_the_admin_scope_erases_nothing() { + let pg = start_postgres().await; + let url = start_vault(pg.dal.clone()).await; + let store = PgPersonalDataRepo::new(pg.dal.clone()); + let (_id, record, _) = a_draft_naming_a_record(&admin(&url), &store).await; + let path = format!("/api/v1/personal-data/{record}"); + + for scope in ["write", "read"] { + let refused = TestClient::new(&url, make_jwt_scoped("op", scope)) + .delete(&path) + .await; + assert_eq!(refused.status(), 403, "a {scope}-scoped key must not erase"); + } + + let unauthenticated = reqwest::Client::new() + .delete(format!("{url}{path}")) + .send() + .await + .expect("request"); + assert_eq!(unauthenticated.status(), 401); + + let held = store + .fetch(&dpp_domain::PersonalDataRecordId::new(record)) + .await + .expect("fetch"); + assert!( + matches!(held, Some(HeldRecord::Present(_))), + "three refusals, and the record is still held: {held:?}" + ); +} + +/// `404` means this node never held the record. A record it held and erased is +/// not that, and a retry of it is `200`. +#[tokio::test(flavor = "multi_thread")] +async fn a_record_never_held_and_one_already_erased_are_told_apart() { + let pg = start_postgres().await; + let url = start_vault(pg.dal.clone()).await; + let client = admin(&url); + let store = PgPersonalDataRepo::new(pg.dal.clone()); + let (_id, record, _) = a_draft_naming_a_record(&client, &store).await; + + // One that could have been minted here, and one that never could. + for stranger in [uuid::Uuid::now_v7().to_string(), "never-minted".to_owned()] { + let resp = client + .delete(&format!("/api/v1/personal-data/{stranger}")) + .await; + assert_eq!(resp.status(), 404, "{stranger}"); + assert_eq!( + resp.headers() + .get("content-type") + .and_then(|v| v.to_str().ok()), + Some("application/problem+json") + ); + } + + assert_eq!( + client + .delete(&format!("/api/v1/personal-data/{record}")) + .await + .status(), + 200 + ); + assert_eq!( + client + .delete(&format!("/api/v1/personal-data/{record}")) + .await + .status(), + 200, + "erased is not unknown" + ); +} + +/// What a `422` from create names, so the dry run can be held to the same words. +async fn refusal(resp: reqwest::Response) -> (u16, Value) { + let status = resp.status().as_u16(); + let mut problem: Value = resp.json().await.expect("a problem document"); + // Per request, so two refusals of one body differ in it and in nothing else. + problem + .as_object_mut() + .expect("an object") + .remove("requestId"); + (status, problem) +} + +/// 🚨 A preview that passes a body create refuses sends an operator into a draft +/// that cannot be stored. Each finding the personal-data check makes is refused +/// by the dry run in the same words create uses, and the same body that clears +/// the check is accepted by both. +#[tokio::test(flavor = "multi_thread")] +async fn the_dry_run_refuses_what_the_personal_data_check_refuses_on_create() { + let pg = start_postgres().await; + let url = start_vault(pg.dal.clone()).await; + let client = admin(&url); + + let unstated = textile_with_a_repair_history(None); + let about_an_unmarked_field = textile_with_a_repair_history(Some(json!({ + FIELD: { "held": "nothing" }, + "productName": { "held": "nothing" }, + }))); + let on_a_basis_the_act_does_not_admit = textile_with_a_repair_history(Some(json!({ + FIELD: { "held": "outside", "lawfulBasis": "contract", "record": "r-1" }, + }))); + let naming_no_record = textile_with_a_repair_history(Some(json!({ + FIELD: { "held": "outside", "lawfulBasis": "consent", "record": "" }, + }))); + + for (what, body, field) in [ + ( + "no statement", + unstated, + "/productGroupData/repairHistoryUrl", + ), + ( + "a statement about a field that is not marked", + about_an_unmarked_field, + "/personalData/productName", + ), + ( + "a basis the governing act does not admit", + on_a_basis_the_act_does_not_admit, + "/personalData/repairHistoryUrl/lawfulBasis", + ), + ( + "an outside statement naming no record", + naming_no_record, + "/personalData/repairHistoryUrl/record", + ), + ] { + let created = refusal(client.post_json("/api/v1/dpp", body.clone()).await).await; + let previewed = refusal(client.post_json("/api/v1/dpp/validate", body).await).await; + + assert_eq!(created.0, 422, "create refuses {what}"); + assert_eq!( + previewed, created, + "the dry run says what create says for {what}" + ); + assert!( + created.1["errors"] + .as_array() + .expect("the findings are addressable") + .iter() + .any(|e| e["field"] == field), + "{what}: names {field}: {}", + created.1 + ); + } + + let stated = textile_with_a_repair_history(Some(json!({ FIELD: { "held": "nothing" } }))); + let preview = client + .post_json("/api/v1/dpp/validate", stated.clone()) + .await; + assert_eq!(preview.status(), 200, "a stated field clears the preview"); + assert_eq!( + client.post_json("/api/v1/dpp", stated).await.status(), + 201, + "and create" + ); +} + +/// The preview now builds the draft create builds, so it refuses a tariff code +/// create refuses. It passed one before, because that refusal lived in the +/// handler's own assembly of the passport. +#[tokio::test(flavor = "multi_thread")] +async fn the_dry_run_refuses_a_tariff_code_create_refuses() { + let pg = start_postgres().await; + let url = start_vault(pg.dal.clone()).await; + let client = admin(&url); + + let mut body = textile_with_a_repair_history(Some(json!({ FIELD: { "held": "nothing" } }))); + body["commodityCode"] = json!("not a tariff code"); + + let created = refusal(client.post_json("/api/v1/dpp", body.clone()).await).await; + let previewed = refusal(client.post_json("/api/v1/dpp/validate", body).await).await; + + assert_eq!(created.0, 422, "{}", created.1); + assert_eq!(previewed, created); +} diff --git a/ops/pg/0043_personal_data_record.sql b/ops/pg/0043_personal_data_record.sql new file mode 100644 index 00000000..16163c96 --- /dev/null +++ b/ops/pg/0043_personal_data_record.sql @@ -0,0 +1,110 @@ +-- ============================================================================ +-- 0043 — personal_data_record: personal data a passport's statement names, +-- held where it can be erased. +-- +-- A published passport is signed, frozen, archived and copied to the back-up +-- provider, so personal data it is not required to carry cannot live in it: a +-- withdrawn consent (GDPR Art. 7(3)) obliges erasure (Art. 17(1)(b)), and +-- nothing the passport is copied to can be erased from. The passport carries a +-- statement naming a record and the basis it is held on; this table is the +-- record. +-- +-- Its own table, and deliberately nothing else's neighbour: +-- +-- * not a column or a key of `passport.doc`, which is archived into +-- `passport_version`, hashed into the audit chain, sealed, snapshotted and +-- copied to the back-up provider; +-- * read by no resolver or view route, and by no query that builds one. The +-- only reader is `PgPersonalDataRepo`, which `dpp-vault` reaches through +-- core's `PersonalDataPort`. +-- +-- A row is in exactly one of two states, enforced below: +-- +-- * PRESENT — `content` holds the data, `erased_at` is NULL; +-- * ERASED — `content` is NULL, `erased_at` says when. The row stays as a +-- tombstone: the identifier, the passport and the field, and the +-- instant. None of the content. +-- +-- **A tombstone is not free of personal data.** A passport identifier can +-- relate to whoever owns the item, and GDPR Recital 26 counts information that +-- can be linked to a person as personal data. So the tombstone is protected +-- like the passport's individual-item data, and is kept only as long as the +-- controller has a basis to keep it — to show that it erased (Art. 5(2)) is +-- one; nothing in this table decides when that ends, and nothing deletes it. +-- +-- Erasure overwrites `content` in place. The database's own write-ahead log, +-- replicas and backups hold earlier row versions, and no application role can +-- reach them; how long those live is the operator's retention, not this table's. +-- +-- Single-tenant: no `operator_id` column. +-- ============================================================================ + +CREATE TABLE odal.personal_data_record ( + -- Minted by the store (UUID v7) and never reused. It goes into a signed + -- passport that outlives the record, so it is derived from nothing in the + -- content or about the person. + id UUID PRIMARY KEY, + + -- The passport the record relates to. A foreign key, so a record cannot be + -- held for a passport this node has no row for; passports are never deleted, + -- so it can never be the thing that blocks anything. + passport_id UUID NOT NULL REFERENCES odal.passport(id), + + -- The marked field, as a dotted path inside `productGroupData` + -- (`usageHistory.negativeEvents`). Whether it is marked is the passport's + -- check, made when the statement naming the record is written. + field TEXT NOT NULL CHECK (field <> ''), + + -- The data, in whatever shape the controller keeps it. NULL once erased. + content JSONB, + + stored_at TIMESTAMPTZ NOT NULL DEFAULT now(), + erased_at TIMESTAMPTZ, + + -- Exactly one of the two states. Without this a row could be erased and still + -- hold its content, or be present and hold none. + CONSTRAINT personal_data_record_one_state CHECK ( + (content IS NOT NULL AND erased_at IS NULL) + OR + (content IS NULL AND erased_at IS NOT NULL) + ) +); + +-- `records_for`: every record a passport has, in the order stored. +CREATE INDEX idx_personal_data_record_passport + ON odal.personal_data_record (passport_id, stored_at, id); + +-- Erasure is one-way. A record may change once — present to erased, content +-- dropped and the instant set, the other columns as they were — and a tombstone +-- never changes again: the receipt a retried erasure returns is the first one, +-- and an erased record is not brought back by anything the app role can run. +CREATE FUNCTION odal.personal_data_record_guard() RETURNS trigger +LANGUAGE plpgsql AS $$ +BEGIN + IF OLD.erased_at IS NOT NULL THEN + RAISE EXCEPTION 'ODAL_PERSONAL_DATA: an erased record is a tombstone and cannot change'; + END IF; + IF NEW.content IS NOT NULL + OR NEW.id <> OLD.id + OR NEW.passport_id <> OLD.passport_id + OR NEW.field <> OLD.field + OR NEW.stored_at <> OLD.stored_at THEN + RAISE EXCEPTION 'ODAL_PERSONAL_DATA: a record can only be erased, not rewritten'; + END IF; + RETURN NEW; +END; +$$; + +CREATE TRIGGER personal_data_record_guard + BEFORE UPDATE ON odal.personal_data_record + FOR EACH ROW EXECUTE FUNCTION odal.personal_data_record_guard(); + +-- 0010's ALL-TABLES grant was a one-time snapshot; tables added later need their +-- own (same pattern as 0017/0021/0022/0023/0028/0038/0039). +-- +-- **No DELETE.** Erasure is an UPDATE that drops the content and keeps the row, +-- because the tombstone is the controller's proof that it erased — so this +-- table does not join the DELETE set in ops/pg/README.md. UPDATE is granted on +-- the two columns erasure writes and no others. +GRANT SELECT, INSERT ON odal.personal_data_record TO odal_app; +GRANT UPDATE (content, erased_at) ON odal.personal_data_record TO odal_app;