Repository navigation
feat(datastore): preserve history through legacy bucket aliases - #836
TimeToBuildBob wants to merge 48 commits into
Conversation
…_id,name)) Adds `device_id` to the `buckets` table so buckets from different devices can share a bucket name without colliding. The new uniqueness constraint is `UNIQUE(device_id, name)`, replacing the old `UNIQUE(name)`. Migration (v6→v7): - Recreates the buckets table with the new schema (SQLite cannot alter UNIQUE constraints in-place). - Stamps all existing rows with `device_id = 'local'` (locally-created). - Also drops the stale `data_deprecated` column (unused since v3). - FK enforcement is disabled around the recreation to avoid the immediate events→buckets FK check firing on DROP TABLE mid-batch. Read-only opens (aw-sync peer DBs at v4-v6): - `get_stored_buckets` falls back to `'local'` literal when `db_version < 7`, so aw-sync can still read older peer databases without migrating them. `create_bucket` stamps `device_id = "local"` for any bucket created without an explicit device_id (local watchers). Peer buckets imported via aw-sync carry their own device_id. Also fixes pre-existing duplicate `#[allow]` attributes in aw-query that were breaking the clippy pre-commit hook under the current toolchain. Git-Session-Id: 77b9
Git-Session-Id: c66cd073-07ec-4980-883b-2f3c24dd5be1
Complete the fix across all test and benchmark Bucket struct literals that weren't updated when device_id was added to the model. Git-Session-Id: da0eaaaf-bdea-5882-addc-bba90b82acea
buckets_cache is keyed by bucket name only. A second insert with the same name but a different device_id would silently overwrite the first cache entry, hiding its history and routing new events to the wrong bucket. Reject until the cache key includes device_id (follow-up refactor). Git-Session-Id: da0eaaaf-bdea-5882-addc-bba90b82acea
…te rejection Add three tests for the storage guarantees introduced by the device_id column: - empty device_id is stamped "local" on creation - explicit device_id survives the round-trip - same-name bucket with a different device_id is rejected (cache guard) Git-Session-Id: da0eaaaf-bdea-5882-addc-bba90b82acea
…new_in_memory(false) Git-Session-Id: dde9bcf8-3b8e-589f-bdc8-7b22fa16c358
…ckets The guard added in d446f37 rejected same-name buckets with a different device_id to protect the name-keyed buckets_cache, but UNIQUE(device_id, name) exists precisely to allow (A, "x") and (B, "x") to coexist — the guard made the feature unusable for its stated purpose. Remove the guard and let the UNIQUE constraint be the sole duplicate guard (same device_id + name still maps to BucketAlreadyExists). Test flipped from asserting rejection to asserting coexistence plus the same-device duplicate rejection path. Git-Session-Id: 836f0e43-c054-59df-bd31-567140d6ae5d
Git-Session-Id: 9295d79d-f715-58c1-aca0-8588355870ff
Git-Session-Id: 18163d54-c6a9-5fd9-a7c3-e85d1aed668f
Git-Session-Id: b59ff0c6-06f5-5ed5-8eaa-6141ff07f3d9
Git-Session-Id: b59ff0c6-06f5-5ed5-8eaa-6141ff07f3d9
… migration lock - rename_bucket checks the destination name inside the UPDATE so a stale cache cannot create an ambiguous (device_id, name) pair - v6->v7 migration carries the AUTOINCREMENT high-water mark over the table rebuild so deleted bucket ids are never reused - re-read user_version after taking the exclusive lock so a concurrent opener's finished migration is not rebuilt over its device IDs Git-Session-Id: b51e5cce-319b-4ec1-8c25-2a1eb9a205e6
After merging master (ActivityWatch#776), _infer_db_version returned 2 for the v7 schema: v7 drops data_deprecated, which the v2/v3 check reads as 'no rename yet'. Recognise device_id first. The early-schema dump-restore test now asserts v7 dropped data_deprecated instead of preserving it. Git-Session-Id: 0c1512d2-dbce-4257-97f6-9b08bbe8dcb8
Git-Session-Id: d8c65c8d-a38a-5d02-bbdc-63034939f80f
Git-Session-Id: b01e1b01-a683-5ca8-805e-c6dd7d8436c0
A concurrent opener that read v5 before the first opener finished the v7 migration would regress user_version back to 6 (or 5) when its own migration step finally took the exclusive lock, causing the v7 rebuild to re-run and overwrite stored device IDs with 'local'. Mirror the existing v6->v7 in-lock recheck in _migrate_v4_to_v5 and _migrate_v5_to_v6: after taking the lock, skip the step if the version is already applied. Git-Session-Id: ea4f2eca-681c-5fde-b350-3502103e7d4c
Git-Session-Id: 9749f553-4dc2-5998-9ffd-3d98a62bae37
…story Git-Session-Id: 0773711b-88a1-4a62-aa84-a1721b7a3cc2
Git-Session-Id: 0773711b-88a1-4a62-aa84-a1721b7a3cc2
[High impact] The changes reach bucket storage and the server’s device-scoped data routes. The PR needs fixes before merging because tests cannot compile and device-scoped writes can remove another device's events. Findings
SummaryAdds hostname aliases and device-scoped APIs while preserving stored bucket names.
Diagram%%{init: {'theme': 'neutral'}}%%
flowchart LR
Request[Device-scoped request] --> Normalize[Normalize local UUID]
Normalize --> Handle[Scoped datastore handle]
Handle --> Worker[Serialized worker]
Worker --> Resolve[Resolve exact name or hostname alias]
Resolve --> Bucket[Stored bucket row]
Bucket --> Events[Read or write events]
Worker --> Export[Grouped export]
Export --> UUID[Replace local sentinel with source UUID]
UUID --> File[Stage JSON in private tempfile]
Reviews (2) · Last reviewed commit: "Merge origin/master into feat/device-add..." · Reviewed by Greptile |
| "UPDATE buckets SET name = ?1 WHERE name = ?2 | ||
| AND NOT EXISTS (SELECT 1 FROM buckets WHERE name = ?1)", |
There was a problem hiding this comment.
Rename redirects existing writes
rename_bucket checks only exact destination names. With window_host on one device and other on another, renaming other to window succeeds even though window already points to window_host. Exact-name precedence then sends later window writes to the renamed bucket, splitting the original history. Apply the alias-shadowing check used by create_bucket before renaming.
Knowledge Base Used: Datastore
There was a problem hiding this comment.
Fixed in 96b1e34, pushed before this review completed. Rename now refreshes stale bucket metadata and resolves the destination before updating it, refusing a destination that resolves to a different stored history. The regression renaming_a_peer_cannot_redirect_an_active_alias covers this exact case and verifies that window still resolves to window_host. The implementation session recorded the regression failing before the fix and the datastore/query/server suite passing afterward.
| match crate::bucket_resolution::resolve_bucket( | ||
| self.buckets_cache.values(), | ||
| &bucket.id, | ||
| Some(&bucket.device_id), | ||
| ) { | ||
| Ok(existing) => return Err(DatastoreError::BucketAlreadyExists(existing.id)), |
There was a problem hiding this comment.
Create peer window with hostname other, then local window_host with hostname host; both are allowed. A local POST for window returns 304 for the local alias window_host here, before checking the global target. But subsequent heartbeats for window select the peer's exact bucket, putting local activity into the peer's history. Before returning BucketAlreadyExists for a scoped alias, check the global target and return a conflict when the targets differ.
Knowledge Base Used: HTTP API and endpoints
There was a problem hiding this comment.
Fixed in 1216619 and retained in 805d908. Before returning BucketAlreadyExists, unscoped creation now verifies that scoped and global resolution select the same row; otherwise it returns AmbiguousBucket. The regression local_create_cannot_acknowledge_an_alias_that_global_writes_redirect failed before the repair and passes afterward. Device-declared imports canonicalize within their selected device before merging. The global duplicate-name guard remains intact.
| if let Some(bucket) = self.buckets_cache.get(bucket_id) { | ||
| return Ok(bucket.clone()); | ||
| } | ||
| crate::bucket_resolution::resolve_bucket(self.buckets_cache.values(), bucket_id, None) |
There was a problem hiding this comment.
Alias lookup now depends on hostname, but the bucket signature does not include it. If an open reader has cached a bucket with hostname unknown, migrate_hostname("host") refreshes only the writer's cache. The reader sees an unchanged signature and keeps rejecting the new hostname alias, even though the writer accepts it. Include hostname in the signature so readers refresh after this supported metadata change.
Knowledge Base Used: Datastore persistence and workers
There was a problem hiding this comment.
Fixed in 1216619. The reader bucket signature includes encoded hostname as well as name, count and maximum row ID, so migrate_hostname repairs invalidate an already-open reader cache. reader_refreshes_aliases_after_hostname_repair reproduced NoSuchBucket before the change and passes afterward. The datastore/query/server suites pass at current head 805d908.
Git-Session-Id: 1fe79215-f972-41ee-9d1a-2ef78e3f9410
…ests Git-Session-Id: 1fe79215-f972-41ee-9d1a-2ef78e3f9410
Git-Session-Id: 1fe79215-f972-41ee-9d1a-2ef78e3f9410
|
The 2/5 summary reviews def5ea9, not current head 1e0a3c5. Its three findings already have pushed repairs and individual replies: rename alias protection in 96b1e34; creation/write target agreement and hostname-sensitive reader refresh in 1216619, retained through the later cache/routing changes. I checked the current source and all three named regression tests; I did not rerun tests in this monitoring pass. The implementation owner is actively extending this draft in the same leased worktree. I left that work untouched and handed back the review refresh for the next verified push. No additional fix or Greptile retrigger from this dispatch; the old score is not evidence that the current head has passed review. |
Git-Session-Id: 73203c26-75f4-4756-b450-2d9f5877e9fc
Git-Session-Id: 73203c26-75f4-4756-b450-2d9f5877e9fc
Git-Session-Id: 73203c26-75f4-4756-b450-2d9f5877e9fc
Git-Session-Id: 73203c26-75f4-4756-b450-2d9f5877e9fc
Git-Session-Id: ee485138-49ef-455b-9c15-d08a96a24bd7
Single conflict in aw-datastore/src/legacy_import.rs: keep device-addressing merge-into-canonical-id (existing.id) and master's ActivityWatch#822 Import error returns instead of panicking. Git-Session-Id: 0bfc
|
@greptileai review |
|
Merged Resolution: keep device-addressing merge-into-canonical-id ( Local verification: |
Master's merge_subwatcher_fields and legacy_clean tests landed without the device_id field this branch added to Bucket. Restore compile after the origin/master merge. Git-Session-Id: 0bfc
|
Follow-up |
| #[serde(default)] | ||
| pub device_id: String, |
There was a problem hiding this comment.
Adding Bucket.device_id leaves two Rust struct literals without the required field: editor_bucket in aw-query/tests/query.rs and the test helper in aw-sync/src/legacy_clean.rs. #[serde(default)] only covers JSON reads; it does not fill Rust struct literals. Both test targets fail to compile. Add device_id to both literals.
| bucket::bucket_events_create_in( | ||
| bucket_id, | ||
| events, | ||
| state, | ||
| &state.bucket_datastore(Some(device_id)), | ||
| ) |
There was a problem hiding this comment.
Scoped writes remove peer events
The new device-scoped event route accepts an event id owned by another device. insert_events uses INSERT OR REPLACE against the globally unique events.id, so posting that ID under device B deletes device A's row and recreates it in B's bucket. Selecting B's bucketrow does not constrain which row SQLite replaces.
Reject IDs owned by another bucket before replacing them. The heartbeat insert path needs the same protection.
| bucket::bucket_new_in( | ||
| bucket_id, | ||
| message, | ||
| state, | ||
| &state.bucket_datastore(Some(device_id)), | ||
| state.normalize_device(device_id), |
There was a problem hiding this comment.
Creation acknowledges another device's bucket
A scoped bucket POST returns 304 Not Modified when only another device owns the exact name. The global-name guard returns BucketAlreadyExists, and the shared handler maps that to 304. But a GET or heartbeat under the requested device returns 404, so the client is told its bucket exists when it cannot use it.
Keep the global-name guard, but return a conflict when the existing bucket belongs to a different device.
|
Convergence adjudication deferred by the freshness gate. Rechecked 2026-10-23 17:44 UTC: current head No findings classified, dismissed or resolved in this adjudication pass; no review retrigger or merge. The detector reports Updated this existing deferral rather than adding another status comment. Earlier checks at |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #836 +/- ##
===========================================
+ Coverage 70.81% 87.03% +16.22%
===========================================
Files 51 93 +42
Lines 2916 14612 +11696
===========================================
+ Hits 2065 12718 +10653
- Misses 851 1894 +1043 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Git-Session-Id: c2d218fa-690b-4e77-96e3-959267ad65f3
Git-Session-Id: c2d218fa-690b-4e77-96e3-959267ad65f3
|
Integration update at ee18948 (fix 6f9efed): reproduced both current correctness findings before repair. A forged explicit event ID moved a different device’s event (200); guarded peer creation returned 304 although scoped GET was 404. Whole-batch ID ownership preflight now precedes insertion, covering event and heartbeat inserts while preserving same-bucket replacement. Rejected batches leave no accepted prefix. Guarded peer creation and import collision return 409. Datastore/query/server suites and production-target strict Clippy pass; 14 device HTTP tests and 14 resolver fixtures pass. ee18948 verifies scoped alias privacy uses the stored bucket name without filtering a different peer name. Compilation was already repaired by a3c3d64. Both duplicate-name guards remain; no full same-name acceptance or release claim. Shared real-HTTP fixtures in ActivityWatch/aw-server#200 pass both transfer directions and alias/query/heartbeat behavior (8 cases). |
Git-Session-Id: c2d218fa-690b-4e77-96e3-959267ad65f3
Git-Session-Id: c2d218fa-690b-4e77-96e3-959267ad65f3
Git-Session-Id: c2d218fa-690b-4e77-96e3-959267ad65f3
|
Further guarded acceptance is pushed through a3bca14. 1a16373 keys heartbeat cache/invalidation/privacy fallback by (device_id, stored id); an internal same-name fixture reproduced cross-device cached event-ID failure before repair. 9a81923 includes ownership in reader signatures (old schemas remain supported), with real open-reader ownership-repair coverage. a3bca14 propagates signature/reload failures instead of serving stale identity: a real-file fixture reproduces a rejected ambiguous catalogue, verifies reads/list fail closed, then verifies recovery after repair. Full datastore/query/server suites and strict production-target Clippy pass; 16 resolver fixtures pass. Both duplicate-name guards remain. Remaining work is coherent same-name legacy-index/list/export/rename integration plus the complete acceptance matrix. #826 remains separately owned; its delete/rename invalidation will need the pair keys. No production migration or complete identity claim. |
Git-Session-Id: e9187c4d-6f8e-4dcd-8348-b5182b82f9d2
Git-Session-Id: e9187c4d-6f8e-4dcd-8348-b5182b82f9d2
Git-Session-Id: 6ce10d23-d723-441f-a1a9-a57de198bdd8
Git-Session-Id: 6ce10d23-d723-441f-a1a9-a57de198bdd8
Git-Session-Id: 6ce10d23-d723-441f-a1a9-a57de198bdd8
Git-Session-Id: 6ce10d23-d723-441f-a1a9-a57de198bdd8
Git-Session-Id: c24867b5-ab58-475a-a9fd-78ac3d0e096f
Git-Session-Id: c24867b5-ab58-475a-a9fd-78ac3d0e096f
Git-Session-Id: c24867b5-ab58-475a-a9fd-78ac3d0e096f
Git-Session-Id: c24867b5-ab58-475a-a9fd-78ac3d0e096f
|
Guarded continuation pushed through d9b1805. Three reproduced repairs: 8c95cad scopes hostname repair and verifies open-reader aliases; 3e9f4ec retains local authority through legacy Python import/canonicalization/dedup with caller-scope restoration; f47b8c9 refreshes migration destinations before mutation, rejecting a newly created peer destination without changing history. d9b1805 adds exact v6→v7 row/name/archive/event-association preservation, including pre-epoch nanoseconds and binary archive bytes; existing migration code passes. Local verification: datastore/query/server 348 passed / 2 ignored, strict production-target Clippy and formatting passed; workspace test targets compile; aw-sync 221 passed; eight shared real-HTTP Python/Rust cases pass (distinct stored names sharing aliases). Normal hooks passed. Both duplicate-name guards remain; this is not enabled same-name coexistence, completed identity acceptance, upstream release or production migration. Contributor-owned #826 remains separate and its eventual invalidation integration must use pair keys. Deferred recovery retains fresh rollback backups on incomplete attempts; no completion markers or retained history were weakened. |
Git-Session-Id: f5ebad7c-cb7d-440b-ad2a-d8782ae33b6e
Git-Session-Id: f5ebad7c-cb7d-440b-ad2a-d8782ae33b6e
Git-Session-Id: f5ebad7c-cb7d-440b-ad2a-d8782ae33b6e
Git-Session-Id: c81281b2-9cbe-4e63-a85e-e227184324d1
Summary
Device-identity integration for ActivityWatch/activitywatch#302, stacked on schema #786 at
d74c27eb425b. Do not merge before #786. Remains draft: Python parity and full identity acceptance are incomplete.(device_id, id)bucket cache; scoped worker requests retain scope through event/list/export/query operations and restore it after errors. Reader signatures include hostname repairs.localselect the same storage namespace; peerhostname="!local"is rejected.87a8619: complete local-plus-stored-peer catalogue and lossless grouped list/export/import. Grouped envelopes use{"devices":{"<source-device-id>":{"buckets":{"<bucket-id>":{...}}}}}. Local sentinel is exported as source UUID, so importing on another server cannot silently localize source history. Events serialize one row at a time from one worker transaction to a private tempfile before HTTP success. Full envelope identity preflight precedes writes; runtime failures can leave earlier groups imported, with cache invalidation on failure. Peer hostname is a deterministic representative of bucket metadata, not a latest-name registry.Routes
GET /api/0/devices— local UUID (including empty server) and stored peersGET /api/0/devices/buckets— grouped metadataGET /api/0/devices/export,POST /api/0/devices/import— grouped transfer/api/0/devices/{device_id}/buckets— GET/api/0/devices/{device_id}/buckets/{id}— GET/POST/DELETE/events,/events/{event_id},/events/count,/events/deleteand/heartbeat?pulsetime=.../api/0/devices/{device_id}/buckets/{id}/exportand/export/csv/api/0/devices/{device_id}/export,/import,/queryBoth global duplicate-name creation and duplicate-cache-load guards remain. Still missing: Python addressing parity, enabled same-name coexistence and full backend-specific migration/reader acceptance. Watcher rollout and V2 segment import remain separate follow-ups. No production database or published package changed.
Verification at
87a8619cargo test -p aw-datastore -p aw-query -p aw-server: passed, including 11 device HTTP fixtures. Additional configured API-key fixture covering all four grouped routes passed separately.cargo clippy -p aw-datastore -p aw-query -p aw-server --lib --bins -- -D warnings, formatting, whitespace and normal commit hooks passed. Web UI not built. Strict all-target Clippy is not claimed green (pre-existing query test-fixture warning).!localmetadata, import UUID normalization and collision status.make test-fast: 1505 passed / 1 skipped;make typecheck: 286 package and 1803 script files passed. These are not Rust coverage. Upstream current-head CI/review remains separate.The Greptile 2/5 summary reviews
def5ea9, not current head. All three named findings already have individual replies and pushed repairs: rename protection96b1e34, creation/write agreement and hostname-sensitive reader refresh1216619, retained through later integration. Heartbeat rename/delete invalidation is separately owned by #826.