Skip to content

fix(export): retain exact archive events through device transfer - #201

Draft
TimeToBuildBob wants to merge 11 commits into
ActivityWatch:masterfrom
TimeToBuildBob:feat/device-archive-fidelity-78c9
Draft

TimeToBuildBob wants to merge 11 commits into
ActivityWatch:masterfrom
TimeToBuildBob:feat/device-archive-fidelity-78c9

Conversation

@TimeToBuildBob

@TimeToBuildBob TimeToBuildBob commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Archive-fidelity continuation for ActivityWatch/activitywatch#302. Stacked on draft #200 at d47d930; depends on draft ActivityWatch/aw-core#184 at exact 4a04a57, itself stacked on aw-core#183/#177. Incremental server commit: a0cdb62. Do not merge before these prerequisites; no production/release claim.

  • Pin/lock the core precision, pre-epoch export and creation/snapshot correctness repairs.
  • Real SQLite HTTP fixtures retain exact timestamps and durations for 1960 and 2300 through scoped JSON/CSV/grouped export, deletion/reimport, source ownership and repeated-import deduplication.
  • Update the existing CSV test to expect the original .123456 input instead of the previous lossy .123000 output.

Verification

  • Both new HTTP cases failed against the previous installed core (.123457 truncated to .123000), then passed with the new dependency.
  • Fresh Poetry lock/install from the pushed Git SHA, no PYTHONPATH override: repository make test 145 passed, make typecheck 31 files passed. Black, Ruff and whitespace checks passed.
  • Core final suite 428 passed / 5 skipped, mypy 37 files; six reproduced prerequisite P1 regression cases pass in the new core. Existing core feat(api): add GET /api/0/devices endpoint (bucket identity phase 2.2) #183 itself remains unchanged until those commits are integrated.
  • Brain fast/type pass 1505 passed / 1 skipped, 286 package / 1803 script files. These do not replace package verification.
  • All databases temporary and isolated. No production service or data touched.

Remaining boundary

This does not complete shared Rust/Python HTTP/migration/same-name/privacy/reader-refresh acceptance. Python supports microseconds, not the entire Rust nanosecond domain; legacy REAL raw-row SQL comparison caveats are documented in the core PR. Both Rust duplicate-name guards remain. Watcher adoption and V2 import remain separate follow-ups. The branch leases on #183/#200 were respected; these sibling drafts are explicit integration candidates, not duplicate merge-ready contracts.

Final continuation (head d97c1b2)

Includes #200's existing same-batch deduplication repair (0278a28, cherry-picked as ffcd3f9) and now pins core 538886d, which excludes SQLite WAL/SHM companions and backup filenames from migration-source discovery. Locked Poetry install with no PYTHONPATH: repository tests 146 passed, mypy 31 files. Core final 429 passed / 5 skipped, mypy 37 files. Both source follow-ups remain drafts, integration and the final cross-backend acceptance matrix remain outstanding.

Three-device HTTP acceptance (d2381cf, current head)

Added real SQLite HTTP reload, scoped event/count/query, grouped export, scoped deletion and already-open-reader invalidation checks for three devices sharing window. Legacy flat list/export returns 409. The fixture fails at third-device creation against the old core and passes with the pinned fix. Explicitly flushes the existing lazy writer before opening a second independent app. Final package slice 147 passed, mypy 31 files; fourth brain fast/type 1505/1 skipped, 286/1803 files. No full cross-backend completion implied.

Verified current head 5b12d1b

Moved the two new DELETE calls outside assertions after CodeQL reported side effects. Final installed package tests 147 passed, mypy 31 files, Black and whitespace clean; fifth brain fast/type 1505/1 skipped, 286/1803 files. All eight current-head CI checks passed, including CodeQL. Core dependency 538886d has all six checks passing. This is source/CI evidence only; the draft integration and Rust/shared acceptance boundaries above remain.

Mirror aw-server-rust's /api/0/devices endpoint in Python aw-server.
Returns the local device id (UUID, stable across restarts) and hostname.
Paired with aw-core#177 which adds device_id to the SQLite schema.

Part of ActivityWatch/activitywatch#302 (bucket identity unification).

Git-Session-Id: 02ce
…stability

Git-Session-Id: aa774620-bec5-581f-adf4-2c8cc3cf4eb6
Git-Session-Id: 931478ca-2d2b-48c5-bb99-0bb5ad608478
Git-Session-Id: 931478ca-2d2b-48c5-bb99-0bb5ad608478
Git-Session-Id: 931478ca-2d2b-48c5-bb99-0bb5ad608478
Device import now deduplicates existing same-device events, matching
Rust, and merges aliases into the stored row instead of rejecting.
Grouped export pins a read-only SQLite snapshot so staging no longer
holds the API write lock. Pins aw-core#183 at 4ac2146.

Git-Session-Id: aeaa4ffa-df33-4fe0-8487-f35c72996602
Git-Session-Id: 0dc232f4-6721-4652-9e55-bd51fe8eec69
Existing-bucket merge compared each incoming event only against stored
history, so two identical new events in one envelope both reached
create_events. Record accepted identities as they land, and execute
DELETE requests before asserting their status in the HTTP tests.

Git-Session-Id: 72dcece2-03b4-59da-979e-db600720e05b
Comment thread tests/test_device_addressing.py Fixed
Comment thread tests/test_device_addressing.py Fixed
Comment thread tests/test_device_addressing.py Fixed
@greptile-apps

greptile-apps Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] The changes reach the Python server’s device API and stored event archives.

Fix duplicate entries within a single device import before merging.

Findings

  1. P1 Imports add duplicate events ▶

Summary

Adds device-scoped bucket operations and grouped archive transfer, backed by a pinned draft aw-core.

  • Callers can address buckets and events by device.
  • Device archives carry their owners and events between servers.
  • Archive transfers keep the event times and durations they receive.

TimeToBuildBob explicitly deferred shared Rust/Python parity, migration, same-name, privacy, and reader-refresh acceptance. TimeToBuildBob also identified nanosecond support, watcher adoption, and V2 import as outside this draft. These acknowledged boundaries are not reported as findings.

Review checks were static; no runtime tests were run.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Device-scoped HTTP request] --> B[Request-local API view]
  B --> C[Scoped SQLite datastore]
  C --> D[Read snapshot]
  D --> E[Private temporary archive]
  E --> F[HTTP download]
  F --> G[Grouped device import]
  G --> H{Bucket already exists?}
  H -->|Yes| I[Filter stored event identities]
  H -->|No| J[Create bucket and insert events]
  I --> K[Insert accepted events]
Loading

Reviews (1) · Last reviewed commit: "fix(export): retain exact archive events..." · Reviewed by Greptile

Comment thread aw_server/api.py Outdated
Comment on lines +265 to +269
new_events = [
event
for event in incoming
if self._import_event_identity(event) not in existing
]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Imports add duplicate events

_merge_imported_events checks every incoming event against the same set of stored events. When an existing bucket receives [e, e] and neither copy is stored yet, both pass the filter and are inserted. This breaks the promise to skip events with the same timestamp, duration, and data, and can inflate event counts and time totals.

Add each accepted identity to the set before checking the next event, and test repeated entries within one import.

Suggested change
new_events = [
event
for event in incoming
if self._import_event_identity(event) not in existing
]
new_events = []
for event in incoming:
identity = self._import_event_identity(event)
if identity not in existing:
existing.add(identity)
new_events.append(event)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already fixed on this branch in ffcd3f9 (cherry-pick of the existing #200 repair 0278a28). Accepted identities enter the set immediately; test_import_skips_duplicate_new_events_in_one_batch passes. Current locked-install suite is 147 passed, mypy31. The review examined a0cdb62, before that integration; no second implementation was added.

Git-Session-Id: 0dc232f4-6721-4652-9e55-bd51fe8eec69
Git-Session-Id: 0dc232f4-6721-4652-9e55-bd51fe8eec69
Comment thread tests/test_device_addressing.py Fixed
Git-Session-Id: 0dc232f4-6721-4652-9e55-bd51fe8eec69
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants