docs: document Smart Memories and sync the backend directory tables - #1433
docs: document Smart Memories and sync the backend directory tables#1433rohan-pandeyy wants to merge 4 commits into
Conversation
- New `backend/backend_python/memories.md`: schema, curation triggers, scoring signals, API, pipeline hooks and preferences. Added to the nav. - Rewrote `frontend/memories.md`, which described the removed implementation (DBSCAN location clustering, reverse geocoding, `/api/memories/*`). - `database.md`: the four memories tables, their keys, cascades and indexes. - `directory-structure.md`: the new modules, the rewritten memories router, and the semantic label tables, which are no longer dormant. - `image-processing.md`: capture-date extraction from EXIF sub-IFDs, Takeout sidecars and video container boxes, and the `date_source` field. - `semantic-search.md`: corrected the pipeline ordering and recorded memory curation as a consumer of embeddings and event labels. - `features.md`, `architecture.md`, `state-management.md`, `ui-components.md` updated to match. Admonition bodies are indented so MkDocs renders them inside the box.
Adds the modules that were never listed: videos and video_frames across database/routes/schemas/utils, the connection context manager, the model and session registries, the semantic vocabulary utils and scripts, and the hardware, model-bootstrap, face-search and face-quality helpers. Drops `microservice.py`, which no longer exists.
|
|
|
Warning Review limit reached
Next review available in: 36 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
WalkthroughThis documentation-only change adds Smart Memories backend and frontend references, updates semantic-search and image-processing documentation, refreshes backend directory descriptions, revises architecture and feature overviews, and adds the backend Memories page to MkDocs navigation. ChangesSmart Memories backend
Frontend Memories
Overview and navigation
Estimated code review effort: 2 (Simple) | ~15 minutes Suggested labels: Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 7
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/backend/backend_python/database.md`:
- Around line 168-171: Update the documentation paragraph describing
images.captured_at and videos.captured_at to scope the non-null requirement
specifically to candidate-media queries over those tables. Remove the claim that
every memories query filters captured_at IS NOT NULL, and clarify that
generated-memory reads use fields such as surface_date and status while
date_created continues to drive gallery display.
In `@docs/backend/backend_python/directory-structure.md`:
- Line 77: Update the folders.py description in the directory structure table to
remove the claim that SigLIP2 embedding is triggered last; describe it as
occurring after YOLO/face processing and before semantic scoring, memory
curation, and video processing, consistent with the semantic-search
documentation and project requirements.
In `@docs/backend/backend_python/semantic-search.md`:
- Around line 357-361: Update the Smart Memories description to state that only
stored image-embedding reads filter by model_version; describe event-label reads
as selecting from the active vocabulary and scored image_classes rows. Verify
the revised documentation against the project requirements and objectives.
- Line 627: Update the documentation entry for
image_util_process_unembedded_images to say the vision session is always closed
even if embedding inference raises, replacing the inaccurate reference to
scoring; verify the wording against the project’s documentation requirements and
objectives.
In `@docs/frontend/memories.md`:
- Around line 226-229: Update the Testing section in the memories documentation
to include the existing Redux slice suite,
frontend/src/features/__tests__/memoriesSlice.test.ts, alongside the listed
MemoryStoryViewer and useRefreshMemories tests.
In `@docs/frontend/ui-components.md`:
- Around line 115-122: Update the MemorySettings documentation to distinguish
persistence events by control type: describe the ToggleRow switches as writing
on their switch-change event, while retaining onValueCommit for slider changes.
Keep the existing descriptions of controls, clamping, and save-state disabling
unchanged.
In `@docs/overview/architecture.md`:
- Around line 34-37: Update the new architecture paragraph to use the canonical
“SQLite” capitalization instead of “sqlite” when referring to the database,
leaving the surrounding wording unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: eb770f0e-2649-4cfe-8fec-fba7ed2d0b93
📒 Files selected for processing (11)
docs/backend/backend_python/database.mddocs/backend/backend_python/directory-structure.mddocs/backend/backend_python/image-processing.mddocs/backend/backend_python/memories.mddocs/backend/backend_python/semantic-search.mddocs/frontend/memories.mddocs/frontend/state-management.mddocs/frontend/ui-components.mddocs/overview/architecture.mddocs/overview/features.mdmkdocs.yml
| ## Testing | ||
|
|
||
| ### Backend Tests | ||
|
|
||
| Located in `backend/tests/`: | ||
|
|
||
| - 100 unit tests covering all routes | ||
| - Run with: `pytest tests/` | ||
|
|
||
| ### Frontend Tests | ||
|
|
||
| Located in `frontend/src/pages/__tests__/`: | ||
|
|
||
| - Page rendering tests | ||
| - Run with: `npm test` | ||
|
|
||
| ### Manual Testing | ||
|
|
||
| Use `backend/test_memories_api.py` for API endpoint testing: | ||
|
|
||
| ```bash | ||
| python test_memories_api.py | ||
| ``` | ||
|
|
||
| ## Performance Considerations | ||
|
|
||
| 1. **Lazy Loading**: Images load on-demand with `loading="lazy"` | ||
| 2. **Thumbnail Optimization**: Uses Tauri's `convertFileSrc()` for efficient file access | ||
| 3. **Redux Memoization**: Uses `React.memo()` for card components | ||
| 4. **Efficient Queries**: SQLite indexes on `latitude`, `longitude`, `captured_at` | ||
| 5. **Background Processing**: Memory generation runs asynchronously | ||
| - `components/Memories/__tests__/MemoryStoryViewer.test.tsx` | ||
| - `hooks/__tests__/useRefreshMemories.test.tsx` |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Include the Redux slice test in the testing index.
The supplied context includes frontend/src/features/__tests__/memoriesSlice.test.ts, which covers critical viewer-state behavior, but this testing section lists only the viewer and refresh-hook suites. Add the slice test so the documentation accurately reflects existing coverage.
As per path instructions, test code must be automated, comprehensive, and cover critical functionality.
Suggested documentation update
- `components/Memories/__tests__/MemoryStoryViewer.test.tsx`
- `hooks/__tests__/useRefreshMemories.test.tsx`
+- `features/__tests__/memoriesSlice.test.ts`📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| ## Testing | |
| ### Backend Tests | |
| Located in `backend/tests/`: | |
| - 100 unit tests covering all routes | |
| - Run with: `pytest tests/` | |
| ### Frontend Tests | |
| Located in `frontend/src/pages/__tests__/`: | |
| - Page rendering tests | |
| - Run with: `npm test` | |
| ### Manual Testing | |
| Use `backend/test_memories_api.py` for API endpoint testing: | |
| ```bash | |
| python test_memories_api.py | |
| ``` | |
| ## Performance Considerations | |
| 1. **Lazy Loading**: Images load on-demand with `loading="lazy"` | |
| 2. **Thumbnail Optimization**: Uses Tauri's `convertFileSrc()` for efficient file access | |
| 3. **Redux Memoization**: Uses `React.memo()` for card components | |
| 4. **Efficient Queries**: SQLite indexes on `latitude`, `longitude`, `captured_at` | |
| 5. **Background Processing**: Memory generation runs asynchronously | |
| - `components/Memories/__tests__/MemoryStoryViewer.test.tsx` | |
| - `hooks/__tests__/useRefreshMemories.test.tsx` | |
| ## Testing | |
| - `components/Memories/__tests__/MemoryStoryViewer.test.tsx` | |
| - `hooks/__tests__/useRefreshMemories.test.tsx` | |
| - `features/__tests__/memoriesSlice.test.ts` |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/frontend/memories.md` around lines 226 - 229, Update the Testing section
in the memories documentation to include the existing Redux slice suite,
frontend/src/features/__tests__/memoriesSlice.test.ts, alongside the listed
MemoryStoryViewer and useRefreshMemories tests.
Source: Path instructions
rohan-pandeyy
added
documentation
- `captured_at IS NOT NULL` applies to the candidate queries over images and videos, not to every memories query; reads of generated memories go through the memories table on surface_date and status. - folders.py no longer described as triggering the embedding pass "last": it runs before semantic scoring, memory curation and the video pass. - Only the embedding reads filter on model_version; the event-label reads take no model_version at all. - image_util_process_unembedded_images closes its session if embedding inference raises, not scoring, which it never performs. - Listed the memories slice and utils suites in the frontend testing index, and gave the backend memories page the test-coverage table its sibling pages already have. - Switches persist on change, sliders on onValueCommit. - SQLite, not sqlite, throughout the architecture page.
|
|
|
|
1 participant
Follow-up to #1429.
Documents the Smart Memories feature shipped in #1430, and brings the backend
directory tables back in line with the tree.
The docs still described the implementation that #1430 deleted — DBSCAN
location clustering, reverse geocoding, "Trip to [City], [Year]" titles, the
/api/memories/*endpoints and four components that no longer exist.New
docs/backend/backend_python/memories.md— the backend reference:the four tables and their vocabularies, the three curation triggers and
their rules, the seven scoring signals, the API, the pipeline hooks and the
preference block. Added to the nav.
Rewritten
docs/frontend/memories.md— replaced wholesale. Now covers the gridpage, the story viewer (segmented progress, autoplay, keyboard and swipe
navigation, video clips, the per-slide sound toggle), the settings page and
the React Query data layer.
Updated
database.mddirectory-structure.mdimage-processing.mddate_sourcefieldsemantic-search.mdfeatures.md,architecture.md,state-management.md,ui-components.mdDirectory tables
The backend tables had drifted well beyond memories. Added the 22 modules that
were never listed —
videosandvideo_framesacross database/routes/schemas/utils, the connection context manager, the model and session registries, the
semantic-vocabulary utils and scripts, and the hardware, model-bootstrap,
face-search and face-quality helpers. Dropped
microservice.py, which nolonger exists. Every table was diffed against the filesystem afterwards: zero
missing, zero stale.
Notes for reviewers
Three things are deliberately not documented as working, because they
aren't yet:
frontend/public/memory-theme.mp3is not bundled,so the toggle is silent.
them (Phase 3).
GET /memories/todayanduseTodayMemoryare implemented but have noconsumer, so nothing currently surfaces "the memory to show now".
Two diagrams need a maintainer, and both carry a temporary note on their page
saying so:
docs/assets/backend-architecture.jpeg— needs a Memories curation boxdownstream of indexing and semantic scoring (it shares the single-worker
executor), trigger arrows from the folder add / sync / AI-tagging pipelines,
and a write into SQLite naming the four tables.
column lists and all five foreign-key relationships are in the new
"Memories tables" section of
database.md, which is written to be usable asthe source for it.
Delete the corresponding note once each diagram is regenerated.
Verification
mkdocs build— no warnings, every internal link resolvesmarkdownlint-cli2with.github/.markdownlint-cli2.jsonc— 0 errorsacross
docs/pre-existing ones in
semantic-search.mdwere rendering as empty boxes andare fixed
Summary by CodeRabbit