Repository navigation
fix: retest items, a search-index memory leak, faster analysis and a clearer Dashboard - #139
Merged
Merged
Conversation
… weak match
A file that fit no folder still went to whichever Smart Folder ranked
first, since every semantic candidate only had to clear 0.4. An
unrelated Markdown report could land in an SEO folder.
Routing now goes to a Smart Folder only on a confident match:
- a learned pattern, learned correction or feedback rule points there
- the content score is 0.55 or more (FOLDER_MATCH_CONFIDENCE) and no
other folder is within 0.03 (NO_SPREAD_THRESHOLD)
- the folder claims the extension, is named after the file type, or,
in the modes that rank by the label, is named outright by it
Otherwise the file needs review: it goes to the default folder, or
stays unassigned when there is none. The explanation says why ("No
Smart Folder matched this file well", "matched this file about
equally").
- Date, workflow and other template defaults no longer pick a folder.
- getSuggestionsForFile returns needsReview and routingDecision;
getBatchSuggestions lists unassigned files.
- Auto-organize, batch organize and the download watcher hold a
needs-review file whatever the threshold.
- The Organize card shows "Needs review" with the reason and where the
file goes, lists the closest folders, and skips name matching for it.
… Folder
Sorting a folder that is, sits inside, or holds a Smart Folder now needs a
choice. Discover asks before it sorts such a folder, names the folder picked
and the Smart Folder with its path, and offers "Sort this folder" or "Pick
another folder" ("Skip this folder" for a drop). The answer is kept on the
Smart Folder (sourceOverlapOk), so it is asked once.
On Organize, files the plan leaves in the folder they are already in need
a choice before Move: leave them there or skip them. Move stays off until
the user chooses, handleOrganizeFiles refuses to run without one, skipped
files are left out of the move and its preview, and the choice is kept on
the Smart Folder (inPlaceFiles) with a way to change it.
Changing a Smart Folder's path to a folder that does not exist yet used to
move the old folder there with its files without asking. The edit now
changes nothing and replies PATH_CHANGE_CHOICE_REQUIRED, and Setup asks:
move the folder and its files, use a new empty folder, or cancel. The edit
takes the answer as an optional third argument ({ pathChange }), checked by
its Zod schema. On macOS a decomposed and a composed spelling of one path
count as the same folder.
…esults from old folder definitions Sample first: when 20 or more added files are about to be analyzed and the current Smart Folder definitions (a hash of names, descriptions and paths) have not been tried, Discover offers to analyze 5 files first, spread across the file types in the batch. The sample uses the normal analysis run and the Organize ranking, shows where each file would go with its confidence and reason, and moves nothing. The user continues with the rest (sample files count as analyzed) or goes to Setup to edit folders. Skipping or reviewing a sample is remembered per definition set in ui.smartFolderSampleHash, so the offer returns only after a folder changes. Stale results: each analysis is stamped with the definitions it was made under. Organize already re-ranks files with the current folders; a file whose folder still comes from the old analysis label is listed with one click to analyze it again, and Move asks before moving it. Documented in organizeDestination.js and docs/USER_GUIDE.md, with a section on writing folder descriptions.
# Conflicts: # src/renderer/components/organize/ReadyFileItem.jsx
…rompts A spreadsheet prompt carried about 3,700 characters sampled from the start, middle and end of the text, so most sheets lost their names and columns and the model read rows of numbers. Table text now gets each sheet's heading, its columns, its first 8 rows and a count of the rows left out, within 2,000 characters. A Search Console workbook goes from 3,739 to 1,517 characters with every sheet covered; CSV exports from about 3,800 to 500 to 750.
…analyzed file lz4-napi's async compress() and uncompress() never free their native work buffers. Compressing a 60 MB index 12 times grew the process from 186 MB to 1,322 MB; compressSync() stayed flat. The search index saves itself every few files during a batch, so a 275-file analysis grew the app from 5.1 GB to 14.6 GB of private memory while the JS heap stayed near 60 MB. Text generation and embeddings were measured flat on their own. The async wrappers now call the sync binding (the async one is kept only as a fallback when the sync API is missing).
A 275-file batch takes about two hours. With a 30-minute expiry, an identical copy reached more than 30 minutes after its twin, or a file analyzed again later in the session, went through the model a second time (a measured second pass over 117 unchanged files hit the cache 0 times). Both caches stay capped (500 file results, 500 documents, 300 images), and their keys already change with the file's size and modified time, the model, the Smart Folders and the instructions, so the expiry is only a backstop. Changelog: the memory fix, this one and the spreadsheet prompt change.
… to 80 Rows of an export are nearly alike, so most spreadsheets reached the 80-passage limit: a 275-file batch of exports stored 14,473 passages, each about 9.5 KB in memory and about 30 KB on disk, and every save of the search index rewrote all of them. chunkOptionsFor() caps text that starts like a table (a sheet heading or a column line) at MAX_TABLE_CHUNKS and is shared by live analysis and both Settings rebuild paths. On 30 exports the passages went from 1,452 to 354 and the median save from 170 ms to 93 ms.
Writing the answer is most of each file's model time. The schema now asks for at most 10, 25 and 15 words for purpose, summary and reasoning. On 30 exports the median answer went from 734 to 618 characters and the median model time from 13.9 s to 11.1 s, with every file in the same category. The changelog notes this and the spreadsheet passage cap.
… load at startup - An empty board said "All caught up", "All clear" and "Nothing waiting". It now reads "Ready for files", explains the three steps, and the pipeline footer only appears when something is waiting or failed. - Stage counts use the sans tabular figures, and each icon chip sits on an opaque base so the connector rail no longer runs through it. The lone ring on the first chip is gone. - Smart folder chips use columns at least 11rem wide and skip the Uncategorized catch-all, so names read in full. - The knowledge card asked for stats once; during startup the handler answers with a bare envelope, which left the card on "unavailable" for the session. A bare answer is now asked again up to six times. A single year reads as "2026". - No row grows into the window's spare height. - The notification collapse button is drawn only when there are notifications; with none it floated over every screen.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes for the open items from the Mac retest of 5.3.1, a memory leak and two speed changes found while benchmarking a 275-file batch, and a pass over the Dashboard.
Retest items
src/main/services/organization/routingDecision.js: a file is routed only on a learned correction, a confident content match (score 0.55 or more, and 0.03 clear of the next folder), or a name match. Otherwise the card says "Needs review" and names the closest folders. Before, any folder that reached 0.4 won. Auto-organize and the Downloads watcher hold these files.Memory and speed
compress/uncompresscalls never released their native buffers: about 35 MB per analyzed file. The sync calls are used, async only as a fallback.Same 275 files (155 XLSX, 85 CSV, 30 PDF, 5 MD) on Windows with a 6 GB GPU:
All 275 files analyzed, each in the same folder as on the leak-fix-only build.
Dashboard
An empty board said "All caught up", "All clear" and "Nothing waiting"; it now says "Ready for files" and explains the three steps. Smart folder names read in full and Uncategorized is no longer listed. The knowledge card no longer stays on "unavailable" when the Dashboard opens during startup (the stats handler answers with a bare envelope until the history service is ready; the card now asks again). The notification collapse button is drawn only when there are notifications; with none it floated over every screen.
Testing
Local, at 23c0448:
npm run ci: 15,347 Jest tests passed, 0 failed, coverage thresholds met, everyverify:*, knip and the build.verify:installed-electronandverify:native-modules.npm run test:e2e: 381 passed, 0 failed.Not yet checked on a Mac: memory and timing on Apple Silicon, and the notarized build.