Skip to content

fix: retest items, a search-index memory leak, faster analysis and a clearer Dashboard - #139

Merged
iLevyTate merged 13 commits into
mainfrom
fix/open-retest-items
Nov 16, 2026
Merged

iLevyTate merged 13 commits into
mainfrom
fix/open-retest-items

Conversation

@iLevyTate

Copy link
Copy Markdown
Owner

Summary

Fixes for the open items from the Mac retest of 5.3.1, a memory leak and two speed changes found while benchmarking a 275-file batch, and a pass over the Dashboard.

Retest items

  • Overlapping folders ask for a choice (item 4). Picking or dropping a folder on Discover that is, sits inside, or holds a Smart Folder opens a choice instead of a warning. On Organize, files that would stay in their own folder need a choice before Move turns on. Changing a Smart Folder's location to a folder that does not exist yet asks whether to move the old folder, start a new one, or cancel (it used to move it without asking).
  • Files no Smart Folder fits go to review (item 7). New src/main/services/organization/routingDecision.js: a file is routed only on a learned correction, a confident content match (score 0.55 or more, and 0.03 clear of the next folder), or a name match. Otherwise the card says "Needs review" and names the closest folders. Before, any folder that reached 0.4 won. Auto-organize and the Downloads watcher hold these files.
  • Try a few files first (item 8). A batch of 20 or more files after the Smart Folders changed offers to analyze 5 first, shows where each would go and why, and moves nothing. Suggestions made before a folder changed are flagged on Organize.

Memory and speed

  • Search index save leak (4043c00). The async LZ4 compress/uncompress calls never released their native buffers: about 35 MB per analyzed file. The sync calls are used, async only as a fallback.
  • Spreadsheet prompts send the head of each sheet (e15f177), analysis results are kept 12 hours instead of 30 minutes (8f013e8), spreadsheets keep 12 search passages instead of up to 80 (15832eb), and the model is asked for a shorter purpose, summary and reasoning (450a1d3).

Same 275 files (155 XLSX, 85 CSV, 30 PDF, 5 MD) on Windows with a 6 GB GPU:

5.3.1 This branch
Batch time 76 min (16.5 s/file) 48 min (10.4 s/file)
App memory at the end 14.6 GB 5.75 GB (peak 6.15 GB)
Stored passages 14,000+ 3,386

All 275 files analyzed, each in the same folder as on the leak-fix-only build.

Dashboard

An empty board said "All caught up", "All clear" and "Nothing waiting"; it now says "Ready for files" and explains the three steps. Smart folder names read in full and Uncategorized is no longer listed. The knowledge card no longer stays on "unavailable" when the Dashboard opens during startup (the stats handler answers with a bare envelope until the history service is ready; the card now asks again). The notification collapse button is drawn only when there are notifications; with none it floated over every screen.

Testing

Local, at 23c0448:

  • npm run ci: 15,347 Jest tests passed, 0 failed, coverage thresholds met, every verify:*, knip and the build.
  • verify:installed-electron and verify:native-modules.
  • npm run test:e2e: 381 passed, 0 failed.
  • Full Jest suite on Linux (WSL, Node 24.22.2): all pass once the Linux LZ4 binding is present.
  • 275-file benchmark and a 21-file business-document routing check (18 of 21 routed on their own; the other 3 are ties held for review).

Not yet checked on a Mac: memory and timing on Apple Silicon, and the notarized build.

… weak match

A file that fit no folder still went to whichever Smart Folder ranked
first, since every semantic candidate only had to clear 0.4. An
unrelated Markdown report could land in an SEO folder.

Routing now goes to a Smart Folder only on a confident match:
- a learned pattern, learned correction or feedback rule points there
- the content score is 0.55 or more (FOLDER_MATCH_CONFIDENCE) and no
  other folder is within 0.03 (NO_SPREAD_THRESHOLD)
- the folder claims the extension, is named after the file type, or,
  in the modes that rank by the label, is named outright by it
Otherwise the file needs review: it goes to the default folder, or
stays unassigned when there is none. The explanation says why ("No
Smart Folder matched this file well", "matched this file about
equally").

- Date, workflow and other template defaults no longer pick a folder.
- getSuggestionsForFile returns needsReview and routingDecision;
  getBatchSuggestions lists unassigned files.
- Auto-organize, batch organize and the download watcher hold a
  needs-review file whatever the threshold.
- The Organize card shows "Needs review" with the reason and where the
  file goes, lists the closest folders, and skips name matching for it.
… Folder

Sorting a folder that is, sits inside, or holds a Smart Folder now needs a
choice. Discover asks before it sorts such a folder, names the folder picked
and the Smart Folder with its path, and offers "Sort this folder" or "Pick
another folder" ("Skip this folder" for a drop). The answer is kept on the
Smart Folder (sourceOverlapOk), so it is asked once.

On Organize, files the plan leaves in the folder they are already in need
a choice before Move: leave them there or skip them. Move stays off until
the user chooses, handleOrganizeFiles refuses to run without one, skipped
files are left out of the move and its preview, and the choice is kept on
the Smart Folder (inPlaceFiles) with a way to change it.

Changing a Smart Folder's path to a folder that does not exist yet used to
move the old folder there with its files without asking. The edit now
changes nothing and replies PATH_CHANGE_CHOICE_REQUIRED, and Setup asks:
move the folder and its files, use a new empty folder, or cancel. The edit
takes the answer as an optional third argument ({ pathChange }), checked by
its Zod schema. On macOS a decomposed and a composed spelling of one path
count as the same folder.
…esults from old folder definitions

Sample first: when 20 or more added files are about to be analyzed and the
current Smart Folder definitions (a hash of names, descriptions and paths)
have not been tried, Discover offers to analyze 5 files first, spread across
the file types in the batch. The sample uses the normal analysis run and the
Organize ranking, shows where each file would go with its confidence and
reason, and moves nothing. The user continues with the rest (sample files
count as analyzed) or goes to Setup to edit folders. Skipping or reviewing a
sample is remembered per definition set in ui.smartFolderSampleHash, so the
offer returns only after a folder changes.

Stale results: each analysis is stamped with the definitions it was made
under. Organize already re-ranks files with the current folders; a file whose
folder still comes from the old analysis label is listed with one click to
analyze it again, and Move asks before moving it. Documented in
organizeDestination.js and docs/USER_GUIDE.md, with a section on writing
folder descriptions.
# Conflicts:
#	src/renderer/components/organize/ReadyFileItem.jsx
…rompts

A spreadsheet prompt carried about 3,700 characters sampled from the start,
middle and end of the text, so most sheets lost their names and columns and
the model read rows of numbers. Table text now gets each sheet's heading, its
columns, its first 8 rows and a count of the rows left out, within 2,000
characters. A Search Console workbook goes from 3,739 to 1,517 characters
with every sheet covered; CSV exports from about 3,800 to 500 to 750.
…analyzed file

lz4-napi's async compress() and uncompress() never free their native work
buffers. Compressing a 60 MB index 12 times grew the process from 186 MB to
1,322 MB; compressSync() stayed flat. The search index saves itself every
few files during a batch, so a 275-file analysis grew the app from 5.1 GB to
14.6 GB of private memory while the JS heap stayed near 60 MB. Text
generation and embeddings were measured flat on their own.

The async wrappers now call the sync binding (the async one is kept only as
a fallback when the sync API is missing).
A 275-file batch takes about two hours. With a 30-minute expiry, an identical
copy reached more than 30 minutes after its twin, or a file analyzed again
later in the session, went through the model a second time (a measured
second pass over 117 unchanged files hit the cache 0 times). Both caches stay
capped (500 file results, 500 documents, 300 images), and their keys already
change with the file's size and modified time, the model, the Smart Folders
and the instructions, so the expiry is only a backstop.

Changelog: the memory fix, this one and the spreadsheet prompt change.
… to 80

Rows of an export are nearly alike, so most spreadsheets reached the
80-passage limit: a 275-file batch of exports stored 14,473 passages, each
about 9.5 KB in memory and about 30 KB on disk, and every save of the search
index rewrote all of them. chunkOptionsFor() caps text that starts like a
table (a sheet heading or a column line) at MAX_TABLE_CHUNKS and is shared by
live analysis and both Settings rebuild paths.

On 30 exports the passages went from 1,452 to 354 and the median save from
170 ms to 93 ms.
Writing the answer is most of each file's model time. The schema now asks
for at most 10, 25 and 15 words for purpose, summary and reasoning. On 30
exports the median answer went from 734 to 618 characters and the median
model time from 13.9 s to 11.1 s, with every file in the same category.
The changelog notes this and the spreadsheet passage cap.
… load at startup

- An empty board said "All caught up", "All clear" and "Nothing waiting".
  It now reads "Ready for files", explains the three steps, and the
  pipeline footer only appears when something is waiting or failed.
- Stage counts use the sans tabular figures, and each icon chip sits on an
  opaque base so the connector rail no longer runs through it. The lone
  ring on the first chip is gone.
- Smart folder chips use columns at least 11rem wide and skip the
  Uncategorized catch-all, so names read in full.
- The knowledge card asked for stats once; during startup the handler
  answers with a bare envelope, which left the card on "unavailable" for
  the session. A bare answer is now asked again up to six times.
  A single year reads as "2026".
- No row grows into the window's spare height.
- The notification collapse button is drawn only when there are
  notifications; with none it floated over every screen.
@iLevyTate
iLevyTate merged commit 40d618e into main Nov 16, 2026
8 checks passed
@iLevyTate
iLevyTate deleted the fix/open-retest-items branch November 16, 2026 23:26
@iLevyTate iLevyTate mentioned this pull request Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant