Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,65 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Added

- **Try a few files before a large batch.** When you start analyzing 20 or more files and your Smart
Folders changed since the last sample, StratoSort offers to analyze 5 of them first, picked across
the file types in the batch. It shows where each would go and why, without moving anything. You
can analyze the rest, or stop and edit your folders. Skipping counts as a choice, so the offer
comes back only after a folder's name, description or path changes. The user guide has a new
section on writing folder descriptions.

### Fixed

- **The Dashboard reads clearly before your first files.** It said "All caught up", "All clear" and
"Nothing waiting" on a board that had never had a file; it now says it is ready for files and
shows the three steps. Smart folder names are no longer cut short, and the Uncategorized catch-all
is no longer listed as one. The knowledge card no longer stays on "unavailable" when the Dashboard
opens during startup, and a single year reads as "2026" instead of "2026 to 2026". Cards keep
their natural height instead of stretching into empty space.
- **A stray close button no longer floats at the top right of every screen.** The notification
list's collapse button was drawn even with no notifications.
- **Memory no longer climbs during a long analysis.** Each save of the search index leaked its
compression buffers, so the app grew by about 35 MB for every file analyzed: from 5 GB to almost
15 GB over 275 files, which pushed the system into swap.
- **A long batch no longer analyzes its early files again.** Analysis results were kept for 30
minutes, so in a batch of a few hundred files, an identical copy or a file analyzed again later
went through the AI a second time. Results are now kept for 12 hours; they still change when the
file, the model or your Smart Folders change.

### Changed

- **A file no Smart Folder fits goes to review instead of the closest folder.** A file now goes to a
Smart Folder only when a learned correction points there, its content matches one folder clearly
(score 0.55 or more, and no other folder within 0.03), or the folder is named after its file type
(or, when the AI model does the routing, after its label). Otherwise the card says "Needs review",
names the closest folders, and the file goes to your default folder unless you pick one. Before,
any folder that reached 0.4 won, so an unrelated Markdown report could land in an SEO folder.
Auto-organize and the Downloads watcher hold such files instead of moving them.
- **A folder you sort that is also a Smart Folder now asks you to choose.** Picking or dropping a
folder on Discover that is, sits inside, or holds a Smart Folder opens a choice instead of a
warning. On Organize, files that would stay in their own folder need a choice (leave them there or
skip them) before Move turns on. Each choice is kept for that Smart Folder.
- **Changing a Smart Folder's location to a folder that does not exist yet asks first.** It moved
the old folder there, files included, without asking. You now choose: move the folder and its
files, start with a new empty folder, or cancel.
- **Spreadsheets and CSVs send the AI the head of each sheet.** The AI saw a sample from the start,
middle and end of the text, so most sheets lost their names and columns. It now gets each sheet's
name, its columns and its first rows, in less than half the text.
- **Analysis is about a fifth faster.** Most of each file's time is the AI writing its answer, so
the purpose, summary and reasoning it writes are now shorter (at most 10, 25 and 15 words). On 30
exports the median model time went from 13.9 s to 11.1 s per file, and every file went to the same
folder as before.
- **Spreadsheets and CSVs keep 12 search passages instead of up to 80.** Their rows are nearly
alike, so most exports reached the limit: a batch of 275 exports stored 14,473 passages, and every
save of the search index rewrote all of them. On 30 exports the passages went from 1,452 to 354
and each save took about half as long. Search by meaning now covers about the first 10,000
characters of a spreadsheet; other documents are unchanged.
- **Suggestions made before you changed your Smart Folders are flagged.** Organize lists the files
that still go where your old folders sent them, with one button to analyze them again, and Move
asks before moving them.

## [5.3.1] - 2026-11-14

### Fixed
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ tree.
`ci`) enforces a monotonic floor and blocks new .js under `COVERED_DIRS` (`src/shared`,
`src/preload`, `src/main/{ipc,services,analysis,errors,utils,workers,transports,core}`,
`src/renderer/{services,hooks,phases,store,utils,components}` today (every `.js` under `src/` is
now opted in — 507/507 as of 2026-11-14, and `GLOBAL_DIRECTIVE_FLOOR` matches at 507) — see
now opted in — 512/512 as of 2026-11-15, and `GLOBAL_DIRECTIVE_FLOOR` matches at 512) — see
`scripts/verify-checkjs-ratchet.js` for the authoritative list) from landing without the
directive. When adding a file to a covered dir: put `// @ts-check` on line 1 and fix the errors
before commit. When opting in a new file elsewhere: bump `GLOBAL_DIRECTIVE_FLOOR` in
Expand Down
53 changes: 53 additions & 0 deletions docs/USER_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,59 @@ content against these descriptions to decide where it fits best.
| **Project Specs** | "Requirements, architecture docs, and project briefs." |
| **Screenshots** | "Screen captures, app screenshots, and UI mockups." |

### Writing folder descriptions

The AI reads each folder's name and description, and nothing else, to decide where a file goes, so a
few words of description decide where many files land. It is normal to find a description too vague
only after seeing where files land; that is what the sample below is for.

What to include:

- The kinds of files that belong here, in the words those files use ("invoice", "W-2", "lease").
- Who they come from or are about, when that tells folders apart ("from suppliers", "for the Garden
Street house").
- The most telling words first. Only the first part of a long description reaches the AI, and Setup
says so when a description is cut off.

What to leave out:

- Words every folder could use, such as "files", "documents", "stuff" or "misc".
- Files that belong in another folder. If two folders are close, say what goes in this one and not
the other ("Invoices from suppliers. Not receipts for things we bought.").

Example: a folder named **Invoices** with the description "Documents" leaves the AI with the name
alone. "Bills and invoices from suppliers, with an amount due and a due date. Not receipts." tells
it what belongs there and what does not.

Setup shows a hint under a description that is missing, short, cut off, or reads like another
folder's. A hint never stops you from saving.

### Try a few files first

When you add 20 or more files and your Smart Folders have not been tried since they last changed,
StratoSort asks before it starts: **Analyze 5 files first**, or **Analyze all** of them. The sample
takes files of each type in the batch. Nothing is moved.

When the sample is done, you see where each file would go, the confidence where there is one, and
why. Then choose **Analyze the other files** to go on, or **Edit Smart Folders** to change a
description first. The sample files count as analyzed, so the rest of the batch does not analyze
them again. Files you have not analyzed yet stay in the queue on Discover.

StratoSort remembers that you saw a sample, or chose **Analyze all**, for the folders you have now.
It asks again only after you change a folder's name, description or location.

### When you change Smart Folders after files were analyzed

When you change a Smart Folder, **Organize** ranks the files waiting there again with the current
folders, without analyzing them again. Most files get a current folder that way.

Some files can still point to a folder picked from your old Smart Folders: the folder the analysis
picked when nothing else ranks the file. StratoSort does not move those unnoticed. **Organize**
lists them above the files, with **Analyze again** to analyze only those files with the current
folders, and **Move** asks before it moves any of them. StratoSort does not analyze them again on
its own, because on a large batch that takes as long as the first run did. A folder you chose
yourself on a card is never treated as out of date.

### Teaching StratoSort

When you change a file's folder on **Organize**, its card says what StratoSort learns from that.
Expand Down
2 changes: 1 addition & 1 deletion scripts/verify-checkjs-ratchet.js
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ const COVERED_DIRS = [
'src/renderer/utils',
'src/renderer/components'
];
const GLOBAL_DIRECTIVE_FLOOR = 507;
const GLOBAL_DIRECTIVE_FLOOR = 512;
// ────────────────────────────────────────────────────────────────────────

const DIRECTIVE_SCAN_LINES = 5;
Expand Down
27 changes: 21 additions & 6 deletions src/main/analysis/chunkEmbedding.js
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@

const path = require('path');
const { chunkText } = require('../utils/textChunking');
const { isTableText } = require('./contentSelector');
const { CHUNKING } = require('../../shared/performanceConstants');
const { AI_DEFAULTS } = require('../../shared/constants');
const { isAutoChunkEnabled } = require('../../shared/experienceMode');
Expand Down Expand Up @@ -42,6 +43,23 @@ async function isChunkingEnabled(loadSettings) {
}
}

/**
* How `text` is cut into passages. A CSV or spreadsheet keeps only its first
* MAX_TABLE_CHUNKS: past the head of each sheet its rows add near-duplicate
* vectors that cost memory and save time for little recall.
* @param {string} text
* @returns {{ chunkSize: number, overlap: number, maxChunks: number }}
*/
function chunkOptionsFor(text) {
return {
chunkSize: CHUNKING.CHUNK_SIZE,
overlap: CHUNKING.OVERLAP,
maxChunks: isTableText(String(text || '').trimStart())
? CHUNKING.MAX_TABLE_CHUNKS
: CHUNKING.MAX_CHUNKS
};
}

/**
* Embed the chunks of `text` into `batchUpsertFileChunks` payloads. Pure with
* respect to the store: nothing is written. A chunk whose embedding fails is
Expand All @@ -58,11 +76,7 @@ async function isChunkingEnabled(loadSettings) {
*/
async function buildFileChunkPayloads({ fileId, filePath, text, embedText, signal }) {
const fileName = path.basename(filePath);
const chunks = chunkText(text, {
chunkSize: CHUNKING.CHUNK_SIZE,
overlap: CHUNKING.OVERLAP,
maxChunks: CHUNKING.MAX_CHUNKS
});
const chunks = chunkText(text, chunkOptionsFor(text));
/** @type {Array<Record<string, any>>} */
const payloads = [];
for (const c of chunks) {
Expand Down Expand Up @@ -179,5 +193,6 @@ async function embedFileChunks({

module.exports = {
embedFileChunks,
buildFileChunkPayloads
buildFileChunkPayloads,
chunkOptionsFor
};
77 changes: 76 additions & 1 deletion src/main/analysis/contentSelector.js
Original file line number Diff line number Diff line change
Expand Up @@ -268,13 +268,74 @@ function sampleMiddleContent(text, startOffset, endOffset, budget, sampleCount)
return samples;
}

// First line of a CSV or spreadsheet as the extractors write it
// (tableRowText in documentExtractors.js).
const TABLE_START_RE = /^(?:## Sheet: .+|Columns: .+\|.+)$/;
const SHEET_HEADING_RE = /^## Sheet: /;

/**
* True for the text of a CSV or spreadsheet whose first table has a header
* row or a sheet heading.
* @param {string} cleaned
*/
function isTableText(cleaned) {
const firstLine = cleaned.slice(0, 400).split('\n', 1)[0].trim();
return TABLE_START_RE.test(firstLine);
}

/**
* The head of every table in a CSV or spreadsheet: its sheet heading, its
* columns, its first rows and how many rows were left out. Exports are told
* apart by their sheets and columns, not by rows of numbers, and the model
* reads every prompt token, so rows past the head of each sheet are dropped.
* Every sheet gets an equal share of the budget; a share a sheet does not use
* goes to the sheets after it.
* @param {string} cleaned
* @param {number} budget
* @returns {string}
*/
function selectTableHeads(cleaned, budget) {
/** @type {string[][]} */
const tables = [];
for (const line of cleaned.split('\n')) {
if (SHEET_HEADING_RE.test(line) || tables.length === 0) tables.push([]);
if (line.trim()) tables[tables.length - 1].push(line);
}
const { TABLE_ROWS_PER_SHEET } = CONTENT_SELECTION;
/** @type {string[]} */
const parts = [];
let left = budget;
tables.forEach((lines, i) => {
const share = Math.floor(left / (tables.length - i));
const headerCount = lines.findIndex((l) => !SHEET_HEADING_RE.test(l) && !/^Columns: /.test(l));
const header = headerCount === -1 ? lines : lines.slice(0, headerCount);
const rows = headerCount === -1 ? [] : lines.slice(headerCount);
const kept = [...header];
let used = header.join('\n').length;
let shown = 0;
while (shown < rows.length && shown < TABLE_ROWS_PER_SHEET) {
const next = rows[shown];
if (used + next.length + 1 > share) break;
kept.push(next);
used += next.length + 1;
shown += 1;
}
if (rows.length > shown) kept.push(`(${rows.length - shown} more rows)`);
const text = kept.join('\n').slice(0, Math.max(share, 0));
parts.push(text);
left -= text.length + 1;
});
return parts.join('\n').slice(0, budget);
}

/**
* Select representative content from a document to maximize information
* density within a fixed character budget.
*
* For documents that fit within the budget, returns the full cleaned text.
* For larger documents, strategically samples beginning, middle, and end
* with a structural outline prepended.
* with a structural outline prepended. A CSV or spreadsheet gets the head of
* each table instead, within at most TABLE_MAX_CHARS.
*
* @param {string} text - Full extracted document text
* @param {number} budget - Maximum character budget for the output
Expand All @@ -289,6 +350,19 @@ function selectRepresentativeContent(text, budget) {
const cleaned = lightClean(text);
const totalLength = cleaned.length;

if (isTableText(cleaned)) {
const tableBudget = Math.min(budget, CONTENT_SELECTION.TABLE_MAX_CHARS);
if (totalLength > tableBudget) {
const content = selectTableHeads(cleaned, tableBudget);
return {
content,
strategy: 'table-heads',
coveragePercent: Math.min(100, Math.round((content.length / totalLength) * 100)),
totalLength
};
}
}

// If text fits within budget, return it all — no selection needed
if (totalLength <= budget) {
return {
Expand Down Expand Up @@ -367,6 +441,7 @@ function selectRepresentativeContent(text, budget) {
}

module.exports = {
isTableText,
selectRepresentativeContent,
extractDocumentOutline,
sampleMiddleContent
Expand Down
5 changes: 4 additions & 1 deletion src/main/analysis/documentAnalysis.js
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,10 @@ const CACHE_CONFIG = {
MAX_FILE_CACHE: 500, // Maximum number of files to cache in memory
FALLBACK_CONFIDENCE: 65, // Confidence score for fallback analysis
DEFAULT_CONFIDENCE: 85, // Default confidence for successful analysis
CACHE_TTL_MS: 30 * 60 * 1000 // 30 minute TTL to prevent memory leaks
// MAX_FILE_CACHE bounds memory, and the key holds the file's size and
// modified time, so the TTL is a backstop: 30 minutes made a long batch
// (275 files take about two hours) analyze its early files again.
CACHE_TTL_MS: 12 * 60 * 60 * 1000
};
// v3: spreadsheet and CSV text now carries sheet names and a column line.
const ANALYSIS_SIGNATURE_VERSION = 'v3';
Expand Down
13 changes: 3 additions & 10 deletions src/main/ipc/semantic.js
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ const { schemas: validationSchemas } = require('./validationSchemas');
const { cosineSimilarity, padOrTruncateVector } = require('../../shared/vectorMath');
const { categorizeModel } = require('../../shared/modelCategorization');
const { chunkText } = require('../utils/textChunking');
const { chunkOptionsFor } = require('../analysis/chunkEmbedding');
const { normalizeText } = require('../../shared/normalization');
const { getFileEmbeddingId } = require('../utils/fileIdUtils');
const { enrichFolderTextForEmbedding } = require('../analysis/semanticExtensionMap');
Expand Down Expand Up @@ -2818,11 +2819,7 @@ async function handleFullRebuild(
// Also process chunks from extracted text
const extractedText = analysis.extractedText || '';
if (shouldRebuildChunks && extractedText.length >= CHUNKING.MIN_TEXT_LENGTH) {
const chunks = chunkText(extractedText, {
chunkSize: CHUNKING.CHUNK_SIZE,
overlap: CHUNKING.OVERLAP,
maxChunks: CHUNKING.MAX_CHUNKS
});
const chunks = chunkText(extractedText, chunkOptionsFor(extractedText));

for (const chunk of chunks) {
throwIfInterrupted();
Expand Down Expand Up @@ -3563,11 +3560,7 @@ async function handleRebuildFiles(
pendingChunkGroups.set(fileId, fileChunkPayloads);

if (extractedText.trim().length >= CHUNKING.MIN_TEXT_LENGTH) {
const chunks = chunkText(extractedText, {
chunkSize: CHUNKING.CHUNK_SIZE,
overlap: CHUNKING.OVERLAP,
maxChunks: CHUNKING.MAX_CHUNKS
});
const chunks = chunkText(extractedText, chunkOptionsFor(extractedText));

for (const c of chunks) {
try {
Expand Down
Loading
Loading