Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,32 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
- **"Watch smart folders" is now "Keep Smart Folders up to date".** Its description says it watches
where files go, not where they come from, and names the ways files come in.

### Fixed

- **Spreadsheets from Visio and .NET tools are read.** Some tools write a workbook's XML with a
prefix on every tag (`<x:row>`), and StratoSort read those workbooks as empty. They are now read
like any other.
- **Cloud analysis is no longer held to one file at a time.** With a cloud model, a batch still took
the concurrency the local graphics card recommends, which on a busy or small card was 1. Documents
analyzed in the cloud now run at the requested concurrency; images and media, which run on local
models, keep the hardware limit.
- **The search index Refresh button stays still during analysis.** Background refreshes of the index
counts, every 1.5 seconds during a batch, spun and disabled the button and faded the counts. They
now update the numbers quietly; a refresh you ask for still shows that it is loading.
- **Search and Chat name files as they are on disk.** A file not yet organized was listed under the
name StratoSort suggested for it, which did not exist yet. Search results and Chat sources now use
the file's current name. Rebuild the search index to update files analyzed earlier.
- **Cloud analysis no longer reloads the local text model after images.** With a cloud provider
doing the document analysis, StratoSort still loaded the local text model after every batch of
images. On a busy machine that pushed the embedding model out of memory, and it reloaded for the
next file. The text model now loads only when something local needs it.
- **The notice for a Smart Folder whose AI provider is not ready is accurate.** It said the local
model would be used; the folder actually falls back to your default provider, which the notice now
names, along with the reason (such as a missing API key). It appears once a minute per folder
instead of for every file.
- **Embedding problems say why.** The warning for a high embedding error rate now names the most
frequent errors.

## [5.4.0] - 2026-11-16

### Added
Expand Down
15 changes: 15 additions & 0 deletions src/main/analysis/documentExtractors.js
Original file line number Diff line number Diff line change
Expand Up @@ -2182,8 +2182,22 @@ async function extractTextFromDocx(filePath, options = {}) {
throw mammothError;
}

/**
* Spreadsheet XML with its element prefixes removed: "<x:row>" becomes
* "<row>". Excel writes the main namespace as the default one, but Visio and
* .NET (Open XML SDK) exports often bind it to a prefix, and the readers below
* match unprefixed tags only, so such a workbook read as empty. Attribute
* names ("r:id") and declarations ("<?xml", "<!") are left as they are.
* @param {string} xml
* @returns {string}
*/
function withoutElementPrefixes(xml) {
return xml.replace(/<(\/?)[A-Za-z_][\w.-]*:(?=[A-Za-z_])/g, '<$1');
}

async function extractSharedStringsFromXlsxXml(/** @type {any} */ sharedStringsXml) {
if (!sharedStringsXml || typeof sharedStringsXml !== 'string') return [];
sharedStringsXml = withoutElementPrefixes(sharedStringsXml);
const pool = getTextExtractionPool();
if (pool && sharedStringsXml.length > 10000) {
try {
Expand Down Expand Up @@ -2224,6 +2238,7 @@ async function extractTextFromXlsxWorksheetXml(
/** @type {any} */ rowBudget
) {
if (!worksheetXml || typeof worksheetXml !== 'string') return { text: '', rowsUsed: 0 };
worksheetXml = withoutElementPrefixes(worksheetXml);
const pool = getTextExtractionPool();
if (pool && worksheetXml.length > 10000) {
try {
Expand Down
23 changes: 22 additions & 1 deletion src/main/analysis/providerStamp.js
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,28 @@ function stampAnalysisProvider(/** @type {any} */ analysis, /** @type {any} */ p
return analysis;
}

/**
* True when document analysis goes to a cloud provider (the active router's
* side-effect-free preview for 'analyze' names one other than local). The
* local text model then does no document work, so it need not be loaded or
* limit concurrency. False when there is no router or on any doubt.
* @returns {boolean}
*/
function isDocumentAnalysisRemote() {
try {
// Lazy: aiProvider requires the services tree.
const aiProvider = require('../ipc/aiProvider');
const router = aiProvider.getActiveAiRouter ? aiProvider.getActiveAiRouter() : null;
if (typeof router?.previewChoiceId !== 'function') return false;
const providerId = router.previewChoiceId({ operation: 'analyze' });
return Boolean(providerId) && providerId !== 'local';
} catch {
return false;
}
}

module.exports = {
resolveProviderIdFromRouter,
stampAnalysisProvider
stampAnalysisProvider,
isDocumentAnalysisRemote
};
17 changes: 16 additions & 1 deletion src/main/core/systemAnalytics.js
Original file line number Diff line number Diff line change
Expand Up @@ -226,11 +226,26 @@ const systemAnalytics = {
}
bucket.lastDegradedAt = now;

// The reasons, most frequent first: a rate alone left a steady 25%
// embedding error rate with no cause in the log.
/** @type {Record<string, number>} */
const reasonCounts = {};
for (const entry of bucket.recent) {
if (entry.outcome === 'primary') continue;
const reason = String(entry.reason || 'unknown').slice(0, 160);
reasonCounts[reason] = (reasonCounts[reason] || 0) + 1;
}
const topReasons = Object.entries(reasonCounts)
.sort((a, b) => b[1] - a[1])
.slice(0, 5)
.map(([reason, count]) => ({ reason, count }));

logger.warn('[ANALYTICS] Pipeline degradation detected', {
pipeline: pipelineName,
recentTotal,
fallbackRate: Number(fallbackRate.toFixed(3)),
errorRate: Number(errorRate.toFixed(3))
errorRate: Number(errorRate.toFixed(3)),
topReasons
});
},

Expand Down
36 changes: 34 additions & 2 deletions src/main/ipc/aiProvider.js
Original file line number Diff line number Diff line change
Expand Up @@ -366,9 +366,23 @@ function registerAiProviderIpc(servicesOrParams) {
safeSendToMain(getMainWindow, IPC_CHANNELS.AI_PROVIDER.STATUS_CHANGED, _routerStatus());
});
_router.on('folder-override-unavailable', (payload) => {
const providerLabel = payload?.configuredProvider || 'cloud';
// Once a minute per folder: the router raises this for every file in
// the folder, and a reanalysis touches hundreds.
const key = `folder:${payload?.folderId}:${payload?.configuredProvider}`;
const now = Date.now();
if (now - (_operationOverrideNoticeAt.get(key) || 0) < OVERRIDE_NOTICE_INTERVAL_MS) return;
_operationOverrideNoticeAt.set(key, now);
// The router falls through to the default provider (then local), not
// straight to local: name the one that actually runs.
const fallbackId =
_router?.previewChoiceId?.({ operation: payload?.operation || 'analyze' }) || 'local';
safeSendToMain(getMainWindow, IPC_EVENTS.NOTIFICATION, {
message: `Smart folder AI override (${providerLabel}) is unavailable. Using the local model.`,
message: folderOverrideNotice({
configured: payload?.configuredProvider,
reason: payload?.reason,
fallbackId,
providers: _providers
}),
severity: 'warning',
duration: 8000,
source: 'ai-provider',
Expand Down Expand Up @@ -823,6 +837,24 @@ function getActiveAiRouter() {
return _router;
}

/**
* The notice for a Smart Folder whose AI provider is not ready. It names the
* provider that runs instead: the router falls through to the default
* provider, and the old text ("Using the local model") was wrong whenever the
* default was a cloud provider.
* @param {{ configured?: string, reason?: string, fallbackId?: string|null, providers?: Record<string, any>|null }} p
* @returns {string}
*/
function folderOverrideNotice({ configured, reason, fallbackId, providers }) {
const label = (/** @type {string|null|undefined} */ id) => providers?.[id || '']?.displayName;
const configuredLabel = label(configured) || configured || 'the chosen provider';
const fallbackLabel =
!fallbackId || fallbackId === 'local' ? 'the local model' : label(fallbackId) || fallbackId;
const why = reason ? ` (${reason})` : '';
return `A Smart Folder is set to ${configuredLabel}, which is not ready${why}. Using ${fallbackLabel} instead. Change it in Settings → AI.`;
}

module.exports = registerAiProviderIpc;
module.exports.folderOverrideNotice = folderOverrideNotice;
module.exports.disposeAiProvider = disposeAiProvider;
module.exports.getActiveAiRouter = getActiveAiRouter;
9 changes: 6 additions & 3 deletions src/main/ipc/semantic.js
Original file line number Diff line number Diff line change
Expand Up @@ -2762,7 +2762,9 @@ async function handleFullRebuild(
continue;
}

const displayName = organization.newName || entry.fileName || path.basename(filePath);
// The name on disk: organization.newName is only a suggestion
// until the file is moved.
const displayName = path.basename(filePath) || entry.fileName;
const ext = (path.extname(filePath) || '').toLowerCase();
const isImage = SUPPORTED_IMAGE_EXTENSIONS.includes(ext);
const fileId = getFileEmbeddingId(filePath, isImage ? 'image' : 'file');
Expand Down Expand Up @@ -3511,8 +3513,9 @@ async function handleRebuildFiles(
// Generate embedding
const { vector, model } = await getFolderMatcher().embedText(summary);

// Use renamed name if available, otherwise fall back to original basename
const displayName = entry.organization?.newName || path.basename(filePath);
// The name on disk: organization.newName is only a suggestion until
// the file is moved.
const displayName = path.basename(filePath);

// Extract rich metadata for graph visualization
const tags = Array.isArray(entry.analysis?.tags) ? entry.analysis.tags : [];
Expand Down
39 changes: 37 additions & 2 deletions src/main/services/BatchAnalysisService.js
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ const { getRecommendedConcurrency } = require('./PerformanceService');
const { getInstance: getModelAccessCoordinator } = require('./ModelAccessCoordinator');

const { getFileTypeSingular } = require('./autoOrganize/fileTypeUtils');
const { isDocumentAnalysisRemote } = require('../analysis/providerStamp');

/**
* BatchAnalysisService
Expand Down Expand Up @@ -202,7 +203,28 @@ class BatchAnalysisService {
}
}

async _getAdaptiveConcurrency(/** @type {any} */ requestedConcurrency, autoConcurrency = true) {
/**
* True when document analysis goes to a cloud provider: the local model is
* not the one doing that work, so the local hardware cap does not apply to
* it. Reads the router's side-effect-free preview; false on any doubt.
* @returns {boolean}
*/
_isDocumentAnalysisRemote() {
return isDocumentAnalysisRemote();
}

/**
* @param {any} requestedConcurrency
* @param {boolean} [autoConcurrency]
* @param {{ remote?: boolean }} [options] remote: the work runs on a cloud
* provider, so the hardware (VRAM/RAM) cap is skipped; queue pressure on
* the local models still applies.
*/
async _getAdaptiveConcurrency(
/** @type {any} */ requestedConcurrency,
autoConcurrency = true,
{ remote = false } = {}
) {
let capped = requestedConcurrency;
let perfRecommendation = null;

Expand All @@ -223,7 +245,9 @@ class BatchAnalysisService {
// When the user has explicitly opted into manual concurrency, we honor
// their value here; the queue-pressure reductions below (runtime
// backpressure) still apply in both modes to protect against OOM.
// Work on a cloud provider does not run on this hardware at all.
if (
!remote &&
autoConcurrency !== false &&
Number.isFinite(perfRecommendation?.maxConcurrent) &&
perfRecommendation.maxConcurrent > 0
Expand Down Expand Up @@ -303,6 +327,15 @@ class BatchAnalysisService {
const concurrency = disableAdaptiveConcurrency
? requestedConcurrency
: await this._getAdaptiveConcurrency(requestedConcurrency, autoConcurrency);
// Documents analyzed by a cloud provider are not limited by local VRAM:
// a reanalysis through a cloud model ran one file at a time because the
// local hardware recommended 1. Images and media still run locally.
const documentConcurrency =
disableAdaptiveConcurrency || !this._isDocumentAnalysisRemote()
? concurrency
: await this._getAdaptiveConcurrency(requestedConcurrency, autoConcurrency, {
remote: true
});

if (!Array.isArray(filePaths) || filePaths.length === 0) {
return {
Expand All @@ -322,6 +355,7 @@ class BatchAnalysisService {
fileCount: filePaths.length,
requestedConcurrency,
concurrency,
documentConcurrency,
sectionOrder: orderedSections,
documentCount: partitioned.documents.length,
imageCount: partitioned.images.length,
Expand Down Expand Up @@ -402,7 +436,8 @@ class BatchAnalysisService {
}
const sectionFiles = partitioned[section];
const sectionIsImage = section === 'images';
const sectionConcurrency = Math.max(1, Math.min(concurrency, sectionFiles.length || 1));
const sectionCap = section === 'documents' ? documentConcurrency : concurrency;
const sectionConcurrency = Math.max(1, Math.min(sectionCap, sectionFiles.length || 1));
const sectionStart = Date.now();
// Vision batch mode is on by default in production. Tests can opt out
// by passing { enableVisionBatchMode: false } in options. The previous
Expand Down
12 changes: 11 additions & 1 deletion src/main/services/LlamaService.js
Original file line number Diff line number Diff line change
Expand Up @@ -4413,6 +4413,17 @@ class LlamaService extends EventEmitter {
// Rapid exit -> enter: batch mode was re-entered, or another batch is on
// its way in, before we even start.
if (this._visionBatchMode || this._visionBatchEntriesPending > 0) return;
const { resetLlamaCircuit } = require('./LlamaResilience');
// Documents go to a cloud provider: nothing will use the local text
// model. Loading it after every image batch (a 7B model on a busy card)
// evicted the embedding model, which then reloaded for the next file.
// A local call that does need it still loads it on demand.
const { isDocumentAnalysisRemote } = require('../analysis/providerStamp');
if (isDocumentAnalysisRemote()) {
for (const circuit of circuitsToReset) resetLlamaCircuit(circuit);
logger.debug('[LlamaService] Skipped text preload after vision: analysis is remote');
return;
}
let loaded;
try {
// speculative: this load does not hold the text lane, so the next
Expand Down Expand Up @@ -4440,7 +4451,6 @@ class LlamaService extends EventEmitter {
}
return;
}
const { resetLlamaCircuit } = require('./LlamaResilience');
for (const circuit of circuitsToReset) {
resetLlamaCircuit(circuit);
}
Expand Down
7 changes: 6 additions & 1 deletion src/main/services/SearchService.js
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ const { LRUCache } = require('../../shared/LRUCache');
const { normalizePathForIndex } = require('../../shared/pathSanitization');
const { validateEmbeddingDimensions, padOrTruncateVector } = require('../../shared/vectorMath');
const { getSemanticFileId } = require('../../shared/fileIdUtils');
const { crossPlatformBasename } = require('../../shared/crossPlatformUtils');
const { createBackgroundWorkTracker } = require('../../shared/promiseUtils');
const { isGhostCleanupDeferred } = require('../../shared/ghostCleanupGuard');
const systemAnalytics = require('../core/systemAnalytics');
Expand Down Expand Up @@ -529,7 +530,11 @@ class SearchService {

// If file was moved/renamed, use the actual destination path
const currentPath = organization.actual || doc.originalPath;
const currentName = organization.newName || doc.fileName || '';
// The name on disk. organization.newName is the suggested name from
// the moment a file is analyzed, before anything is renamed: search
// and chat cited files by names that did not exist yet.
const currentName =
(currentPath ? crossPlatformBasename(currentPath) : '') || doc.fileName || '';

// Use normalizePathForIndex for Windows case-insensitivity consistency
// This ensures BM25 index keys match vector DB lookups
Expand Down
11 changes: 8 additions & 3 deletions src/renderer/components/search/hooks/useSearchModalStats.js
Original file line number Diff line number Diff line change
Expand Up @@ -82,10 +82,15 @@ export function useSearchModalStats({
// state written by a newer one (rapid open/close + rebuild races).
const statsRequestCounterRef = useRef(0);

// `quiet`: a background refresh (batch progress, a file move) updates the
// counts without the loading state. Batch progress refreshed every 1.5 s,
// and each one spun and disabled the Refresh button and faded the counts:
// the button flashed for the whole run.
const refreshStats = useCallback(async (/** @type {any} */ options = {}) => {
const forceRefresh = options === true || options?.force === true;
const quiet = options?.quiet === true;
const requestId = (statsRequestCounterRef.current += 1);
if (isMountedRef.current) setIsLoadingStats(true);
if (isMountedRef.current && !quiet) setIsLoadingStats(true);
try {
/** @type {any} */
let res = null;
Expand Down Expand Up @@ -169,7 +174,7 @@ export function useSearchModalStats({
});
onFileOperation?.(data);
embeddingsIpc.invalidateStatsCache();
refreshStats(true);
refreshStats({ force: true, quiet: true });
}
};

Expand All @@ -194,7 +199,7 @@ export function useSearchModalStats({
const elapsed = now - lastRefreshAt;
const invoke = () => {
lastRefreshAt = Date.now();
refreshStats({ force });
refreshStats({ force, quiet: true });
};

if (force || elapsed >= MIN_REFRESH_INTERVAL_MS) {
Expand Down
Loading
Loading