Harden GVFS.Mount against native ProjFS failures under memory pressure (0xC000CE01)

[tyrielv]

Problem

GVFS.Mount crashes are being reported that surface downstream as GetOfficialBranch.exe (and other tools) exiting with -1073689087 = 0xC000CE01 = STATUS_FILE_SYSTEM_VIRTUALIZATION_UNAVAILABLE. The chain is: GVFS.Mount crashes → the ProjFS virtualization root is orphaned → every subsequent placeholder access on that enlistment returns 0xC000CE01 until remount.

Watson attributes the crashes to radar_high_memory projectedfslib.dll!prjcompletecommand, alongside System.OutOfMemoryException buckets and GC access violations. The dominant managed signatures are NullReferenceException surfaced from the native ProjFS command-completion (PrjCompleteCommand, via StartDirectoryEnumerationAsyncHandler) and delete (PrjDeleteFile) paths — the native code faults under memory exhaustion and the CLR raises a (catchable) NullReferenceException on the worker thread, which today reaches ExecuteFileOrNetworkRequest and calls Environment.Exit.

VFS for Git is already on the latest Microsoft.Windows.ProjFS (2.1.0), so the native projectedfslib.dll fault cannot be fixed from here. The only managed-side levers are (A) reduce the memory pressure that triggers the native fault, and (B) stop converting a single failed native ProjFS call into a whole-mount crash. This PR does both. It does not claim to eliminate the native OOM fault itself.

Not a v2.0 regression: the crash-prone call sites and the activeEnumerations leak date to 2018 (785ccb4a, .NET Framework era). The v2.0 pure-C# ProjFS rewrite (66334474) made the same latent native fault legible as a managed NRE in GVFS telemetry — so the recent spike in reports is largely improved visibility, which B’s telemetry is designed to quantify.

A — Bound the activeEnumerations leak

ProjFS does not always deliver EndDirectoryEnumeration for a StartDirectoryEnumeration (the long-standing // TODO: Need ProjFS 13150199 note). The corresponding ActiveEnumeration — and the List<ProjectedFileInfo> it pins — then leaks for the life of the mount.

• ActiveEnumeration tracks a monotonic LastActivityTickCount (Environment.TickCount64), set on creation and every GetDirectoryEnumeration read.
• Once activeEnumerations grows past a threshold far larger than any realistic count of concurrent live enumerations (1024), a throttled sweep (≤ once/min) evicts entries idle longer than a timeout (5 min).
• Evicting a live-but-idle enumeration only fails that one directory listing with InternalError — never a crash — and every eviction is reported via telemetry.
• Monotonic clocks are used for both the throttle and the staleness check so NTP/wall-clock steps can’t disturb them.

B — Don’t crash the mount on a native ProjFS failure (generalized)

A single boundary helper, TryInvokeProjFS(Func<HResult>, operationName, …), wraps each GVFS-initiated native ProjFS call: it catches the exception, emits high-signal telemetry (*_NativeFailure, with activeEnumerations/activeCommands counts and GC.GetTotalMemory), and returns HResult.InternalError so the caller maps it to a failure result and keeps serving.

Routed through the swallow-and-continue helper: CompleteCommand (via TryCompleteCommand — covers the completion leg of all async callbacks: enumeration, placeholder-info, file-data), DeleteFile, WritePlaceholderFile, WritePlaceholderDirectory, UpdatePlaceholderIfNeeded, ClearNegativePathCache.

MarkDirectoryAsPlaceholder mutates projection state, so swallowing is unsafe. It uses a sibling helper InvokeProjFSOrThrow that emits the same *_NativeFailure telemetry (so the memory diagnostics are captured) and then rethrows — the exception still reaches LogUnhandledExceptionAndExit, preserving fail-fast. Both helpers share one LogProjFSNativeFailure.

Left as-is (already non-fatal): GetFileStreamHandlerAsyncHandler already fails a single hydration on any exception (completes the command with FileNotAvailable, no exit), so CreateWriteBuffer/WriteFileData are already non-fatal.

The exception is demonstrably catchable — it already reaches GVFS’s LogUnhandledExceptionAndExit today (the source of the existing VFS.Error "exiting process" events). B intercepts it one frame earlier and either declines to exit or (for state-mutating ops) enriches the crash with memory telemetry first.

Tests

• DeleteFileReturnsIOErrorWhenVirtualizationInstanceThrows, OutboundProjFSOperationsReturnFailureWhenVirtualizationInstanceThrows (ClearNegativePathCache, UpdatePlaceholderIfNeeded), WritePlaceholderOperationsReturnFailureWhenVirtualizationInstanceThrows (tester-based) — a throwing native call yields IOError, not a propagated exception.
• RecordActivityUpdatesLastActivityTime — covers the A monotonic-timestamp building block.
• (InvokeProjFSOrThrow’s rethrow path ends in Environment.Exit, so it isn’t safely unit-testable in-process; it’s symmetric with the tested TryInvokeProjFS.)
• Full unit suite: 881 passed, 0 failed (11 skipped); clean build.

Open questions for review (draft)

1. Eviction heuristic. Threshold (1024) / stale-timeout (5 min) are conservative guesses. Tunable, or tied to a real signal? Is the rare "evict a live idle enumeration → one failed listing" tradeoff acceptable?
2. Swallow vs. controlled remount. B fails the single op and keeps serving. If the native context is genuinely dead, every op fails 0xC000CE01 anyway — would a deliberate remount on repeated *_NativeFailure be better recovery than limping?
3. Is A even the dominant leak? The B telemetry (activeEnumerations/memory) is partly here to answer that. Worth landing B first purely for diagnostics before committing to A’s heuristic?

---

Assisted-by: Claude Opus 4.8