fix(index): report distance comparisons from HNSW search

[wombatu-kun]

HNSW::search takes _metrics: &dyn MetricsCollector and never touches it, so every HNSW query reports zero distance comparisons in per-query metrics while FlatIndex reports real numbers for the same workload. That makes the comparison count useless for exactly the case it is most interesting in: judging whether a graph traversal is doing more work than a scan would.

Change

The traversal primitives in graph.rs now return how many distances they computed alongside their results, and HNSW::search sums the paths it actually took and reports the total in a single record_comparisons call.

Counting happens where the distance is computed, so all four dispatch paths are covered: the unfiltered graph traversal, the filtered one, the ACORN traversal, and the exact flat scan. Two details worth calling out:

• The flat scan needs no loop instrumentation, since it scores every set bit in the mask exactly once, so its count is count_ones().
• When ACORN under-delivers and falls back to the basic traversal, both traversals' distances are summed. The abandoned work was still paid for.

The entry point distance is computed outside the traversal, so it is added explicitly rather than being silently dropped.

Cost

The hot loops gain one usize increment per distance computation, which is the accumulate-locally-report-once shape the other index types already use (wand.rs does the same with its comparisons field). Index construction is unaffected: the build path goes through beam_search and greedy_search, the query path through the _borrowed variants and beam_search_acorn, and the two sets are disjoint.

search_basic, search_acorn, and search_inner keep their existing signatures, since they are public and have callers in the benches and examples. The counted variants sit underneath them.

Tests

test_search_reports_distance_comparisons covers all four dispatch paths. The sparse-mask case asserts an exact count rather than merely a non-zero one, since the flat scan's work is known up front. The test fails on main (every path reports 0) and passes here.

MaskPreFilter moved from inside test_subindex_prefilter_dispatch to the enclosing test module so both tests can use it.

[lance-gatekeeper]

Gate recommendation: approve with a non-blocking risk. The implementation accounts for the entry point, every scored traversal candidate, exact flat-scan work, and both ACORN and fallback traversals. Exact graph-path totals remain source-audited rather than independently measured; the proposed deterministic calculator-backed regression would make that contract durable.

The public graph helper return types are source-breaking. Please mark this PR with the breaking-change label.

[lance-gatekeeper]

The graph-path checks only require >= k, so they would still pass if the implementation reported a constant k; this case also does not force the ACORN under-delivery branch where two traversals must be summed. Please compare the metric with an independently counting DistCalculator on deterministic inputs, including a forced fallback, so missed or double-counted distance calls are caught.

[wombatu-kun]

Done 98e1178

[lance-gatekeeper]

Gate recommendation: approve. The deterministic counting-storage oracle now matches reported distances across unfiltered, all-pass, sparse flat-scan, filtered basic, filtered ACORN, and forced ACORN-fallback searches, including the abandoned traversal’s work. This resolves the earlier evidence gap.

The public graph helper return types remain source-breaking. Please mark this PR with the breaking-change label.

[codecov]

Codecov Report

❌ Patch coverage is 85.38961% with 45 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-index/src/vector/hnsw/builder.rs	84.28%	39 Missing and 5 partials ⚠️
rust/lance-index/src/vector/graph.rs	95.83%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[wombatu-kun]

Rebased onto current main.

@BubbleCal mind taking a look when you get a chance? It is a small change, but the metric it adds is load-bearing: HNSW::search takes a MetricsCollector and ignores it, so an HNSW query reports zero distance comparisons while FlatIndex reports the real count. That leaves wall-clock as the only instrument for HNSW search work, and it is far too noisy to settle the few-percent differences that keep coming up.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The rebase preserves the new base’s level-descent and entry-point semantics while retaining exact accounting across the main search paths, including forced ACORN fallback. No new acceptance risk was introduced.

The public graph helper return types remain source-breaking. Please mark this PR with the breaking-change label.