perf(ivfpq): speed up transposed PQ8 scans

[jerry-024]

Summary

Speed up transposed 8-bit IVF-PQ scans by exposing each 256-entry lookup table as a fixed-size array, allowing the compiler to eliminate bounds checks in the hot loop. Search results and the existing 12.5% sparse/dense crossover remain unchanged.

Changes

• Add fixed-array fast paths for initializing and accumulating transposed PQ8 distances.
• Keep the slice-based fallback for non-PQ8 lookup tables.
• Cover scan results around the configured sparse/dense crossover.
• Add a deterministic filter-scan benchmark using the production query shape (nq=64, nprobe=64).

Testing

• cargo fmt --all --check
• cargo check -p paimon-vindex-core --tests --target x86_64-unknown-linux-gnu
• cargo clippy -p paimon-vindex-core --benches --target x86_64-unknown-linux-gnu -- -D warnings

Single-thread benchmark on the same SSH development host:

Filter density	p50
6.25%	541.739 ms
12.5%	907.507 ms
18.75%	1058.931 ms
25%	1040.456 ms
50%	1071.382 ms
100%	1066.809 ms

Historical scalar results on the same host and shape were 1890.700 ms at 50% density and 1849.500 ms at 100%, directional improvements of 43.3% and 42.3%. These were not interleaved A/B runs; production impact still needs a Spark run on the target AMD executor.

Notes

• No unsafe code, architecture-specific SIMD, index-format changes, or user-facing configuration are added.
• Recall, filtering semantics, and the number of PQ codes evaluated are unchanged.

[shyjsarah]

This guard checks a stale environment variable name. The exercised search path currently reads PAIMON_VINDEX_LOG_IVFPQ_BATCH_TIMING and PAIMON_VINDEX_LOG_IVFPQ_BATCH_REUSE, so enabling either diagnostic still passes this assertion and can add instrumentation/logging overhead to the measured samples. Could we assert that both current variables are unset before running the benchmark?

[shyjsarah]

+1