Poc index layout by thorfour · Pull Request #9706 · vortex-data/vortex

thorfour · 2026-08-31T20:05:13Z

Summary

Tracking Issue: Tracking Issue: Index Layouts #9024

This is a PoC of the Indexed Layout described in the above issue.

Changes

This implements the IndexVTable trait that Index implementations can implement to be utilized in an IndexLayout.

Adds an example ReverseIndex that implements the trait and it's use is demonstrated in a couple unit tests. Additionally adds a datafusion test that validates that fewer bytes are read from a Vortex file when an indexed layout is successfully utilized.

Adds an indexed layout to the Vortex layout crate based on vortex-data#9024

Intended as an example implementation of a indexed-layout index.

codspeed-hq · 2026-09-01T10:13:28Z

Merging this PR will degrade performance by 16.97%

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 3 improved benchmarks
❌ 8 regressed benchmarks
✅ 2079 untouched benchmarks
⏩ 206 skipped benchmarks¹
🗄️ 4 archived benchmarks run²

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	WallTime	`multiply_shapes_neon[(128, PerRowPerRow)]`	2 µs	3.8 µs	-48.18%
❌	WallTime	`add_shapes_neon[(128, PerRowPerRow)]`	1.8 µs	3.1 µs	-41.31%
❌	WallTime	`arrow_checked_add_u32_neon[16384]`	12.4 µs	20.3 µs	-38.91%
❌	WallTime	`subtract_shapes_neon[(128, PerRowPerRow)]`	1.9 µs	2.9 µs	-34.22%
❌	Simulation	`cached_indices_i128[0.01]`	48 µs	61.6 µs	-22.01%
❌	WallTime	`add_u32_nonnull_neon`	6.6 µs	7.6 µs	-12.99%
❌	WallTime	`add_i64_nonnull_neon`	9.8 µs	11.2 µs	-12.95%
❌	Simulation	`compress_fsst[(500, 64, 8)]`	482.1 µs	536.2 µs	-10.09%
⚡	Simulation	`cached_indices_i32[0.01]`	51.6 µs	37.5 µs	+37.36%
⚡	WallTime	`arrow_checked_add_u32_avx512[16384]`	21.3 µs	17.6 µs	+20.77%
⚡	WallTime	`arrow_checked_add_u32_avx2[16384]`	21.3 µs	17.7 µs	+20.16%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing polarsignals:poc-index-layout (f81f922) with develop (97953be)}

206 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
4 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩

joseph-isaacs · 2026-09-01T10:12:34Z

+        let data_eval = self
+            .data_child()?
+            .pruning_evaluation(row_range, expr, mask.clone())?;
+
+        let Some(probe) = self.probe(expr)? else {
+            return Ok(data_eval);
+        };


is it worth checking if the probe is cached and not checking data child pruning eval if it prunes the whole range?

joseph-isaacs · 2026-09-01T10:17:05Z

+    /// Decide whether this index can serve `expr`, a single conjunct scoped to the data child's
+    /// dtype.
+    ///
+    /// `None` means "no claim" and is always safe: the scan falls back to the data child.
+    fn plan(
+        &self,
+        expr: &BoundExpression,
+        dtype: &DType,
+        options: &[u8],
+    ) -> VortexResult<Option<IndexQueryPlan>>;


do you expect the plan to be a pruning expr or a regular expression?

See

vortex/vortex-array/src/stats/rewrite/builtins.rs

Line 56 in 93b0535

pub(crate) fn register_builtins(session: &StatsSession) {

We currently convert expressions into pruning expression before calling prune_expression.

I am not sure on this design but this is what happens now.

joseph-isaacs · 2026-09-01T10:27:02Z

+    fn builder(
+        &self,
+        dtype: &DType,
+        options: &[u8],


what is this?

It lets the writer configure a specific attachment of that index kind — e.g. a bloom filter's bit width or hash count, a trigram index's n-gram size.

joseph-isaacs · 2026-09-01T10:32:04Z

+//! A prototype of the `vortex.indexed` layout proposed in
+//! [vortex-data/vortex#9024](https://github.com/vortex-data/vortex/issues/9024).
+//!
+//! A `vortex.indexed` layout wraps a data layout with zero or more *locating indexes* held as
+//! auxiliary children. Writers build indexes through the pluggable [`IndexVTable`] registry while
+//! streaming; readers probe those indexes to prune (or outright answer) filter predicates, and
+//! fall back to a plain scan of the data child whenever an index is missing, unknown, or has no
+//! claim on the expression.
+//!
+//! Indexes are optional at every stage. A builder that finds nothing worth keeping declines once


do you expect nested indexes, or a single index from the root for each leaf?

We would for example have struct({a: Indexed(struct{b: Indexed(Flat)})}). This is important since it informs how we probe children and merge indexes.

It would be that Index outer or inner is more precise.

Worth a comment on how they compose

joseph-isaacs · 2026-09-01T10:36:49Z

+            RowLocator::Rows(rows) => {
+                let start = u32::try_from(row_range.start)?;
+                let end = u32::try_from(row_range.end)?;
+                let mut pos = row_range.start;
+                for row in rows.range(start..end) {
+                    let row = u64::from(row);
+                    bits.append_n(false, usize::try_from(row - pos)?);
+                    bits.append_n(true, 1);
+                    pos = row + 1;
+                }
+                bits.append_n(false, usize::try_from(row_range.end - pos)?);
+            }
+            // Broadcast each block's bit across the rows it covers, clipped to `row_range`.
+            RowLocator::Blocks { block_len, ids } => {
+                let mut row = row_range.start;
+                while row < row_range.end {
+                    let block = row / block_len;
+                    let block_end = ((block + 1) * block_len).min(row_range.end);
+                    let hit = ids.contains(u32::try_from(block)?);
+                    bits.append_n(hit, usize::try_from(block_end - row)?);
+                    row = block_end;
+                }
+            }
+        }


This looks quite low. append_n is pretty expensive. Can be optimised later

joseph-isaacs · 2026-09-01T10:42:26Z

+/// [`IndexSession`](crate::layouts::indexed::session::IndexSession) under a stable string id,
+/// which is what gets written into the layout metadata. A reader that does not have the kind
+/// registered drops the index child and reads the data child directly.
+pub trait IndexVTable: 'static + Send + Sync + Debug {


Where do we define what layout children are allowed for each index?
At some point we convert a SendableArrayStream into a sterilised layout and later back..

What is the code that validate serde for this

joseph-isaacs · 2026-09-01T10:42:44Z

Looking promising

joseph-isaacs · 2026-09-01T14:43:59Z

+pub trait IndexResolve: 'static + Send + Sync {
+    /// `postings` are the index-child rows that survived [`IndexQueryPlan::filter`], projected in
+    /// the index child's own schema.
+    fn resolve(
+        &self,
+        postings: &ArrayRef,
+        data_row_count: u64,
+        ctx: &mut ExecutionCtx,
+    ) -> VortexResult<RowLocator>;


should we allow this to specify a row range and avoid loading in data for values outside that row_range?

Then we could use this index with conjunction with a selection to only load some of the index

joseph-isaacs · 2026-09-01T14:47:16Z

I wonder if should have the following:

a way to specify if the index data is contiguous or spread out

a way to chunk index are different size (a bloom filter over the whole file, tri-gram index of every 250k row)

Two support the second we would need to change the layout strategy to support chunking of some of the children layouts (likely have a strategy that chunks that layout child at the boundary specified so reading back that row range actually reads only that chunk).

I believe we can already perform number 2 if we simply perform nesting of the layouts:

e.g: Indexed(Indexed(data, ChunkedLayout), FlatLayout)

I believe the first bullet can be implemented with composition as well right? Sorry if these are stupid questions.

Chunked(ChunkBy::RowCount(100_000), Indexed(...))

Or is this not quite what you meant?

vortex-reverse-index was a standalone worked example of the vortex.indexed IndexVTable contract; move it to vortex-layout/examples/reverse_index, following the plain cargo-example convention used by vortex-ffi/examples instead of shipping it as its own workspace crate. vortex-datafusion could no longer dev-depend on it as a library once it became a cargo example, so its DataFusion integration test moves along with it (examples/reverse_index/datafusion_tests.rs), rebuilt against vortex-datafusion's public API and a manually assembled session instead of the vortex facade crate, which vortex-layout cannot depend on. Signed-off-by: "Thor" <thor.hansen@dash0.com>

IndexedReader::plan_probe returned on the first spec that claimed an expression, regardless of exactness. A Superset claim from an earlier spec could shadow a later, more precise Exact claim, and multiple Superset claims on the same expression never got combined. CachedProbe is now Exact(locator) or Superset(Vec<locator>): plan_probe scans every spec, preferring any Exact claim outright and otherwise collecting all Superset claims. pruning_evaluation intersects every Superset locator (short-circuiting once nothing survives), and filter_evaluation only fast-paths on an Exact claim. Signed-off-by: "Thor" <thor.hansen@dash0.com>

thorfour added 4 commits August 31, 2026 14:58

PoC: IndexedLayout

930da9c

Adds an indexed layout to the Vortex layout crate based on vortex-data#9024

vortex-reverse-index: An example index

f80bc95

Intended as an example implementation of a indexed-layout index.

Datafusion tests with an indexed layout

df008c6

Cargo.toml

f81f922

joseph-isaacs reviewed Sep 1, 2026

View reviewed changes

Comment thread vortex-layout/examples/reverse_index/main.rs

joseph-isaacs reviewed Sep 1, 2026

View reviewed changes

thorfour added 2 commits September 1, 2026 15:57

Uh oh!

Conversation

thorfour commented Aug 31, 2026

Summary

Changes

Uh oh!

codspeed-hq Bot commented Sep 1, 2026

Merging this PR will degrade performance by 16.97%

Performance Changes

Footnotes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joseph-isaacs commented Sep 1, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants