Skip to content

Allow benchmarking larger-than-memory datasets - #721

Open
ashkrisk wants to merge 1 commit into
ltm-ds/fvec-ravvfrom
ltm-ds/ltm-ds
Open

Allow benchmarking larger-than-memory datasets#721
ashkrisk wants to merge 1 commit into
ltm-ds/fvec-ravvfrom
ltm-ds/ltm-ds

Conversation

@ashkrisk

@ashkrisk ashkrisk commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Build on patches introduced earlier in the series to allow BenchYAML to process that don't fit in physical memory, by memory-mapping the index vectors instead of fully loading them upfront.

This behaviour is opt-in and requires you to set -Djvector.bench.dataset.mmap.enable=true to enable.

Additionally, fixes a concurrency issue with RAVV handling in Grid, and makes the output directories configurable

Summary of changes:

  • Add a RavvDataSet wherein the index vectors are backed by a RAVV.
  • Add a function to DataSetInfo to return a DataSet backed by memory-mapped files, using RavvDataSet.
  • Consume the new DataSet from BenchYAML
  • Fix a concurrency issue in Grid where RAVV#getVector was being called from multiple threads at once
  • Make Grid's work directory and index cache directory configurable via environment variables. (Useful if you have an SSD on a separate mount)

search_heap_use

@github-actions

github-actions Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Before you submit for review:

  • Does your PR follow guidelines from CONTRIBUTIONS.md?
  • Did you summarize what this PR does clearly and concisely?
  • Did you include performance data for changes which may be performance impacting?
  • Did you include useful docs for any user-facing changes or features?
  • Did you include useful javadocs for developer oriented changes, explaining new concepts or key changes?
  • Did you rebase your branch onto the latest main for regression testing and PR submission?
  • Did you trigger regression testing via Run Bench Main and review results?
  • Did you adhere to the code formatting guidelines (TBD)
  • Did you group your changes for easy review, providing meaningful descriptions for each commit?
  • Did you ensure that all files contain the correct copyright header?
  • Did you add documentation for this feature to the release notes directory?

If you did not complete any of these, then please explain below.

@ashkrisk
ashkrisk force-pushed the ltm-ds/ltm-ds branch 3 times, most recently from 47442fb to f11246a Compare August 30, 2026 12:40
Build on patches introduced earlier in the series to allow BenchYAML to
process that don't fit in physical memory, by memory-mapping the index
vectors instead of fully loading them upfront.

This behaviour is opt-in and requires you to set
`-Djvector.bench.dataset.mmap.enable=true` to enable.

Additionally, fixes a concurrency issue with RAVV handling in Grid, and
makes the output directories configurable

Summary of changes:
- Add a `RavvDataSet` wherein the index vectors are backed by a RAVV.
- Add a function to `DataSetInfo` to return a `DataSet` backed by
  memory-mapped files, using `RavvDataSet`.
- Consume the new `DataSet` from `BenchYAML`
- Fix a concurrency issue in Grid where `RAVV#getVector` was being
  called from multiple threads at once
- Make Grid's work directory and index cache directory configurable via
  environment variables. (Useful if you have an SSD on a separate mount)
@ashkrisk
ashkrisk marked this pull request as ready for review August 31, 2026 06:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant