Introduce the InMemoryDataSet interface. - #718
Draft
ashkrisk wants to merge 1 commit into
Draft
Conversation
Separate part of functionality provided by the `DataSet` interface into the `InMemoryDataSet` interface to allow for DataSets that are not held in-memory. The current `DataSet` interface requires all the base vectors in the corpus to be made available as a `List<VectorFloat<?>>`. This makes it difficult to create a `DataSet` implementation that reads data from disk without manifesting it in-memory. To solve this, the new `InMemoryDataSet` interface is created and the scope of the original `DataSet` interface is reduced. The `DataSet` interface no longer provides all base vectors in memory, however, it retains the ability to acces base vectors through a RAVV. Summary of changes: - Split-off `getBaseVectors` from the `DataSet` interface into the `InMemoryDataSet` interface - References to `getBaseVectors` are scrubbed from `Grid` to allow it to use the reduced `DataSet` interface. No performance impact is expected since `Grid` only used the base vector list for counting the number of vectors. - `CompactorBenchmark` (and by extension, `AutoBenchYAML`) now uses `InMemoryDataSet` instead of the plain `DataSet`. The list of base vectors is used for partitioning, and porting this to the reduced `DataSet` interface is left to future patches.
Contributor
|
Before you submit for review:
If you did not complete any of these, then please explain below. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Separate part of functionality provided by the
DataSetinterface into theInMemoryDataSetinterface to allow for DataSets that are not held in-memory.The current
DataSetinterface requires all the base vectors in the corpus to be made available as aList<VectorFloat<?>>. This makes it difficult to create aDataSetimplementation that reads data from disk without manifesting it in-memory.To solve this, the new
InMemoryDataSetinterface is created and the scope of the originalDataSetinterface is reduced. TheDataSetinterface no longer provides all base vectors in memory, however, it retains the ability to acces base vectors through a RAVV.Summary of changes:
getBaseVectorsfrom theDataSetinterface into theInMemoryDataSetinterfacegetBaseVectorsare scrubbed fromGridto allow it to use the reducedDataSetinterface. No performance impact is expected sinceGridonly used the base vector list for counting the number of vectors.CompactorBenchmark(and by extension,AutoBenchYAML) now usesInMemoryDataSetinstead of the plainDataSet. The list of base vectors is used for partitioning, and porting this to the reducedDataSetinterface is left to future patches.