Repository navigation
Test vector index - #6716
Closed
jerry-024 wants to merge 14 commits into
Closed
Test vector index#6716jerry-024 wants to merge 14 commits into
jerry-024 wants to merge 14 commits into
Conversation
jerry-024
marked this pull request as draft
December 1, 2025 09:37
* upstream/master: [test] Fix test SparkWriteITCase.testTruncatePartitionValueNull [orc] Limiting Memory Usage of OrcBulkWriter When Writing VectorizedRowBatch (apache#6590) [arrow] Improve customization capabilities for data type conversion. (apache#6695) [spark] Fix NPE in spark truncate null partitions [core] Exclude .class files from sources.jar (apache#6707) [core] DataEvolutionFileStoreScan should not filter files by read type when it contains no physical columns. (apache#6714) [spark] Update scalafmt version to 3.10.2 (apache#6709) [variant] Extract only required columns when reading shredded variants (apache#6720) [python] Fix read large volume of blob data (apache#6701) [flink] Support cdc source (apache#6606) [hive] fix splitting for bucket tables (apache#6594) [spark] Update spark build topology for global index (2) (apache#6703) [test][spark] Fix the flaky test setDefaultDatabase (apache#6696) [spark] Update global index build topology (apache#6700) [spark] Introduce global file index builder on spark (apache#6684) [python] Fix with_shard feature for blob data (apache#6691) [test][spark] Add alter with incompatible col type test case (apache#6689) [variant] Introduce withVariantAccess in ReadBuilder (apache#6685) [python] Fix file name prefix in postpone mode. (apache#6668)
* upstream/master: [core] format table: throw exception when get partiiton from file system (apache#6730) [core] Enrich static creation methods for GlobalIndexResult [spark] Fix duplicate column error when merging on _ROW_ID (apache#6727) [core] Remove Builder of SplitReadProvider.Context [core] Intruduce bitmap64 for GlobalIndexResult and support with ranges filter push down (apache#6725) [spark] Push down partition filter when compactUnAwareBucketTable (apache#6663) [core] Support the new split description for reading data (apache#6711) [python] support paimon as ray datasource for distributed processing (apache#6686) [docs] Add missing PaimonSparkSessionExtensions to Spark configs (apache#6729) Revert "[flink] Flink batch job support specify partition with max_pt() and max_two_pt()" without review [flink] Flink batch job support specify partition with max_pt() and max_two_pt() [spark] Update the display of size and timing metrics (apache#6722) [core] Append commit should check bucket number if latest commit user is different (apache#6723) [core] ShardScanner should not keep partition parameter (apache#6724)
jerry-024
force-pushed
the
test_vector_index
branch
from
December 4, 2025 09:17
1179721 to
213332f
Compare
jerry-024
force-pushed
the
test_vector_index
branch
from
December 5, 2025 06:34
a61e47d to
fec2404
Compare
jerry-024
force-pushed
the
test_vector_index
branch
from
December 5, 2025 09:11
aef25b9 to
0219ab8
Compare
* upstream/master: [core] Upgrade LZ4 dependency to 1.8.1 (apache#6737) [core] Update the result of show global system tables (apache#6751) [python] Introduce lance file format support to pypaimon (apache#6746) [Python] Support DLF OSS endpoint override in RESTTokenFileIO (apache#6749) [python] fix pypaimon manifest entry incomplete identifier issue (apache#6748) [spark] Merge into supports _ROW_ID shortcut (apache#6745) [core] Fix BlockIterator may not move forward after calling `next()` (apache#6738)
jerry-024
force-pushed
the
test_vector_index
branch
2 times, most recently
from
December 8, 2025 02:41
049fb3c to
f136243
Compare
jerry-024
force-pushed
the
test_vector_index
branch
from
December 8, 2025 03:30
0a7a3f4 to
72c0389
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Introduce vector index support for Paimon using Apache Lucene 9.x to enable:
Implementation
Tests
API and Format
Documentation