[Feature] Support independent vector indexes on multiple columns in primary-key tables #10658

Search before asking

• I searched in the issues and found nothing similar.

Motivation

I would like to create independent vector indexes on multiple vector columns in the same primary-key table.

A typical use case is a record containing both image_embedding and text_embedding, or embeddings produced by different models. Each column needs independent ANN search and may have a different dimension, distance metric, and index configuration.

Currently, pk-vector.index.columns accepts exactly one column. As a result, multiple vector fields belonging to the same entity cannot all have primary-key vector indexes in one table. Splitting them into separate tables adds data duplication and coordination work for updates and deletes.

Solution

Allow pk-vector.index.columns to contain multiple columns, with each column owning an independent index configured through the existing fields.<column>.pk-vector.* options.

For example, the following Flink SQL illustrates the desired configuration:

CREATE TABLE items (
    id BIGINT,
    image_embedding ARRAY<FLOAT> COMMENT '__VECTOR_FIELD;512',
    text_embedding ARRAY<FLOAT> COMMENT '__VECTOR_FIELD;768',
    PRIMARY KEY (id) NOT ENFORCED
) WITH (
    'bucket' = '4',
    'merge-engine' = 'deduplicate',
    'deletion-vectors.enabled' = 'true',
    'deletion-vectors.merge-on-read' = 'false',
    'pk-vector.index.columns' = 'image_embedding,text_embedding',
    'fields.image_embedding.pk-vector.index.type' = 'ivf-flat',
    'fields.image_embedding.pk-vector.distance.metric' = 'cosine',
    'fields.text_embedding.pk-vector.index.type' = 'ivf-flat',
    'fields.text_embedding.pk-vector.distance.metric' = 'l2'
);

Expected behavior:

• Both columns can be indexed, with validation of each column's type, dimension, and index settings.
• Queries explicitly select a vector column and use that column's index and options.
• Index maintenance, compaction, recovery, and deletion-vector filtering work correctly for every configured field under the existing snapshot and index-coverage semantics.
• Existing single-column configurations continue to work unchanged.

The initial scope is independent per-column indexing and querying. Combining results from multiple vector fields can be discussed separately.

Anything else?

The example above describes the requested behavior; it is currently rejected by the single-column schema check.

The current limitation is documented here:
https://paimon.apache.org/docs/master/primary-key-table/global-index/#limitations

PR #8549 already describes the plural option name as preparation for independent indexes on multiple vector fields. This request follows that direction.

Are you willing to submit a PR?

• I'm willing to submit a PR!

Implementation PRs

• [core] Prepare PK vector index lifecycle for multiple columns #10688 — Prepare the vector index maintainer lifecycle.
• [core] Enable multiple vector index columns on primary-key tables #10755 — Enable multiple vector index columns.
• [core] Support bounded parallel builds for primary-key indexes #10756 — Support bounded parallel primary-key index builds.

Merge order: #10688 → #10755 → #10756.