[SteNicholas]

Purpose

Linked issue: close #405

A full-text search in Paimon C++ could not carry a row filter, so a caller had to filter after
ranking and got fewer than limit rows. This adds FullTextSearchBuilder::WithFilter(), aligned
with Java FullTextSearchBuilder#withFilter (apache/paimon#9855): only the matching rows are
ranked, so the result is the top k among them, and they reach the existing index as include row
ids, so BM25 statistics stay those of the full corpus.

• FullTextSearchBuilder: validates the filter and splits it as Java
  PartitionPredicate#splitPartitionPredicatesAndDataPredicates does. Conjuncts over partition
  fields alone are ANDed into the partition filter, and the rest become the data filter. A conjunct
  over both partition and data fields stays in the data filter and also restricts the partitions it
  implies, projected as Java PredicateProjectionConverter does, so
  (p = 1 AND a > 0) OR (p = 2 AND a > 0) also searches only p = 1 OR p = 2. Repeated calls are
  ANDed.
• DataEvolutionFullTextScan: also loads the index files whose indexed field is a filter
  field, and attaches to each split the intersecting files of the single-field scalar indexes on
  the filter fields. Unlike in Java, the full-text indexes of every column are left out, as
  lucene-fts answers every scalar predicate with all of its rows, and so are multi-field indexes,
  which the read leaves out of the evaluation anyway. A scalar index file whose row range is
  negative, reversed or overflows int64 fails the plan.
• IndexFullTextSearchSplit: version 2 of the encoding appends the scalar index files, as
  version 2 of Java IndexFullTextSearchSplit does; version 1 splits are still read.
• DataEvolutionFullTextRead::MatchedRows(): resolves the data filter before ranking with the
  scalar indexes of the splits, read per field as one union (GlobalIndexScanImpl:: CreateForScalarFilters() and ScanWithCoverage(), as Java DataEvolutionGlobalIndexScanner).
  • Rows that no scalar index covers are excluded, as in the fast scalar-index.search-mode. If
    no index can evaluate the filter at all, nothing is searched and a warning is logged.
  • If the indexes return only a superset, the candidates are excluded with a warning by default,
    so the search never returns a non-matching row. With the new
    global-index.filter.refine-from-data option (default false, Java description), they are
    verified against the data instead.
  • Each split searches with include row ids searchRowRanges ∧ liveRows ∧ matchedRows.
• GlobalIndexEvaluatorImpl: reports the fields that decided a result
  (EvaluateWithContributingFields(), as Java GlobalIndexEvaluator.Evaluation), so that a
  superset can be told from an exact answer: not every filter field contributed, or there is a
  Contains, EndsWith or Like leaf. Unlike in Java, a field with a conjunct that no index could
  evaluate does not count even if another of its conjuncts was evaluated, as the result is then
  only a superset for that field.
• FilteredRowIdReader: as in Java, reads only the filter columns and _ROW_ID of the
  candidates, with prefetch, pinned to the planned snapshot and with the schema that the search
  resolved, so a column renamed or replaced since then cannot point the filter at another field.
  Unlike in Java, once the planned snapshot has expired, a search of scan.tag-name reads the tag
  instead, which must still refer to the planned snapshot.
• StaticFromTagStartingScanner: reads tags from the branch of the scan, as in Java, instead
  of always from the main branch.
• Options: scalar-index.search-mode falls back on global-index.search-mode, as Java
  CoreOptions#scalarIndexSearchMode does. Only fast is supported so far; a row filter in full
  or detail fails with NotImplemented, as does a filter with a full-text match.

Tests

UT:

• src/paimon/core/table/source/data_evolution_full_text_read_test.cpp, with fake scalar indexers:
  TestFilterRanksOnlyMatchedRows, TestFilterExcludesRowsThatIndexesCannotDecide (no scalar
  file, a declining one, a multi-field one that the read leaves out, a partly indexed conjunction,
  Contains/EndsWith/Like, rows that no scalar file covers),
  TestFilterUnionsIndexesOfDifferentTypes, and filter validation in
  TestBuilderValidatesSettings (unknown field, _ROW_ID, mismatched type, full-text match, a
  scalar search mode other than fast, null filter).
• src/paimon/core/table/source/data_evolution_full_text_scan_test.cpp: TestScalarPreFilterFiles
  (only single-field indexes that do not support full-text search, without multi-field indexes or
  an empty filter) and TestIntersectingFiles, including invalid row ranges.
• src/paimon/core/table/source/full_text_search_split_test.cpp: TestSerializeRoundTrip (the
  version 2 bytes are pinned and version 1 bytes still read), TestSerializeScalarIndexFiles, and
  invalid scalar index files in TestDeserializeRejectsInvalidBytes,
  TestDeserializeRejectsCorruptEntries and TestCreateValidatesArguments.
• src/paimon/core/table/source/filtered_row_id_reader_test.cpp:
  TestExactWhenEveryFilterFieldContributed and TestCandidateOnlyLeavesAreNotExact.
• src/paimon/core/global_index/global_index_evaluator_impl_test.cpp: ReportsContributingFields,
  AndWithEmptyResultShortCircuits and CollectFieldIds.
• src/paimon/core/core_options_test.cpp: TestParseIndexSearchMode for the full-text and scalar
  families, and global-index.filter.refine-from-data in TestDefaultValue and TestFromMap.

IT (test/inte/global_index_test.cpp), aligned with Java NativeFullTextRowFilterTest and
FullTextSearchBuilderTest:

• TestFullTextTableFullTextSearchWithFilter and TestLuceneTableFullTextSearchWithFilter:
  equality, range, IN, NotEqual, OR across fields, no and all matching rows, equality on a
  BTree of the text column, accumulated filters, the top row among matching rows with limit 1,
  a column without an index or with only a full-text index, a partly indexed conjunction and
  Contains/EndsWith/Like with refine on and off, scalar index files attached only for filtered
  columns, serialized splits, and refine with scan.snapshot-id, scan.tag-name and
  scan.timestamp-millis.
• TestFullTextTableFullTextSearchWithPartitionAndDataFilter and
  TestLuceneTableFullTextSearchWithPartitionAndDataFilter: partition and data predicates combined,
  with SetPartitionFilter() and refine, and the partitions that a mixed conjunct implies.
• TestFullTextSearchSkipsRowsDeletedByDeletionVectors and
  TestLuceneSearchSkipsRowsDeletedByDeletionVectors: a filter with deletion vectors, a scalar index
  file per shard, and refine across shards.
• TestFilteredRowIdReader: deletion vectors, a pinned older snapshot, a tag moved to another
  snapshot, the tag fallback once the snapshot has expired (also with an empty plan), a branch tag,
  and a column renamed or replaced after planning.

API and Format

New public API: FullTextSearchBuilder::WithFilter(), and Options::SCALAR_INDEX_SEARCH_MODE and
Options::GLOBAL_INDEX_FILTER_REFINE_FROM_DATA. The serialized form of full-text search splits,
which is specific to Paimon C++, moves to version 2 and still reads version 1. No storage format or
protocol change.

Documentation

Yes. docs/source/user_guide/global_index.rst adds a "Row Filters" section covering the predicate
split, implied partitions, the fast mode, excluded candidates and
global-index.filter.refine-from-data, and the docs of FullTextSearchBuilder, FullTextScan,
FullTextRead and the new options describe the filter.

Generative AI tooling

Generated-by: Claude Code 2.1.311 (Claude Opus 5.5)

🤖 Generated with Claude Code