Repository navigation
(Calculated fields) Allow aggregations on calculated fields - #6905
Draft
fulmicoton wants to merge 1 commit into
Draft
fulmicoton wants to merge 1 commit into
fulmicoton wants to merge 1 commit into
Conversation
Add `calculated_fields` to `SearchRequest`: named jitexpr expressions computed from fast fields. Aggregations can use their names like fast field names. - Bump tantivy to get the jitexpr `ValueSource` for aggregations. - At the leaf, the calculated fields are registered in the aggregation context as `JitExprValueSourceProvider`s. - Warmup replaces calculated fields by the fast fields their expressions read. - Root validation rejects invalid expressions and checks that the expression inputs are fast fields.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This plugs tantivy's jitexpr value sources for aggregations (quickwit-oss/tantivy#3162) into Quickwit.
SearchRequestgets a newcalculated_fieldsfield: a list of named jitexpr expressions, computed from fast fields. Aggregations can refer to these names like fast field names. This follows #6777, which added calculated-field predicates to the query AST.Changes
JitExprValueSourceProvider.quickwit-search/src/calculated_fields.rs. It parses and type checks the calculated fields, and rejects empty and duplicate names.calculated_fieldsis not empty, the collector registers them inAggContextParamswithwith_value_sources. The merge collector does not need them, because merging intermediate results does not read columns.calculated_fields, together with the aggregation.A calculated field shadows a fast field with the same name, so a prefix such as
#is recommended.Not in this PR
calculated_fieldsyet. Only the proto/gRPC API does._keyordering for text outputs, andmin_doc_count: 0.Tests
test_single_node_aggregation_on_calculated_fields: a terms aggregation on(UPPER color)with a sum on(MULTIPLY price quantity).test_single_node_aggregation_calculated_field_requires_fast_inputs.quickwit-doc-mapper'stest_concatenate_multiple_fieldfails, but it fails onmaintoo. It is not related to this PR.🤖 Generated with Claude Code