Skip to content

(Calculated fields) Allow aggregations on calculated fields - #6905

Draft
fulmicoton wants to merge 1 commit into
mainfrom
paul.masurel/agg-calculated-fields
Draft

fulmicoton wants to merge 1 commit into
mainfrom
paul.masurel/agg-calculated-fields

Conversation

@fulmicoton

Copy link
Copy Markdown
Collaborator

This plugs tantivy's jitexpr value sources for aggregations (quickwit-oss/tantivy#3162) into Quickwit.

SearchRequest gets a new calculated_fields field: a list of named jitexpr expressions, computed from fast fields. Aggregations can refer to these names like fast field names. This follows #6777, which added calculated-field predicates to the query AST.

"calculated_fields": [
  { "name": "#total", "expression": "(MULTIPLY price quantity)" }
],
"aggregation_request": "{\"by_color\": {\"terms\": {\"field\": \"color\"}, \"aggs\": {\"total\": {\"sum\": {\"field\": \"#total\"}}}}}"

Changes

  • Bump tantivy to the tip of main, to get JitExprValueSourceProvider.
  • New quickwit-search/src/calculated_fields.rs. It parses and type checks the calculated fields, and rejects empty and duplicate names.
  • Leaf: when calculated_fields is not empty, the collector registers them in AggContextParams with with_value_sources. The merge collector does not need them, because merging intermediate results does not read columns.
  • Warmup: the collector replaces the calculated fields used by aggregations with the fast fields their expressions read. Unused calculated fields are not warmed up.
  • Root: invalid expressions are rejected before the fan-out. The inputs of a calculated field must be fast fields.
  • Scroll requests drop calculated_fields, together with the aggregation.

A calculated field shadows a fast field with the same name, so a prefix such as # is recommended.

Not in this PR

  • The REST search API and the Elasticsearch-compatible API do not expose calculated_fields yet. Only the proto/gRPC API does.
  • Tantivy rejects some terms aggregation options on computed fields: _key ordering for text outputs, and min_doc_count: 0.

Tests

  • Unit tests for parsing, the warmup resolution and the collector errors.
  • test_single_node_aggregation_on_calculated_fields: a terms aggregation on (UPPER color) with a sum on (MULTIPLY price quantity).
  • test_single_node_aggregation_calculated_field_requires_fast_inputs.

quickwit-doc-mapper's test_concatenate_multiple_field fails, but it fails on main too. It is not related to this PR.

🤖 Generated with Claude Code

Add `calculated_fields` to `SearchRequest`: named jitexpr expressions
computed from fast fields. Aggregations can use their names like fast
field names.

- Bump tantivy to get the jitexpr `ValueSource` for aggregations.
- At the leaf, the calculated fields are registered in the aggregation
  context as `JitExprValueSourceProvider`s.
- Warmup replaces calculated fields by the fast fields their expressions
  read.
- Root validation rejects invalid expressions and checks that the
  expression inputs are fast fields.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants