What is the problem the feature request solves?
Spark 3.5+ exposes the Apache DataSketches HyperLogLog functions, which give approximate distinct counting with a sketch that can be persisted and merged later:
hll_sketch_agghll_union_agghll_sketch_estimatehll_union
Comet falls back to Spark for all four, so any query that builds or reads an HLL sketch column loses native execution for that part of the plan.
docs/source/user-guide/latest/expressions.md listed the hll_* family under Not currently planned, alongside a note that the families there "may be reconsidered based on demand". This issue is that reconsideration: unlike the other sketch families, the HLL sketch has a pure-Rust Apache DataSketches implementation available, which removes the main cost objection.
Describe the potential solution
Implement all four natively for Spark 4.0+, backed by the pure-Rust datasketches crate so that no C++ toolchain enters the build.
Cross-engine compatibility is the interesting constraint here, because a sketch column is data a user persists and reads back. The crate uses MurmurHash3-x64-128 with the standard DataSketches update seed (9001) and hashes input bytes the same way datasketches-java does, so the sketches are mutually readable: Spark can read a Comet-produced sketch and vice versa. For a high-cardinality HLL-array sketch the serialized bytes are identical.
The point estimate is not bit-identical, though. After a partial/final merge the sketch is flagged out-of-order and estimated with a composite estimator whose interpolation and bias tables differ slightly between the Rust crate and datasketches-java — around 0.7% observed, well inside HLL's ~1.6% standard error at the default lgConfigK. So the functions should be reported as Incompatible: Comet falls back to Spark for exact results by default, and users opt in per expression with spark.comet.expression.<name>.allowIncompatible=true to accept the small difference in exchange for native execution.
Scoping to Spark 4.0+ keeps the shared serde tree compiling on 3.4, where the functions do not exist.
Additional context
Implemented in #4802.
What is the problem the feature request solves?
Spark 3.5+ exposes the Apache DataSketches HyperLogLog functions, which give approximate distinct counting with a sketch that can be persisted and merged later:
hll_sketch_agghll_union_agghll_sketch_estimatehll_unionComet falls back to Spark for all four, so any query that builds or reads an HLL sketch column loses native execution for that part of the plan.
docs/source/user-guide/latest/expressions.mdlisted thehll_*family under Not currently planned, alongside a note that the families there "may be reconsidered based on demand". This issue is that reconsideration: unlike the other sketch families, the HLL sketch has a pure-Rust Apache DataSketches implementation available, which removes the main cost objection.Describe the potential solution
Implement all four natively for Spark 4.0+, backed by the pure-Rust
datasketchescrate so that no C++ toolchain enters the build.Cross-engine compatibility is the interesting constraint here, because a sketch column is data a user persists and reads back. The crate uses MurmurHash3-x64-128 with the standard DataSketches update seed (9001) and hashes input bytes the same way datasketches-java does, so the sketches are mutually readable: Spark can read a Comet-produced sketch and vice versa. For a high-cardinality HLL-array sketch the serialized bytes are identical.
The point estimate is not bit-identical, though. After a partial/final merge the sketch is flagged out-of-order and estimated with a composite estimator whose interpolation and bias tables differ slightly between the Rust crate and datasketches-java — around 0.7% observed, well inside HLL's ~1.6% standard error at the default
lgConfigK. So the functions should be reported asIncompatible: Comet falls back to Spark for exact results by default, and users opt in per expression withspark.comet.expression.<name>.allowIncompatible=trueto accept the small difference in exchange for native execution.Scoping to Spark 4.0+ keeps the shared serde tree compiling on 3.4, where the functions do not exist.
Additional context
Implemented in #4802.