test: add CometExpressionCoverageSuite for runtime expression coverage audit

[andygrove]

Which issue does this PR close?

There is no dedicated issue. This adds a developer utility for auditing expression coverage. Related to the expression coverage epic #240.

Rationale for this change

Finding Spark expressions that Comet does not accelerate is currently a manual, error-prone exercise. Searching the source is unreliable because Comet dispatches expressions through several mechanisms: the serde maps, the StaticInvoke method map, RuntimeReplaceable rewrites to other handled expressions, and the Spark 4.x expression shims. A grep-based audit misses all but the first and produces false positives (for example flagging from_xml, aes_encrypt, or array_prepend as unsupported when they are handled via a shim or a rewrite).

This adds a runtime audit that determines coverage by actually planning each function and inspecting the physical plan, which accounts for every dispatch path. It doubles as a per-Spark-version coverage report.

What changes are included in this PR?

• CometExpressionCoverageSuite: for every function in Spark's registry it runs the function's own documented example (ExpressionInfo.getExamples) over a Comet-scanned table, then uses findFirstNonCometOperator and ExtendedExplainInfo.getFallbackReasons to classify the result. Categories: NATIVE, INCOMPATIBLE (supported but opt-in), LITERAL_PROBE (supported; the all-literal example tripped an all-constant guard that would not fire for column inputs), GAP (genuine fallback, with the fallback reason and probed query recorded), SKIP, and ERROR. The report is written to spark/target/expression-coverage-report-spark-<version>.md.
• A note in the contributor guide (adding_a_new_expression.md) on how to run it and read the report, including running it under each Spark profile to compare versions.

On Spark 4.1 it currently reports roughly 330 native, a handful incompatible/opt-in, and a GAP list dominated by families already documented as not-planned (variant, geospatial, sketches, reflection) plus a few genuine candidates.

How are these changes tested?

The suite is itself a test and asserts that it probes a non-empty set of functions. It was run against the Spark 4.1 profile to produce the coverage report and validate the classification (including confirming that expressions handled via shims and rewrites are correctly reported as native).

[andygrove]

Coverage report generated by CometExpressionCoverageSuite against the Spark 4.1 profile:

Comet expression coverage report (Spark 4.1.2)

Probed 524 registered functions using their documented examples. Rerun under each Maven Spark profile (-Pspark-3.4, -Pspark-3.5, -Pspark-4.0, -Pspark-4.1) to compare coverage across versions.

Status	Count
NATIVE	330
INCOMPATIBLE	6
LITERAL_PROBE	4
GAP	42
SKIP	136
ERROR	6

Scalar gaps (Project fallback) - 38

Actionable candidates: scalar expressions Comet did not convert.

Function	Expression class	Fallback reason	Probed query
assert_true	org.apache.spark.sql.catalyst.expressions.AssertTrue	raise_error is not supported	SELECT assert_true(0 < 1) FROM comet_coverage_probe
bitmap_bit_position	org.apache.spark.sql.catalyst.expressions.BitmapBitPosition	Static invoke expression: bitmapBitPosition is not supported	SELECT bitmap_bit_position(1) FROM comet_coverage_probe
bitmap_bucket_number	org.apache.spark.sql.catalyst.expressions.BitmapBucketNumber	Static invoke expression: bitmapBucketNumber is not supported	SELECT bitmap_bucket_number(123) FROM comet_coverage_probe
bitmap_count	org.apache.spark.sql.catalyst.expressions.BitmapCount	Static invoke expression: bitmapCount is not supported	SELECT bitmap_count(X '1010') FROM comet_coverage_probe
e	org.apache.spark.sql.catalyst.expressions.EulerNumber	E is not supported	SELECT e() FROM comet_coverage_probe
encode	org.apache.spark.sql.catalyst.expressions.Encode	Static invoke expression: encode is not supported	SELECT encode('abc', 'utf-8') FROM comet_coverage_probe
input_file_block_length	org.apache.spark.sql.catalyst.expressions.InputFileBlockLength	Native DataFusion scan is not compatible with input_file_name, input_file_block_start, or input_file_block_length	SELECT input_file_block_length() FROM comet_coverage_probe
input_file_block_start	org.apache.spark.sql.catalyst.expressions.InputFileBlockStart	Native DataFusion scan is not compatible with input_file_name, input_file_block_start, or input_file_block_length	SELECT input_file_block_start() FROM comet_coverage_probe
input_file_name	org.apache.spark.sql.catalyst.expressions.InputFileName	Native DataFusion scan is not compatible with input_file_name, input_file_block_start, or input_file_block_length	SELECT input_file_name() FROM comet_coverage_probe
is_valid_utf8	org.apache.spark.sql.catalyst.expressions.IsValidUTF8	invoke is not supported	SELECT is_valid_utf8('Spark') FROM comet_coverage_probe
is_variant_null	org.apache.spark.sql.catalyst.expressions.variant.IsVariantNull	Static invoke expression: isVariantNull is not supported	SELECT is_variant_null(parse_json('null')) FROM comet_coverage_probe
java_method	org.apache.spark.sql.catalyst.expressions.CallMethodViaReflection	java_method is not supported	SELECT java_method('java.util.UUID', 'randomUUID') FROM comet_coverage_probe
make_interval	org.apache.spark.sql.catalyst.expressions.MakeInterval	make_interval is not supported	SELECT make_interval(100, 11, 1, 1, 12, 30, 01.001001) FROM comet_coverage_probe
make_valid_utf8	org.apache.spark.sql.catalyst.expressions.MakeValidUTF8	invoke is not supported	SELECT make_valid_utf8('Spark') FROM comet_coverage_probe
parse_json	org.apache.spark.sql.catalyst.expressions.variant.ParseJsonExpressionBuilder	Static invoke expression: parseJson is not supported	SELECT parse_json('{"a":1,"b":0.8}') FROM comet_coverage_probe
quote	org.apache.spark.sql.catalyst.expressions.Quote	Static invoke expression: quote is not supported	SELECT quote('Don\'t') FROM comet_coverage_probe
raise_error	org.apache.spark.sql.catalyst.expressions.RaiseErrorExpressionBuilder	raise_error is not supported	SELECT raise_error('custom error message') FROM comet_coverage_probe
randstr	org.apache.spark.sql.catalyst.expressions.RandStr	randstr is not supported	SELECT randstr(3, 0) AS result FROM comet_coverage_probe
reflect	org.apache.spark.sql.catalyst.expressions.CallMethodViaReflection	reflect is not supported	SELECT reflect('java.util.UUID', 'randomUUID') FROM comet_coverage_probe
schema_of_variant	org.apache.spark.sql.catalyst.expressions.variant.SchemaOfVariant	Static invoke expression: schemaOfVariant is not supported	SELECT schema_of_variant(parse_json('null')) FROM comet_coverage_probe
sentences	org.apache.spark.sql.catalyst.expressions.Sentences	Static invoke expression: getSentences is not supported	SELECT sentences('Hi there! Good morning.') FROM comet_coverage_probe
st_asbinary	org.apache.spark.sql.catalyst.expressions.st.ST_AsBinary	Static invoke expression: stAsBinary is not supported	SELECT hex(st_asbinary(st_geogfromwkb(X'0101000000000000000000F03F0000000000000040'))) FROM comet_coverage_probe
st_geogfromwkb	org.apache.spark.sql.catalyst.expressions.st.ST_GeogFromWKB	Static invoke expression: stAsBinary is not supported	SELECT hex(st_asbinary(st_geogfromwkb(X'0101000000000000000000F03F0000000000000040'))) FROM comet_coverage_probe
st_geomfromwkb	org.apache.spark.sql.catalyst.expressions.st.ST_GeomFromWKB	Static invoke expression: stAsBinary is not supported	SELECT hex(st_asbinary(st_geomfromwkb(X'0101000000000000000000F03F0000000000000040'))) FROM comet_coverage_probe
st_setsrid	org.apache.spark.sql.catalyst.expressions.st.ST_SetSrid	Static invoke expression: stSrid is not supported	SELECT st_srid(st_setsrid(ST_GeogFromWKB(X'0101000000000000000000F03F0000000000000040'), 4326)) FROM comet_coverage_probe
st_srid	org.apache.spark.sql.catalyst.expressions.st.ST_Srid	Static invoke expression: stSrid is not supported	SELECT st_srid(st_geogfromwkb(X'0101000000000000000000F03F0000000000000040')) FROM comet_coverage_probe
to_binary	org.apache.spark.sql.catalyst.expressions.ToBinary	Static invoke expression: encode is not supported	SELECT to_binary('abc', 'utf-8') FROM comet_coverage_probe
to_variant_object	org.apache.spark.sql.catalyst.expressions.variant.ToVariantObject	to_variant_object is not supported	SELECT to_variant_object(named_struct('a', 1, 'b', 2)) FROM comet_coverage_probe
try_make_interval	org.apache.spark.sql.catalyst.expressions.TryMakeInterval	make_interval is not supported	SELECT try_make_interval(100, 11, 1, 1, 12, 30, 01.001001) FROM comet_coverage_probe
try_parse_json	org.apache.spark.sql.catalyst.expressions.variant.TryParseJsonExpressionBuilder	Static invoke expression: parseJson is not supported	SELECT try_parse_json('{"a":1,"b":0.8}') FROM comet_coverage_probe
try_reflect	org.apache.spark.sql.catalyst.expressions.TryReflect	reflect is not supported	SELECT try_reflect('java.util.UUID', 'randomUUID') FROM comet_coverage_probe
try_validate_utf8	org.apache.spark.sql.catalyst.expressions.TryValidateUTF8	Static invoke expression: tryValidateUTF8String is not supported	SELECT try_validate_utf8('Spark') FROM comet_coverage_probe
try_variant_get	org.apache.spark.sql.catalyst.expressions.variant.TryVariantGetExpressionBuilder	try_variant_get is not supported	SELECT try_variant_get(parse_json('{"a": 1}'), '$.a', 'int') FROM comet_coverage_probe
typeof	org.apache.spark.sql.catalyst.expressions.TypeOf	typeof is not supported	SELECT typeof(1) FROM comet_coverage_probe
uuid	org.apache.spark.sql.catalyst.expressions.Uuid	uuid is not supported	SELECT uuid() FROM comet_coverage_probe
validate_utf8	org.apache.spark.sql.catalyst.expressions.ValidateUTF8	Static invoke expression: validateUTF8String is not supported	SELECT validate_utf8('Spark') FROM comet_coverage_probe
variant_get	org.apache.spark.sql.catalyst.expressions.variant.VariantGetExpressionBuilder	variant_get is not supported	SELECT variant_get(parse_json('{"a": 1}'), '$.a', 'int') FROM comet_coverage_probe
version	org.apache.spark.sql.catalyst.expressions.SparkVersion	Static invoke expression: getSparkVersion is not supported	SELECT version() FROM comet_coverage_probe

Non-scalar gaps (Generate / Aggregate / Window / etc.) - 4

Function	Expression class	Operator
inline	org.apache.spark.sql.catalyst.expressions.InlineExpressionBuilder	Generate
inline_outer	org.apache.spark.sql.catalyst.expressions.InlineExpressionBuilder	Generate
json_tuple	org.apache.spark.sql.catalyst.expressions.JsonTuple	Generate
stack	org.apache.spark.sql.catalyst.expressions.Stack	Generate