[SPARK-58199][SQL] Add FileIndex fullyPushedFilter hint for physical plan

[szehon-ho]

What changes were proposed in this pull request?

This PR adds an optional fullyPushedFilters hint to the FileIndex trait and
teaches FileSourceStrategy to use it when deciding which filters can be removed
from the row-level FilterExec placed above the scan.

• FileIndex.fullyPushedFilters: Option[Seq[Expression]] defaults to None.
• In FileSourceStrategy, the set of post-scan (after-scan) filters is now computed as:
  • When the FileIndex returns Some(fullyPushedFilters), those filters are the
    only basis for removing predicates from the FilterExec above the scan
    (filterSet -- fullyPushedFilters).
  • When it returns None, planning keeps the existing default behavior
    (filterSet -- partitionKeyFilters.filter(_.references.nonEmpty)).

This lets a FileIndex implementation that guarantees certain predicates for all
returned rows (e.g. an index that fully prunes on non-partition columns) tell the
planner exactly which filters are safe to drop, instead of relying only on the
default partition-key heuristic.

Note: in DSV2 , the scalar subquery partition filters are kept on FIlterExec for correctness, but they are removed in DSV1 world.

Why are the changes needed?

Today FileSourceStrategy assumes only partition-key filters are fully satisfied by
file listing, so any other predicate a FileIndex may fully guarantee must still be
re-evaluated in a FilterExec. Custom FileIndex implementations have no way to
communicate stronger pushdown guarantees to the physical plan. The new hint provides
an opt-in mechanism, with no behavior change for existing indexes (default None).

Does this PR introduce any user-facing change?

No. The new trait method has a default implementation returning None, which
preserves the existing planning behavior for all built-in FileIndex implementations.

How was this patch tested?

Added a unit test in FileSourceStrategySuite
("FileIndex fullyPushedFilters controls final FilterExec removal") that exercises a
test FileIndex returning None, Some(Nil), and Some(<data filter>), and asserts
the resulting FilterExec predicates. Ran:

build/sbt 'sql/testOnly *FileSourceStrategySuite -- -z "fullyPushedFilters"'

All tests passed.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 4.8)

[yyanyy]

nit/nonblocker - I wonder if we can always set partition key filters in index when we see it the first time to avoid the fallback; is that possible?

[szehon-ho]

yes, let me address with the other concern (that we dont call listFiles yet until physical execution..)