[dborowitz]

Summary

Modernizes Python and PySpark version handling across packaging metadata, runtime session checks, unit tests, and CI:

1. Declares Requires-Python: >=3.10 and tests non-default Python versions (setup.py, README.md, pyproject.toml, .github/workflows/tests.yaml, .github/workflows/vermin.yaml)

   • No prior release declared Requires-Python in setup.py (and wheels before 2.1.1 were tagged py2.py3-none-any). Because session.py uses match/case syntax (Python 3.10+), pip on Python 3.9 (where pyspark 4.0 is still installable) would install the wheel and fail at import time with SyntaxError.
   • Sets python_requires=">=3.10" and adds Python 3 / 3.10–3.14 classifiers in setup.py, and documents supported Python versions (and the UDF Python minor-version matching requirement) in README.md.
   • Keeps the full unit test suite on Python 3.12 (the default runtime's Python) across Linux, macOS, and Windows for all extras, and adds Linux [client] smoke rows for Python 3.10, 3.11, 3.13, and 3.14 running pytest tests/unit -m "not slow" -n auto (which imports every module under google/).
   • Adds a slow pytest marker for the 39 unit tests that take ≥1s each (mostly sleeping): the nonzero backend_wait/client_wait params of test_proxy.py's timing tests (the zero-wait params still cover every socket-timeout combination), plus 3 slow tests in test_session.py and test_pypi_artifacts.py. Per-test coverage contexts show these take ~80% of serial suite time while covering only 6 lines nothing else covers, so the smoke subset reaches 99.8% of the lines the full suite covers in ~15–20s under -n 4 (vs. ~50–60s for the full suite). This catches standard library APIs missing from older Pythons (e.g. datetime.UTC, itertools.batched) or removed in newer ones, not just syntax.
   • Adds a Vermin workflow running vermin --target=3.10- over google, tests, and setup.py, which statically catches newer-only language features and stdlib modules/functions/kwargs even in lines no test runs.

2. Centralizes runtime version metadata and warns when PySpark is newer than the runtime's Spark (_runtimes.py, session.py, tests/unit/test_runtimes.py, tests/unit/test_init.py)

   • Spark Connect servers support clients with an older Spark minor version, but not a newer one. Verified against live sessions with a 28-operation probe (DataFrame/SQL ops, toPandas/toArrow, Python/pandas UDFs, mapInPandas/applyInPandas, UDTFs, caching, catalog, tables, streaming, interruptAll):
     • pyspark-client 4.0.3 → a Spark 4.2 server: everything that works with a matched client works.
     • pyspark-client 4.2.0 → Spark 4.0 (runtime 3.0): createDataFrame() from local data (rows, explicit schema, or pandas) fails with SQL_CONF_NOT_FOUND for spark.sql.session.localRelationSizeLimit, a config Spark 4.0 does not have.
   • Adds google/cloud/managed_spark_connect/_runtimes.py (RuntimeInfo, _RUNTIMES = {(3, 0): RuntimeInfo(spark=(4, 0), python=(3, 12))}) indexed by (major, minor) so subminor runtime strings such as "3.0.12" are checked as well, while unknown runtime versions are skipped gracefully.
   • Uses _runtimes.get_runtime_info() in both _check_python_version_compatibility() and a new _check_pyspark_version_compatibility() check that warns (never raises) only when the installed pyspark minor version is newer than the target runtime's Spark version (recommending pyspark-client~=X.Y.0 or pyspark[connect]~=X.Y.0 based on is_remote_only()).
   • Adds a unit test in tests/unit/test_runtimes.py keeping _DEFAULT_RUNTIME_VERSION and setup.py's [client]/[spark] pyspark pins in sync.

3. Fixes unit tests under newer pyspark-client and tests them in CI (tests/unit/test_instrumentation.py, tests/unit/test_init.py, .github/workflows/tests.yaml)

   • While [client] and [spark] pin ~=4.0.0 to match the default 3.0 runtime, newer runtimes and preinstalled environments may have a newer pyspark-client (such as 4.2.x).
   • Implements _set_relation_in_plan() on FakeClient in tests/unit/test_instrumentation.py (invoked by LogicalPlan.to_proto() in pyspark-client 4.2+).
   • Passes subprocess probe scripts via stdin (python -) rather than -c in tests/unit/test_init.py to avoid pyspark 4.2's check_dependencies(), which treats a __main__ module without __file__ as a pyspark doctest run and exits when SPARK_HOME is unset.
   • Adds a test-newest-pyspark CI job in .github/workflows/tests.yaml running the full unit test suite on Ubuntu + Python 3.13 against the latest pyspark-client (serially, as pytest-xdist workers also bootstrap without __main__.__file__).

Commits in this PR

1. build: declare Requires-Python >=3.10 and smoke test other Pythons
2. feat: warn when installed PySpark is newer than the runtime's Spark
3. test: fix unit tests under newer pyspark-client and run them in CI

Testing

• Verified built wheel metadata (Tag: py3-none-any, Requires-Python: >=3.10) and confirmed uv pip install rejects Python 3.9 and succeeds on Python 3.10–3.14.
• Ran the -m "not slow" smoke subset (535 passed) on Python 3.10, 3.11, 3.12, 3.13, and 3.14.
• Ran vermin --target=3.10- (passes; verified it flags datetime.UTC, asyncio.timeout, itertools.batched, shutil.rmtree(onexc=), typing.Self, enum.StrEnum, and Path.walk in a canary file).
• Ran the full unit test suite (587 passed) against both pyspark-client==4.0.3 (Python 3.12) and pyspark-client==4.2.0 (Python 3.13).

[gemini-code-assist]

Code Review

This pull request introduces PySpark version compatibility checks and updates Python version requirements to 3.10 or later. It adds a new _runtimes module to map Managed Spark runtime versions to their respective Spark and Python versions, integrates PySpark version validation into the session builder, and updates the documentation, setup configuration, and unit tests accordingly. The review feedback suggests enhancing robustness by adding a defensive type check for the version string in major_minor and wrapping the import of is_remote_only in a try-except block to gracefully handle older or custom PySpark installations.