Skip to content

Arrow reader cross-thread close can race idle SQLCancel and surface ProgrammingError #890

Closed

Description

@Theekshna

Summary

tests/test_004_cursor_arrow.py::test_arrow_reader_cancel_from_other_thread still fails after the HY008 handling added in microsoft/mssql-python#857, which closed microsoft/mssql-python#856.

The evidence points to another close/cancel interleaving: the test sleeps for 50 ms but does not establish that an ODBC fetch is active when reader.close() calls SQLCancel. When cancellation lands between calls, idle SQLCancel closes the cursor rather than being a harmless no-op. A subsequent metadata request can then raise ProgrammingError, which the test does not accept.

This should be investigated as a reader-close sequencing/test synchronization issue, not addressed by broadly swallowing ProgrammingError.

Observed CI failure

  • Pipeline: mssql-rs build 191239.
  • Job: mssql-python suite on mssql-odbc driver (cross-repo).
  • Pinned mssql-python revision: bd0aaf89c724a92c934b68bca41469655bfc733c (already includes the previous deflake fix).
  • Test: tests/test_004_cursor_arrow.py::test_arrow_reader_cancel_from_other_thread.
  • Exception: mssql_python.exceptions.ProgrammingError, SQLSTATE HY010, Function sequence error.
  • Stack: _ArrowReader.__next__ → PyArrow read_next_batch → batch_generator → arrow_batch → DDBCSQLFetchArrowBatch → helpers.check_error.
  • Arrow test file: 1 failed, 49 passed, 1 skipped. Overall harness: 70/71 files passed, no crashes or timeouts.
  • The job explicitly runs with -m "not slow", so this stress-marked test is included.

Source evidence

At the pinned revision:

  • _ArrowReader.__next__ checks _closed before entering the inner reader; that check does not synchronize the full batch operation with concurrent close.
  • _ArrowReader.close marks the reader closed and calls _cancel(). Its comment describes idle SQLCancel as a safe no-op, which does not match the measured drivers. Closing an already-executing generator may leave it alive for later cleanup.
  • batch_generator has no close-request check before fetching another batch.
  • FetchArrowBatch_wrap requests result metadata before fetching. SQLNumResultCols_wrap propagates errors through CheckFetchError.

These paths expose a close-versus-batch window; the failing CI log does not contain a per-ODBC-call trace proving the exact scheduling of that particular run.

Controlled native reproduction

Using live local SQL Server on Windows, direct driver entry points, valid retained handles/buffers, and the same million-row query with a 64-row bound rowset:

  1. Execute select top 1000000 1 a from sys.objects o1, sys.objects o2, sys.objects o3.
  2. Fetch one 64-row batch.
  3. Call SQLCancel from a second thread while no ODBC operation is active; wait for it to return.
  4. Call SQLNumResultCols on the same statement.
  5. Close/reset and execute select 5 to verify cursor reuse.
Observation Rust mssql-odbc Microsoft ODBC Driver 18
Idle SQLCancel SQL_SUCCESS_WITH_INFO, 01S05 (Cancel treated as FreeStmt/Close) Same
Following SQLNumResultCols SQL_ERROR, HY010, 20/20 runs SQL_ERROR, HY007, 20/20 runs
Cursor reusable afterward Yes Yes

The Microsoft reference was retail 18.7.1.1, reporting SQL_DRIVER_VER=18.07.0001. The Rust driver was built from source matching the failing build's cancellation, close, fetch, operation-completion, statement-state, and result-column-count implementations.

Both HY007 and HY010 map to ProgrammingError in mssql-python. Consequently, merely changing the Rust diagnostic to HY007 would not make the test accept this ordering. The different SQLSTATE is a separate observed driver-parity discrepancy.

Separate active-execution and active-fetch cancellation probes produced HY008 diagnostics on both drivers and left the cursor reusable. This does not claim that every cancellation return code is identical between drivers.

Validation limit: this is a controlled native reproduction on Windows, not a replay of the full Linux/PyArrow stress test. The complete CI thread interleaving remains unproven.

Expected follow-up

Coordinate reader close with the complete batch operation so a close request cannot lead to another metadata/fetch sequence on an already-closed cursor. Make the test establish the intended in-flight cancellation point rather than relying on a 50 ms sleep, and cover the between-batch close case separately.

Retain the existing postconditions: the closer finishes, no unexpected close exception occurs, the reader is closed, and the parent cursor remains reusable.

Activity

  1. github-actions commented on Nov 12, 2026

    @github-actions

    Hi ttk (@Theekshna), thank you for opening this issue!

    Our team will review it shortly. We aim to triage all new issues within 24-48 hours and get back to you.

    If you have additional information to share, please feel free to update the issue.

    Thank you for your patience!

  2. Theekshna commented on Nov 12, 2026

    @Theekshna
    ContributorAuthor

    Confirmed with the actual, unchanged Python test (not just a forced native call sequence).

    Environment: Windows x64, local SQL Server 2022 over TCP, Python 3.13.17, PyArrow 26.0.1, pytest 8.3.4. Python source and native bindings were built from bd0aaf8. The test file, 50 ms sleep, assertions, and Python reader implementation were not modified. The same native bindings and package-loading path were used for both drivers, swapping only the packaged driver binary as the mssql-rs job does.

    Command, once per fresh process:

    python -m pytest tests\test_004_cursor_arrow.py::test_arrow_reader_cancel_from_other_thread -m stress -q --tb=short -p no:azurepipelines

    Results without tracing or native instrumentation

    Driver Passed Failed Failure
    Bundled msodbcsql 18.6.2.1 (SQL_DRIVER_VER=18.06.0002) 26 4 3x HY007 / associated statement not prepared; 1x 24000 / invalid cursor state
    Rust mssql-odbc, optimized --release build from the failing pipeline's source 29 1 HY010 / function sequence error, matching CI

    These are observed counts from one local workload, not stable failure-rate estimates.

    Actual test failures captured with native ODBC call tracing

    For a separate run, a session-local build of the bindings recorded native entry/exit timestamps, thread IDs, return codes, and SQLSTATEs returned by the existing diagnostic calls. Records were buffered in memory and written only at process exit; the observer added no sleeps or ODBC calls. No test or Python reader changes were made. Tracing can still perturb scheduling, so its counts are separate from the uninstrumented baseline.

    • Classic: 5 failures / 30 runs (4x HY007, 1x 24000).
    • Rust optimized build: 1 failure / 180 runs (HY010).
    • The selected failing traces have zero dropped records.

    Classic HY007 trace: the preceding SQLFetch has completed. Thread B enters SQLCancel between batches; thread A enters SQLNumResultCols 0.0204 ms later. SQLCancel returns SQL_SUCCESS_WITH_INFO after 5.2309 ms. SQLNumResultCols then returns SQL_ERROR, and the normal diagnostic read reports HY007. This is the idle-cancel-as-close ordering discussed in this issue.

    Rust HY010 trace: the preceding SQLFetch has completed. Thread B enters SQLCancel; thread A enters SQLNumResultCols 0.0298 ms later and receives SQL_ERROR, then HY010 from its diagnostic read. SQLCancel later returns SQL_SUCCESS_WITH_INFO. This reproduces the CI exception with the actual reader/test, not an artificial post-cancel metadata call.

    Second classic failure path: cancellation lands during SQLFetch. That fetch returns SQL_SUCCESS_WITH_INFO and its diagnostic contains HY008. The native Arrow wrapper continues with another SQLFetch, which returns SQL_ERROR / 24000. The test's HY008 OperationalError handler does not cover this subsequent ProgrammingError.

    Conclusion and limits

    The test/reader close race demonstrably occurs with both drivers. It is not evidence that idle SQLCancel must be changed to a no-op, or that all ProgrammingError exceptions should be ignored. Reader shutdown must account for the complete batch operation and partial-success cancellation, and tests should cover both in-flight and between-batch close.

    The Rust HY010 versus classic HY007 diagnostic difference remains a separate parity issue tracked by https://sqlclientdrivers.visualstudio.com/mssql-rs/_workitems/edit/50265. Fixing that SQLSTATE alone cannot resolve the test because both map to ProgrammingError.

    This reproduction is Windows, not the original Linux/Python 3.12 CI environment; the original failed CI run itself still has no per-call trace. However, the exact unchanged test now fails on both drivers locally, and a matching Rust HY010 failure has been traced.

  3. added
    bugSomething isn't working
    triage doneIssues that are triaged by dev team and are in investigation.
    and removed
    triage neededFor new issues, not triaged yet.
    on Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

bugSomething isn't workinginADOtriage doneIssues that are triaged by dev team and are in investigation.under development

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions