Skip to content

FEAT: Add native FLOAT32 and FLOAT16 vector support - #902

Open
gargsaumya wants to merge 11 commits into
mainfrom
saumya/vector-support
Open

gargsaumya wants to merge 11 commits into
mainfrom
saumya/vector-support

Conversation

@gargsaumya

@gargsaumya gargsaumya commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Work Item / Issue Reference

AB#50380


Summary

Add opt-in synchronous native FLOAT32/FLOAT16 vector support.

  • Introduce immutable Vector(values, base_type="float32" | "float16"), with .base_type, .dimensions, and normal sequence operations including list(v).
  • Normalize elements to float32 at construction. Reject non-numeric elements, NaN/infinity, float32 overflow, and conservatively out-of-range FLOAT16 inputs before native binding. Explicitly hinted numeric lists receive the same validation during binding.
  • Support native scalar and batch binding, typed/all-NULL arrays, explicitly hinted lists, parameter styles, streamed companion text, row fetch APIs, metadata, output converters, transactions, and cursor reuse.
  • Implement buffered callproc() (1-3 part names, optionally bracket-quoted) with OutputParameter(sql_type, size, scale) / InputOutputParameter(vector) wrappers, multiple results and repeated NULL inputs. Output parameters take the setinputsizes() triple; only SQL_SS_VECTOR is supported today and other types raise NotSupportedError.
  • Preserve existing unhinted-list and default connection behavior. Pass through VectorTypeSupport=off/v1/v2; both vector base types use float32 application buffers, while ODBC owns FLOAT16 conversion and native serialization.
  • Preserve the latest parameter-binding reuse and native-fetch behavior. Buffered procedure rows stay on the Python fetch path.
  • Add documentation, native type declarations, public consumer typing checks, and unit/live SQL coverage. Gate integration tests using server type metadata and the qualified provider, without suppressing connection/query errors on supported configurations.

API example: native reads and writes

connection_string is supplied externally. This FLOAT16 example requires ODBC 18.7.1.1+ and server support; FLOAT32 uses base_type="float32" and a matching SQL column.

from mssql_python import Vector, connect

with connect(connection_string, VectorTypeSupport="v2") as connection:
    with connection.cursor() as cursor:
        cursor.execute("CREATE TABLE #VectorData (value VECTOR(3, FLOAT16) NULL)")
        v = Vector([1.0, 2.0, 3.0], base_type="float16")
        cursor.execute("INSERT INTO #VectorData VALUES (?)", v)
        cursor.execute("SELECT value FROM #VectorData")
        result = cursor.fetchone()[0]  # Vector; SQL NULL returns None
        values = list(result)

The same values work with executemany() and procedure parameter wrappers, e.g. OutputParameter(SQL_SS_VECTOR, 3, 1). Vector bulk copy and Arrow are covered by the follow-up PRs, not this PR.

Validation

Validated on Windows x64 / Python 3.13 with a rebuilt native extension and an isolated source-built Microsoft ODBC 18.7.1.1 provider:

Coverage Result
Vector unit tests, including numeric boundaries and CI capability gates 144 passed
Azure SQL vector E2E: v1/FLOAT32, v2/FLOAT32, v2/FLOAT16 194 passed, zero skipped
Existing Azure input-size, prepared-statement reuse and native-fetch regressions 16 passed
Existing executemany/setinputsizes/decimal/time cursor tests (local SQL Server 2022) 162 passed, 3 pre-existing skips
Strict source/stub typing and consumer API tests 6 passed
  • Native Release build, full repository-pinned Black check, and whitespace checks passed.
  • Reran all three vector modes through the final server metadata gate successfully.
  • Provisioned and tested through Azure CLI, then deleted the isolated SQL server/databases and temporary credentials. Existing resources were untouched.
  • This is targeted Windows/Azure validation, not a claim that the full suite or cross-platform/provider matrix has completed.

Scope and follow-up

  • Vector bulk copy and Arrow vector support are intentionally deferred to the next PR. Existing unsupported boundaries raise explicit errors.
  • New async vector integration and alternate Rust ODBC provider qualification are not included.
  • Table-valued parameters are intentionally not part of this PR; vector TVP columns will build on the shared TableValuedParameter definition from FEAT: Add table-valued parameter support #789.
  • Procedure output parameters currently support SQL_SS_VECTOR only; procedure result sets are buffered in memory so output values are final when callproc() returns.
  • FLOAT32 requires Microsoft ODBC 18.6.1.1+; FLOAT16/v2 requires 18.7.1.1+ and server support. This PR does not replace bundled binaries.

gargsaumya and others added 2 commits November 17, 2026 11:01
Add immutable base_type-aware vectors, hinted list binding, TVPs and buffered procedure outputs. Include numeric validation, documentation and unit/live SQL coverage. Vector bulk copy and Arrow remain explicit follow-up work.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use server type metadata and the qualified provider to skip unsupported CI legs without hiding connection failures. Cover both gate outcomes and cleanup.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 17, 2026 05:43
@github-actions github-actions Bot added the pr-size: large Substantial code update label Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

PR Performance Report

✅ No regression detected

No consistent slowdowns detected across all 2 environments.

0 IMPROVEMENTS 0 SLOWDOWNS 2/2 ENVIRONMENTS

Coverage: 2 of 2 environments completed. Advisory result; does not block merging.

Performance diagnostics

Phase times are inclusive diagnostics and must not be added together. They identify where measured time changed, not why it changed.

No affected phases or call-count changes were recorded.

All database tasks and timings

Unix / SQL Server 2022

Database task Before After Paired change Result
Connection opening 10.535 ms 10.352 ms +0.3% no signal
SELECT queries 1.118 ms 0.977 ms -12.6% no signal
Row insertion 30.880 ms 31.258 ms +1.8% no signal
Executemany inserts 163.097 ms 162.120 ms -0.4% no signal
Fetch-all queries 119.425 ms 121.605 ms +0.3% no signal
Row-by-row fetching 4.945 ms 4.978 ms +1.0% no signal
Batched row fetching 125.211 ms 133.870 ms +10.2% no signal
Transaction commit and rollback 106.279 ms 106.261 ms -1.1% no signal
Arrow row fetching 88.502 ms 89.964 ms +0.9% no signal
100,000-row insertion 354.996 ms 386.201 ms +12.1% no signal
Row fetching in batches of 100 120.409 ms 120.493 ms -0.6% no signal
Row fetching in batches of 10,000 127.446 ms 134.597 ms +5.7% no signal
Repeated positional queries 32.872 ms 33.274 ms +1.4% no signal
Repeated named-parameter queries 33.068 ms 32.965 ms +0.7% no signal
Legacy 100,000-row insertion 265.589 ms 301.803 ms +12.3% no signal
Insertion with explicit input sizes 384.331 ms 384.612 ms +0.4% no signal
Joined aggregation queries 178.014 ms 177.398 ms -0.3% no signal
Large joined-result fetching 190.510 ms 188.408 ms +1.3% no signal
1.2-million-row fetching 3489.262 ms 3438.058 ms -1.4% no signal
Common table expression queries 5.103 ms 5.108 ms +0.1% no signal
256 KiB VARCHAR(MAX) / fetchall() 1.559 ms 1.559 ms -1.4% no signal
10,000 scalar values / fetchval() (debug disabled) 16.883 ms 16.813 ms +0.6% no signal
10,000 output-converted rows / fetchall() 19.478 ms 19.306 ms -0.5% no signal
10,000 output-converted rows / fetchmany(1000) 23.327 ms 23.314 ms +0.4% no signal
10,000 INT/NVARCHAR rows / fetchmany(1) 24.928 ms 24.444 ms -1.7% no signal
10,000 rows x 32 columns / dict(row._mapping) (warm) 32.283 ms 31.295 ms -1.1% no signal

Unix / SQL Server 2025

Database task Before After Paired change Result
Connection opening 81.649 ms 82.064 ms +0.7% no signal
SELECT queries 1.187 ms 1.046 ms -11.9% no signal
Row insertion 31.037 ms 31.688 ms +2.1% no signal
Executemany inserts 161.230 ms 160.963 ms -1.6% no signal
Fetch-all queries 121.293 ms 122.198 ms +2.2% no signal
Row-by-row fetching 4.998 ms 5.106 ms +1.4% no signal
Batched row fetching 118.082 ms 119.589 ms +1.1% no signal
Transaction commit and rollback 108.770 ms 108.452 ms -0.3% no signal
Arrow row fetching 91.736 ms 91.125 ms -0.5% no signal
100,000-row insertion 359.083 ms 386.421 ms +8.3% no signal
Row fetching in batches of 100 125.764 ms 122.830 ms -2.9% no signal
Row fetching in batches of 10,000 141.385 ms 150.984 ms +8.7% no signal
Repeated positional queries 31.855 ms 32.929 ms +2.7% no signal
Repeated named-parameter queries 32.332 ms 33.492 ms +3.4% no signal
Legacy 100,000-row insertion 266.121 ms 298.038 ms +14.1% no signal
Insertion with explicit input sizes 392.649 ms 410.103 ms +3.5% no signal
Joined aggregation queries 167.337 ms 187.297 ms +5.9% no signal
Large joined-result fetching 192.829 ms 214.508 ms +13.7% no signal
1.2-million-row fetching 3563.669 ms 3471.660 ms -1.2% no signal
Common table expression queries 5.182 ms 5.349 ms +3.2% no signal
256 KiB VARCHAR(MAX) / fetchall() 1.955 ms 1.960 ms +9.7% no signal
10,000 scalar values / fetchval() (debug disabled) 16.963 ms 16.976 ms -0.8% no signal
10,000 output-converted rows / fetchall() 19.468 ms 20.229 ms +3.7% no signal
10,000 output-converted rows / fetchmany(1000) 23.884 ms 23.623 ms +0.6% no signal
10,000 INT/NVARCHAR rows / fetchmany(1) 24.610 ms 25.094 ms +2.1% no signal
10,000 rows x 32 columns / dict(row._mapping) (warm) 33.025 ms 32.219 ms -2.6% no signal
Build and measurement details

ADO build 193304

PR head: 97939010a285f03eef89cc9f5c0b390eb2810027
Base: d257fdd86dc0c8f09462981cde0cf76e38b7d0d7
Measured merge: bbafc6df5a67211252c69acd04b323badca2b23a

  • Unix / SQL Server 2022: Python 3.12.3, x86_64, SQL 16.0.4325.4; 5 paired comparisons and 1 warmup.
  • Unix / SQL Server 2025: Python 3.12.3, x86_64, SQL 17.0.5065.9; 5 paired comparisons and 1 warmup.

A consistent change requires more than 20% median paired movement, at least 1 ms between the median runtimes, and at least 80% of pairs exceeding the relative threshold in the same direction. A slowdown without enough pair agreement is reported as inconsistent.

The displayed change is the median of paired before-and-after ratios. It is not recalculated from the two displayed median runtimes.

Both revisions use profiling-enabled builds on the same agent and database, with alternating order and discarded warmups. Results are diagnostic and do not represent production-wheel latency.

Raw samples and logs are attached to the ADO run as profiler-* artifacts.

Comment thread tests/test_025_vector.py Dismissed
Comment thread tests/test_025_vector.py Dismissed
Comment thread tests/test_025_vector.py Dismissed
Comment thread tests/test_025_vector.py Dismissed
Comment thread tests/test_025_vector.py Dismissed
Comment thread tests/test_025_vector.py Dismissed
Comment thread tests/test_025_vector.py Dismissed

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A failing callproc() can leave stale cache state, causing subsequent execution of the previous SQL to skip preparation even though the handle is prepared for the procedure.

1 open issue
What changed in this PR

Adds opt-in FLOAT32/FLOAT16 vector support to the synchronous Python API and native ODBC bindings, extending parameter binding, row fetching, and stored-procedure support.

Changes:

  • Introduces immutable Vector values with numeric validation, explicit list hints, and connection-level vector negotiation.
  • Adds scalar and batch binding, typed table-valued parameters, and buffered callproc() with vector output wrappers.
  • Adds documentation, type declarations, and unit/live SQL coverage. Vector bulk copy, Arrow fetches, and async integration remain out of scope.
File Description
tests/​test_typing.py Checks public vector API call shapes.
tests/​test_025_vector.py Adds vector unit contracts and live SQL coverage.
tests/​test_004_cursor.py Replaces the unsupported-callproc test with name validation.
README.md Documents vector APIs, requirements, limitations, and testing.
mssql_python/​vector.py Defines validated vectors and procedure output wrappers.
mssql_python/​table_parameter.py Defines explicit table schemas and row snapshots.
mssql_python/​pybind/​vector_support.hpp Implements native vector buffers and conversion.
mssql_python/​pybind/​py_type_cache.hpp Caches vector and table-parameter classes.
mssql_python/​pybind/​param_detect.hpp Detects vector/table parameters and applies vector hints.
mssql_python/​pybind/​ddbc_bindings.h Adds vector and table-parameter ODBC constants.
mssql_python/​pybind/​ddbc_bindings.cpp Implements binding, vector fetching, and buffered procedure execution.
mssql_python/​mssql_python.pyi Exposes new public types and the vector constant.
mssql_python/​ddbc_bindings.py Adds typing imports for native vector helpers.
mssql_python/​cursor.py Integrates vector batches, buffered procedures, and unsupported-operation guards.
mssql_python/​constants.py Registers vector types and the negotiation keyword.
mssql_python/​connection_string_parser.py Validates and normalizes VectorTypeSupport.
mssql_python/​_native_fetch.py Keeps buffered procedure results on Python fetch paths.
mssql_python/​_ddbc_types.pyi Declares native vector helpers and procedure results.
mssql_python/​__init__.py Exports the new public API.
CHANGELOG.md Records vector support and its current limitations.

Review effort: Balanced

Comment thread mssql_python/cursor.py Resolved
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

📊 Code Coverage Report

🔥 Diff Coverage

90%


🎯 Overall Coverage

85%


📈 Total Lines Covered: 12034 out of 14134
📁 Project: mssql-python


Diff Coverage

Diff: main...HEAD, staged and unstaged changes

  • mssql_python/init.py (100%)
  • mssql_python/_native_fetch.py (100%)
  • mssql_python/connection_string_parser.py (100%)
  • mssql_python/constants.py (100%)
  • mssql_python/cursor.py (98.0%): Missing lines 2844,4689,4695
  • mssql_python/pybind/ddbc_bindings.cpp (84.5%): Missing lines 492-493,603-604,607-611,2464-2467,2477,2492-2494,3720-3722,3724-3726,6794-6796,6804-6805
  • mssql_python/pybind/ddbc_bindings.h (100%)
  • mssql_python/pybind/param_detect.hpp (89.7%): Missing lines 253-254,262
  • mssql_python/pybind/py_type_cache.hpp (100%)
  • mssql_python/pybind/vector_support.hpp (59.5%): Missing lines 29,31,52,56,65-73,76-79
  • mssql_python/vector.py (100%)

Summary

  • Total: 536 lines
  • Missing: 51 lines
  • Coverage: 90%

mssql_python/cursor.py

Lines 2840-2848

  2840                     if sql_type == ddbc_sql_const.SQL_SS_VECTOR.value:
  2841                         from mssql_python.vector import _check_shape
  2842 
  2843                         if decimal_digits not in (0, 1):
! 2844                             raise ValueError("Vector subtype must be 0 (FLOAT32) or 1 (FLOAT16)")
  2845                         _check_shape(column_size, "float16" if decimal_digits else "float32")
  2846                         paraminfo.columnSize = ddbc_bindings._vector_buffer_size(column_size)
  2847                     paraminfo.isDAE = is_dae

Lines 4685-4693

  4685         if not isinstance(procname, str) or not procname or "\0" in procname:
  4686             raise ValueError("procname must be a non-empty procedure name")
  4687         name = _quote_procedure_name(procname)
  4688         if isinstance(parameters, (str, bytes, bytearray)):
! 4689             raise TypeError("parameters must be a sequence of parameter values")
  4690         values = list(parameters) if parameters is not None else []
  4691         query = "{CALL " + name + "(" + ",".join("?" for _ in values) + ")}"
  4692         self._soft_reset_cursor()
  4693         self.messages = []

Lines 4691-4699

  4691         query = "{CALL " + name + "(" + ",".join("?" for _ in values) + ")}"
  4692         self._soft_reset_cursor()
  4693         self.messages = []
  4694         if self._cached_decoding_generation != self._connection._decoding_generation:
! 4695             self._refresh_decoding_cache()
  4696         self.last_executed_stmt = ""
  4697         self.is_stmt_prepared = [False]
  4698         outputs, results = ddbc_bindings.DDBCCallProc(
  4699             self.hstmt,

mssql_python/pybind/ddbc_bindings.cpp

Lines 488-497

  488         auto resolved = ResolveNullParamType(handle, hStmt, static_cast<int>(paramIndex));
  489         paramInfo.paramSQLType = resolved.sqlType;
  490         paramInfo.columnSize = resolved.columnSize;
  491         paramInfo.decimalDigits = resolved.decimalDigits;
! 492         if (resolved.sqlType == SQL_SS_VECTOR) {
! 493             if (resolved.columnSize < sizeof(SQL_SS_VECTOR_STRUCT) ||
  494                 (resolved.columnSize - sizeof(SQL_SS_VECTOR_STRUCT)) % sizeof(float) != 0)
  495                 throw py::value_error("ODBC returned invalid vector parameter metadata");
  496             size_t dimensions =
  497                 (resolved.columnSize - sizeof(SQL_SS_VECTOR_STRUCT)) / sizeof(float);

Lines 599-615

  599 
  600         switch (paramInfo.paramCType) {
  601             case SQL_C_SS_VECTOR: {
  602                 if (!paramInfo.vectorBinding) {
! 603                     if (param.is_none())
! 604                         throw py::value_error("NULL vector parameters require dimensions and subtype");
  605                     paramInfo.vectorBinding = std::make_shared<VectorBinding>(param);
  606                 }
! 607                 auto binding = paramInfo.vectorBinding;
! 608                 paramBuffers.push_back(binding);
! 609                 dataPtr = &binding->value;
! 610                 bufferLength = static_cast<SQLLEN>(VectorBufferSize(binding->elements.size()));
! 611                 strLenOrIndPtr = &binding->indicator;
  612                 break;
  613             }
  614             case SQL_C_CHAR: {
  615                 if (!py::isinstance<py::str>(param) && !py::isinstance<py::bytearray>(param) &&

Lines 2460-2471

  2460             SQLLEN bufferLength = 0;
  2461             switch (info.paramCType) {
  2462                 case SQL_C_SS_VECTOR: {
  2463                     size_t dimensions = 0;
! 2464                     int subtype = info.decimalDigits;
! 2465                     for (auto item : columnValues) {
! 2466                         if (!item.is_none()) {
! 2467                             dimensions = item.attr("dimensions").cast<size_t>();
  2468                             subtype = item.attr("base_type").cast<std::string>() == "float16" ? 1 : 0;
  2469                             break;
  2470                         }
  2471                     }

Lines 2473-2481

  2473                         if (info.columnSize < sizeof(SQL_SS_VECTOR_STRUCT))
  2474                             throw py::value_error("NULL vector arrays require vector metadata");
  2475                         dimensions = (info.columnSize - sizeof(SQL_SS_VECTOR_STRUCT)) / sizeof(float);
  2476                     }
! 2477                     ValidateVectorShape(dimensions, subtype);
  2478                     if (info.columnSize != VectorBufferSize(dimensions) ||
  2479                         info.decimalDigits != subtype)
  2480                         throw py::value_error("Vector array must match its declared dimensions and subtype");
  2481                     auto structs = AllocateParamBufferArray<SQL_SS_VECTOR_STRUCT>(

Lines 2488-2498

  2488                         if (binding->value.dimension != dimensions || binding->value.type != subtype)
  2489                             throw py::value_error("Vector array dimensions and subtype must agree");
  2490                         structs[i] = binding->value;
  2491                         strLenOrIndArray[i] = binding->indicator;
! 2492                         tempBuffers.push_back(binding);
! 2493                     }
! 2494                     dataPtr = structs;
  2495                     bufferLength = static_cast<SQLLEN>(VectorBufferSize(dimensions));
  2496                     info.columnSize = static_cast<SQLULEN>(bufferLength);
  2497                     info.decimalDigits = static_cast<SQLSMALLINT>(subtype);
  2498                     break;

Lines 3716-3730

  3716             VectorBinding binding(dimensions, static_cast<int>(subtype));
  3717             {
  3718                 py::gil_scoped_release release;
  3719                 ret = SQLGetData_ptr(hStmt, i, SQL_C_SS_VECTOR, &binding.value,
! 3720                     static_cast<SQLLEN>(VectorBufferSize(dimensions)), &binding.indicator);
! 3721             }
! 3722             CaptureFetchDiagnostics(hStmt, ret, messages);
  3723             if (!SQL_SUCCEEDED(ret)) return ret;
! 3724             row.append(binding.toPython());
! 3725             continue;
! 3726         }
  3727 
  3728         // Preprocess sql_variant: detect underlying type to route to correct conversion logic
  3729         SQLSMALLINT effectiveDataType = dataType;
  3730         if (dataType == SQL_SS_VARIANT) {

Lines 6790-6800

  6790         ~ParameterGuard() {
  6791             if (!active) return;
  6792             SQLRETURN rc = ResetParamBindings(handle, statement);
  6793             if (!SQL_SUCCEEDED(rc)) LOG_ERROR("Failed to reset procedure parameter bindings");
! 6794         }
! 6795     } reset{*statement, hStmt};
! 6796     std::string encoding = "utf-8";
  6797     if (encodingSettings.contains("encoding") && encodingSettings.contains("ctype") &&
  6798         encodingSettings["ctype"].cast<int>() == SQL_C_CHAR)
  6799         encoding = encodingSettings["encoding"].cast<std::string>();
  6800     rc = BindParameters(*statement, hStmt, values, infos, buffers, encoding);

Lines 6800-6809

  6800     rc = BindParameters(*statement, hStmt, values, infos, buffers, encoding);
  6801     CheckFetchError(statement, rc);
  6802     {
  6803         py::gil_scoped_release release;
! 6804         rc = SQLExecute_ptr(hStmt);
! 6805     }
  6806     CheckFetchError(statement, rc);
  6807     CaptureFetchDiagnostics(hStmt, rc, messages);
  6808     py::list results;
  6809     while (rc != SQL_NO_DATA) {

mssql_python/pybind/param_detect.hpp

Lines 249-258

  249         auto dimensions = static_cast<size_t>(info.columnSize);
  250         py::object vector = py::module_::import("mssql_python.vector")
  251             .attr("_vector_from_hint")(borrow(obj), dimensions, info.decimalDigits);
  252         if (PyList_SetItem(params, index, vector.release().ptr()) != 0) {
! 253             throw py::error_already_set();
! 254         }
  255         obj = PyList_GET_ITEM(params, index);
  256         info.paramCType = SQL_C_SS_VECTOR;
  257         info.vectorBinding = obj == Py_None
  258             ? std::make_shared<VectorBinding>(dimensions, info.decimalDigits)

Lines 258-266

  258             ? std::make_shared<VectorBinding>(dimensions, info.decimalDigits)
  259             : std::make_shared<VectorBinding>(borrow(obj));
  260         if (info.vectorBinding->value.dimension != dimensions ||
  261             info.vectorBinding->value.type != info.decimalDigits)
! 262             throw py::value_error("Vector dimensions and subtype must match setinputsizes");
  263         info.columnSize = VectorBufferSize(dimensions);
  264         info.isDAE = false;
  265         return;
  266     }

mssql_python/pybind/vector_support.hpp

Lines 25-35

  25 }
  26 
  27 inline void ValidateVectorShape(size_t dimensions, int subtype) {
  28     if (subtype != 0 && subtype != 1)
! 29         throw py::value_error("Invalid vector subtype; expected FLOAT32 (0) or FLOAT16 (1)");
  30     if (dimensions == 0 || dimensions > (subtype == 0 ? 1998u : 3996u))
! 31         throw py::value_error("Invalid vector dimensions for the selected subtype");
  32 }
  33 
  34 struct VectorBinding {
  35     std::vector<float> elements;

Lines 48-60

  48         : VectorBinding(source.attr("dimensions").cast<size_t>(),
  49                         source.attr("base_type").cast<std::string>() == "float16" ? 1 : 0) {
  50         auto values = source.attr("values").cast<py::tuple>();
  51         if (values.size() != elements.size())
! 52             throw py::value_error("Vector dimensions do not match its values");
  53         for (size_t i = 0; i < elements.size(); ++i) {
  54             double number = values[i].cast<double>();
  55             if (!std::isfinite(number) || std::abs(number) > std::numeric_limits<float>::max())
! 56                 throw py::value_error("Vector elements must be finite float32 values");
  57             elements[i] = static_cast<float>(number);
  58         }
  59         indicator = static_cast<SQLLEN>(VectorBufferSize(elements.size()));
  60     }

Lines 61-81

  61 
  62     VectorBinding(const VectorBinding&) = delete;
  63     VectorBinding& operator=(const VectorBinding&) = delete;
  64 
! 65     py::object toPython() const {
! 66         if (indicator == SQL_NULL_DATA) return py::none();
! 67         ValidateVectorShape(value.dimension, value.type);
! 68         if (static_cast<size_t>(value.dimension) > elements.size() ||
! 69             indicator != static_cast<SQLLEN>(VectorBufferSize(value.dimension)))
! 70             throw py::value_error("ODBC returned an invalid vector length");
! 71         py::tuple values(value.dimension);
! 72         for (size_t i = 0; i < static_cast<size_t>(value.dimension); ++i)
! 73             values[i] = py::float_(elements[i]);
  74         // The shape check above covers what ODBC can get wrong, and SQL Server cannot
  75         // store non-finite elements, so skip the public constructor's per-element checks.
! 76         PyTypeCache::initialize();
! 77         return py::reinterpret_borrow<py::object>(PyTypeCache::vector_from_trusted)(
! 78             values, value.type == SQL_VECTOR_TYPE_FLOAT16 ? "float16" : "float32");
! 79     }
  80 };


📋 Files Needing Attention

📉 Files with overall lowest coverage (click to expand)
mssql_python.pybind.performance_counter.hpp: 0.7%
mssql_python.pybind.logger_bridge.cpp: 57.9%
mssql_python.pybind.vector_support.hpp: 59.5%
mssql_python.pybind.ddbc_bindings.h: 64.8%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python._native_fetch.py: 78.1%
mssql_python.pybind.ddbc_bindings.cpp: 78.3%
mssql_python.pybind.row_mapping_cache.hpp: 81.5%
mssql_python.pybind.connection.connection_pool.cpp: 82.5%
mssql_python.pybind.connection.connection.cpp: 83.1%

🔗 Quick Links

⚙️ Build Summary 📋 Coverage Details

View Azure DevOps Build

Browse Full Coverage Report

Invalidate prepared-query caches before calling procedures and reject overwritten non-vector INPUT_OUTPUT values before native binding. Add unit and Azure regressions and remove a stale TODO.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 17, 2026 08:00
@gargsaumya

Copy link
Copy Markdown
Contributor Author

Pushed review fixes in 9c515b5 and re-reviewed the updated feature diff.

  • Fixed the valid prepared-statement cache finding: both caches are invalidated before DDBCCallProc, including failure paths. Added the exact execute -> failing callproc -> execute regression, plus live missing-procedure and execution-error cases for all three vector configurations.
  • Re-review also found that overwriting InputOutputParameter.value with a non-vector could reach ODBC without validation. Native code now rejects it before preparation/binding; added 15 live cases covering invalid values and subsequent cursor reuse.
  • Removed the carried-over TODO comment. Replied inline to all seven localhost warnings with the parser/mock evidence; those are test-only false positives, so the fixtures were not changed.

Validation passed:

  • Release native build.
  • 114 vector unit tests and 6 strict source/stub/API typing tests.
  • All 188 Azure vector E2E cases, zero skipped: v1/FLOAT32, v2/FLOAT32 and v2/FLOAT16 using isolated ODBC 18.7.1.1.
  • 16 existing live input-size, prepared-statement and native-fetch regressions.
  • Pinned full-directory Black check, commit/push hooks and whitespace checks.

Both regressions were observed failing before their respective fixes. Temporary Azure servers were deleted and deletion was verified.

The previous ADO validation failures are separate setup blockers, not failing vector assertions:

  • Nine jobs fail while installing mssql-python-rs==0.5.0.dev20261117: the public nightly feed reports either no matching distribution or HTTP 403 (anonymous identity lacks Reader permission). The same 403 occurs on main in build 192647.
  • macOS x86_64 SQL2025 fails before tests because the SQL Server container exits with "Failed to load LSA: 0xc0000001", including its retry.
  • The Debian result-copy failures are secondary to the dependency installation failure.

Evidence: PR build 192677, main build 192647.

I have not changed runtime pins, disabled jobs or weakened validation to hide these failures. The feed's intended public access/authentication and the macOS SQL container/runner need to be restored by their owners before the standard matrix can go green.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

An empty buffered procedure result consumed by fetchall() leaves rowcount at -1 instead of 0; exhaustion handling needs correction and a regression test.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Empty buffered fetchall leaves rowcount at -1

mssql_python/​cursor.py:3299

For an empty buffered procedure result set, fetchall() calls fetchmany(0), which returns before updating rowcount. This leaves rowcount at -1 after the result is consumed, unlike fetchone() and fetchmany(), which set it to 0. Request at least one row to process exhaustion, and add an empty buffered-result regression.

1 resolved since last review

Review effort: Balanced

Comment thread mssql_python/pybind/ddbc_bindings.cpp Resolved
Comment thread mssql_python/pybind/ddbc_bindings.cpp Resolved
Comment thread mssql_python/cursor.py Resolved
Copilot AI balanced review requested due to automatic review settings November 18, 2026 05:08
@gargsaumya
gargsaumya enabled auto-merge (squash) November 18, 2026 05:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

An empty buffered procedure result leaves rowcount at -1 after fetchall() instead of finalizing it to 0, requiring a fetch-path fix and regression coverage.

1 open issue

Review effort: Balanced

Comment thread mssql_python/cursor.py Resolved
Bind TVP TIME columns as text so fractional seconds survive, size numeric/time text buffers separately from SQL precision, normalize non-Decimal DECIMAL/NUMERIC TVP cells like executemany(), and finalize rowcount when fetchall() exhausts an empty buffered procedure result. Add unit and live regressions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 18, 2026 07:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Native ABI, TVP handling, and buffer cleanup across new ODBC paths need human review because reported validation is Windows-only and this environment cannot perform cross-platform or live SQL/fault-injection checks.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings November 18, 2026 08:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Platform-sensitive native ABI and buffer-ownership changes need cross-platform runtime and failure-path validation beyond the reported Windows x64 checks, which could not be independently verified.

No open issues 🎉

Review effort: Balanced

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vector reads and writes work end to end for float32, including batches and procedure outputs. two things need to change before merge: the table-valued parameter API here collides with the one already in review, and batch inserts got slower for users who never touch vectors. requesting changes.

Comment thread mssql_python/table_parameter.py Resolved
Comment thread mssql_python/cursor.py Resolved
Comment thread mssql_python/pybind/vector_support.hpp Resolved
Comment thread mssql_python/vector.py Resolved
Comment thread mssql_python/vector.py Resolved
Comment thread mssql_python/cursor.py Resolved
Drop the PR-local TableValuedParameter and TVP binder so the shared TVP definition (#789) is the single public API; vector TVP columns will build on it. Restore main's array-binding buffer ownership.

OutputParameter now takes the setinputsizes (sql_type, size, scale) triple; only SQL_SS_VECTOR is supported and other types raise NotSupportedError. InputOutputParameter infers its type from the initial Vector.

Avoid per-cell ABC isinstance scans in executemany() and bulkcopy() by gating on exact Vector type. Build fetched and output vectors through a cached trusted constructor instead of revalidating driver-produced values. Add a bulk fast path for list/tuple float inputs with identical values and errors. Accept bracket-quoted callproc name parts and reject malformed brackets.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 18, 2026 13:05
Complete dropping the PR-local TVP API in favor of the shared definition (#789); nothing imports this module any more.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The table-valued parameter class lacks a public export and execution/binding integration, leaving the advertised support unimplemented and inconsistent with the README’s deferred scope.

No open issues 🎉

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings November 18, 2026 13:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Native vector ABI, pointer-bearing bindings, and procedure-output cleanup require qualified live-driver validation beyond the reported Windows x64 coverage, leaving cross-platform/provider compatibility and native failure paths unverified.

No open issues 🎉

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Native ODBC vector ABI, array binding, and procedure-output buffer lifetimes require cross-platform runtime validation beyond the reported Windows-only testing, including failure-path cleanup.

No open issues 🎉

Review effort: Balanced

gargsaumya and others added 2 commits November 18, 2026 22:00
FLOAT16 is preview-only on SQL Server 2025 (PREVIEW_FEATURES) and GA on Azure SQL DB, MI and Fabric. The driver never converts between vector base types; SQL Server converts native FLOAT32/FLOAT16 parameters to the column type, matching Microsoft.Data.SqlClient. Add a live test pinning conversion, rounding and the out-of-range error.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 18, 2026 18:00
gargsaumya added a commit that referenced this pull request Nov 18, 2026
Pick up the reviewed #902 changes (TVP removal, OutputParameter triple, trusted vector construction, exact Vector type checks, SQL Server 2025 FLOAT16 preview docs).

Align vector bulk copy and Arrow docs/tests with Microsoft.Data.SqlClient: the client never converts between base types; columns are declared with their source base type (JSON text for FLOAT16 columns as text) and SQL Server converts them. Replace base-type mismatch rejection tests with live conversion, rounding, out-of-range and mixed-column tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because buffered callproc() results can leave skip() and scroll() out of sync with subsequent fetches and row-position counters.

No open issues 🎉

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-size: large Substantial code update

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants