Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,14 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
the previous prepared-query cache even when the procedure fails; overwritten
non-vector INPUT_OUTPUT values are rejected before binding. FLOAT16 requires
Microsoft ODBC Driver 18.7.1.1+ (not supplied by the current bundled driver); ODBC owns
half-precision conversion. Vector bulk copy remains pending pycore's contract,
and Arrow vector fetches are not supported.
half-precision conversion.
- FLOAT32/FLOAT16 vector bulk copy through a typed pycore bridge and Arrow
read/write support using fixed-size float32 lists with SQL base-type metadata.
As in Microsoft.Data.SqlClient, bulk copy never converts between base types:
each column is declared with its source base type (JSON text for FLOAT16
columns is sent as text) and SQL Server converts it to the destination. The
companion vector-capable pycore build is required and the released runtime pin
is unchanged pending publication.
- New feature: Support for macOS and Linux.
- Documentation: Added API documentation in the Wiki.
- New `token_provider=` parameter on `connect()` / `Connection` for Microsoft
Expand Down
76 changes: 68 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -287,11 +287,69 @@ print(output.value)
Procedure text/binary inputs
must fit the inline binding limits. Table-valued parameters are not part of this
vector support; vector TVP columns will build on the shared TVP definition.
Arrow vector fetches and Arrow access to
buffered procedure results are not supported; use row fetch methods instead.
**Vector bulk copy is pending the pycore vector input contract.** `bulkcopy()`
explicitly rejects `Vector` cells rather than guessing a serialization format.
Once that backend contract is available, pycore will own its FLOAT16 conversion.
Arrow access to buffered procedure results is not supported; use row fetch
methods for those results.

**Bulk copy and Arrow:** `bulkcopy()` accepts `Vector` cells in tuple or `Row`
sources, including generators and SQL NULLs. A typed bridge preserves each
vector's dimensions and `base_type`; dimensions must match the mapped
destination column. Pycore owns TDS serialization, not the Python wrapper. This
path requires the companion vector-capable pycore build exposing
`BulkCopyVector`; an older backend raises `NotSupportedError` for `Vector` cells.
The released runtime pin is not changed until that companion build is published.
As with other bulk-copy types, this uses a separate connection: commit table
creation before loading, do not use connection-local temporary tables, and do
not assume rollback on the ODBC connection undoes a bulk load. Its native
TDS connection negotiates vector support independently of the ODBC connection's
`VectorTypeSupport` setting.

Base types follow Microsoft.Data.SqlClient's SqlBulkCopy: the client never
converts between FLOAT32 and FLOAT16. Each vector column is declared with the
base type of its first row's `Vector`, and SQL Server converts it to the
destination column (rounding to nearest; values outside the FLOAT16 range fail
on the server). Every `Vector` in a column must then share that base type. JSON
text for a FLOAT16 column is sent as text and parsed by SQL Server; JSON for a
FLOAT32 column and plain numeric lists are converted to the destination base
type, as before.

`arrow_batch()`, `arrow()`, and `arrow_reader()` represent a native SQL vector as
an Arrow **fixed-size list of float32** for both SQL base types. The list size is
the number of dimensions, and field metadata
`b"mssql.vector.base_type": b"float32"` (or `b"float16"`) preserves SQL storage.
NULLs belong to the parent list; individual vector elements cannot be NULL.
The native fetch path copies float32 buffers directly without creating Python
floats per element. Vector-containing results use rowwise ODBC retrieval;
ordinary Arrow results retain the existing column-binding path.

`bulkcopy_arrow()` accepts the same representation with tables, record batches,
readers, iterables of batches, and Arrow C stream/array producers. It requires
the companion pycore Arrow vector adapter. The destination schema validates
dimensions. A column tagged with `mssql.vector.base_type` metadata is sent with
that base type and SQL Server converts it to the destination, as for row bulk
copy; untagged columns are converted to the destination base type, like lists.
Variable-size lists and non-float32 child arrays are not
implicitly converted. For raw Arrow input, finite/range and non-NULL-element
checks occur during native bulk serialization. Earlier batches may already have
been written when a later batch fails.

```python
import pyarrow as pa

# dbo.Embeddings already exists: (id INT, embedding VECTOR(3, FLOAT16)).
cursor.bulkcopy(
"dbo.Embeddings",
[(1, Vector([0.1, 0.2, 0.3], base_type="float16")), (2, None)],
)
cursor.execute("SELECT id, embedding FROM dbo.Embeddings")
table = cursor.arrow()
assert table.schema.field("embedding").type == pa.list_(
pa.field("item", pa.float32(), nullable=False), 3
)
# Use a different, existing table with a matching schema for streaming copies.
cursor.execute("SELECT id, embedding FROM dbo.Embeddings")
with cursor.arrow_reader() as reader:
cursor.bulkcopy_arrow("dbo.EmbeddingsCopy", reader)
```

Vector unit tests run without SQL Server. End-to-end tests in
`tests/test_025_vector.py` exercise v1/FLOAT32, v2/FLOAT32, and v2/FLOAT16:
Expand All @@ -300,9 +358,11 @@ typed/all-NULL arrays and streamed text), numeric boundaries and error recovery,
every row fetch method including `fetchval()`, metadata/row access, output converters,
commit/rollback, stored procedure outputs/result sets, and error recovery.
Negotiation tests assert that v1 rejects native FLOAT16, `off` rejects native
vector inputs, and `off` retains JSON/NULL reads. Arrow and bulk-copy cases assert
the current explicit unsupported boundaries; they are not positive vector
bulk-copy tests.
vector inputs, and `off` retains JSON/NULL reads. Vector-capable pycore builds
add positive row/Arrow bulk-copy round trips, column mappings, streaming sources,
NULLs, dimensional/range errors, and Arrow ownership/schema checks.
Bulk-copy integration cases skip explicitly when the companion pycore vector
bridge is absent; native Arrow fetch coverage does not require that bridge.

Set `DB_CONNECTION_STRING` outside source control to enable the live tests.
The database principal needs permission to create tables, procedures, and table
Expand Down
37 changes: 23 additions & 14 deletions mssql_python/cursor.py
Original file line number Diff line number Diff line change
Expand Up @@ -3447,6 +3447,8 @@ def arrow_batch(self, batch_size: int = 8192) -> "pyarrow.RecordBatch":

Returns:
A pyarrow RecordBatch object containing up to batch_size rows.
Vector columns use fixed-size float32 lists with
``mssql.vector.base_type`` field metadata preserving SQL storage.
"""
batch = self._fetch_arrow_batch(batch_size)
assert batch is not None
Expand All @@ -3456,12 +3458,10 @@ def _fetch_arrow_batch(
self, batch_size: int, reader_state: ddbc_bindings._ArrowReaderState | None = None
) -> Optional["pyarrow.RecordBatch"]:
self._check_closed() # Check if the cursor is closed
if getattr(self, "_procedure_rows", None) is not None or any(
column[1] is Vector for column in (self.description or ())
):
if getattr(self, "_procedure_rows", None) is not None:
raise NotSupportedError(
driver_error="Use row fetch methods for vectors and buffered callproc results",
ddbc_error="Arrow vector conversion is not implemented",
driver_error="Use row fetch methods for buffered callproc results",
ddbc_error="Arrow conversion of buffered procedure results is not supported",
)
pa = self._ensure_pyarrow()

Expand Down Expand Up @@ -3969,6 +3969,8 @@ def bulkcopy(
Row objects from fetchone/fetchmany/fetchall are automatically
converted to tuples. Lists and other types are not accepted.
Rows must use the same representation throughout the iterable.
Vector cells preserve their dimensions and base_type, which must
match the destination column. FLOAT16 encoding is handled by pycore.

Data Format Requirements:
- Each element in the iterable represents one row
Expand Down Expand Up @@ -4057,15 +4059,15 @@ def bulkcopy(

pycore_connection = None
pycore_cursor = None
vector_validation_error: NotSupportedError | None = None
vector_validation_error: NotSupportedError | TypeError | ValueError | None = None

def _validated_vector_rows(
rows: Iterable[tuple[Any, ...]],
) -> Iterator[tuple[Any, ...]]:
nonlocal vector_validation_error
try:
yield from self._validate_bulkcopy_vectors(rows)
except NotSupportedError as error:
yield from self._adapt_bulkcopy_vectors(rows, mssql_py_core)
except (NotSupportedError, TypeError, ValueError) as error:
vector_validation_error = error
raise

Expand Down Expand Up @@ -4629,16 +4631,23 @@ def tables(
raise

@staticmethod
def _validate_bulkcopy_vectors(
rows: Iterable[tuple[Any, ...]],
def _adapt_bulkcopy_vectors(
rows: Iterable[tuple[Any, ...]], core: PyCoreModule
) -> Iterator[tuple[Any, ...]]:
vector_type = getattr(core, "BulkCopyVector", None)
for row in rows:
if any(type(value) is Vector for value in row):
raise NotSupportedError(
driver_error="Vector bulk copy requires the developing pycore vector contract",
ddbc_error="No vector serialization is performed by the Python wrapper",
if vector_type is None:
raise NotSupportedError(
driver_error="Vector bulk copy requires a vector-capable pycore build",
ddbc_error="The installed pycore does not expose BulkCopyVector",
)
yield tuple(
(vector_type(value.values, value.base_type) if type(value) is Vector else value)
for value in row
)
yield row
else:
yield row

def _next_procedure_result(self) -> bool:
self._cached_column_map = None
Expand Down
Loading
Loading