diff --git a/CHANGELOG.md b/CHANGELOG.md index db7616423..a2d0d5f7b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,8 +19,14 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), the previous prepared-query cache even when the procedure fails; overwritten non-vector INPUT_OUTPUT values are rejected before binding. FLOAT16 requires Microsoft ODBC Driver 18.7.1.1+ (not supplied by the current bundled driver); ODBC owns - half-precision conversion. Vector bulk copy remains pending pycore's contract, - and Arrow vector fetches are not supported. + half-precision conversion. +- FLOAT32/FLOAT16 vector bulk copy through a typed pycore bridge and Arrow + read/write support using fixed-size float32 lists with SQL base-type metadata. + As in Microsoft.Data.SqlClient, bulk copy never converts between base types: + each column is declared with its source base type (JSON text for FLOAT16 + columns is sent as text) and SQL Server converts it to the destination. The + companion vector-capable pycore build is required and the released runtime pin + is unchanged pending publication. - New feature: Support for macOS and Linux. - Documentation: Added API documentation in the Wiki. - New `token_provider=` parameter on `connect()` / `Connection` for Microsoft diff --git a/README.md b/README.md index 4d34f940d..04ac36dba 100644 --- a/README.md +++ b/README.md @@ -287,11 +287,69 @@ print(output.value) Procedure text/binary inputs must fit the inline binding limits. Table-valued parameters are not part of this vector support; vector TVP columns will build on the shared TVP definition. -Arrow vector fetches and Arrow access to -buffered procedure results are not supported; use row fetch methods instead. -**Vector bulk copy is pending the pycore vector input contract.** `bulkcopy()` -explicitly rejects `Vector` cells rather than guessing a serialization format. -Once that backend contract is available, pycore will own its FLOAT16 conversion. +Arrow access to buffered procedure results is not supported; use row fetch +methods for those results. + +**Bulk copy and Arrow:** `bulkcopy()` accepts `Vector` cells in tuple or `Row` +sources, including generators and SQL NULLs. A typed bridge preserves each +vector's dimensions and `base_type`; dimensions must match the mapped +destination column. Pycore owns TDS serialization, not the Python wrapper. This +path requires the companion vector-capable pycore build exposing +`BulkCopyVector`; an older backend raises `NotSupportedError` for `Vector` cells. +The released runtime pin is not changed until that companion build is published. +As with other bulk-copy types, this uses a separate connection: commit table +creation before loading, do not use connection-local temporary tables, and do +not assume rollback on the ODBC connection undoes a bulk load. Its native +TDS connection negotiates vector support independently of the ODBC connection's +`VectorTypeSupport` setting. + +Base types follow Microsoft.Data.SqlClient's SqlBulkCopy: the client never +converts between FLOAT32 and FLOAT16. Each vector column is declared with the +base type of its first row's `Vector`, and SQL Server converts it to the +destination column (rounding to nearest; values outside the FLOAT16 range fail +on the server). Every `Vector` in a column must then share that base type. JSON +text for a FLOAT16 column is sent as text and parsed by SQL Server; JSON for a +FLOAT32 column and plain numeric lists are converted to the destination base +type, as before. + +`arrow_batch()`, `arrow()`, and `arrow_reader()` represent a native SQL vector as +an Arrow **fixed-size list of float32** for both SQL base types. The list size is +the number of dimensions, and field metadata +`b"mssql.vector.base_type": b"float32"` (or `b"float16"`) preserves SQL storage. +NULLs belong to the parent list; individual vector elements cannot be NULL. +The native fetch path copies float32 buffers directly without creating Python +floats per element. Vector-containing results use rowwise ODBC retrieval; +ordinary Arrow results retain the existing column-binding path. + +`bulkcopy_arrow()` accepts the same representation with tables, record batches, +readers, iterables of batches, and Arrow C stream/array producers. It requires +the companion pycore Arrow vector adapter. The destination schema validates +dimensions. A column tagged with `mssql.vector.base_type` metadata is sent with +that base type and SQL Server converts it to the destination, as for row bulk +copy; untagged columns are converted to the destination base type, like lists. +Variable-size lists and non-float32 child arrays are not +implicitly converted. For raw Arrow input, finite/range and non-NULL-element +checks occur during native bulk serialization. Earlier batches may already have +been written when a later batch fails. + +```python +import pyarrow as pa + +# dbo.Embeddings already exists: (id INT, embedding VECTOR(3, FLOAT16)). +cursor.bulkcopy( + "dbo.Embeddings", + [(1, Vector([0.1, 0.2, 0.3], base_type="float16")), (2, None)], +) +cursor.execute("SELECT id, embedding FROM dbo.Embeddings") +table = cursor.arrow() +assert table.schema.field("embedding").type == pa.list_( + pa.field("item", pa.float32(), nullable=False), 3 +) +# Use a different, existing table with a matching schema for streaming copies. +cursor.execute("SELECT id, embedding FROM dbo.Embeddings") +with cursor.arrow_reader() as reader: + cursor.bulkcopy_arrow("dbo.EmbeddingsCopy", reader) +``` Vector unit tests run without SQL Server. End-to-end tests in `tests/test_025_vector.py` exercise v1/FLOAT32, v2/FLOAT32, and v2/FLOAT16: @@ -300,9 +358,11 @@ typed/all-NULL arrays and streamed text), numeric boundaries and error recovery, every row fetch method including `fetchval()`, metadata/row access, output converters, commit/rollback, stored procedure outputs/result sets, and error recovery. Negotiation tests assert that v1 rejects native FLOAT16, `off` rejects native -vector inputs, and `off` retains JSON/NULL reads. Arrow and bulk-copy cases assert -the current explicit unsupported boundaries; they are not positive vector -bulk-copy tests. +vector inputs, and `off` retains JSON/NULL reads. Vector-capable pycore builds +add positive row/Arrow bulk-copy round trips, column mappings, streaming sources, +NULLs, dimensional/range errors, and Arrow ownership/schema checks. +Bulk-copy integration cases skip explicitly when the companion pycore vector +bridge is absent; native Arrow fetch coverage does not require that bridge. Set `DB_CONNECTION_STRING` outside source control to enable the live tests. The database principal needs permission to create tables, procedures, and table diff --git a/mssql_python/cursor.py b/mssql_python/cursor.py index 3b81bf612..df9b1acc4 100644 --- a/mssql_python/cursor.py +++ b/mssql_python/cursor.py @@ -3447,6 +3447,8 @@ def arrow_batch(self, batch_size: int = 8192) -> "pyarrow.RecordBatch": Returns: A pyarrow RecordBatch object containing up to batch_size rows. + Vector columns use fixed-size float32 lists with + ``mssql.vector.base_type`` field metadata preserving SQL storage. """ batch = self._fetch_arrow_batch(batch_size) assert batch is not None @@ -3456,12 +3458,10 @@ def _fetch_arrow_batch( self, batch_size: int, reader_state: ddbc_bindings._ArrowReaderState | None = None ) -> Optional["pyarrow.RecordBatch"]: self._check_closed() # Check if the cursor is closed - if getattr(self, "_procedure_rows", None) is not None or any( - column[1] is Vector for column in (self.description or ()) - ): + if getattr(self, "_procedure_rows", None) is not None: raise NotSupportedError( - driver_error="Use row fetch methods for vectors and buffered callproc results", - ddbc_error="Arrow vector conversion is not implemented", + driver_error="Use row fetch methods for buffered callproc results", + ddbc_error="Arrow conversion of buffered procedure results is not supported", ) pa = self._ensure_pyarrow() @@ -3969,6 +3969,8 @@ def bulkcopy( Row objects from fetchone/fetchmany/fetchall are automatically converted to tuples. Lists and other types are not accepted. Rows must use the same representation throughout the iterable. + Vector cells preserve their dimensions and base_type, which must + match the destination column. FLOAT16 encoding is handled by pycore. Data Format Requirements: - Each element in the iterable represents one row @@ -4057,15 +4059,15 @@ def bulkcopy( pycore_connection = None pycore_cursor = None - vector_validation_error: NotSupportedError | None = None + vector_validation_error: NotSupportedError | TypeError | ValueError | None = None def _validated_vector_rows( rows: Iterable[tuple[Any, ...]], ) -> Iterator[tuple[Any, ...]]: nonlocal vector_validation_error try: - yield from self._validate_bulkcopy_vectors(rows) - except NotSupportedError as error: + yield from self._adapt_bulkcopy_vectors(rows, mssql_py_core) + except (NotSupportedError, TypeError, ValueError) as error: vector_validation_error = error raise @@ -4629,16 +4631,23 @@ def tables( raise @staticmethod - def _validate_bulkcopy_vectors( - rows: Iterable[tuple[Any, ...]], + def _adapt_bulkcopy_vectors( + rows: Iterable[tuple[Any, ...]], core: PyCoreModule ) -> Iterator[tuple[Any, ...]]: + vector_type = getattr(core, "BulkCopyVector", None) for row in rows: if any(type(value) is Vector for value in row): - raise NotSupportedError( - driver_error="Vector bulk copy requires the developing pycore vector contract", - ddbc_error="No vector serialization is performed by the Python wrapper", + if vector_type is None: + raise NotSupportedError( + driver_error="Vector bulk copy requires a vector-capable pycore build", + ddbc_error="The installed pycore does not expose BulkCopyVector", + ) + yield tuple( + (vector_type(value.values, value.base_type) if type(value) is Vector else value) + for value in row ) - yield row + else: + yield row def _next_procedure_result(self) -> bool: self._cached_column_map = None diff --git a/mssql_python/pybind/ddbc_bindings.cpp b/mssql_python/pybind/ddbc_bindings.cpp index 7bb165334..e8bdb4b3e 100644 --- a/mssql_python/pybind/ddbc_bindings.cpp +++ b/mssql_python/pybind/ddbc_bindings.cpp @@ -136,6 +136,19 @@ inline int EffectiveCharCtypeForFetch(int charCtype, const std::string& charEnco // Struct to hold parameter information for binding. Used by SQLBindParameter. +struct ArrowArray; +struct ArrowSchema; + +template +struct ArrowColumnDeleter { + void operator()(T* value) const { + if (value) { + if (value->release) value->release(value); + delete value; + } + } +}; + struct ArrowArrayPrivateData { std::unique_ptr valid; @@ -162,11 +175,16 @@ struct ArrowArrayPrivateData { // Points to one of the typed *Val buffers above. Since the buffer pointers // don't change, this can be set once during batch initialization. void* ptrValueBuffer; + std::unique_ptr child; + std::array children{}; }; struct ArrowSchemaPrivateData { std::unique_ptr name; std::unique_ptr format; + std::string metadata; + std::unique_ptr child; + std::array children{}; }; #ifndef ARROW_C_DATA_INTERFACE @@ -5615,6 +5633,8 @@ SQLRETURN FetchArrowBatch_wrap(SqlHandlePtr StatementHandle, py::list& capsules, std::vector columnNullable(numCols); std::vector columnVarLen(numCols, false); std::vector nullCounts(numCols, 0); + std::vector> vectorBindings(numCols); + std::vector vectorSubtypes(numCols, 0); std::vector> arrowArrayPrivateData(numCols); std::vector> arrowSchemaPrivateData(numCols); @@ -5649,6 +5669,33 @@ SQLRETURN FetchArrowBatch_wrap(SqlHandlePtr StatementHandle, py::list& capsules, std::string format = ""; switch (dataType) { + case SQL_SS_VECTOR: { + if (columnSize < sizeof(SQL_SS_VECTOR_STRUCT) || + (columnSize - sizeof(SQL_SS_VECTOR_STRUCT)) % sizeof(float) != 0) + throw py::value_error("ODBC returned invalid vector column metadata"); + size_t dimensions = (columnSize - sizeof(SQL_SS_VECTOR_STRUCT)) / sizeof(float); + int subtype = colMeta["DecimalDigits"].cast(); + vectorSubtypes[i] = subtype; + vectorBindings[i] = std::make_unique(dimensions, subtype); + hasLobColumns = true; // Vector ABI pointers cannot be copied as bound columns. + fetchSize = 1; + format = "+w:" + std::to_string(dimensions); + arrowColumnProducer->float32Val = + std::make_unique(static_cast(arrowBatchSize) * dimensions); + arrowColumnProducer->ptrValueBuffer = nullptr; + auto& metadata = arrowSchemaPrivateData[i]->metadata; + auto appendInt = [&](int32_t value) { + metadata.append(reinterpret_cast(&value), sizeof(value)); + }; + const std::string key = "mssql.vector.base_type"; + const std::string value = subtype == 1 ? "float16" : "float32"; + appendInt(1); + appendInt(static_cast(key.size())); + metadata.append(key); + appendInt(static_cast(value.size())); + metadata.append(value); + break; + } case SQL_CHAR: case SQL_VARCHAR: case SQL_LONGVARCHAR: @@ -5850,6 +5897,25 @@ SQLRETURN FetchArrowBatch_wrap(SqlHandlePtr StatementHandle, py::list& capsules, assert(idxRowSql == 0 && "GetData only works one row at a time"); switch (dataType) { + case SQL_SS_VECTOR: { + auto& binding = *vectorBindings[idxCol]; + { + py::gil_scoped_release release; + ret = SQLGetData_ptr( + hStmt, idxCol + 1, SQL_C_SS_VECTOR, &binding.value, + static_cast(VectorBufferSize(binding.elements.size())), + &binding.indicator); + } + fetchStateGuard.observe(ret); + if (!SQL_SUCCEEDED(ret)) return ret; + binding.validateResult(); + if (binding.indicator != SQL_NULL_DATA && + (static_cast(binding.value.dimension) != binding.elements.size() || + binding.value.type != vectorSubtypes[idxCol])) + throw py::value_error("ODBC returned inconsistent vector metadata"); + buffers.indicators[idxCol][0] = binding.indicator; + break; + } case SQL_SS_UDT: case SQL_BINARY: case SQL_VARBINARY: @@ -6117,6 +6183,13 @@ SQLRETURN FetchArrowBatch_wrap(SqlHandlePtr StatementHandle, py::list& capsules, auto dataLen = static_cast(indicator); switch (dataType) { + case SQL_SS_VECTOR: { + const auto& binding = *vectorBindings[idxCol]; + std::memcpy(arrowColumnProducer->float32Val.get() + + static_cast(idxRowArrow) * binding.elements.size(), + binding.elements.data(), binding.elements.size() * sizeof(float)); + break; + } case SQL_SS_UDT: case SQL_BINARY: case SQL_VARBINARY: @@ -6342,9 +6415,10 @@ SQLRETURN FetchArrowBatch_wrap(SqlHandlePtr StatementHandle, py::list& capsules, auto arrowSchemaBatch = std::make_unique(); auto arrowSchemaBatchChildren = std::make_unique(numCols); - auto arrowSchemaBatchChildPointers = std::make_unique[]>(numCols); + using SchemaOwner = std::unique_ptr>; + auto arrowSchemaBatchChildPointers = std::make_unique(numCols); for (SQLSMALLINT i = 0; i < numCols; i++) { - arrowSchemaBatchChildPointers[i] = std::make_unique(); + arrowSchemaBatchChildPointers[i] = SchemaOwner(new ArrowSchema{}); } // Second, transfer ownership to arrowSchemaBatch @@ -6354,7 +6428,8 @@ SQLRETURN FetchArrowBatch_wrap(SqlHandlePtr StatementHandle, py::list& capsules, *arrowSchemaBatchChildPointers[i] = { arrowSchemaPrivateData[i]->format.get(), arrowSchemaPrivateData[i]->name.get(), - nullptr, + arrowSchemaPrivateData[i]->metadata.empty() + ? nullptr : arrowSchemaPrivateData[i]->metadata.data(), static_cast(columnNullable[i] ? ARROW_FLAG_NULLABLE : 0), 0, nullptr, @@ -6363,12 +6438,33 @@ SQLRETURN FetchArrowBatch_wrap(SqlHandlePtr StatementHandle, py::list& capsules, assert(schema != nullptr); assert(schema->release != nullptr); assert(schema->private_data != nullptr); - assert(schema->children == nullptr && schema->n_children == 0); + for (int64_t child = 0; child < schema->n_children; ++child) { + if (schema->children[child]->release) + schema->children[child]->release(schema->children[child]); + } delete schema->private_data; // Frees format and name schema->release = nullptr; }, arrowSchemaPrivateData[i].release(), }; + if (vectorBindings[i]) { + auto* parent = arrowSchemaBatchChildPointers[i].get(); + auto* owner = parent->private_data; + auto childOwner = std::make_unique(); + childOwner->format = std::make_unique(2); + std::memcpy(childOwner->format.get(), "f", 2); + childOwner->name = std::make_unique(5); + std::memcpy(childOwner->name.get(), "item", 5); + owner->child = std::make_unique(); + *owner->child = { + childOwner->format.get(), childOwner->name.get(), nullptr, + 0, 0, nullptr, nullptr, parent->release, childOwner.get(), + }; + childOwner.release(); + owner->children[0] = owner->child.get(); + parent->n_children = 1; + parent->children = owner->children.data(); + } } for (SQLSMALLINT i = 0; i < numCols; i++) { @@ -6430,9 +6526,10 @@ SQLRETURN FetchArrowBatch_wrap(SqlHandlePtr StatementHandle, py::list& capsules, arrowArrayBatchBuffers[0] = nullptr; auto arrowArrayBatchChildren = std::make_unique(numCols); - auto arrowArrayBatchChildPointers = std::make_unique[]>(numCols); + using ArrayOwner = std::unique_ptr>; + auto arrowArrayBatchChildPointers = std::make_unique(numCols); for (SQLSMALLINT i = 0; i < numCols; i++) { - arrowArrayBatchChildPointers[i] = std::make_unique(); + arrowArrayBatchChildPointers[i] = ArrayOwner(new ArrowArray{}); } // Second, transfer ownership to arrowArrayBatch @@ -6456,14 +6553,36 @@ SQLRETURN FetchArrowBatch_wrap(SqlHandlePtr StatementHandle, py::list& capsules, assert(array != nullptr); assert(array->private_data != nullptr); assert(array->release != nullptr); - assert(array->children == nullptr); - assert(array->n_children == 0); + for (int64_t child = 0; child < array->n_children; ++child) { + if (array->children[child]->release) + array->children[child]->release(array->children[child]); + } delete array->private_data; // Frees all buffer entries assert(array->buffers != nullptr); array->release = nullptr; }, arrowArrayPrivateData[col].release(), }; + if (vectorBindings[col]) { + auto* parent = arrowArrayBatchChildPointers[col].get(); + auto* owner = parent->private_data; + auto childOwner = std::make_unique(); + childOwner->float32Val = std::move(owner->float32Val); + childOwner->buffers[0] = nullptr; + childOwner->buffers[1] = childOwner->float32Val.get(); + owner->child = std::make_unique(); + *owner->child = { + static_cast(idxRowArrow) * + static_cast(vectorBindings[col]->elements.size()), + 0, 0, 2, 0, (const void**)childOwner->buffers.data(), + nullptr, nullptr, parent->release, childOwner.get(), + }; + childOwner.release(); + owner->children[0] = owner->child.get(); + parent->n_buffers = 1; + parent->n_children = 1; + parent->children = owner->children.data(); + } } for (SQLSMALLINT i = 0; i < numCols; i++) { diff --git a/mssql_python/pybind/vector_support.hpp b/mssql_python/pybind/vector_support.hpp index 9c6640b40..5a24db6aa 100644 --- a/mssql_python/pybind/vector_support.hpp +++ b/mssql_python/pybind/vector_support.hpp @@ -62,12 +62,17 @@ struct VectorBinding { VectorBinding(const VectorBinding&) = delete; VectorBinding& operator=(const VectorBinding&) = delete; - py::object toPython() const { - if (indicator == SQL_NULL_DATA) return py::none(); + void validateResult() const { + if (indicator == SQL_NULL_DATA) return; ValidateVectorShape(value.dimension, value.type); if (static_cast(value.dimension) > elements.size() || indicator != static_cast(VectorBufferSize(value.dimension))) throw py::value_error("ODBC returned an invalid vector length"); + } + + py::object toPython() const { + if (indicator == SQL_NULL_DATA) return py::none(); + validateResult(); py::tuple values(value.dimension); for (size_t i = 0; i < static_cast(value.dimension); ++i) values[i] = py::float_(elements[i]); diff --git a/tests/test_025_vector.py b/tests/test_025_vector.py index cd336ca82..e7f8e271a 100644 --- a/tests/test_025_vector.py +++ b/tests/test_025_vector.py @@ -608,9 +608,23 @@ def test_executemany_rejects_mixed_vector_columns(unit_cursor, values, error): def test_bulkcopy_does_not_invent_pycore_serialization(): rows = [(1, "x"), (2, None)] - assert list(Cursor._validate_bulkcopy_vectors(rows)) == rows + core = SimpleNamespace() + assert list(Cursor._adapt_bulkcopy_vectors(rows, core)) == rows with pytest.raises(mssql_python.NotSupportedError, match="pycore"): - list(Cursor._validate_bulkcopy_vectors([(1, Vector([1, 2, 3]))])) + list(Cursor._adapt_bulkcopy_vectors([(1, Vector([1, 2, 3]))], core)) + + +@pytest.mark.parametrize("base_type", ["float32", "float16"]) +def test_bulkcopy_adapts_only_vectors_and_preserves_base_type(base_type): + value = Vector([0.1, 2, 3], base_type) + converted = object() + factory = Mock(return_value=converted) + core = SimpleNamespace(BulkCopyVector=factory) + ordinary = (1, [1, 2, 3], None) + rows = list(Cursor._adapt_bulkcopy_vectors(iter([ordinary, (value, None)]), core)) + assert rows[0] is ordinary + assert rows[1] == (converted, None) + factory.assert_called_once_with(value.values, base_type) @pytest.mark.parametrize("leading_row", [False, True]) @@ -718,6 +732,16 @@ def live_vector_connection(request, conn_str, vector_server): connection.close() +@pytest.fixture +def vector_bulk_core(): + core = pytest.importorskip("mssql_py_core") + if not hasattr(core, "BulkCopyVector"): + pytest.skip( + "Vector bulk-copy integration requires the companion vector-capable pycore build" + ) + return core + + def _expected_stored_vector(values, base_type): normalized = [struct.unpack("=f", struct.pack("=f", value))[0] for value in values] if base_type == "float16": @@ -1251,28 +1275,364 @@ def test_live_vector_output_converters(live_vector_connection): @pytest.mark.parametrize("method", ["arrow_batch", "arrow", "arrow_reader"]) -def test_live_vector_arrow_boundary(live_vector_connection, method): - pytest.importorskip("pyarrow") +def test_live_vector_arrow_roundtrip(live_vector_connection, method): + pa = pytest.importorskip("pyarrow") connection, dtype = live_vector_connection value = Vector([1, 2, 3], dtype) with connection.cursor() as cursor: cursor.execute("SELECT ? AS embedding", value) - with pytest.raises(mssql_python.NotSupportedError, match="row fetch"): - getattr(cursor, method)() - assert cursor.fetchone()[0] == value + result = getattr(cursor, method)(batch_size=2) + if method == "arrow_reader": + with result: + result = result.read_all() + assert result.schema.field("embedding").type == pa.list_( + pa.field("item", pa.float32(), nullable=False), 3 + ) + assert result.schema.field("embedding").metadata == { + b"mssql.vector.base_type": dtype.encode() + } + assert result.column(0).to_pylist() == [list(value)] -def test_live_vector_bulkcopy_boundary(live_vector_connection): +@pytest.mark.parametrize("source_kind", ["tuples", "generator", "rows", "lists"]) +def test_live_vector_bulkcopy_roundtrip(live_vector_connection, vector_bulk_core, source_kind): connection, dtype = live_vector_connection table_name = f"dbo.pytest_vector_bulk_{uuid.uuid4().hex}" connection.autocommit = True + with connection.cursor() as cursor: + cursor.execute(f"CREATE TABLE {table_name} (id INT, embedding VECTOR(3, {dtype}))") + try: + value = Vector([0.1, -0.0, 1.0001], dtype) + data = [(1, None), (2, value), (3, value)] + if source_kind == "generator": + data = (row for row in data) + elif source_kind == "lists": + data = [(row_id, list(v) if v is not None else None) for row_id, v in data] + elif source_kind == "rows": + cursor.execute( + "SELECT 1 AS id, CAST(NULL AS VECTOR(3, " + dtype + ")) AS embedding " + "UNION ALL SELECT 2, ? UNION ALL SELECT 3, ?", + value, + value, + ) + data = cursor.fetchall() + result = cursor.bulkcopy( + table_name, + data, + batch_size=2, + column_mappings=[(0, "id"), (1, "embedding")], + ) + assert result["rows_copied"] == 3 + cursor.execute(f"SELECT id, embedding FROM {table_name} ORDER BY id") + rows = cursor.fetchall() + expected = _expected_stored_vector(value, dtype) + assert [tuple(row) for row in rows] == [(1, None), (2, expected), (3, expected)] + finally: + cursor.execute(f"DROP TABLE {table_name}") + + +@pytest.mark.parametrize("batch_size", [0, 1, 7, 64]) +def test_live_vector_arrow_nulls_slices_and_ownership(live_vector_connection, batch_size): + import gc + + pa = pytest.importorskip("pyarrow") + connection, dtype = live_vector_connection + value = Vector([0.1, 2.5, -0.0], dtype) + expected = list(_expected_stored_vector(value, dtype)) + with connection.cursor() as cursor: + cursor.execute( + "SELECT TOP (71) ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) AS id, " + "CASE WHEN ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) % 2 = 0 " + f"THEN ? ELSE CAST(NULL AS VECTOR(3, {dtype})) END AS v, " + "REPLICATE(CAST(N'x' AS NVARCHAR(MAX)), 9000) AS text_value " + "FROM sys.all_objects a", + value, + ) + schema_only = cursor.arrow_batch(0) + assert schema_only.num_rows == 0 + assert schema_only.schema.field("v").type == pa.list_( + pa.field("item", pa.float32(), nullable=False), 3 + ) + table = cursor.arrow(batch_size or 7) + assert table.num_rows == 71 + assert all(v is None or v == expected for v in table.column("v").to_pylist()) + assert table.column("v").null_count > 0 + assert table.column("text_value").to_pylist() == ["x" * 9000] * 71 + sliced = table.slice(1, 5) + retained = sliced.column("v").chunk(0) + retained_children = retained.values + expected_children = retained_children.to_pylist() + cursor.execute("SELECT 1") + del table, sliced + gc.collect() + assert all(v is None or v == expected for v in retained.to_pylist()) + del retained + gc.collect() + assert retained_children.to_pylist() == expected_children + + +@pytest.mark.parametrize( + "source_kind", ["table", "batch", "reader", "batches", "c_stream", "c_array"] +) +def test_live_vector_bulkcopy_arrow_roundtrip( + live_vector_connection, vector_bulk_core, source_kind +): + pa = pytest.importorskip("pyarrow") + connection, dtype = live_vector_connection + connection.autocommit = True + table_name = f"dbo.pytest_vector_arrow_{uuid.uuid4().hex}" + value = [0.1, -0.0, 1.0001] + with connection.cursor() as cursor: + cursor.execute(f"CREATE TABLE {table_name} (id INT, embedding VECTOR(3, {dtype}))") + try: + field = pa.field( + "v", + pa.list_(pa.float32(), 3), + metadata={b"mssql.vector.base_type": dtype.encode()}, + ) + batch = pa.record_batch( + [ + pa.array([99, 1, 2, 3], type=pa.int32()), + pa.array([value, None, value, value], type=field.type), + ], + schema=pa.schema([pa.field("i", pa.int32()), field]), + ).slice(1) + if source_kind == "table": + source = pa.Table.from_batches([batch.slice(0, 1), batch.slice(1)]) + elif source_kind == "reader": + source = pa.RecordBatchReader.from_batches(batch.schema, [batch]) + elif source_kind == "batches": + source = iter([batch]) + elif source_kind == "c_stream": + + class Stream: + def __arrow_c_stream__(self, requested_schema=None): + return pa.Table.from_batches([batch]).__arrow_c_stream__(requested_schema) + + source = Stream() + elif source_kind == "c_array": + + class Array: + def __arrow_c_array__(self, requested_schema=None): + return batch.__arrow_c_array__(requested_schema) + + source = Array() + else: + source = batch + result = cursor.bulkcopy_arrow( + table_name, + source, + batch_size=2, + column_mappings=[(1, "embedding"), (0, "id")], + ) + assert result["rows_copied"] == 3 + cursor.execute(f"SELECT id, embedding FROM {table_name} ORDER BY id") + expected = _expected_stored_vector(value, dtype) + assert [tuple(row) for row in cursor.fetchall()] == [ + (1, None), + (2, expected), + (3, expected), + ] + copy_name = table_name + "_copy" + cursor.execute(f"CREATE TABLE {copy_name} (id INT, embedding VECTOR(3, {dtype}))") + cursor.execute(f"SELECT id, embedding FROM {table_name} ORDER BY id") + with cursor.arrow_reader(batch_size=2) as reader: + copied = cursor.bulkcopy_arrow( + copy_name, reader, column_mappings=["id", "embedding"] + ) + assert copied["rows_copied"] == 3 + finally: + cursor.execute(f"DROP TABLE IF EXISTS {table_name}_copy") + cursor.execute(f"DROP TABLE {table_name}") + + +@pytest.mark.parametrize( + "method,failure", + [ + ("bulkcopy", "dimensions"), + ("bulkcopy_arrow", "dimensions"), + ("bulkcopy_arrow", "nonfinite"), + ("bulkcopy_arrow", "null_element"), + ("bulkcopy_arrow", "range"), + ], +) +def test_live_vector_bulkcopy_validation(live_vector_connection, vector_bulk_core, method, failure): + pa = pytest.importorskip("pyarrow") + connection, dtype = live_vector_connection + connection.autocommit = True + table_name = f"dbo.pytest_vector_invalid_{uuid.uuid4().hex}" with connection.cursor() as cursor: cursor.execute(f"CREATE TABLE {table_name} (embedding VECTOR(3, {dtype}))") try: - with pytest.raises(mssql_python.NotSupportedError, match="pycore"): - cursor.bulkcopy(table_name, [(Vector([1, 2, 3], dtype),)]) + if method == "bulkcopy": + source = [(Vector([1, 2], dtype),)] + else: + values = { + "dimensions": [1, 2], + "nonfinite": [math.inf, 2, 3], + "null_element": [None, 2, 3], + "range": [65505, 2, 3], + }[failure] + arrow_type = pa.list_(pa.float32(), len(values)) + field = pa.field( + "embedding", + arrow_type, + metadata={b"mssql.vector.base_type": dtype.encode()}, + ) + source = pa.record_batch( + [pa.array([values], type=arrow_type)], schema=pa.schema([field]) + ) + expected_rows = 0 + if failure == "range" and dtype == "float32": + assert getattr(cursor, method)(table_name, source)["rows_copied"] == 1 + expected_rows = 1 + else: + with pytest.raises( + ValueError, match="(?i)vector|dimension|base_type|range|finite|NULL" + ): + getattr(cursor, method)(table_name, source, use_internal_transaction=True) + cursor.execute(f"SELECT COUNT(*) FROM {table_name}") + assert cursor.fetchval() == expected_rows + cursor.execute("SELECT 1") + assert cursor.fetchval() == 1 + finally: + cursor.execute(f"DROP TABLE {table_name}") + + +@pytest.mark.parametrize("source", ["vectors", "arrow", "json"]) +def test_live_vector_bulkcopy_server_converts_base_types( + live_vector_connection, vector_bulk_core, source +): + """Like SqlBulkCopy: declare the source base type and let SQL Server convert.""" + pa = pytest.importorskip("pyarrow") + connection, dtype = live_vector_connection + if os.getenv("MSSQL_TEST_VECTOR_FLOAT16") != "1": + pytest.skip("Cross-base conversion needs a FLOAT16-capable server") + if source == "json" and dtype != "float16": + pytest.skip("JSON text is sent as text only for FLOAT16 destinations") + other = "float32" if dtype == "float16" else "float16" + connection.autocommit = True + table_name = f"dbo.pytest_vector_cross_{uuid.uuid4().hex}" + values = [0.1, 2.0, 1e-8] + # One side is always FLOAT16, so stored values carry FLOAT16 rounding. + expected = Vector(_expected_stored_vector(values, "float16").values, dtype) + if source == "vectors": + data = [(Vector(values, other),), (None,), (Vector(values, other),)] + elif source == "json": + data = [("[0.1, 2, 1e-8]",), (None,), ("[0.1,2,0.00000001]",)] + else: + field = pa.field( + "embedding", + pa.list_(pa.float32(), 3), + metadata={b"mssql.vector.base_type": other.encode()}, + ) + data = pa.Table.from_arrays( + [pa.array([values, None, values], type=field.type)], schema=pa.schema([field]) + ) + method = "bulkcopy_arrow" if source == "arrow" else "bulkcopy" + with connection.cursor() as cursor: + cursor.execute(f"CREATE TABLE {table_name} (id INT IDENTITY, embedding VECTOR(3, {dtype}))") + try: + assert ( + getattr(cursor, method)(table_name, data, column_mappings=[(0, "embedding")])[ + "rows_copied" + ] + == 3 + ) + cursor.execute(f"SELECT embedding FROM {table_name} ORDER BY id") + rows = [row[0] for row in cursor.fetchall()] + assert rows == [expected, None, expected] + assert rows[0].base_type == dtype + finally: + cursor.execute(f"DROP TABLE {table_name}") + + +def test_live_vector_bulkcopy_cross_base_errors(live_vector_connection, vector_bulk_core): + connection, dtype = live_vector_connection + if dtype != "float16": + pytest.skip("FLOAT16 destination cases") + connection.autocommit = True + table_name = f"dbo.pytest_vector_cross_err_{uuid.uuid4().hex}" + with connection.cursor() as cursor: + cursor.execute(f"CREATE TABLE {table_name} (embedding VECTOR(3, float16))") + try: + # SQL Server rejects FLOAT32 values outside the FLOAT16 range. + with pytest.raises(Exception, match="out-of-range"): + cursor.bulkcopy( + table_name, [(Vector([70000.0, 0, 0]),)], use_internal_transaction=True + ) + # The first row's Vector declares the column's base type for the load. + for rows in ( + [(Vector([1, 2, 3]),), (Vector([1, 2, 3], "float16"),)], + [(None,), (Vector([1, 2, 3]),)], + ): + with pytest.raises(ValueError, match="(?i)base.?type"): + cursor.bulkcopy(table_name, rows, use_internal_transaction=True) cursor.execute(f"SELECT COUNT(*) FROM {table_name}") - assert cursor.fetchone()[0] == 0 + assert cursor.fetchval() == 0 + finally: + cursor.execute(f"DROP TABLE {table_name}") + + +@pytest.mark.parametrize("method", ["bulkcopy", "bulkcopy_arrow"]) +def test_live_vector_bulkcopy_maximum_dimensions(live_vector_connection, vector_bulk_core, method): + pa = pytest.importorskip("pyarrow") + connection, dtype = live_vector_connection + connection.autocommit = True + dimensions = 3996 if dtype == "float16" else 1998 + table_name = f"dbo.pytest_vector_max_bulk_{uuid.uuid4().hex}" + value = Vector([0.125] * dimensions, dtype) + with connection.cursor() as cursor: + cursor.execute(f"CREATE TABLE {table_name} (embedding VECTOR({dimensions}, {dtype}))") + try: + if method == "bulkcopy": + source = [(None,), (value,)] + else: + source = pa.table( + { + "embedding": pa.array( + [None, list(value)], type=pa.list_(pa.float32(), dimensions) + ) + } + ) + assert getattr(cursor, method)(table_name, source)["rows_copied"] == 2 + cursor.execute(f"SELECT embedding FROM {table_name} WHERE embedding IS NOT NULL") + table = cursor.arrow() + assert table.schema.field("embedding").type == pa.list_( + pa.field("item", pa.float32(), nullable=False), dimensions + ) + assert table.column(0).to_pylist() == [list(value)] + finally: + cursor.execute(f"DROP TABLE {table_name}") + + +@pytest.mark.parametrize("method", ["bulkcopy", "bulkcopy_arrow"]) +@pytest.mark.parametrize("empty", [False, True]) +def test_live_vector_bulkcopy_empty_and_all_null( + live_vector_connection, vector_bulk_core, method, empty +): + pa = pytest.importorskip("pyarrow") + connection, dtype = live_vector_connection + connection.autocommit = True + table_name = f"dbo.pytest_vector_null_bulk_{uuid.uuid4().hex}" + count = 0 if empty else 2 + with connection.cursor() as cursor: + cursor.execute(f"CREATE TABLE {table_name} (embedding VECTOR(3, {dtype}))") + try: + if method == "bulkcopy": + source = [(None,)] * count + else: + # Hidden child values under NULL parents must not be validated as data. + values = pa.array([math.nan] * (3 * count), type=pa.float32()) + vectors = pa.FixedSizeListArray.from_arrays( + values, 3, mask=pa.array([True] * count, type=pa.bool_()) + ) + source = pa.table({"embedding": vectors}) + assert getattr(cursor, method)(table_name, source)["rows_copied"] == count + cursor.execute(f"SELECT embedding FROM {table_name}") + table = cursor.arrow() + assert table.num_rows == count + assert table.column(0).to_pylist() == [None] * count finally: cursor.execute(f"DROP TABLE {table_name}")