Repository navigation
FEAT: Support FLOAT32/FLOAT16 vectors in bulk copy and Arrow - #903
Draft
gargsaumya wants to merge 4 commits into
Draft
gargsaumya wants to merge 4 commits into
gargsaumya wants to merge 4 commits into
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
3 of 5 tasks
Gaurav Sharma (bewithgaurav)
left a comment
Collaborator
There was a problem hiding this comment.
bulk copy and arrow both round trip vectors correctly for me, including sliced tables and reader to bulk copy, and plain bulk copy throughput is unchanged. this still needs the latest native vector changes merged in and the vector-capable runtime published before it leaves draft. commented, no blockers.
Pick up the reviewed #902 changes (TVP removal, OutputParameter triple, trusted vector construction, exact Vector type checks, SQL Server 2025 FLOAT16 preview docs). Align vector bulk copy and Arrow docs/tests with Microsoft.Data.SqlClient: the client never converts between base types; columns are declared with their source base type (JSON text for FLOAT16 columns as text) and SQL Server converts them. Replace base-type mismatch rejection tests with live conversion, rounding, out-of-range and mixed-column tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Arrow fixed-size float32 lists represent both base types, so untagged columns convert to the destination like numeric lists; only mssql.vector.base_type-tagged columns declare their own base type for server-side conversion. This keeps FLOAT16 columns above 1998 dimensions loadable from untagged Arrow data. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Work Item / Issue Reference
Summary
Add FLOAT32/FLOAT16 row bulk-copy adaptation and native Arrow vector reads/writes.
Vector; Arrowmssql.vector.base_typemetadata) and SQL Server converts it (round-to-nearest; out-of-range FLOAT16 values fail on the server). JSON text for FLOAT16 columns is sent as text. Lists and untagged Arrow data use the destination base type.mssql.vector.base_typefield metadata.callproc()results remain row-fetch only.API examples: bulk copy and both Arrow flows
These examples assume a cursor connected with
VectorTypeSupport="v2", ODBC 18.7.1.1+, the companion vector-capable pycore runtime, and existingdbo.VectorData/dbo.VectorDataCopytables with one nullableVECTOR(3, FLOAT16)column namedvalue. FLOAT32 uses the same APIs with a matching base type/schema.Row bulk copy: pass the same public
Vectorobjects inside rows.SQL -> Arrow fetching, then Arrow -> SQL bulk loading:
arrow_batch()and streamingarrow_reader()also support vectors; a reader can be passed directly tobulkcopy_arrow(). Both SQL base types use Arrow fixed-size lists of float32, withmssql.vector.base_typefield metadata preservingfloat32versusfloat16. Parent NULLs are supported; vector elements cannot be NULL.Validation
Dependencies / scope