GH-184: Export Arrow arrays through the C Data Interface#605
Draft
samtalki wants to merge 4 commits into
Draft
Conversation
samtalki
force-pushed
the
agent/cdata-export
branch
2 times, most recently
from
26667a7 to
775bc7d
Compare
samtalki
force-pushed
the
agent/cdata-export
branch
from
775bc7d to
c8d7f55
Compare
Add immutable ArrowSchema and ArrowArray ABI structs plus C Data import for null and primitive arrays. Validate pointer layout, child counts, buffer counts, offsets, known flag bits, dictionary pointer consistency, release idempotency, and reads after release. Trust declared null counts like Arrow C++ and arrow-rs, require a validity bitmap for unknown null counts like nanoarrow, and ignore reserved flag bits for forward compatibility. Move the base ArrowSchema and ArrowArray structures into Julia owned storage at import and mark the sources released without calling their callbacks, following the C Data Interface move semantics used by arrow-rs from_raw and nanoarrow ArrowArrayMove. Callers may free or reuse the passed structures once from_c_data returns; release_c_data or finalization releases the moved copies through the producer callbacks. Keep imported buffers rooted behind a shared release owner. Copy misaligned fixed width buffers into aligned Julia storage before typed access, and materialize Julia owned data from copy, collect, deepcopy, and serialize so no read path bypasses the release liveness gate. Co-authored-by: Robert Buessow <robert.buessow@relational.ai> Co-authored-by: Olle Martensson <olle.martensson@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Generated-by: OpenAI Codex
Add C Data import for struct arrays on top of the null and primitive array importer. Root structs are exposed as Tables.jl column tables with DataAPI metadata; nested structs import as NamedTuple element vectors with parent validity masking and parent offsets applied to the children. Parse field names and schema metadata from the C schema for the imported table. Validate child pointer layout, child counts against the schema, struct child lengths, name and metadata bounds, and nesting depth, bound validation with a total node budget so aliased or cyclic child pointers cannot make it explode combinatorially, and reject duplicate struct field names, which NamedTuple rows and column lookup cannot represent. Adjust dictionary encoding offsets for non 1-based reference pools so categorical pools with missing values keep correct dictionary indices. Co-authored-by: Robert Buessow <robert.buessow@relational.ai> Co-authored-by: Olle Martensson <olle.martensson@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Generated-by: OpenAI Codex
Add import support for boolean, string, binary, list, fixed size, temporal, timestamp, duration, interval, and decimal C Data formats. Extend malformed input, offset, null count, release, copy, collect, metadata, and GC rooting coverage for the broader importer. Co-authored-by: Robert Buessow <robert.buessow@relational.ai> Co-authored-by: Olle Martensson <olle.martensson@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Generated-by: OpenAI Codex
Add array and table export entry points with independent schema and array owners. Release children that were already exported when a later child fails to build, so partially constructed exports do not leak owner registry entries. Cover primitive and struct round trips, release order, moved child ownership, GC, metadata, malformed layout, unsupported export cases, and exported buffer layout checks for nested, string, and fixed size arrays. Co-authored-by: Robert Buessow <robert.buessow@relational.ai> Co-authored-by: Olle Martensson <olle.martensson@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Generated-by: OpenAI Codex
samtalki
force-pushed
the
agent/cdata-export
branch
from
c8d7f55 to
3304de3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR builds on #604 and adds Arrow C Data Interface export.
Arrow.to_c_data(col::ArrowVector; name="").Arrow.to_c_data(tbl; names=String.(Tables.columnnames(Tables.columns(tbl)))).Stack
Base remains
mainbecause these cross fork PRs are not represented as true stacked bases on GitHub.Depends on #604, currently
e84c39d.Incremental review link:
samtalki/arrow-julia@agent/cdata-import-breadth-v2...agent/cdata-export
Size
Incremental over #604: 3 files changed, 1111 insertions, 2 deletions.
Total against
main: 5 files changed, 3770 insertions, 4 deletions.Validation
git diff --check agent/cdata-import-breadth-v2...HEAD: passed.AI Assistance
Co-authored-by: Robert Buessow robert.buessow@relational.ai
Co-authored-by: Olle Martensson olle.martensson@gmail.com
Generated-by: OpenAI Codex