GH-184: Add Arrow C Data Interface import for null and primitive arrays by samtalki · Pull Request #607 · apache/arrow-julia

samtalki · 2026-07-12T20:02:05Z

Summary

First PR of the C Data Interface stack for GH-184, extracting null and primitive array import from #603 as requested there.

Immutable ArrowSchema and ArrowArray C layout structs with layout assertions.
Arrow.from_c_data(schema_ptr, array_ptr; convert=true) for null and primitive arrays.
Validates pointer layout, child and buffer counts, offset arithmetic, known flag bits, dictionary pointer consistency, release idempotency, and reads after release. Declared null counts are trusted like Arrow C++ and arrow-rs; null_count == -1 requires the validity bitmap it is resolved from, like nanoarrow; reserved flag bits are ignored.
Import moves the base ArrowSchema and ArrowArray structures into Julia owned storage and marks the sources released without calling their callbacks; callers may free or reuse the passed structures once from_c_data returns.
Aligned buffers are viewed zero copy behind a shared release owner; misaligned fixed width buffers are copied into aligned storage; copy, collect, deepcopy, and serialize materialize Julia owned data through the release liveness gate.

Struct, boolean, string, binary, list, and export support land in the follow up PRs.

Stack

Base is main.

Follow up PRs:

GH-184: Add Arrow C Data Interface import foundation #603 adds struct array import as Tables.jl column tables.
GH-184: Expand Arrow C Data Interface import coverage #604 expands import coverage.
GH-184: Export Arrow arrays through the C Data Interface #605 adds C Data export.
GH-184: Harden Arrow C Data Interface validation #606 adds hardening and smoke tests.

Size

Against main: 6 files changed, 1246 insertions, 1 deletion.

Validation

git diff --check main...HEAD: passed.
ABI smoke: ArrowSchema and ArrowArray are isbits structs with expected C layout sizes.
Focused C Data tests: 93/93 passed.
Full package tests (bounds checked): 66278/66278 passed.

AI Assistance

Co-authored-by: Robert Buessow robert.buessow@relational.ai
Co-authored-by: Olle Martensson olle.martensson@gmail.com
Generated-by: OpenAI Codex

Add immutable ArrowSchema and ArrowArray ABI structs plus C Data import for null and primitive arrays. Validate pointer layout, child counts, buffer counts, offsets, known flag bits, dictionary pointer consistency, release idempotency, and reads after release. Trust declared null counts like Arrow C++ and arrow-rs, require a validity bitmap for unknown null counts like nanoarrow, and ignore reserved flag bits for forward compatibility. Move the base ArrowSchema and ArrowArray structures into Julia owned storage at import and mark the sources released without calling their callbacks, following the C Data Interface move semantics used by arrow-rs from_raw and nanoarrow ArrowArrayMove. Callers may free or reuse the passed structures once from_c_data returns; release_c_data or finalization releases the moved copies through the producer callbacks. Keep imported buffers rooted behind a shared release owner. Copy misaligned fixed width buffers into aligned Julia storage before typed access, and materialize Julia owned data from copy, collect, deepcopy, and serialize so no read path bypasses the release liveness gate. Co-authored-by: Robert Buessow <robert.buessow@relational.ai> Co-authored-by: Olle Martensson <olle.martensson@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Generated-by: OpenAI Codex

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

codecov-commenter · 2026-07-12T20:26:53Z

Codecov Report

❌ Patch coverage is 97.52650% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.16%. Comparing base (3712291) to head (23de5c2).
⚠️ Report is 54 commits behind head on main.

Files with missing lines	Patch %	Lines
src/cdata.jl	97.52%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #607      +/-   ##
==========================================
+ Coverage   87.43%   88.16%   +0.72%     
==========================================
  Files          26       28       +2     
  Lines        3288     3666     +378     
==========================================
+ Hits         2875     3232     +357     
- Misses        413      434      +21

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

kou · 2026-07-12T21:09:05Z

+# On 32 bit ABIs with 8 byte Int64 alignment the C structs contain padding, so
+# the packed size formulas hold only where pointer and Int64 sizes agree.
+@static if Sys.WORD_SIZE == 64
+    @assert sizeof(ArrowSchema) == 7 * _CDATA_PTR_SIZE + 2 * sizeof(Int64)
+    @assert sizeof(ArrowArray) == 5 * _CDATA_PTR_SIZE + 5 * sizeof(Int64)


Does this mean that sizeof(ArrowSchema)/sizeof(ArrowArray) may be different on 32 bit environment?

kou · 2026-07-12T21:14:32Z

+const ARROW_FLAG_NULLABLE = Int64(2)
+const ARROW_FLAG_MAP_KEYS_SORTED = Int64(4)
+
+const _CDATA_MAX_FORMAT_BYTES = 4096


I don't object this for safety but is 4096 reasonable limit?

kou · 2026-07-12T21:20:54Z

+            unlock(owner.lock)
+        end
+    else
+        finalizer(_finalize_c_data, owner)


I'm not familiar with Julia but why can we retry the finalizer? If finalizer() is called on owner, is finalization of the owner cancelled?

kou · 2026-07-12T21:31:27Z

+function _count_nulls(bytes::Vector{UInt8}, bitoffset::Int, len::Int)
+    len == 0 && return 0
+    firstbit = bitoffset
+    lastbit = bitoffset + len - 1
+    firstbyte = firstbit >>> 3
+    lastbyte = lastbit >>> 3
+    firstmask = 0xff << (firstbit & 7)
+    lastmask = 0xff >>> (7 - (lastbit & 7))
+    set = 0
+    if firstbyte == lastbyte
+        set = count_ones(@inbounds(bytes[firstbyte + 1]) & firstmask & lastmask)
+    else
+        set = count_ones(@inbounds(bytes[firstbyte + 1]) & firstmask)
+        @inbounds for b = (firstbyte + 2):lastbyte
+            set += count_ones(bytes[b])
+        end
+        set += count_ones(@inbounds(bytes[lastbyte + 1]) & lastmask)
+    end
+    return len - set
+end


Do we need to implement this? (Do we have any existing feature for this?)

kou · 2026-07-12T21:40:15Z

+@propagate_inbounds function Base.getindex(x::CDataNull, i::Integer)
+    return _with_live(x) do
+        @boundscheck checkbounds(x, i)
+        return missing
+    end
+end
+
+@propagate_inbounds function Base.getindex(x::CDataPrimitive{T}, i::Integer) where {T}
+    return _with_live(x) do
+        @boundscheck checkbounds(x, i)
+        if !_valid(x.validity, i)
+            return missing
+        end
+        return @inbounds ArrowTypes.fromarrow(T, x.data[i])
+    end
+end
+
+function Base.collect(x::CDataVector{T}) where {T}
+    return _with_live(x) do
+        out = Vector{T}(undef, length(x))
+        for i in eachindex(x)
+            @inbounds out[i] = x[i]
+        end
+        return out
+    end
+end


Why do we need to implement them?

Can users access the the Arrow data after they import the Arrow data?

kou · 2026-07-13T02:42:43Z

+    unsafe_store!(
+        ptr,
+        ArrowArray(
+            array.length,
+            array.null_count,
+            array.offset,
+            array.n_buffers,
+            array.n_children,
+            array.buffers,
+            array.children,
+            array.dictionary,
+            C_NULL,
+            array.private_data,
+        ),
+    )


I'm not familiar with Julia but we can't update only array.release with Julia API, right?

kou · 2026-07-13T02:48:33Z

+function _load_buffers(array::ArrowArray, expected::Int)
+    n_buffers = _checked_nonnegative(array.n_buffers, "ArrowArray.n_buffers")
+    n_buffers == expected ||
+        throw(ArgumentError("ArrowArray.n_buffers does not match the format"))


How about adding the actual and expected N buffers to the error messages for easy to debug?

kou · 2026-07-13T02:53:16Z

+    schema_children = _checked_nonnegative(schema.n_children, "ArrowSchema.n_children")
+    schema_children == n_children ||


Suggested change

schema_children = _checked_nonnegative(schema.n_children, "ArrowSchema.n_children")

schema_children == n_children ||

schema_n_children = _checked_nonnegative(schema.n_children, "ArrowSchema.n_children")

schema_n_children == n_children ||

kou · 2026-07-13T02:53:33Z

+    n_children = _checked_nonnegative(array.n_children, "ArrowArray.n_children")
+    schema_children = _checked_nonnegative(schema.n_children, "ArrowSchema.n_children")
+    schema_children == n_children ||
+        throw(ArgumentError("ArrowSchema and ArrowArray child counts differ"))


Can we show actual/expected values in the error message?

kou · 2026-07-13T02:55:07Z

+        throw(ArgumentError("ArrowArray.null_count is out of range"))
+    end
+    if (schema.dictionary == C_NULL) != (array.dictionary == C_NULL)
+        throw(ArgumentError("ArrowSchema and ArrowArray dictionary pointers differ"))


Is this error message correct?
It seems that this checks only whether dictionary is NULL or not.

kou requested a review from Copilot July 12, 2026 20:25

Copilot AI reviewed Jul 12, 2026

samtalki mentioned this pull request Jul 12, 2026

GH-184: Add Arrow C Data Interface import foundation #603

Open

kou reviewed Jul 13, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

GH-184: Add Arrow C Data Interface import for null and primitive arrays#607

GH-184: Add Arrow C Data Interface import for null and primitive arrays#607
samtalki wants to merge 1 commit into
apache:mainfrom
samtalki:agent/cdata-import-primitive

samtalki commented Jul 12, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

codecov-commenter commented Jul 12, 2026 •

edited

Loading

Uh oh!

kou Jul 12, 2026

Uh oh!

kou Jul 12, 2026

Uh oh!

kou Jul 12, 2026

Uh oh!

kou Jul 12, 2026

Uh oh!

kou Jul 12, 2026

Uh oh!

kou Jul 13, 2026

Uh oh!

kou Jul 13, 2026

Uh oh!

kou Jul 13, 2026

Uh oh!

kou Jul 13, 2026

Uh oh!

kou Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		schema_children = _checked_nonnegative(schema.n_children, "ArrowSchema.n_children")
		schema_children == n_children \|\|

Uh oh!

Conversation

samtalki commented Jul 12, 2026

Summary

Stack

Size

Validation

AI Assistance

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Jul 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov-commenter commented Jul 12, 2026 •

edited

Loading