Skip to content

[cpyrt] Parse buffer formats properly instead of character search - #140

Merged
guitargeek merged 1 commit into
compiler-research:mainfrom
guitargeek:buffer-format-compat
Oct 19, 2026
Merged

guitargeek merged 1 commit into
compiler-research:mainfrom
guitargeek:buffer-format-compat

Conversation

@guitargeek

Copy link
Copy Markdown
Collaborator

numpy and other buffer producers canonicalize the PEP 3118 format char of 64-bit integer dtypes (e.g. int64 is always reported as 'l' on 64-bit platforms), but C++ parameters declared as int64_t resolve to 'long long' on macOS and 'long' on Linux. The exact format-char matching in the array converters therefore rejected valid buffers when the resolved type and the canonicalized buffer format differed, breaking e.g. passing numpy int64 arrays to 'const int64_t*' parameters on macOS.

Replace the ad-hoc buffer-format matching with a small PEP 3118 parser in Utility::BufferFormatCompatible(). Formats are reduced to their element type (skipping byte-order prefixes like ctypes' '<q', repeat counts, and the 'Z' complex marker), classified by kind (signed/unsigned integer, float, complex, ...) and native size, and accepted when both match the requested type code. Use it where buffer formats are matched: Utility::GetBuffer() (standard buffer protocol), CArraySetArg() (low-level views), the 1- and 2-dimensional branches of the array converter, and StdSpanConverter (std::span parameters).

This subsumes all previously scattered special cases:

  • int/long equivalence on Windows and 32-bit Linux in GetBuffer()
  • long/long long interchange for numpy's canonicalized int64 format
  • 'z'/'Zf' complex-float naming
  • signed char ('b') as bool, kept as an explicit documented quirk

As side effects, ctypes' explicitly-endian '<q'/'<d' buffers are now accepted natively, and complex buffers no longer leak into float parameters through the 'Zd'-contains-'d' substring accident (std::span, which performs no item-size check, previously read complex128 buffers as double).

@guitargeek
guitargeek force-pushed the buffer-format-compat branch 3 times, most recently from 06c6385 to eae190d Compare October 7, 2026 15:57
aaronj0
aaronj0 previously approved these changes Oct 19, 2026

@aaronj0 aaronj0 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! This is a nice improvement over the previous heuristic approach and now supports all formats under the PEP 3118 as shown in the test. Can you fix the merge conflicts?

numpy and other buffer producers canonicalize the PEP 3118 format char
of 64-bit integer dtypes (e.g. int64 is always reported as 'l' on 64-bit
platforms), but C++ parameters declared as int64_t resolve to 'long long'
on macOS and 'long' on Linux. The exact format-char matching in the
array converters therefore rejected valid buffers when the resolved type
and the canonicalized buffer format differed, breaking e.g. passing
numpy int64 arrays to 'const int64_t*' parameters on macOS.

Replace the ad-hoc buffer-format matching with a small PEP 3118 parser
in Utility::BufferFormatCompatible(). Formats are reduced to their
element type (skipping byte-order prefixes like ctypes' '<q', repeat
counts, and the 'Z' complex marker), classified by kind
(signed/unsigned integer, float, complex, ...) and native size, and
accepted when both match the requested type code. Use it where buffer
formats are matched: Utility::GetBuffer() (standard buffer protocol),
CArraySetArg() (low-level views), the 1- and 2-dimensional branches of
the array converter, and StdSpanConverter (std::span parameters).

This subsumes all previously scattered special cases:

- int/long equivalence on Windows and 32-bit Linux in GetBuffer()
- long/long long interchange for numpy's canonicalized int64 format
- 'z'/'Zf' complex-float naming
- signed char ('b') as bool, kept as an explicit documented quirk

As side effects, ctypes' explicitly-endian '<q'/'<d' buffers are now
accepted natively, and complex buffers no longer leak into float
parameters through the 'Zd'-contains-'d' substring accident (std::span,
which performs no item-size check, previously read complex128 buffers
as double).
@guitargeek

Copy link
Copy Markdown
Collaborator Author

Merge conflicts fixed! Could you re-review?

@guitargeek
guitargeek merged commit 871ff34 into compiler-research:main Oct 19, 2026
24 checks passed
@guitargeek
guitargeek deleted the buffer-format-compat branch October 19, 2026 17:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants