fix(numpy): accept PYBIND11_TYPE-wrapped types in the dtype macros

[advitrocks9]

Description

Fixes #4018.

PYBIND11_TYPE is the documented way to pass a type with a comma in its spelling to a macro,
but it doesn't work with the dtype macros:

template <typename T1, typename T2>
struct TestStruct {
    T1 a;
    T2 b;
};

PYBIND11_MODULE(repro, m) {
    PYBIND11_NUMPY_DTYPE(PYBIND11_TYPE(TestStruct<float, double>), a, b);  // error: expected ','
}

The only workaround was a using alias.

This happens because PYBIND11_TYPE is itself a macro. PYBIND11_NUMPY_DTYPE builds its field
list through PYBIND11_MAP_LIST, which rescans its output repeatedly, and the first rescan
expands the wrapper away. After that the bare comma splits the type in two at the next argument
collection. Re-wrapping at every step, the way PYBIND11_OVERRIDE does, gives one correct field
and then a field named "double>".

Plain parentheses survive rescanning, so the dtype macros now pass (Type) through the map
machinery, and a new PYBIND11_FIELD_DESCRIPTOR_EX_IMPL removes the parentheses where the type
is actually used (#define PYBIND11_UNPAREN_TYPE(T) PYBIND11_TYPE T). The public
PYBIND11_FIELD_DESCRIPTOR and PYBIND11_FIELD_DESCRIPTOR_EX still take a bare type and add
the parentheses themselves before forwarding to the _IMPL macro, so projects that call them
directly are unaffected, and they now accept a PYBIND11_TYPE-wrapped type as well. Existing
call sites preprocess to the same tokens as before.

The tests register a two-parameter struct through both dtype macros and check the resulting
dtypes, with two different field types in the _EX case so each renamed field is checked
against its own type. A second test calls the field descriptor macros directly, with bare and
wrapped types.

Suggested changelog entry:

• PYBIND11_NUMPY_DTYPE and PYBIND11_NUMPY_DTYPE_EX now accept a type whose spelling contains
  a comma, when it is wrapped in PYBIND11_TYPE. Previously the wrapper was silently dropped
  during expansion and only a using alias worked.

---

📚 Documentation preview 📚: https://pybind11--6148.org.readthedocs.build/

[rwgk]

Hi @advitrocks9 , below is a codex gpt-5.6-sol ultra review (I'm posting it as generated).

It would be very easy for me to ask codex to implement the suggested fix. Please let me know if I should do that and add the commit(s) here.

---

Summary

This is a focused preprocessor fix for #4018. PYBIND11_NUMPY_DTYPE registers a C++ struct as a NumPy structured dtype, but a specialization such as TemplatedStruct<int, float> currently cannot be passed directly because the preprocessor mistakes its comma for a macro-argument separator.

Although pybind11 documents PYBIND11_TYPE(...) for shielding comma-containing types, recursive expansion inside the dtype macros removes that wrapper too early. Users therefore need a using alias as a workaround.

This PR fixes that by:

• transporting the type through the recursive field mapping inside ordinary parentheses, which survive macro rescanning;
• removing those parentheses only when constructing each field descriptor;
• testing both PYBIND11_NUMPY_DTYPE and PYBIND11_NUMPY_DTYPE_EX; and
• documenting the PYBIND11_TYPE spelling.

This is entirely a preprocessor fix. It does not change NumPy conversion behavior, runtime representation, internals layout, or ABI. The central design looks sound.

Important finding: source-compatibility regression

I think one change is needed before merging.

The new implementation makes direct calls to PYBIND11_FIELD_DESCRIPTOR and PYBIND11_FIELD_DESCRIPTOR_EX require an already-parenthesized type. For example, this previously valid code now fails:

PYBIND11_FIELD_DESCRIPTOR(MyStruct, field)

At numpy.h:1803, that expands into the invalid tokens PYBIND11_TYPE MyStruct.

Although the PR describes these as undocumented expansion internals, direct use is real: Rosetta uses PYBIND11_FIELD_DESCRIPTOR to assemble descriptor lists, as does bbstrader.

This can be fixed without changing the main design:

• Put the new parenthesized-type implementation in a separate _IMPL macro.
• Keep the existing leaf macros as wrappers that add parentheses around their bare T.
• Use a map-specific adapter when the type is already parenthesized.
• Add compile coverage for direct calls to both existing leaf macros.

I tested that structure with GCC 13 and Clang 18; it supports legacy direct calls, wrapped template types, nested templates, and both recursive mapping variants.

Tests and other observations

The new tests cover the actual feature well: both public dtype macros, multiple fields, and the resulting dtype values. Existing ordinary dtype registrations remain heavily exercised.

One optional strengthening: the _EX test currently uses int16_t for both fields. Using two distinguishable types, such as int16_t and uint16_t, would independently verify that each renamed field retains the correct type.

Current CI has no reported failures and includes successful Windows/MSVC, macOS/Clang, and Ubuntu builds. Because the PR is still draft and several jobs were skipped or cancelled, the full matrix should still run before merging.

Overall: this is a good, narrowly designed fix. I would be comfortable approving it once the direct field-descriptor calling convention is preserved.

[advitrocks9]

Thanks, the compat finding is right, direct callers like Rosetta would break. Fixed in
1fee84e, it was quick enough to do by hand. The parenthesized-type body now lives in an
_IMPL macro that the map expansions call, and the public macros take a bare type again and
add the parens themselves. As a side effect direct calls now also work with
PYBIND11_TYPE-wrapped types, which they didn't before this PR either.

Also took the test suggestion, the _EX case now uses int16_t/uint16_t. There's a new
test_direct_field_descriptor that calls the macros directly both ways, and with the fix
reverted it fails to compile with unknown type name 'PYBIND11_TYPE', so it pins this exact
regression.

I'll mark the PR ready so the rest of CI runs.

[rwgk]

Awesome, thanks; approving. codex is happy now, except:

The only actionable item is the PR description, which is now stale. It still says:

• direct leaf-macro calls require a parenthesized type and may break downstream users; and
• PYBIND11_FIELD_DESCRIPTOR_EX performs the unwrapping.

Those statements describe the previous head. The description should instead say that the public leaf macros retain their bare-type interface, while PYBIND11_FIELD_DESCRIPTOR_EX_IMPL receives and unwraps the protected type. Its test paragraph should also mention the new direct-call regression coverage.

[advitrocks9]

Updated the description to match the current version.