JIT: vpshufbitqmb is missing INS_FLAGS_IsDstDstSrcAVXInstruction

[jamesburton]

Description

VPSHUFBITQMB has the form k1 {k2}, xmm2, xmm3/m128 — a kmask destination plus two source operands. Its row in instrsxarch.h is missing INS_FLAGS_IsDstDstSrcAVXInstruction, which every other three-operand mask-producing EVEX instruction in the table carries.

https://github.com/dotnet/runtime/blob/main/src/coreclr/jit/instrsxarch.h#L991

INST3(vpshufbitqmb, "vpshufbitqmb", IUM_WR, BAD_CODE, BAD_CODE, SSE38(0x8F), 6C, 1C, INS_TT_FULL_MEM,
      Input_8Bit | KMask_Base16 | REX_W0 | Encoding_EVEX)

Compare its siblings, all of which have the flag:

instruction	form	IsDstDstSrcAVX
vptestmb/d/q/w	k1 {k2}, xmm2, xmm3/m	✔
vptestnmb/d/q/w	k1 {k2}, xmm2, xmm3/m	✔
vpcmpb, vpcmpub, vpcmpw, vpcmpuw	k1 {k2}, xmm2, xmm3/m, imm8	✔
vpblendmb	xmm1 {k1}, xmm2, xmm3/m	✔
vpshufbitqmb	k1 {k2}, xmm2, xmm3/m	✘

Impact

IsThreeOperandAVXInstruction is a pure lookup of INS_FLAGS_Is3OperandInstructionMask, so without the flag it returns false for this instruction. emitIns_R_R_S, emitIns_R_R_R_I and emitIns_R_R_S_I all assert on that predicate:

Assertion failed 'IsThreeOperandAVXInstruction(ins) || IsApxExtendedEvexInstruction(ins)'
    File: src\coreclr\jit\emitxarch.cpp:8572

No shipping code is affected today. No HARDWARE_INTRINSIC entry maps to INS_vpshufbitqmb, so the JIT can never currently be asked to emit it — the row has been dormant since it was added. This is a latent defect that will bite the first consumer, not a user-visible bug.

How it was found

While implementing the BITALG intrinsics for #96162. Avx512BitAlg.ShuffleBits is the first consumer of this row, and hit the assert immediately on a Checked JIT.

Note on vpcmpd / vpcmpq / vpcmpud / vpcmpuq

These four also lack the flag while their b/w/ub/uw siblings have it, and unlike vpshufbitqmb they are reachable. I checked this empirically — Vector512.LessThanOrEqual<int>, GreaterThan<uint>, LessThan<ulong> etc. with both register and memory operands, on a Checked JIT — and they do not reach any of the asserting emit paths, so this appears to be a harmless inconsistency rather than a second bug. Flagging it in case it should be made consistent anyway.

Fix

One line — add INS_FLAGS_IsDstDstSrcAVXInstruction to the vpshufbitqmb row.

Verified on Zen 5 (AMD Strix Halo, which has BITALG): with the flag added, the full JIT/HardwareIntrinsics/X86_Avx512 suite passes 1775/1775 on a Checked JIT, including new ShuffleBits coverage.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.