[release/11.0] Fix scalar-typed SIMD operands in Dot and mask inversion lowering

[github-actions]

Backport of #133709 to release/11.0

/cc @tannergooding

Customer Impact

• Customer reported
• Found internally

This fixes #133551, where Vector128.Dot or Vector256.Dot with a CreateScalarUnsafe operand leaves an invalid scalar/vector shape during lowering and can fail JIT compilation. The repro fails on .NET 11 RC2 but behaves correctly on .NET 8, 9, and 10. The same lowering-shape fix also addresses the Checked-JIT assertion reported in #133554 for a Vector512 conditional select.

Regression

• Yes
• No

The #133551 failure was introduced during the .NET 11 development cycle; the exact introducing change was not identified. #133554 is included because it exercises the same scalar-typed SIMD representation invariant.

Testing

The source PR added focused float and double Dot regressions at 128 and 256 bits in both operand orders, scalar multiply/divide controls, and 512-bit conditional-select cases covering register and byref inputs, equal and unequal values, and NaNs. All eight focused cases passed with native ISA support, AVX-512 disabled, AVX disabled, hardware intrinsics disabled, and register stress. Fourteen captured leaf disassemblies confirmed the intended packed operations, scalar-width loads, and masked zeroing without changing the scalar controls.

Risk

Low. The product diff is small and corrects established scalar/SIMD representation invariants in two downstream lowering paths without changing the shared builder semantics. Focused coverage exercises operand orders, vector widths, ISA fallbacks, memory and register forms, NaNs, and register stress, with before/after disassembly confirming the intended codegen.

IMPORTANT: If this backport is for a servicing release, please verify that:

• For .NET 8 and .NET 9: The PR target branch is release/X.0-staging, not release/X.0.
• For .NET 10+: The PR target branch is release/X.0 (no -staging suffix).

Package authoring no longer needed in .NET 9

IMPORTANT: Starting with .NET 9, you no longer need to edit a NuGet package's csproj to enable building and bump the version.
Keep in mind that we still need package authoring in .NET 8 and older versions.

Note

This backport template was completed by GitHub Copilot at the user's direction.

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[JulieLeeMSFT]

Approved. Please check test failures once CI is done.