diff --git a/.cargo/config-avx.toml b/.cargo/config-avx.toml new file mode 100644 index 00000000..60e3177d --- /dev/null +++ b/.cargo/config-avx.toml @@ -0,0 +1,14 @@ +# AVX WITHOUT AVX2 (Sandy Bridge class) — the `simd_avx.rs` realization. +# +# Usage: +# env -u RUSTFLAGS cargo --config .cargo/config-avx.toml +# bash scripts/masking-parity.sh avx-qemu (runs under qemu -cpu SandyBridge) +# +# Selects the arm `all(target_feature = "avx", not(target_feature = "avx2"))`: +# Sandy Bridge / Ivy Bridge, AMD Bulldozer–Jaguar, VMs that hide AVX2. Plan: +# `.claude/plans/simd-avx1-backend-v1.md`. Overlay semantics as in +# `config-v3.toml`: only the target-cpu is set here; the crypto cfgs come from +# `.cargo/config.toml` through cargo's rustflags join. `env -u RUSTFLAGS` is +# load-bearing for the same reason. +[target.'cfg(target_arch = "x86_64")'] +rustflags = ["-Ctarget-cpu=sandybridge"] diff --git a/.claude/blackboard.md b/.claude/blackboard.md index 0dbfb623..350e88fd 100644 --- a/.claude/blackboard.md +++ b/.claude/blackboard.md @@ -1,3 +1,23 @@ +## 2026-10-10 — `simd_avx.rs`: AVX-without-AVX2 realization (Sandy/Ivy Bridge, Bulldozer–Jaguar), plus three pre-existing x86 fixes + +- **What:** a seventh realization, selected at compile time by `all(target_feature = "avx", not(target_feature = "avx2"))`. You get it with `target-cpu=native` on such a CPU, or with `.cargo/config-avx.toml` (`sandybridge`). The types and the `simd.rs` exports are unchanged; the 49 AVX2-only items in `simd_avx2.rs` and 19 in `simd_avx512.rs` are cfg-gated, and `simd_avx.rs` supplies the same methods from SSE2–SSE4.1 halves plus AVX1 float-domain ops. There are no runtime feature checks. The `cpu_guard` AVX2 startup floor applies only when the AVX2 arm is compiled in. Baseline builds (no `avx` cfg) are UNCHANGED: they keep the AVX2 arm and the floor. Whether the SSE tier should take them over is an open operator question. +- **Evidence:** + - `scripts/masking-parity.sh avx-qemu` passes all 15 groups under `qemu-x86_64-static -cpu SandyBridge`, with header `avx2=false avx512f=false`. + - 422 `simd` lib tests pass on that arm under qemu. + - The CI `realization/avx` row runs both, plus an objdump witness. The witness checks the parity binary for AVX2 integer ymm ops, FMA3 and F16C, and finds none. It can fire: the AVX2-op check finds 20,162 hits on a v3 binary. Its FMA/F16C check has never been seen to fire, because the parity program never calls `mul_add`; the unit test under qemu covers that case instead. + - Clippy `-D warnings` is clean on v3, v4, avx and baseline. Lib tests pass on v3 (2510) and v4 (2561). Codegen witness avx2/avx512 and parity v3/v4 PASS. +- **sentinel-qa BLOCK, fixed:** `F32x8::mul_add` called `_mm256_fmadd_ps` ungated, and F32x8 is exported on this arm, so a Sandy Bridge build faulted. It now has `cfg(target_feature = "fma")` with a per-lane `f32::mul_add` fallback (fused, one rounding). Test `f32x8_mul_add_is_fused` separates fused from unfused rounding. Disable: forcing the FMA body on the AVX build gives SIGILL under qemu. The F16C sites in `simd_avx512.rs` are runtime-detected and need nothing. +- **Pre-existing bugs found and fixed:** + - `simd_avx2::dot_i8`, a safe `pub fn` with no in-tree callers: `vpmaddubsw` saturated at `a = b = 127` (4080 instead of 516128), and a shorter `b` was read past its end. This is inventory row 410. Both arms now use `pmovsxbw` + `pmaddwd`, fold into an i64 every 16,384 chunks, and sum over `min` length. Test `simd_avx2_dot_i8_is_exact` is red on the old code on both arms. + - Five x86 256-bit methods had **no test on any stable arm**: `U16x16::{mullo, blend_epi32}`, `U8x32::shuffle_bytes`, `I8x32::cmp_gt` and `I16x16::cmp_gt`. Found because their disable runs stayed green. Test `x86_256bit_integer_methods_match_scalar` now checks them against scalar references on every x86 arm. +- **Disable runs on `simd_avx.rs`, all red, each restored:** cmpgt bias dropped (6 tests), `reduce_min` shuffle, saturating sub → wrapping, `transpose8` unpack swap, blend operand swap, I8 cmp_gt operand swap, I16 cmp_gt half swap, shuffle operand swap, mullo → mulhi. The first run of the I16 half swap crashed qemu itself (`QEMU internal SIGSEGV`) under the multithreaded harness. That is not a detection; single-threaded, the assertion fails as it should. +- **Unsafe:** 29 new blocks in `simd_avx.rs`, all SAFETY-commented, verdict IRREDUCIBLE. They are appended to `.claude/knowledge/unsafe-inventory/sites.tsv` with a README addendum. +- **Open:** + - The SSE-tier scope decision (all non-AVX builds vs only v2 builds). + - The `saturating_abs` SSSE3 patch, awaiting approval. + - `dot_i8` changed its results; it has no in-tree callers, so the effect on external consumers is unknown. +- **Plan:** `.claude/plans/simd-avx1-backend-v1.md`. + ## 2026-10-08 — `PillarReport::deferred` + `certified()`; `Cascade::query` documents that Stroke 1 can drop true hits - **Pillars:** Pillar-15/16/17 run no probe but report `passed = true`. A gate on `passed` counted them as certified. `PillarReport` now has `deferred: bool` (true only for those three) and `certified() = passed && !deferred`. `print()` shows `DEFERRED`. `passed` keeps its meaning, so existing callers do not change. @@ -3630,3 +3650,11 @@ qemu `-cpu Nehalem` 132, `SandyBridge` 132, `IvyBridge` 132, `Haswell` 0, `max` 0. Side effect: a baseline build on a PRE-AVX CPU now gets the message too (the guard is not VEX-encoded there). The optional pre-AVX to-do above remains only for builds compiled with `target-cpu` v3/v4/native. + +## simd_avx.rs (AVX-without-AVX2 backend) — started 2026-10-10, plan `.claude/plans/simd-avx1-backend-v1.md` + +Operator chose Option A after #348. P0 landed: `.cargo/config-avx.toml` +(`-Ctarget-cpu=sandybridge`) and `scripts/masking-parity.sh avx-qemu` (run under +`qemu-x86_64-static -cpu SandyBridge`). Before the backend exists the arm FAILS +with exit 132 from `cpu_guard`'s AVX2 floor (measured) — the expected red. +No CI row yet; it is added when the arm goes green. diff --git a/.claude/knowledge/unsafe-inventory/README.md b/.claude/knowledge/unsafe-inventory/README.md index cdf7f010..25fd3334 100644 --- a/.claude/knowledge/unsafe-inventory/README.md +++ b/.claude/knowledge/unsafe-inventory/README.md @@ -144,3 +144,32 @@ A flag is a lead, not a verdict. The first four were re-read at source and hold. per type to two. 4. SAFETY comments, macro-first. 5. `SAFE-API` swaps; `SAFE-CRATE` only with approval. + +## Addendum 2026-10-10 — `src/simd_avx.rs` (AVX-without-AVX2 arm) + +The snapshot above predates this file. Its **29 sites** are appended to +`sites.tsv` (rows at the end, line numbers from the commit that added them). +All are `unsafe { }` blocks over SSE2–SSE4.1 / AVX1 intrinsics, all carry a +`// SAFETY:` comment, and all get verdict **IRREDUCIBLE**: per the probe table, +an x86 intrinsic in a plain fn is still E0133 on rustc 1.99 even when the +feature is compiled in. Their footing is the arm predicate +`all(target_feature = "avx", not(target_feature = "avx2"))`: the module only +exists in builds where AVX (and, through it, SSE4.2) is enabled for the whole +binary, and `cpu_guard` checks those features at startup. + +Also changed by the same arc: `F32x8::mul_add` in `src/simd_avx512.rs` now +issues `_mm256_fmadd_ps` only under `cfg(target_feature = "fma")` and otherwise +uses `f32::mul_add` per lane. Before, it was an ungated FMA3 intrinsic in a +type exported on the AVX arm (sentinel-qa BLOCK; a Sandy Bridge build faulted, +verified under `qemu -cpu SandyBridge`). The site count for that file is +unchanged; its line numbers in `sites.tsv` have moved by a few rows. + +**Resolved by the same arc — `src/simd_avx2.rs` row 410 (`dot_i8`).** The +bounds flag (a shorter `b` read 32 bytes past its end inside a safe `pub fn`) +is fixed: both x86 arms now sum over `min(a.len(), b.len())` bytes with +`chunks_exact`, so every load is in bounds. The same commit fixes a wrong-value +bug the flag did not name: the `vpmaddubsw` form saturated i16 at +`a = b = 127` (returned 4080 instead of 516128 over 32 bytes). The rows' second +flag ("safe pub fn uses AVX2 with no cfg") is now covered: the AVX2 bodies are +cfg-gated off the AVX-without-AVX2 arm, and a baseline (no `avx` cfg) build +keeps `cpu_guard`'s AVX2 startup floor. diff --git a/.claude/knowledge/unsafe-inventory/sites.tsv b/.claude/knowledge/unsafe-inventory/sites.tsv index 8047175f..cd0676b1 100644 --- a/.claude/knowledge/unsafe-inventory/sites.tsv +++ b/.claude/knowledge/unsafe-inventory/sites.tsv @@ -1300,3 +1300,32 @@ src/zip/ndproducer.rs 360 UPSTREAM fn src UNSAFE-FN-API missing - unsafe fn: cal src/zip/ndproducer.rs 364 UPSTREAM fn src UNSAFE-FN-API missing - unsafe fn: caller upholds pointer/bounds/validity precondition; raw-pointer core design ptr_arith unsafe fn uget_ptr(&self, i: &Self::Dim) -> *const A { src/zip/ndproducer.rs 409 UPSTREAM fn src UNSAFE-FN-API missing - unsafe fn: caller upholds pointer/bounds/validity precondition; raw-pointer core design ptr_arith unsafe fn as_ref(&self, ptr: *mut A) -> *mut A { src/zip/ndproducer.rs 413 UPSTREAM fn src UNSAFE-FN-API missing - unsafe fn: caller upholds pointer/bounds/validity precondition; raw-pointer core design ptr_arith unsafe fn uget_ptr(&self, i: &Self::Dim) -> *mut A { +src/simd_avx.rs 42 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic unsafe { (_mm256_castsi256_si128(v), _mm256_extractf128_si256::<1>(v)) } +src/simd_avx.rs 49 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic unsafe { _mm256_set_m128i(hi, lo) } +src/simd_avx.rs 59 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic unsafe { join($op(al, bl), $op(ah, bh)) } +src/simd_avx.rs 69 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic unsafe { _mm256_castps_si256($op(_mm256_castsi256_ps($a), _mm256_castsi256_ps($b))) } +src/simd_avx.rs 79 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic unsafe { (_mm_movemask_epi8(lo) as u32 & 0xFFFF) | ((_mm_movemask_epi8(hi) as u32) << 16) +src/simd_avx.rs 93 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic let mut sum = unsafe { +src/simd_avx.rs 133 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic let mut result = unsafe { +src/simd_avx.rs 174 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic unsafe { +src/simd_avx.rs 192 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic,raw_load_store,ptr_arith unsafe { +src/simd_avx.rs 212 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic,raw_load_store,ptr_arith unsafe { _mm256_loadu_pd(v.0.as_ptr().add(4 * half) as *const f64) } +src/simd_avx.rs 219 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic let c = unsafe { +src/simd_avx.rs 235 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic,raw_load_store,ptr_arith unsafe { _mm256_storeu_pd(out[4 * bj + k].0.as_mut_ptr().add(4 * bi) as *mut f64, v) }; +src/simd_avx.rs 276 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic,raw_load_store,ptr_arith core::array::from_fn(|k| unsafe { _mm_loadu_si128(self.0.as_ptr().add(4 * k) as *const __m +src/simd_avx.rs 284 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic unsafe { +src/simd_avx.rs 297 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic unsafe { +src/simd_avx.rs 314 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic let q = unsafe { _mm_movemask_ps(_mm_castsi128_ps(_mm_cmpgt_epi32(a[k], b[k]))) as u32 }; +src/simd_avx.rs 332 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic unsafe { +src/simd_avx.rs 343 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic unsafe { +src/simd_avx.rs 360 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic Self(unsafe { _mm256_permute2f128_si256::(self.0, other.0) }) +src/simd_avx.rs 370 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic Self(unsafe { +src/simd_avx.rs 394 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic unsafe { +src/simd_avx.rs 478 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic let ones = unsafe { _mm256_set1_epi16(-1) }; +src/simd_avx.rs 496 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic,raw_load_store,ptr_arith unsafe { +src/simd_avx.rs 527 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic let bias = unsafe { _mm256_set1_epi8(i8::MIN) }; +src/simd_avx.rs 562 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic unsafe { +src/simd_avx.rs 573 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic unsafe { +src/simd_avx.rs 605 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic Self(unsafe { join(_mm_blendv_epi8(bl, al, ml), _mm_blendv_epi8(bh, ah, mh)) }) +src/simd_avx.rs 688 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic unsafe { +src/simd_avx.rs 755 FORK block src IRREDUCIBLE present - SSE2-SSE4.1/AVX1 intrinsics on the AVX-without-AVX2 arm; still E0133 on rustc 1.99 (see probe table), so the block cannot be removed x86_intrinsic unsafe { diff --git a/.claude/plans/simd-avx1-backend-v1.md b/.claude/plans/simd-avx1-backend-v1.md new file mode 100644 index 00000000..41a18aab --- /dev/null +++ b/.claude/plans/simd-avx1-backend-v1.md @@ -0,0 +1,70 @@ +# Plan: `simd_avx.rs` — AVX-without-AVX2 backend (v1) + +Operator decision 2026-10-10 (after PR #348): Option A, a real backend for CPUs +with AVX but not AVX2 (Sandy Bridge / Ivy Bridge, AMD Bulldozer–Jaguar, VMs that +hide AVX2), selected at compile time. No runtime feature checks in the SIMD code. + +## Selection + +* Arm predicate: `all(target_arch = "x86_64", target_feature = "avx", + not(target_feature = "avx2"))` — written out at each gate (no build.rs, no + cfg-alias crate). +* Reached by: `target-cpu=native` on such a CPU, or the new + `.cargo/config-avx.toml` (`-Ctarget-cpu=sandybridge`). +* Unchanged: AVX-512 builds, AVX2 builds, and BASELINE builds (no `avx` cfg). + A baseline build keeps the AVX2 arm and the `cpu_guard` AVX2 startup floor — + switching it to this arm would slow every baseline build on AVX2 hosts. + (Open question for the operator; default is "unchanged".) +* `cpu_guard::BACKEND_FLOOR` applies only when the AVX2 arm is compiled in. On + this arm the normal compiled-feature table already requires `avx`. + +## Shape: impl blocks, not types + +Measured: the AVX2-only intrinsics sit in impl blocks; the TYPES are shared and +referenced across the arm (`F32x16`→`I32x16`/`U32x16`, `F64x8`→`U64x8`, +`U16x32`→`U8x64`, `U8x32`→`U16x16`). Redefining types would cascade, so: + +* Type definitions and `simd.rs` export lists do not change. +* In `simd_avx2.rs` / `simd_avx512.rs`, every method that calls an AVX2-only + intrinsic moves into an impl block gated `not()`. Mixed blocks + are split; their portable methods stay shared (no duplication). +* `simd_avx.rs` provides the same methods, same signatures, gated on the arm: + two `__m128i` halves via SSE2/SSSE3/SSE4.1 (all present on every AVX CPU), + plus AVX1 for loads/stores/float-domain bit ops. + +Affected — ⊘ the first table here counted doc-comment mentions and listed +`U8x64`, `U32x16`, `U32x8`, `U16x8`, `palette_lookup_u8x8` and a 1-op `F32x8` +item. Re-measured with comments stripped (`tools/avx2_gate.py`): those are +scalar or comment-only and needed nothing. What actually needed a gate: + +| item | gated in | notes | +|---|---|---| +| `U8x32` | `simd_avx2.rs` | 15 methods + 5 traits; biggest | +| `U16x16` | `simd_avx2.rs` | 8 methods + 12 traits | +| `U64x8` | `simd_avx2.rs` | `rotl_half`, `mul_lo32`, `transpose8`, `Shl`/`Shr` (per-lane: no SSE `sllv`) | +| `I32x16` | `simd_avx2.rs` | `reduce_min/max`, `gt_bitmask` | +| `popcount`, `dot_i8` (free fns) | `simd_avx2.rs` | re-exported from `simd_avx` on the arm | +| `I8x32`, `I16x16` | `simd_avx512.rs` | 6 + 5 methods, 4 traits each | +| `F32x8::mul_add` | `simd_avx512.rs` | FMA3, not AVX2 — missed by the AVX2-only scan, caught by sentinel-qa; now `cfg(target_feature = "fma")` with a per-lane `f32::mul_add` fallback | + +49 gates in `simd_avx2.rs`, 19 in `simd_avx512.rs`. F16C sites in +`simd_avx512.rs` are runtime-detected (`is_x86_feature_detected!`) and need none. + +## Work items + +- [x] P0 TESTS FIRST: a `native-avx` arm in `crates/simd-masking-parity` + (`--config .cargo/config-avx.toml`, run under `qemu-x86_64-static -cpu + SandyBridge`), wired into `scripts/masking-parity.sh` and the + `simd-matrix` CI workflow. Expect it to SIGILL before the backend exists. +- [x] P1 `.cargo/config-avx.toml`; arm predicate; `cpu_guard` floor gated. +- [x] P2 split the affected impl blocks (no behaviour change on other arms; + codegen-witness avx2/avx512 must stay byte-for-byte green). +- [x] P3 `simd_avx.rs` per type, one type per chunk (U8x32, U16x16, U64x8, + U32x8, U32x16, I32x16, U8x64, popcount/dot_i8, I8x32, I16x16, U16x8, + palette_lookup_u8x8). Every `unsafe` gets SAFETY + sentinel-qa audit. +- [x] P4 gates: lib tests on the avx arm under qemu SandyBridge; parity on all + six existing arms + the new one; v3/v4 clippy; disable-run per type. +- [x] P5 docs: CLAUDE.md realization list, contract doc, blackboard, unsafe + inventory rows for the new file. + +Estimate: ~1,000–1,500 lines, 2–3 sessions. diff --git a/.github/workflows/simd-matrix.yaml b/.github/workflows/simd-matrix.yaml index 7ab27bbb..b7834310 100644 --- a/.github/workflows/simd-matrix.yaml +++ b/.github/workflows/simd-matrix.yaml @@ -218,6 +218,54 @@ jobs: echo "::warning::runner lacks avx512f — v4 parity run and probe self-check SKIPPED; asserting the emitted assembly only" env -u RUSTFLAGS WITNESS_NO_RUN=1 CARGO_ARGS='--config .cargo/config-v4.toml' bash scripts/codegen-witness.sh avx512 + avx: + # AVX-without-AVX2 realization (`src/simd_avx.rs`, Sandy Bridge class). + # Built for `sandybridge` (`.cargo/config-avx.toml`) and RUN under + # `qemu-x86_64-static -cpu SandyBridge`, so an AVX2 instruction anywhere + # on the executed path faults instead of passing silently on an AVX2 + # runner. Three rungs: parity (bits), an assembly witness that the parity + # binary contains no AVX2 integer instruction at all (LLVM never emits + # AVX2 for this target, so any `vp*` on a ymm register other than the + # AVX1 `vptest` came from an intrinsic that escaped its gate), and the + # SIMD unit tests under emulation. + runs-on: ubuntu-latest + name: realization/avx × x86_64 + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - uses: dtolnay/rust-toolchain@stable + - uses: Swatinem/rust-cache@v2 + with: + key: avx + - name: install qemu-user + run: sudo apt-get update && sudo apt-get install -y qemu-user-static + - name: masking parity (sandybridge build, qemu -cpu SandyBridge) + run: bash scripts/masking-parity.sh avx-qemu + - name: no AVX2 instruction in the parity binary + run: | + bin=crates/simd-masking-parity/target/release/simd-masking-parity + # Sandy Bridge has AVX but neither AVX2 (integer ops on ymm), FMA3 + # nor F16C. The parity binary has no runtime-gated paths, so any of + # them here came from an intrinsic the AVX arm failed to gate. + dis=$(objdump -d --no-show-raw-insn "$bin") + bad=$( { echo "$dis" | grep -E '\svp[a-z0-9]+\s.*%ymm' | grep -vE '\svptest\s'; + echo "$dis" | grep -E '\sv(fn?m(add|sub)|fmaddsub|fmsubadd)[0-9a-z]*\s|\svcvt(ph2ps|ps2ph)\s'; } || true) + if [ -n "$bad" ]; then echo "$bad" | head -20; echo "::error::AVX2/FMA/F16C instruction in the AVX-arm parity binary"; exit 1; fi + echo "no AVX2, FMA or F16C instruction found" + - name: SIMD unit tests under qemu -cpu SandyBridge + env: + CARGO_PROFILE_DEV_DEBUG: "0" + CARGO_PROFILE_TEST_DEBUG: "0" + CARGO_PROFILE_DEV_OPT_LEVEL: "1" + CARGO_PROFILE_TEST_OPT_LEVEL: "1" + run: | + bin=$(env -u RUSTFLAGS cargo --config .cargo/config-avx.toml test --lib --no-run --message-format=json \ + | jq -r 'select(.executable != null and .target.kind == ["lib"] and .target.name == "ndarray") | .executable') + # One thread: qemu-user itself has crashed (internal SIGSEGV) under + # the multithreaded harness while a test was panicking. + qemu-x86_64-static -cpu SandyBridge "$bin" simd --test-threads=1 + neon: # aarch64 realization: cross-build on the x86 runner, run under qemu-user. # Three rungs: parity under qemu (bits), the codegen witness (opt-3 NEON diff --git a/CLAUDE.md b/CLAUDE.md index f3dfc2b9..673f8805 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -196,7 +196,7 @@ src/ ### The parity arms are ALL reachable here — never report one as blocked without apt `scripts/masking-parity.sh` takes `native | nightly | wasm | wasm-scalar | -neon-qemu`, and the cross arms are an `apt-get` away, not an environment +neon-qemu | avx-qemu`, and the cross arms are an `apt-get` away, not an environment limit. Measured 2026-09-16: `neon-qemu` failed with a bare `No such file or directory (os error 2)` — which reads as "this target is not available here" and is in fact a **missing linker**, then a missing @@ -239,8 +239,11 @@ env -u RUSTFLAGS cargo --config ../../.cargo/config-v4.toml run --release Read the program's own header line to confirm which arm you actually got (`avx512f=true`, `neon=true`, …) — that line exists precisely because the -config can silently not apply. Five of the six realizations are reachable -without nightly (AVX2, AVX-512, NEON, wasm-simd128, wasm-scalar); only +config can silently not apply. Six of the seven realizations are reachable +without nightly (AVX2, AVX-512, AVX-without-AVX2, NEON, wasm-simd128, +wasm-scalar); the AVX arm (`src/simd_avx.rs`, `.cargo/config-avx.toml`) runs +under `qemu-x86_64-static -cpu SandyBridge` (`apt-get install -y +qemu-user-static`); only `nightly-simd` needs a toolchain this repo does not pin. The same lesson the sibling `lance-graph-java` records for the JDK: **a stale index or a missing helper binary reporting absence is not evidence of absence.** diff --git a/scripts/masking-parity.sh b/scripts/masking-parity.sh index 787c652b..a8d1418c 100755 --- a/scripts/masking-parity.sh +++ b/scripts/masking-parity.sh @@ -8,12 +8,13 @@ # scripts/masking-parity.sh wasm wasm32 +simd128 under node (simd_wasm arm) # scripts/masking-parity.sh wasm-scalar wasm32 WITHOUT simd128 under node = the scalar arm # scripts/masking-parity.sh neon-qemu aarch64 cross-build run under qemu-aarch64-static +# scripts/masking-parity.sh avx-qemu AVX-without-AVX2 build (config-avx) under qemu -cpu SandyBridge # # Extra cargo arguments (e.g. `--config .cargo/config-v4.toml` for the v4 # realization on the native row) pass through CARGO_ARGS, see # scripts/codegen-witness.sh for why that form and not the env var. set -euo pipefail -ARM="${1:?native|nightly|wasm|wasm-scalar|neon-qemu}" +ARM="${1:?native|nightly|wasm|wasm-scalar|neon-qemu|avx-qemu}" ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" MANIFEST="$ROOT/crates/simd-masking-parity/Cargo.toml" TD="${CARGO_TARGET_DIR:-$ROOT/crates/simd-masking-parity/target}" @@ -61,6 +62,14 @@ case "$ARM" in cargo ${CARGO_ARGS:-} build --release --manifest-path "$MANIFEST" --bin simd-masking-parity --target aarch64-unknown-linux-gnu "$QEMU" -L "$SYSROOT" "$TD/aarch64-unknown-linux-gnu/release/simd-masking-parity" ;; + avx-qemu) + # The `simd_avx.rs` arm: AVX but no AVX2. Built for Sandy Bridge and run on + # an emulated Sandy Bridge, so an AVX2 instruction anywhere in the binary + # faults (SIGILL) instead of silently passing on an AVX2 host. + QEMU="${QEMU_X86_64:-qemu-x86_64-static}" + env -u RUSTFLAGS cargo ${CARGO_ARGS:-} --config "$ROOT/.cargo/config-avx.toml" build --release --manifest-path "$MANIFEST" --bin simd-masking-parity + "$QEMU" -cpu SandyBridge "$TD/release/simd-masking-parity" + ;; *) echo "unknown arm: $ARM"; exit 2 ;; esac echo "masking parity ($ARM): PASS" diff --git a/src/cpu_guard.rs b/src/cpu_guard.rs index b1be1990..547d67ee 100644 --- a/src/cpu_guard.rs +++ b/src/cpu_guard.rs @@ -15,9 +15,10 @@ //! missing. x86_64 only for now; aarch64 is documented below as not covered. //! //! It also enforces one floor that is NOT a compile-time feature: **AVX2 on -//! every x86_64 build.** `crate::simd` selects its AVX2 realization whenever -//! AVX-512 is absent, and that code calls AVX2 intrinsics without a runtime -//! check, so a build compiled without `avx2` still needs AVX2 the moment it +//! every x86_64 build that uses the AVX2 realization.** `crate::simd` selects +//! that realization whenever AVX-512 is absent, except on the AVX-without-AVX2 +//! arm (`simd_avx.rs`), and it calls AVX2 intrinsics without a runtime check, +//! so a baseline build compiled without `avx2` still needs AVX2 the moment it //! touches the SIMD types. The floor is checked once, here, at startup, and //! never per call: dispatch stays compile-time (operator, 2026-10-10, after //! codex review on PR #348). @@ -92,11 +93,12 @@ impl fmt::Display for BuildCpuMismatch { self.missing.join(", ") )?; if self.missing.contains(&"avx2") { - // Rebuilding cannot help: the AVX2 floor applies to every x86_64 build. + // The AVX2 realization was compiled in; only a different build helps. write!( f, - " ndarray's x86_64 SIMD backend needs AVX2 whatever the build flags, \ - so no build of this program can run on this CPU." + " This build uses ndarray's AVX2 SIMD backend. For a CPU with AVX but \ + not AVX2, rebuild with `target-cpu=native` on it or with \ + `--config .cargo/config-avx.toml`; a CPU without AVX is not supported." ) } else { write!( @@ -270,11 +272,16 @@ const FEATURES: &[FeatureRow] = { ) }; -/// Features required on every x86_64 build, whatever it was compiled with: -/// the floor of `crate::simd`'s AVX2 realization (see the module docs). The -/// `compiled` column is `true` by definition. +/// The floor of `crate::simd`'s AVX2 realization (see the module docs): +/// required on every x86_64 build that compiles that realization in, whatever +/// its target features. The one x86_64 build that does not is the +/// AVX-without-AVX2 arm (`simd_avx.rs`, `.cargo/config-avx.toml`), whose +/// `avx` requirement the compiled-feature table already enforces. #[cfg(target_arch = "x86_64")] -const BACKEND_FLOOR: &[FeatureRow] = &[("avx2", true, || cpuid::bit(7, 0, 'b', 5) && cpuid::os_avx())]; +const BACKEND_FLOOR: &[FeatureRow] = + &[("avx2", cfg!(not(all(target_feature = "avx", not(target_feature = "avx2")))), || { + cpuid::bit(7, 0, 'b', 5) && cpuid::os_avx() + })]; #[cfg(not(target_arch = "x86_64"))] const BACKEND_FLOOR: &[FeatureRow] = &[]; @@ -437,24 +444,26 @@ mod tests { #[test] #[cfg(target_arch = "x86_64")] - fn avx2_is_required_on_every_x86_64_build() { - // The floor must not depend on the build flags: a baseline build - // (no `avx2` compiled in) still reaches the AVX2 SIMD backend. + fn avx2_is_required_unless_the_avx_arm_is_compiled_in() { + // A baseline build (no `avx2` compiled in) still reaches the AVX2 SIMD + // backend, so the floor applies to it; only the AVX-without-AVX2 arm + // is exempt, because it never selects that backend. + let avx_arm = cfg!(all(target_feature = "avx", not(target_feature = "avx2"))); assert!(BACKEND_FLOOR .iter() - .any(|&(n, compiled, _)| n == "avx2" && compiled)); + .any(|&(n, compiled, _)| n == "avx2" && compiled == !avx_arm)); } #[test] fn the_message_explains_the_avx2_floor_only_when_avx2_is_missing() { let floor = BuildCpuMismatch { missing: vec!["avx2"] }.to_string(); - assert!(floor.contains("needs AVX2 whatever the build"), "{floor}"); - assert!(!floor.contains("Rebuild"), "rebuilding cannot help: {floor}"); + assert!(floor.contains("AVX2 SIMD backend"), "{floor}"); + assert!(floor.contains("config-avx.toml"), "{floor}"); let other = BuildCpuMismatch { missing: vec!["avx512f"], } .to_string(); - assert!(!other.contains("needs AVX2"), "{other}"); + assert!(!other.contains("AVX2 SIMD backend"), "{other}"); assert!(other.contains("Rebuild"), "{other}"); } diff --git a/src/lib.rs b/src/lib.rs index 74d2fda3..482cb9af 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -244,6 +244,15 @@ pub(crate) mod simd_avx512; #[cfg(all(feature = "std", target_arch = "x86_64"))] #[allow(clippy::all, missing_docs, dead_code, unused_variables, unused_imports)] pub mod simd_avx2; +/// AVX-without-AVX2 realization of the methods `simd_avx2` / `simd_avx512` +/// gate on AVX2; see the module docs. +#[cfg(all( + feature = "std", + target_arch = "x86_64", + target_feature = "avx", + not(target_feature = "avx2") +))] +mod simd_avx; // (The standalone `simd_crypto` ChaCha20 primitive was RETIRED — superseded by // the AdaWorldAPI `chacha20` fork whose backend rides `ndarray::simd::U32x16`, diff --git a/src/simd.rs b/src/simd.rs index a3735dad..0904c2ab 100644 --- a/src/simd.rs +++ b/src/simd.rs @@ -218,6 +218,11 @@ pub const PREFERRED_I16_LANES: usize = 16; // * v3 / GitHub-CI default → `target_feature = "avx2"` only → // simd_avx2 backend (F32x16 = two-half (f32x8, f32x8), int wrappers // are scalar polyfills via the `avx2_int_type!` macro). +// * AVX without AVX2 (`.cargo/config-avx.toml`, or native on a Sandy +// Bridge-class host) → the SAME types and exports as the AVX2 arm, but +// every method that would need AVX2 is realized in `simd_avx.rs` (two +// SSE halves / AVX1 forms); `simd_avx2.rs` and `simd_avx512.rs` gate +// those methods on `not(all(avx, not(avx2)))`. // * v4 (or native on AVX-512 host) → `target_feature = "avx512f"` → // simd_avx512 backend with native __m512 / __m512d / __m512i. // * aarch64 → simd_neon backend. @@ -1078,6 +1083,115 @@ mod tests { /// transpose that drops, duplicates or misroutes any word fails, and the /// fixture is asserted not to be symmetric, so an identity "transpose" /// fails too. Transposing twice must return the input. + /// The x86 256-bit integer methods that had no test on any stable arm, + /// each checked lane-by-lane against a scalar reference on asymmetric + /// data. They run on the AVX2, AVX-512 and AVX-without-AVX2 realizations; + /// a swapped operand, swapped 128-bit halves or a wrong intrinsic fails. + #[cfg(all(feature = "std", target_arch = "x86_64"))] + #[test] + fn x86_256bit_integer_methods_match_scalar() { + use crate::simd_avx2::{U16x16, U8x32}; + use crate::simd_avx512::{I16x16, I8x32}; + + let a16: [u16; 16] = core::array::from_fn(|i| (i as u16).wrapping_mul(4_001).wrapping_add(7)); + let b16: [u16; 16] = core::array::from_fn(|i| (i as u16).wrapping_mul(331).wrapping_add(60_013)); + let (va, vb) = (U16x16::from_array(a16), U16x16::from_array(b16)); + let want: [u16; 16] = core::array::from_fn(|i| a16[i].wrapping_mul(b16[i])); + assert_eq!(va.mullo(vb).to_array(), want, "U16x16::mullo"); + + // Bit i of IMM takes dword i (u16 lanes 2i, 2i+1) from `other`. + const IMM: i32 = 0b1010_0110; + let want: [u16; 16] = core::array::from_fn(|i| if IMM >> (i / 2) & 1 == 1 { b16[i] } else { a16[i] }); + assert_eq!(va.blend_epi32::(vb).to_array(), want, "U16x16::blend_epi32"); + + let table: [u8; 32] = core::array::from_fn(|i| (i as u8).wrapping_mul(3).wrapping_add(1)); + let idx: [u8; 32] = core::array::from_fn(|i| match i % 5 { + 0 => 0x80 | i as u8, // high bit set: zero + _ => (i as u8).wrapping_mul(7) & 0x0f, + }); + let want: [u8; 32] = core::array::from_fn(|j| { + if idx[j] & 0x80 != 0 { + 0 + } else { + table[(j / 16) * 16 + (idx[j] & 0x0f) as usize] + } + }); + let got = U8x32::from_array(table) + .shuffle_bytes(U8x32::from_array(idx)) + .to_array(); + assert_eq!(got, want, "U8x32::shuffle_bytes"); + + let x8: [i8; 32] = core::array::from_fn(|i| (i as i8).wrapping_mul(37).wrapping_sub(60)); + let y8: [i8; 32] = core::array::from_fn(|i| (i as i8).wrapping_mul(-23).wrapping_add(5)); + let want = (0..32).fold(0u32, |m, i| m | ((x8[i] > y8[i]) as u32) << i); + assert!(want != 0 && want != u32::MAX, "fixture must mix both outcomes"); + assert_eq!(I8x32::from_array(x8).cmp_gt(I8x32::from_array(y8)), want, "I8x32::cmp_gt"); + + let x16: [i16; 16] = core::array::from_fn(|i| (i as i16) * 2_111 - 15_000); + let y16: [i16; 16] = core::array::from_fn(|i| if i < 8 { 1_000 - (i as i16) * 900 } else { -4_000 }); + let want = (0..16).fold(0u16, |m, i| m | ((x16[i] > y16[i]) as u16) << i); + assert!(want & 0xff != want >> 8, "halves must differ so a half swap shows"); + assert_eq!(I16x16::from_array(x16).cmp_gt(I16x16::from_array(y16)), want, "I16x16::cmp_gt"); + } + + /// `simd_avx2::dot_i8` is an exact signed dot product of the first + /// `min(a.len(), b.len())` bytes, read as `i8`. Three failure modes are + /// pinned: the old `maddubs` form saturated i16 at `a = b = 127` + /// (`255 * 127 * 2 > i16::MAX` after the `^ 0x80` bias), an unfolded i32 + /// accumulator wraps on long inputs, and a shorter `b` was read past its + /// end. On the AVX-without-AVX2 arm the same name is `simd_avx::dot_i8`. + #[cfg(target_arch = "x86_64")] + #[test] + fn simd_avx2_dot_i8_is_exact() { + use crate::simd_avx2::dot_i8 as dot; + let reference = |a: &[u8], b: &[u8]| -> i64 { + a.iter() + .zip(b) + .map(|(&x, &y)| (x as i8 as i64) * (y as i8 as i64)) + .sum() + }; + for &(x, y) in &[(127u8, 127u8), (0x80, 0x80), (127, 0x80), (0x80, 127), (0xff, 1)] { + for len in [0, 1, 15, 16, 17, 31, 32, 33, 64, 100] { + let (a, b) = (vec![x; len], vec![y; len]); + assert_eq!(dot(&a, &b), reference(&a, &b), "x={x:#x} y={y:#x} len={len}"); + } + } + let a: Vec = (0..1000u32) + .map(|i| (i.wrapping_mul(2_654_435_761) >> 13) as u8) + .collect(); + let b: Vec = (0..1000u32) + .map(|i| (i.wrapping_mul(40_503) >> 5) as u8) + .collect(); + assert_eq!(dot(&a, &b), reference(&a, &b)); + assert_eq!(dot(&a[..999], &b), reference(&a[..999], &b[..999])); + assert_eq!(dot(&a, &b[..40]), reference(&a[..40], &b[..40])); + // 1.5 Mi products of 16_384 sum to ~2.6e10, past i32::MAX. + let n = 3 << 19; + let big = vec![0x80u8; n]; + assert!(16_384 * n as i64 > i32::MAX as i64); + assert_eq!(dot(&big, &big), 16_384 * n as i64); + } + + /// `F32x8::mul_add` is FUSED on every arm: one rounding, not two. With + /// `x = 1 + 2^-12`, `x * x = 1 + 2^-11 + 2^-24` exactly; rounded to f32 + /// that is `1 + 2^-11` (a tie, to even), so an unfused `x * x - (1 + 2^-11)` + /// gives 0 while the fused form keeps the `2^-24`. On the AVX-without-FMA + /// arm this runs the per-lane fallback, and under `qemu -cpu SandyBridge` + /// an ungated FMA3 instruction would fault instead. + #[test] + fn f32x8_mul_add_is_fused() { + let x = 1.0f32 + f32::EPSILON * 2048.0; // 1 + 2^-12 + let c = -(1.0f32 + f32::EPSILON * 4096.0); // -(1 + 2^-11) + assert_eq!(x * x + c, 0.0, "fixture must separate fused from unfused"); + let got = F32x8::splat(x).mul_add(F32x8::splat(x), F32x8::splat(c)); + assert_eq!(got.to_array(), [f32::EPSILON / 2.0; 8]); // 2^-24 + let lanes = F32x8::from_array([1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0]); + let r = lanes + .mul_add(F32x8::splat(2.0), F32x8::splat(0.5)) + .to_array(); + assert_eq!(r, [2.5, 4.5, 6.5, 8.5, 10.5, 12.5, 14.5, 16.5]); + } + #[test] fn u64x8_transpose8_matches_the_index_map() { use super::U64x8; diff --git a/src/simd_avx.rs b/src/simd_avx.rs new file mode 100644 index 00000000..77657c7a --- /dev/null +++ b/src/simd_avx.rs @@ -0,0 +1,791 @@ +//! AVX-without-AVX2 realization (Sandy Bridge / Ivy Bridge, AMD Bulldozer to +//! Jaguar, VMs that hide AVX2). Plan: `.claude/plans/simd-avx1-backend-v1.md`. +//! +//! Compiled only on `all(target_feature = "avx", not(target_feature = "avx2"))` +//! (`target-cpu=native` on such a CPU, or `.cargo/config-avx.toml`). It is not a +//! second set of types: `simd.rs` exports the same types on this arm as on the +//! AVX2 arm. What differs is the realization of the methods that would need +//! AVX2. `simd_avx2.rs` and `simd_avx512.rs` gate exactly those methods with +//! `cfg(not())`, and this file supplies them with the same signatures +//! and the same results, built from: +//! +//! * two 128-bit SSE2/SSSE3/SSE4.1 halves for 256-bit integer arithmetic, and +//! * AVX1's float-domain forms where they are bit-exact (`and`/`or`/`xor` via +//! `_ps`, `permute2f128`, `blend_ps`, 64-bit unpacks via `_pd`). +//! +//! Selection is compile-time only; nothing here checks the CPU at run time. +//! +//! # Safety footing (every `unsafe` block below) +//! +//! On this arm `avx` is a compiled-in target feature, and rustc's feature +//! implications make `sse2`, `ssse3`, `sse4.1` and `sse4.2` compiled in with it. +//! `cpu_guard` refuses to start a binary whose compiled-in features the CPU +//! lacks, so every intrinsic used here is supported at run time. Each block +//! states anything beyond that (memory bounds, shift-count ranges). + +use core::arch::x86_64::*; +use core::ops::{ + Add, AddAssign, BitAnd, BitAndAssign, BitOr, BitOrAssign, BitXor, BitXorAssign, Mul, Not, Shl, Shr, Sub, SubAssign, +}; + +use crate::simd_avx2::{I32x16, U16x16, U64x8, U8x32}; +use crate::simd_avx512::{F32x8, I16x16, I8x32}; + +// --------------------------------------------------------------------------- +// 256 ⇄ 2 × 128 +// --------------------------------------------------------------------------- + +/// Low and high 128-bit halves of a 256-bit integer register. +#[inline(always)] +fn split(v: __m256i) -> (__m128i, __m128i) { + // SAFETY: AVX is compiled in (module docs); register-only ops. + unsafe { (_mm256_castsi256_si128(v), _mm256_extractf128_si256::<1>(v)) } +} + +/// Inverse of [`split`]. +#[inline(always)] +fn join(lo: __m128i, hi: __m128i) -> __m256i { + // SAFETY: AVX is compiled in (module docs); register-only op. + unsafe { _mm256_set_m128i(hi, lo) } +} + +/// Applies a 128-bit binary op to both halves of two 256-bit registers. +macro_rules! halves2 { + ($op:ident, $a:expr, $b:expr) => {{ + let (al, ah) = split($a); + let (bl, bh) = split($b); + // SAFETY: the SSE op is compiled in on this arm (module docs); + // register-only op. + unsafe { join($op(al, bl), $op(ah, bh)) } + }}; +} + +/// Applies an AVX1 float-domain bitwise op to two 256-bit integer registers. +/// Bit-exact: `and`/`or`/`xor` do not look at the bits as floats. +macro_rules! bits_ps { + ($op:ident, $a:expr, $b:expr) => {{ + // SAFETY: AVX is compiled in (module docs); casts and a bitwise op on + // registers only. + unsafe { _mm256_castps_si256($op(_mm256_castsi256_ps($a), _mm256_castsi256_ps($b))) } + }}; +} + +/// `movemask_epi8` over a 256-bit register: low half → bits 0..16, high half → +/// bits 16..32, the same order `_mm256_movemask_epi8` produces. +#[inline(always)] +fn movemask_epi8_256(v: __m256i) -> u32 { + let (lo, hi) = split(v); + // SAFETY: SSE2 is compiled in (module docs); register-only op. + unsafe { (_mm_movemask_epi8(lo) as u32 & 0xFFFF) | ((_mm_movemask_epi8(hi) as u32) << 16) } +} + +// --------------------------------------------------------------------------- +// Free functions (re-exported from `simd_avx2` on this arm) +// --------------------------------------------------------------------------- + +/// Popcount of a byte slice: SSSE3 `pshufb` nibble lookup over 16-byte chunks, +/// `psadbw` to fold. Same result as the AVX2 realization. +pub fn popcount(a: &[u8]) -> u64 { + let chunks = a.len() / 16; + // SAFETY: SSE2/SSSE3 are compiled in (module docs). Every load reads the + // 16 bytes at `i * 16` with `i < chunks = len / 16`, so it is in bounds; + // `loadu` needs no alignment. + let mut sum = unsafe { + let low_mask = _mm_set1_epi8(0x0f); + let lookup = _mm_setr_epi8(0, 1, 1, 2, 1, 2, 2, 3, 1, 2, 2, 3, 2, 3, 3, 4); + let mut total = _mm_setzero_si128(); + let mut i = 0; + while i < chunks { + // Byte counters reach at most 8 per chunk; 31 chunks stay below + // 256, so fold before they could wrap. + let end = (i + 31).min(chunks); + let mut local = _mm_setzero_si128(); + while i < end { + let v = _mm_loadu_si128(a.as_ptr().add(i * 16) as *const __m128i); + let lo = _mm_and_si128(v, low_mask); + let hi = _mm_and_si128(_mm_srli_epi16::<4>(v), low_mask); + let cnt = _mm_add_epi8(_mm_shuffle_epi8(lookup, lo), _mm_shuffle_epi8(lookup, hi)); + local = _mm_add_epi8(local, cnt); + i += 1; + } + total = _mm_add_epi64(total, _mm_sad_epu8(local, _mm_setzero_si128())); + } + let mut lanes = [0u64; 2]; + _mm_storeu_si128(lanes.as_mut_ptr() as *mut __m128i, total); + lanes[0] + lanes[1] + }; + for &byte in &a[chunks * 16..] { + sum += byte.count_ones() as u64; + } + sum +} + +/// Signed int8 dot product, `Σ a[i] as i8 × b[i] as i8`, exact, over the first +/// `min(a.len(), b.len())` bytes. SSE4.1 `pmovsxbw` sign-extends each 16-byte +/// chunk to i16 and SSE2 `pmaddwd` multiplies pairwise into i32, exact for +/// every i8 pair; the i32 lanes are folded into an i64 every +/// [`crate::simd_avx2::DOT_I8_FOLD`] chunks, so no length can wrap them. +pub fn dot_i8(a: &[u8], b: &[u8]) -> i64 { + let len = a.len().min(b.len()); + let (a, b) = (&a[..len], &b[..len]); + let fold = crate::simd_avx2::DOT_I8_FOLD; + let mut total = 0i64; + for (ba, bb) in a.chunks(16 * fold).zip(b.chunks(16 * fold)) { + let (ca, cb) = (ba.chunks_exact(16), bb.chunks_exact(16)); + let (ta, tb) = (ca.remainder(), cb.remainder()); + // SAFETY: SSE2/SSE4.1 are compiled in (module docs). Every chunk is + // exactly 16 bytes, so each 16-byte `loadu` is in bounds; `loadu` + // needs no alignment. + let lanes = unsafe { + let mut acc = _mm_setzero_si128(); + for (xa, xb) in ca.zip(cb) { + let av = _mm_loadu_si128(xa.as_ptr() as *const __m128i); + let bv = _mm_loadu_si128(xb.as_ptr() as *const __m128i); + let lo = _mm_madd_epi16(_mm_cvtepi8_epi16(av), _mm_cvtepi8_epi16(bv)); + let hi = _mm_madd_epi16( + _mm_cvtepi8_epi16(_mm_srli_si128::<8>(av)), + _mm_cvtepi8_epi16(_mm_srli_si128::<8>(bv)), + ); + acc = _mm_add_epi32(acc, _mm_add_epi32(lo, hi)); + } + let mut v = [0i32; 4]; + _mm_storeu_si128(v.as_mut_ptr() as *mut __m128i, acc); + v + }; + total += lanes.iter().map(|&v| v as i64).sum::(); + total += ta + .iter() + .zip(tb) + .map(|(&x, &y)| (x as i8 as i64) * (y as i8 as i64)) + .sum::(); + } + total +} + +// --------------------------------------------------------------------------- +// U64x8 (array-backed; only these methods need AVX2 on the AVX2 arm) +// --------------------------------------------------------------------------- + +impl U64x8 { + /// `(x << n) | (x >> (64 - n))` per 64-bit lane of one 256-bit half, with + /// `1 <= n <= 63` guaranteed by the callers (see the AVX2 realization). + #[inline(always)] + pub(crate) fn rotl_half(v: __m256i, n: u32) -> __m256i { + debug_assert!((1..=63).contains(&n)); + let (lo, hi) = split(v); + // SAFETY: SSE2 is compiled in (module docs). Counts are in `1..=63`, + // so neither shift is by 64 or more. + unsafe { + let l = _mm_cvtsi32_si128(n as i32); + let r = _mm_cvtsi32_si128((64 - n) as i32); + join( + _mm_or_si128(_mm_sll_epi64(lo, l), _mm_srl_epi64(lo, r)), + _mm_or_si128(_mm_sll_epi64(hi, l), _mm_srl_epi64(hi, r)), + ) + } + } + + /// Lane-wise `lo32(self) × lo32(rhs)` as an exact `u64` (`pmuludq`, four + /// per vector). The high 32 bits of every input lane are ignored. + #[inline(always)] + pub fn mul_lo32(self, rhs: Self) -> Self { + let mut o = [0u64; 8]; + // SAFETY: SSE2 is compiled in (module docs). `self.0`, `rhs.0` and `o` + // are `[u64; 8]` (64 bytes); the four 16-byte loads and stores at byte + // offsets 0, 16, 32, 48 are in bounds. + unsafe { + for k in 0..4 { + let a = _mm_loadu_si128(self.0.as_ptr().add(2 * k) as *const __m128i); + let b = _mm_loadu_si128(rhs.0.as_ptr().add(2 * k) as *const __m128i); + _mm_storeu_si128(o.as_mut_ptr().add(2 * k) as *mut __m128i, _mm_mul_epu32(a, b)); + } + } + Self(o) + } + + /// 8×8 transpose of `u64` words across eight registers: + /// `out[i]` lane `j` == `rows[j]` lane `i`. AVX1 forms of the AVX2 + /// realization: `unpack{lo,hi}_pd` and `permute2f128`, which move the same + /// 64-bit words without inspecting them. + #[inline(always)] + pub fn transpose8(rows: [Self; 8]) -> [Self; 8] { + let load = |v: &Self, half: usize| -> __m256d { + // SAFETY: AVX is compiled in (module docs); `v.0` is `[u64; 8]` + // (64 bytes), so the 32-byte load at byte offset 0 or 32 is in + // bounds; `loadu` needs no alignment. + unsafe { _mm256_loadu_pd(v.0.as_ptr().add(4 * half) as *const f64) } + }; + let mut out = [Self([0u64; 8]); 8]; + for bi in 0..2 { + for bj in 0..2 { + let h = |k: usize| load(&rows[4 * bi + k], bj); + // SAFETY: AVX is compiled in (module docs); register-only ops. + let c = unsafe { + let t0 = _mm256_unpacklo_pd(h(0), h(1)); + let t1 = _mm256_unpackhi_pd(h(0), h(1)); + let t2 = _mm256_unpacklo_pd(h(2), h(3)); + let t3 = _mm256_unpackhi_pd(h(2), h(3)); + [ + _mm256_permute2f128_pd::<0x20>(t0, t2), + _mm256_permute2f128_pd::<0x20>(t1, t3), + _mm256_permute2f128_pd::<0x31>(t0, t2), + _mm256_permute2f128_pd::<0x31>(t1, t3), + ] + }; + for (k, v) in c.into_iter().enumerate() { + // SAFETY: AVX is compiled in (module docs); `out[..].0` is + // `[u64; 8]`, so the 32-byte store at byte offset 0 or 32 + // is in bounds. + unsafe { _mm256_storeu_pd(out[4 * bj + k].0.as_mut_ptr().add(4 * bi) as *mut f64, v) }; + } + } + } + out + } +} + +/// Lane-wise variable left shift. Counts of 64 or more zero the lane, as +/// `VPSLLVQ` does on the AVX2 arm; callers keep counts below 64 (see the AVX2 +/// realization for the contract). No SSE/AVX1 form exists, so this is per lane. +impl Shl for U64x8 { + type Output = Self; + #[inline(always)] + fn shl(self, rhs: Self) -> Self { + debug_assert!(rhs.0.iter().all(|&n| n < 64), "U64x8 shift counts are a caller contract: < 64"); + Self(core::array::from_fn(|i| if rhs.0[i] < 64 { self.0[i] << rhs.0[i] } else { 0 })) + } +} + +/// Lane-wise variable right shift; see the `Shl` impl above. +impl Shr for U64x8 { + type Output = Self; + #[inline(always)] + fn shr(self, rhs: Self) -> Self { + debug_assert!(rhs.0.iter().all(|&n| n < 64), "U64x8 shift counts are a caller contract: < 64"); + Self(core::array::from_fn(|i| if rhs.0[i] < 64 { self.0[i] >> rhs.0[i] } else { 0 })) + } +} + +// --------------------------------------------------------------------------- +// I32x16 (array-backed) +// --------------------------------------------------------------------------- + +impl I32x16 { + /// Four 128-bit quarters of the 16 lanes. + #[inline(always)] + fn quarters(self) -> [__m128i; 4] { + // SAFETY: SSE2 is compiled in (module docs); `self.0` is `[i32; 16]` + // (64 bytes), so the 16-byte loads at byte offsets 0, 16, 32, 48 are in + // bounds; `loadu` needs no alignment. + core::array::from_fn(|k| unsafe { _mm_loadu_si128(self.0.as_ptr().add(4 * k) as *const __m128i) }) + } + + /// Horizontal signed minimum (`pminsd` tree, SSE4.1). Exact. + #[inline(always)] + pub fn reduce_min(self) -> i32 { + let [a, b, c, d] = self.quarters(); + // SAFETY: SSE2/SSE4.1 are compiled in (module docs); register-only ops. + unsafe { + let m4 = _mm_min_epi32(_mm_min_epi32(a, b), _mm_min_epi32(c, d)); + let m2 = _mm_min_epi32(m4, _mm_shuffle_epi32::<0b01_00_11_10>(m4)); + let m1 = _mm_min_epi32(m2, _mm_shuffle_epi32::<0b00_00_00_01>(m2)); + _mm_cvtsi128_si32(m1) + } + } + + /// Horizontal signed maximum, the `pmaxsd` twin of [`Self::reduce_min`]. + #[inline(always)] + pub fn reduce_max(self) -> i32 { + let [a, b, c, d] = self.quarters(); + // SAFETY: SSE2/SSE4.1 are compiled in (module docs); register-only ops. + unsafe { + let m4 = _mm_max_epi32(_mm_max_epi32(a, b), _mm_max_epi32(c, d)); + let m2 = _mm_max_epi32(m4, _mm_shuffle_epi32::<0b01_00_11_10>(m4)); + let m1 = _mm_max_epi32(m2, _mm_shuffle_epi32::<0b00_00_00_01>(m2)); + _mm_cvtsi128_si32(m1) + } + } + + /// Lane-wise signed greater-than as a 16-bit mask, LSB-first (lane 0 is + /// bit 0). Same contract as the AVX2 realization. + #[inline(always)] + pub fn gt_bitmask(self, other: Self) -> u16 { + let a = self.quarters(); + let b = other.quarters(); + let mut m = 0u32; + for k in 0..4 { + // SAFETY: SSE/SSE2 are compiled in (module docs); register-only ops. + let q = unsafe { _mm_movemask_ps(_mm_castsi128_ps(_mm_cmpgt_epi32(a[k], b[k]))) as u32 }; + m |= q << (4 * k); + } + m as u16 + } +} + +// --------------------------------------------------------------------------- +// U16x16 (native __m256i) +// --------------------------------------------------------------------------- + +impl U16x16 { + /// Logical right shift of each 16-bit lane by `imm` (any count; 16 or more + /// gives 0, as on the AVX2 arm). + #[inline(always)] + pub fn shr(self, imm: u32) -> Self { + let (lo, hi) = split(self.0); + // SAFETY: SSE2 is compiled in (module docs); register-only ops. + unsafe { + let c = _mm_cvtsi32_si128(imm as i32); + Self(join(_mm_srl_epi16(lo, c), _mm_srl_epi16(hi, c))) + } + } + + /// Logical left shift of each 16-bit lane by `imm` (any count). + #[inline(always)] + pub fn shl(self, imm: u32) -> Self { + let (lo, hi) = split(self.0); + // SAFETY: SSE2 is compiled in (module docs); register-only ops. + unsafe { + let c = _mm_cvtsi32_si128(imm as i32); + Self(join(_mm_sll_epi16(lo, c), _mm_sll_epi16(hi, c))) + } + } + + /// Multiply, keep the low 16 bits (wrapping). + #[inline(always)] + pub fn mullo(self, other: Self) -> Self { + Self(halves2!(_mm_mullo_epi16, self.0, other.0)) + } + + /// Cross-128-bit-lane permute; `IMM` has `_mm256_permute2x128_si256`'s + /// meaning. AVX1's `permute2f128` moves the same 128-bit halves. + #[inline(always)] + pub fn permute2x128(self, other: Self) -> Self { + // SAFETY: AVX is compiled in (module docs); register-only op. + Self(unsafe { _mm256_permute2f128_si256::(self.0, other.0) }) + } + + /// Blend 32-bit dwords from `self`/`other` per `IMM` + /// (`_mm256_blend_epi32`'s meaning; bit `i` set takes dword `i` from + /// `other`). AVX1's `blend_ps` selects the same dwords. + #[inline(always)] + pub fn blend_epi32(self, other: Self) -> Self { + // SAFETY: AVX is compiled in (module docs); casts and a blend on + // registers only. + Self(unsafe { + _mm256_castps_si256(_mm256_blend_ps::(_mm256_castsi256_ps(self.0), _mm256_castsi256_ps(other.0))) + }) + } + + /// Zero-extend the low 8 × u16 lanes to f32. + #[inline(always)] + pub fn to_f32x8_lo(self) -> F32x8 { + F32x8(widen_u16_to_f32(split(self.0).0)) + } + + /// Zero-extend the high 8 × u16 lanes to f32. + #[inline(always)] + pub fn to_f32x8_hi(self) -> F32x8 { + F32x8(widen_u16_to_f32(split(self.0).1)) + } +} + +/// 8 × u16 (one xmm) → 8 × f32 (one ymm), exact. +#[inline(always)] +fn widen_u16_to_f32(v: __m128i) -> __m256 { + // SAFETY: SSE2/SSE4.1 and AVX are compiled in (module docs); register-only + // ops. `pmovzxwd` widens the low four lanes; the byte shift moves the high + // four into place first. + unsafe { + let lo = _mm_cvtepu16_epi32(v); + let hi = _mm_cvtepu16_epi32(_mm_srli_si128::<8>(v)); + _mm256_cvtepi32_ps(join(lo, hi)) + } +} + +impl Add for U16x16 { + type Output = Self; + #[inline(always)] + fn add(self, rhs: Self) -> Self { + Self(halves2!(_mm_add_epi16, self.0, rhs.0)) + } +} +impl Sub for U16x16 { + type Output = Self; + #[inline(always)] + fn sub(self, rhs: Self) -> Self { + Self(halves2!(_mm_sub_epi16, self.0, rhs.0)) + } +} +impl Mul for U16x16 { + type Output = Self; + #[inline(always)] + fn mul(self, rhs: Self) -> Self { + self.mullo(rhs) + } +} +impl AddAssign for U16x16 { + #[inline(always)] + fn add_assign(&mut self, rhs: Self) { + *self = *self + rhs; + } +} +impl SubAssign for U16x16 { + #[inline(always)] + fn sub_assign(&mut self, rhs: Self) { + *self = *self - rhs; + } +} +impl BitAnd for U16x16 { + type Output = Self; + #[inline(always)] + fn bitand(self, rhs: Self) -> Self { + Self(bits_ps!(_mm256_and_ps, self.0, rhs.0)) + } +} +impl BitOr for U16x16 { + type Output = Self; + #[inline(always)] + fn bitor(self, rhs: Self) -> Self { + Self(bits_ps!(_mm256_or_ps, self.0, rhs.0)) + } +} +impl BitXor for U16x16 { + type Output = Self; + #[inline(always)] + fn bitxor(self, rhs: Self) -> Self { + Self(bits_ps!(_mm256_xor_ps, self.0, rhs.0)) + } +} +impl BitAndAssign for U16x16 { + #[inline(always)] + fn bitand_assign(&mut self, rhs: Self) { + *self = *self & rhs; + } +} +impl BitOrAssign for U16x16 { + #[inline(always)] + fn bitor_assign(&mut self, rhs: Self) { + *self = *self | rhs; + } +} +impl BitXorAssign for U16x16 { + #[inline(always)] + fn bitxor_assign(&mut self, rhs: Self) { + *self = *self ^ rhs; + } +} +impl Not for U16x16 { + type Output = Self; + #[inline(always)] + fn not(self) -> Self { + // SAFETY: AVX is compiled in (module docs); register-only op. + let ones = unsafe { _mm256_set1_epi16(-1) }; + Self(bits_ps!(_mm256_xor_ps, self.0, ones)) + } +} + +// --------------------------------------------------------------------------- +// U8x32 (native __m256i) +// --------------------------------------------------------------------------- + +impl U8x32 { + /// Horizontal byte sum as u64 (does not wrap at 2^8). + #[inline(always)] + pub fn sum_bytes_u64(self) -> u64 { + let (lo, hi) = split(self.0); + let mut t = [0u64; 4]; + // SAFETY: SSE2 is compiled in (module docs); `t` is `[u64; 4]` + // (32 bytes), so the two 16-byte stores at byte offsets 0 and 16 are + // in bounds. + unsafe { + let z = _mm_setzero_si128(); + _mm_storeu_si128(t.as_mut_ptr() as *mut __m128i, _mm_sad_epu8(lo, z)); + _mm_storeu_si128(t.as_mut_ptr().add(2) as *mut __m128i, _mm_sad_epu8(hi, z)); + } + t[0] + t[1] + t[2] + t[3] + } + + /// Lane-wise unsigned min. + #[inline(always)] + pub fn simd_min(self, other: Self) -> Self { + Self(halves2!(_mm_min_epu8, self.0, other.0)) + } + + /// Lane-wise unsigned max. + #[inline(always)] + pub fn simd_max(self, other: Self) -> Self { + Self(halves2!(_mm_max_epu8, self.0, other.0)) + } + + /// Per-lane equality as a 32-bit mask (bit `i` set iff `self[i] == other[i]`). + #[inline(always)] + pub fn cmpeq_mask(self, other: Self) -> u32 { + movemask_epi8_256(halves2!(_mm_cmpeq_epi8, self.0, other.0)) + } + + /// Per-lane unsigned greater-than as a 32-bit mask (signed compare after + /// flipping the sign bit of both operands, as on the AVX2 arm). + #[inline(always)] + pub fn cmpgt_mask(self, other: Self) -> u32 { + // SAFETY: AVX is compiled in (module docs); register-only op. + let bias = unsafe { _mm256_set1_epi8(i8::MIN) }; + let a = bits_ps!(_mm256_xor_ps, self.0, bias); + let b = bits_ps!(_mm256_xor_ps, other.0, bias); + movemask_epi8_256(halves2!(_mm_cmpgt_epi8, a, b)) + } + + /// MSB of each lane as a 32-bit mask. + #[inline(always)] + pub fn movemask(self) -> u32 { + movemask_epi8_256(self.0) + } + + /// Per-lane saturating unsigned add. + #[inline(always)] + pub fn saturating_add(self, other: Self) -> Self { + Self(halves2!(_mm_adds_epu8, self.0, other.0)) + } + + /// Per-lane saturating unsigned sub. + #[inline(always)] + pub fn saturating_sub(self, other: Self) -> Self { + Self(halves2!(_mm_subs_epu8, self.0, other.0)) + } + + /// Per-lane unsigned rounded average, `(a + b + 1) >> 1`. + #[inline(always)] + pub fn pairwise_avg(self, other: Self) -> Self { + Self(halves2!(_mm_avg_epu8, self.0, other.0)) + } + + /// Right shift each 16-bit lane by `imm` bits (any count). + #[inline(always)] + pub fn shr_epi16(self, imm: u32) -> Self { + let (lo, hi) = split(self.0); + // SAFETY: SSE2 is compiled in (module docs); register-only ops. + unsafe { + let c = _mm_cvtsi32_si128(imm as i32); + Self(join(_mm_srl_epi16(lo, c), _mm_srl_epi16(hi, c))) + } + } + + /// Left shift each 16-bit lane by `imm` bits (any count). + #[inline(always)] + pub fn shl_epi16(self, imm: u32) -> Self { + let (lo, hi) = split(self.0); + // SAFETY: SSE2 is compiled in (module docs); register-only ops. + unsafe { + let c = _mm_cvtsi32_si128(imm as i32); + Self(join(_mm_sll_epi16(lo, c), _mm_sll_epi16(hi, c))) + } + } + + /// Within-128-bit-lane byte shuffle with `_mm256_shuffle_epi8`'s meaning: + /// `pshufb` on each half is exactly that. + #[inline(always)] + pub fn shuffle_bytes(self, idx: Self) -> Self { + Self(halves2!(_mm_shuffle_epi8, self.0, idx.0)) + } + + /// Interleave the low 8 bytes of each 128-bit half. + #[inline(always)] + pub fn unpack_lo_epi8(self, other: Self) -> Self { + Self(halves2!(_mm_unpacklo_epi8, self.0, other.0)) + } + + /// Interleave the high 8 bytes of each 128-bit half. + #[inline(always)] + pub fn unpack_hi_epi8(self, other: Self) -> Self { + Self(halves2!(_mm_unpackhi_epi8, self.0, other.0)) + } + + /// Select `a` where the mask lane's MSB is set, else `b`. + #[inline(always)] + pub fn mask_blend(mask: Self, a: Self, b: Self) -> Self { + let (ml, mh) = split(mask.0); + let (al, ah) = split(a.0); + let (bl, bh) = split(b.0); + // SAFETY: SSE4.1 is compiled in (module docs); register-only ops. + Self(unsafe { join(_mm_blendv_epi8(bl, al, ml), _mm_blendv_epi8(bh, ah, mh)) }) + } +} + +impl BitAnd for U8x32 { + type Output = Self; + #[inline(always)] + fn bitand(self, rhs: Self) -> Self { + Self(bits_ps!(_mm256_and_ps, self.0, rhs.0)) + } +} +impl BitOr for U8x32 { + type Output = Self; + #[inline(always)] + fn bitor(self, rhs: Self) -> Self { + Self(bits_ps!(_mm256_or_ps, self.0, rhs.0)) + } +} +impl BitXor for U8x32 { + type Output = Self; + #[inline(always)] + fn bitxor(self, rhs: Self) -> Self { + Self(bits_ps!(_mm256_xor_ps, self.0, rhs.0)) + } +} +/// Wrapping add (use `saturating_add` to clamp). +impl Add for U8x32 { + type Output = Self; + #[inline(always)] + fn add(self, rhs: Self) -> Self { + Self(halves2!(_mm_add_epi8, self.0, rhs.0)) + } +} +/// Wrapping sub (use `saturating_sub` to clamp). +impl Sub for U8x32 { + type Output = Self; + #[inline(always)] + fn sub(self, rhs: Self) -> Self { + Self(halves2!(_mm_sub_epi8, self.0, rhs.0)) + } +} + +// --------------------------------------------------------------------------- +// I8x32 / I16x16 (native __m256i, defined in simd_avx512.rs) +// --------------------------------------------------------------------------- + +impl I8x32 { + /// Wrapping lane-wise add. + #[inline(always)] + pub fn add(self, other: Self) -> Self { + Self(halves2!(_mm_add_epi8, self.0, other.0)) + } + + /// Wrapping lane-wise sub. + #[inline(always)] + pub fn sub(self, other: Self) -> Self { + Self(halves2!(_mm_sub_epi8, self.0, other.0)) + } + + /// Lane-wise signed min (SSE4.1). + #[inline(always)] + pub fn min(self, other: Self) -> Self { + Self(halves2!(_mm_min_epi8, self.0, other.0)) + } + + /// Lane-wise signed max (SSE4.1). + #[inline(always)] + pub fn max(self, other: Self) -> Self { + Self(halves2!(_mm_max_epi8, self.0, other.0)) + } + + /// Lane-wise signed `self > other` as a 32-bit mask (lane 0 is bit 0). + #[inline(always)] + pub fn cmp_gt(self, other: Self) -> u32 { + movemask_epi8_256(halves2!(_mm_cmpgt_epi8, self.0, other.0)) + } + + /// Lane-wise `|x|` clamped to `i8::MAX` (`i8::MIN` → `i8::MAX`), via + /// SSSE3 `pabsb` then an unsigned min with `0x7f`. + #[inline(always)] + pub fn saturating_abs(self) -> Self { + let (lo, hi) = split(self.0); + // SAFETY: SSE2/SSSE3 are compiled in (module docs); register-only ops. + unsafe { + let cap = _mm_set1_epi8(0x7f); + Self(join(_mm_min_epu8(_mm_abs_epi8(lo), cap), _mm_min_epu8(_mm_abs_epi8(hi), cap))) + } + } +} + +impl Add for I8x32 { + type Output = Self; + #[inline(always)] + fn add(self, rhs: Self) -> Self { + I8x32::add(self, rhs) + } +} +impl Sub for I8x32 { + type Output = Self; + #[inline(always)] + fn sub(self, rhs: Self) -> Self { + I8x32::sub(self, rhs) + } +} +impl AddAssign for I8x32 { + #[inline(always)] + fn add_assign(&mut self, rhs: Self) { + *self = I8x32::add(*self, rhs); + } +} +impl SubAssign for I8x32 { + #[inline(always)] + fn sub_assign(&mut self, rhs: Self) { + *self = I8x32::sub(*self, rhs); + } +} + +impl I16x16 { + /// Wrapping lane-wise add. + #[inline(always)] + pub fn add(self, other: Self) -> Self { + Self(halves2!(_mm_add_epi16, self.0, other.0)) + } + + /// Wrapping lane-wise sub. + #[inline(always)] + pub fn sub(self, other: Self) -> Self { + Self(halves2!(_mm_sub_epi16, self.0, other.0)) + } + + /// Lane-wise signed min. + #[inline(always)] + pub fn min(self, other: Self) -> Self { + Self(halves2!(_mm_min_epi16, self.0, other.0)) + } + + /// Lane-wise signed max. + #[inline(always)] + pub fn max(self, other: Self) -> Self { + Self(halves2!(_mm_max_epi16, self.0, other.0)) + } + + /// Lane-wise signed `self > other` as a 16-bit mask (lane 0 is bit 0). + #[inline(always)] + pub fn cmp_gt(self, other: Self) -> u16 { + let (al, ah) = split(self.0); + let (bl, bh) = split(other.0); + // SAFETY: SSE2 is compiled in (module docs); register-only ops. + // `packs` narrows each all-ones/all-zeros 16-bit lane to one byte, + // low half's lanes first, so the byte mask is the lane mask in order. + unsafe { + let packed = _mm_packs_epi16(_mm_cmpgt_epi16(al, bl), _mm_cmpgt_epi16(ah, bh)); + _mm_movemask_epi8(packed) as u16 + } + } +} + +impl Add for I16x16 { + type Output = Self; + #[inline(always)] + fn add(self, rhs: Self) -> Self { + I16x16::add(self, rhs) + } +} +impl Sub for I16x16 { + type Output = Self; + #[inline(always)] + fn sub(self, rhs: Self) -> Self { + I16x16::sub(self, rhs) + } +} +impl AddAssign for I16x16 { + #[inline(always)] + fn add_assign(&mut self, rhs: Self) { + *self = I16x16::add(*self, rhs); + } +} +impl SubAssign for I16x16 { + #[inline(always)] + fn sub_assign(&mut self, rhs: Self) { + *self = I16x16::sub(*self, rhs); + } +} diff --git a/src/simd_avx2.rs b/src/simd_avx2.rs index 13adc6ae..ff8bb795 100644 --- a/src/simd_avx2.rs +++ b/src/simd_avx2.rs @@ -350,6 +350,12 @@ pub fn hamming_top_k( (indices, top_distances) } +// On the AVX-without-AVX2 arm these two come from `simd_avx.rs` (same +// signatures, SSSE3 realization); the AVX2 versions below are gated out. +#[cfg(all(target_feature = "avx", not(target_feature = "avx2")))] +pub use crate::simd_avx::{dot_i8, popcount}; + +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// AVX2 popcount using Harley-Seal vpshufb nibble lookup. pub fn popcount(a: &[u8]) -> u64 { #[cfg(target_arch = "x86_64")] @@ -402,53 +408,61 @@ pub fn popcount(a: &[u8]) -> u64 { } } -/// AVX2 int8 dot product using VPMADDUBSW + VPMADDWD with XOR-0x80 bias correction. +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] +/// Signed int8 dot product: the bytes of `a` and `b` are read as `i8`, and +/// the exact sum of products over the first `min(a.len(), b.len())` bytes is +/// returned. +/// +/// Each 32-byte chunk is sign-extended to i16 (`vpmovsxbw`) and multiplied +/// pairwise into i32 (`vpmaddwd`), which is exact for every i8 pair. The i32 +/// lanes are folded into an i64 every [`DOT_I8_FOLD`] chunks, so no input +/// length can wrap them. (The earlier `vpmaddubsw` form saturated i16 at +/// `a = b = 127`, and read `b` past its end when `b` was the shorter slice.) pub fn dot_i8(a: &[u8], b: &[u8]) -> i64 { - #[cfg(target_arch = "x86_64")] - { - use core::arch::x86_64::*; - unsafe { - let len = a.len(); - let chunks = len / 32; - let bias = _mm256_set1_epi8(-128i8); - let ones_u8 = _mm256_set1_epi8(1); - let ones_i16 = _mm256_set1_epi16(1); + use core::arch::x86_64::*; + let len = a.len().min(b.len()); + let (a, b) = (&a[..len], &b[..len]); + let mut total = 0i64; + for (ba, bb) in a.chunks(32 * DOT_I8_FOLD).zip(b.chunks(32 * DOT_I8_FOLD)) { + let (ca, cb) = (ba.chunks_exact(32), bb.chunks_exact(32)); + let (ta, tb) = (ca.remainder(), cb.remainder()); + // SAFETY: AVX2 is compiled in (this fn is gated off the + // AVX-without-AVX2 arm). Every chunk is exactly 32 bytes, so each + // 32-byte `loadu` is in bounds; `loadu` needs no alignment. + let lanes = unsafe { let mut acc = _mm256_setzero_si256(); - let mut b_sum = _mm256_setzero_si256(); - for i in 0..chunks { - let base = i * 32; - let av = _mm256_loadu_si256(a[base..].as_ptr() as *const __m256i); - let bv = _mm256_loadu_si256(b[base..].as_ptr() as *const __m256i); - let av_u = _mm256_xor_si256(av, bias); - let prod = _mm256_maddubs_epi16(av_u, bv); - let widened = _mm256_madd_epi16(prod, ones_i16); - acc = _mm256_add_epi32(acc, widened); - let b_abs = _mm256_maddubs_epi16(ones_u8, bv); - let b_wide = _mm256_madd_epi16(b_abs, ones_i16); - b_sum = _mm256_add_epi32(b_sum, b_wide); - } - let mut acc_vals = [0i32; 8]; - _mm256_storeu_si256(acc_vals.as_mut_ptr() as *mut __m256i, acc); - let total_biased: i64 = acc_vals.iter().map(|&v| v as i64).sum(); - let mut bsum_vals = [0i32; 8]; - _mm256_storeu_si256(bsum_vals.as_mut_ptr() as *mut __m256i, b_sum); - let total_b: i64 = bsum_vals.iter().map(|&v| v as i64).sum(); - let mut result = total_biased - 128 * total_b; - for i in (chunks * 32)..len { - result += (a[i] as i8 as i64) * (b[i] as i8 as i64); + for (xa, xb) in ca.zip(cb) { + let av = _mm256_loadu_si256(xa.as_ptr() as *const __m256i); + let bv = _mm256_loadu_si256(xb.as_ptr() as *const __m256i); + let lo = _mm256_madd_epi16( + _mm256_cvtepi8_epi16(_mm256_castsi256_si128(av)), + _mm256_cvtepi8_epi16(_mm256_castsi256_si128(bv)), + ); + let hi = _mm256_madd_epi16( + _mm256_cvtepi8_epi16(_mm256_extracti128_si256::<1>(av)), + _mm256_cvtepi8_epi16(_mm256_extracti128_si256::<1>(bv)), + ); + acc = _mm256_add_epi32(acc, _mm256_add_epi32(lo, hi)); } - result - } - } - #[cfg(not(target_arch = "x86_64"))] - { - a.iter() - .zip(b.iter()) + let mut v = [0i32; 8]; + _mm256_storeu_si256(v.as_mut_ptr() as *mut __m256i, acc); + v + }; + total += lanes.iter().map(|&v| v as i64).sum::(); + total += ta + .iter() + .zip(tb) .map(|(&x, &y)| (x as i8 as i64) * (y as i8 as i64)) - .sum() + .sum::(); } + total } +/// Chunks between i32 → i64 folds in [`dot_i8`]. One chunk adds at most four +/// products of magnitude `128 * 128` to an i32 lane (65_536), so 16_384 chunks +/// stay below `2^30`. +pub(crate) const DOT_I8_FOLD: usize = 16_384; + // ============================================================================ // GEMM — AVX2 fallback (delegates to scalar for now) // ============================================================================ @@ -1687,6 +1701,7 @@ impl U64x8 { Self(o) } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// `(x << n) | (x >> (64 - n))` per 64-bit lane on one 256-bit half, /// with `1 <= n <= 63` guaranteed by the callers. `_mm256_sll_epi64` / /// `_mm256_srl_epi64` take the count from the low 64 bits of an xmm @@ -1728,6 +1743,7 @@ impl U64x8 { Self::from_avx2_halves(Self::rotl_half(lo, 64 - n), Self::rotl_half(hi, 64 - n)) } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Lane-wise `lo32(self) × lo32(rhs)` as an exact `u64` — the widening /// 32×32→64 multiply (`VPMULUDQ`, one per 256-bit half). The high 32 /// bits of every input lane are ignored; the product cannot overflow. @@ -1742,6 +1758,7 @@ impl U64x8 { unsafe { Self::from_avx2_halves(_mm256_mul_epu32(a_lo, b_lo), _mm256_mul_epu32(a_hi, b_hi)) } } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// 8×8 transpose of `u64` words across eight registers: /// `out[i]` lane `j` == `rows[j]` lane `i`. /// @@ -1796,6 +1813,7 @@ impl U64x8 { } } +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Lane-wise variable shifts for the mask family's word ops (the Morton hex /// neighbour shift composes `and`/`shl`/`or` over `U64x8`). Same signature as /// the AVX-512 / NEON / WASM / scalar backends: the count is a per-lane @@ -1827,6 +1845,7 @@ impl Shl for U64x8 { } } +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Lane-wise variable right shift; see the `Shl` impl above for the /// count contract and a worked example (the two are inverses below 64). impl Shr for U64x8 { @@ -2099,6 +2118,7 @@ impl U16x16 { unsafe { _mm256_storeu_si256(s.as_mut_ptr() as *mut __m256i, self.0) }; } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Logical right shift each 16-bit lane by `imm` (matches `U16x32::shr`). #[inline(always)] pub fn shr(self, imm: u32) -> Self { @@ -2109,6 +2129,7 @@ impl U16x16 { Self(unsafe { _mm256_srl_epi16(self.0, _mm_cvtsi32_si128(imm as i32)) }) } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Logical left shift each 16-bit lane by `imm` (matches `U16x32::shl`). #[inline(always)] pub fn shl(self, imm: u32) -> Self { @@ -2118,6 +2139,7 @@ impl U16x16 { Self(unsafe { _mm256_sll_epi16(self.0, _mm_cvtsi32_si128(imm as i32)) }) } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Multiply, keep low 16 bits (wrapping) — `_mm256_mullo_epi16`. #[inline(always)] pub fn mullo(self, other: Self) -> Self { @@ -2132,6 +2154,7 @@ impl U16x16 { // ── FastScan flush-epilogue helpers (PQ4-ADC u16→f32 cross-lane combine) ── + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Cross-128-bit-lane permute (`_mm256_permute2x128_si256`). `IMM` selects /// which 128-bit halves of `self`/`other` land in each output half. Used /// (with `IMM=0x21`) by the FastScan SUB-trick to bring the two blocks' @@ -2143,6 +2166,7 @@ impl U16x16 { Self(unsafe { _mm256_permute2x128_si256::(self.0, other.0) }) } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Blend 32-bit dwords from `self`/`other` per the `IMM` mask /// (`_mm256_blend_epi32`). Companion to `permute2x128` in the FastScan /// lane combine (with `IMM=0xF0`). @@ -2153,6 +2177,7 @@ impl U16x16 { Self(unsafe { _mm256_blend_epi32::(self.0, other.0) }) } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Zero-extend the low 8 × u16 lanes to f32 (`_mm256_cvtepu16_epi32` then /// `_mm256_cvtepi32_ps`). The PQ4-ADC accumulators are ≤ `FLUSH_EVERY·127` /// so they fit exactly in f32; this is the lossless u16→f32 step before the @@ -2164,6 +2189,7 @@ impl U16x16 { crate::simd_avx512::F32x8(unsafe { _mm256_cvtepi32_ps(_mm256_cvtepu16_epi32(_mm256_castsi256_si128(self.0))) }) } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Zero-extend the high 8 × u16 lanes to f32 (sibling of `to_f32x8_lo`). #[inline(always)] pub fn to_f32x8_hi(self) -> crate::simd_avx512::F32x8 { @@ -2182,6 +2208,7 @@ impl Default for U16x16 { } } +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] impl Add for U16x16 { type Output = Self; #[inline(always)] @@ -2189,6 +2216,7 @@ impl Add for U16x16 { Self(unsafe { _mm256_add_epi16(self.0, rhs.0) }) } } +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] impl Sub for U16x16 { type Output = Self; #[inline(always)] @@ -2196,6 +2224,7 @@ impl Sub for U16x16 { Self(unsafe { _mm256_sub_epi16(self.0, rhs.0) }) } } +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] impl Mul for U16x16 { type Output = Self; #[inline(always)] @@ -2203,18 +2232,21 @@ impl Mul for U16x16 { Self(unsafe { _mm256_mullo_epi16(self.0, rhs.0) }) } } +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] impl AddAssign for U16x16 { #[inline(always)] fn add_assign(&mut self, rhs: Self) { self.0 = unsafe { _mm256_add_epi16(self.0, rhs.0) }; } } +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] impl SubAssign for U16x16 { #[inline(always)] fn sub_assign(&mut self, rhs: Self) { self.0 = unsafe { _mm256_sub_epi16(self.0, rhs.0) }; } } +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] impl BitAnd for U16x16 { type Output = Self; #[inline(always)] @@ -2222,6 +2254,7 @@ impl BitAnd for U16x16 { Self(unsafe { _mm256_and_si256(self.0, rhs.0) }) } } +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] impl BitOr for U16x16 { type Output = Self; #[inline(always)] @@ -2229,6 +2262,7 @@ impl BitOr for U16x16 { Self(unsafe { _mm256_or_si256(self.0, rhs.0) }) } } +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] impl BitXor for U16x16 { type Output = Self; #[inline(always)] @@ -2236,24 +2270,28 @@ impl BitXor for U16x16 { Self(unsafe { _mm256_xor_si256(self.0, rhs.0) }) } } +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] impl BitAndAssign for U16x16 { #[inline(always)] fn bitand_assign(&mut self, rhs: Self) { self.0 = unsafe { _mm256_and_si256(self.0, rhs.0) }; } } +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] impl BitOrAssign for U16x16 { #[inline(always)] fn bitor_assign(&mut self, rhs: Self) { self.0 = unsafe { _mm256_or_si256(self.0, rhs.0) }; } } +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] impl BitXorAssign for U16x16 { #[inline(always)] fn bitxor_assign(&mut self, rhs: Self) { self.0 = unsafe { _mm256_xor_si256(self.0, rhs.0) }; } } +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] impl Not for U16x16 { type Output = Self; #[inline(always)] @@ -2601,6 +2639,7 @@ impl I32x16 { } } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Horizontal signed minimum. `iter().min()` measured fully scalar on /// the codegen oracle (17 `cmpl` on GPRs, 0 packed), so this is a /// `vpminsd` tree: 16 → 8 → 4 → 2 → 1 lanes. Exact — min is order-free. @@ -2617,6 +2656,7 @@ impl I32x16 { } } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Horizontal signed maximum — the `vpmaxsd` twin of [`Self::reduce_min`]. #[inline(always)] pub fn reduce_max(self) -> i32 { @@ -2702,6 +2742,7 @@ impl I32x16 { !(neg as u16) } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Lane-wise **signed** greater-than as a packed 16-bit bitmask. /// /// Bit `i` of the result is set iff `self.lane(i) > other.lane(i)` under @@ -2904,6 +2945,7 @@ impl U8x32 { *arr.iter().max().unwrap() } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Sum-of-absolute-differences against zero ⇒ horizontal byte sum /// folded into the low 64 bits of each 128-bit lane, then combined. /// Returns the total as u64 (does NOT wrap at 2^8). Useful for @@ -2922,6 +2964,7 @@ impl U8x32 { // ── Min / max (lane-wise) ─────────────────────────────────────── + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Lane-wise unsigned min. #[inline(always)] pub fn simd_min(self, other: Self) -> Self { @@ -2930,6 +2973,7 @@ impl U8x32 { Self(unsafe { _mm256_min_epu8(self.0, other.0) }) } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Lane-wise unsigned max. #[inline(always)] pub fn simd_max(self, other: Self) -> Self { @@ -2940,6 +2984,7 @@ impl U8x32 { // ── Comparison → bitmask ──────────────────────────────────────── + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Per-lane equality. Returns a 32-bit mask: bit `i` set iff /// `self[i] == other[i]`. (Matches the shape of `U8x64::cmpeq_mask` /// at the natural AVX2 width.) @@ -2953,6 +2998,7 @@ impl U8x32 { unsafe { _mm256_movemask_epi8(eq) as u32 } } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Per-lane unsigned greater-than. Returns a 32-bit mask. /// AVX2 only has signed `_mm256_cmpgt_epi8`, so we XOR both /// operands with `0x80` to convert unsigned ↔ signed (preserves @@ -2970,6 +3016,7 @@ impl U8x32 { } } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Extract MSB of each lane as a 32-bit mask (matches /// `U8x64::movemask` at AVX2 width). #[inline(always)] @@ -2981,6 +3028,7 @@ impl U8x32 { // ── Saturating arithmetic ──────────────────────────────────────── + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Per-lane saturating unsigned add: `min(a + b, 255)`. #[inline(always)] pub fn saturating_add(self, other: Self) -> Self { @@ -2989,6 +3037,7 @@ impl U8x32 { Self(unsafe { _mm256_adds_epu8(self.0, other.0) }) } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Per-lane saturating unsigned sub: `max(a - b, 0)`. #[inline(always)] pub fn saturating_sub(self, other: Self) -> Self { @@ -2997,6 +3046,7 @@ impl U8x32 { Self(unsafe { _mm256_subs_epu8(self.0, other.0) }) } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Per-lane unsigned rounded average: `(a + b + 1) >> 1`. #[inline(always)] pub fn pairwise_avg(self, other: Self) -> Self { @@ -3007,6 +3057,7 @@ impl U8x32 { // ── 16-bit-lane shifts (used by nibble pack/unpack) ───────────── + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Right shift each 16-bit lane by `imm` bits. (AVX2 has no native /// 8-bit shift; 16-bit shift + mask is the standard idiom.) #[inline(always)] @@ -3017,6 +3068,7 @@ impl U8x32 { Self(unsafe { _mm256_srl_epi16(self.0, _mm_cvtsi32_si128(imm as i32)) }) } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Left shift each 16-bit lane by `imm` bits. #[inline(always)] pub fn shl_epi16(self, imm: u32) -> Self { @@ -3027,6 +3079,7 @@ impl U8x32 { // ── Lane shuffles ─────────────────────────────────────────────── + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Within-128-bit-lane byte shuffle. `idx[i]` (0..16) selects the /// source byte within the SAME 128-bit half; high-bit set in /// `idx[i]` zeroes the output lane. Matches `_mm256_shuffle_epi8`. @@ -3056,6 +3109,7 @@ impl U8x32 { Self::from_array(out) } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Interleave low 8 bytes of each 128-bit half (`_mm256_unpacklo_epi8`). /// Output: `[a0,b0, a1,b1, ..., a7,b7]` within each 128-bit half. #[inline(always)] @@ -3065,6 +3119,7 @@ impl U8x32 { Self(unsafe { _mm256_unpacklo_epi8(self.0, other.0) }) } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Interleave high 8 bytes of each 128-bit half (`_mm256_unpackhi_epi8`). #[inline(always)] pub fn unpack_hi_epi8(self, other: Self) -> Self { @@ -3075,6 +3130,7 @@ impl U8x32 { // ── Conditional move via bit mask ─────────────────────────────── + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Select `a` where mask bit is set, else `b`. The mask is a /// `U8x32` whose lane MSB acts as the boolean (matches /// `_mm256_blendv_epi8` semantics — different from the @@ -3113,6 +3169,7 @@ impl U8x32 { // Bitwise + arithmetic operator impls so consumers can use natural // `a + b`, `a & b`, etc. without method chaining. Match the U8x64 shape. +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] #[cfg(target_arch = "x86_64")] impl core::ops::BitAnd for U8x32 { type Output = Self; @@ -3124,6 +3181,7 @@ impl core::ops::BitAnd for U8x32 { } } +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] #[cfg(target_arch = "x86_64")] impl core::ops::BitOr for U8x32 { type Output = Self; @@ -3135,6 +3193,7 @@ impl core::ops::BitOr for U8x32 { } } +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] #[cfg(target_arch = "x86_64")] impl core::ops::BitXor for U8x32 { type Output = Self; @@ -3146,6 +3205,7 @@ impl core::ops::BitXor for U8x32 { } } +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] #[cfg(target_arch = "x86_64")] impl core::ops::Add for U8x32 { type Output = Self; @@ -3157,6 +3217,7 @@ impl core::ops::Add for U8x32 { } } +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] #[cfg(target_arch = "x86_64")] impl core::ops::Sub for U8x32 { type Output = Self; diff --git a/src/simd_avx512.rs b/src/simd_avx512.rs index 4ff2aaf9..31f3b3ef 100644 --- a/src/simd_avx512.rs +++ b/src/simd_avx512.rs @@ -1410,9 +1410,20 @@ impl F32x8 { /// ``` #[inline(always)] pub fn mul_add(self, a: Self, b: Self) -> Self { - // SAFETY: FMA3 intrinsic; reached only on FMA-capable targets via the - // consumer's runtime dispatch / `#[target_feature(enable = "fma")]`. - Self(unsafe { _mm256_fmadd_ps(self.0, a.0, b.0) }) + #[cfg(target_feature = "fma")] + { + // SAFETY: `_mm256_fmadd_ps` needs FMA3, which `cfg(target_feature + // = "fma")` proves is enabled for this whole build, so every CPU + // the binary may run on has it (`cpu_guard` checks at startup). + Self(unsafe { _mm256_fmadd_ps(self.0, a.0, b.0) }) + } + // AVX without FMA (the `simd_avx.rs` arm, e.g. Sandy Bridge): a fused + // multiply-add per lane, same single rounding as the FMA3 form. + #[cfg(not(target_feature = "fma"))] + { + let (x, y, z) = (self.to_array(), a.to_array(), b.to_array()); + Self::from_array(core::array::from_fn(|i| x[i].mul_add(y[i], z[i]))) + } } /// Lane-wise `self > other` as an 8-bit mask: bit `i` set iff @@ -2115,26 +2126,31 @@ impl I8x32 { unsafe { _mm256_storeu_si256(s.as_mut_ptr() as *mut __m256i, self.0) }; } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] #[inline(always)] pub fn add(self, other: Self) -> Self { Self(unsafe { _mm256_add_epi8(self.0, other.0) }) } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] #[inline(always)] pub fn sub(self, other: Self) -> Self { Self(unsafe { _mm256_sub_epi8(self.0, other.0) }) } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] #[inline(always)] pub fn min(self, other: Self) -> Self { Self(unsafe { _mm256_min_epi8(self.0, other.0) }) } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] #[inline(always)] pub fn max(self, other: Self) -> Self { Self(unsafe { _mm256_max_epi8(self.0, other.0) }) } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Compare-greater-than: returns 32-bit mask via packed-byte movemask. /// Bit i set where self[i] > other[i]. #[inline(always)] @@ -2143,6 +2159,7 @@ impl I8x32 { } } +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] impl Add for I8x32 { type Output = Self; #[inline(always)] @@ -2150,6 +2167,7 @@ impl Add for I8x32 { Self(unsafe { _mm256_add_epi8(self.0, rhs.0) }) } } +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] impl Sub for I8x32 { type Output = Self; #[inline(always)] @@ -2157,12 +2175,14 @@ impl Sub for I8x32 { Self(unsafe { _mm256_sub_epi8(self.0, rhs.0) }) } } +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] impl AddAssign for I8x32 { #[inline(always)] fn add_assign(&mut self, rhs: Self) { self.0 = unsafe { _mm256_add_epi8(self.0, rhs.0) }; } } +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] impl SubAssign for I8x32 { #[inline(always)] fn sub_assign(&mut self, rhs: Self) { @@ -2315,26 +2335,31 @@ impl I16x16 { unsafe { _mm256_storeu_si256(s.as_mut_ptr() as *mut __m256i, self.0) }; } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] #[inline(always)] pub fn add(self, other: Self) -> Self { Self(unsafe { _mm256_add_epi16(self.0, other.0) }) } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] #[inline(always)] pub fn sub(self, other: Self) -> Self { Self(unsafe { _mm256_sub_epi16(self.0, other.0) }) } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] #[inline(always)] pub fn min(self, other: Self) -> Self { Self(unsafe { _mm256_min_epi16(self.0, other.0) }) } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] #[inline(always)] pub fn max(self, other: Self) -> Self { Self(unsafe { _mm256_max_epi16(self.0, other.0) }) } + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Compare-greater-than: returns 16-bit mask via packed-word movemask. /// Bit i set where self[i] > other[i]. #[inline(always)] @@ -2352,6 +2377,7 @@ impl I16x16 { } } +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] impl Add for I16x16 { type Output = Self; #[inline(always)] @@ -2359,6 +2385,7 @@ impl Add for I16x16 { Self(unsafe { _mm256_add_epi16(self.0, rhs.0) }) } } +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] impl Sub for I16x16 { type Output = Self; #[inline(always)] @@ -2366,12 +2393,14 @@ impl Sub for I16x16 { Self(unsafe { _mm256_sub_epi16(self.0, rhs.0) }) } } +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] impl AddAssign for I16x16 { #[inline(always)] fn add_assign(&mut self, rhs: Self) { self.0 = unsafe { _mm256_add_epi16(self.0, rhs.0) }; } } +#[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] impl SubAssign for I16x16 { #[inline(always)] fn sub_assign(&mut self, rhs: Self) { @@ -3285,6 +3314,7 @@ pub fn palette_lookup_u8x8(idx_v: U16x8, lut: &[u8]) -> U8x8 { // 2. _mm256_min_epu8 (VPMINUB) clamps 0x80 → 0x7f. impl I8x32 { + #[cfg(not(all(target_feature = "avx", not(target_feature = "avx2"))))] /// Lane-wise saturating absolute value. /// /// `saturating_abs(i8::MIN) == i8::MAX` (127). Uses the VPABSB +