Reject NaN and out-of-range blur radii

[lazerg]

Fixes #9900. Alternative to #9906

Changes proposed in this pull request:

• ImagingBoxBlur() only rejected negative radii, so NaN slipped through and (int)floatRadius was undefined behaviour, giving a garbage radius and the out of bounds write in ImagingLineBoxBlur32(). Infinity and any radius of 2**31 or more crash the same way, since they are not representable as an int either.
• The radius is now rejected unless it is at least 0 and small enough to convert to an int, which covers GaussianBlur too.

[akx]

To a casual reader, this looks like a very odd way to write the thing. If the idea is that this catches NaNs by... off the top of my head, some quirk of how IEEE floats work, a comment would be nice.

EDIT: a comment got added just as I was writing this. Nice!

Even nicer, though, I think, some sort of reusable float-validation function since I'm quite sure there are many other instances of this same class of bug in Pillow.

[fallenmi]

[P1] Avoid signed overflow below the new radius limit

The new 2**31 guard keeps the float-to-int conversion in range, but it still accepts radii for which the existing integer arithmetic immediately overflows. ImagingHorizontalBoxBlur() computes (radius * 2 + 1) * ww with radius as an int; for radius = 2**30, radius * 2 is already outside the range of signed int even though the new check accepts it.

I reproduced this on exact head ed5a26fb0caa987f95b539494ae6b26fc5f6c551 with a UBSan build using a 3x3 L image and ImageFilter.BoxBlur(2**30):

src/libImaging/BoxBlur.c:179:38: runtime error: signed integer overflow: 1073741824 * 2 cannot be represented in type 'int'

The three new out-of-range tests pass, but they only exercise NaN, infinity, and 2**31; the same UBSan failure remains after a clean merge onto current main e41083f383c9cd3db95de52564cc0b6452313d4a. Please either reject radii before this multiplication can overflow (for the current expression, no later than (INT_MAX - 1) / 2) or perform the arithmetic in a type that cannot overflow, and add a regression around 2**30.

This review was prepared with OpenAI Codex assistance under the account owner's authorization. The exact diff, source path, merge result, and test evidence above were inspected and verified in the account owner's Codex session before submission.

[lazerg]

Confirmed. A UBSan build reproduces BoxBlur.c:179:38 at radius 2**30.

The same problem also starts lower down. acc = lineIn[0] * (radius + 1) (lines 43-46, 55-58, 126 and 130) overflows once 255 * (radius + 1) passes INT_MAX, so from radius 8421504. A (INT_MAX - 1) / 2 bound would still leave those in place.

I made the arithmetic unsigned instead of lowering the limit. acc, ww and fw are already UINT32, so only the int operands needed a cast, and every radius the guard accepts keeps the same result.

Added test_large_blur_filter_radius over 2**24 and 2**30 for L and RGB. It aborts under UBSan on the previous commit and passes now. Full suite still green.

ac5f69b

[fallenmi]

Thanks — this resolves my prior UBSan blocker. At exact head ac5f69b73c6086d2dc0865389d5afed322c73dc1, the multiplications that previously promoted through signed int now enter unsigned arithmetic before the operation in both the 8-bit and 32-bit paths. The existing binary32 conversion and 2**31 guard keep the remaining cast, radius + 1, edge calculations, and indices representable for every admitted radius.

The new 2**24 and 2**30 regressions cover both L and RGB. I rechecked current main c9e4cff871b19688789a9da4827d5e993204174c and synthetic merge 35501a1a5cd91d64051f36bb28c49ec84b83552d; the merge preserves the fix and exact test blob. All 55 check runs, 11 suites, nine Actions workflows, and two legacy statuses are green.

Disclosure: I used OpenAI Codex to assist this changed-head source/API review. I verified the exact public diff, source paths, refs, interactions, policy, merge, and CI metadata; I did not rerun a local build for this follow-up.