Skip to content

Util.invPow2: limit exp to 0..1022, the range it computes exactly - #816

Open
leerho wants to merge 1 commit into
mainfrom
invpow2-range
Open

leerho wants to merge 1 commit into
mainfrom
invpow2-range

Conversation

@leerho

@leerho leerho commented Nov 22, 2026

Copy link
Copy Markdown
Member

Util.invPow2(exp) builds 2^(-exp) from its IEEE 754 bits: Double.longBitsToDouble((1023L - exp) << 52). That is exact for exp 0..1022. At exp = 1023 the exponent field becomes 0 and the result is 0.0, not 2^(-1023), which is a subnormal and can't be built this way. The javadoc and the range check allowed 1023.

Change:

  • the range check now rejects exp > 1022 ((exp | (1022 - exp)) < 0), with the message updated;
  • the javadoc gives the range 0..1022, says why, and adds @throws.

Callers: HLL, HllMap and CPC all pass exp below 64, so their behavior is unchanged.

Tests (UtilTest):

  • 1023 and Integer.MAX_VALUE are rejected;
  • invPow2(1022) equals Double.MIN_NORMAL;
  • every exp from 0 to 1022 matches Math.scalb(1.0, -exp) bit for bit.

The full suite passes (2,449). javac -Xlint:all -Xdoclint:all,-missing adds no warnings to Util.java.

🤖 Generated with Claude Code

At exp = 1023 the exponent field is 0, so the method returned 0.0 instead
of the subnormal 2^-1023. All callers use exp < 64.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant