Repository navigation
Collection of small fixes part two - #765
Merged
Merged
Conversation
solidpixel
force-pushed
the
opt-two-plane-channel-weights
branch
2 times, most recently
from
November 13, 2026 07:32
206efe4 to
10f9d7e
Compare
RGB-scale endpoints share a chroma direction and differ only in magnitude. Refitting these magnitudes must account for each channel's error weight and interpolation plane. The previous two-plane fit used plane 1 for all RGB channels, even when an RGB channel used plane 2. The one-plane and alpha-separated fits also multiplied unweighted RGB projections by the sum of channel weights, which cancelled out of the solve and ignored relative channel importance. Keep the mean RGB direction and solve for the low and high magnitudes using channel-weighted least squares. Weight each matrix contribution by the channel error weight and squared direction component, selecting the matrix terms from that channel's actual weight plane. Form the right-hand side by projecting the existing channel-weighted color sums onto the RGB direction. This also removes the separate per-texel accumulation for the scale fit. Run the RGB-scale solve independently of the first plane's endpoint solve. A varying RGB second plane can constrain the magnitudes even when plane 1 is constant, so only force a unit scale when the RGB plane is constant and plane 2 contains alpha. Retain the determinant and validity checks and the projected-range fallback for rejected fits.
Partition ranking combines color-fitting error with an estimate of error introduced by weight quantization. The fitting term uses the configured channel weights, but the quantization term assumed uniform weights. This biased candidate selection when channel importance differed, and even allowed a common scale applied to all channel weights to change rankings. Apply the channel weights to the quantization estimate before summing the squared line components. Replicate the weights into each candidate quad in the batched RGB and grayscale RGBA paths, and apply the same weighting in the scalar RGB and RGBA paths. Preserve arithmetic ordering between scalar and SIMD scoring. Unit channel weights retain the existing scores.
HDR RGB delta modes encode C as an unsigned difference between the major channel endpoints. When that channel decreases across the fitted ramp, clamping C to zero can erase variation and produce a poor endpoint fit. The direct encoding can preserve this variation, but its coarser precision can increase error when C is only slightly negative. For negative C, decode the accepted clamped delta encoding and the direct fallback, then select the one with lower RGB endpoint squared error in LNS space. Keep the delta encoding on ties and leave selection for nonnegative C unchanged.
Convert normalized weight error into color error using the squared endpoint distance and the channel-weighted squared fitted direction. The two-, three-, and four-component paths previously used the average channel weight, treating equal-length red and green ranges as equally important even when their channel weights differ. Compare the resulting weighted scales when checking whether the scale is constant across partitions. This allows the shared-scale decimation path only when the partitions have equal error sensitivity, keeping weight-grid fitting and candidate ranking consistent with the color-error metric.
When a partition's RGB weights are constant, endpoint separation is indeterminate and the best constant color is the partition mean. The RGB-scale refit instead retained the maximum projected color while setting the scale ratio to one, biasing the reconstructed colors toward brighter texels. Use the mean RGB color in the single-plane and alpha-separated two-plane refits, and the mean luminance in the L+A specialization. This primarily improves low-bitrate encodings, especially larger footprints such as 12x12, where coarse weight grids more often collapse varying texels to constant weights.
Project texels onto the fitted color line using the channel error weights when computing ideal interpolation weights. Normalize by the directional weighted error so the projection minimizes the configured squared error for the existing endpoint segment. This primarily benefits normal maps and perceptual modes, where channel weights differ substantially.
The HDR RGB endpoint encoder quantizes signed D0 and D1 color deltas using a rounding helper that assumes nonnegative inputs. Adding 0.5 and then truncating toward zero biases negative deltas upward by one quantization step, even for exactly representable integer values. Use floor after adding the rounding bias to handle both signs correctly while preserving the existing upward rounding of halfway cases. Check the asymmetric signed field bounds explicitly. An n-bit signed field includes -2^(n-1) but excludes +2^(n-1); the absolute-value check incorrectly rejected the most negative encodable delta. Retaining these legal values allows higher-precision HDR endpoint modes to remain usable.
Constant weights within a partition cannot determine both the RGB base color and the shared offset. The singular normal-equation solve can still produce finite, incorrect coefficients, which pass its NaN-only fallback check and discard useful HDR endpoint candidates. Use the mean RGB color with zero offset before solving these systems, following the existing endpoint refit's weight-constancy threshold. Apply this to both single-plane and dual-plane refits. Alpha variation cannot constrain the RGB offset, while variation in either RGB weight plane must retain the analytic fit.
Quantize both large-range HDR luminance candidates before comparing their reconstructed endpoint errors. Quantization can reverse the error ranking of the unquantized candidates, leading to a worse encoded representation. Reconstruct the candidates using the decoder's endpoint-ordering rule, including equal stored values selecting the upper submode. Select and store the quantized candidate with the smaller endpoint error.
solidpixel
force-pushed
the
opt-two-plane-channel-weights
branch
from
November 13, 2026 07:37
10f9d7e to
1a3c2e6
Compare
The ASTC specification does not require endpoint ordering for LDR luminance direct mode (CEM 0). Preserve descending endpoints instead of collapsing them to their average.
RGB and RGBA direct modes use the relative RGB endpoint sums to select blue contraction. Even at QUANT_256, rounded endpoints can violate the direct ordering constraint, forcing the fallback encoder to adjust their values. Try blue contraction for these endpoint pairs and compare its decoded error with direct encoding. Skip the full-precision attempt when the rounded endpoints already satisfy the direct ordering, and preserve the existing candidate search at lower quantization levels.
Endpoint formats are selected before iterative refinement, but the general refitters also compute auxiliary endpoint models that the chosen formats cannot use. Pass the selected formats into the one- and two-plane refitters. Only fit RGB-scale endpoints for the LDR scale formats, and RGB-offset endpoints for the HDR offset format. Preserve the ordinary endpoint solve and existing luminance-alpha specialization.
solidpixel
force-pushed
the
opt-two-plane-channel-weights
branch
from
November 13, 2026 12:52
1a3c2e6 to
f7b85c2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This collection of fixes focusses on correct encoding steering choices, fixing some "forever bugs" in the compressor search heuristics and error weighting. These give a small, but measurable, improvement in PSNR*. Some images even manage to compress a little faster.
Many improvements have been found in encodings with unequally weighted channels, where previous behavior was not always considering the channel error weight on all selection and refinement paths. This improves encoding quality for compression using
-normal,-perceptual,-alphaweight, and manually set-cwor-sswchannel weighting options.Many improvements have also been found for HDR endpoint fitting and selection.
Smaller number of minor improvements to the generic code path help RGB and RGBA LDR textures, but to a lesser degree.
* some images regress, as always, but average improvement is positive. Expect to need to update image references due to the small PSNR regressions.
Do not squash ...