Correct AV1 12-bit inverse SIMD arithmetic

HEIF_IMPLEMENTATION_PLAN.md

@@ -29,13 +29,15 @@ Checkboxes may be marked complete only when the implementation and the verificat
 
 ## Delivery dashboard
 
-Last reconciled with the source tree on 2026-08-29 against production checkpoint `019ac5648b4380a6ff4865072c96fde29ce09fad`. Committed checkpoints include the AV1 transform architecture, OBU framing, intra-block copy, 12-profile reconstruction matrix, layered-item properties, layered reference/header/CDF/motion-field state, inter-frame intra blocks, SIMD-first translational prediction, complete single-reference inter reconstruction, compound reference trees and modes, paired reference-MV derivation, reference-dependent bounded sequence decoding, allocation-free SIMD-first equal averaging, selected inter-intra prediction, selectable compound blending, OBMC, scaled-reference reconstruction, local warped prediction, non-translational global prediction, official motion-vector conformance, and selected spatial-layer presentation. This dashboard is the authoritative delivery order. The detailed phase checklists below provide subsystem evidence; they do not override the current-stage marker or permit work to skip ahead.
+Last reconciled with the source tree on 2026-08-29 against production checkpoint `beb2ab86ce5e4052c667df30af8a48a860814031`. Committed checkpoints include the AV1 transform architecture, OBU framing, intra-block copy, 12-profile reconstruction matrix, layered-item properties, layered reference/header/CDF/motion-field state, inter-frame intra blocks, SIMD-first translational prediction, complete single-reference inter reconstruction, compound reference trees and modes, paired reference-MV derivation, reference-dependent bounded sequence decoding, allocation-free SIMD-first equal averaging, selected inter-intra prediction, selectable compound blending, OBMC, scaled-reference reconstruction, local warped prediction, non-translational global prediction, official motion-vector conformance, selected spatial-layer presentation, and progressive color and auxiliary-alpha conformance. This dashboard is the authoritative delivery order. The detailed phase checklists below provide subsystem evidence; they do not override the current-stage marker or permit work to skip ahead.
 
 Commit `1c58d855f70b024170ced9eb0a7005f0f9c955ad` records the complete official motion-vector conformance checkpoint. The official IVF has SHA-1 `F064290D7FCD3B3DE19020E8AEC6C43C88D3A505`, matching the pinned libaom test-data manifest, and SHA-256 `222A9050059B254DAB17CFB802FF829C778E3F93AF18961A622C8268576C1395`; its pinned-libaom Y4M has SHA-256 `D97AC78C81782CF1507549368047769DC677DBE205706458D1EE9C807DE6EC78`. Both source targets build with zero warnings and errors, the `net10.0` test-project analyzer build completes with zero errors and 1,014 pre-existing repository warnings, Roslynk reports zero compiler errors, 3,985 focused `net10.0` cases pass without failures or skips, and `git diff --check` is clean.
 
 Commit `019ac5648b4380a6ff4865072c96fde29ce09fad` records the complete selected-spatial-layer presentation checkpoint. An essential `lsel` selecting the genuine 40x40 base layer of the committed 40x40-to-80x80 progressive AVIF returns that exact pinned-libaom native reconstruction, then scales its YUV444 planes to the 80x80 `ispe` extent before color conversion and matches pinned libavif's RGBA output exactly. Retained reconstructed planes remain at coded dimensions; presentation planes use group-safe allocator-owned rows and are returned exactly once under a 1,024-byte constrained allocator. Portable `Vector128` and scalar two-times filters match pinned libyuv edge, quarter-sample, and rounding results at 8 and 12 bits through `FeatureTestRunner`. Both source targets build with zero warnings and errors, the `net10.0` test-project analyzer build completes with zero errors and 1,014 pre-existing repository warnings, Roslynk reports zero compiler errors, 156 focused reconstruction and color cases pass without failures or skips, and `git diff --check` is clean.
 
-The working tree contains a complete, verified progressive color and auxiliary-alpha coverage checkpoint awaiting commit. Both pinned progressive AVIF fixtures pass their exact native-plane and pinned-libavif presentation comparisons through normal dispatch, AVX-512-disabled, AVX-disabled, and scalar `FeatureTestRunner` configurations. The YUV444-alpha fixture also compares every composed alpha sample with the independent final native auxiliary plane. Both complete production presentation paths pass under a 1,024-byte constrained tracked allocator with every allocation returned exactly once. Both source targets build with zero warnings and errors, the `net10.0` test-project analyzer build completes with zero errors and 1,014 pre-existing repository warnings, Roslynk reports zero compiler errors, 156 focused reconstruction and color cases pass without failures or skips, and `git diff --check` is clean.
+Commit `beb2ab86ce5e4052c667df30af8a48a860814031` records the complete progressive color and auxiliary-alpha conformance checkpoint. Both pinned progressive AVIF fixtures pass their exact native-plane and pinned-libavif presentation comparisons through normal dispatch, AVX-512-disabled, AVX-disabled, and scalar `FeatureTestRunner` configurations. The YUV444-alpha fixture also compares every composed alpha sample with the independent final native auxiliary plane. Both complete production presentation paths pass under a 1,024-byte constrained tracked allocator with every allocation returned exactly once. Both source targets build with zero warnings and errors, the `net10.0` test-project analyzer build completes with zero errors and 1,014 pre-existing repository warnings, Roslynk reports zero compiler errors, 156 focused reconstruction and color cases pass without failures or skips, and `git diff --check` is clean.
+
+The working tree contains a complete, verified twelve-bit inverse-transform arithmetic checkpoint awaiting commit. ADST4 retains pinned libaom's signed 32-bit sine products and factorized sums and widens only its terminal rounding; Identity4 and Identity16 widen their fixed-point product and rounding bias only for the 20-bit twelve-bit row stage. The established 8/10-bit and twelve-bit column paths remain unchanged. Exact boundary vectors cover both 128-bit and 256-bit operators through normal, AVX-512-disabled, AVX-disabled, and scalar `FeatureTestRunner` configurations. Both source targets build with zero warnings and errors, the `net10.0` test-project analyzer build completes with zero errors and 1,014 pre-existing repository warnings, Roslynk reports zero compiler errors, 510 focused inverse-transform cases and 44 production reconstruction cases pass without failures or skips, and `git diff --check` is clean.
 
 Status meanings:
 
@@ -47,13 +49,13 @@ Status meanings:
 
 Current development stage: **Stage 3 — complete AV1 still-image decoding.** The decoder retains reference/header/CDF/motion-field state, derives frame-level skip-mode references, consumes temporal segment prediction, decodes intra-coded blocks inside inter frames, and reconstructs translational single-reference, compound, inter-intra, OBMC, scaled-reference, local warped, and non-translational global prediction before residual traversal. Commit `1c58d855f70b024170ced9eb0a7005f0f9c955ad` adds exact official four-frame coverage through all ordinary inter modes, all three motion modes, every regular/smooth/sharp dual-filter pair, sub-8x8 chroma prediction, and no-round compound intermediates. Commit `019ac5648b4380a6ff4865072c96fde29ce09fad` adds exact selected-layer native reconstruction and scaled presentation. Neither AV1 nor HEVC production encoding is implemented.
 
-Immediate checkpoint: **commit the verified progressive color and auxiliary-alpha coverage before advancing.** Both pinned progressive fixtures now pass exact production presentation under every required dispatch fallback and constrained tracked allocation; the alpha-bearing fixture additionally passes an independent sample-for-sample auxiliary-plane comparison.
+Immediate checkpoint: **commit the verified twelve-bit inverse-transform arithmetic correction before advancing.** The corrected operators match pinned libaom at the exact overflow boundaries, preserve narrower stage paths, and pass the complete focused transform and production reconstruction evidence.
 
 | Order | Delivery stage | State | Delivered state | Gate that remains open |
 | --- | --- | --- | --- | --- |
 | 1 | Baseline, provenance, documentation, and public contract | In progress | Pinned codec references, a bounded image-only scope, encoder options, typed bit depth, decoder-option propagation, and extensive HEIF documentation exist. | Complete the all-file documentation audit, record a fresh Release baseline, finish distinct public HEIC/AVIF save boundaries, and close API review. |
 | 2 | Bounded HEIF item and image-sequence container | In progress | Still-item parsing, grids, auxiliary alpha, metadata properties, bounded image-sequence tracks, Identify, and all-sync AV1 sequence presentation are connected. | Complete adversarial boundary coverage, remaining item/property behavior, reference-dependent sequence reconstruction, and the bounded sequence writer. |
-| 3 | Still-image AV1 and HEVC decoding | **Current** | HEVC reconstruction reaches exact HM/libheif fixtures across the recorded 8/10/12-bit and chroma cases. AV1 work includes bounded OBU framing, reconstruction, filters, grain, color, transforms, intra-block copy, an exact independent 12-profile bit-depth/chroma matrix through every dispatch tier, retained layered reference/header/CDF/motion-field state, temporal segment prediction, inter-frame intra blocks, SIMD-first translational prediction, exact single-reference inter reconstruction, exact equal-average and selectable compound reconstruction, inter-intra prediction, committed OBMC, scaled-reference reconstruction, local warped prediction, non-translational global prediction, selected spatial-layer presentation, and verified progressive color and auxiliary-alpha output. | Correct the audited 12-bit inverse-transform arithmetic, remove every other valid AV1 still-image unsupported branch with independent vectors, and then complete the remaining HEVC profile and Range Extensions matrix. |
+| 3 | Still-image AV1 and HEVC decoding | **Current** | HEVC reconstruction reaches exact HM/libheif fixtures across the recorded 8/10/12-bit and chroma cases. AV1 work includes bounded OBU framing, reconstruction, filters, grain, color, corrected twelve-bit inverse transforms, intra-block copy, an exact independent 12-profile bit-depth/chroma matrix through every dispatch tier, retained layered reference/header/CDF/motion-field state, temporal segment prediction, inter-frame intra blocks, SIMD-first translational prediction, exact single-reference inter reconstruction, exact equal-average and selectable compound reconstruction, inter-intra prediction, committed OBMC, scaled-reference reconstruction, local warped prediction, non-translational global prediction, selected spatial-layer presentation, and verified progressive color and auxiliary-alpha output. | Remove every other valid AV1 still-image unsupported branch with independent vectors; then complete the remaining HEVC profile and Range Extensions matrix. |
 | 4 | Complete decoded presentation and animation | In progress | Shared SIMD-first AV1/HEVC color conversion, ICC application, grids, transforms, direct planar alpha composition, frame metadata, repetition, and independently decodable AV1 sequence samples exist. | Close the full color/ICC cross-product, HEVC sequence decoding, AV1/HEVC reference-dependent samples, frame-local metadata/alpha behavior, and independent animated decode vectors. |
 | 5 | AV1/AVIF encoding | Not started | RGB-to-planar conversion, forward transforms, OBU writer foundations, options, and container-writing infrastructure exist. | `HeifEncoderCore` still rejects AV1. Implement a real independently decodable lossy/lossless AV1 payload and the complete AVIF item/metadata matrix. |
 | 6 | HEVC/HEIC encoding | Not started | Shared input color conversion, options, and HEIF writer infrastructure exist. | `HeifEncoderCore` still rejects HEVC. Implement a real independently decodable lossy/lossless HEVC payload and the complete HEIC item/metadata matrix. |
@@ -104,8 +106,8 @@ Immediate checkpoint: **commit the verified progressive color and auxiliary-alph
     - [x] Verify every connected inter mode and filter with exact independent content. The official pinned-libaom sequence supplies complete bit-mask coverage for all twelve ordinary inter modes, all three motion modes, and all nine regular/smooth/sharp dual-filter pairs. The committed average, selectable compound, inter-intra, OBMC, scaled-reference, local-warp, and global-warp AVIF fixtures separately require every connected reconstruction variant and pass exact native-plane comparison; the presentation-bearing fixtures also pass exact pinned-libavif output.
   - [x] Return the explicitly selected spatial layer or the final displayed layer, keeping reference reconstruction separate from display-only film grain. The essential-`lsel` production fixture reconstructs the selected 40x40 YUV444 base exactly, scales native component planes to the 80x80 item extent with pinned-libyuv integer rounding, and matches pinned-libavif RGBA presentation under normal/scalar dispatch. The committed final-layer fixture and film-grain matrix remain exact. Constrained tracked allocation returns every short-lived presentation plane exactly once. Commit `019ac5648b4380a6ff4865072c96fde29ce09fad` records the checkpoint.
   - [x] Verify color and auxiliary-alpha output exactly against both pinned libavif progressive fixtures under normal SIMD dispatch and all required `FeatureTestRunner` fallbacks. Both fixtures pass exact public presentation under normal, AVX-512-disabled, AVX-disabled, and scalar dispatch; the alpha-bearing fixture additionally matches every composed alpha sample with its pinned native auxiliary plane. Both full production paths pass under a 1,024-byte constrained tracked allocator with exactly-once returns. Both Release source targets build with zero warnings and errors, the test-project analyzer build completes with zero errors and 1,014 pre-existing warnings, Roslynk reports zero compiler errors, and all 156 focused reconstruction and color cases pass without failures or skips.
-- [ ] **Queued until the progressive color and auxiliary-alpha checkpoint commit:** correct the audited 12-bit inverse ADST4, Identity4, and Identity16 SIMD arithmetic by widening only the libaom-widened multiply/accumulate operations, with exact conformant-range vectors and `FeatureTestRunner` coverage.
-- [ ] Continue inventorying and removing every remaining valid AV1 still-image unsupported branch, adding exact independent compression-tool fixtures to the profile-matrix regression gate.
+- [x] Correct the audited 12-bit inverse ADST4, Identity4, and Identity16 SIMD arithmetic by widening only the libaom-widened multiply/accumulate operations, with exact conformant-range vectors and `FeatureTestRunner` coverage. Both 128-bit and 256-bit operators match pinned outputs through every required hardware fallback; 510 focused inverse-transform cases, 44 production reconstruction cases, both zero-warning source builds, the zero-error analyzer build, zero Roslynk compiler errors, and clean `git diff --check` complete the verification.
+- [ ] **Queued until the twelve-bit inverse-transform checkpoint commit:** continue inventorying and removing every remaining valid AV1 still-image unsupported branch, adding exact independent compression-tool fixtures to the profile-matrix regression gate.
 - [ ] Complete the remaining HEVC still-image profile and Range Extensions matrix with exact independent native-plane and presentation evidence.
 - [ ] Close shared decoded presentation, ICC, alpha, grid, transform, metadata, and animated AV1/HEVC decode gates.
 - [ ] Implement and independently verify real AV1/AVIF still encoding.
@@ -838,7 +840,8 @@ The dashboard and immediate execution queue define the remaining critical path.
 - [x] Implement and independently verify local warped motion through pinned projection, direct 8/10/12-bit predictor coverage, a genuine traced two-frame bounded AVIF, exact final native and presentation comparisons, constrained allocation, and normal/scalar dispatch. Commit `0b2db86809e26757460e719a6b59c0fe58bb0586` records the checkpoint.
 - [x] Implement and independently verify non-translational global prediction with a genuine traced bounded sequence, exact native and presentation comparisons, constrained allocation, normal/scalar dispatch, and direct 8/10/12-bit compound production coverage. Commit `c5637ea0187df35b385bf43e2fe85cd955f01099` records the checkpoint.
 - [x] Complete progressive color and auxiliary-alpha verification through exact native and presentation comparisons, every required `FeatureTestRunner` fallback, and constrained tracked allocation.
-- [ ] **Queued until the progressive color and auxiliary-alpha checkpoint commit:** correct the audited 12-bit inverse-transform arithmetic, remove every other unsupported valid AV1 still-image syntax path, and prove the complete AVIF decode matrix with independent inputs and scalar/SIMD parity.
+- [x] Correct the audited 12-bit inverse ADST4, Identity4, and Identity16 arithmetic through exact pinned boundary vectors and SIMD/scalar feature isolation.
+- [ ] **Queued until the twelve-bit inverse-transform checkpoint commit:** remove every other unsupported valid AV1 still-image syntax path and prove the complete AVIF decode matrix with independent inputs and scalar/SIMD parity.
 - [ ] Close Phase 4 by completing the remaining HEVC profile and Range Extensions matrix with exact native-plane and presented-image evidence.
 - [ ] Close Phase 5 and the decode portion of the bounded sequence ledger: color, ICC, alpha, grids, presentation transforms, reference-dependent samples, and complete animated AVIF/HEIC decode.
 - [ ] Close the still-image portions of Phases 0, 1, and 2 that remain as release gates: documentation, provenance, public format boundaries, API review, parser hardening, and malformed-input coverage.

src/ImageSharp/Formats/Heif/Av1/Transform/Av1IdentityTransform1d.cs

@@ -70,6 +70,40 @@ internal static class Av1IdentityTransform1d
         }
     }
 
+    /// <summary>
+    /// Scales four independent identity-transform axes with widened fixed-point products.
+    /// </summary>
+    /// <param name="input">The source values for four transform axes.</param>
+    /// <param name="output">The destination values for four transform axes.</param>
+    /// <param name="length">The number of values in each axis.</param>
+    /// <param name="multiplier">The fixed-point identity scale.</param>
+    /// <param name="fractionalBits">The number of fractional bits in <paramref name="multiplier"/>.</param>
+    [MethodImpl(MethodImplOptions.AggressiveInlining)]
+    public static void TransformWidened(ref Av1TransformVector<Vector128<int>> input, ref Av1TransformVector<Vector128<int>> output, int length, int multiplier, int fractionalBits)
+    {
+        for (int i = 0; i < length; i++)
+        {
+            output[i] = Av1Transform1dMath.MultiplyRoundWidened(input[i], multiplier, fractionalBits);
+        }
+    }
+
+    /// <summary>
+    /// Scales eight independent identity-transform axes with widened fixed-point products.
+    /// </summary>
+    /// <param name="input">The source values for eight transform axes.</param>
+    /// <param name="output">The destination values for eight transform axes.</param>
+    /// <param name="length">The number of values in each axis.</param>
+    /// <param name="multiplier">The fixed-point identity scale.</param>
+    /// <param name="fractionalBits">The number of fractional bits in <paramref name="multiplier"/>.</param>
+    [MethodImpl(MethodImplOptions.AggressiveInlining)]
+    public static void TransformWidened(ref Av1TransformVector<Vector256<int>> input, ref Av1TransformVector<Vector256<int>> output, int length, int multiplier, int fractionalBits)
+    {
+        for (int i = 0; i < length; i++)
+        {
+            output[i] = Av1Transform1dMath.MultiplyRoundWidened(input[i], multiplier, fractionalBits);
+        }
+    }
+
     /// <summary>
     /// Scales sixteen independent identity-transform axes in parallel.
     /// </summary>

src/ImageSharp/Formats/Heif/Av1/Transform/Av1Inverse2dTransformer.Adst4Operator.cs

@@ -84,9 +84,8 @@ internal static partial class Av1Inverse2dTransformer
             int cosBit,
             Av1TransformStageRange stageRange)
         {
-            TransformCore(ref input, ref output, cosBit);
+            TransformCore(ref input, ref output, cosBit, stageRange[0] >= WidenedIntermediateBitCount);
             _ = step;
-            _ = stageRange;
         }
 
         /// <inheritdoc/>
@@ -97,9 +96,8 @@ internal static partial class Av1Inverse2dTransformer
             int cosBit,
             Av1TransformStageRange stageRange)
         {
-            TransformCore(ref input, ref output, cosBit);
+            TransformCore(ref input, ref output, cosBit, stageRange[0] >= WidenedIntermediateBitCount);
             _ = step;
-            _ = stageRange;
         }
 
         /// <summary>
@@ -108,7 +106,12 @@ internal static partial class Av1Inverse2dTransformer
         /// <param name="input">The source values for four transform axes.</param>
         /// <param name="output">The destination values for four transform axes.</param>
         /// <param name="cosBit">The fixed-point precision of the sine constants.</param>
-        private static void TransformCore(ref Av1TransformVector<Vector128<int>> input, ref Av1TransformVector<Vector128<int>> output, int cosBit)
+        /// <param name="widenedRound">Whether the terminal fixed-point rounding requires signed 64-bit lanes.</param>
+        private static void TransformCore(
+            ref Av1TransformVector<Vector128<int>> input,
+            ref Av1TransformVector<Vector128<int>> output,
+            int cosBit,
+            bool widenedRound)
         {
             ReadOnlySpan<int> sinpi = Av1SinusConstants.SinusPi(cosBit);
             Vector128<int> x0 = input.V0;
@@ -116,8 +119,18 @@ internal static partial class Av1Inverse2dTransformer
             Vector128<int> x2 = input.V2;
             Vector128<int> x3 = input.V3;
 
-            // The products retain the sine-table scale across the complete matrix. The bounded transform inputs make
-            // the optimized kernels' wrapping 32-bit multiply/add sequence valid until the terminal rounding shift.
+            // Pinned libaom retains the sine-table scale in Int32 products and sums, but performs the twelve-bit row
+            // kernel's terminal scaling and rounding in Int64. This is the only stage whose rounding bias can overflow
+            // a valid Int32 fixed-point sum.
+            if (widenedRound)
+            {
+                output.V0 = Av1Transform1dMath.MultiplyAdd4WidenedRound(sinpi[1], x0, sinpi[3], x1, sinpi[4], x2, sinpi[2], x3, cosBit);
+                output.V1 = Av1Transform1dMath.MultiplyAdd4WidenedRound(sinpi[2], x0, sinpi[3], x1, -sinpi[1], x2, -sinpi[4], x3, cosBit);
+                output.V2 = Av1Transform1dMath.MultiplyAdd4WidenedRound(sinpi[3], x0, 0, x1, -sinpi[3], x2, sinpi[3], x3, cosBit);
+                output.V3 = Av1Transform1dMath.MultiplyAdd4WidenedRound(sinpi[1] + sinpi[2], x0, -sinpi[3], x1, sinpi[4] - sinpi[1], x2, sinpi[2] - sinpi[4], x3, cosBit);
+                return;
+            }
+
             output.V0 = Av1Transform1dMath.MultiplyAdd4(sinpi[1], x0, sinpi[3], x1, sinpi[4], x2, sinpi[2], x3, cosBit);
             output.V1 = Av1Transform1dMath.MultiplyAdd4(sinpi[2], x0, sinpi[3], x1, -sinpi[1], x2, -sinpi[4], x3, cosBit);
             output.V2 = Av1Transform1dMath.MultiplyAdd4(sinpi[3], x0, 0, x1, -sinpi[3], x2, sinpi[3], x3, cosBit);
@@ -130,7 +143,12 @@ internal static partial class Av1Inverse2dTransformer
         /// <param name="input">The source values for eight transform axes.</param>
         /// <param name="output">The destination values for eight transform axes.</param>
         /// <param name="cosBit">The fixed-point precision of the sine constants.</param>
-        private static void TransformCore(ref Av1TransformVector<Vector256<int>> input, ref Av1TransformVector<Vector256<int>> output, int cosBit)
+        /// <param name="widenedRound">Whether the terminal fixed-point rounding requires signed 64-bit lanes.</param>
+        private static void TransformCore(
+            ref Av1TransformVector<Vector256<int>> input,
+            ref Av1TransformVector<Vector256<int>> output,
+            int cosBit,
+            bool widenedRound)
         {
             ReadOnlySpan<int> sinpi = Av1SinusConstants.SinusPi(cosBit);
             Vector256<int> x0 = input.V0;
@@ -138,6 +156,15 @@ internal static partial class Av1Inverse2dTransformer
             Vector256<int> x2 = input.V2;
             Vector256<int> x3 = input.V3;
 
+            if (widenedRound)
+            {
+                output.V0 = Av1Transform1dMath.MultiplyAdd4WidenedRound(sinpi[1], x0, sinpi[3], x1, sinpi[4], x2, sinpi[2], x3, cosBit);
+                output.V1 = Av1Transform1dMath.MultiplyAdd4WidenedRound(sinpi[2], x0, sinpi[3], x1, -sinpi[1], x2, -sinpi[4], x3, cosBit);
+                output.V2 = Av1Transform1dMath.MultiplyAdd4WidenedRound(sinpi[3], x0, 0, x1, -sinpi[3], x2, sinpi[3], x3, cosBit);
+                output.V3 = Av1Transform1dMath.MultiplyAdd4WidenedRound(sinpi[1] + sinpi[2], x0, -sinpi[3], x1, sinpi[4] - sinpi[1], x2, sinpi[2] - sinpi[4], x3, cosBit);
+                return;
+            }
+
             output.V0 = Av1Transform1dMath.MultiplyAdd4(sinpi[1], x0, sinpi[3], x1, sinpi[4], x2, sinpi[2], x3, cosBit);
             output.V1 = Av1Transform1dMath.MultiplyAdd4(sinpi[2], x0, sinpi[3], x1, -sinpi[1], x2, -sinpi[4], x3, cosBit);
             output.V2 = Av1Transform1dMath.MultiplyAdd4(sinpi[3], x0, 0, x1, -sinpi[3], x2, sinpi[3], x3, cosBit);

src/ImageSharp/Formats/Heif/Av1/Transform/Av1Inverse2dTransformer.Identity16Operator.cs

@@ -48,10 +48,19 @@ internal static partial class Av1Inverse2dTransformer
             int cosBit,
             Av1TransformStageRange stageRange)
         {
-            Av1IdentityTransform1d.Transform(ref input, ref output, 16, 2 * Av1Transform1dMath.NewSqrt2, Av1Transform1dMath.NewSqrt2Bits);
+            // The doubled scale exceeds Int32 only for the 20-bit twelve-bit row range. Widen that exact product and
+            // rounding sequence, matching libaom without changing the established lower-range SIMD path.
+            if (stageRange[0] >= WidenedIntermediateBitCount)
+            {
+                Av1IdentityTransform1d.TransformWidened(ref input, ref output, 16, 2 * Av1Transform1dMath.NewSqrt2, Av1Transform1dMath.NewSqrt2Bits);
+            }
+            else
+            {
+                Av1IdentityTransform1d.Transform(ref input, ref output, 16, 2 * Av1Transform1dMath.NewSqrt2, Av1Transform1dMath.NewSqrt2Bits);
+            }
+
             _ = step;
             _ = cosBit;
-            _ = stageRange;
         }
 
         /// <inheritdoc/>
@@ -62,10 +71,17 @@ internal static partial class Av1Inverse2dTransformer
             int cosBit,
             Av1TransformStageRange stageRange)
         {
-            Av1IdentityTransform1d.Transform(ref input, ref output, 16, 2 * Av1Transform1dMath.NewSqrt2, Av1Transform1dMath.NewSqrt2Bits);
+            if (stageRange[0] >= WidenedIntermediateBitCount)
+            {
+                Av1IdentityTransform1d.TransformWidened(ref input, ref output, 16, 2 * Av1Transform1dMath.NewSqrt2, Av1Transform1dMath.NewSqrt2Bits);
+            }
+            else
+            {
+                Av1IdentityTransform1d.Transform(ref input, ref output, 16, 2 * Av1Transform1dMath.NewSqrt2, Av1Transform1dMath.NewSqrt2Bits);
+            }
+
             _ = step;
             _ = cosBit;
-            _ = stageRange;
         }
     }
 }

src/ImageSharp/Formats/Heif/Av1/Transform/Av1Inverse2dTransformer.Identity4Operator.cs

@@ -48,10 +48,19 @@ internal static partial class Av1Inverse2dTransformer
             int cosBit,
             Av1TransformStageRange stageRange)
         {
-            Av1IdentityTransform1d.Transform(ref input, ref output, 4, Av1Transform1dMath.NewSqrt2, Av1Transform1dMath.NewSqrt2Bits);
+            // Only a twelve-bit row transform has the 20-bit input range that can overflow this fixed-point product.
+            // Match libaom's high-bit-depth kernel there while retaining the compact Int32 path for narrower ranges.
+            if (stageRange[0] >= WidenedIntermediateBitCount)
+            {
+                Av1IdentityTransform1d.TransformWidened(ref input, ref output, 4, Av1Transform1dMath.NewSqrt2, Av1Transform1dMath.NewSqrt2Bits);
+            }
+            else
+            {
+                Av1IdentityTransform1d.Transform(ref input, ref output, 4, Av1Transform1dMath.NewSqrt2, Av1Transform1dMath.NewSqrt2Bits);
+            }
+
             _ = step;
             _ = cosBit;
-            _ = stageRange;
         }
 
         /// <inheritdoc/>
@@ -62,10 +71,17 @@ internal static partial class Av1Inverse2dTransformer
             int cosBit,
             Av1TransformStageRange stageRange)
         {
-            Av1IdentityTransform1d.Transform(ref input, ref output, 4, Av1Transform1dMath.NewSqrt2, Av1Transform1dMath.NewSqrt2Bits);
+            if (stageRange[0] >= WidenedIntermediateBitCount)
+            {
+                Av1IdentityTransform1d.TransformWidened(ref input, ref output, 4, Av1Transform1dMath.NewSqrt2, Av1Transform1dMath.NewSqrt2Bits);
+            }
+            else
+            {
+                Av1IdentityTransform1d.Transform(ref input, ref output, 4, Av1Transform1dMath.NewSqrt2, Av1Transform1dMath.NewSqrt2Bits);
+            }
+
             _ = step;
             _ = cosBit;
-            _ = stageRange;
         }
     }
 }

src/ImageSharp/Formats/Heif/Av1/Transform/Av1Inverse2dTransformer.Operator.cs

@@ -18,6 +18,11 @@ namespace SixLabors.ImageSharp.Formats.Heif.Av1.Transform;
 /// </remarks>
 internal static partial class Av1Inverse2dTransformer
 {
+    /// <summary>
+    /// The signed stage width whose fixed-point terminal operations require widened SIMD intermediates.
+    /// </summary>
+    private const byte WidenedIntermediateBitCount = 20;
+
     /// <summary>
     /// Defines the scalar and SIMD arithmetic for one AV1 one-dimensional inverse transform.
     /// </summary>

src/ImageSharp/Formats/Heif/Av1/Transform/Av1Transform1dMath.cs

@@ -396,6 +396,44 @@ internal static class Av1Transform1dMath
     public static Vector512<int> MultiplyRound(Vector512<int> value, int multiplier, int fractionalBits)
         => HalfButterfly(multiplier, value, 0, Vector512<int>.Zero, fractionalBits);
 
+    /// <summary>
+    /// Multiplies and rounds four fixed-point values with signed sixty-four-bit intermediate lanes.
+    /// </summary>
+    /// <param name="value">The four values to scale.</param>
+    /// <param name="multiplier">The fixed-point multiplier.</param>
+    /// <param name="fractionalBits">The number of fractional bits in the multiplier.</param>
+    /// <returns>The four rounded results.</returns>
+    [MethodImpl(MethodImplOptions.AggressiveInlining)]
+    public static Vector128<int> MultiplyRoundWidened(Vector128<int> value, int multiplier, int fractionalBits)
+    {
+        (Vector128<long> lower, Vector128<long> upper) = Vector128.Widen(value);
+        Vector128<long> rounding = Vector128.Create(1L << (fractionalBits - 1));
+
+        // Pinned libaom's high-bit-depth identity kernels multiply in signed 64-bit lanes. Widen before both the
+        // product and rounding addition so a valid 20-bit twelve-bit row value cannot wrap through Int32.
+        lower = ((lower * multiplier) + rounding) >> fractionalBits;
+        upper = ((upper * multiplier) + rounding) >> fractionalBits;
+        return Vector128.Narrow(lower, upper);
+    }
+
+    /// <summary>
+    /// Multiplies and rounds eight fixed-point values with signed sixty-four-bit intermediate lanes.
+    /// </summary>
+    /// <param name="value">The eight values to scale.</param>
+    /// <param name="multiplier">The fixed-point multiplier.</param>
+    /// <param name="fractionalBits">The number of fractional bits in the multiplier.</param>
+    /// <returns>The eight rounded results.</returns>
+    [MethodImpl(MethodImplOptions.AggressiveInlining)]
+    public static Vector256<int> MultiplyRoundWidened(Vector256<int> value, int multiplier, int fractionalBits)
+    {
+        (Vector256<long> lower, Vector256<long> upper) = Vector256.Widen(value);
+        Vector256<long> rounding = Vector256.Create(1L << (fractionalBits - 1));
+
+        lower = ((lower * multiplier) + rounding) >> fractionalBits;
+        upper = ((upper * multiplier) + rounding) >> fractionalBits;
+        return Vector256.Narrow(lower, upper);
+    }
+
     /// <summary>
     /// Multiplies four scalar inputs by fixed-point weights and rounds their sum.
     /// </summary>
@@ -504,4 +542,64 @@ internal static class Av1Transform1dMath
         Vector512<int> weightedSum = (input0 * weight0) + (input1 * weight1) + (input2 * weight2) + (input3 * weight3);
         return (weightedSum + Vector512.Create(1 << (fractionalBits - 1))) >> fractionalBits;
     }
+
+    /// <summary>
+    /// Multiplies four sets of four inputs in signed thirty-two-bit lanes, then widens the terminal rounding step.
+    /// </summary>
+    /// <param name="weight0">The first fixed-point weight.</param>
+    /// <param name="input0">The first four input values.</param>
+    /// <param name="weight1">The second fixed-point weight.</param>
+    /// <param name="input1">The second four input values.</param>
+    /// <param name="weight2">The third fixed-point weight.</param>
+    /// <param name="input2">The third four input values.</param>
+    /// <param name="weight3">The fourth fixed-point weight.</param>
+    /// <param name="input3">The fourth four input values.</param>
+    /// <param name="fractionalBits">The number of fractional bits in each weight.</param>
+    /// <returns>The four rounded fixed-point sums.</returns>
+    [MethodImpl(MethodImplOptions.AggressiveInlining)]
+    public static Vector128<int> MultiplyAdd4WidenedRound(
+        int weight0,
+        Vector128<int> input0,
+        int weight1,
+        Vector128<int> input1,
+        int weight2,
+        Vector128<int> input2,
+        int weight3,
+        Vector128<int> input3,
+        int fractionalBits)
+    {
+        // libaom keeps conformant ADST4 sine products and their factorized sums in Int32, then widens the terminal
+        // scaling and rounding. Preserve that exact boundary instead of widening every transform multiplication.
+        Vector128<int> weightedSum = (input0 * weight0) + (input1 * weight1) + (input2 * weight2) + (input3 * weight3);
+        return MultiplyRoundWidened(weightedSum, 1, fractionalBits);
+    }
+
+    /// <summary>
+    /// Multiplies four sets of eight inputs in signed thirty-two-bit lanes, then widens the terminal rounding step.
+    /// </summary>
+    /// <param name="weight0">The first fixed-point weight.</param>
+    /// <param name="input0">The first eight input values.</param>
+    /// <param name="weight1">The second fixed-point weight.</param>
+    /// <param name="input1">The second eight input values.</param>
+    /// <param name="weight2">The third fixed-point weight.</param>
+    /// <param name="input2">The third eight input values.</param>
+    /// <param name="weight3">The fourth fixed-point weight.</param>
+    /// <param name="input3">The fourth eight input values.</param>
+    /// <param name="fractionalBits">The number of fractional bits in each weight.</param>
+    /// <returns>The eight rounded fixed-point sums.</returns>
+    [MethodImpl(MethodImplOptions.AggressiveInlining)]
+    public static Vector256<int> MultiplyAdd4WidenedRound(
+        int weight0,
+        Vector256<int> input0,
+        int weight1,
+        Vector256<int> input1,
+        int weight2,
+        Vector256<int> input2,
+        int weight3,
+        Vector256<int> input3,
+        int fractionalBits)
+    {
+        Vector256<int> weightedSum = (input0 * weight0) + (input1 * weight1) + (input2 * weight2) + (input3 * weight3);
+        return MultiplyRoundWidened(weightedSum, 1, fractionalBits);
+    }
 }

tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InverseTransformTests.cs

@@ -41,6 +41,13 @@ public class Av1InverseTransformTests
     public void IdentityOperatorsProduceIdenticalScalarAndSimdResults()
         => FeatureTestRunner.RunWithHwIntrinsicsFeature(AssertIdentityOperatorParity, TransformConfigurations);
 
+    /// <summary>
+    /// Verifies the pinned-libaom widened operations at the twelve-bit inverse row-stage bounds.
+    /// </summary>
+    [Fact]
+    public void TwelveBitWideIntermediatesMatchPinnedLibaom()
+        => FeatureTestRunner.RunWithHwIntrinsicsFeature(AssertTwelveBitWideIntermediateParity, TransformConfigurations);
+
     /// <summary>
     /// Verifies the inverse DCT operators against their scalar implementations.
     /// </summary>
@@ -74,6 +81,133 @@ public class Av1InverseTransformTests
         AssertOperatorParity<Av1Inverse2dTransformer.Identity32Operator>(32);
     }
 
+    /// <summary>
+    /// Exercises the exact ADST4 rounding and identity-product overflows that are possible at a twenty-bit row range.
+    /// </summary>
+    private static void AssertTwelveBitWideIntermediateParity()
+    {
+        const int cosBit = 12;
+        Av1TransformStageRange stageRange = default;
+        for (int index = 0; index < Av1Transform2dFlipConfiguration.MaxStageNumber; index++)
+        {
+            stageRange[index] = 20;
+        }
+
+        Av1TransformVector<Vector128<int>> adstInput128 = default;
+        adstInput128.V0 = Vector128.Create(196_118, -196_118, 196_117, -196_117);
+        adstInput128.V1 = Vector128.Create(196_117, -196_117, 196_117, -196_117);
+        adstInput128.V2 = Vector128.Create(196_117, -196_117, 196_117, -196_117);
+        adstInput128.V3 = Vector128.Create(196_117, -196_117, 196_117, -196_117);
+        Av1TransformVector<Vector256<int>> adstInput256 = default;
+        adstInput256.V0 = Vector256.Create(196_118, -196_118, 196_117, -196_117, 196_118, -196_118, 196_117, -196_117);
+        adstInput256.V1 = Vector256.Create(196_117, -196_117, 196_117, -196_117, 196_117, -196_117, 196_117, -196_117);
+        adstInput256.V2 = adstInput256.V1;
+        adstInput256.V3 = adstInput256.V1;
+        Av1TransformVector<Vector128<int>> adstOutput128 = default;
+        Av1TransformVector<Vector128<int>> adstStep128 = default;
+        Av1TransformVector<Vector256<int>> adstOutput256 = default;
+        Av1TransformVector<Vector256<int>> adstStep256 = default;
+
+        Av1Inverse2dTransformer.Adst4Operator.Transform(
+            ref adstInput128,
+            ref adstOutput128,
+            ref adstStep128,
+            cosBit,
+            stageRange);
+
+        Av1Inverse2dTransformer.Adst4Operator.Transform(
+            ref adstInput256,
+            ref adstOutput256,
+            ref adstStep256,
+            cosBit,
+            stageRange);
+
+        // These are the exact outputs of pinned libaom's signed Int64 terminal round. The first positive lane has an
+        // Int32 fixed-point sum of 2,147,482,471, so adding the 2,048 rounding bias in Int32 would wrap.
+        Vector128<int> adstExpected0 = Vector128.Create(524_288, -524_288, 524_287, -524_287);
+        Vector128<int> adstExpected1 = Vector128.Create(33_612, -33_612, 33_612, -33_612);
+        Vector128<int> adstExpected2 = Vector128.Create(160_112, -160_112, 160_111, -160_111);
+        Vector128<int> adstExpected3 = Vector128.Create(77_567, -77_567, 77_566, -77_566);
+        Assert.Equal(adstExpected0, adstOutput128.V0);
+        Assert.Equal(adstExpected1, adstOutput128.V1);
+        Assert.Equal(adstExpected2, adstOutput128.V2);
+        Assert.Equal(adstExpected3, adstOutput128.V3);
+        Assert.Equal(Vector256.Create(adstExpected0, adstExpected0), adstOutput256.V0);
+        Assert.Equal(Vector256.Create(adstExpected1, adstExpected1), adstOutput256.V1);
+        Assert.Equal(Vector256.Create(adstExpected2, adstExpected2), adstOutput256.V2);
+        Assert.Equal(Vector256.Create(adstExpected3, adstExpected3), adstOutput256.V3);
+
+        Vector128<int> identityInput128 = Vector128.Create(524_287, -524_288, 524_286, -524_287);
+        Vector256<int> identityInput256 = Vector256.Create(
+            524_287,
+            -524_288,
+            524_286,
+            -524_287,
+            370_727,
+            -370_728,
+            262_143,
+            -262_144);
+
+        AssertWidenedIdentityOperator<Av1Inverse2dTransformer.Identity4Operator>(
+            4,
+            identityInput128,
+            Vector128.Create(741_503, -741_504, 741_501, -741_503),
+            identityInput256,
+            Vector256.Create(741_503, -741_504, 741_501, -741_503, 524_322, -524_323, 370_751, -370_752),
+            stageRange);
+
+        AssertWidenedIdentityOperator<Av1Inverse2dTransformer.Identity16Operator>(
+            16,
+            identityInput128,
+            Vector128.Create(1_483_005, -1_483_008, 1_483_002, -1_483_005),
+            identityInput256,
+            Vector256.Create(1_483_005, -1_483_008, 1_483_002, -1_483_005, 1_048_643, -1_048_646, 741_501, -741_504),
+            stageRange);
+    }
+
+    /// <summary>
+    /// Verifies one identity operator against exact pinned-libaom widened fixed-point results.
+    /// </summary>
+    /// <typeparam name="TOperator">The inverse identity operator.</typeparam>
+    /// <param name="length">The identity-transform length.</param>
+    /// <param name="input128">The four-lane bounded input.</param>
+    /// <param name="expected128">The exact four-lane result.</param>
+    /// <param name="input256">The eight-lane bounded input.</param>
+    /// <param name="expected256">The exact eight-lane result.</param>
+    /// <param name="stageRange">The twelve-bit inverse row-stage range.</param>
+    private static void AssertWidenedIdentityOperator<TOperator>(
+        int length,
+        Vector128<int> input128,
+        Vector128<int> expected128,
+        Vector256<int> input256,
+        Vector256<int> expected256,
+        Av1TransformStageRange stageRange)
+        where TOperator : struct, Av1Inverse2dTransformer.IAv1InverseTransform1dOperator
+    {
+        const int cosBit = 12;
+        Av1TransformVector<Vector128<int>> values128 = default;
+        Av1TransformVector<Vector128<int>> output128 = default;
+        Av1TransformVector<Vector128<int>> step128 = default;
+        Av1TransformVector<Vector256<int>> values256 = default;
+        Av1TransformVector<Vector256<int>> output256 = default;
+        Av1TransformVector<Vector256<int>> step256 = default;
+
+        for (int index = 0; index < length; index++)
+        {
+            values128[index] = input128;
+            values256[index] = input256;
+        }
+
+        TOperator.Transform(ref values128, ref output128, ref step128, cosBit, stageRange);
+        TOperator.Transform(ref values256, ref output256, ref step256, cosBit, stageRange);
+
+        for (int index = 0; index < length; index++)
+        {
+            Assert.Equal(expected128, output128[index]);
+            Assert.Equal(expected256, output256[index]);
+        }
+    }
+
     [Theory]
     [InlineData((int)Av1TransformSize.Size4x4, 0, -4)]
     [InlineData((int)Av1TransformSize.Size8x8, -1, -4)]