Complete AV1 predictor architecture checkpoint

HEIF_IMPLEMENTATION_PLAN.md

@@ -29,7 +29,7 @@ Checkboxes may be marked complete only when the implementation and the verificat
 
 ## Delivery dashboard
 
-Last reconciled with the source tree on 2026-08-29 against production checkpoint `beb2ab86ce5e4052c667df30af8a48a860814031`. Committed checkpoints include the AV1 transform architecture, OBU framing, intra-block copy, 12-profile reconstruction matrix, layered-item properties, layered reference/header/CDF/motion-field state, inter-frame intra blocks, SIMD-first translational prediction, complete single-reference inter reconstruction, compound reference trees and modes, paired reference-MV derivation, reference-dependent bounded sequence decoding, allocation-free SIMD-first equal averaging, selected inter-intra prediction, selectable compound blending, OBMC, scaled-reference reconstruction, local warped prediction, non-translational global prediction, official motion-vector conformance, selected spatial-layer presentation, and progressive color and auxiliary-alpha conformance. This dashboard is the authoritative delivery order. The detailed phase checklists below provide subsystem evidence; they do not override the current-stage marker or permit work to skip ahead.
+Last reconciled with the source tree on 2026-08-29 against production checkpoint `ef93d584511055f8e91e8d801662c44a5b0984d9`. Committed checkpoints include the AV1 transform architecture, OBU framing, intra-block copy, 12-profile reconstruction matrix, corrected twelve-bit inverse-transform SIMD arithmetic, layered-item properties, layered reference/header/CDF/motion-field state, inter-frame intra blocks, SIMD-first translational prediction, complete single-reference inter reconstruction, compound reference trees and modes, paired reference-MV derivation, reference-dependent bounded sequence decoding, allocation-free SIMD-first equal averaging, selected inter-intra prediction, selectable compound blending, OBMC, scaled-reference reconstruction, local warped prediction, non-translational global prediction, official motion-vector conformance, selected spatial-layer presentation, and progressive color and auxiliary-alpha conformance. This dashboard is the authoritative delivery order. The detailed phase checklists below provide subsystem evidence; they do not override the current-stage marker or permit work to skip ahead.
 
 Commit `1c58d855f70b024170ced9eb0a7005f0f9c955ad` records the complete official motion-vector conformance checkpoint. The official IVF has SHA-1 `F064290D7FCD3B3DE19020E8AEC6C43C88D3A505`, matching the pinned libaom test-data manifest, and SHA-256 `222A9050059B254DAB17CFB802FF829C778E3F93AF18961A622C8268576C1395`; its pinned-libaom Y4M has SHA-256 `D97AC78C81782CF1507549368047769DC677DBE205706458D1EE9C807DE6EC78`. Both source targets build with zero warnings and errors, the `net10.0` test-project analyzer build completes with zero errors and 1,014 pre-existing repository warnings, Roslynk reports zero compiler errors, 3,985 focused `net10.0` cases pass without failures or skips, and `git diff --check` is clean.
 
@@ -37,7 +37,9 @@ Commit `019ac5648b4380a6ff4865072c96fde29ce09fad` records the complete selected-
 
 Commit `beb2ab86ce5e4052c667df30af8a48a860814031` records the complete progressive color and auxiliary-alpha conformance checkpoint. Both pinned progressive AVIF fixtures pass their exact native-plane and pinned-libavif presentation comparisons through normal dispatch, AVX-512-disabled, AVX-disabled, and scalar `FeatureTestRunner` configurations. The YUV444-alpha fixture also compares every composed alpha sample with the independent final native auxiliary plane. Both complete production presentation paths pass under a 1,024-byte constrained tracked allocator with every allocation returned exactly once. Both source targets build with zero warnings and errors, the `net10.0` test-project analyzer build completes with zero errors and 1,014 pre-existing repository warnings, Roslynk reports zero compiler errors, 156 focused reconstruction and color cases pass without failures or skips, and `git diff --check` is clean.
 
-The working tree contains a complete, verified twelve-bit inverse-transform arithmetic checkpoint awaiting commit. ADST4 retains pinned libaom's signed 32-bit sine products and factorized sums and widens only its terminal rounding; Identity4 and Identity16 widen their fixed-point product and rounding bias only for the 20-bit twelve-bit row stage. The established 8/10-bit and twelve-bit column paths remain unchanged. Exact boundary vectors cover both 128-bit and 256-bit operators through normal, AVX-512-disabled, AVX-disabled, and scalar `FeatureTestRunner` configurations. Both source targets build with zero warnings and errors, the `net10.0` test-project analyzer build completes with zero errors and 1,014 pre-existing repository warnings, Roslynk reports zero compiler errors, 510 focused inverse-transform cases and 44 production reconstruction cases pass without failures or skips, and `git diff --check` is clean.
+Commit `ef93d584511055f8e91e8d801662c44a5b0984d9` records the complete twelve-bit inverse-transform arithmetic checkpoint. ADST4 retains pinned libaom's signed 32-bit sine products and factorized sums and widens only its terminal rounding; Identity4 and Identity16 widen their fixed-point product and rounding bias only for the 20-bit twelve-bit row stage. The established 8/10-bit and twelve-bit column paths remain unchanged. Exact boundary vectors cover both 128-bit and 256-bit operators through normal, AVX-512-disabled, AVX-disabled, and scalar `FeatureTestRunner` configurations. Both source targets build with zero warnings and errors, the `net10.0` test-project analyzer build completes with zero errors and 1,014 pre-existing repository warnings, Roslynk reports zero compiler errors, 510 focused inverse-transform cases and 44 production reconstruction cases pass without failures or skips, and `git diff --check` is clean.
+
+The working tree contains a complete, verified official syntax-coverage and predictor-architecture checkpoint awaiting commit. The official all-intra, CDF-update, and temporal motion-field IVF files have SHA-1 `A9F7EA6312A533CC6426A6145EDD190D45813C37`, `AFCA5502A489692B0A3C120370B0F43B8FC572A1`, and `B48A717C7C003B8DD23C3C2CAED1AC673380FDB3`, exactly matching the pinned libaom manifest. Their SHA-256 values are `5FCD265FD9F9BDD0D3179340B4C4532F1422CA5E5D97741C7481B84CB5DC122F`, `14A3DBF537B6BF15EFC003182D9916D61438C93624A8BD26E6E3AE7EAF33EA82`, and `B59BF9586D8546DFDA81DFEC4EE4E32CEB502C9D22412AB0B63A2ABB534A1F14`; the pinned-libaom Y4M references have SHA-256 `1211EBEFBC9CCEF9ED19BE4CCE3F807D69FFFE338E95CCA1B5F4CA8023482175`, `4FBFF73FF0DE2D9084DAE557D1D4BD677B0486516525BF4D327D2D795D5A7779`, and `F7DB607694818C19E62FD9A27F53E1A3E2D00B72C39C0430C1B26399CC76777D`. Exact native-plane comparison covers all 39 all-intra frames, every intra mode, seven selected transform types, both tile-local and frame-end adaptive CDF publication, all four temporal motion-field frames under normal/scalar dispatch, and constrained tracked motion-field allocation with exactly-once returns. Every distinct AV1 predictor traversal now owns a family-named JPEG-style static operator contract and SIMD traversal instead of nesting separate predictors beneath broad intra/inter families. Compound reference convolution, equal averaging, distance weighting, alpha-mask blending, difference-weighted mask construction, and each intermediate reconstruction or mask traversal have separate family owners and matching `.Operator.cs` contracts; inter-intra mask construction has its own non-operator builder. Both Release source targets build with zero warnings and errors, Roslynk reports zero compiler errors, 179 focused prediction, reconstruction, reference, ownership, and lifecycle cases pass without failures or skips, and `git diff --check` is clean.
 
 Status meanings:
 
@@ -49,7 +51,7 @@ Status meanings:
 
 Current development stage: **Stage 3 — complete AV1 still-image decoding.** The decoder retains reference/header/CDF/motion-field state, derives frame-level skip-mode references, consumes temporal segment prediction, decodes intra-coded blocks inside inter frames, and reconstructs translational single-reference, compound, inter-intra, OBMC, scaled-reference, local warped, and non-translational global prediction before residual traversal. Commit `1c58d855f70b024170ced9eb0a7005f0f9c955ad` adds exact official four-frame coverage through all ordinary inter modes, all three motion modes, every regular/smooth/sharp dual-filter pair, sub-8x8 chroma prediction, and no-round compound intermediates. Commit `019ac5648b4380a6ff4865072c96fde29ce09fad` adds exact selected-layer native reconstruction and scaled presentation. Neither AV1 nor HEVC production encoding is implemented.
 
-Immediate checkpoint: **commit the verified twelve-bit inverse-transform arithmetic correction before advancing.** The corrected operators match pinned libaom at the exact overflow boundaries, preserve narrower stage paths, and pass the complete focused transform and production reconstruction evidence.
+Immediate checkpoint: **inventory and remove the next remaining production-reachable unsupported valid AV1 still-image syntax path.** Trace every explicit rejection through the decoder, distinguish malformed or out-of-scope syntax from valid still-image behavior, select the first valid gap in source order, and close it with exact independent native-plane and presentation evidence before advancing.
 
 | Order | Delivery stage | State | Delivered state | Gate that remains open |
 | --- | --- | --- | --- | --- |
@@ -107,7 +109,8 @@ Immediate checkpoint: **commit the verified twelve-bit inverse-transform arithme
   - [x] Return the explicitly selected spatial layer or the final displayed layer, keeping reference reconstruction separate from display-only film grain. The essential-`lsel` production fixture reconstructs the selected 40x40 YUV444 base exactly, scales native component planes to the 80x80 item extent with pinned-libyuv integer rounding, and matches pinned-libavif RGBA presentation under normal/scalar dispatch. The committed final-layer fixture and film-grain matrix remain exact. Constrained tracked allocation returns every short-lived presentation plane exactly once. Commit `019ac5648b4380a6ff4865072c96fde29ce09fad` records the checkpoint.
   - [x] Verify color and auxiliary-alpha output exactly against both pinned libavif progressive fixtures under normal SIMD dispatch and all required `FeatureTestRunner` fallbacks. Both fixtures pass exact public presentation under normal, AVX-512-disabled, AVX-disabled, and scalar dispatch; the alpha-bearing fixture additionally matches every composed alpha sample with its pinned native auxiliary plane. Both full production paths pass under a 1,024-byte constrained tracked allocator with exactly-once returns. Both Release source targets build with zero warnings and errors, the test-project analyzer build completes with zero errors and 1,014 pre-existing warnings, Roslynk reports zero compiler errors, and all 156 focused reconstruction and color cases pass without failures or skips.
 - [x] Correct the audited 12-bit inverse ADST4, Identity4, and Identity16 SIMD arithmetic by widening only the libaom-widened multiply/accumulate operations, with exact conformant-range vectors and `FeatureTestRunner` coverage. Both 128-bit and 256-bit operators match pinned outputs through every required hardware fallback; 510 focused inverse-transform cases, 44 production reconstruction cases, both zero-warning source builds, the zero-error analyzer build, zero Roslynk compiler errors, and clean `git diff --check` complete the verification.
-- [ ] **Queued until the twelve-bit inverse-transform checkpoint commit:** continue inventorying and removing every remaining valid AV1 still-image unsupported branch, adding exact independent compression-tool fixtures to the profile-matrix regression gate.
+- [x] Verify the official libaom all-intra, CDF-update, and temporal motion-field sequences through exact native output, selected syntax coverage, normal/scalar dispatch, constrained tracked allocation, and the family-owned JPEG-style predictor operator architecture.
+- [ ] **Current:** continue inventorying and removing every remaining valid AV1 still-image unsupported branch, adding exact independent compression-tool fixtures to the profile-matrix regression gate.
 - [ ] Complete the remaining HEVC still-image profile and Range Extensions matrix with exact independent native-plane and presentation evidence.
 - [ ] Close shared decoded presentation, ICC, alpha, grid, transform, metadata, and animated AV1/HEVC decode gates.
 - [ ] Implement and independently verify real AV1/AVIF still encoding.
@@ -819,6 +822,7 @@ No valid HEVC or AV1 color, compression, or bit-depth row may remain `unsupporte
 ## Working rules for implementation
 
 - Keep changes vertical and reviewable. A slice should add one behavior, its focused tests, independent evidence, and any required notice update.
+- Keep every AV1 prediction family on the established JPEG color-converter operator architecture. Each distinct traversal contract owns a family-named predictor type; its `.Operator.cs` defines the static interface, and its family-named files own the closed generic widest-to-narrowest SIMD traversal. Semantic `readonly struct` operators implement scalar, `Vector128`, `Vector256`, and `Vector512` arithmetic through that contract. Only modes which share the same traversal and contract may share a predictor family; do not nest a separate predictor beneath a broad intra/inter family or create hardware-width-specific class hierarchies.
 - Design SIMD-suitable codec work SIMD-first. Establish vector-friendly storage, operator boundaries, scratch ownership, traversal, every applicable lane width, and benchmark-gated dispatch before implementing the equivalent scalar fallback; never build a scalar production architecture and bolt SIMD onto it later.
 - Inspect every owning method and upstream invariant before adding guards. Validate external file data at the parser/model boundary and rely on those established invariants internally.
 - Do not extract one-use helpers merely to label code. Extract shared primitives only when they have genuine reuse or remove substantial complexity.
@@ -841,7 +845,7 @@ The dashboard and immediate execution queue define the remaining critical path.
 - [x] Implement and independently verify non-translational global prediction with a genuine traced bounded sequence, exact native and presentation comparisons, constrained allocation, normal/scalar dispatch, and direct 8/10/12-bit compound production coverage. Commit `c5637ea0187df35b385bf43e2fe85cd955f01099` records the checkpoint.
 - [x] Complete progressive color and auxiliary-alpha verification through exact native and presentation comparisons, every required `FeatureTestRunner` fallback, and constrained tracked allocation.
 - [x] Correct the audited 12-bit inverse ADST4, Identity4, and Identity16 arithmetic through exact pinned boundary vectors and SIMD/scalar feature isolation.
-- [ ] **Queued until the twelve-bit inverse-transform checkpoint commit:** remove every other unsupported valid AV1 still-image syntax path and prove the complete AVIF decode matrix with independent inputs and scalar/SIMD parity.
+- [ ] **Current:** remove every other unsupported valid AV1 still-image syntax path and prove the complete AVIF decode matrix with independent inputs and scalar/SIMD parity.
 - [ ] Close Phase 4 by completing the remaining HEVC profile and Range Extensions matrix with exact native-plane and presented-image evidence.
 - [ ] Close Phase 5 and the decode portion of the bounded sequence ledger: color, ICC, alpha, grids, presentation transforms, reference-dependent samples, and complete animated AVIF/HEIC decode.
 - [ ] Close the still-image portions of Phases 0, 1, and 2 that remain as release gates: documentation, provenance, public format boundaries, API review, parser hardening, and malformed-input coverage.

shared-infrastructure

@@ -1 +1 @@
-Subproject commit 03471c6b458a2c11a0b1df1f5bf2777c8c773b99
+Subproject commit a835a9d74e82b2d32b580a7902eb2699ebc47098

src/ImageSharp/Formats/Heif/Av1/Prediction/Av1DcIntraPredictor.Operator.cs

@@ -0,0 +1,434 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.CompilerServices;
+using System.Runtime.InteropServices;
+using System.Runtime.Intrinsics;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction;
+
+/// <content>
+/// Defines reference reduction and rounded mean arithmetic for AV1 DC intra prediction.
+/// </content>
+internal static class Av1DcIntraPredictor
+{
+    /// <summary>
+    /// Defines scalar and SIMD reference reduction for AV1 DC intra prediction.
+    /// </summary>
+    internal interface IDcPredictionOperator
+    {
+        /// <summary>
+        /// Sums one 8-bit reference sample.
+        /// </summary>
+        /// <param name="sample">The reference sample.</param>
+        /// <returns>The sample value.</returns>
+        public static abstract int Sum(byte sample);
+
+        /// <summary>
+        /// Sums sixteen 8-bit reference samples.
+        /// </summary>
+        /// <param name="samples">The reference samples.</param>
+        /// <returns>The exact sum.</returns>
+        public static abstract int Sum(Vector128<byte> samples);
+
+        /// <summary>
+        /// Sums thirty-two 8-bit reference samples.
+        /// </summary>
+        /// <param name="samples">The reference samples.</param>
+        /// <returns>The exact sum.</returns>
+        public static abstract int Sum(Vector256<byte> samples);
+
+        /// <summary>
+        /// Sums sixty-four 8-bit reference samples.
+        /// </summary>
+        /// <param name="samples">The reference samples.</param>
+        /// <returns>The exact sum.</returns>
+        public static abstract int Sum(Vector512<byte> samples);
+
+        /// <summary>
+        /// Sums one high-bit-depth reference sample.
+        /// </summary>
+        /// <param name="sample">The reference sample.</param>
+        /// <returns>The sample value.</returns>
+        public static abstract int Sum(short sample);
+
+        /// <summary>
+        /// Sums eight high-bit-depth reference samples.
+        /// </summary>
+        /// <param name="samples">The reference samples.</param>
+        /// <returns>The exact sum.</returns>
+        public static abstract int Sum(Vector128<short> samples);
+
+        /// <summary>
+        /// Sums sixteen high-bit-depth reference samples.
+        /// </summary>
+        /// <param name="samples">The reference samples.</param>
+        /// <returns>The exact sum.</returns>
+        public static abstract int Sum(Vector256<short> samples);
+
+        /// <summary>
+        /// Sums thirty-two high-bit-depth reference samples.
+        /// </summary>
+        /// <param name="samples">The reference samples.</param>
+        /// <returns>The exact sum.</returns>
+        public static abstract int Sum(Vector512<short> samples);
+
+        /// <summary>
+        /// Calculates the 8-bit DC prediction.
+        /// </summary>
+        /// <param name="sum">The sum of available reference samples.</param>
+        /// <param name="count">The number of available reference samples.</param>
+        /// <returns>The rounded DC prediction.</returns>
+        public static abstract byte Predict(int sum, int count);
+
+        /// <summary>
+        /// Calculates the high-bit-depth DC prediction.
+        /// </summary>
+        /// <param name="sum">The sum of available reference samples.</param>
+        /// <param name="count">The number of available reference samples.</param>
+        /// <param name="bitDepth">The reconstructed sample precision.</param>
+        /// <returns>The rounded DC prediction.</returns>
+        public static abstract short Predict(int sum, int count, int bitDepth);
+    }
+
+    /// <summary>
+    /// Predicts an 8-bit DC block.
+    /// </summary>
+    public static void Predict(bool hasLeft, bool hasAbove, Span<byte> destination, int destinationStride, ReadOnlySpan<byte> above, ReadOnlySpan<byte> left, int width, int height)
+        => Predictor<DcOperator>.Predict(hasLeft, hasAbove, destination, destinationStride, above, left, width, height);
+
+    /// <summary>
+    /// Predicts a high-bit-depth DC block.
+    /// </summary>
+    public static void Predict(bool hasLeft, bool hasAbove, Span<short> destination, int destinationStride, ReadOnlySpan<short> above, ReadOnlySpan<short> left, int width, int height, int bitDepth)
+        => Predictor<DcOperator>.Predict(hasLeft, hasAbove, destination, destinationStride, above, left, width, height, bitDepth);
+
+    /// <summary>
+    /// Predicts an 8-bit DC block without hardware intrinsics.
+    /// </summary>
+    public static void PredictScalar(bool hasLeft, bool hasAbove, Span<byte> destination, int destinationStride, ReadOnlySpan<byte> above, ReadOnlySpan<byte> left, int width, int height)
+        => Predictor<DcOperator>.PredictScalar(hasLeft, hasAbove, destination, destinationStride, above, left, width, height);
+
+    /// <summary>
+    /// Predicts a high-bit-depth DC block without hardware intrinsics.
+    /// </summary>
+    public static void PredictScalar(bool hasLeft, bool hasAbove, Span<short> destination, int destinationStride, ReadOnlySpan<short> above, ReadOnlySpan<short> left, int width, int height, int bitDepth)
+        => Predictor<DcOperator>.PredictScalar(hasLeft, hasAbove, destination, destinationStride, above, left, width, height, bitDepth);
+
+    /// <summary>
+    /// Calculates the DC value from the available neighboring samples.
+    /// </summary>
+    internal readonly struct DcOperator : IDcPredictionOperator
+    {
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static int Sum(byte sample) => sample;
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static int Sum(Vector128<byte> samples)
+        {
+            (Vector128<ushort> lower, Vector128<ushort> upper) = Vector128.Widen(samples);
+
+            return Vector128.Sum(lower) + Vector128.Sum(upper);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static int Sum(Vector256<byte> samples)
+        {
+            (Vector256<ushort> lower, Vector256<ushort> upper) = Vector256.Widen(samples);
+
+            return Vector256.Sum(lower) + Vector256.Sum(upper);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static int Sum(Vector512<byte> samples)
+        {
+            (Vector512<ushort> lower, Vector512<ushort> upper) = Vector512.Widen(samples);
+
+            return Vector512.Sum(lower) + Vector512.Sum(upper);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static int Sum(short sample) => sample;
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static int Sum(Vector128<short> samples)
+        {
+            (Vector128<int> lower, Vector128<int> upper) = Vector128.Widen(samples);
+
+            return Vector128.Sum(lower) + Vector128.Sum(upper);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static int Sum(Vector256<short> samples)
+        {
+            (Vector256<int> lower, Vector256<int> upper) = Vector256.Widen(samples);
+
+            return Vector256.Sum(lower) + Vector256.Sum(upper);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static int Sum(Vector512<short> samples)
+        {
+            (Vector512<int> lower, Vector512<int> upper) = Vector512.Widen(samples);
+
+            return Vector512.Sum(lower) + Vector512.Sum(upper);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static byte Predict(int sum, int count) => count == 0 ? (byte)128 : (byte)((sum + (count >> 1)) / count);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static short Predict(int sum, int count, int bitDepth)
+            => count == 0 ? (short)(1 << (bitDepth - 1)) : (short)((sum + (count >> 1)) / count);
+    }
+
+    /// <summary>
+    /// Reconstructs DC blocks through one closed reduction operator.
+    /// </summary>
+    /// <typeparam name="TOperator">The reference reduction and rounded mean arithmetic.</typeparam>
+    private static class Predictor<TOperator>
+        where TOperator : struct, IDcPredictionOperator
+    {
+        /// <summary>
+        /// Predicts an 8-bit DC block.
+        /// </summary>
+        /// <param name="hasLeft">Whether the left reference is available.</param>
+        /// <param name="hasAbove">Whether the top reference is available.</param>
+        /// <param name="destination">The destination block.</param>
+        /// <param name="destinationStride">The distance between destination rows.</param>
+        /// <param name="above">The top reference.</param>
+        /// <param name="left">The left reference.</param>
+        /// <param name="width">The block width.</param>
+        /// <param name="height">The block height.</param>
+        public static void Predict(bool hasLeft, bool hasAbove, Span<byte> destination, int destinationStride, ReadOnlySpan<byte> above, ReadOnlySpan<byte> left, int width, int height)
+        {
+            int count = (hasAbove ? width : 0) + (hasLeft ? height : 0);
+            int sum = (hasAbove ? Sum(above[..width]) : 0) + (hasLeft ? Sum(left[..height]) : 0);
+            byte prediction = TOperator.Predict(sum, count);
+
+            for (int row = 0; row < height; row++)
+            {
+                destination.Slice(row * destinationStride, width).Fill(prediction);
+            }
+        }
+
+        /// <summary>
+        /// Predicts a high-bit-depth DC block.
+        /// </summary>
+        /// <param name="hasLeft">Whether the left reference is available.</param>
+        /// <param name="hasAbove">Whether the top reference is available.</param>
+        /// <param name="destination">The destination block.</param>
+        /// <param name="destinationStride">The distance between destination rows.</param>
+        /// <param name="above">The top reference.</param>
+        /// <param name="left">The left reference.</param>
+        /// <param name="width">The block width.</param>
+        /// <param name="height">The block height.</param>
+        /// <param name="bitDepth">The reconstructed sample precision.</param>
+        public static void Predict(bool hasLeft, bool hasAbove, Span<short> destination, int destinationStride, ReadOnlySpan<short> above, ReadOnlySpan<short> left, int width, int height, int bitDepth)
+        {
+            int count = (hasAbove ? width : 0) + (hasLeft ? height : 0);
+            int sum = (hasAbove ? Sum(above[..width]) : 0) + (hasLeft ? Sum(left[..height]) : 0);
+            short prediction = TOperator.Predict(sum, count, bitDepth);
+
+            for (int row = 0; row < height; row++)
+            {
+                destination.Slice(row * destinationStride, width).Fill(prediction);
+            }
+        }
+
+        /// <summary>
+        /// Predicts an 8-bit DC block without hardware intrinsics.
+        /// </summary>
+        /// <param name="hasLeft">Whether the left reference is available.</param>
+        /// <param name="hasAbove">Whether the top reference is available.</param>
+        /// <param name="destination">The destination block.</param>
+        /// <param name="destinationStride">The distance between destination rows.</param>
+        /// <param name="above">The top reference.</param>
+        /// <param name="left">The left reference.</param>
+        /// <param name="width">The block width.</param>
+        /// <param name="height">The block height.</param>
+        public static void PredictScalar(bool hasLeft, bool hasAbove, Span<byte> destination, int destinationStride, ReadOnlySpan<byte> above, ReadOnlySpan<byte> left, int width, int height)
+        {
+            int count = (hasAbove ? width : 0) + (hasLeft ? height : 0);
+            int sum = (hasAbove ? SumScalar(above[..width]) : 0) + (hasLeft ? SumScalar(left[..height]) : 0);
+            byte prediction = TOperator.Predict(sum, count);
+
+            for (int row = 0; row < height; row++)
+            {
+                ref byte destinationRow = ref destination[row * destinationStride];
+                for (int column = 0; column < width; column++)
+                {
+                    Unsafe.Add(ref destinationRow, column) = prediction;
+                }
+            }
+        }
+
+        /// <summary>
+        /// Predicts a high-bit-depth DC block without hardware intrinsics.
+        /// </summary>
+        /// <param name="hasLeft">Whether the left reference is available.</param>
+        /// <param name="hasAbove">Whether the top reference is available.</param>
+        /// <param name="destination">The destination block.</param>
+        /// <param name="destinationStride">The distance between destination rows.</param>
+        /// <param name="above">The top reference.</param>
+        /// <param name="left">The left reference.</param>
+        /// <param name="width">The block width.</param>
+        /// <param name="height">The block height.</param>
+        /// <param name="bitDepth">The reconstructed sample precision.</param>
+        public static void PredictScalar(bool hasLeft, bool hasAbove, Span<short> destination, int destinationStride, ReadOnlySpan<short> above, ReadOnlySpan<short> left, int width, int height, int bitDepth)
+        {
+            int count = (hasAbove ? width : 0) + (hasLeft ? height : 0);
+            int sum = (hasAbove ? SumScalar(above[..width]) : 0) + (hasLeft ? SumScalar(left[..height]) : 0);
+            short prediction = TOperator.Predict(sum, count, bitDepth);
+
+            for (int row = 0; row < height; row++)
+            {
+                ref short destinationRow = ref destination[row * destinationStride];
+                for (int column = 0; column < width; column++)
+                {
+                    Unsafe.Add(ref destinationRow, column) = prediction;
+                }
+            }
+        }
+
+        /// <summary>
+        /// Sums 8-bit references through the widest available SIMD widths.
+        /// </summary>
+        /// <param name="samples">The reference samples.</param>
+        /// <returns>The exact sum.</returns>
+        private static int Sum(ReadOnlySpan<byte> samples)
+        {
+            ref byte samplesBase = ref MemoryMarshal.GetReference(samples);
+            int sum = 0;
+            int index = 0;
+
+            // The shared index deliberately continues through narrower widths. This handles every legal AV1 edge
+            // length without a separate dispatch tree and leaves only an incomplete final vector to scalar code.
+            if (Vector512.IsHardwareAccelerated)
+            {
+                int oneVectorFromEnd = samples.Length - Vector512<byte>.Count;
+                for (; index <= oneVectorFromEnd; index += Vector512<byte>.Count)
+                {
+                    sum += TOperator.Sum(Vector512.LoadUnsafe(ref samplesBase, (nuint)index));
+                }
+            }
+
+            if (Vector256.IsHardwareAccelerated)
+            {
+                int oneVectorFromEnd = samples.Length - Vector256<byte>.Count;
+                for (; index <= oneVectorFromEnd; index += Vector256<byte>.Count)
+                {
+                    sum += TOperator.Sum(Vector256.LoadUnsafe(ref samplesBase, (nuint)index));
+                }
+            }
+
+            if (Vector128.IsHardwareAccelerated)
+            {
+                int oneVectorFromEnd = samples.Length - Vector128<byte>.Count;
+                for (; index <= oneVectorFromEnd; index += Vector128<byte>.Count)
+                {
+                    sum += TOperator.Sum(Vector128.LoadUnsafe(ref samplesBase, (nuint)index));
+                }
+            }
+
+            for (; index < samples.Length; index++)
+            {
+                sum += TOperator.Sum(Unsafe.Add(ref samplesBase, index));
+            }
+
+            return sum;
+        }
+
+        /// <summary>
+        /// Sums high-bit-depth references through the widest available SIMD widths.
+        /// </summary>
+        /// <param name="samples">The reference samples.</param>
+        /// <returns>The exact sum.</returns>
+        private static int Sum(ReadOnlySpan<short> samples)
+        {
+            ref short samplesBase = ref MemoryMarshal.GetReference(samples);
+            int sum = 0;
+            int index = 0;
+
+            if (Vector512.IsHardwareAccelerated)
+            {
+                int oneVectorFromEnd = samples.Length - Vector512<short>.Count;
+                for (; index <= oneVectorFromEnd; index += Vector512<short>.Count)
+                {
+                    sum += TOperator.Sum(Vector512.LoadUnsafe(ref samplesBase, (nuint)index));
+                }
+            }
+
+            if (Vector256.IsHardwareAccelerated)
+            {
+                int oneVectorFromEnd = samples.Length - Vector256<short>.Count;
+                for (; index <= oneVectorFromEnd; index += Vector256<short>.Count)
+                {
+                    sum += TOperator.Sum(Vector256.LoadUnsafe(ref samplesBase, (nuint)index));
+                }
+            }
+
+            if (Vector128.IsHardwareAccelerated)
+            {
+                int oneVectorFromEnd = samples.Length - Vector128<short>.Count;
+                for (; index <= oneVectorFromEnd; index += Vector128<short>.Count)
+                {
+                    sum += TOperator.Sum(Vector128.LoadUnsafe(ref samplesBase, (nuint)index));
+                }
+            }
+
+            for (; index < samples.Length; index++)
+            {
+                sum += TOperator.Sum(Unsafe.Add(ref samplesBase, index));
+            }
+
+            return sum;
+        }
+
+        /// <summary>
+        /// Sums 8-bit references without hardware intrinsics.
+        /// </summary>
+        /// <param name="samples">The reference samples.</param>
+        /// <returns>The exact sum.</returns>
+        private static int SumScalar(ReadOnlySpan<byte> samples)
+        {
+            ref byte samplesBase = ref MemoryMarshal.GetReference(samples);
+            int sum = 0;
+
+            for (int index = 0; index < samples.Length; index++)
+            {
+                sum += TOperator.Sum(Unsafe.Add(ref samplesBase, index));
+            }
+
+            return sum;
+        }
+
+        /// <summary>
+        /// Sums high-bit-depth references without hardware intrinsics.
+        /// </summary>
+        /// <param name="samples">The reference samples.</param>
+        /// <returns>The exact sum.</returns>
+        private static int SumScalar(ReadOnlySpan<short> samples)
+        {
+            ref short samplesBase = ref MemoryMarshal.GetReference(samples);
+            int sum = 0;
+
+            for (int index = 0; index < samples.Length; index++)
+            {
+                sum += TOperator.Sum(Unsafe.Add(ref samplesBase, index));
+            }
+
+            return sum;
+        }
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/Av1DcIntraPredictor.cs

@@ -1,254 +0,0 @@
-// Copyright (c) Six Labors.
-// Licensed under the Six Labors Split License.
-
-using System.Runtime.CompilerServices;
-using System.Runtime.InteropServices;
-using System.Runtime.Intrinsics;
-
-namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction;
-
-/// <summary>
-/// Reconstructs AV1 DC intra-prediction blocks from the available neighboring samples.
-/// </summary>
-/// <remarks>
-/// Reference reduction uses the widest available integer lanes, then one rounded scalar mean is broadcast across each
-/// destination row. Row filling is delegated to span operations so the runtime supplies its native vectorized store;
-/// explicit scalar entry points remain available for feature-disabled conformance tests.
-/// </remarks>
-internal static class Av1DcIntraPredictor
-{
-    /// <summary>
-    /// Predicts an 8-bit DC block.
-    /// </summary>
-    /// <param name="hasLeft">Whether the prepared left reference is available.</param>
-    /// <param name="hasAbove">Whether the prepared top reference is available.</param>
-    /// <param name="destination">The destination block origin.</param>
-    /// <param name="destinationStride">The destination row stride in samples.</param>
-    /// <param name="above">The prepared top reference.</param>
-    /// <param name="left">The prepared left reference.</param>
-    /// <param name="width">The block width in samples.</param>
-    /// <param name="height">The block height in samples.</param>
-    public static void Predict(bool hasLeft, bool hasAbove, Span<byte> destination, int destinationStride, ReadOnlySpan<byte> above, ReadOnlySpan<byte> left, int width, int height)
-    {
-        int count = (hasAbove ? width : 0) + (hasLeft ? height : 0);
-        int sum = (hasAbove ? Sum(above[..width]) : 0) + (hasLeft ? Sum(left[..height]) : 0);
-
-        // Section 7.11.2.2 defines the unsigned midpoint when no reference is available. Otherwise adding half
-        // the reference count implements the specified rounded mean before extending it across the whole block.
-        byte prediction = count == 0 ? (byte)128 : (byte)((sum + (count >> 1)) / count);
-        for (int row = 0; row < height; row++)
-        {
-            destination.Slice(row * destinationStride, width).Fill(prediction);
-        }
-    }
-
-    /// <summary>
-    /// Predicts a high-bit-depth DC block.
-    /// </summary>
-    /// <param name="hasLeft">Whether the prepared left reference is available.</param>
-    /// <param name="hasAbove">Whether the prepared top reference is available.</param>
-    /// <param name="destination">The destination block origin.</param>
-    /// <param name="destinationStride">The destination row stride in samples.</param>
-    /// <param name="above">The prepared top reference.</param>
-    /// <param name="left">The prepared left reference.</param>
-    /// <param name="width">The block width in samples.</param>
-    /// <param name="height">The block height in samples.</param>
-    /// <param name="bitDepth">The reconstructed sample precision.</param>
-    public static void Predict(bool hasLeft, bool hasAbove, Span<short> destination, int destinationStride, ReadOnlySpan<short> above, ReadOnlySpan<short> left, int width, int height, int bitDepth)
-    {
-        int count = (hasAbove ? width : 0) + (hasLeft ? height : 0);
-        int sum = (hasAbove ? Sum(above[..width]) : 0) + (hasLeft ? Sum(left[..height]) : 0);
-        short prediction = count == 0 ? (short)(1 << (bitDepth - 1)) : (short)((sum + (count >> 1)) / count);
-        for (int row = 0; row < height; row++)
-        {
-            destination.Slice(row * destinationStride, width).Fill(prediction);
-        }
-    }
-
-    /// <summary>
-    /// Predicts an 8-bit DC block without hardware intrinsics.
-    /// </summary>
-    /// <param name="hasLeft">Whether the prepared left reference is available.</param>
-    /// <param name="hasAbove">Whether the prepared top reference is available.</param>
-    /// <param name="destination">The destination block origin.</param>
-    /// <param name="destinationStride">The destination row stride in samples.</param>
-    /// <param name="above">The prepared top reference.</param>
-    /// <param name="left">The prepared left reference.</param>
-    /// <param name="width">The block width in samples.</param>
-    /// <param name="height">The block height in samples.</param>
-    public static void PredictScalar(bool hasLeft, bool hasAbove, Span<byte> destination, int destinationStride, ReadOnlySpan<byte> above, ReadOnlySpan<byte> left, int width, int height)
-    {
-        int count = (hasAbove ? width : 0) + (hasLeft ? height : 0);
-        int sum = (hasAbove ? SumScalar(above[..width]) : 0) + (hasLeft ? SumScalar(left[..height]) : 0);
-        byte prediction = count == 0 ? (byte)128 : (byte)((sum + (count >> 1)) / count);
-        for (int row = 0; row < height; row++)
-        {
-            ref byte destinationRow = ref destination[row * destinationStride];
-            for (int column = 0; column < width; column++)
-            {
-                Unsafe.Add(ref destinationRow, column) = prediction;
-            }
-        }
-    }
-
-    /// <summary>
-    /// Predicts a high-bit-depth DC block without hardware intrinsics.
-    /// </summary>
-    /// <param name="hasLeft">Whether the prepared left reference is available.</param>
-    /// <param name="hasAbove">Whether the prepared top reference is available.</param>
-    /// <param name="destination">The destination block origin.</param>
-    /// <param name="destinationStride">The destination row stride in samples.</param>
-    /// <param name="above">The prepared top reference.</param>
-    /// <param name="left">The prepared left reference.</param>
-    /// <param name="width">The block width in samples.</param>
-    /// <param name="height">The block height in samples.</param>
-    /// <param name="bitDepth">The reconstructed sample precision.</param>
-    public static void PredictScalar(bool hasLeft, bool hasAbove, Span<short> destination, int destinationStride, ReadOnlySpan<short> above, ReadOnlySpan<short> left, int width, int height, int bitDepth)
-    {
-        int count = (hasAbove ? width : 0) + (hasLeft ? height : 0);
-        int sum = (hasAbove ? SumScalar(above[..width]) : 0) + (hasLeft ? SumScalar(left[..height]) : 0);
-        short prediction = count == 0 ? (short)(1 << (bitDepth - 1)) : (short)((sum + (count >> 1)) / count);
-        for (int row = 0; row < height; row++)
-        {
-            ref short destinationRow = ref destination[row * destinationStride];
-            for (int column = 0; column < width; column++)
-            {
-                Unsafe.Add(ref destinationRow, column) = prediction;
-            }
-        }
-    }
-
-    /// <summary>
-    /// Sums 8-bit neighboring samples using the widest available SIMD width.
-    /// </summary>
-    /// <param name="samples">The samples to sum.</param>
-    /// <returns>The exact sum.</returns>
-    private static int Sum(ReadOnlySpan<byte> samples)
-    {
-        ref byte samplesBase = ref MemoryMarshal.GetReference(samples);
-        int sum = 0;
-        int index = 0;
-
-        // Widening prevents the packed-byte reduction from overflowing before the horizontal sum. The shared index
-        // advances through every available vector width and leaves only the incomplete final group to the scalar loop.
-        if (Vector512.IsHardwareAccelerated)
-        {
-            int oneVectorFromEnd = samples.Length - Vector512<byte>.Count;
-            for (; index <= oneVectorFromEnd; index += Vector512<byte>.Count)
-            {
-                (Vector512<ushort> low, Vector512<ushort> high) = Vector512.Widen(Vector512.LoadUnsafe(ref samplesBase, (nuint)index));
-                sum += Vector512.Sum(low) + Vector512.Sum(high);
-            }
-        }
-
-        if (Vector256.IsHardwareAccelerated)
-        {
-            int oneVectorFromEnd = samples.Length - Vector256<byte>.Count;
-            for (; index <= oneVectorFromEnd; index += Vector256<byte>.Count)
-            {
-                (Vector256<ushort> low, Vector256<ushort> high) = Vector256.Widen(Vector256.LoadUnsafe(ref samplesBase, (nuint)index));
-                sum += Vector256.Sum(low) + Vector256.Sum(high);
-            }
-        }
-
-        if (Vector128.IsHardwareAccelerated)
-        {
-            int oneVectorFromEnd = samples.Length - Vector128<byte>.Count;
-            for (; index <= oneVectorFromEnd; index += Vector128<byte>.Count)
-            {
-                (Vector128<ushort> low, Vector128<ushort> high) = Vector128.Widen(Vector128.LoadUnsafe(ref samplesBase, (nuint)index));
-                sum += Vector128.Sum(low) + Vector128.Sum(high);
-            }
-        }
-
-        for (; index < samples.Length; index++)
-        {
-            sum += Unsafe.Add(ref samplesBase, index);
-        }
-
-        return sum;
-    }
-
-    /// <summary>
-    /// Sums high-bit-depth neighboring samples using the widest available SIMD width.
-    /// </summary>
-    /// <param name="samples">The samples to sum.</param>
-    /// <returns>The exact sum.</returns>
-    private static int Sum(ReadOnlySpan<short> samples)
-    {
-        ref short samplesBase = ref MemoryMarshal.GetReference(samples);
-        int sum = 0;
-        int index = 0;
-
-        // Signed 16-bit storage is nonnegative for supported bit depths. Widening to Int32 preserves the exact sum of
-        // the largest permitted reference edge before the rounded mean is calculated once outside this loop.
-        if (Vector512.IsHardwareAccelerated)
-        {
-            int oneVectorFromEnd = samples.Length - Vector512<short>.Count;
-            for (; index <= oneVectorFromEnd; index += Vector512<short>.Count)
-            {
-                (Vector512<int> low, Vector512<int> high) = Vector512.Widen(Vector512.LoadUnsafe(ref samplesBase, (nuint)index));
-                sum += Vector512.Sum(low) + Vector512.Sum(high);
-            }
-        }
-
-        if (Vector256.IsHardwareAccelerated)
-        {
-            int oneVectorFromEnd = samples.Length - Vector256<short>.Count;
-            for (; index <= oneVectorFromEnd; index += Vector256<short>.Count)
-            {
-                (Vector256<int> low, Vector256<int> high) = Vector256.Widen(Vector256.LoadUnsafe(ref samplesBase, (nuint)index));
-                sum += Vector256.Sum(low) + Vector256.Sum(high);
-            }
-        }
-
-        if (Vector128.IsHardwareAccelerated)
-        {
-            int oneVectorFromEnd = samples.Length - Vector128<short>.Count;
-            for (; index <= oneVectorFromEnd; index += Vector128<short>.Count)
-            {
-                (Vector128<int> low, Vector128<int> high) = Vector128.Widen(Vector128.LoadUnsafe(ref samplesBase, (nuint)index));
-                sum += Vector128.Sum(low) + Vector128.Sum(high);
-            }
-        }
-
-        for (; index < samples.Length; index++)
-        {
-            sum += Unsafe.Add(ref samplesBase, index);
-        }
-
-        return sum;
-    }
-
-    /// <summary>
-    /// Sums 8-bit neighboring samples without hardware intrinsics.
-    /// </summary>
-    /// <param name="samples">The samples to sum.</param>
-    /// <returns>The exact sum.</returns>
-    private static int SumScalar(ReadOnlySpan<byte> samples)
-    {
-        int sum = 0;
-        foreach (byte sample in samples)
-        {
-            sum += sample;
-        }
-
-        return sum;
-    }
-
-    /// <summary>
-    /// Sums high-bit-depth neighboring samples without hardware intrinsics.
-    /// </summary>
-    /// <param name="samples">The samples to sum.</param>
-    /// <returns>The exact sum.</returns>
-    private static int SumScalar(ReadOnlySpan<short> samples)
-    {
-        int sum = 0;
-        foreach (short sample in samples)
-        {
-            sum += sample;
-        }
-
-        return sum;
-    }
-}

src/ImageSharp/Formats/Heif/Av1/Prediction/Av1DirectionalIntraPredictor.Operator.cs

@@ -0,0 +1,239 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.CompilerServices;
+using System.Runtime.Intrinsics;
+using SixLabors.ImageSharp.Formats.Heif.Av1.Transform;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction;
+
+/// <content>
+/// Defines Q5 interpolation for AV1 directional intra prediction.
+/// </content>
+internal static partial class Av1DirectionalIntraPredictor
+{
+    /// <summary>
+    /// The largest number of samples required to transpose a directional prediction block.
+    /// </summary>
+    public const int ScratchLength = 64 * 64;
+
+    /// <summary>
+    /// Defines scalar and SIMD interpolation for AV1 directional intra prediction.
+    /// </summary>
+    internal interface IDirectionalPredictionOperator
+    {
+        /// <summary>
+        /// Interpolates one 8-bit pair.
+        /// </summary>
+        /// <param name="left">The first reference sample.</param>
+        /// <param name="right">The second reference sample.</param>
+        /// <param name="weight">The Q5 weight of <paramref name="right"/>.</param>
+        /// <returns>The interpolated sample.</returns>
+        public static abstract byte Interpolate(byte left, byte right, int weight);
+
+        /// <summary>
+        /// Interpolates one high-bit-depth pair.
+        /// </summary>
+        /// <param name="left">The first reference sample.</param>
+        /// <param name="right">The second reference sample.</param>
+        /// <param name="weight">The Q5 weight of <paramref name="right"/>.</param>
+        /// <returns>The interpolated sample.</returns>
+        public static abstract short Interpolate(short left, short right, int weight);
+
+        /// <summary>
+        /// Interpolates sixteen 8-bit pairs.
+        /// </summary>
+        /// <param name="left">The first reference samples.</param>
+        /// <param name="right">The second reference samples.</param>
+        /// <param name="weight">The Q5 weight of <paramref name="right"/>.</param>
+        /// <returns>The interpolated samples.</returns>
+        public static abstract Vector128<byte> Interpolate(Vector128<byte> left, Vector128<byte> right, int weight);
+
+        /// <summary>
+        /// Interpolates thirty-two 8-bit pairs.
+        /// </summary>
+        /// <param name="left">The first reference samples.</param>
+        /// <param name="right">The second reference samples.</param>
+        /// <param name="weight">The Q5 weight of <paramref name="right"/>.</param>
+        /// <returns>The interpolated samples.</returns>
+        public static abstract Vector256<byte> Interpolate(Vector256<byte> left, Vector256<byte> right, int weight);
+
+        /// <summary>
+        /// Interpolates sixty-four 8-bit pairs.
+        /// </summary>
+        /// <param name="left">The first reference samples.</param>
+        /// <param name="right">The second reference samples.</param>
+        /// <param name="weight">The Q5 weight of <paramref name="right"/>.</param>
+        /// <returns>The interpolated samples.</returns>
+        public static abstract Vector512<byte> Interpolate(Vector512<byte> left, Vector512<byte> right, int weight);
+
+        /// <summary>
+        /// Interpolates eight high-bit-depth pairs.
+        /// </summary>
+        /// <param name="left">The first reference samples.</param>
+        /// <param name="right">The second reference samples.</param>
+        /// <param name="weight">The Q5 weight of <paramref name="right"/>.</param>
+        /// <returns>The interpolated samples.</returns>
+        public static abstract Vector128<short> Interpolate(Vector128<short> left, Vector128<short> right, int weight);
+
+        /// <summary>
+        /// Interpolates sixteen high-bit-depth pairs.
+        /// </summary>
+        /// <param name="left">The first reference samples.</param>
+        /// <param name="right">The second reference samples.</param>
+        /// <param name="weight">The Q5 weight of <paramref name="right"/>.</param>
+        /// <returns>The interpolated samples.</returns>
+        public static abstract Vector256<short> Interpolate(Vector256<short> left, Vector256<short> right, int weight);
+
+        /// <summary>
+        /// Interpolates thirty-two high-bit-depth pairs.
+        /// </summary>
+        /// <param name="left">The first reference samples.</param>
+        /// <param name="right">The second reference samples.</param>
+        /// <param name="weight">The Q5 weight of <paramref name="right"/>.</param>
+        /// <returns>The interpolated samples.</returns>
+        public static abstract Vector512<short> Interpolate(Vector512<short> left, Vector512<short> right, int weight);
+
+        /// <summary>
+        /// Interpolates four widened pairs with independent weights.
+        /// </summary>
+        /// <param name="left">The first reference samples.</param>
+        /// <param name="right">The second reference samples.</param>
+        /// <param name="weights">The Q5 weights of <paramref name="right"/>.</param>
+        /// <returns>The interpolated samples.</returns>
+        public static abstract Vector128<int> Interpolate(Vector128<int> left, Vector128<int> right, Vector128<int> weights);
+    }
+
+    /// <summary>
+    /// Gets the horizontal Q8 projection derivative for an adjusted angle.
+    /// </summary>
+    public static int GetDeltaX(int angle) => Predictor<DirectionalOperator>.GetDeltaX(angle);
+
+    /// <summary>
+    /// Gets the vertical Q8 projection derivative for an adjusted angle.
+    /// </summary>
+    public static int GetDeltaY(int angle) => Predictor<DirectionalOperator>.GetDeltaY(angle);
+
+    /// <summary>
+    /// Predicts an 8-bit directional block.
+    /// </summary>
+    public static void Predict(Span<byte> destination, int destinationStride, Av1TransformSize transformSize, ReadOnlySpan<byte> above, ReadOnlySpan<byte> left, bool upsampleAbove, bool upsampleLeft, int angle, Span<byte> scratch)
+        => Predictor<DirectionalOperator>.Predict(destination, destinationStride, transformSize, above, left, upsampleAbove, upsampleLeft, angle, scratch);
+
+    /// <summary>
+    /// Predicts a high-bit-depth directional block.
+    /// </summary>
+    public static void Predict(Span<short> destination, int destinationStride, Av1TransformSize transformSize, ReadOnlySpan<short> above, ReadOnlySpan<short> left, bool upsampleAbove, bool upsampleLeft, int angle, Span<short> scratch)
+        => Predictor<DirectionalOperator>.Predict(destination, destinationStride, transformSize, above, left, upsampleAbove, upsampleLeft, angle, scratch);
+
+    /// <summary>
+    /// Predicts an 8-bit directional block without hardware intrinsics.
+    /// </summary>
+    public static void PredictScalar(Span<byte> destination, int destinationStride, Av1TransformSize transformSize, ReadOnlySpan<byte> above, ReadOnlySpan<byte> left, bool upsampleAbove, bool upsampleLeft, int angle)
+        => Predictor<DirectionalOperator>.PredictScalar(destination, destinationStride, transformSize, above, left, upsampleAbove, upsampleLeft, angle);
+
+    /// <summary>
+    /// Predicts a high-bit-depth directional block without hardware intrinsics.
+    /// </summary>
+    public static void PredictScalar(Span<short> destination, int destinationStride, Av1TransformSize transformSize, ReadOnlySpan<short> above, ReadOnlySpan<short> left, bool upsampleAbove, bool upsampleLeft, int angle)
+        => Predictor<DirectionalOperator>.PredictScalar(destination, destinationStride, transformSize, above, left, upsampleAbove, upsampleLeft, angle);
+
+    /// <summary>
+    /// Interpolates projected neighboring samples for all directional prediction zones.
+    /// </summary>
+    internal readonly struct DirectionalOperator : IDirectionalPredictionOperator
+    {
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static byte Interpolate(byte left, byte right, int weight)
+            => (byte)(((left * (32 - weight)) + (right * weight) + 16) >> 5);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static short Interpolate(short left, short right, int weight)
+            => (short)(((left * (32 - weight)) + (right * weight) + 16) >> 5);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector128<byte> Interpolate(Vector128<byte> left, Vector128<byte> right, int weight)
+        {
+            (Vector128<ushort> leftLow, Vector128<ushort> leftHigh) = Vector128.Widen(left);
+            (Vector128<ushort> rightLow, Vector128<ushort> rightHigh) = Vector128.Widen(right);
+            Vector128<ushort> rounding = Vector128.Create((ushort)16);
+            Vector128<ushort> low = ((leftLow * (ushort)(32 - weight)) + (rightLow * (ushort)weight) + rounding) >> 5;
+            Vector128<ushort> high = ((leftHigh * (ushort)(32 - weight)) + (rightHigh * (ushort)weight) + rounding) >> 5;
+
+            return Vector128.Narrow(low, high);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector256<byte> Interpolate(Vector256<byte> left, Vector256<byte> right, int weight)
+        {
+            (Vector256<ushort> leftLow, Vector256<ushort> leftHigh) = Vector256.Widen(left);
+            (Vector256<ushort> rightLow, Vector256<ushort> rightHigh) = Vector256.Widen(right);
+            Vector256<ushort> rounding = Vector256.Create((ushort)16);
+            Vector256<ushort> low = ((leftLow * (ushort)(32 - weight)) + (rightLow * (ushort)weight) + rounding) >> 5;
+            Vector256<ushort> high = ((leftHigh * (ushort)(32 - weight)) + (rightHigh * (ushort)weight) + rounding) >> 5;
+
+            return Vector256.Narrow(low, high);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector512<byte> Interpolate(Vector512<byte> left, Vector512<byte> right, int weight)
+        {
+            (Vector512<ushort> leftLow, Vector512<ushort> leftHigh) = Vector512.Widen(left);
+            (Vector512<ushort> rightLow, Vector512<ushort> rightHigh) = Vector512.Widen(right);
+            Vector512<ushort> rounding = Vector512.Create((ushort)16);
+            Vector512<ushort> low = ((leftLow * (ushort)(32 - weight)) + (rightLow * (ushort)weight) + rounding) >> 5;
+            Vector512<ushort> high = ((leftHigh * (ushort)(32 - weight)) + (rightHigh * (ushort)weight) + rounding) >> 5;
+
+            return Vector512.Narrow(low, high);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector128<short> Interpolate(Vector128<short> left, Vector128<short> right, int weight)
+        {
+            (Vector128<int> leftLow, Vector128<int> leftHigh) = Vector128.Widen(left);
+            (Vector128<int> rightLow, Vector128<int> rightHigh) = Vector128.Widen(right);
+            Vector128<int> rounding = Vector128.Create(16);
+            Vector128<int> low = ((leftLow * (32 - weight)) + (rightLow * weight) + rounding) >> 5;
+            Vector128<int> high = ((leftHigh * (32 - weight)) + (rightHigh * weight) + rounding) >> 5;
+
+            return Vector128.Narrow(low, high);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector256<short> Interpolate(Vector256<short> left, Vector256<short> right, int weight)
+        {
+            (Vector256<int> leftLow, Vector256<int> leftHigh) = Vector256.Widen(left);
+            (Vector256<int> rightLow, Vector256<int> rightHigh) = Vector256.Widen(right);
+            Vector256<int> rounding = Vector256.Create(16);
+            Vector256<int> low = ((leftLow * (32 - weight)) + (rightLow * weight) + rounding) >> 5;
+            Vector256<int> high = ((leftHigh * (32 - weight)) + (rightHigh * weight) + rounding) >> 5;
+
+            return Vector256.Narrow(low, high);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector512<short> Interpolate(Vector512<short> left, Vector512<short> right, int weight)
+        {
+            (Vector512<int> leftLow, Vector512<int> leftHigh) = Vector512.Widen(left);
+            (Vector512<int> rightLow, Vector512<int> rightHigh) = Vector512.Widen(right);
+            Vector512<int> rounding = Vector512.Create(16);
+            Vector512<int> low = ((leftLow * (32 - weight)) + (rightLow * weight) + rounding) >> 5;
+            Vector512<int> high = ((leftHigh * (32 - weight)) + (rightHigh * weight) + rounding) >> 5;
+
+            return Vector512.Narrow(low, high);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector128<int> Interpolate(Vector128<int> left, Vector128<int> right, Vector128<int> weights)
+            => ((left * (Vector128.Create(32) - weights)) + (right * weights) + Vector128.Create(16)) >> 5;
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/Av1DirectionalIntraPredictor.Transpose.cs

@@ -17,303 +17,310 @@ namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction;
 internal static partial class Av1DirectionalIntraPredictor
 {
     /// <summary>
-    /// Transposes a contiguous 8-bit prediction block into the reconstructed destination.
+    /// Implements the directional traversal for one closed interpolation operator.
     /// </summary>
-    /// <param name="source">The contiguous source block.</param>
-    /// <param name="destination">The destination block origin.</param>
-    /// <param name="sourceWidth">The source row width.</param>
-    /// <param name="sourceHeight">The number of source rows.</param>
-    /// <param name="destinationStride">The destination row stride.</param>
-    private static void Transpose(ReadOnlySpan<byte> source, Span<byte> destination, int sourceWidth, int sourceHeight, int destinationStride)
+    private static partial class Predictor<TOperator>
+        where TOperator : struct, IDirectionalPredictionOperator
     {
-        if (Vector128.IsHardwareAccelerated)
+        /// <summary>
+        /// Transposes a contiguous 8-bit prediction block into the reconstructed destination.
+        /// </summary>
+        /// <param name="source">The contiguous source block.</param>
+        /// <param name="destination">The destination block origin.</param>
+        /// <param name="sourceWidth">The source row width.</param>
+        /// <param name="sourceHeight">The number of source rows.</param>
+        /// <param name="destinationStride">The destination row stride.</param>
+        private static void Transpose(ReadOnlySpan<byte> source, Span<byte> destination, int sourceWidth, int sourceHeight, int destinationStride)
         {
-            // Selecting one tile size for the complete block keeps both loop increments aligned with the AV1 block
-            // dimensions. No partial SIMD tile reaches a neighboring prediction block.
-            int tileSize = sourceWidth >= 8 && sourceHeight >= 8 ? 8 : 4;
-            for (int y = 0; y < sourceHeight; y += tileSize)
+            if (Vector128.IsHardwareAccelerated)
             {
-                for (int x = 0; x < sourceWidth; x += tileSize)
+                // Selecting one tile size for the complete block keeps both loop increments aligned with the AV1 block
+                // dimensions. No partial SIMD tile reaches a neighboring prediction block.
+                int tileSize = sourceWidth >= 8 && sourceHeight >= 8 ? 8 : 4;
+                for (int y = 0; y < sourceHeight; y += tileSize)
                 {
-                    if (tileSize == 8)
+                    for (int x = 0; x < sourceWidth; x += tileSize)
                     {
-                        Transpose8x8(source, destination, sourceWidth, destinationStride, x, y);
-                    }
-                    else
-                    {
-                        Transpose4x4(source, destination, sourceWidth, destinationStride, x, y);
+                        if (tileSize == 8)
+                        {
+                            Transpose8x8(source, destination, sourceWidth, destinationStride, x, y);
+                        }
+                        else
+                        {
+                            Transpose4x4(source, destination, sourceWidth, destinationStride, x, y);
+                        }
                     }
                 }
-            }
 
-            return;
-        }
+                return;
+            }
 
-        for (int y = 0; y < sourceHeight; y++)
-        {
-            for (int x = 0; x < sourceWidth; x++)
+            for (int y = 0; y < sourceHeight; y++)
             {
-                destination[(x * destinationStride) + y] = source[(y * sourceWidth) + x];
+                for (int x = 0; x < sourceWidth; x++)
+                {
+                    destination[(x * destinationStride) + y] = source[(y * sourceWidth) + x];
+                }
             }
         }
-    }
 
-    /// <summary>
-    /// Transposes a contiguous high-bit-depth prediction block into the reconstructed destination.
-    /// </summary>
-    /// <param name="source">The contiguous source block.</param>
-    /// <param name="destination">The destination block origin.</param>
-    /// <param name="sourceWidth">The source row width.</param>
-    /// <param name="sourceHeight">The number of source rows.</param>
-    /// <param name="destinationStride">The destination row stride.</param>
-    private static void Transpose(ReadOnlySpan<short> source, Span<short> destination, int sourceWidth, int sourceHeight, int destinationStride)
-    {
-        if (Vector128.IsHardwareAccelerated)
+        /// <summary>
+        /// Transposes a contiguous high-bit-depth prediction block into the reconstructed destination.
+        /// </summary>
+        /// <param name="source">The contiguous source block.</param>
+        /// <param name="destination">The destination block origin.</param>
+        /// <param name="sourceWidth">The source row width.</param>
+        /// <param name="sourceHeight">The number of source rows.</param>
+        /// <param name="destinationStride">The destination row stride.</param>
+        private static void Transpose(ReadOnlySpan<short> source, Span<short> destination, int sourceWidth, int sourceHeight, int destinationStride)
         {
-            // The same tiling invariant applies to two-byte samples; only the register unpack granularity differs.
-            int tileSize = sourceWidth >= 8 && sourceHeight >= 8 ? 8 : 4;
-            for (int y = 0; y < sourceHeight; y += tileSize)
+            if (Vector128.IsHardwareAccelerated)
             {
-                for (int x = 0; x < sourceWidth; x += tileSize)
+                // The same tiling invariant applies to two-byte samples; only the register unpack granularity differs.
+                int tileSize = sourceWidth >= 8 && sourceHeight >= 8 ? 8 : 4;
+                for (int y = 0; y < sourceHeight; y += tileSize)
                 {
-                    if (tileSize == 8)
+                    for (int x = 0; x < sourceWidth; x += tileSize)
                     {
-                        Transpose8x8(source, destination, sourceWidth, destinationStride, x, y);
-                    }
-                    else
-                    {
-                        Transpose4x4(source, destination, sourceWidth, destinationStride, x, y);
+                        if (tileSize == 8)
+                        {
+                            Transpose8x8(source, destination, sourceWidth, destinationStride, x, y);
+                        }
+                        else
+                        {
+                            Transpose4x4(source, destination, sourceWidth, destinationStride, x, y);
+                        }
                     }
                 }
-            }
 
-            return;
-        }
+                return;
+            }
 
-        for (int y = 0; y < sourceHeight; y++)
-        {
-            for (int x = 0; x < sourceWidth; x++)
+            for (int y = 0; y < sourceHeight; y++)
             {
-                destination[(x * destinationStride) + y] = source[(y * sourceWidth) + x];
+                for (int x = 0; x < sourceWidth; x++)
+                {
+                    destination[(x * destinationStride) + y] = source[(y * sourceWidth) + x];
+                }
             }
         }
-    }
 
-    /// <summary>
-    /// Transposes one eight-by-eight tile of 8-bit prediction samples.
-    /// </summary>
-    /// <param name="source">The contiguous source block.</param>
-    /// <param name="destination">The destination block origin.</param>
-    /// <param name="sourceStride">The source row stride.</param>
-    /// <param name="destinationStride">The destination row stride.</param>
-    /// <param name="x">The tile X coordinate.</param>
-    /// <param name="y">The tile Y coordinate.</param>
-    private static void Transpose8x8(ReadOnlySpan<byte> source, Span<byte> destination, int sourceStride, int destinationStride, int x, int y)
-    {
-        ref byte sourceBase = ref MemoryMarshal.GetReference(source);
-        ref byte destinationBase = ref MemoryMarshal.GetReference(destination);
-        Vector128<byte> row0 = LoadEightBytes(ref sourceBase, ((y + 0) * sourceStride) + x);
-        Vector128<byte> row1 = LoadEightBytes(ref sourceBase, ((y + 1) * sourceStride) + x);
-        Vector128<byte> row2 = LoadEightBytes(ref sourceBase, ((y + 2) * sourceStride) + x);
-        Vector128<byte> row3 = LoadEightBytes(ref sourceBase, ((y + 3) * sourceStride) + x);
-        Vector128<byte> row4 = LoadEightBytes(ref sourceBase, ((y + 4) * sourceStride) + x);
-        Vector128<byte> row5 = LoadEightBytes(ref sourceBase, ((y + 5) * sourceStride) + x);
-        Vector128<byte> row6 = LoadEightBytes(ref sourceBase, ((y + 6) * sourceStride) + x);
-        Vector128<byte> row7 = LoadEightBytes(ref sourceBase, ((y + 7) * sourceStride) + x);
+        /// <summary>
+        /// Transposes one eight-by-eight tile of 8-bit prediction samples.
+        /// </summary>
+        /// <param name="source">The contiguous source block.</param>
+        /// <param name="destination">The destination block origin.</param>
+        /// <param name="sourceStride">The source row stride.</param>
+        /// <param name="destinationStride">The destination row stride.</param>
+        /// <param name="x">The tile X coordinate.</param>
+        /// <param name="y">The tile Y coordinate.</param>
+        private static void Transpose8x8(ReadOnlySpan<byte> source, Span<byte> destination, int sourceStride, int destinationStride, int x, int y)
+        {
+            ref byte sourceBase = ref MemoryMarshal.GetReference(source);
+            ref byte destinationBase = ref MemoryMarshal.GetReference(destination);
+            Vector128<byte> row0 = LoadEightBytes(ref sourceBase, ((y + 0) * sourceStride) + x);
+            Vector128<byte> row1 = LoadEightBytes(ref sourceBase, ((y + 1) * sourceStride) + x);
+            Vector128<byte> row2 = LoadEightBytes(ref sourceBase, ((y + 2) * sourceStride) + x);
+            Vector128<byte> row3 = LoadEightBytes(ref sourceBase, ((y + 3) * sourceStride) + x);
+            Vector128<byte> row4 = LoadEightBytes(ref sourceBase, ((y + 4) * sourceStride) + x);
+            Vector128<byte> row5 = LoadEightBytes(ref sourceBase, ((y + 5) * sourceStride) + x);
+            Vector128<byte> row6 = LoadEightBytes(ref sourceBase, ((y + 6) * sourceStride) + x);
+            Vector128<byte> row7 = LoadEightBytes(ref sourceBase, ((y + 7) * sourceStride) + x);
 
-        // Three unpack stages exchange one, two, then four byte coordinates. Each final vector contains two
-        // complete source columns, which are written as two contiguous eight-byte destination rows.
-        Vector128<byte> pair0 = Vector128_.UnpackLow(row0, row1);
-        Vector128<byte> pair2 = Vector128_.UnpackLow(row2, row3);
-        Vector128<byte> pair4 = Vector128_.UnpackLow(row4, row5);
-        Vector128<byte> pair6 = Vector128_.UnpackLow(row6, row7);
-        Vector128<short> quad0 = Vector128_.UnpackLow(pair0.AsInt16(), pair2.AsInt16());
-        Vector128<short> quad1 = Vector128_.UnpackHigh(pair0.AsInt16(), pair2.AsInt16());
-        Vector128<short> quad4 = Vector128_.UnpackLow(pair4.AsInt16(), pair6.AsInt16());
-        Vector128<short> quad5 = Vector128_.UnpackHigh(pair4.AsInt16(), pair6.AsInt16());
-        Vector128<int> columns01 = Vector128_.UnpackLow(quad0.AsInt32(), quad4.AsInt32());
-        Vector128<int> columns23 = Vector128_.UnpackHigh(quad0.AsInt32(), quad4.AsInt32());
-        Vector128<int> columns45 = Vector128_.UnpackLow(quad1.AsInt32(), quad5.AsInt32());
-        Vector128<int> columns67 = Vector128_.UnpackHigh(quad1.AsInt32(), quad5.AsInt32());
+            // Three unpack stages exchange one, two, then four byte coordinates. Each final vector contains two
+            // complete source columns, which are written as two contiguous eight-byte destination rows.
+            Vector128<byte> pair0 = Vector128_.UnpackLow(row0, row1);
+            Vector128<byte> pair2 = Vector128_.UnpackLow(row2, row3);
+            Vector128<byte> pair4 = Vector128_.UnpackLow(row4, row5);
+            Vector128<byte> pair6 = Vector128_.UnpackLow(row6, row7);
+            Vector128<short> quad0 = Vector128_.UnpackLow(pair0.AsInt16(), pair2.AsInt16());
+            Vector128<short> quad1 = Vector128_.UnpackHigh(pair0.AsInt16(), pair2.AsInt16());
+            Vector128<short> quad4 = Vector128_.UnpackLow(pair4.AsInt16(), pair6.AsInt16());
+            Vector128<short> quad5 = Vector128_.UnpackHigh(pair4.AsInt16(), pair6.AsInt16());
+            Vector128<int> columns01 = Vector128_.UnpackLow(quad0.AsInt32(), quad4.AsInt32());
+            Vector128<int> columns23 = Vector128_.UnpackHigh(quad0.AsInt32(), quad4.AsInt32());
+            Vector128<int> columns45 = Vector128_.UnpackLow(quad1.AsInt32(), quad5.AsInt32());
+            Vector128<int> columns67 = Vector128_.UnpackHigh(quad1.AsInt32(), quad5.AsInt32());
 
-        StoreEightBytes(columns01.AsUInt64().GetElement(0), ref destinationBase, ((x + 0) * destinationStride) + y);
-        StoreEightBytes(columns01.AsUInt64().GetElement(1), ref destinationBase, ((x + 1) * destinationStride) + y);
-        StoreEightBytes(columns23.AsUInt64().GetElement(0), ref destinationBase, ((x + 2) * destinationStride) + y);
-        StoreEightBytes(columns23.AsUInt64().GetElement(1), ref destinationBase, ((x + 3) * destinationStride) + y);
-        StoreEightBytes(columns45.AsUInt64().GetElement(0), ref destinationBase, ((x + 4) * destinationStride) + y);
-        StoreEightBytes(columns45.AsUInt64().GetElement(1), ref destinationBase, ((x + 5) * destinationStride) + y);
-        StoreEightBytes(columns67.AsUInt64().GetElement(0), ref destinationBase, ((x + 6) * destinationStride) + y);
-        StoreEightBytes(columns67.AsUInt64().GetElement(1), ref destinationBase, ((x + 7) * destinationStride) + y);
-    }
+            StoreEightBytes(columns01.AsUInt64().ToScalar(), ref destinationBase, ((x + 0) * destinationStride) + y);
+            StoreEightBytes(columns01.AsUInt64().GetElement(1), ref destinationBase, ((x + 1) * destinationStride) + y);
+            StoreEightBytes(columns23.AsUInt64().ToScalar(), ref destinationBase, ((x + 2) * destinationStride) + y);
+            StoreEightBytes(columns23.AsUInt64().GetElement(1), ref destinationBase, ((x + 3) * destinationStride) + y);
+            StoreEightBytes(columns45.AsUInt64().ToScalar(), ref destinationBase, ((x + 4) * destinationStride) + y);
+            StoreEightBytes(columns45.AsUInt64().GetElement(1), ref destinationBase, ((x + 5) * destinationStride) + y);
+            StoreEightBytes(columns67.AsUInt64().ToScalar(), ref destinationBase, ((x + 6) * destinationStride) + y);
+            StoreEightBytes(columns67.AsUInt64().GetElement(1), ref destinationBase, ((x + 7) * destinationStride) + y);
+        }
 
-    /// <summary>
-    /// Transposes one four-by-four tile of 8-bit prediction samples.
-    /// </summary>
-    /// <param name="source">The contiguous source block.</param>
-    /// <param name="destination">The destination block origin.</param>
-    /// <param name="sourceStride">The source row stride.</param>
-    /// <param name="destinationStride">The destination row stride.</param>
-    /// <param name="x">The tile X coordinate.</param>
-    /// <param name="y">The tile Y coordinate.</param>
-    private static void Transpose4x4(ReadOnlySpan<byte> source, Span<byte> destination, int sourceStride, int destinationStride, int x, int y)
-    {
-        ref byte sourceBase = ref MemoryMarshal.GetReference(source);
-        ref byte destinationBase = ref MemoryMarshal.GetReference(destination);
-        Vector128<byte> row0 = LoadFourBytes(ref sourceBase, ((y + 0) * sourceStride) + x);
-        Vector128<byte> row1 = LoadFourBytes(ref sourceBase, ((y + 1) * sourceStride) + x);
-        Vector128<byte> row2 = LoadFourBytes(ref sourceBase, ((y + 2) * sourceStride) + x);
-        Vector128<byte> row3 = LoadFourBytes(ref sourceBase, ((y + 3) * sourceStride) + x);
-        Vector128<byte> pair0 = Vector128_.UnpackLow(row0, row1);
-        Vector128<byte> pair1 = Vector128_.UnpackLow(row2, row3);
-        Vector128<short> columns = Vector128_.UnpackLow(pair0.AsInt16(), pair1.AsInt16());
+        /// <summary>
+        /// Transposes one four-by-four tile of 8-bit prediction samples.
+        /// </summary>
+        /// <param name="source">The contiguous source block.</param>
+        /// <param name="destination">The destination block origin.</param>
+        /// <param name="sourceStride">The source row stride.</param>
+        /// <param name="destinationStride">The destination row stride.</param>
+        /// <param name="x">The tile X coordinate.</param>
+        /// <param name="y">The tile Y coordinate.</param>
+        private static void Transpose4x4(ReadOnlySpan<byte> source, Span<byte> destination, int sourceStride, int destinationStride, int x, int y)
+        {
+            ref byte sourceBase = ref MemoryMarshal.GetReference(source);
+            ref byte destinationBase = ref MemoryMarshal.GetReference(destination);
+            Vector128<byte> row0 = LoadFourBytes(ref sourceBase, ((y + 0) * sourceStride) + x);
+            Vector128<byte> row1 = LoadFourBytes(ref sourceBase, ((y + 1) * sourceStride) + x);
+            Vector128<byte> row2 = LoadFourBytes(ref sourceBase, ((y + 2) * sourceStride) + x);
+            Vector128<byte> row3 = LoadFourBytes(ref sourceBase, ((y + 3) * sourceStride) + x);
+            Vector128<byte> pair0 = Vector128_.UnpackLow(row0, row1);
+            Vector128<byte> pair1 = Vector128_.UnpackLow(row2, row3);
+            Vector128<short> columns = Vector128_.UnpackLow(pair0.AsInt16(), pair1.AsInt16());
 
-        Vector128<uint> packedColumns = columns.AsUInt32();
-        StoreFourBytes(packedColumns.GetElement(0), ref destinationBase, ((x + 0) * destinationStride) + y);
-        StoreFourBytes(packedColumns.GetElement(1), ref destinationBase, ((x + 1) * destinationStride) + y);
-        StoreFourBytes(packedColumns.GetElement(2), ref destinationBase, ((x + 2) * destinationStride) + y);
-        StoreFourBytes(packedColumns.GetElement(3), ref destinationBase, ((x + 3) * destinationStride) + y);
-    }
+            Vector128<uint> packedColumns = columns.AsUInt32();
+            StoreFourBytes(packedColumns.GetElement(0), ref destinationBase, ((x + 0) * destinationStride) + y);
+            StoreFourBytes(packedColumns.GetElement(1), ref destinationBase, ((x + 1) * destinationStride) + y);
+            StoreFourBytes(packedColumns.GetElement(2), ref destinationBase, ((x + 2) * destinationStride) + y);
+            StoreFourBytes(packedColumns.GetElement(3), ref destinationBase, ((x + 3) * destinationStride) + y);
+        }
 
-    /// <summary>
-    /// Transposes one eight-by-eight tile of high-bit-depth prediction samples.
-    /// </summary>
-    /// <param name="source">The contiguous source block.</param>
-    /// <param name="destination">The destination block origin.</param>
-    /// <param name="sourceStride">The source row stride.</param>
-    /// <param name="destinationStride">The destination row stride.</param>
-    /// <param name="x">The tile X coordinate.</param>
-    /// <param name="y">The tile Y coordinate.</param>
-    private static void Transpose8x8(ReadOnlySpan<short> source, Span<short> destination, int sourceStride, int destinationStride, int x, int y)
-    {
-        ref short sourceBase = ref MemoryMarshal.GetReference(source);
-        ref short destinationBase = ref MemoryMarshal.GetReference(destination);
-        Vector128<short> row0 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 0) * sourceStride) + x));
-        Vector128<short> row1 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 1) * sourceStride) + x));
-        Vector128<short> row2 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 2) * sourceStride) + x));
-        Vector128<short> row3 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 3) * sourceStride) + x));
-        Vector128<short> row4 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 4) * sourceStride) + x));
-        Vector128<short> row5 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 5) * sourceStride) + x));
-        Vector128<short> row6 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 6) * sourceStride) + x));
-        Vector128<short> row7 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 7) * sourceStride) + x));
-        Vector128<short> pair0 = Vector128_.UnpackLow(row0, row1);
-        Vector128<short> pair1 = Vector128_.UnpackHigh(row0, row1);
-        Vector128<short> pair2 = Vector128_.UnpackLow(row2, row3);
-        Vector128<short> pair3 = Vector128_.UnpackHigh(row2, row3);
-        Vector128<short> pair4 = Vector128_.UnpackLow(row4, row5);
-        Vector128<short> pair5 = Vector128_.UnpackHigh(row4, row5);
-        Vector128<short> pair6 = Vector128_.UnpackLow(row6, row7);
-        Vector128<short> pair7 = Vector128_.UnpackHigh(row6, row7);
-        Vector128<int> quad0 = Vector128_.UnpackLow(pair0.AsInt32(), pair2.AsInt32());
-        Vector128<int> quad1 = Vector128_.UnpackHigh(pair0.AsInt32(), pair2.AsInt32());
-        Vector128<int> quad2 = Vector128_.UnpackLow(pair1.AsInt32(), pair3.AsInt32());
-        Vector128<int> quad3 = Vector128_.UnpackHigh(pair1.AsInt32(), pair3.AsInt32());
-        Vector128<int> quad4 = Vector128_.UnpackLow(pair4.AsInt32(), pair6.AsInt32());
-        Vector128<int> quad5 = Vector128_.UnpackHigh(pair4.AsInt32(), pair6.AsInt32());
-        Vector128<int> quad6 = Vector128_.UnpackLow(pair5.AsInt32(), pair7.AsInt32());
-        Vector128<int> quad7 = Vector128_.UnpackHigh(pair5.AsInt32(), pair7.AsInt32());
-        Vector128<short> column0 = Vector128_.UnpackLow(quad0.AsInt64(), quad4.AsInt64()).AsInt16();
-        Vector128<short> column1 = Vector128_.UnpackHigh(quad0.AsInt64(), quad4.AsInt64()).AsInt16();
-        Vector128<short> column2 = Vector128_.UnpackLow(quad1.AsInt64(), quad5.AsInt64()).AsInt16();
-        Vector128<short> column3 = Vector128_.UnpackHigh(quad1.AsInt64(), quad5.AsInt64()).AsInt16();
-        Vector128<short> column4 = Vector128_.UnpackLow(quad2.AsInt64(), quad6.AsInt64()).AsInt16();
-        Vector128<short> column5 = Vector128_.UnpackHigh(quad2.AsInt64(), quad6.AsInt64()).AsInt16();
-        Vector128<short> column6 = Vector128_.UnpackLow(quad3.AsInt64(), quad7.AsInt64()).AsInt16();
-        Vector128<short> column7 = Vector128_.UnpackHigh(quad3.AsInt64(), quad7.AsInt64()).AsInt16();
-        column0.StoreUnsafe(ref destinationBase, (nuint)(((x + 0) * destinationStride) + y));
-        column1.StoreUnsafe(ref destinationBase, (nuint)(((x + 1) * destinationStride) + y));
-        column2.StoreUnsafe(ref destinationBase, (nuint)(((x + 2) * destinationStride) + y));
-        column3.StoreUnsafe(ref destinationBase, (nuint)(((x + 3) * destinationStride) + y));
-        column4.StoreUnsafe(ref destinationBase, (nuint)(((x + 4) * destinationStride) + y));
-        column5.StoreUnsafe(ref destinationBase, (nuint)(((x + 5) * destinationStride) + y));
-        column6.StoreUnsafe(ref destinationBase, (nuint)(((x + 6) * destinationStride) + y));
-        column7.StoreUnsafe(ref destinationBase, (nuint)(((x + 7) * destinationStride) + y));
-    }
+        /// <summary>
+        /// Transposes one eight-by-eight tile of high-bit-depth prediction samples.
+        /// </summary>
+        /// <param name="source">The contiguous source block.</param>
+        /// <param name="destination">The destination block origin.</param>
+        /// <param name="sourceStride">The source row stride.</param>
+        /// <param name="destinationStride">The destination row stride.</param>
+        /// <param name="x">The tile X coordinate.</param>
+        /// <param name="y">The tile Y coordinate.</param>
+        private static void Transpose8x8(ReadOnlySpan<short> source, Span<short> destination, int sourceStride, int destinationStride, int x, int y)
+        {
+            ref short sourceBase = ref MemoryMarshal.GetReference(source);
+            ref short destinationBase = ref MemoryMarshal.GetReference(destination);
+            Vector128<short> row0 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 0) * sourceStride) + x));
+            Vector128<short> row1 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 1) * sourceStride) + x));
+            Vector128<short> row2 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 2) * sourceStride) + x));
+            Vector128<short> row3 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 3) * sourceStride) + x));
+            Vector128<short> row4 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 4) * sourceStride) + x));
+            Vector128<short> row5 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 5) * sourceStride) + x));
+            Vector128<short> row6 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 6) * sourceStride) + x));
+            Vector128<short> row7 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 7) * sourceStride) + x));
+            Vector128<short> pair0 = Vector128_.UnpackLow(row0, row1);
+            Vector128<short> pair1 = Vector128_.UnpackHigh(row0, row1);
+            Vector128<short> pair2 = Vector128_.UnpackLow(row2, row3);
+            Vector128<short> pair3 = Vector128_.UnpackHigh(row2, row3);
+            Vector128<short> pair4 = Vector128_.UnpackLow(row4, row5);
+            Vector128<short> pair5 = Vector128_.UnpackHigh(row4, row5);
+            Vector128<short> pair6 = Vector128_.UnpackLow(row6, row7);
+            Vector128<short> pair7 = Vector128_.UnpackHigh(row6, row7);
+            Vector128<int> quad0 = Vector128_.UnpackLow(pair0.AsInt32(), pair2.AsInt32());
+            Vector128<int> quad1 = Vector128_.UnpackHigh(pair0.AsInt32(), pair2.AsInt32());
+            Vector128<int> quad2 = Vector128_.UnpackLow(pair1.AsInt32(), pair3.AsInt32());
+            Vector128<int> quad3 = Vector128_.UnpackHigh(pair1.AsInt32(), pair3.AsInt32());
+            Vector128<int> quad4 = Vector128_.UnpackLow(pair4.AsInt32(), pair6.AsInt32());
+            Vector128<int> quad5 = Vector128_.UnpackHigh(pair4.AsInt32(), pair6.AsInt32());
+            Vector128<int> quad6 = Vector128_.UnpackLow(pair5.AsInt32(), pair7.AsInt32());
+            Vector128<int> quad7 = Vector128_.UnpackHigh(pair5.AsInt32(), pair7.AsInt32());
+            Vector128<short> column0 = Vector128_.UnpackLow(quad0.AsInt64(), quad4.AsInt64()).AsInt16();
+            Vector128<short> column1 = Vector128_.UnpackHigh(quad0.AsInt64(), quad4.AsInt64()).AsInt16();
+            Vector128<short> column2 = Vector128_.UnpackLow(quad1.AsInt64(), quad5.AsInt64()).AsInt16();
+            Vector128<short> column3 = Vector128_.UnpackHigh(quad1.AsInt64(), quad5.AsInt64()).AsInt16();
+            Vector128<short> column4 = Vector128_.UnpackLow(quad2.AsInt64(), quad6.AsInt64()).AsInt16();
+            Vector128<short> column5 = Vector128_.UnpackHigh(quad2.AsInt64(), quad6.AsInt64()).AsInt16();
+            Vector128<short> column6 = Vector128_.UnpackLow(quad3.AsInt64(), quad7.AsInt64()).AsInt16();
+            Vector128<short> column7 = Vector128_.UnpackHigh(quad3.AsInt64(), quad7.AsInt64()).AsInt16();
+            column0.StoreUnsafe(ref destinationBase, (nuint)(((x + 0) * destinationStride) + y));
+            column1.StoreUnsafe(ref destinationBase, (nuint)(((x + 1) * destinationStride) + y));
+            column2.StoreUnsafe(ref destinationBase, (nuint)(((x + 2) * destinationStride) + y));
+            column3.StoreUnsafe(ref destinationBase, (nuint)(((x + 3) * destinationStride) + y));
+            column4.StoreUnsafe(ref destinationBase, (nuint)(((x + 4) * destinationStride) + y));
+            column5.StoreUnsafe(ref destinationBase, (nuint)(((x + 5) * destinationStride) + y));
+            column6.StoreUnsafe(ref destinationBase, (nuint)(((x + 6) * destinationStride) + y));
+            column7.StoreUnsafe(ref destinationBase, (nuint)(((x + 7) * destinationStride) + y));
+        }
 
-    /// <summary>
-    /// Transposes one four-by-four tile of high-bit-depth prediction samples.
-    /// </summary>
-    /// <param name="source">The contiguous source block.</param>
-    /// <param name="destination">The destination block origin.</param>
-    /// <param name="sourceStride">The source row stride.</param>
-    /// <param name="destinationStride">The destination row stride.</param>
-    /// <param name="x">The tile X coordinate.</param>
-    /// <param name="y">The tile Y coordinate.</param>
-    private static void Transpose4x4(ReadOnlySpan<short> source, Span<short> destination, int sourceStride, int destinationStride, int x, int y)
-    {
-        ref short sourceBase = ref MemoryMarshal.GetReference(source);
-        ref short destinationBase = ref MemoryMarshal.GetReference(destination);
-        Vector128<short> row0 = LoadFourShorts(ref sourceBase, ((y + 0) * sourceStride) + x);
-        Vector128<short> row1 = LoadFourShorts(ref sourceBase, ((y + 1) * sourceStride) + x);
-        Vector128<short> row2 = LoadFourShorts(ref sourceBase, ((y + 2) * sourceStride) + x);
-        Vector128<short> row3 = LoadFourShorts(ref sourceBase, ((y + 3) * sourceStride) + x);
-        Vector128<short> pair0 = Vector128_.UnpackLow(row0, row1);
-        Vector128<short> pair1 = Vector128_.UnpackLow(row2, row3);
-        Vector128<int> columns01 = Vector128_.UnpackLow(pair0.AsInt32(), pair1.AsInt32());
-        Vector128<int> columns23 = Vector128_.UnpackHigh(pair0.AsInt32(), pair1.AsInt32());
+        /// <summary>
+        /// Transposes one four-by-four tile of high-bit-depth prediction samples.
+        /// </summary>
+        /// <param name="source">The contiguous source block.</param>
+        /// <param name="destination">The destination block origin.</param>
+        /// <param name="sourceStride">The source row stride.</param>
+        /// <param name="destinationStride">The destination row stride.</param>
+        /// <param name="x">The tile X coordinate.</param>
+        /// <param name="y">The tile Y coordinate.</param>
+        private static void Transpose4x4(ReadOnlySpan<short> source, Span<short> destination, int sourceStride, int destinationStride, int x, int y)
+        {
+            ref short sourceBase = ref MemoryMarshal.GetReference(source);
+            ref short destinationBase = ref MemoryMarshal.GetReference(destination);
+            Vector128<short> row0 = LoadFourShorts(ref sourceBase, ((y + 0) * sourceStride) + x);
+            Vector128<short> row1 = LoadFourShorts(ref sourceBase, ((y + 1) * sourceStride) + x);
+            Vector128<short> row2 = LoadFourShorts(ref sourceBase, ((y + 2) * sourceStride) + x);
+            Vector128<short> row3 = LoadFourShorts(ref sourceBase, ((y + 3) * sourceStride) + x);
+            Vector128<short> pair0 = Vector128_.UnpackLow(row0, row1);
+            Vector128<short> pair1 = Vector128_.UnpackLow(row2, row3);
+            Vector128<int> columns01 = Vector128_.UnpackLow(pair0.AsInt32(), pair1.AsInt32());
+            Vector128<int> columns23 = Vector128_.UnpackHigh(pair0.AsInt32(), pair1.AsInt32());
 
-        StoreFourShorts(columns01.AsUInt64().GetElement(0), ref destinationBase, ((x + 0) * destinationStride) + y);
-        StoreFourShorts(columns01.AsUInt64().GetElement(1), ref destinationBase, ((x + 1) * destinationStride) + y);
-        StoreFourShorts(columns23.AsUInt64().GetElement(0), ref destinationBase, ((x + 2) * destinationStride) + y);
-        StoreFourShorts(columns23.AsUInt64().GetElement(1), ref destinationBase, ((x + 3) * destinationStride) + y);
-    }
+            StoreFourShorts(columns01.AsUInt64().ToScalar(), ref destinationBase, ((x + 0) * destinationStride) + y);
+            StoreFourShorts(columns01.AsUInt64().GetElement(1), ref destinationBase, ((x + 1) * destinationStride) + y);
+            StoreFourShorts(columns23.AsUInt64().ToScalar(), ref destinationBase, ((x + 2) * destinationStride) + y);
+            StoreFourShorts(columns23.AsUInt64().GetElement(1), ref destinationBase, ((x + 3) * destinationStride) + y);
+        }
 
-    /// <summary>
-    /// Loads eight bytes into the lower half of a vector without reading past a tile row.
-    /// </summary>
-    /// <param name="source">The source buffer origin.</param>
-    /// <param name="offset">The source offset.</param>
-    /// <returns>The loaded bytes.</returns>
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector128<byte> LoadEightBytes(ref byte source, int offset)
-        => Vector128.Create(Unsafe.As<byte, ulong>(ref Unsafe.Add(ref source, offset)), 0UL).AsByte();
+        /// <summary>
+        /// Loads eight bytes into the lower half of a vector without reading past a tile row.
+        /// </summary>
+        /// <param name="source">The source buffer origin.</param>
+        /// <param name="offset">The source offset.</param>
+        /// <returns>The loaded bytes.</returns>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        private static Vector128<byte> LoadEightBytes(ref byte source, int offset)
+            => Vector128.Create(Unsafe.As<byte, ulong>(ref Unsafe.Add(ref source, offset)), 0UL).AsByte();
 
-    /// <summary>
-    /// Loads four bytes into the low vector lanes without reading past a tile row.
-    /// </summary>
-    /// <param name="source">The source buffer origin.</param>
-    /// <param name="offset">The source offset.</param>
-    /// <returns>The loaded bytes.</returns>
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector128<byte> LoadFourBytes(ref byte source, int offset)
-        => Vector128.Create(Unsafe.As<byte, uint>(ref Unsafe.Add(ref source, offset)), 0U, 0U, 0U).AsByte();
+        /// <summary>
+        /// Loads four bytes into the low vector lanes without reading past a tile row.
+        /// </summary>
+        /// <param name="source">The source buffer origin.</param>
+        /// <param name="offset">The source offset.</param>
+        /// <returns>The loaded bytes.</returns>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        private static Vector128<byte> LoadFourBytes(ref byte source, int offset)
+            => Vector128.Create(Unsafe.As<byte, uint>(ref Unsafe.Add(ref source, offset)), 0U, 0U, 0U).AsByte();
 
-    /// <summary>
-    /// Loads four high-bit-depth samples into the lower half of a vector without reading past a tile row.
-    /// </summary>
-    /// <param name="source">The source buffer origin.</param>
-    /// <param name="offset">The source offset.</param>
-    /// <returns>The loaded samples.</returns>
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector128<short> LoadFourShorts(ref short source, int offset)
-        => Vector128.Create(Unsafe.As<short, ulong>(ref Unsafe.Add(ref source, offset)), 0UL).AsInt16();
+        /// <summary>
+        /// Loads four high-bit-depth samples into the lower half of a vector without reading past a tile row.
+        /// </summary>
+        /// <param name="source">The source buffer origin.</param>
+        /// <param name="offset">The source offset.</param>
+        /// <returns>The loaded samples.</returns>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        private static Vector128<short> LoadFourShorts(ref short source, int offset)
+            => Vector128.Create(Unsafe.As<short, ulong>(ref Unsafe.Add(ref source, offset)), 0UL).AsInt16();
 
-    /// <summary>
-    /// Stores the lower eight bytes of a vector.
-    /// </summary>
-    /// <param name="source">The packed bytes.</param>
-    /// <param name="destination">The destination buffer origin.</param>
-    /// <param name="offset">The destination offset.</param>
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static void StoreEightBytes(ulong source, ref byte destination, int offset)
-        => Unsafe.As<byte, ulong>(ref Unsafe.Add(ref destination, offset)) = source;
+        /// <summary>
+        /// Stores the lower eight bytes of a vector.
+        /// </summary>
+        /// <param name="source">The packed bytes.</param>
+        /// <param name="destination">The destination buffer origin.</param>
+        /// <param name="offset">The destination offset.</param>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        private static void StoreEightBytes(ulong source, ref byte destination, int offset)
+            => Unsafe.As<byte, ulong>(ref Unsafe.Add(ref destination, offset)) = source;
 
-    /// <summary>
-    /// Stores four bytes from the low vector lanes.
-    /// </summary>
-    /// <param name="source">The packed bytes.</param>
-    /// <param name="destination">The destination buffer origin.</param>
-    /// <param name="offset">The destination offset.</param>
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static void StoreFourBytes(uint source, ref byte destination, int offset)
-        => Unsafe.As<byte, uint>(ref Unsafe.Add(ref destination, offset)) = source;
+        /// <summary>
+        /// Stores four bytes from the low vector lanes.
+        /// </summary>
+        /// <param name="source">The packed bytes.</param>
+        /// <param name="destination">The destination buffer origin.</param>
+        /// <param name="offset">The destination offset.</param>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        private static void StoreFourBytes(uint source, ref byte destination, int offset)
+            => Unsafe.As<byte, uint>(ref Unsafe.Add(ref destination, offset)) = source;
 
-    /// <summary>
-    /// Stores four high-bit-depth samples from the low vector lanes.
-    /// </summary>
-    /// <param name="source">The packed samples.</param>
-    /// <param name="destination">The destination buffer origin.</param>
-    /// <param name="offset">The destination offset.</param>
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static void StoreFourShorts(ulong source, ref short destination, int offset)
-        => Unsafe.As<short, ulong>(ref Unsafe.Add(ref destination, offset)) = source;
+        /// <summary>
+        /// Stores four high-bit-depth samples from the low vector lanes.
+        /// </summary>
+        /// <param name="source">The packed samples.</param>
+        /// <param name="destination">The destination buffer origin.</param>
+        /// <param name="offset">The destination offset.</param>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        private static void StoreFourShorts(ulong source, ref short destination, int offset)
+            => Unsafe.As<short, ulong>(ref Unsafe.Add(ref destination, offset)) = source;
+    }
 }

src/ImageSharp/Formats/Heif/Av1/Prediction/Av1DirectionalIntraPredictor.cs

@@ -1,443 +0,0 @@
-// Copyright (c) Six Labors.
-// Licensed under the Six Labors Split License.
-
-using System.Runtime.CompilerServices;
-using SixLabors.ImageSharp.Formats.Heif.Av1.Transform;
-
-namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction;
-
-/// <summary>
-/// Reconstructs AV1 directional intra-prediction blocks from prepared neighboring samples.
-/// </summary>
-/// <remarks>
-/// The three projection zones implement the directional prediction process in section 7.11.2.4 of the AV1 specification.
-/// </remarks>
-internal static partial class Av1DirectionalIntraPredictor
-{
-    /// <summary>
-    /// The largest number of samples required to transpose a directional prediction block.
-    /// </summary>
-    public const int ScratchLength = 64 * 64;
-
-    /// <summary>
-    /// Gets the Q8 directional derivatives indexed by acute prediction angle.
-    /// </summary>
-    private static ReadOnlySpan<int> DirectionalIntraDerivative =>
-    [
-
-        // Zero entries represent angles which AV1 never signals. Direct indexing avoids a search or division in
-        // each directional block while retaining the exact fixed-point projections from the normative table.
-        0, 0, 0, 1023, 0, 0, 547, 0, 0, 372, 0, 0, 0, 0, 273, 0, 0, 215, 0, 0, 178, 0, 0,
-        151, 0, 0, 132, 0, 0, 116, 0, 0, 102, 0, 0, 0, 90, 0, 0, 80, 0, 0, 71, 0, 0, 64, 0, 0,
-        57, 0, 0, 51, 0, 0, 45, 0, 0, 0, 40, 0, 0, 35, 0, 0, 31, 0, 0, 27, 0, 0, 23, 0, 0,
-        19, 0, 0, 15, 0, 0, 0, 0, 11, 0, 0, 7, 0, 0, 3, 0, 0,
-    ];
-
-    /// <summary>
-    /// Predicts an 8-bit directional block using the widest available SIMD path.
-    /// </summary>
-    /// <param name="destination">The destination block origin.</param>
-    /// <param name="destinationStride">The destination row stride in samples.</param>
-    /// <param name="transformSize">The predicted block dimensions.</param>
-    /// <param name="above">The prepared top reference, including any required extension.</param>
-    /// <param name="left">The prepared left reference, including any required extension.</param>
-    /// <param name="upsampleAbove">Whether the top edge contains half-sample positions.</param>
-    /// <param name="upsampleLeft">Whether the left edge contains half-sample positions.</param>
-    /// <param name="angle">The adjusted prediction angle.</param>
-    /// <param name="scratch">The caller-owned block transposition workspace.</param>
-    public static void Predict(Span<byte> destination, int destinationStride, Av1TransformSize transformSize, ReadOnlySpan<byte> above, ReadOnlySpan<byte> left, bool upsampleAbove, bool upsampleLeft, int angle, Span<byte> scratch)
-    {
-        int width = transformSize.GetWidth();
-        int height = transformSize.GetHeight();
-
-        if (angle is > 0 and < 90)
-        {
-            PredictZone1(destination, destinationStride, above, upsampleAbove, GetDeltaX(angle), width, height);
-        }
-        else if (angle is > 90 and < 180)
-        {
-            PredictZone2(destination, destinationStride, above, left, upsampleAbove, upsampleLeft, GetDeltaX(angle), GetDeltaY(angle), width, height);
-        }
-        else if (angle is > 180 and < 270)
-        {
-            // libaom computes zone 3 as a zone 1 block with swapped dimensions, then transposes it. This preserves
-            // contiguous reference reads and destination stores in both hot stages instead of scattering columns.
-            Span<byte> transposed = scratch[..(width * height)];
-            PredictZone1(transposed, height, left, upsampleLeft, GetDeltaY(angle), height, width);
-            Transpose(transposed, destination, height, width, destinationStride);
-        }
-        else
-        {
-            Av1PredictionMode mode = angle == 90 ? Av1PredictionMode.Vertical : Av1PredictionMode.Horizontal;
-            Av1IntraPredictorBase.GetPredictor(mode).Predict(destination, destinationStride, above, left, width, height);
-        }
-    }
-
-    /// <summary>
-    /// Predicts a high-bit-depth directional block using the widest available SIMD path.
-    /// </summary>
-    /// <param name="destination">The destination block origin.</param>
-    /// <param name="destinationStride">The destination row stride in samples.</param>
-    /// <param name="transformSize">The predicted block dimensions.</param>
-    /// <param name="above">The prepared top reference, including any required extension.</param>
-    /// <param name="left">The prepared left reference, including any required extension.</param>
-    /// <param name="upsampleAbove">Whether the top edge contains half-sample positions.</param>
-    /// <param name="upsampleLeft">Whether the left edge contains half-sample positions.</param>
-    /// <param name="angle">The adjusted prediction angle.</param>
-    /// <param name="scratch">The caller-owned block transposition workspace.</param>
-    public static void Predict(Span<short> destination, int destinationStride, Av1TransformSize transformSize, ReadOnlySpan<short> above, ReadOnlySpan<short> left, bool upsampleAbove, bool upsampleLeft, int angle, Span<short> scratch)
-    {
-        int width = transformSize.GetWidth();
-        int height = transformSize.GetHeight();
-
-        if (angle is > 0 and < 90)
-        {
-            PredictZone1(destination, destinationStride, above, upsampleAbove, GetDeltaX(angle), width, height);
-        }
-        else if (angle is > 90 and < 180)
-        {
-            PredictZone2(destination, destinationStride, above, left, upsampleAbove, upsampleLeft, GetDeltaX(angle), GetDeltaY(angle), width, height);
-        }
-        else if (angle is > 180 and < 270)
-        {
-            Span<short> transposed = scratch[..(width * height)];
-            PredictZone1(transposed, height, left, upsampleLeft, GetDeltaY(angle), height, width);
-            Transpose(transposed, destination, height, width, destinationStride);
-        }
-        else
-        {
-            Av1PredictionMode mode = angle == 90 ? Av1PredictionMode.Vertical : Av1PredictionMode.Horizontal;
-            Av1IntraPredictorBase.GetPredictor(mode).Predict(destination, destinationStride, above, left, width, height);
-        }
-    }
-
-    /// <summary>
-    /// Predicts an 8-bit directional block without hardware intrinsics.
-    /// </summary>
-    /// <param name="destination">The destination block origin.</param>
-    /// <param name="destinationStride">The destination row stride in samples.</param>
-    /// <param name="transformSize">The predicted block dimensions.</param>
-    /// <param name="above">The prepared top reference, including any required extension.</param>
-    /// <param name="left">The prepared left reference, including any required extension.</param>
-    /// <param name="upsampleAbove">Whether the top edge contains half-sample positions.</param>
-    /// <param name="upsampleLeft">Whether the left edge contains half-sample positions.</param>
-    /// <param name="angle">The adjusted prediction angle.</param>
-    public static void PredictScalar(Span<byte> destination, int destinationStride, Av1TransformSize transformSize, ReadOnlySpan<byte> above, ReadOnlySpan<byte> left, bool upsampleAbove, bool upsampleLeft, int angle)
-    {
-        int width = transformSize.GetWidth();
-        int height = transformSize.GetHeight();
-
-        if (angle is > 0 and < 90)
-        {
-            PredictZone1Scalar(destination, destinationStride, above, upsampleAbove, GetDeltaX(angle), width, height);
-        }
-        else if (angle is > 90 and < 180)
-        {
-            PredictZone2Scalar(destination, destinationStride, above, left, upsampleAbove, upsampleLeft, GetDeltaX(angle), GetDeltaY(angle), width, height);
-        }
-        else if (angle is > 180 and < 270)
-        {
-            PredictZone3Scalar(destination, destinationStride, left, upsampleLeft, GetDeltaY(angle), width, height);
-        }
-        else
-        {
-            Av1PredictionMode mode = angle == 90 ? Av1PredictionMode.Vertical : Av1PredictionMode.Horizontal;
-            Av1IntraPredictorBase.GetPredictor(mode).PredictScalar(destination, destinationStride, above, left, width, height);
-        }
-    }
-
-    /// <summary>
-    /// Predicts a high-bit-depth directional block without hardware intrinsics.
-    /// </summary>
-    /// <param name="destination">The destination block origin.</param>
-    /// <param name="destinationStride">The destination row stride in samples.</param>
-    /// <param name="transformSize">The predicted block dimensions.</param>
-    /// <param name="above">The prepared top reference, including any required extension.</param>
-    /// <param name="left">The prepared left reference, including any required extension.</param>
-    /// <param name="upsampleAbove">Whether the top edge contains half-sample positions.</param>
-    /// <param name="upsampleLeft">Whether the left edge contains half-sample positions.</param>
-    /// <param name="angle">The adjusted prediction angle.</param>
-    public static void PredictScalar(Span<short> destination, int destinationStride, Av1TransformSize transformSize, ReadOnlySpan<short> above, ReadOnlySpan<short> left, bool upsampleAbove, bool upsampleLeft, int angle)
-    {
-        int width = transformSize.GetWidth();
-        int height = transformSize.GetHeight();
-
-        if (angle is > 0 and < 90)
-        {
-            PredictZone1Scalar(destination, destinationStride, above, upsampleAbove, GetDeltaX(angle), width, height);
-        }
-        else if (angle is > 90 and < 180)
-        {
-            PredictZone2Scalar(destination, destinationStride, above, left, upsampleAbove, upsampleLeft, GetDeltaX(angle), GetDeltaY(angle), width, height);
-        }
-        else if (angle is > 180 and < 270)
-        {
-            PredictZone3Scalar(destination, destinationStride, left, upsampleLeft, GetDeltaY(angle), width, height);
-        }
-        else
-        {
-            Av1PredictionMode mode = angle == 90 ? Av1PredictionMode.Vertical : Av1PredictionMode.Horizontal;
-            Av1IntraPredictorBase.GetPredictor(mode).PredictScalar(destination, destinationStride, above, left, width, height);
-        }
-    }
-
-    /// <summary>
-    /// Gets the horizontal Q8 projection derivative for an adjusted angle.
-    /// </summary>
-    /// <param name="angle">The adjusted prediction angle.</param>
-    /// <returns>The horizontal derivative, or one when the selected zone does not consume it.</returns>
-    public static int GetDeltaX(int angle)
-        => angle switch
-        {
-            > 0 and < 90 => DirectionalIntraDerivative[angle],
-            > 90 and < 180 => DirectionalIntraDerivative[180 - angle],
-            _ => 1,
-        };
-
-    /// <summary>
-    /// Gets the vertical Q8 projection derivative for an adjusted angle.
-    /// </summary>
-    /// <param name="angle">The adjusted prediction angle.</param>
-    /// <returns>The vertical derivative, or one when the selected zone does not consume it.</returns>
-    public static int GetDeltaY(int angle)
-        => angle switch
-        {
-            > 90 and < 180 => DirectionalIntraDerivative[angle - 90],
-            > 180 and < 270 => DirectionalIntraDerivative[270 - angle],
-            _ => 1,
-        };
-
-    /// <summary>
-    /// Predicts one 8-bit zone 1 block without hardware intrinsics.
-    /// </summary>
-    /// <param name="destination">The destination block origin.</param>
-    /// <param name="destinationStride">The destination row stride.</param>
-    /// <param name="above">The projected top reference.</param>
-    /// <param name="upsample">Whether the reference contains half-sample positions.</param>
-    /// <param name="derivative">The Q8 projection derivative.</param>
-    /// <param name="width">The block width.</param>
-    /// <param name="height">The block height.</param>
-    private static void PredictZone1Scalar(Span<byte> destination, int destinationStride, ReadOnlySpan<byte> above, bool upsample, int derivative, int width, int height)
-    {
-        int upsampleShift = upsample ? 1 : 0;
-        int maximumBasis = (width + height - 1) << upsampleShift;
-        int fractionBits = 6 - upsampleShift;
-        int basisIncrement = 1 << upsampleShift;
-        int projection = derivative;
-        ref byte aboveBase = ref Unsafe.AsRef(in above[0]);
-
-        for (int row = 0; row < height; row++, projection += derivative)
-        {
-            int basis = projection >> fractionBits;
-            int weight = ((projection << upsampleShift) & 0x3F) >> 1;
-            ref byte destinationRow = ref destination[row * destinationStride];
-
-            for (int column = 0; column < width; column++, basis += basisIncrement)
-            {
-                Unsafe.Add(ref destinationRow, column) = basis < maximumBasis
-                    ? (byte)(((Unsafe.Add(ref aboveBase, basis) * (32 - weight)) + (Unsafe.Add(ref aboveBase, basis + 1) * weight) + 16) >> 5)
-                    : Unsafe.Add(ref aboveBase, maximumBasis);
-            }
-        }
-    }
-
-    /// <summary>
-    /// Predicts one high-bit-depth zone 1 block without hardware intrinsics.
-    /// </summary>
-    /// <param name="destination">The destination block origin.</param>
-    /// <param name="destinationStride">The destination row stride.</param>
-    /// <param name="above">The projected top reference.</param>
-    /// <param name="upsample">Whether the reference contains half-sample positions.</param>
-    /// <param name="derivative">The Q8 projection derivative.</param>
-    /// <param name="width">The block width.</param>
-    /// <param name="height">The block height.</param>
-    private static void PredictZone1Scalar(Span<short> destination, int destinationStride, ReadOnlySpan<short> above, bool upsample, int derivative, int width, int height)
-    {
-        int upsampleShift = upsample ? 1 : 0;
-        int maximumBasis = (width + height - 1) << upsampleShift;
-        int fractionBits = 6 - upsampleShift;
-        int basisIncrement = 1 << upsampleShift;
-        int projection = derivative;
-        ref short aboveBase = ref Unsafe.AsRef(in above[0]);
-
-        for (int row = 0; row < height; row++, projection += derivative)
-        {
-            int basis = projection >> fractionBits;
-            int weight = ((projection << upsampleShift) & 0x3F) >> 1;
-            ref short destinationRow = ref destination[row * destinationStride];
-
-            for (int column = 0; column < width; column++, basis += basisIncrement)
-            {
-                Unsafe.Add(ref destinationRow, column) = basis < maximumBasis
-                    ? (short)(((Unsafe.Add(ref aboveBase, basis) * (32 - weight)) + (Unsafe.Add(ref aboveBase, basis + 1) * weight) + 16) >> 5)
-                    : Unsafe.Add(ref aboveBase, maximumBasis);
-            }
-        }
-    }
-
-    /// <summary>
-    /// Predicts one 8-bit zone 2 block without hardware intrinsics.
-    /// </summary>
-    /// <param name="destination">The destination block origin.</param>
-    /// <param name="destinationStride">The destination row stride.</param>
-    /// <param name="above">The projected top reference.</param>
-    /// <param name="left">The projected left reference.</param>
-    /// <param name="upsampleAbove">Whether the top reference contains half-sample positions.</param>
-    /// <param name="upsampleLeft">Whether the left reference contains half-sample positions.</param>
-    /// <param name="dx">The horizontal Q8 derivative.</param>
-    /// <param name="dy">The vertical Q8 derivative.</param>
-    /// <param name="width">The block width.</param>
-    /// <param name="height">The block height.</param>
-    private static void PredictZone2Scalar(Span<byte> destination, int destinationStride, ReadOnlySpan<byte> above, ReadOnlySpan<byte> left, bool upsampleAbove, bool upsampleLeft, int dx, int dy, int width, int height)
-    {
-        int aboveShift = upsampleAbove ? 1 : 0;
-        int leftShift = upsampleLeft ? 1 : 0;
-        int minimumTopBasis = -(1 << aboveShift);
-        int topFractionBits = 6 - aboveShift;
-        int leftFractionBits = 6 - leftShift;
-        int topBasisIncrement = 1 << aboveShift;
-        int topProjection = -dx;
-        ref byte aboveBase = ref Unsafe.AsRef(in above[0]);
-        ref byte leftBase = ref Unsafe.AsRef(in left[0]);
-
-        for (int row = 0; row < height; row++, topProjection -= dx)
-        {
-            int topBasis = topProjection >> topFractionBits;
-            int topWeight = ((topProjection << aboveShift) & 0x3F) >> 1;
-            int leftProjection = (row << 6) - dy;
-            ref byte destinationRow = ref destination[row * destinationStride];
-
-            for (int column = 0; column < width; column++, topBasis += topBasisIncrement, leftProjection -= dy)
-            {
-                int prediction;
-                if (topBasis >= minimumTopBasis)
-                {
-                    prediction = (Unsafe.Add(ref aboveBase, topBasis) * (32 - topWeight)) + (Unsafe.Add(ref aboveBase, topBasis + 1) * topWeight);
-                }
-                else
-                {
-                    int leftBasis = leftProjection >> leftFractionBits;
-                    int leftWeight = ((leftProjection << leftShift) & 0x3F) >> 1;
-                    prediction = (Unsafe.Add(ref leftBase, leftBasis) * (32 - leftWeight)) + (Unsafe.Add(ref leftBase, leftBasis + 1) * leftWeight);
-                }
-
-                Unsafe.Add(ref destinationRow, column) = (byte)((prediction + 16) >> 5);
-            }
-        }
-    }
-
-    /// <summary>
-    /// Predicts one high-bit-depth zone 2 block without hardware intrinsics.
-    /// </summary>
-    /// <param name="destination">The destination block origin.</param>
-    /// <param name="destinationStride">The destination row stride.</param>
-    /// <param name="above">The projected top reference.</param>
-    /// <param name="left">The projected left reference.</param>
-    /// <param name="upsampleAbove">Whether the top reference contains half-sample positions.</param>
-    /// <param name="upsampleLeft">Whether the left reference contains half-sample positions.</param>
-    /// <param name="dx">The horizontal Q8 derivative.</param>
-    /// <param name="dy">The vertical Q8 derivative.</param>
-    /// <param name="width">The block width.</param>
-    /// <param name="height">The block height.</param>
-    private static void PredictZone2Scalar(Span<short> destination, int destinationStride, ReadOnlySpan<short> above, ReadOnlySpan<short> left, bool upsampleAbove, bool upsampleLeft, int dx, int dy, int width, int height)
-    {
-        int aboveShift = upsampleAbove ? 1 : 0;
-        int leftShift = upsampleLeft ? 1 : 0;
-        int minimumTopBasis = -(1 << aboveShift);
-        int topFractionBits = 6 - aboveShift;
-        int leftFractionBits = 6 - leftShift;
-        int topBasisIncrement = 1 << aboveShift;
-        int topProjection = -dx;
-        ref short aboveBase = ref Unsafe.AsRef(in above[0]);
-        ref short leftBase = ref Unsafe.AsRef(in left[0]);
-
-        for (int row = 0; row < height; row++, topProjection -= dx)
-        {
-            int topBasis = topProjection >> topFractionBits;
-            int topWeight = ((topProjection << aboveShift) & 0x3F) >> 1;
-            int leftProjection = (row << 6) - dy;
-            ref short destinationRow = ref destination[row * destinationStride];
-
-            for (int column = 0; column < width; column++, topBasis += topBasisIncrement, leftProjection -= dy)
-            {
-                int prediction;
-                if (topBasis >= minimumTopBasis)
-                {
-                    prediction = (Unsafe.Add(ref aboveBase, topBasis) * (32 - topWeight)) + (Unsafe.Add(ref aboveBase, topBasis + 1) * topWeight);
-                }
-                else
-                {
-                    int leftBasis = leftProjection >> leftFractionBits;
-                    int leftWeight = ((leftProjection << leftShift) & 0x3F) >> 1;
-                    prediction = (Unsafe.Add(ref leftBase, leftBasis) * (32 - leftWeight)) + (Unsafe.Add(ref leftBase, leftBasis + 1) * leftWeight);
-                }
-
-                Unsafe.Add(ref destinationRow, column) = (short)((prediction + 16) >> 5);
-            }
-        }
-    }
-
-    /// <summary>
-    /// Predicts one 8-bit zone 3 block without hardware intrinsics.
-    /// </summary>
-    /// <param name="destination">The destination block origin.</param>
-    /// <param name="destinationStride">The destination row stride.</param>
-    /// <param name="left">The projected left reference.</param>
-    /// <param name="upsample">Whether the reference contains half-sample positions.</param>
-    /// <param name="derivative">The Q8 projection derivative.</param>
-    /// <param name="width">The block width.</param>
-    /// <param name="height">The block height.</param>
-    private static void PredictZone3Scalar(Span<byte> destination, int destinationStride, ReadOnlySpan<byte> left, bool upsample, int derivative, int width, int height)
-    {
-        int upsampleShift = upsample ? 1 : 0;
-        int maximumBasis = (width + height - 1) << upsampleShift;
-        int fractionBits = 6 - upsampleShift;
-        int basisIncrement = 1 << upsampleShift;
-        int projection = derivative;
-        ref byte leftBase = ref Unsafe.AsRef(in left[0]);
-
-        for (int column = 0; column < width; column++, projection += derivative)
-        {
-            int basis = projection >> fractionBits;
-            int weight = ((projection << upsampleShift) & 0x3F) >> 1;
-            for (int row = 0; row < height; row++, basis += basisIncrement)
-            {
-                destination[(row * destinationStride) + column] = basis < maximumBasis
-                    ? (byte)(((Unsafe.Add(ref leftBase, basis) * (32 - weight)) + (Unsafe.Add(ref leftBase, basis + 1) * weight) + 16) >> 5)
-                    : Unsafe.Add(ref leftBase, maximumBasis);
-            }
-        }
-    }
-
-    /// <summary>
-    /// Predicts one high-bit-depth zone 3 block without hardware intrinsics.
-    /// </summary>
-    /// <param name="destination">The destination block origin.</param>
-    /// <param name="destinationStride">The destination row stride.</param>
-    /// <param name="left">The projected left reference.</param>
-    /// <param name="upsample">Whether the reference contains half-sample positions.</param>
-    /// <param name="derivative">The Q8 projection derivative.</param>
-    /// <param name="width">The block width.</param>
-    /// <param name="height">The block height.</param>
-    private static void PredictZone3Scalar(Span<short> destination, int destinationStride, ReadOnlySpan<short> left, bool upsample, int derivative, int width, int height)
-    {
-        int upsampleShift = upsample ? 1 : 0;
-        int maximumBasis = (width + height - 1) << upsampleShift;
-        int fractionBits = 6 - upsampleShift;
-        int basisIncrement = 1 << upsampleShift;
-        int projection = derivative;
-        ref short leftBase = ref Unsafe.AsRef(in left[0]);
-
-        for (int column = 0; column < width; column++, projection += derivative)
-        {
-            int basis = projection >> fractionBits;
-            int weight = ((projection << upsampleShift) & 0x3F) >> 1;
-            for (int row = 0; row < height; row++, basis += basisIncrement)
-            {
-                destination[(row * destinationStride) + column] = basis < maximumBasis
-                    ? (short)(((Unsafe.Add(ref leftBase, basis) * (32 - weight)) + (Unsafe.Add(ref leftBase, basis + 1) * weight) + 16) >> 5)
-                    : Unsafe.Add(ref leftBase, maximumBasis);
-            }
-        }
-    }
-}

src/ImageSharp/Formats/Heif/Av1/Prediction/Av1FilterIntraPredictor.Operator.cs

@@ -452,7 +452,7 @@ internal abstract partial class Av1FilterIntraPredictorBase
         private static void StoreEightShorts(Vector256<int> prediction, ref short buffer, int firstRowOffset, int secondRowOffset)
         {
             Vector256<short> narrowed = Vector256.Narrow(prediction, Vector256<int>.Zero);
-            Unsafe.As<short, ulong>(ref Unsafe.Add(ref buffer, firstRowOffset)) = narrowed.AsUInt64().GetElement(0);
+            Unsafe.As<short, ulong>(ref Unsafe.Add(ref buffer, firstRowOffset)) = narrowed.AsUInt64().ToScalar();
             Unsafe.As<short, ulong>(ref Unsafe.Add(ref buffer, secondRowOffset)) = narrowed.AsUInt64().GetElement(1);
         }
 
@@ -466,7 +466,7 @@ internal abstract partial class Av1FilterIntraPredictorBase
         private static void StoreFourShorts(Vector128<int> prediction, ref short buffer, int offset)
         {
             Vector128<short> narrowed = Vector128.Narrow(prediction, Vector128<int>.Zero);
-            Unsafe.As<short, ulong>(ref Unsafe.Add(ref buffer, offset)) = narrowed.AsUInt64().GetElement(0);
+            Unsafe.As<short, ulong>(ref Unsafe.Add(ref buffer, offset)) = narrowed.AsUInt64().ToScalar();
         }
 
         /// <summary>

src/ImageSharp/Formats/Heif/Av1/Prediction/Av1IntraPredictor.Operator.cs

@@ -4,6 +4,7 @@
 using System.Runtime.CompilerServices;
 using System.Runtime.InteropServices;
 using System.Runtime.Intrinsics;
+using SixLabors.ImageSharp.Formats.Heif.Av1.Transform;
 
 namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction;
 

src/ImageSharp/Formats/Heif/Av1/Prediction/Av1IntraPredictorBase.cs

@@ -1,6 +1,8 @@
 // Copyright (c) Six Labors.
 // Licensed under the Six Labors Split License.
 
+using SixLabors.ImageSharp.Formats.Heif.Av1.Transform;
+
 namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction;
 
 /// <summary>

src/ImageSharp/Formats/Heif/Av1/Prediction/Av1PalettePredictor.Operator.cs

@@ -0,0 +1,356 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.CompilerServices;
+using System.Runtime.InteropServices;
+using System.Runtime.Intrinsics;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction;
+
+/// <content>
+/// Defines the closed scalar/SIMD operator contract and traversal for AV1 palette prediction.
+/// </content>
+internal static class Av1PalettePredictor
+{
+    /// <summary>
+    /// Multiplies a palette index by two to select the low byte of a high-bit-depth entry.
+    /// </summary>
+    private const ushort PaletteByteOffsetMultiplier = 0x0202;
+
+    /// <summary>
+    /// Adds one to each odd control byte so each shuffled high-bit-depth sample retains both bytes.
+    /// </summary>
+    private const ushort PaletteHighByteOffset = 0x0100;
+
+    /// <summary>
+    /// Defines scalar and SIMD palette-index lookup.
+    /// </summary>
+    private interface IPaletteOperator
+    {
+        /// <summary>
+        /// Predicts one 8-bit sample.
+        /// </summary>
+        /// <param name="palette">The first palette entry.</param>
+        /// <param name="index">The palette index.</param>
+        /// <returns>The selected sample.</returns>
+        public static abstract byte Predict(ref byte palette, byte index);
+
+        /// <summary>
+        /// Predicts sixteen 8-bit samples.
+        /// </summary>
+        /// <param name="palette">The palette entries repeated in each 128-bit lane.</param>
+        /// <param name="indices">The palette indices.</param>
+        /// <returns>The selected samples.</returns>
+        public static abstract Vector128<byte> Predict(Vector128<byte> palette, Vector128<byte> indices);
+
+        /// <summary>
+        /// Predicts thirty-two 8-bit samples.
+        /// </summary>
+        /// <param name="palette">The palette entries repeated in each 128-bit lane.</param>
+        /// <param name="indices">The palette indices.</param>
+        /// <returns>The selected samples.</returns>
+        public static abstract Vector256<byte> Predict(Vector256<byte> palette, Vector256<byte> indices);
+
+        /// <summary>
+        /// Predicts sixty-four 8-bit samples.
+        /// </summary>
+        /// <param name="palette">The palette entries repeated in each 128-bit lane.</param>
+        /// <param name="indices">The palette indices.</param>
+        /// <returns>The selected samples.</returns>
+        public static abstract Vector512<byte> Predict(Vector512<byte> palette, Vector512<byte> indices);
+
+        /// <summary>
+        /// Predicts one high-bit-depth sample.
+        /// </summary>
+        /// <param name="palette">The first palette entry.</param>
+        /// <param name="index">The palette index.</param>
+        /// <returns>The selected sample.</returns>
+        public static abstract short Predict(ref ushort palette, byte index);
+
+        /// <summary>
+        /// Predicts eight high-bit-depth samples.
+        /// </summary>
+        /// <param name="palette">The palette bytes repeated in each 128-bit lane.</param>
+        /// <param name="indices">The palette indices.</param>
+        /// <returns>The selected samples.</returns>
+        public static abstract Vector128<short> Predict(Vector128<byte> palette, Vector128<ushort> indices);
+
+        /// <summary>
+        /// Predicts sixteen high-bit-depth samples.
+        /// </summary>
+        /// <param name="palette">The palette bytes repeated in each 128-bit lane.</param>
+        /// <param name="indices">The palette indices.</param>
+        /// <returns>The selected samples.</returns>
+        public static abstract Vector256<short> Predict(Vector256<byte> palette, Vector256<ushort> indices);
+
+        /// <summary>
+        /// Predicts thirty-two high-bit-depth samples.
+        /// </summary>
+        /// <param name="palette">The palette bytes repeated in each 128-bit lane.</param>
+        /// <param name="indices">The palette indices.</param>
+        /// <returns>The selected samples.</returns>
+        public static abstract Vector512<short> Predict(Vector512<byte> palette, Vector512<ushort> indices);
+    }
+
+    /// <summary>
+    /// Reconstructs an 8-bit palette-predicted block.
+    /// </summary>
+    public static void Predict(
+        ReadOnlySpan<ushort> paletteColors,
+        ReadOnlySpan<byte> colorIndexMap,
+        int colorIndexMapStride,
+        Span<byte> destination,
+        int destinationStride,
+        int width,
+        int height)
+        => Predictor<PaletteOperator>.Predict(paletteColors, colorIndexMap, colorIndexMapStride, destination, destinationStride, width, height);
+
+    /// <summary>
+    /// Reconstructs a high-bit-depth palette-predicted block.
+    /// </summary>
+    public static void Predict(
+        ReadOnlySpan<ushort> paletteColors,
+        ReadOnlySpan<byte> colorIndexMap,
+        int colorIndexMapStride,
+        Span<short> destination,
+        int destinationStride,
+        int width,
+        int height)
+        => Predictor<PaletteOperator>.Predict(paletteColors, colorIndexMap, colorIndexMapStride, destination, destinationStride, width, height);
+
+    /// <summary>
+    /// Maps decoded palette indices to reconstructed samples.
+    /// </summary>
+    private readonly struct PaletteOperator : IPaletteOperator
+    {
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static byte Predict(ref byte palette, byte index) => Unsafe.Add(ref palette, index);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector128<byte> Predict(Vector128<byte> palette, Vector128<byte> indices)
+            => Vector128.ShuffleNative(palette, indices);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector256<byte> Predict(Vector256<byte> palette, Vector256<byte> indices)
+            => Vector256.ShuffleNative(palette, indices);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector512<byte> Predict(Vector512<byte> palette, Vector512<byte> indices)
+            => Vector512.ShuffleNative(palette, indices);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static short Predict(ref ushort palette, byte index) => (short)Unsafe.Add(ref palette, index);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector128<short> Predict(Vector128<byte> palette, Vector128<ushort> indices)
+        {
+            Vector128<ushort> controls = (indices * Vector128.Create(PaletteByteOffsetMultiplier)) + Vector128.Create(PaletteHighByteOffset);
+
+            return Vector128.ShuffleNative(palette, controls.AsByte()).AsInt16();
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector256<short> Predict(Vector256<byte> palette, Vector256<ushort> indices)
+        {
+            Vector256<ushort> controls = (indices * Vector256.Create(PaletteByteOffsetMultiplier)) + Vector256.Create(PaletteHighByteOffset);
+
+            return Vector256.ShuffleNative(palette, controls.AsByte()).AsInt16();
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector512<short> Predict(Vector512<byte> palette, Vector512<ushort> indices)
+        {
+            Vector512<ushort> controls = (indices * Vector512.Create(PaletteByteOffsetMultiplier)) + Vector512.Create(PaletteHighByteOffset);
+
+            return Vector512.ShuffleNative(palette, controls.AsByte()).AsInt16();
+        }
+    }
+
+    /// <summary>
+    /// Traverses palette blocks through one closed lookup operator.
+    /// </summary>
+    /// <typeparam name="TOperator">The palette lookup arithmetic.</typeparam>
+    private static class Predictor<TOperator>
+        where TOperator : struct, IPaletteOperator
+    {
+        /// <summary>
+        /// Reconstructs an 8-bit palette block.
+        /// </summary>
+        public static void Predict(
+            ReadOnlySpan<ushort> paletteColors,
+            ReadOnlySpan<byte> colorIndexMap,
+            int colorIndexMapStride,
+            Span<byte> destination,
+            int destinationStride,
+            int width,
+            int height)
+        {
+            ref byte mapBase = ref MemoryMarshal.GetReference(colorIndexMap);
+            ref byte destinationBase = ref MemoryMarshal.GetReference(destination);
+
+            // AV1 palettes contain at most eight colors. Repeating all eight entries in every 128-bit lane keeps native
+            // table lookup lane-local at every SIMD width and removes palette bounds work from the reconstruction loop.
+            ulong packedPalette = 0;
+            for (int index = 0; index < paletteColors.Length; index++)
+            {
+                packedPalette |= (ulong)(byte)paletteColors[index] << (index * 8);
+            }
+
+            ref byte paletteBase = ref Unsafe.As<ulong, byte>(ref packedPalette);
+            Vector128<byte> palette128 = Vector128.Create(packedPalette, packedPalette).AsByte();
+
+            for (int row = 0; row < height; row++)
+            {
+                ref byte mapRow = ref Unsafe.Add(ref mapBase, row * colorIndexMapStride);
+                ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
+                int column = 0;
+
+                if (Vector512.IsHardwareAccelerated)
+                {
+                    Vector256<byte> palette256 = Vector256.Create(palette128, palette128);
+                    Vector512<byte> palette512 = Vector512.Create(palette256, palette256);
+                    int oneVectorFromEnd = width - Vector512<byte>.Count;
+
+                    for (; column <= oneVectorFromEnd; column += Vector512<byte>.Count)
+                    {
+                        Vector512<byte> indices = Vector512.LoadUnsafe(ref mapRow, (nuint)column);
+                        TOperator.Predict(palette512, indices).StoreUnsafe(ref destinationRow, (nuint)column);
+                    }
+                }
+
+                if (Vector256.IsHardwareAccelerated)
+                {
+                    Vector256<byte> palette256 = Vector256.Create(palette128, palette128);
+                    int oneVectorFromEnd = width - Vector256<byte>.Count;
+
+                    for (; column <= oneVectorFromEnd; column += Vector256<byte>.Count)
+                    {
+                        Vector256<byte> indices = Vector256.LoadUnsafe(ref mapRow, (nuint)column);
+                        TOperator.Predict(palette256, indices).StoreUnsafe(ref destinationRow, (nuint)column);
+                    }
+                }
+
+                if (Vector128.IsHardwareAccelerated)
+                {
+                    int oneVectorFromEnd = width - Vector128<byte>.Count;
+                    for (; column <= oneVectorFromEnd; column += Vector128<byte>.Count)
+                    {
+                        Vector128<byte> indices = Vector128.LoadUnsafe(ref mapRow, (nuint)column);
+                        TOperator.Predict(palette128, indices).StoreUnsafe(ref destinationRow, (nuint)column);
+                    }
+
+                    int remaining = width - column;
+                    if (remaining >= 8)
+                    {
+                        ulong packedIndices = Unsafe.ReadUnaligned<ulong>(ref Unsafe.Add(ref mapRow, column));
+                        Vector128<byte> prediction = TOperator.Predict(palette128, Vector128.CreateScalarUnsafe(packedIndices).AsByte());
+                        Unsafe.WriteUnaligned(ref Unsafe.Add(ref destinationRow, column), prediction.AsUInt64().ToScalar());
+                        column += 8;
+                        remaining -= 8;
+                    }
+
+                    if (remaining >= 4)
+                    {
+                        uint packedIndices = Unsafe.ReadUnaligned<uint>(ref Unsafe.Add(ref mapRow, column));
+                        Vector128<byte> prediction = TOperator.Predict(palette128, Vector128.CreateScalarUnsafe(packedIndices).AsByte());
+                        Unsafe.WriteUnaligned(ref Unsafe.Add(ref destinationRow, column), prediction.AsUInt32().ToScalar());
+                        column += 4;
+                    }
+                }
+
+                for (; column < width; column++)
+                {
+                    Unsafe.Add(ref destinationRow, column) = TOperator.Predict(ref paletteBase, Unsafe.Add(ref mapRow, column));
+                }
+            }
+        }
+
+        /// <summary>
+        /// Reconstructs a high-bit-depth palette block.
+        /// </summary>
+        public static void Predict(
+            ReadOnlySpan<ushort> paletteColors,
+            ReadOnlySpan<byte> colorIndexMap,
+            int colorIndexMapStride,
+            Span<short> destination,
+            int destinationStride,
+            int width,
+            int height)
+        {
+            ref byte mapBase = ref MemoryMarshal.GetReference(colorIndexMap);
+            ref short destinationBase = ref MemoryMarshal.GetReference(destination);
+            InlineArray8<ushort> paletteStorage = default;
+            paletteColors.CopyTo(paletteStorage);
+
+            ref ushort paletteBase = ref paletteStorage[0];
+            Vector128<byte> palette128 = Vector128.LoadUnsafe(ref paletteBase).AsByte();
+
+            for (int row = 0; row < height; row++)
+            {
+                ref byte mapRow = ref Unsafe.Add(ref mapBase, row * colorIndexMapStride);
+                ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
+                int column = 0;
+
+                if (Vector512.IsHardwareAccelerated)
+                {
+                    Vector256<byte> palette256 = Vector256.Create(palette128, palette128);
+                    Vector512<byte> palette512 = Vector512.Create(palette256, palette256);
+                    int oneVectorFromEnd = width - Vector512<short>.Count;
+
+                    for (; column <= oneVectorFromEnd; column += Vector512<short>.Count)
+                    {
+                        (Vector256<ushort> lower, Vector256<ushort> upper) = Vector256.Widen(Vector256.LoadUnsafe(ref mapRow, (nuint)column));
+                        Vector512<ushort> indices = Vector512.Create(lower, upper);
+                        TOperator.Predict(palette512, indices).StoreUnsafe(ref destinationRow, (nuint)column);
+                    }
+                }
+
+                if (Vector256.IsHardwareAccelerated)
+                {
+                    Vector256<byte> palette256 = Vector256.Create(palette128, palette128);
+                    int oneVectorFromEnd = width - Vector256<short>.Count;
+
+                    for (; column <= oneVectorFromEnd; column += Vector256<short>.Count)
+                    {
+                        (Vector128<ushort> lower, Vector128<ushort> upper) = Vector128.Widen(Vector128.LoadUnsafe(ref mapRow, (nuint)column));
+                        Vector256<ushort> indices = Vector256.Create(lower, upper);
+                        TOperator.Predict(palette256, indices).StoreUnsafe(ref destinationRow, (nuint)column);
+                    }
+                }
+
+                if (Vector128.IsHardwareAccelerated)
+                {
+                    int oneVectorFromEnd = width - Vector128<short>.Count;
+                    for (; column <= oneVectorFromEnd; column += Vector128<short>.Count)
+                    {
+                        ulong packedIndices = Unsafe.ReadUnaligned<ulong>(ref Unsafe.Add(ref mapRow, column));
+                        Vector128<ushort> indices = Vector128.WidenLower(Vector128.CreateScalarUnsafe(packedIndices).AsByte());
+                        TOperator.Predict(palette128, indices).StoreUnsafe(ref destinationRow, (nuint)column);
+                    }
+
+                    if (width - column >= 4)
+                    {
+                        uint packedIndices = Unsafe.ReadUnaligned<uint>(ref Unsafe.Add(ref mapRow, column));
+                        Vector128<ushort> indices = Vector128.WidenLower(Vector128.CreateScalarUnsafe(packedIndices).AsByte());
+                        Vector128<short> prediction = TOperator.Predict(palette128, indices);
+                        Unsafe.WriteUnaligned(ref Unsafe.As<short, byte>(ref Unsafe.Add(ref destinationRow, column)), prediction.AsUInt64().ToScalar());
+                        column += 4;
+                    }
+                }
+
+                for (; column < width; column++)
+                {
+                    Unsafe.Add(ref destinationRow, column) = TOperator.Predict(ref paletteBase, Unsafe.Add(ref mapRow, column));
+                }
+            }
+        }
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/Av1PalettePredictor.cs

@@ -1,274 +0,0 @@
-// Copyright (c) Six Labors.
-// Licensed under the Six Labors Split License.
-
-using System.Runtime.CompilerServices;
-using System.Runtime.InteropServices;
-using System.Runtime.Intrinsics;
-
-namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction;
-
-/// <summary>
-/// Reconstructs AV1 palette-predicted sample blocks from decoded color-index maps.
-/// </summary>
-/// <remarks>
-/// AV1 palettes contain at most eight entries, so a complete 8-bit palette fits in each native shuffle lane and a
-/// complete high-bit-depth palette fits as sixteen bytes. Color indices become byte-shuffle controls; replicating the
-/// table per 128-bit lane keeps every lookup lane-local at 128, 256, and 512 bits. Exact-width tail loads and stores
-/// avoid requiring writable padding around small transform blocks.
-/// </remarks>
-internal static class Av1PalettePredictor
-{
-    /// <summary>
-    /// The repeated byte offsets that select both bytes of eight 16-bit palette entries.
-    /// </summary>
-    private const ushort PaletteByteOffsetMultiplier = 0x0202;
-
-    /// <summary>
-    /// The high-byte increment that selects the second byte of each 16-bit palette entry.
-    /// </summary>
-    private const ushort PaletteHighByteOffset = 0x0100;
-
-    /// <summary>
-    /// Reconstructs an 8-bit palette-predicted block.
-    /// </summary>
-    /// <param name="paletteColors">The decoded palette colors in prediction-index order.</param>
-    /// <param name="colorIndexMap">The color-index map beginning at the prediction block origin.</param>
-    /// <param name="colorIndexMapStride">The distance, in indices, between map rows.</param>
-    /// <param name="destination">The destination beginning at the prediction block origin.</param>
-    /// <param name="destinationStride">The distance, in samples, between destination rows.</param>
-    /// <param name="width">The prediction width in samples.</param>
-    /// <param name="height">The prediction height in samples.</param>
-    public static void Predict(
-        ReadOnlySpan<ushort> paletteColors,
-        ReadOnlySpan<byte> colorIndexMap,
-        int colorIndexMapStride,
-        Span<byte> destination,
-        int destinationStride,
-        int width,
-        int height)
-    {
-        ref byte mapBase = ref MemoryMarshal.GetReference(colorIndexMap);
-        ref byte destinationBase = ref MemoryMarshal.GetReference(destination);
-
-        // An AV1 palette contains at most eight colors. Packing it once into the low 64 bits and repeating it in
-        // every 128-bit lane turns reconstruction into the lane-local table lookup implemented by ShuffleNative.
-        ulong packedPalette = 0;
-        for (int index = 0; index < paletteColors.Length; index++)
-        {
-            packedPalette |= (ulong)(byte)paletteColors[index] << (index * 8);
-        }
-
-        Vector128<byte> palette128 = Vector128.Create(packedPalette, packedPalette).AsByte();
-
-        if (Vector512.IsHardwareAccelerated && width >= Vector512<byte>.Count)
-        {
-            Vector256<byte> palette256 = Vector256.Create(palette128, palette128);
-            Vector512<byte> palette512 = Vector512.Create(palette256, palette256);
-
-            for (int row = 0; row < height; row++)
-            {
-                ref byte mapRow = ref Unsafe.Add(ref mapBase, row * colorIndexMapStride);
-                ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
-
-                for (int column = 0; column < width; column += Vector512<byte>.Count)
-                {
-                    Vector512<byte> indices = Vector512.LoadUnsafe(ref mapRow, (nuint)column);
-                    Vector512.ShuffleNative(palette512, indices).StoreUnsafe(ref destinationRow, (nuint)column);
-                }
-            }
-
-            return;
-        }
-
-        if (Vector256.IsHardwareAccelerated && width >= Vector256<byte>.Count)
-        {
-            Vector256<byte> palette256 = Vector256.Create(palette128, palette128);
-            for (int row = 0; row < height; row++)
-            {
-                ref byte mapRow = ref Unsafe.Add(ref mapBase, row * colorIndexMapStride);
-                ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
-
-                for (int column = 0; column < width; column += Vector256<byte>.Count)
-                {
-                    Vector256<byte> indices = Vector256.LoadUnsafe(ref mapRow, (nuint)column);
-                    Vector256.ShuffleNative(palette256, indices).StoreUnsafe(ref destinationRow, (nuint)column);
-                }
-            }
-
-            return;
-        }
-
-        if (Vector128.IsHardwareAccelerated)
-        {
-            for (int row = 0; row < height; row++)
-            {
-                ref byte mapRow = ref Unsafe.Add(ref mapBase, row * colorIndexMapStride);
-                ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
-                int column = 0;
-
-                for (; column <= width - Vector128<byte>.Count; column += Vector128<byte>.Count)
-                {
-                    Vector128<byte> indices = Vector128.LoadUnsafe(ref mapRow, (nuint)column);
-                    Vector128.ShuffleNative(palette128, indices).StoreUnsafe(ref destinationRow, (nuint)column);
-                }
-
-                if (column < width)
-                {
-                    // Transform widths are powers of two. After complete vector chunks, only a four- or eight-byte
-                    // row tail remains, so an exact-width load and store keeps neighboring transforms untouched.
-                    int remaining = width - column;
-                    ulong packedIndices = remaining == 4
-                        ? Unsafe.ReadUnaligned<uint>(ref Unsafe.Add(ref mapRow, column))
-                        : Unsafe.ReadUnaligned<ulong>(ref Unsafe.Add(ref mapRow, column));
-
-                    Vector128<byte> result = Vector128.ShuffleNative(palette128, Vector128.CreateScalarUnsafe(packedIndices).AsByte());
-
-                    if (remaining == 4)
-                    {
-                        Unsafe.WriteUnaligned(ref Unsafe.Add(ref destinationRow, column), result.AsUInt32().ToScalar());
-                    }
-                    else
-                    {
-                        Unsafe.WriteUnaligned(ref Unsafe.Add(ref destinationRow, column), result.AsUInt64().ToScalar());
-                    }
-
-                    column = width;
-                }
-            }
-
-            return;
-        }
-
-        for (int row = 0; row < height; row++)
-        {
-            ref byte mapRow = ref Unsafe.Add(ref mapBase, row * colorIndexMapStride);
-            ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
-            for (int column = 0; column < width; column++)
-            {
-                Unsafe.Add(ref destinationRow, column) = (byte)paletteColors[Unsafe.Add(ref mapRow, column)];
-            }
-        }
-    }
-
-    /// <summary>
-    /// Reconstructs a high-bit-depth palette-predicted block.
-    /// </summary>
-    /// <param name="paletteColors">The decoded palette colors in prediction-index order.</param>
-    /// <param name="colorIndexMap">The color-index map beginning at the prediction block origin.</param>
-    /// <param name="colorIndexMapStride">The distance, in indices, between map rows.</param>
-    /// <param name="destination">The destination beginning at the prediction block origin.</param>
-    /// <param name="destinationStride">The distance, in samples, between destination rows.</param>
-    /// <param name="width">The prediction width in samples.</param>
-    /// <param name="height">The prediction height in samples.</param>
-    public static void Predict(
-        ReadOnlySpan<ushort> paletteColors,
-        ReadOnlySpan<byte> colorIndexMap,
-        int colorIndexMapStride,
-        Span<short> destination,
-        int destinationStride,
-        int width,
-        int height)
-    {
-        ref byte mapBase = ref MemoryMarshal.GetReference(colorIndexMap);
-        ref short destinationBase = ref MemoryMarshal.GetReference(destination);
-        InlineArray8<ushort> paletteStorage = default;
-
-        paletteColors.CopyTo(paletteStorage);
-
-        // Each index is expanded to the byte offsets 2n and 2n+1. Repeating the complete 16-byte palette in every
-        // 128-bit lane then permits the same native byte-table shuffle on x86, Arm, and WebAssembly.
-        Vector128<byte> palette128 = Vector128.LoadUnsafe(ref paletteStorage[0]).AsByte();
-
-        if (Vector512.IsHardwareAccelerated && width >= Vector512<ushort>.Count)
-        {
-            Vector256<byte> palette256 = Vector256.Create(palette128, palette128);
-            Vector512<byte> palette512 = Vector512.Create(palette256, palette256);
-            Vector512<ushort> multiplier = Vector512.Create(PaletteByteOffsetMultiplier);
-            Vector512<ushort> increment = Vector512.Create(PaletteHighByteOffset);
-
-            for (int row = 0; row < height; row++)
-            {
-                ref byte mapRow = ref Unsafe.Add(ref mapBase, row * colorIndexMapStride);
-                ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
-
-                for (int column = 0; column < width; column += Vector512<ushort>.Count)
-                {
-                    (Vector256<ushort> lower, Vector256<ushort> upper) = Vector256.Widen(Vector256.LoadUnsafe(ref mapRow, (nuint)column));
-
-                    Vector512<ushort> indices = Vector512.Create(lower, upper);
-                    Vector512<byte> controls = ((indices * multiplier) + increment).AsByte();
-                    Vector512.ShuffleNative(palette512, controls).AsInt16().StoreUnsafe(ref destinationRow, (nuint)column);
-                }
-            }
-
-            return;
-        }
-
-        if (Vector256.IsHardwareAccelerated && width >= Vector256<ushort>.Count)
-        {
-            Vector256<byte> palette256 = Vector256.Create(palette128, palette128);
-            Vector256<ushort> multiplier = Vector256.Create(PaletteByteOffsetMultiplier);
-            Vector256<ushort> increment = Vector256.Create(PaletteHighByteOffset);
-
-            for (int row = 0; row < height; row++)
-            {
-                ref byte mapRow = ref Unsafe.Add(ref mapBase, row * colorIndexMapStride);
-                ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
-
-                for (int column = 0; column < width; column += Vector256<ushort>.Count)
-                {
-                    (Vector128<ushort> lower, Vector128<ushort> upper) = Vector128.Widen(Vector128.LoadUnsafe(ref mapRow, (nuint)column));
-
-                    Vector256<ushort> indices = Vector256.Create(lower, upper);
-                    Vector256<byte> controls = ((indices * multiplier) + increment).AsByte();
-                    Vector256.ShuffleNative(palette256, controls).AsInt16().StoreUnsafe(ref destinationRow, (nuint)column);
-                }
-            }
-
-            return;
-        }
-
-        if (Vector128.IsHardwareAccelerated)
-        {
-            Vector128<ushort> multiplier = Vector128.Create(PaletteByteOffsetMultiplier);
-            Vector128<ushort> increment = Vector128.Create(PaletteHighByteOffset);
-
-            for (int row = 0; row < height; row++)
-            {
-                ref byte mapRow = ref Unsafe.Add(ref mapBase, row * colorIndexMapStride);
-                ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
-                int column = 0;
-
-                for (; column <= width - Vector128<ushort>.Count; column += Vector128<ushort>.Count)
-                {
-                    ulong packedIndices = Unsafe.ReadUnaligned<ulong>(ref Unsafe.Add(ref mapRow, column));
-                    Vector128<ushort> indices = Vector128.WidenLower(Vector128.CreateScalarUnsafe(packedIndices).AsByte());
-                    Vector128<byte> controls = ((indices * multiplier) + increment).AsByte();
-                    Vector128.ShuffleNative(palette128, controls).AsInt16().StoreUnsafe(ref destinationRow, (nuint)column);
-                }
-
-                if (column < width)
-                {
-                    uint packedIndices = Unsafe.ReadUnaligned<uint>(ref Unsafe.Add(ref mapRow, column));
-                    Vector128<ushort> indices = Vector128.WidenLower(Vector128.CreateScalarUnsafe(packedIndices).AsByte());
-                    Vector128<byte> controls = ((indices * multiplier) + increment).AsByte();
-                    Vector128<short> result = Vector128.ShuffleNative(palette128, controls).AsInt16();
-                    Unsafe.WriteUnaligned(ref Unsafe.As<short, byte>(ref Unsafe.Add(ref destinationRow, column)), result.AsUInt64().ToScalar());
-                    column += 4;
-                }
-            }
-
-            return;
-        }
-
-        for (int row = 0; row < height; row++)
-        {
-            ref byte mapRow = ref Unsafe.Add(ref mapBase, row * colorIndexMapStride);
-            ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
-            for (int column = 0; column < width; column++)
-            {
-                Unsafe.Add(ref destinationRow, column) = (short)paletteColors[Unsafe.Add(ref mapRow, column)];
-            }
-        }
-    }
-}

src/ImageSharp/Formats/Heif/Av1/Prediction/Av1PredictionDecoder.cs

@@ -1390,7 +1390,7 @@ internal class Av1PredictionDecoder
                 Vector128<byte> interpolated = InterpolateEightBytes(ref inputBase, i);
                 Vector128<byte> originals = Vector128.LoadUnsafe(ref inputBase, (nuint)(i + 2));
                 Vector128<byte> interleaved = Vector128_.UnpackLow(interpolated, originals);
-                Unsafe.As<byte, ulong>(ref Unsafe.Add(ref bufferBase, (2 * i) - 1)) = interleaved.AsUInt64().GetElement(0);
+                Unsafe.As<byte, ulong>(ref Unsafe.Add(ref bufferBase, (2 * i) - 1)) = interleaved.AsUInt64().ToScalar();
                 i += 4;
             }
         }
@@ -1610,7 +1610,7 @@ internal class Av1PredictionDecoder
                         Vector128<ushort> source1 = WidenLower(Vector128.LoadUnsafe(ref edge, (nuint)(processed + 2)));
                         Vector128<ushort> source2 = WidenLower(Vector128.LoadUnsafe(ref edge, (nuint)(processed + 3)));
                         Vector128<byte> result = Vector128.Narrow(FilterEdgeStrength1(source0, source1, source2), Vector128<ushort>.Zero);
-                        Unsafe.As<byte, ulong>(ref Unsafe.Add(ref buffer, processed + 1)) = result.AsUInt64().GetElement(0);
+                        Unsafe.As<byte, ulong>(ref Unsafe.Add(ref buffer, processed + 1)) = result.AsUInt64().ToScalar();
                     }
 
                     break;
@@ -1621,7 +1621,7 @@ internal class Av1PredictionDecoder
                         Vector128<ushort> source1 = WidenLower(Vector128.LoadUnsafe(ref edge, (nuint)(processed + 2)));
                         Vector128<ushort> source2 = WidenLower(Vector128.LoadUnsafe(ref edge, (nuint)(processed + 3)));
                         Vector128<byte> result = Vector128.Narrow(FilterEdgeStrength2(source0, source1, source2), Vector128<ushort>.Zero);
-                        Unsafe.As<byte, ulong>(ref Unsafe.Add(ref buffer, processed + 1)) = result.AsUInt64().GetElement(0);
+                        Unsafe.As<byte, ulong>(ref Unsafe.Add(ref buffer, processed + 1)) = result.AsUInt64().ToScalar();
                     }
 
                     break;
@@ -1634,7 +1634,7 @@ internal class Av1PredictionDecoder
                         Vector128<ushort> source3 = WidenLower(Vector128.LoadUnsafe(ref edge, (nuint)(processed + 3)));
                         Vector128<ushort> source4 = WidenLower(Vector128.LoadUnsafe(ref edge, (nuint)(processed + 4)));
                         Vector128<byte> result = Vector128.Narrow(FilterEdgeStrength3(source0, source1, source2, source3, source4), Vector128<ushort>.Zero);
-                        Unsafe.As<byte, ulong>(ref Unsafe.Add(ref buffer, processed + 1)) = result.AsUInt64().GetElement(0);
+                        Unsafe.As<byte, ulong>(ref Unsafe.Add(ref buffer, processed + 1)) = result.AsUInt64().ToScalar();
                     }
 
                     break;

src/ImageSharp/Formats/Heif/Av1/Prediction/ChromaFromLuma/Av1ChromaFromLumaPredictor.Operator.cs

@@ -0,0 +1,324 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.CompilerServices;
+using System.Runtime.InteropServices;
+using System.Runtime.Intrinsics;
+using System.Runtime.Intrinsics.Arm;
+using System.Runtime.Intrinsics.X86;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.ChromaFromLuma;
+
+/// <content>
+/// Defines the closed scalar/SIMD operator contract and traversal for AV1 chroma-from-luma prediction.
+/// </content>
+internal static partial class Av1ChromaFromLumaPredictor
+{
+    /// <summary>
+    /// The fixed row stride of the AV1 chroma-from-luma scratch buffer.
+    /// </summary>
+    private const int BufferLine = 32;
+
+    /// <summary>
+    /// Defines scalar and SIMD signed Q3 arithmetic for AV1 chroma-from-luma prediction.
+    /// </summary>
+    private interface IChromaFromLumaOperator
+    {
+        /// <summary>
+        /// Predicts one chroma sample.
+        /// </summary>
+        /// <param name="lumaQ3">The zero-mean Q3 luma sample.</param>
+        /// <param name="dc">The chroma DC prediction.</param>
+        /// <param name="alphaQ3">The signed Q3 chroma scaling factor.</param>
+        /// <param name="maximum">The maximum sample value.</param>
+        /// <returns>The predicted chroma sample.</returns>
+        public static abstract short Predict(short lumaQ3, short dc, int alphaQ3, short maximum);
+
+        /// <summary>
+        /// Predicts eight chroma samples.
+        /// </summary>
+        /// <param name="lumaQ3">The zero-mean Q3 luma samples.</param>
+        /// <param name="dc">The chroma DC prediction.</param>
+        /// <param name="alphaQ3">The signed Q3 chroma scaling factor.</param>
+        /// <param name="maximum">The maximum sample value.</param>
+        /// <returns>The predicted chroma samples.</returns>
+        public static abstract Vector128<short> Predict(Vector128<short> lumaQ3, short dc, int alphaQ3, short maximum);
+
+        /// <summary>
+        /// Predicts sixteen chroma samples.
+        /// </summary>
+        /// <param name="lumaQ3">The zero-mean Q3 luma samples.</param>
+        /// <param name="dc">The chroma DC prediction.</param>
+        /// <param name="alphaQ3">The signed Q3 chroma scaling factor.</param>
+        /// <param name="maximum">The maximum sample value.</param>
+        /// <returns>The predicted chroma samples.</returns>
+        public static abstract Vector256<short> Predict(Vector256<short> lumaQ3, short dc, int alphaQ3, short maximum);
+
+        /// <summary>
+        /// Predicts thirty-two chroma samples.
+        /// </summary>
+        /// <param name="lumaQ3">The zero-mean Q3 luma samples.</param>
+        /// <param name="dc">The chroma DC prediction.</param>
+        /// <param name="alphaQ3">The signed Q3 chroma scaling factor.</param>
+        /// <param name="maximum">The maximum sample value.</param>
+        /// <returns>The predicted chroma samples.</returns>
+        public static abstract Vector512<short> Predict(Vector512<short> lumaQ3, short dc, int alphaQ3, short maximum);
+    }
+
+    /// <summary>
+    /// Applies chroma-from-luma prediction to an 8-bit chroma block.
+    /// </summary>
+    /// <param name="lumaQ3">The zero-mean Q3 luma surface.</param>
+    /// <param name="destination">The DC-predicted chroma block that receives the luma adjustment.</param>
+    /// <param name="destinationStride">The distance, in samples, between destination rows.</param>
+    /// <param name="alphaQ3">The signed Q3 chroma scaling factor.</param>
+    /// <param name="width">The block width in samples.</param>
+    /// <param name="height">The block height in samples.</param>
+    public static void Predict(ReadOnlySpan<short> lumaQ3, Span<byte> destination, int destinationStride, int alphaQ3, int width, int height)
+        => Predictor<ChromaFromLumaOperator>.Predict(lumaQ3, destination, destinationStride, alphaQ3, width, height);
+
+    /// <summary>
+    /// Applies chroma-from-luma prediction to a high-bit-depth chroma block.
+    /// </summary>
+    /// <param name="lumaQ3">The zero-mean Q3 luma surface.</param>
+    /// <param name="destination">The DC-predicted chroma block that receives the luma adjustment.</param>
+    /// <param name="destinationStride">The distance, in samples, between destination rows.</param>
+    /// <param name="alphaQ3">The signed Q3 chroma scaling factor.</param>
+    /// <param name="bitDepth">The number of bits used to represent each sample.</param>
+    /// <param name="width">The block width in samples.</param>
+    /// <param name="height">The block height in samples.</param>
+    public static void Predict(ReadOnlySpan<short> lumaQ3, Span<short> destination, int destinationStride, int alphaQ3, int bitDepth, int width, int height)
+        => Predictor<ChromaFromLumaOperator>.Predict(lumaQ3, destination, destinationStride, alphaQ3, bitDepth, width, height);
+
+    /// <summary>
+    /// Applies the decoded chroma scaling factor to zero-mean luma samples.
+    /// </summary>
+    private readonly struct ChromaFromLumaOperator : IChromaFromLumaOperator
+    {
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static short Predict(short lumaQ3, short dc, int alphaQ3, short maximum)
+        {
+            int scaledLumaQ0 = Av1Math.RoundPowerOf2Signed(alphaQ3 * lumaQ3, 6);
+
+            return (short)Math.Clamp(dc + scaledLumaQ0, (short)0, maximum);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector128<short> Predict(Vector128<short> lumaQ3, short dc, int alphaQ3, short maximum)
+        {
+            Vector128<short> dcVector = Vector128.Create(dc);
+            Vector128<short> scaledLumaQ0;
+
+            if (Ssse3.IsSupported)
+            {
+                Vector128<short> alphaSign = Vector128.Create((short)alphaQ3);
+                Vector128<short> alphaQ12 = Vector128.Create((short)(Math.Abs(alphaQ3) << 9));
+                scaledLumaQ0 = Ssse3.MultiplyHighRoundScale(Ssse3.Abs(lumaQ3).AsInt16(), alphaQ12);
+                Vector128<short> signMask = (lumaQ3 ^ alphaSign) >> 15;
+                scaledLumaQ0 = (scaledLumaQ0 ^ signMask) - signMask;
+            }
+            else if (AdvSimd.IsSupported)
+            {
+                Vector128<short> alphaSign = Vector128.Create((short)alphaQ3);
+                Vector128<short> alphaQ12 = Vector128.Create((short)(Math.Abs(alphaQ3) << 9));
+                scaledLumaQ0 = AdvSimd.MultiplyRoundedDoublingSaturateHigh(Vector128.Abs(lumaQ3), alphaQ12);
+                Vector128<short> signMask = (lumaQ3 ^ alphaSign) >> 15;
+                scaledLumaQ0 = (scaledLumaQ0 ^ signMask) - signMask;
+            }
+            else
+            {
+                // WebAssembly and other Vector128 targets do not expose packed rounded-high multiply. Widening keeps
+                // the same signed rounding rule without introducing a second scalar traversal.
+                (Vector128<int> lower, Vector128<int> upper) = Vector128.Widen(lumaQ3);
+                Vector128<int> alpha = Vector128.Create(alphaQ3);
+                lower *= alpha;
+                upper *= alpha;
+                lower = (lower + Vector128.Create(32) + (lower >> 31)) >> 6;
+                upper = (upper + Vector128.Create(32) + (upper >> 31)) >> 6;
+                scaledLumaQ0 = Vector128.Narrow(lower, upper);
+            }
+
+            return Vector128.Clamp(scaledLumaQ0 + dcVector, Vector128<short>.Zero, Vector128.Create(maximum));
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector256<short> Predict(Vector256<short> lumaQ3, short dc, int alphaQ3, short maximum)
+        {
+            Vector256<short> dcVector = Vector256.Create(dc);
+            Vector256<short> scaledLumaQ0;
+
+            if (Avx2.IsSupported)
+            {
+                Vector256<short> alphaSign = Vector256.Create((short)alphaQ3);
+                Vector256<short> alphaQ12 = Vector256.Create((short)(Math.Abs(alphaQ3) << 9));
+                scaledLumaQ0 = Avx2.MultiplyHighRoundScale(Avx2.Abs(lumaQ3).AsInt16(), alphaQ12);
+                Vector256<short> signMask = (lumaQ3 ^ alphaSign) >> 15;
+                scaledLumaQ0 = (scaledLumaQ0 ^ signMask) - signMask;
+            }
+            else
+            {
+                (Vector256<int> lower, Vector256<int> upper) = Vector256.Widen(lumaQ3);
+                Vector256<int> alpha = Vector256.Create(alphaQ3);
+                lower *= alpha;
+                upper *= alpha;
+                lower = (lower + Vector256.Create(32) + (lower >> 31)) >> 6;
+                upper = (upper + Vector256.Create(32) + (upper >> 31)) >> 6;
+                scaledLumaQ0 = Vector256.Narrow(lower, upper);
+            }
+
+            return Vector256.Clamp(scaledLumaQ0 + dcVector, Vector256<short>.Zero, Vector256.Create(maximum));
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector512<short> Predict(Vector512<short> lumaQ3, short dc, int alphaQ3, short maximum)
+        {
+            (Vector512<int> lower, Vector512<int> upper) = Vector512.Widen(lumaQ3);
+            Vector512<int> alpha = Vector512.Create(alphaQ3);
+            lower *= alpha;
+            upper *= alpha;
+            lower = (lower + Vector512.Create(32) + (lower >> 31)) >> 6;
+            upper = (upper + Vector512.Create(32) + (upper >> 31)) >> 6;
+            Vector512<short> scaledLumaQ0 = Vector512.Narrow(lower, upper);
+
+            return Vector512.Clamp(scaledLumaQ0 + Vector512.Create(dc), Vector512<short>.Zero, Vector512.Create(maximum));
+        }
+    }
+
+    /// <summary>
+    /// Traverses a chroma block through one closed prediction operator.
+    /// </summary>
+    /// <typeparam name="TOperator">The signed Q3 prediction arithmetic.</typeparam>
+    private static class Predictor<TOperator>
+        where TOperator : struct, IChromaFromLumaOperator
+    {
+        /// <summary>
+        /// Applies chroma-from-luma prediction to an 8-bit block.
+        /// </summary>
+        public static void Predict(ReadOnlySpan<short> lumaQ3, Span<byte> destination, int destinationStride, int alphaQ3, int width, int height)
+        {
+            ref short lumaBase = ref MemoryMarshal.GetReference(lumaQ3);
+            ref byte destinationBase = ref MemoryMarshal.GetReference(destination);
+            short dc = destinationBase;
+
+            // CfL follows DC prediction, so one sample supplies the base value for the complete block. The fixed
+            // scratch stride also makes exact-width Vector128 loads safe for the four-sample AV1 tail.
+            for (int row = 0; row < height; row++)
+            {
+                int lumaRowOffset = row * BufferLine;
+                int destinationRowOffset = row * destinationStride;
+                ref short lumaRow = ref Unsafe.Add(ref lumaBase, lumaRowOffset);
+                ref byte destinationRow = ref Unsafe.Add(ref destinationBase, destinationRowOffset);
+                int column = 0;
+
+                if (Vector512.IsHardwareAccelerated)
+                {
+                    int oneVectorFromEnd = width - Vector512<short>.Count;
+                    for (; column <= oneVectorFromEnd; column += Vector512<short>.Count)
+                    {
+                        Vector512<short> prediction = TOperator.Predict(Vector512.LoadUnsafe(ref lumaRow, (nuint)column), dc, alphaQ3, byte.MaxValue);
+                        Vector256<byte> packed = Vector512.Narrow(prediction.AsUInt16(), Vector512<ushort>.Zero).GetLower();
+                        packed.StoreUnsafe(ref destinationRow, (nuint)column);
+                    }
+                }
+
+                if (Vector256.IsHardwareAccelerated)
+                {
+                    int oneVectorFromEnd = width - Vector256<short>.Count;
+                    for (; column <= oneVectorFromEnd; column += Vector256<short>.Count)
+                    {
+                        Vector256<short> prediction = TOperator.Predict(Vector256.LoadUnsafe(ref lumaRow, (nuint)column), dc, alphaQ3, byte.MaxValue);
+                        Vector128<byte> packed = Vector256.Narrow(prediction.AsUInt16(), Vector256<ushort>.Zero).GetLower();
+                        packed.StoreUnsafe(ref destinationRow, (nuint)column);
+                    }
+                }
+
+                if (Vector128.IsHardwareAccelerated)
+                {
+                    int oneVectorFromEnd = width - Vector128<short>.Count;
+                    for (; column <= oneVectorFromEnd; column += Vector128<short>.Count)
+                    {
+                        Vector128<short> prediction = TOperator.Predict(Vector128.LoadUnsafe(ref lumaRow, (nuint)column), dc, alphaQ3, byte.MaxValue);
+                        Vector64<byte> packed = Vector128.Narrow(prediction.AsUInt16(), Vector128<ushort>.Zero).GetLower();
+                        packed.StoreUnsafe(ref destinationRow, (nuint)column);
+                    }
+
+                    if (width - column >= 4)
+                    {
+                        Vector128<short> prediction = TOperator.Predict(Vector128.LoadUnsafe(ref lumaRow, (nuint)column), dc, alphaQ3, byte.MaxValue);
+                        Vector64<byte> packed = Vector128.Narrow(prediction.AsUInt16(), Vector128<ushort>.Zero).GetLower();
+                        Unsafe.WriteUnaligned(ref Unsafe.Add(ref destinationRow, column), packed.AsUInt32().ToScalar());
+                        column += 4;
+                    }
+                }
+
+                for (; column < width; column++)
+                {
+                    Unsafe.Add(ref destinationRow, column) = (byte)TOperator.Predict(Unsafe.Add(ref lumaRow, column), dc, alphaQ3, byte.MaxValue);
+                }
+            }
+        }
+
+        /// <summary>
+        /// Applies chroma-from-luma prediction to a high-bit-depth block.
+        /// </summary>
+        public static void Predict(ReadOnlySpan<short> lumaQ3, Span<short> destination, int destinationStride, int alphaQ3, int bitDepth, int width, int height)
+        {
+            ref short lumaBase = ref MemoryMarshal.GetReference(lumaQ3);
+            ref short destinationBase = ref MemoryMarshal.GetReference(destination);
+            short dc = destinationBase;
+            short maximum = (short)((1 << bitDepth) - 1);
+
+            for (int row = 0; row < height; row++)
+            {
+                int lumaRowOffset = row * BufferLine;
+                int destinationRowOffset = row * destinationStride;
+                ref short lumaRow = ref Unsafe.Add(ref lumaBase, lumaRowOffset);
+                ref short destinationRow = ref Unsafe.Add(ref destinationBase, destinationRowOffset);
+                int column = 0;
+
+                if (Vector512.IsHardwareAccelerated)
+                {
+                    int oneVectorFromEnd = width - Vector512<short>.Count;
+                    for (; column <= oneVectorFromEnd; column += Vector512<short>.Count)
+                    {
+                        TOperator.Predict(Vector512.LoadUnsafe(ref lumaRow, (nuint)column), dc, alphaQ3, maximum).StoreUnsafe(ref destinationRow, (nuint)column);
+                    }
+                }
+
+                if (Vector256.IsHardwareAccelerated)
+                {
+                    int oneVectorFromEnd = width - Vector256<short>.Count;
+                    for (; column <= oneVectorFromEnd; column += Vector256<short>.Count)
+                    {
+                        TOperator.Predict(Vector256.LoadUnsafe(ref lumaRow, (nuint)column), dc, alphaQ3, maximum).StoreUnsafe(ref destinationRow, (nuint)column);
+                    }
+                }
+
+                if (Vector128.IsHardwareAccelerated)
+                {
+                    int oneVectorFromEnd = width - Vector128<short>.Count;
+                    for (; column <= oneVectorFromEnd; column += Vector128<short>.Count)
+                    {
+                        TOperator.Predict(Vector128.LoadUnsafe(ref lumaRow, (nuint)column), dc, alphaQ3, maximum).StoreUnsafe(ref destinationRow, (nuint)column);
+                    }
+
+                    if (width - column >= 4)
+                    {
+                        Vector128<short> prediction = TOperator.Predict(Vector128.LoadUnsafe(ref lumaRow, (nuint)column), dc, alphaQ3, maximum);
+                        Unsafe.WriteUnaligned(ref Unsafe.As<short, byte>(ref Unsafe.Add(ref destinationRow, column)), prediction.AsUInt64().ToScalar());
+                        column += 4;
+                    }
+                }
+
+                for (; column < width; column++)
+                {
+                    Unsafe.Add(ref destinationRow, column) = TOperator.Predict(Unsafe.Add(ref lumaRow, column), dc, alphaQ3, maximum);
+                }
+            }
+        }
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/ChromaFromLuma/Av1ChromaFromLumaPredictor.cs

@@ -1,237 +0,0 @@
-// Copyright (c) Six Labors.
-// Licensed under the Six Labors Split License.
-
-using System.Runtime.CompilerServices;
-using System.Runtime.InteropServices;
-using System.Runtime.Intrinsics;
-using System.Runtime.Intrinsics.Arm;
-using System.Runtime.Intrinsics.X86;
-
-namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.ChromaFromLuma;
-
-/// <summary>
-/// Applies an AV1 chroma-from-luma residual to a DC-predicted chroma block.
-/// </summary>
-/// <remarks>
-/// Each signed 16-bit lane contains one zero-mean Q3 luma value. Packed rounded-high multiplication converts the product
-/// with the Q3 alpha parameter directly to a signed integer adjustment; the alpha/luma sign mask restores the product
-/// sign after the magnitude operation. The common DC prediction is broadcast, then results are clipped and narrowed to
-/// the destination sample representation.
-/// </remarks>
-internal static class Av1ChromaFromLumaPredictor
-{
-    /// <summary>
-    /// The fixed row stride of the AV1 chroma-from-luma scratch buffer.
-    /// </summary>
-    private const int BufferLine = 32;
-
-    /// <summary>
-    /// Applies chroma-from-luma prediction to an 8-bit chroma block.
-    /// </summary>
-    /// <param name="lumaQ3">The zero-mean Q3 luma surface.</param>
-    /// <param name="destination">The DC-predicted chroma block that receives the luma adjustment.</param>
-    /// <param name="destinationStride">The distance, in samples, between destination rows.</param>
-    /// <param name="alphaQ3">The signed Q3 chroma scaling factor.</param>
-    /// <param name="width">The block width in samples.</param>
-    /// <param name="height">The block height in samples.</param>
-    public static void Predict(ReadOnlySpan<short> lumaQ3, Span<byte> destination, int destinationStride, int alphaQ3, int width, int height)
-    {
-        ref short lumaBase = ref MemoryMarshal.GetReference(lumaQ3);
-        ref byte destinationBase = ref MemoryMarshal.GetReference(destination);
-        short dc = destinationBase;
-
-        // CfL always follows DC prediction, so the first sample is the common base value for every lane. This
-        // mirrors libaom and avoids loading a block that is known to contain a single repeated prediction value.
-        if (Avx2.IsSupported && width >= Vector256<short>.Count)
-        {
-            Vector256<short> alphaSign = Vector256.Create((short)alphaQ3);
-            Vector256<short> alphaQ12 = Vector256.Create((short)(Math.Abs(alphaQ3) << 9));
-            Vector256<short> dcVector = Vector256.Create(dc);
-            Vector256<short> maximum = Vector256.Create((short)byte.MaxValue);
-
-            for (int row = 0; row < height; row++)
-            {
-                int lumaRowOffset = row * BufferLine;
-                int destinationRowOffset = row * destinationStride;
-
-                for (int column = 0; column < width; column += Vector256<short>.Count)
-                {
-                    Vector256<short> prediction = Predict(Vector256.LoadUnsafe(ref lumaBase, (nuint)(lumaRowOffset + column)), alphaSign, alphaQ12, dcVector);
-                    prediction = Vector256.Clamp(prediction, Vector256<short>.Zero, maximum);
-                    Vector256.Narrow(prediction.AsUInt16(), Vector256<ushort>.Zero).GetLower().StoreUnsafe(ref destinationBase, (nuint)(destinationRowOffset + column));
-                }
-            }
-
-            return;
-        }
-
-        if (Vector128.IsHardwareAccelerated)
-        {
-            Vector128<short> alphaSign = Vector128.Create((short)alphaQ3);
-            Vector128<short> alphaQ12 = Vector128.Create((short)(Math.Abs(alphaQ3) << 9));
-            Vector128<short> dcVector = Vector128.Create(dc);
-            Vector128<short> maximum = Vector128.Create((short)byte.MaxValue);
-
-            for (int row = 0; row < height; row++)
-            {
-                int lumaRowOffset = row * BufferLine;
-                int destinationRowOffset = row * destinationStride;
-                int column = 0;
-
-                for (; column <= width - Vector128<short>.Count; column += Vector128<short>.Count)
-                {
-                    Vector128<short> prediction = Predict(Vector128.LoadUnsafe(ref lumaBase, (nuint)(lumaRowOffset + column)), alphaSign, alphaQ12, dcVector, alphaQ3);
-                    prediction = Vector128.Clamp(prediction, Vector128<short>.Zero, maximum);
-                    Vector128.Narrow(prediction.AsUInt16(), Vector128<ushort>.Zero).GetLower().StoreUnsafe(ref destinationBase, (nuint)(destinationRowOffset + column));
-                }
-
-                // Four-wide CfL blocks still have a complete padded scratch row, so reading eight residuals is
-                // valid. Only the four active predictions are stored to the image buffer.
-                if (column < width)
-                {
-                    Vector128<short> prediction = Predict(Vector128.LoadUnsafe(ref lumaBase, (nuint)(lumaRowOffset + column)), alphaSign, alphaQ12, dcVector, alphaQ3);
-                    prediction = Vector128.Clamp(prediction, Vector128<short>.Zero, maximum);
-                    Vector64<byte> packed = Vector128.Narrow(prediction.AsUInt16(), Vector128<ushort>.Zero).GetLower();
-                    Unsafe.WriteUnaligned(ref Unsafe.Add(ref destinationBase, destinationRowOffset + column), packed.AsUInt32().ToScalar());
-                }
-            }
-
-            return;
-        }
-
-        for (int row = 0; row < height; row++)
-        {
-            int lumaRowOffset = row * BufferLine;
-            int destinationRowOffset = row * destinationStride;
-            for (int column = 0; column < width; column++)
-            {
-                int scaledLumaQ0 = Av1Math.RoundPowerOf2Signed(alphaQ3 * Unsafe.Add(ref lumaBase, lumaRowOffset + column), 6);
-                Unsafe.Add(ref destinationBase, destinationRowOffset + column) = (byte)Math.Clamp(dc + scaledLumaQ0, byte.MinValue, byte.MaxValue);
-            }
-        }
-    }
-
-    /// <summary>
-    /// Applies chroma-from-luma prediction to a high-bit-depth chroma block.
-    /// </summary>
-    /// <param name="lumaQ3">The zero-mean Q3 luma surface.</param>
-    /// <param name="destination">The DC-predicted chroma block that receives the luma adjustment.</param>
-    /// <param name="destinationStride">The distance, in samples, between destination rows.</param>
-    /// <param name="alphaQ3">The signed Q3 chroma scaling factor.</param>
-    /// <param name="bitDepth">The number of bits used to represent each sample.</param>
-    /// <param name="width">The block width in samples.</param>
-    /// <param name="height">The block height in samples.</param>
-    public static void Predict(ReadOnlySpan<short> lumaQ3, Span<short> destination, int destinationStride, int alphaQ3, int bitDepth, int width, int height)
-    {
-        ref short lumaBase = ref MemoryMarshal.GetReference(lumaQ3);
-        ref short destinationBase = ref MemoryMarshal.GetReference(destination);
-        short dc = destinationBase;
-        short maximum = (short)((1 << bitDepth) - 1);
-
-        if (Avx2.IsSupported && width >= Vector256<short>.Count)
-        {
-            Vector256<short> alphaSign = Vector256.Create((short)alphaQ3);
-            Vector256<short> alphaQ12 = Vector256.Create((short)(Math.Abs(alphaQ3) << 9));
-            Vector256<short> dcVector = Vector256.Create(dc);
-            Vector256<short> maximumVector = Vector256.Create(maximum);
-
-            for (int row = 0; row < height; row++)
-            {
-                int lumaRowOffset = row * BufferLine;
-                int destinationRowOffset = row * destinationStride;
-
-                for (int column = 0; column < width; column += Vector256<short>.Count)
-                {
-                    Vector256<short> prediction = Predict(Vector256.LoadUnsafe(ref lumaBase, (nuint)(lumaRowOffset + column)), alphaSign, alphaQ12, dcVector);
-                    Vector256.Clamp(prediction, Vector256<short>.Zero, maximumVector).StoreUnsafe(ref destinationBase, (nuint)(destinationRowOffset + column));
-                }
-            }
-
-            return;
-        }
-
-        if (Vector128.IsHardwareAccelerated)
-        {
-            Vector128<short> alphaSign = Vector128.Create((short)alphaQ3);
-            Vector128<short> alphaQ12 = Vector128.Create((short)(Math.Abs(alphaQ3) << 9));
-            Vector128<short> dcVector = Vector128.Create(dc);
-            Vector128<short> maximumVector = Vector128.Create(maximum);
-
-            for (int row = 0; row < height; row++)
-            {
-                int lumaRowOffset = row * BufferLine;
-                int destinationRowOffset = row * destinationStride;
-                int column = 0;
-
-                for (; column <= width - Vector128<short>.Count; column += Vector128<short>.Count)
-                {
-                    Vector128<short> prediction = Predict(Vector128.LoadUnsafe(ref lumaBase, (nuint)(lumaRowOffset + column)), alphaSign, alphaQ12, dcVector, alphaQ3);
-                    Vector128.Clamp(prediction, Vector128<short>.Zero, maximumVector).StoreUnsafe(ref destinationBase, (nuint)(destinationRowOffset + column));
-                }
-
-                if (column < width)
-                {
-                    Vector128<short> prediction = Predict(Vector128.LoadUnsafe(ref lumaBase, (nuint)(lumaRowOffset + column)), alphaSign, alphaQ12, dcVector, alphaQ3);
-                    prediction = Vector128.Clamp(prediction, Vector128<short>.Zero, maximumVector);
-                    Unsafe.WriteUnaligned(ref Unsafe.As<short, byte>(ref Unsafe.Add(ref destinationBase, destinationRowOffset + column)), prediction.AsUInt64().ToScalar());
-                }
-            }
-
-            return;
-        }
-
-        for (int row = 0; row < height; row++)
-        {
-            int lumaRowOffset = row * BufferLine;
-            int destinationRowOffset = row * destinationStride;
-            for (int column = 0; column < width; column++)
-            {
-                int scaledLumaQ0 = Av1Math.RoundPowerOf2Signed(alphaQ3 * Unsafe.Add(ref lumaBase, lumaRowOffset + column), 6);
-                Unsafe.Add(ref destinationBase, destinationRowOffset + column) = (short)Math.Clamp(dc + scaledLumaQ0, 0, maximum);
-            }
-        }
-    }
-
-    /// <summary>
-    /// Calculates sixteen chroma predictions using the packed Q3 arithmetic defined by AV1.
-    /// </summary>
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector256<short> Predict(Vector256<short> lumaQ3, Vector256<short> alphaSign, Vector256<short> alphaQ12, Vector256<short> dc)
-    {
-        Vector256<short> scaledLumaQ0 = Avx2.MultiplyHighRoundScale(Avx2.Abs(lumaQ3).AsInt16(), alphaQ12);
-        Vector256<short> signMask = (lumaQ3 ^ alphaSign) >> 15;
-        return ((scaledLumaQ0 ^ signMask) - signMask) + dc;
-    }
-
-    /// <summary>
-    /// Calculates eight chroma predictions using the packed Q3 arithmetic defined by AV1.
-    /// </summary>
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector128<short> Predict(Vector128<short> lumaQ3, Vector128<short> alphaSign, Vector128<short> alphaQ12, Vector128<short> dc, int alphaQ3)
-    {
-        Vector128<short> scaledLumaQ0;
-        if (Ssse3.IsSupported)
-        {
-            scaledLumaQ0 = Ssse3.MultiplyHighRoundScale(Ssse3.Abs(lumaQ3).AsInt16(), alphaQ12);
-        }
-        else if (AdvSimd.IsSupported)
-        {
-            scaledLumaQ0 = AdvSimd.MultiplyRoundedDoublingSaturateHigh(Vector128.Abs(lumaQ3), alphaQ12);
-        }
-        else
-        {
-            // WebAssembly and other Vector128 targets do not expose packed rounded-high multiply. Widening retains
-            // SIMD traversal while reproducing the same signed nearest-integer result in ordinary integer lanes.
-            (Vector128<int> lower, Vector128<int> upper) = Vector128.Widen(lumaQ3);
-            Vector128<int> alpha = Vector128.Create(alphaQ3);
-            lower *= alpha;
-            upper *= alpha;
-            lower = (lower + Vector128.Create(32) + (lower >> 31)) >> 6;
-            upper = (upper + Vector128.Create(32) + (upper >> 31)) >> 6;
-            return Vector128.Narrow(lower, upper) + dc;
-        }
-
-        Vector128<short> signMask = (lumaQ3 ^ alphaSign) >> 15;
-        return ((scaledLumaQ0 ^ signMask) - signMask) + dc;
-    }
-}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundAveragePredictor.Operator.cs

@@ -0,0 +1,132 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.CompilerServices;
+using System.Runtime.Intrinsics;
+
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
+
+/// <content>
+/// Defines equal-weight compound prediction arithmetic.
+/// </content>
+internal static partial class Av1CompoundAveragePredictor
+{
+    /// <summary>
+    /// Defines equal-weight compound averaging for scalar and SIMD lane groups.
+    /// </summary>
+    private interface IAv1CompoundAverageOperator
+    {
+        /// <summary>
+        /// Averages two 8-bit samples.
+        /// </summary>
+        /// <param name="first">The first sample.</param>
+        /// <param name="second">The second sample.</param>
+        /// <returns>The rounded average.</returns>
+        public static abstract byte Blend(byte first, byte second);
+
+        /// <summary>
+        /// Averages two high-bit-depth samples.
+        /// </summary>
+        /// <param name="first">The first sample.</param>
+        /// <param name="second">The second sample.</param>
+        /// <returns>The rounded average.</returns>
+        public static abstract ushort Blend(ushort first, ushort second);
+
+        /// <summary>
+        /// Averages two 128-bit vectors of 8-bit samples.
+        /// </summary>
+        /// <param name="first">The first samples.</param>
+        /// <param name="second">The second samples.</param>
+        /// <returns>The rounded averages.</returns>
+        public static abstract Vector128<byte> Blend(Vector128<byte> first, Vector128<byte> second);
+
+        /// <summary>
+        /// Averages two 256-bit vectors of 8-bit samples.
+        /// </summary>
+        /// <param name="first">The first samples.</param>
+        /// <param name="second">The second samples.</param>
+        /// <returns>The rounded averages.</returns>
+        public static abstract Vector256<byte> Blend(Vector256<byte> first, Vector256<byte> second);
+
+        /// <summary>
+        /// Averages two 512-bit vectors of 8-bit samples.
+        /// </summary>
+        /// <param name="first">The first samples.</param>
+        /// <param name="second">The second samples.</param>
+        /// <returns>The rounded averages.</returns>
+        public static abstract Vector512<byte> Blend(Vector512<byte> first, Vector512<byte> second);
+
+        /// <summary>
+        /// Averages two 128-bit vectors of high-bit-depth samples.
+        /// </summary>
+        /// <param name="first">The first samples.</param>
+        /// <param name="second">The second samples.</param>
+        /// <returns>The rounded averages.</returns>
+        public static abstract Vector128<ushort> Blend(Vector128<ushort> first, Vector128<ushort> second);
+
+        /// <summary>
+        /// Averages two 256-bit vectors of high-bit-depth samples.
+        /// </summary>
+        /// <param name="first">The first samples.</param>
+        /// <param name="second">The second samples.</param>
+        /// <returns>The rounded averages.</returns>
+        public static abstract Vector256<ushort> Blend(Vector256<ushort> first, Vector256<ushort> second);
+
+        /// <summary>
+        /// Averages two 512-bit vectors of high-bit-depth samples.
+        /// </summary>
+        /// <param name="first">The first samples.</param>
+        /// <param name="second">The second samples.</param>
+        /// <returns>The rounded averages.</returns>
+        public static abstract Vector512<ushort> Blend(Vector512<ushort> first, Vector512<ushort> second);
+    }
+
+    /// <summary>
+    /// Implements equal-weight rounded averaging for scalar and SIMD lane groups.
+    /// </summary>
+    private readonly struct CompoundAverageOperator : IAv1CompoundAverageOperator
+    {
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static byte Blend(byte first, byte second) => (byte)((first + second + 1) >> 1);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static ushort Blend(ushort first, ushort second) => (ushort)((first + second + 1) >> 1);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector128<byte> Blend(Vector128<byte> first, Vector128<byte> second)
+            => (first | second) - ((first ^ second) >> 1);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector256<byte> Blend(Vector256<byte> first, Vector256<byte> second)
+            => (first | second) - ((first ^ second) >> 1);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector512<byte> Blend(Vector512<byte> first, Vector512<byte> second)
+        {
+            // This identity is exactly (a + b + 1) >> 1 but cannot overflow unsigned lanes at any SIMD width.
+            return (first | second) - ((first ^ second) >> 1);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector128<ushort> Blend(Vector128<ushort> first, Vector128<ushort> second)
+            => (first | second) - ((first ^ second) >> 1);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector256<ushort> Blend(Vector256<ushort> first, Vector256<ushort> second)
+            => (first | second) - ((first ^ second) >> 1);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector512<ushort> Blend(Vector512<ushort> first, Vector512<ushort> second)
+            => (first | second) - ((first ^ second) >> 1);
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundAveragePredictor.cs

@@ -0,0 +1,281 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.InteropServices;
+using System.Runtime.Intrinsics;
+
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
+
+/// <content>
+/// Combines two AV1 inter predictors with equal-weight rounded averaging.
+/// </content>
+internal static partial class Av1CompoundAveragePredictor
+{
+    /// <summary>
+    /// Averages an 8-bit predictor into an existing prediction block.
+    /// </summary>
+    /// <param name="destination">The first predictor and combined output.</param>
+    /// <param name="destinationStride">The distance between destination rows in samples.</param>
+    /// <param name="second">The second predictor.</param>
+    /// <param name="secondStride">The distance between second-predictor rows in samples.</param>
+    /// <param name="width">The active block width.</param>
+    /// <param name="height">The active block height.</param>
+    public static void Average(
+        Span<byte> destination,
+        int destinationStride,
+        ReadOnlySpan<byte> second,
+        int secondStride,
+        int width,
+        int height)
+        => Average<CompoundAverageOperator>(destination, destinationStride, second, secondStride, width, height);
+
+    /// <summary>
+    /// Averages an 8-bit predictor through one closed compound operator.
+    /// </summary>
+    /// <typeparam name="TOperator">The equal-weight averaging arithmetic.</typeparam>
+    /// <param name="destination">The first predictor and combined output.</param>
+    /// <param name="destinationStride">The distance between destination rows.</param>
+    /// <param name="second">The second predictor.</param>
+    /// <param name="secondStride">The distance between second-predictor rows.</param>
+    /// <param name="width">The active block width.</param>
+    /// <param name="height">The active block height.</param>
+    private static void Average<TOperator>(
+        Span<byte> destination,
+        int destinationStride,
+        ReadOnlySpan<byte> second,
+        int secondStride,
+        int width,
+        int height)
+        where TOperator : struct, IAv1CompoundAverageOperator
+    {
+        for (int row = 0; row < height; row++)
+        {
+            Span<byte> destinationRow = destination.Slice(row * destinationStride, width);
+            ReadOnlySpan<byte> secondRow = second.Slice(row * secondStride, width);
+            ref byte destinationReference = ref MemoryMarshal.GetReference(destinationRow);
+            ref byte secondReference = ref MemoryMarshal.GetReference(secondRow);
+            int column = 0;
+
+            if (Vector512.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector512<byte>.Count;
+                for (; column <= vectorEnd; column += Vector512<byte>.Count)
+                {
+                    Vector512<byte> firstVector = Vector512.LoadUnsafe(ref destinationReference, (nuint)column);
+                    Vector512<byte> secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column);
+                    TOperator.Blend(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column);
+                }
+            }
+
+            if (Vector256.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector256<byte>.Count;
+                for (; column <= vectorEnd; column += Vector256<byte>.Count)
+                {
+                    Vector256<byte> firstVector = Vector256.LoadUnsafe(ref destinationReference, (nuint)column);
+                    Vector256<byte> secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column);
+                    TOperator.Blend(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column);
+                }
+            }
+
+            if (Vector128.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector128<byte>.Count;
+                for (; column <= vectorEnd; column += Vector128<byte>.Count)
+                {
+                    Vector128<byte> firstVector = Vector128.LoadUnsafe(ref destinationReference, (nuint)column);
+                    Vector128<byte> secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column);
+                    TOperator.Blend(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column);
+                }
+            }
+
+            for (; column < width; column++)
+            {
+                destinationRow[column] = TOperator.Blend(destinationRow[column], secondRow[column]);
+            }
+        }
+    }
+
+    /// <summary>
+    /// Averages a high-bit-depth predictor into an existing prediction block.
+    /// </summary>
+    /// <param name="destination">The first predictor and combined output.</param>
+    /// <param name="destinationStride">The distance between destination rows in samples.</param>
+    /// <param name="second">The second predictor.</param>
+    /// <param name="secondStride">The distance between second-predictor rows in samples.</param>
+    /// <param name="width">The active block width.</param>
+    /// <param name="height">The active block height.</param>
+    public static void Average(
+        Span<ushort> destination,
+        int destinationStride,
+        ReadOnlySpan<ushort> second,
+        int secondStride,
+        int width,
+        int height)
+        => Average<CompoundAverageOperator>(destination, destinationStride, second, secondStride, width, height);
+
+    /// <summary>
+    /// Averages a high-bit-depth predictor through one closed compound operator.
+    /// </summary>
+    /// <typeparam name="TOperator">The equal-weight averaging arithmetic.</typeparam>
+    /// <param name="destination">The first predictor and combined output.</param>
+    /// <param name="destinationStride">The distance between destination rows.</param>
+    /// <param name="second">The second predictor.</param>
+    /// <param name="secondStride">The distance between second-predictor rows.</param>
+    /// <param name="width">The active block width.</param>
+    /// <param name="height">The active block height.</param>
+    private static void Average<TOperator>(
+        Span<ushort> destination,
+        int destinationStride,
+        ReadOnlySpan<ushort> second,
+        int secondStride,
+        int width,
+        int height)
+        where TOperator : struct, IAv1CompoundAverageOperator
+    {
+        for (int row = 0; row < height; row++)
+        {
+            Span<ushort> destinationRow = destination.Slice(row * destinationStride, width);
+            ReadOnlySpan<ushort> secondRow = second.Slice(row * secondStride, width);
+            ref ushort destinationReference = ref MemoryMarshal.GetReference(destinationRow);
+            ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow);
+            int column = 0;
+
+            if (Vector512.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector512<ushort>.Count;
+                for (; column <= vectorEnd; column += Vector512<ushort>.Count)
+                {
+                    Vector512<ushort> firstVector = Vector512.LoadUnsafe(ref destinationReference, (nuint)column);
+                    Vector512<ushort> secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column);
+                    TOperator.Blend(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column);
+                }
+            }
+
+            if (Vector256.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector256<ushort>.Count;
+                for (; column <= vectorEnd; column += Vector256<ushort>.Count)
+                {
+                    Vector256<ushort> firstVector = Vector256.LoadUnsafe(ref destinationReference, (nuint)column);
+                    Vector256<ushort> secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column);
+                    TOperator.Blend(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column);
+                }
+            }
+
+            if (Vector128.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector128<ushort>.Count;
+                for (; column <= vectorEnd; column += Vector128<ushort>.Count)
+                {
+                    Vector128<ushort> firstVector = Vector128.LoadUnsafe(ref destinationReference, (nuint)column);
+                    Vector128<ushort> secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column);
+                    TOperator.Blend(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column);
+                }
+            }
+
+            for (; column < width; column++)
+            {
+                destinationRow[column] = TOperator.Blend(destinationRow[column], secondRow[column]);
+            }
+        }
+    }
+
+    /// <summary>
+    /// Averages an 8-bit predictor without explicit hardware intrinsics.
+    /// </summary>
+    /// <param name="destination">The first predictor and combined output.</param>
+    /// <param name="destinationStride">The distance between destination rows in samples.</param>
+    /// <param name="second">The second predictor.</param>
+    /// <param name="secondStride">The distance between second-predictor rows in samples.</param>
+    /// <param name="width">The active block width.</param>
+    /// <param name="height">The active block height.</param>
+    public static void AverageScalar(
+        Span<byte> destination,
+        int destinationStride,
+        ReadOnlySpan<byte> second,
+        int secondStride,
+        int width,
+        int height)
+        => AverageScalar<CompoundAverageOperator>(destination, destinationStride, second, secondStride, width, height);
+
+    /// <summary>
+    /// Averages an 8-bit predictor through one closed scalar compound operator.
+    /// </summary>
+    /// <typeparam name="TOperator">The equal-weight averaging arithmetic.</typeparam>
+    /// <param name="destination">The first predictor and combined output.</param>
+    /// <param name="destinationStride">The distance between destination rows.</param>
+    /// <param name="second">The second predictor.</param>
+    /// <param name="secondStride">The distance between second-predictor rows.</param>
+    /// <param name="width">The active block width.</param>
+    /// <param name="height">The active block height.</param>
+    private static void AverageScalar<TOperator>(
+        Span<byte> destination,
+        int destinationStride,
+        ReadOnlySpan<byte> second,
+        int secondStride,
+        int width,
+        int height)
+        where TOperator : struct, IAv1CompoundAverageOperator
+    {
+        for (int row = 0; row < height; row++)
+        {
+            Span<byte> destinationRow = destination.Slice(row * destinationStride, width);
+            ReadOnlySpan<byte> secondRow = second.Slice(row * secondStride, width);
+            for (int column = 0; column < width; column++)
+            {
+                destinationRow[column] = TOperator.Blend(destinationRow[column], secondRow[column]);
+            }
+        }
+    }
+
+    /// <summary>
+    /// Averages a high-bit-depth predictor without explicit hardware intrinsics.
+    /// </summary>
+    /// <param name="destination">The first predictor and combined output.</param>
+    /// <param name="destinationStride">The distance between destination rows in samples.</param>
+    /// <param name="second">The second predictor.</param>
+    /// <param name="secondStride">The distance between second-predictor rows in samples.</param>
+    /// <param name="width">The active block width.</param>
+    /// <param name="height">The active block height.</param>
+    public static void AverageScalar(
+        Span<ushort> destination,
+        int destinationStride,
+        ReadOnlySpan<ushort> second,
+        int secondStride,
+        int width,
+        int height)
+        => AverageScalar<CompoundAverageOperator>(destination, destinationStride, second, secondStride, width, height);
+
+    /// <summary>
+    /// Averages a high-bit-depth predictor through one closed scalar compound operator.
+    /// </summary>
+    /// <typeparam name="TOperator">The equal-weight averaging arithmetic.</typeparam>
+    /// <param name="destination">The first predictor and combined output.</param>
+    /// <param name="destinationStride">The distance between destination rows.</param>
+    /// <param name="second">The second predictor.</param>
+    /// <param name="secondStride">The distance between second-predictor rows.</param>
+    /// <param name="width">The active block width.</param>
+    /// <param name="height">The active block height.</param>
+    private static void AverageScalar<TOperator>(
+        Span<ushort> destination,
+        int destinationStride,
+        ReadOnlySpan<ushort> second,
+        int secondStride,
+        int width,
+        int height)
+        where TOperator : struct, IAv1CompoundAverageOperator
+    {
+        for (int row = 0; row < height; row++)
+        {
+            Span<ushort> destinationRow = destination.Slice(row * destinationStride, width);
+            ReadOnlySpan<ushort> secondRow = second.Slice(row * secondStride, width);
+            for (int column = 0; column < width; column++)
+            {
+                destinationRow[column] = TOperator.Blend(destinationRow[column], secondRow[column]);
+            }
+        }
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundDistanceWeightedPredictor.Operator.cs

@@ -0,0 +1,211 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.CompilerServices;
+using System.Runtime.Intrinsics;
+
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor;
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
+
+/// <content>
+/// Defines distance-weighted compound prediction arithmetic.
+/// </content>
+internal static partial class Av1CompoundDistanceWeightedPredictor
+{
+    /// <summary>
+    /// Defines distance-weighted compound blending for scalar and SIMD lane groups.
+    /// </summary>
+    private interface IAv1CompoundDistanceWeightedOperator
+    {
+        /// <summary>
+        /// Blends two 8-bit samples with display-distance weights.
+        /// </summary>
+        /// <param name="first">The first sample.</param>
+        /// <param name="second">The second sample.</param>
+        /// <param name="firstWeight">The first predictor weight.</param>
+        /// <param name="secondWeight">The second predictor weight.</param>
+        /// <returns>The weighted sample.</returns>
+        public static abstract byte Blend(byte first, byte second, int firstWeight, int secondWeight);
+
+        /// <summary>
+        /// Blends two high-bit-depth samples with display-distance weights.
+        /// </summary>
+        /// <param name="first">The first sample.</param>
+        /// <param name="second">The second sample.</param>
+        /// <param name="firstWeight">The first predictor weight.</param>
+        /// <param name="secondWeight">The second predictor weight.</param>
+        /// <returns>The weighted sample.</returns>
+        public static abstract ushort Blend(ushort first, ushort second, int firstWeight, int secondWeight);
+
+        /// <summary>
+        /// Blends 128-bit vectors of 8-bit samples with display-distance weights.
+        /// </summary>
+        /// <param name="first">The first samples.</param>
+        /// <param name="second">The second samples.</param>
+        /// <param name="firstWeight">The first predictor weight.</param>
+        /// <param name="secondWeight">The second predictor weight.</param>
+        /// <returns>The weighted samples.</returns>
+        public static abstract Vector128<byte> Blend(Vector128<byte> first, Vector128<byte> second, int firstWeight, int secondWeight);
+
+        /// <summary>
+        /// Blends 256-bit vectors of 8-bit samples with display-distance weights.
+        /// </summary>
+        /// <param name="first">The first samples.</param>
+        /// <param name="second">The second samples.</param>
+        /// <param name="firstWeight">The first predictor weight.</param>
+        /// <param name="secondWeight">The second predictor weight.</param>
+        /// <returns>The weighted samples.</returns>
+        public static abstract Vector256<byte> Blend(Vector256<byte> first, Vector256<byte> second, int firstWeight, int secondWeight);
+
+        /// <summary>
+        /// Blends 512-bit vectors of 8-bit samples with display-distance weights.
+        /// </summary>
+        /// <param name="first">The first samples.</param>
+        /// <param name="second">The second samples.</param>
+        /// <param name="firstWeight">The first predictor weight.</param>
+        /// <param name="secondWeight">The second predictor weight.</param>
+        /// <returns>The weighted samples.</returns>
+        public static abstract Vector512<byte> Blend(Vector512<byte> first, Vector512<byte> second, int firstWeight, int secondWeight);
+
+        /// <summary>
+        /// Blends 128-bit vectors of high-bit-depth samples with display-distance weights.
+        /// </summary>
+        /// <param name="first">The first samples.</param>
+        /// <param name="second">The second samples.</param>
+        /// <param name="firstWeight">The first predictor weight.</param>
+        /// <param name="secondWeight">The second predictor weight.</param>
+        /// <returns>The weighted samples.</returns>
+        public static abstract Vector128<ushort> Blend(Vector128<ushort> first, Vector128<ushort> second, int firstWeight, int secondWeight);
+
+        /// <summary>
+        /// Blends 256-bit vectors of high-bit-depth samples with display-distance weights.
+        /// </summary>
+        /// <param name="first">The first samples.</param>
+        /// <param name="second">The second samples.</param>
+        /// <param name="firstWeight">The first predictor weight.</param>
+        /// <param name="secondWeight">The second predictor weight.</param>
+        /// <returns>The weighted samples.</returns>
+        public static abstract Vector256<ushort> Blend(Vector256<ushort> first, Vector256<ushort> second, int firstWeight, int secondWeight);
+
+        /// <summary>
+        /// Blends 512-bit vectors of high-bit-depth samples with display-distance weights.
+        /// </summary>
+        /// <param name="first">The first samples.</param>
+        /// <param name="second">The second samples.</param>
+        /// <param name="firstWeight">The first predictor weight.</param>
+        /// <param name="secondWeight">The second predictor weight.</param>
+        /// <returns>The weighted samples.</returns>
+        public static abstract Vector512<ushort> Blend(Vector512<ushort> first, Vector512<ushort> second, int firstWeight, int secondWeight);
+    }
+
+    /// <summary>
+    /// Implements display-distance-weighted blending for scalar and SIMD lane groups.
+    /// </summary>
+    private readonly struct CompoundDistanceWeightedOperator : IAv1CompoundDistanceWeightedOperator
+    {
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static byte Blend(byte first, byte second, int firstWeight, int secondWeight)
+            => (byte)(((first * firstWeight) + (second * secondWeight) + 8) >> DistanceWeightBits);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static ushort Blend(ushort first, ushort second, int firstWeight, int secondWeight)
+            => (ushort)(((first * firstWeight) + (second * secondWeight) + 8) >> DistanceWeightBits);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector128<byte> Blend(Vector128<byte> first, Vector128<byte> second, int firstWeight, int secondWeight)
+        {
+            Av1IntraPredictorBase.Widen(first, out Vector128<int> first0, out Vector128<int> first1, out Vector128<int> first2, out Vector128<int> first3);
+            Av1IntraPredictorBase.Widen(second, out Vector128<int> second0, out Vector128<int> second1, out Vector128<int> second2, out Vector128<int> second3);
+            return Av1IntraPredictorBase.Narrow(
+                Blend(first0, second0, firstWeight, secondWeight),
+                Blend(first1, second1, firstWeight, secondWeight),
+                Blend(first2, second2, firstWeight, secondWeight),
+                Blend(first3, second3, firstWeight, secondWeight));
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector256<byte> Blend(Vector256<byte> first, Vector256<byte> second, int firstWeight, int secondWeight)
+        {
+            Av1IntraPredictorBase.Widen(first, out Vector256<int> first0, out Vector256<int> first1, out Vector256<int> first2, out Vector256<int> first3);
+            Av1IntraPredictorBase.Widen(second, out Vector256<int> second0, out Vector256<int> second1, out Vector256<int> second2, out Vector256<int> second3);
+            return Av1IntraPredictorBase.Narrow(
+                Blend(first0, second0, firstWeight, secondWeight),
+                Blend(first1, second1, firstWeight, secondWeight),
+                Blend(first2, second2, firstWeight, secondWeight),
+                Blend(first3, second3, firstWeight, secondWeight));
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector512<byte> Blend(Vector512<byte> first, Vector512<byte> second, int firstWeight, int secondWeight)
+        {
+            Av1IntraPredictorBase.Widen(first, out Vector512<int> first0, out Vector512<int> first1, out Vector512<int> first2, out Vector512<int> first3);
+            Av1IntraPredictorBase.Widen(second, out Vector512<int> second0, out Vector512<int> second1, out Vector512<int> second2, out Vector512<int> second3);
+            return Av1IntraPredictorBase.Narrow(
+                Blend(first0, second0, firstWeight, secondWeight),
+                Blend(first1, second1, firstWeight, secondWeight),
+                Blend(first2, second2, firstWeight, secondWeight),
+                Blend(first3, second3, firstWeight, secondWeight));
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector128<ushort> Blend(Vector128<ushort> first, Vector128<ushort> second, int firstWeight, int secondWeight)
+        {
+            Av1IntraPredictorBase.Widen(first.AsInt16(), out Vector128<int> first0, out Vector128<int> first1);
+            Av1IntraPredictorBase.Widen(second.AsInt16(), out Vector128<int> second0, out Vector128<int> second1);
+            return Av1IntraPredictorBase.Narrow(
+                Blend(first0, second0, firstWeight, secondWeight),
+                Blend(first1, second1, firstWeight, secondWeight)).AsUInt16();
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector256<ushort> Blend(Vector256<ushort> first, Vector256<ushort> second, int firstWeight, int secondWeight)
+        {
+            Av1IntraPredictorBase.Widen(first.AsInt16(), out Vector256<int> first0, out Vector256<int> first1);
+            Av1IntraPredictorBase.Widen(second.AsInt16(), out Vector256<int> second0, out Vector256<int> second1);
+            return Av1IntraPredictorBase.Narrow(
+                Blend(first0, second0, firstWeight, secondWeight),
+                Blend(first1, second1, firstWeight, secondWeight)).AsUInt16();
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector512<ushort> Blend(Vector512<ushort> first, Vector512<ushort> second, int firstWeight, int secondWeight)
+        {
+            Av1IntraPredictorBase.Widen(first.AsInt16(), out Vector512<int> first0, out Vector512<int> first1);
+            Av1IntraPredictorBase.Widen(second.AsInt16(), out Vector512<int> second0, out Vector512<int> second1);
+            return Av1IntraPredictorBase.Narrow(
+                Blend(first0, second0, firstWeight, secondWeight),
+                Blend(first1, second1, firstWeight, secondWeight)).AsUInt16();
+        }
+
+        /// <summary>
+        /// Applies display-distance weighting to 128-bit vectors of widened samples.
+        /// </summary>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        private static Vector128<int> Blend(Vector128<int> first, Vector128<int> second, int firstWeight, int secondWeight)
+            => ((first * Vector128.Create(firstWeight)) + (second * Vector128.Create(secondWeight)) + Vector128.Create(8)) >> DistanceWeightBits;
+
+        /// <summary>
+        /// Applies display-distance weighting to 256-bit vectors of widened samples.
+        /// </summary>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        private static Vector256<int> Blend(Vector256<int> first, Vector256<int> second, int firstWeight, int secondWeight)
+            => ((first * Vector256.Create(firstWeight)) + (second * Vector256.Create(secondWeight)) + Vector256.Create(8)) >> DistanceWeightBits;
+
+        /// <summary>
+        /// Applies display-distance weighting to 512-bit vectors of widened samples.
+        /// </summary>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        private static Vector512<int> Blend(Vector512<int> first, Vector512<int> second, int firstWeight, int secondWeight)
+            => ((first * Vector512.Create(firstWeight)) + (second * Vector512.Create(secondWeight)) + Vector512.Create(8)) >> DistanceWeightBits;
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundDistanceWeightedPredictor.cs

@@ -0,0 +1,235 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.CompilerServices;
+using System.Runtime.InteropServices;
+using System.Runtime.Intrinsics;
+
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor;
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
+
+/// <content>
+/// Reconstructs display-distance-weighted compound prediction.
+/// </content>
+internal static partial class Av1CompoundDistanceWeightedPredictor
+{
+    /// <summary>
+    /// Combines two 8-bit predictors with AV1 display-distance weights.
+    /// </summary>
+    public static void DistanceWeighted(
+        Span<byte> destination,
+        int destinationStride,
+        ReadOnlySpan<byte> second,
+        int secondStride,
+        int width,
+        int height,
+        int firstWeight,
+        int secondWeight)
+        => DistanceWeighted<CompoundDistanceWeightedOperator>(
+            destination,
+            destinationStride,
+            second,
+            secondStride,
+            width,
+            height,
+            firstWeight,
+            secondWeight);
+
+    /// <summary>
+    /// Executes one closed 8-bit distance-weighted compound operator.
+    /// </summary>
+    /// <typeparam name="TOperator">The compound arithmetic operator.</typeparam>
+    private static void DistanceWeighted<TOperator>(
+        Span<byte> destination,
+        int destinationStride,
+        ReadOnlySpan<byte> second,
+        int secondStride,
+        int width,
+        int height,
+        int firstWeight,
+        int secondWeight)
+        where TOperator : struct, IAv1CompoundDistanceWeightedOperator
+    {
+        for (int row = 0; row < height; row++)
+        {
+            Span<byte> destinationRow = destination.Slice(row * destinationStride, width);
+            ReadOnlySpan<byte> secondRow = second.Slice(row * secondStride, width);
+            ref byte destinationReference = ref MemoryMarshal.GetReference(destinationRow);
+            ref byte secondReference = ref MemoryMarshal.GetReference(secondRow);
+            int column = 0;
+
+            if (Vector512.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector512<byte>.Count;
+                for (; column <= vectorEnd; column += Vector512<byte>.Count)
+                {
+                    Vector512<byte> firstVector = Vector512.LoadUnsafe(ref destinationReference, (nuint)column);
+                    Vector512<byte> secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column);
+                    TOperator.Blend(firstVector, secondVector, firstWeight, secondWeight).StoreUnsafe(ref destinationReference, (nuint)column);
+                }
+            }
+
+            if (Vector256.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector256<byte>.Count;
+                for (; column <= vectorEnd; column += Vector256<byte>.Count)
+                {
+                    Vector256<byte> firstVector = Vector256.LoadUnsafe(ref destinationReference, (nuint)column);
+                    Vector256<byte> secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column);
+                    TOperator.Blend(firstVector, secondVector, firstWeight, secondWeight).StoreUnsafe(ref destinationReference, (nuint)column);
+                }
+            }
+
+            if (Vector128.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector128<byte>.Count;
+                for (; column <= vectorEnd; column += Vector128<byte>.Count)
+                {
+                    Vector128<byte> firstVector = Vector128.LoadUnsafe(ref destinationReference, (nuint)column);
+                    Vector128<byte> secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column);
+                    TOperator.Blend(firstVector, secondVector, firstWeight, secondWeight).StoreUnsafe(ref destinationReference, (nuint)column);
+                }
+            }
+
+            for (; column < width; column++)
+            {
+                destinationRow[column] = TOperator.Blend(destinationRow[column], secondRow[column], firstWeight, secondWeight);
+            }
+        }
+    }
+
+    /// <summary>
+    /// Combines two high-bit-depth predictors with AV1 display-distance weights.
+    /// </summary>
+    public static void DistanceWeighted(
+        Span<ushort> destination,
+        int destinationStride,
+        ReadOnlySpan<ushort> second,
+        int secondStride,
+        int width,
+        int height,
+        int firstWeight,
+        int secondWeight)
+        => DistanceWeighted<CompoundDistanceWeightedOperator>(
+            destination,
+            destinationStride,
+            second,
+            secondStride,
+            width,
+            height,
+            firstWeight,
+            secondWeight);
+
+    /// <summary>
+    /// Executes one closed high-bit-depth distance-weighted compound operator.
+    /// </summary>
+    /// <typeparam name="TOperator">The compound arithmetic operator.</typeparam>
+    private static void DistanceWeighted<TOperator>(
+        Span<ushort> destination,
+        int destinationStride,
+        ReadOnlySpan<ushort> second,
+        int secondStride,
+        int width,
+        int height,
+        int firstWeight,
+        int secondWeight)
+        where TOperator : struct, IAv1CompoundDistanceWeightedOperator
+    {
+        for (int row = 0; row < height; row++)
+        {
+            Span<ushort> destinationRow = destination.Slice(row * destinationStride, width);
+            ReadOnlySpan<ushort> secondRow = second.Slice(row * secondStride, width);
+            ref ushort destinationReference = ref MemoryMarshal.GetReference(destinationRow);
+            ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow);
+            int column = 0;
+
+            if (Vector512.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector512<ushort>.Count;
+                for (; column <= vectorEnd; column += Vector512<ushort>.Count)
+                {
+                    Vector512<ushort> firstVector = Vector512.LoadUnsafe(ref destinationReference, (nuint)column);
+                    Vector512<ushort> secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column);
+                    TOperator.Blend(firstVector, secondVector, firstWeight, secondWeight).StoreUnsafe(ref destinationReference, (nuint)column);
+                }
+            }
+
+            if (Vector256.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector256<ushort>.Count;
+                for (; column <= vectorEnd; column += Vector256<ushort>.Count)
+                {
+                    Vector256<ushort> firstVector = Vector256.LoadUnsafe(ref destinationReference, (nuint)column);
+                    Vector256<ushort> secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column);
+                    TOperator.Blend(firstVector, secondVector, firstWeight, secondWeight).StoreUnsafe(ref destinationReference, (nuint)column);
+                }
+            }
+
+            if (Vector128.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector128<ushort>.Count;
+                for (; column <= vectorEnd; column += Vector128<ushort>.Count)
+                {
+                    Vector128<ushort> firstVector = Vector128.LoadUnsafe(ref destinationReference, (nuint)column);
+                    Vector128<ushort> secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column);
+                    TOperator.Blend(firstVector, secondVector, firstWeight, secondWeight).StoreUnsafe(ref destinationReference, (nuint)column);
+                }
+            }
+
+            for (; column < width; column++)
+            {
+                destinationRow[column] = TOperator.Blend(destinationRow[column], secondRow[column], firstWeight, secondWeight);
+            }
+        }
+    }
+
+    /// <summary>
+    /// Combines two 8-bit predictors with display-distance weights without explicit hardware intrinsics.
+    /// </summary>
+    public static void DistanceWeightedScalar(
+        Span<byte> destination,
+        int destinationStride,
+        ReadOnlySpan<byte> second,
+        int secondStride,
+        int width,
+        int height,
+        int firstWeight,
+        int secondWeight)
+        => DistanceWeightedScalar<CompoundDistanceWeightedOperator>(
+            destination,
+            destinationStride,
+            second,
+            secondStride,
+            width,
+            height,
+            firstWeight,
+            secondWeight);
+
+    /// <summary>
+    /// Executes one closed 8-bit distance-weighted compound operator without explicit hardware intrinsics.
+    /// </summary>
+    /// <typeparam name="TOperator">The compound arithmetic operator.</typeparam>
+    private static void DistanceWeightedScalar<TOperator>(
+        Span<byte> destination,
+        int destinationStride,
+        ReadOnlySpan<byte> second,
+        int secondStride,
+        int width,
+        int height,
+        int firstWeight,
+        int secondWeight)
+        where TOperator : struct, IAv1CompoundDistanceWeightedOperator
+    {
+        for (int row = 0; row < height; row++)
+        {
+            Span<byte> destinationRow = destination.Slice(row * destinationStride, width);
+            ReadOnlySpan<byte> secondRow = second.Slice(row * secondStride, width);
+            for (int column = 0; column < width; column++)
+            {
+                destinationRow[column] = TOperator.Blend(destinationRow[column], secondRow[column], firstWeight, secondWeight);
+            }
+        }
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.Arithmetic.cs

@@ -0,0 +1,112 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.Intrinsics;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
+
+/// <content>
+/// Provides shared final-rounding arithmetic for compound intermediate reconstruction.
+/// </content>
+internal static partial class Av1CompoundInterPredictor
+{
+    /// <summary>
+    /// Derives the bias and remaining fractional precision of a compound intermediate.
+    /// </summary>
+    public static void GetIntermediateRounding(int bitDepth, out int roundBits, out int roundOffset)
+    {
+        int intermediateRange = bitDepth + 7 - 3 + 2;
+        int round0 = 3 + Math.Max(intermediateRange - 16, 0);
+        int offsetBits = bitDepth + 14 - round0;
+        roundBits = 14 - round0 - CompoundRound1Bits;
+        roundOffset = (1 << (offsetBits - CompoundRound1Bits)) +
+            (1 << (offsetBits - CompoundRound1Bits - 1));
+    }
+
+    /// <summary>
+    /// Removes the compound bias and final fractional precision from 128-bit unsigned lanes.
+    /// </summary>
+    public static Vector128<ushort> FinalizeIntermediate(Vector128<ushort> value, int roundBits, int roundOffset)
+    {
+        Vector128<short> result = (value - Vector128.Create((ushort)roundOffset)).AsInt16();
+        if (roundBits != 0)
+        {
+            result = (result + Vector128.Create((short)(1 << (roundBits - 1)))) >> roundBits;
+        }
+
+        result = Vector128.Max(Vector128<short>.Zero, Vector128.Min(Vector128.Create((short)byte.MaxValue), result));
+        return result.AsUInt16();
+    }
+
+    /// <summary>
+    /// Removes the compound bias and final fractional precision from 256-bit unsigned lanes.
+    /// </summary>
+    public static Vector256<ushort> FinalizeIntermediate(Vector256<ushort> value, int roundBits, int roundOffset)
+    {
+        Vector256<short> result = (value - Vector256.Create((ushort)roundOffset)).AsInt16();
+        if (roundBits != 0)
+        {
+            result = (result + Vector256.Create((short)(1 << (roundBits - 1)))) >> roundBits;
+        }
+
+        result = Vector256.Max(Vector256<short>.Zero, Vector256.Min(Vector256.Create((short)byte.MaxValue), result));
+        return result.AsUInt16();
+    }
+
+    /// <summary>
+    /// Removes the compound bias and final fractional precision from 512-bit unsigned lanes.
+    /// </summary>
+    public static Vector512<ushort> FinalizeIntermediate(Vector512<ushort> value, int roundBits, int roundOffset)
+    {
+        Vector512<short> result = (value - Vector512.Create((ushort)roundOffset)).AsInt16();
+        if (roundBits != 0)
+        {
+            result = (result + Vector512.Create((short)(1 << (roundBits - 1)))) >> roundBits;
+        }
+
+        result = Vector512.Max(Vector512<short>.Zero, Vector512.Min(Vector512.Create((short)byte.MaxValue), result));
+        return result.AsUInt16();
+    }
+
+    /// <summary>
+    /// Removes the compound bias and final fractional precision from 128-bit widened lanes.
+    /// </summary>
+    public static Vector128<int> FinalizeIntermediate(Vector128<int> value, int roundBits, int roundOffset)
+    {
+        Vector128<int> result = value - Vector128.Create(roundOffset);
+        if (roundBits != 0)
+        {
+            result = (result + Vector128.Create(1 << (roundBits - 1))) >> roundBits;
+        }
+
+        return Vector128.Max(Vector128<int>.Zero, Vector128.Min(Vector128.Create((int)byte.MaxValue), result));
+    }
+
+    /// <summary>
+    /// Removes the compound bias and final fractional precision from 256-bit widened lanes.
+    /// </summary>
+    public static Vector256<int> FinalizeIntermediate(Vector256<int> value, int roundBits, int roundOffset)
+    {
+        Vector256<int> result = value - Vector256.Create(roundOffset);
+        if (roundBits != 0)
+        {
+            result = (result + Vector256.Create(1 << (roundBits - 1))) >> roundBits;
+        }
+
+        return Vector256.Max(Vector256<int>.Zero, Vector256.Min(Vector256.Create((int)byte.MaxValue), result));
+    }
+
+    /// <summary>
+    /// Removes the compound bias and final fractional precision from 512-bit widened lanes.
+    /// </summary>
+    public static Vector512<int> FinalizeIntermediate(Vector512<int> value, int roundBits, int roundOffset)
+    {
+        Vector512<int> result = value - Vector512.Create(roundOffset);
+        if (roundBits != 0)
+        {
+            result = (result + Vector512.Create(1 << (roundBits - 1))) >> roundBits;
+        }
+
+        return Vector512.Max(Vector512<int>.Zero, Vector512.Min(Vector512.Create((int)byte.MaxValue), result));
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.Blending.cs

@@ -1,501 +0,0 @@
-// Copyright (c) Six Labors.
-// Licensed under the Six Labors Split License.
-
-using System.Runtime.CompilerServices;
-using System.Runtime.InteropServices;
-using System.Runtime.Intrinsics;
-
-namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
-
-/// <content>
-/// Provides distance-weighted and per-sample masked compound blending.
-/// </content>
-internal static partial class Av1CompoundInterPredictor
-{
-    private const int DistanceWeightBits = 4;
-    private const int MaskWeightBits = 6;
-    private const int MaximumMaskAlpha = 1 << MaskWeightBits;
-
-    /// <summary>
-    /// Combines two 8-bit predictors with AV1 display-distance weights.
-    /// </summary>
-    public static void DistanceWeighted(
-        Span<byte> destination,
-        int destinationStride,
-        ReadOnlySpan<byte> second,
-        int secondStride,
-        int width,
-        int height,
-        int firstWeight,
-        int secondWeight)
-    {
-        for (int row = 0; row < height; row++)
-        {
-            Span<byte> destinationRow = destination.Slice(row * destinationStride, width);
-            ReadOnlySpan<byte> secondRow = second.Slice(row * secondStride, width);
-            ref byte destinationReference = ref MemoryMarshal.GetReference(destinationRow);
-            ref byte secondReference = ref MemoryMarshal.GetReference(secondRow);
-            int column = 0;
-
-            if (Vector512.IsHardwareAccelerated)
-            {
-                int vectorEnd = width - Vector512<byte>.Count;
-                for (; column <= vectorEnd; column += Vector512<byte>.Count)
-                {
-                    Vector512<byte> firstVector = Vector512.LoadUnsafe(ref destinationReference, (nuint)column);
-                    Vector512<byte> secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column);
-                    DistanceWeighted(firstVector, secondVector, firstWeight, secondWeight).StoreUnsafe(ref destinationReference, (nuint)column);
-                }
-            }
-
-            if (Vector256.IsHardwareAccelerated)
-            {
-                int vectorEnd = width - Vector256<byte>.Count;
-                for (; column <= vectorEnd; column += Vector256<byte>.Count)
-                {
-                    Vector256<byte> firstVector = Vector256.LoadUnsafe(ref destinationReference, (nuint)column);
-                    Vector256<byte> secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column);
-                    DistanceWeighted(firstVector, secondVector, firstWeight, secondWeight).StoreUnsafe(ref destinationReference, (nuint)column);
-                }
-            }
-
-            if (Vector128.IsHardwareAccelerated)
-            {
-                int vectorEnd = width - Vector128<byte>.Count;
-                for (; column <= vectorEnd; column += Vector128<byte>.Count)
-                {
-                    Vector128<byte> firstVector = Vector128.LoadUnsafe(ref destinationReference, (nuint)column);
-                    Vector128<byte> secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column);
-                    DistanceWeighted(firstVector, secondVector, firstWeight, secondWeight).StoreUnsafe(ref destinationReference, (nuint)column);
-                }
-            }
-
-            for (; column < width; column++)
-            {
-                destinationRow[column] = (byte)(((destinationRow[column] * firstWeight) + (secondRow[column] * secondWeight) + 8) >> DistanceWeightBits);
-            }
-        }
-    }
-
-    /// <summary>
-    /// Combines two high-bit-depth predictors with AV1 display-distance weights.
-    /// </summary>
-    public static void DistanceWeighted(
-        Span<ushort> destination,
-        int destinationStride,
-        ReadOnlySpan<ushort> second,
-        int secondStride,
-        int width,
-        int height,
-        int firstWeight,
-        int secondWeight)
-    {
-        for (int row = 0; row < height; row++)
-        {
-            Span<ushort> destinationRow = destination.Slice(row * destinationStride, width);
-            ReadOnlySpan<ushort> secondRow = second.Slice(row * secondStride, width);
-            ref ushort destinationReference = ref MemoryMarshal.GetReference(destinationRow);
-            ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow);
-            int column = 0;
-
-            if (Vector512.IsHardwareAccelerated)
-            {
-                int vectorEnd = width - Vector512<ushort>.Count;
-                for (; column <= vectorEnd; column += Vector512<ushort>.Count)
-                {
-                    Vector512<ushort> firstVector = Vector512.LoadUnsafe(ref destinationReference, (nuint)column);
-                    Vector512<ushort> secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column);
-                    DistanceWeighted(firstVector, secondVector, firstWeight, secondWeight).StoreUnsafe(ref destinationReference, (nuint)column);
-                }
-            }
-
-            if (Vector256.IsHardwareAccelerated)
-            {
-                int vectorEnd = width - Vector256<ushort>.Count;
-                for (; column <= vectorEnd; column += Vector256<ushort>.Count)
-                {
-                    Vector256<ushort> firstVector = Vector256.LoadUnsafe(ref destinationReference, (nuint)column);
-                    Vector256<ushort> secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column);
-                    DistanceWeighted(firstVector, secondVector, firstWeight, secondWeight).StoreUnsafe(ref destinationReference, (nuint)column);
-                }
-            }
-
-            if (Vector128.IsHardwareAccelerated)
-            {
-                int vectorEnd = width - Vector128<ushort>.Count;
-                for (; column <= vectorEnd; column += Vector128<ushort>.Count)
-                {
-                    Vector128<ushort> firstVector = Vector128.LoadUnsafe(ref destinationReference, (nuint)column);
-                    Vector128<ushort> secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column);
-                    DistanceWeighted(firstVector, secondVector, firstWeight, secondWeight).StoreUnsafe(ref destinationReference, (nuint)column);
-                }
-            }
-
-            for (; column < width; column++)
-            {
-                destinationRow[column] = (ushort)(((destinationRow[column] * firstWeight) + (secondRow[column] * secondWeight) + 8) >> DistanceWeightBits);
-            }
-        }
-    }
-
-    /// <summary>
-    /// Blends two 8-bit predictors through a contiguous AV1 alpha mask.
-    /// </summary>
-    public static void Blend(
-        Span<byte> destination,
-        int destinationStride,
-        ReadOnlySpan<byte> second,
-        int secondStride,
-        ReadOnlySpan<byte> mask,
-        int maskStride,
-        int width,
-        int height)
-    {
-        for (int row = 0; row < height; row++)
-        {
-            Span<byte> destinationRow = destination.Slice(row * destinationStride, width);
-            ReadOnlySpan<byte> secondRow = second.Slice(row * secondStride, width);
-            ReadOnlySpan<byte> maskRow = mask.Slice(row * maskStride, width);
-            ref byte destinationReference = ref MemoryMarshal.GetReference(destinationRow);
-            ref byte secondReference = ref MemoryMarshal.GetReference(secondRow);
-            ref byte maskReference = ref MemoryMarshal.GetReference(maskRow);
-            int column = 0;
-
-            if (Vector512.IsHardwareAccelerated)
-            {
-                int vectorEnd = width - Vector512<byte>.Count;
-                for (; column <= vectorEnd; column += Vector512<byte>.Count)
-                {
-                    Vector512<byte> firstVector = Vector512.LoadUnsafe(ref destinationReference, (nuint)column);
-                    Vector512<byte> secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column);
-                    Vector512<byte> maskVector = Vector512.LoadUnsafe(ref maskReference, (nuint)column);
-                    Blend(firstVector, secondVector, maskVector).StoreUnsafe(ref destinationReference, (nuint)column);
-                }
-            }
-
-            if (Vector256.IsHardwareAccelerated)
-            {
-                int vectorEnd = width - Vector256<byte>.Count;
-                for (; column <= vectorEnd; column += Vector256<byte>.Count)
-                {
-                    Vector256<byte> firstVector = Vector256.LoadUnsafe(ref destinationReference, (nuint)column);
-                    Vector256<byte> secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column);
-                    Vector256<byte> maskVector = Vector256.LoadUnsafe(ref maskReference, (nuint)column);
-                    Blend(firstVector, secondVector, maskVector).StoreUnsafe(ref destinationReference, (nuint)column);
-                }
-            }
-
-            if (Vector128.IsHardwareAccelerated)
-            {
-                int vectorEnd = width - Vector128<byte>.Count;
-                for (; column <= vectorEnd; column += Vector128<byte>.Count)
-                {
-                    Vector128<byte> firstVector = Vector128.LoadUnsafe(ref destinationReference, (nuint)column);
-                    Vector128<byte> secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column);
-                    Vector128<byte> maskVector = Vector128.LoadUnsafe(ref maskReference, (nuint)column);
-                    Blend(firstVector, secondVector, maskVector).StoreUnsafe(ref destinationReference, (nuint)column);
-                }
-            }
-
-            for (; column < width; column++)
-            {
-                int alpha = maskRow[column];
-                destinationRow[column] = (byte)(((alpha * destinationRow[column]) + ((MaximumMaskAlpha - alpha) * secondRow[column]) + 32) >> MaskWeightBits);
-            }
-        }
-    }
-
-    /// <summary>
-    /// Blends two high-bit-depth predictors through a contiguous AV1 alpha mask.
-    /// </summary>
-    public static void Blend(
-        Span<ushort> destination,
-        int destinationStride,
-        ReadOnlySpan<ushort> second,
-        int secondStride,
-        ReadOnlySpan<byte> mask,
-        int maskStride,
-        int width,
-        int height)
-    {
-        for (int row = 0; row < height; row++)
-        {
-            Span<ushort> destinationRow = destination.Slice(row * destinationStride, width);
-            ReadOnlySpan<ushort> secondRow = second.Slice(row * secondStride, width);
-            ReadOnlySpan<byte> maskRow = mask.Slice(row * maskStride, width);
-            ref ushort destinationReference = ref MemoryMarshal.GetReference(destinationRow);
-            ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow);
-            ref byte maskReference = ref MemoryMarshal.GetReference(maskRow);
-            int column = 0;
-
-            if (Vector512.IsHardwareAccelerated)
-            {
-                int vectorEnd = width - Vector512<ushort>.Count;
-                for (; column <= vectorEnd; column += Vector512<ushort>.Count)
-                {
-                    Vector512<ushort> firstVector = Vector512.LoadUnsafe(ref destinationReference, (nuint)column);
-                    Vector512<ushort> secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column);
-                    Vector512<ushort> maskVector = LoadMask512(ref maskReference, column);
-                    Blend(firstVector, secondVector, maskVector).StoreUnsafe(ref destinationReference, (nuint)column);
-                }
-            }
-
-            if (Vector256.IsHardwareAccelerated)
-            {
-                int vectorEnd = width - Vector256<ushort>.Count;
-                for (; column <= vectorEnd; column += Vector256<ushort>.Count)
-                {
-                    Vector256<ushort> firstVector = Vector256.LoadUnsafe(ref destinationReference, (nuint)column);
-                    Vector256<ushort> secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column);
-                    Vector256<ushort> maskVector = LoadMask256(ref maskReference, column);
-                    Blend(firstVector, secondVector, maskVector).StoreUnsafe(ref destinationReference, (nuint)column);
-                }
-            }
-
-            if (Vector128.IsHardwareAccelerated)
-            {
-                int vectorEnd = width - Vector128<ushort>.Count;
-                for (; column <= vectorEnd; column += Vector128<ushort>.Count)
-                {
-                    Vector128<ushort> firstVector = Vector128.LoadUnsafe(ref destinationReference, (nuint)column);
-                    Vector128<ushort> secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column);
-                    Vector128<ushort> maskVector = LoadMask128(ref maskReference, column);
-                    Blend(firstVector, secondVector, maskVector).StoreUnsafe(ref destinationReference, (nuint)column);
-                }
-            }
-
-            for (; column < width; column++)
-            {
-                int alpha = maskRow[column];
-                destinationRow[column] = (ushort)(((alpha * destinationRow[column]) + ((MaximumMaskAlpha - alpha) * secondRow[column]) + 32) >> MaskWeightBits);
-            }
-        }
-    }
-
-    /// <summary>
-    /// Combines two 8-bit predictors with display-distance weights without explicit hardware intrinsics.
-    /// </summary>
-    public static void DistanceWeightedScalar(
-        Span<byte> destination,
-        int destinationStride,
-        ReadOnlySpan<byte> second,
-        int secondStride,
-        int width,
-        int height,
-        int firstWeight,
-        int secondWeight)
-    {
-        for (int row = 0; row < height; row++)
-        {
-            Span<byte> destinationRow = destination.Slice(row * destinationStride, width);
-            ReadOnlySpan<byte> secondRow = second.Slice(row * secondStride, width);
-            for (int column = 0; column < width; column++)
-            {
-                destinationRow[column] = (byte)(((destinationRow[column] * firstWeight) + (secondRow[column] * secondWeight) + 8) >> DistanceWeightBits);
-            }
-        }
-    }
-
-    /// <summary>
-    /// Blends two 8-bit predictors through an alpha mask without explicit hardware intrinsics.
-    /// </summary>
-    public static void BlendScalar(
-        Span<byte> destination,
-        int destinationStride,
-        ReadOnlySpan<byte> second,
-        int secondStride,
-        ReadOnlySpan<byte> mask,
-        int maskStride,
-        int width,
-        int height)
-    {
-        for (int row = 0; row < height; row++)
-        {
-            Span<byte> destinationRow = destination.Slice(row * destinationStride, width);
-            ReadOnlySpan<byte> secondRow = second.Slice(row * secondStride, width);
-            ReadOnlySpan<byte> maskRow = mask.Slice(row * maskStride, width);
-            for (int column = 0; column < width; column++)
-            {
-                int alpha = maskRow[column];
-                destinationRow[column] = (byte)(((alpha * destinationRow[column]) + ((MaximumMaskAlpha - alpha) * secondRow[column]) + 32) >> MaskWeightBits);
-            }
-        }
-    }
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector128<byte> DistanceWeighted(Vector128<byte> first, Vector128<byte> second, int firstWeight, int secondWeight)
-    {
-        Av1IntraPredictorBase.Widen(first, out Vector128<int> first0, out Vector128<int> first1, out Vector128<int> first2, out Vector128<int> first3);
-        Av1IntraPredictorBase.Widen(second, out Vector128<int> second0, out Vector128<int> second1, out Vector128<int> second2, out Vector128<int> second3);
-        return Av1IntraPredictorBase.Narrow(
-            DistanceWeighted(first0, second0, firstWeight, secondWeight),
-            DistanceWeighted(first1, second1, firstWeight, secondWeight),
-            DistanceWeighted(first2, second2, firstWeight, secondWeight),
-            DistanceWeighted(first3, second3, firstWeight, secondWeight));
-    }
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector256<byte> DistanceWeighted(Vector256<byte> first, Vector256<byte> second, int firstWeight, int secondWeight)
-    {
-        Av1IntraPredictorBase.Widen(first, out Vector256<int> first0, out Vector256<int> first1, out Vector256<int> first2, out Vector256<int> first3);
-        Av1IntraPredictorBase.Widen(second, out Vector256<int> second0, out Vector256<int> second1, out Vector256<int> second2, out Vector256<int> second3);
-        return Av1IntraPredictorBase.Narrow(
-            DistanceWeighted(first0, second0, firstWeight, secondWeight),
-            DistanceWeighted(first1, second1, firstWeight, secondWeight),
-            DistanceWeighted(first2, second2, firstWeight, secondWeight),
-            DistanceWeighted(first3, second3, firstWeight, secondWeight));
-    }
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector512<byte> DistanceWeighted(Vector512<byte> first, Vector512<byte> second, int firstWeight, int secondWeight)
-    {
-        Av1IntraPredictorBase.Widen(first, out Vector512<int> first0, out Vector512<int> first1, out Vector512<int> first2, out Vector512<int> first3);
-        Av1IntraPredictorBase.Widen(second, out Vector512<int> second0, out Vector512<int> second1, out Vector512<int> second2, out Vector512<int> second3);
-        return Av1IntraPredictorBase.Narrow(
-            DistanceWeighted(first0, second0, firstWeight, secondWeight),
-            DistanceWeighted(first1, second1, firstWeight, secondWeight),
-            DistanceWeighted(first2, second2, firstWeight, secondWeight),
-            DistanceWeighted(first3, second3, firstWeight, secondWeight));
-    }
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector128<ushort> DistanceWeighted(Vector128<ushort> first, Vector128<ushort> second, int firstWeight, int secondWeight)
-    {
-        Av1IntraPredictorBase.Widen(first.AsInt16(), out Vector128<int> first0, out Vector128<int> first1);
-        Av1IntraPredictorBase.Widen(second.AsInt16(), out Vector128<int> second0, out Vector128<int> second1);
-        return Av1IntraPredictorBase.Narrow(
-            DistanceWeighted(first0, second0, firstWeight, secondWeight),
-            DistanceWeighted(first1, second1, firstWeight, secondWeight)).AsUInt16();
-    }
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector256<ushort> DistanceWeighted(Vector256<ushort> first, Vector256<ushort> second, int firstWeight, int secondWeight)
-    {
-        Av1IntraPredictorBase.Widen(first.AsInt16(), out Vector256<int> first0, out Vector256<int> first1);
-        Av1IntraPredictorBase.Widen(second.AsInt16(), out Vector256<int> second0, out Vector256<int> second1);
-        return Av1IntraPredictorBase.Narrow(
-            DistanceWeighted(first0, second0, firstWeight, secondWeight),
-            DistanceWeighted(first1, second1, firstWeight, secondWeight)).AsUInt16();
-    }
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector512<ushort> DistanceWeighted(Vector512<ushort> first, Vector512<ushort> second, int firstWeight, int secondWeight)
-    {
-        Av1IntraPredictorBase.Widen(first.AsInt16(), out Vector512<int> first0, out Vector512<int> first1);
-        Av1IntraPredictorBase.Widen(second.AsInt16(), out Vector512<int> second0, out Vector512<int> second1);
-        return Av1IntraPredictorBase.Narrow(
-            DistanceWeighted(first0, second0, firstWeight, secondWeight),
-            DistanceWeighted(first1, second1, firstWeight, secondWeight)).AsUInt16();
-    }
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector128<int> DistanceWeighted(Vector128<int> first, Vector128<int> second, int firstWeight, int secondWeight)
-        => ((first * Vector128.Create(firstWeight)) + (second * Vector128.Create(secondWeight)) + Vector128.Create(8)) >> DistanceWeightBits;
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector256<int> DistanceWeighted(Vector256<int> first, Vector256<int> second, int firstWeight, int secondWeight)
-        => ((first * Vector256.Create(firstWeight)) + (second * Vector256.Create(secondWeight)) + Vector256.Create(8)) >> DistanceWeightBits;
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector512<int> DistanceWeighted(Vector512<int> first, Vector512<int> second, int firstWeight, int secondWeight)
-        => ((first * Vector512.Create(firstWeight)) + (second * Vector512.Create(secondWeight)) + Vector512.Create(8)) >> DistanceWeightBits;
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector128<byte> Blend(Vector128<byte> first, Vector128<byte> second, Vector128<byte> mask)
-    {
-        Av1IntraPredictorBase.Widen(first, out Vector128<int> first0, out Vector128<int> first1, out Vector128<int> first2, out Vector128<int> first3);
-        Av1IntraPredictorBase.Widen(second, out Vector128<int> second0, out Vector128<int> second1, out Vector128<int> second2, out Vector128<int> second3);
-        Av1IntraPredictorBase.Widen(mask, out Vector128<int> mask0, out Vector128<int> mask1, out Vector128<int> mask2, out Vector128<int> mask3);
-        return Av1IntraPredictorBase.Narrow(
-            Blend(first0, second0, mask0),
-            Blend(first1, second1, mask1),
-            Blend(first2, second2, mask2),
-            Blend(first3, second3, mask3));
-    }
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector256<byte> Blend(Vector256<byte> first, Vector256<byte> second, Vector256<byte> mask)
-    {
-        Av1IntraPredictorBase.Widen(first, out Vector256<int> first0, out Vector256<int> first1, out Vector256<int> first2, out Vector256<int> first3);
-        Av1IntraPredictorBase.Widen(second, out Vector256<int> second0, out Vector256<int> second1, out Vector256<int> second2, out Vector256<int> second3);
-        Av1IntraPredictorBase.Widen(mask, out Vector256<int> mask0, out Vector256<int> mask1, out Vector256<int> mask2, out Vector256<int> mask3);
-        return Av1IntraPredictorBase.Narrow(
-            Blend(first0, second0, mask0),
-            Blend(first1, second1, mask1),
-            Blend(first2, second2, mask2),
-            Blend(first3, second3, mask3));
-    }
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector512<byte> Blend(Vector512<byte> first, Vector512<byte> second, Vector512<byte> mask)
-    {
-        Av1IntraPredictorBase.Widen(first, out Vector512<int> first0, out Vector512<int> first1, out Vector512<int> first2, out Vector512<int> first3);
-        Av1IntraPredictorBase.Widen(second, out Vector512<int> second0, out Vector512<int> second1, out Vector512<int> second2, out Vector512<int> second3);
-        Av1IntraPredictorBase.Widen(mask, out Vector512<int> mask0, out Vector512<int> mask1, out Vector512<int> mask2, out Vector512<int> mask3);
-        return Av1IntraPredictorBase.Narrow(
-            Blend(first0, second0, mask0),
-            Blend(first1, second1, mask1),
-            Blend(first2, second2, mask2),
-            Blend(first3, second3, mask3));
-    }
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector128<ushort> Blend(Vector128<ushort> first, Vector128<ushort> second, Vector128<ushort> mask)
-    {
-        Av1IntraPredictorBase.Widen(first.AsInt16(), out Vector128<int> first0, out Vector128<int> first1);
-        Av1IntraPredictorBase.Widen(second.AsInt16(), out Vector128<int> second0, out Vector128<int> second1);
-        Av1IntraPredictorBase.Widen(mask.AsInt16(), out Vector128<int> mask0, out Vector128<int> mask1);
-        return Av1IntraPredictorBase.Narrow(Blend(first0, second0, mask0), Blend(first1, second1, mask1)).AsUInt16();
-    }
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector256<ushort> Blend(Vector256<ushort> first, Vector256<ushort> second, Vector256<ushort> mask)
-    {
-        Av1IntraPredictorBase.Widen(first.AsInt16(), out Vector256<int> first0, out Vector256<int> first1);
-        Av1IntraPredictorBase.Widen(second.AsInt16(), out Vector256<int> second0, out Vector256<int> second1);
-        Av1IntraPredictorBase.Widen(mask.AsInt16(), out Vector256<int> mask0, out Vector256<int> mask1);
-        return Av1IntraPredictorBase.Narrow(Blend(first0, second0, mask0), Blend(first1, second1, mask1)).AsUInt16();
-    }
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector512<ushort> Blend(Vector512<ushort> first, Vector512<ushort> second, Vector512<ushort> mask)
-    {
-        Av1IntraPredictorBase.Widen(first.AsInt16(), out Vector512<int> first0, out Vector512<int> first1);
-        Av1IntraPredictorBase.Widen(second.AsInt16(), out Vector512<int> second0, out Vector512<int> second1);
-        Av1IntraPredictorBase.Widen(mask.AsInt16(), out Vector512<int> mask0, out Vector512<int> mask1);
-        return Av1IntraPredictorBase.Narrow(Blend(first0, second0, mask0), Blend(first1, second1, mask1)).AsUInt16();
-    }
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector128<int> Blend(Vector128<int> first, Vector128<int> second, Vector128<int> mask)
-        => ((mask * first) + ((Vector128.Create(MaximumMaskAlpha) - mask) * second) + Vector128.Create(32)) >> MaskWeightBits;
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector256<int> Blend(Vector256<int> first, Vector256<int> second, Vector256<int> mask)
-        => ((mask * first) + ((Vector256.Create(MaximumMaskAlpha) - mask) * second) + Vector256.Create(32)) >> MaskWeightBits;
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector512<int> Blend(Vector512<int> first, Vector512<int> second, Vector512<int> mask)
-        => ((mask * first) + ((Vector512.Create(MaximumMaskAlpha) - mask) * second) + Vector512.Create(32)) >> MaskWeightBits;
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector128<ushort> LoadMask128(ref byte source, int offset)
-    {
-        Vector64<byte> packed = Unsafe.As<byte, Vector64<byte>>(ref Unsafe.Add(ref source, offset));
-        return Vector128.WidenLower(Vector128.Create(packed, Vector64<byte>.Zero));
-    }
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector256<ushort> LoadMask256(ref byte source, int offset)
-    {
-        Vector128<byte> packed = Vector128.LoadUnsafe(ref source, (nuint)offset);
-        return Vector256.WidenLower(Vector256.Create(packed, Vector128<byte>.Zero));
-    }
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector512<ushort> LoadMask512(ref byte source, int offset)
-    {
-        Vector256<byte> packed = Vector256.LoadUnsafe(ref source, (nuint)offset);
-        return Vector512.WidenLower(Vector512.Create(packed, Vector256<byte>.Zero));
-    }
-}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.Intermediate.cs

@@ -1,529 +0,0 @@
-// Copyright (c) Six Labors.
-// Licensed under the Six Labors Split License.
-
-using System.Runtime.InteropServices;
-using System.Runtime.Intrinsics;
-using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling;
-
-namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
-
-/// <content>
-/// Combines high-precision compound convolution intermediates into reconstructed samples.
-/// </content>
-internal static partial class Av1CompoundInterPredictor
-{
-    /// <summary>
-    /// Combines two compound intermediates by equal averaging.
-    /// </summary>
-    public static void AverageIntermediate(
-        Span<byte> destination,
-        int destinationStride,
-        ReadOnlySpan<ushort> first,
-        int firstStride,
-        ReadOnlySpan<ushort> second,
-        int secondStride,
-        int width,
-        int height,
-        int bitDepth)
-    {
-        GetIntermediateRounding(bitDepth, out int roundBits, out int roundOffset);
-        for (int row = 0; row < height; row++)
-        {
-            Span<byte> destinationRow = destination.Slice(row * destinationStride, width);
-            ReadOnlySpan<ushort> firstRow = first.Slice(row * firstStride, width);
-            ReadOnlySpan<ushort> secondRow = second.Slice(row * secondStride, width);
-            ref byte destinationReference = ref MemoryMarshal.GetReference(destinationRow);
-            ref ushort firstReference = ref MemoryMarshal.GetReference(firstRow);
-            ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow);
-            int column = 0;
-
-            if (Vector128.IsHardwareAccelerated)
-            {
-                int vectorEnd = width - Vector128<byte>.Count;
-                for (; column <= vectorEnd; column += Vector128<byte>.Count)
-                {
-                    Vector128<ushort> first0 = Vector128.LoadUnsafe(ref firstReference, (nuint)column);
-                    Vector128<ushort> first1 = Vector128.LoadUnsafe(
-                        ref firstReference,
-                        (nuint)(column + Vector128<ushort>.Count));
-
-                    Vector128<ushort> second0 = Vector128.LoadUnsafe(ref secondReference, (nuint)column);
-                    Vector128<ushort> second1 = Vector128.LoadUnsafe(
-                        ref secondReference,
-                        (nuint)(column + Vector128<ushort>.Count));
-
-                    AverageIntermediate(first0, first1, second0, second1, roundBits, roundOffset).StoreUnsafe(
-                        ref destinationReference,
-                        (nuint)column);
-                }
-            }
-
-            for (; column < width; column++)
-            {
-                // The reference average deliberately truncates here. The sole rounding step follows bias removal,
-                // preventing the double rounding that occurs when each reference is first converted to pixels.
-                int result = ((firstRow[column] + secondRow[column]) >> 1) - roundOffset;
-                destinationRow[column] = (byte)Math.Clamp(RoundPowerOfTwo(result, roundBits), 0, byte.MaxValue);
-            }
-        }
-    }
-
-    /// <summary>
-    /// Combines two compound intermediates using the decoded temporal-distance weights.
-    /// </summary>
-    public static void DistanceWeightedIntermediate(
-        Span<byte> destination,
-        int destinationStride,
-        ReadOnlySpan<ushort> first,
-        int firstStride,
-        ReadOnlySpan<ushort> second,
-        int secondStride,
-        int width,
-        int height,
-        int firstWeight,
-        int secondWeight,
-        int bitDepth)
-    {
-        GetIntermediateRounding(bitDepth, out int roundBits, out int roundOffset);
-        for (int row = 0; row < height; row++)
-        {
-            Span<byte> destinationRow = destination.Slice(row * destinationStride, width);
-            ReadOnlySpan<ushort> firstRow = first.Slice(row * firstStride, width);
-            ReadOnlySpan<ushort> secondRow = second.Slice(row * secondStride, width);
-            ref byte destinationReference = ref MemoryMarshal.GetReference(destinationRow);
-            ref ushort firstReference = ref MemoryMarshal.GetReference(firstRow);
-            ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow);
-            int column = 0;
-
-            if (Vector128.IsHardwareAccelerated)
-            {
-                int vectorEnd = width - Vector128<byte>.Count;
-                for (; column <= vectorEnd; column += Vector128<byte>.Count)
-                {
-                    Vector128<ushort> first0 = Vector128.LoadUnsafe(ref firstReference, (nuint)column);
-                    Vector128<ushort> first1 = Vector128.LoadUnsafe(
-                        ref firstReference,
-                        (nuint)(column + Vector128<ushort>.Count));
-
-                    Vector128<ushort> second0 = Vector128.LoadUnsafe(ref secondReference, (nuint)column);
-                    Vector128<ushort> second1 = Vector128.LoadUnsafe(
-                        ref secondReference,
-                        (nuint)(column + Vector128<ushort>.Count));
-
-                    DistanceWeightedIntermediate(
-                        first0,
-                        first1,
-                        second0,
-                        second1,
-                        firstWeight,
-                        secondWeight,
-                        roundBits,
-                        roundOffset).StoreUnsafe(ref destinationReference, (nuint)column);
-                }
-            }
-
-            for (; column < width; column++)
-            {
-                int result = ((firstRow[column] * firstWeight) + (secondRow[column] * secondWeight)) >> DistanceWeightBits;
-                result -= roundOffset;
-                destinationRow[column] = (byte)Math.Clamp(RoundPowerOfTwo(result, roundBits), 0, byte.MaxValue);
-            }
-        }
-    }
-
-    /// <summary>
-    /// Fills a luma-resolution difference-weighted mask from compound intermediates.
-    /// </summary>
-    public static void FillDifferenceWeightedIntermediateMask(
-        Span<byte> mask,
-        int maskStride,
-        ReadOnlySpan<ushort> first,
-        int firstStride,
-        ReadOnlySpan<ushort> second,
-        int secondStride,
-        int width,
-        int height,
-        int bitDepth,
-        Av1DifferenceWeightedMaskType maskType)
-    {
-        bool invert = maskType == Av1DifferenceWeightedMaskType.Type38Inverse;
-        GetIntermediateRounding(bitDepth, out int roundBits, out _);
-        int differenceRound = roundBits + bitDepth - 8;
-        for (int row = 0; row < height; row++)
-        {
-            Span<byte> maskRow = mask.Slice(row * maskStride, width);
-            ReadOnlySpan<ushort> firstRow = first.Slice(row * firstStride, width);
-            ReadOnlySpan<ushort> secondRow = second.Slice(row * secondStride, width);
-            ref byte maskReference = ref MemoryMarshal.GetReference(maskRow);
-            ref ushort firstReference = ref MemoryMarshal.GetReference(firstRow);
-            ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow);
-            int column = 0;
-
-            if (Vector128.IsHardwareAccelerated)
-            {
-                int vectorEnd = width - Vector128<byte>.Count;
-                for (; column <= vectorEnd; column += Vector128<byte>.Count)
-                {
-                    Vector128<ushort> first0 = Vector128.LoadUnsafe(ref firstReference, (nuint)column);
-                    Vector128<ushort> first1 = Vector128.LoadUnsafe(
-                        ref firstReference,
-                        (nuint)(column + Vector128<ushort>.Count));
-
-                    Vector128<ushort> second0 = Vector128.LoadUnsafe(ref secondReference, (nuint)column);
-                    Vector128<ushort> second1 = Vector128.LoadUnsafe(
-                        ref secondReference,
-                        (nuint)(column + Vector128<ushort>.Count));
-
-                    DifferenceWeightedIntermediate(
-                        first0,
-                        first1,
-                        second0,
-                        second1,
-                        differenceRound,
-                        invert).StoreUnsafe(ref maskReference, (nuint)column);
-                }
-            }
-
-            for (; column < width; column++)
-            {
-                int difference = Math.Abs(firstRow[column] - secondRow[column]);
-                difference = RoundPowerOfTwo(difference, differenceRound);
-                int alpha = Math.Min(MaximumMaskAlpha, 38 + (difference >> 4));
-                maskRow[column] = (byte)(invert ? MaximumMaskAlpha - alpha : alpha);
-            }
-        }
-    }
-
-    /// <summary>
-    /// Blends two compound intermediates through a luma-resolution mask.
-    /// </summary>
-    public static void BlendIntermediate(
-        Span<byte> destination,
-        int destinationStride,
-        ReadOnlySpan<ushort> first,
-        int firstStride,
-        ReadOnlySpan<ushort> second,
-        int secondStride,
-        ReadOnlySpan<byte> mask,
-        int maskStride,
-        int width,
-        int height,
-        int subX,
-        int subY,
-        int bitDepth)
-    {
-        GetIntermediateRounding(bitDepth, out int roundBits, out int roundOffset);
-        for (int row = 0; row < height; row++)
-        {
-            Span<byte> destinationRow = destination.Slice(row * destinationStride, width);
-            ReadOnlySpan<ushort> firstRow = first.Slice(row * firstStride, width);
-            ReadOnlySpan<ushort> secondRow = second.Slice(row * secondStride, width);
-            ref byte destinationReference = ref MemoryMarshal.GetReference(destinationRow);
-            ref ushort firstReference = ref MemoryMarshal.GetReference(firstRow);
-            ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow);
-            int column = 0;
-
-            if (Vector128.IsHardwareAccelerated && subX == 0 && subY == 0)
-            {
-                ref byte maskReference = ref MemoryMarshal.GetReference(mask);
-                int maskRowOffset = row * maskStride;
-                int vectorEnd = width - Vector128<byte>.Count;
-                for (; column <= vectorEnd; column += Vector128<byte>.Count)
-                {
-                    Vector128<ushort> first0 = Vector128.LoadUnsafe(ref firstReference, (nuint)column);
-                    Vector128<ushort> first1 = Vector128.LoadUnsafe(
-                        ref firstReference,
-                        (nuint)(column + Vector128<ushort>.Count));
-
-                    Vector128<ushort> second0 = Vector128.LoadUnsafe(ref secondReference, (nuint)column);
-                    Vector128<ushort> second1 = Vector128.LoadUnsafe(
-                        ref secondReference,
-                        (nuint)(column + Vector128<ushort>.Count));
-
-                    Vector128<byte> alpha = Vector128.LoadUnsafe(
-                        ref maskReference,
-                        (nuint)(maskRowOffset + column));
-
-                    BlendIntermediate(
-                        first0,
-                        first1,
-                        second0,
-                        second1,
-                        alpha,
-                        roundBits,
-                        roundOffset).StoreUnsafe(ref destinationReference, (nuint)column);
-                }
-            }
-
-            for (; column < width; column++)
-            {
-                int alpha = GetSubsampledMaskAlpha(mask, maskStride, row, column, subX, subY);
-
-                // Mask blending also truncates its Q6 result because final pixel rounding is still pending. Adding
-                // a half-unit here would produce a second rounding step and diverge from pinned libaom.
-                int result = ((alpha * firstRow[column]) + ((MaximumMaskAlpha - alpha) * secondRow[column])) >> MaskWeightBits;
-                result -= roundOffset;
-                destinationRow[column] = (byte)Math.Clamp(RoundPowerOfTwo(result, roundBits), 0, byte.MaxValue);
-            }
-        }
-    }
-
-    /// <summary>
-    /// Equal-averages sixteen compound lanes and converts them to final 8-bit samples.
-    /// </summary>
-    private static Vector128<byte> AverageIntermediate(
-        Vector128<ushort> first0,
-        Vector128<ushort> first1,
-        Vector128<ushort> second0,
-        Vector128<ushort> second1,
-        int roundBits,
-        int roundOffset)
-        => Vector128.Narrow(
-            FinalizeIntermediate(
-                (first0 & second0) + ((first0 ^ second0) >> 1),
-                roundBits,
-                roundOffset),
-            FinalizeIntermediate(
-                (first1 & second1) + ((first1 ^ second1) >> 1),
-                roundBits,
-                roundOffset));
-
-    /// <summary>
-    /// Distance-weights sixteen compound lanes and converts them to final 8-bit samples.
-    /// </summary>
-    private static Vector128<byte> DistanceWeightedIntermediate(
-        Vector128<ushort> first0,
-        Vector128<ushort> first1,
-        Vector128<ushort> second0,
-        Vector128<ushort> second1,
-        int firstWeight,
-        int secondWeight,
-        int roundBits,
-        int roundOffset)
-        => Vector128.Narrow(
-            DistanceWeightedIntermediate(first0, second0, firstWeight, secondWeight, roundBits, roundOffset),
-            DistanceWeightedIntermediate(first1, second1, firstWeight, secondWeight, roundBits, roundOffset));
-
-    /// <summary>
-    /// Distance-weights eight compound lanes without overflowing the unsigned intermediate range.
-    /// </summary>
-    private static Vector128<ushort> DistanceWeightedIntermediate(
-        Vector128<ushort> first,
-        Vector128<ushort> second,
-        int firstWeight,
-        int secondWeight,
-        int roundBits,
-        int roundOffset)
-    {
-        Vector128<int> firstLower = Vector128.WidenLower(first).AsInt32();
-        Vector128<int> firstUpper = Vector128.WidenUpper(first).AsInt32();
-        Vector128<int> secondLower = Vector128.WidenLower(second).AsInt32();
-        Vector128<int> secondUpper = Vector128.WidenUpper(second).AsInt32();
-        Vector128<int> lower =
-            ((firstLower * firstWeight) + (secondLower * secondWeight)) >> DistanceWeightBits;
-
-        Vector128<int> upper =
-            ((firstUpper * firstWeight) + (secondUpper * secondWeight)) >> DistanceWeightBits;
-
-        return Vector128.Narrow(
-            FinalizeIntermediate(lower, roundBits, roundOffset),
-            FinalizeIntermediate(upper, roundBits, roundOffset)).AsUInt16();
-    }
-
-    /// <summary>
-    /// Creates sixteen difference-weighted mask values from compound intermediates.
-    /// </summary>
-    private static Vector128<byte> DifferenceWeightedIntermediate(
-        Vector128<ushort> first0,
-        Vector128<ushort> first1,
-        Vector128<ushort> second0,
-        Vector128<ushort> second1,
-        int differenceRound,
-        bool invert)
-        => Vector128.Narrow(
-            DifferenceWeightedIntermediate(first0, second0, differenceRound, invert),
-            DifferenceWeightedIntermediate(first1, second1, differenceRound, invert));
-
-    /// <summary>
-    /// Creates eight difference-weighted mask values without losing the required pre-alpha rounding.
-    /// </summary>
-    private static Vector128<ushort> DifferenceWeightedIntermediate(
-        Vector128<ushort> first,
-        Vector128<ushort> second,
-        int differenceRound,
-        bool invert)
-    {
-        Vector128<ushort> difference = Vector128.Max(first, second) - Vector128.Min(first, second);
-        Vector128<int> lower = DifferenceWeightedIntermediate(
-            Vector128.WidenLower(difference).AsInt32(),
-            differenceRound,
-            invert);
-
-        Vector128<int> upper = DifferenceWeightedIntermediate(
-            Vector128.WidenUpper(difference).AsInt32(),
-            differenceRound,
-            invert);
-
-        return Vector128.Narrow(lower, upper).AsUInt16();
-    }
-
-    /// <summary>
-    /// Converts four intermediate differences to the decoded type-38 mask range.
-    /// </summary>
-    private static Vector128<int> DifferenceWeightedIntermediate(
-        Vector128<int> difference,
-        int differenceRound,
-        bool invert)
-    {
-        if (differenceRound != 0)
-        {
-            difference = (difference + Vector128.Create(1 << (differenceRound - 1))) >> differenceRound;
-        }
-
-        Vector128<int> maximum = Vector128.Create(MaximumMaskAlpha);
-        Vector128<int> alpha = Vector128.Min(maximum, (difference >> 4) + Vector128.Create(38));
-        return invert ? maximum - alpha : alpha;
-    }
-
-    /// <summary>
-    /// Mask-blends sixteen compound lanes and converts them to final 8-bit samples.
-    /// </summary>
-    private static Vector128<byte> BlendIntermediate(
-        Vector128<ushort> first0,
-        Vector128<ushort> first1,
-        Vector128<ushort> second0,
-        Vector128<ushort> second1,
-        Vector128<byte> alpha,
-        int roundBits,
-        int roundOffset)
-        => Vector128.Narrow(
-            BlendIntermediate(
-                first0,
-                second0,
-                Vector128.WidenLower(alpha),
-                roundBits,
-                roundOffset),
-            BlendIntermediate(
-                first1,
-                second1,
-                Vector128.WidenUpper(alpha),
-                roundBits,
-                roundOffset));
-
-    /// <summary>
-    /// Mask-blends eight compound lanes after widening every product to signed 32-bit precision.
-    /// </summary>
-    private static Vector128<ushort> BlendIntermediate(
-        Vector128<ushort> first,
-        Vector128<ushort> second,
-        Vector128<ushort> alpha,
-        int roundBits,
-        int roundOffset)
-    {
-        Vector128<int> firstLower = Vector128.WidenLower(first).AsInt32();
-        Vector128<int> firstUpper = Vector128.WidenUpper(first).AsInt32();
-        Vector128<int> secondLower = Vector128.WidenLower(second).AsInt32();
-        Vector128<int> secondUpper = Vector128.WidenUpper(second).AsInt32();
-        Vector128<int> alphaLower = Vector128.WidenLower(alpha).AsInt32();
-        Vector128<int> alphaUpper = Vector128.WidenUpper(alpha).AsInt32();
-        Vector128<int> maximum = Vector128.Create(MaximumMaskAlpha);
-        Vector128<int> lower =
-            ((alphaLower * firstLower) + ((maximum - alphaLower) * secondLower)) >> MaskWeightBits;
-
-        Vector128<int> upper =
-            ((alphaUpper * firstUpper) + ((maximum - alphaUpper) * secondUpper)) >> MaskWeightBits;
-
-        return Vector128.Narrow(
-            FinalizeIntermediate(lower, roundBits, roundOffset),
-            FinalizeIntermediate(upper, roundBits, roundOffset)).AsUInt16();
-    }
-
-    /// <summary>
-    /// Removes the compound bias and final fractional precision from eight unsigned lanes.
-    /// </summary>
-    private static Vector128<ushort> FinalizeIntermediate(
-        Vector128<ushort> value,
-        int roundBits,
-        int roundOffset)
-    {
-        Vector128<short> result = (value - Vector128.Create((ushort)roundOffset)).AsInt16();
-        if (roundBits != 0)
-        {
-            result = (result + Vector128.Create((short)(1 << (roundBits - 1)))) >> roundBits;
-        }
-
-        result = Vector128.Max(Vector128<short>.Zero, Vector128.Min(Vector128.Create((short)byte.MaxValue), result));
-        return result.AsUInt16();
-    }
-
-    /// <summary>
-    /// Removes the compound bias and final fractional precision from four widened lanes.
-    /// </summary>
-    private static Vector128<int> FinalizeIntermediate(
-        Vector128<int> value,
-        int roundBits,
-        int roundOffset)
-    {
-        Vector128<int> result = value - Vector128.Create(roundOffset);
-        if (roundBits != 0)
-        {
-            result = (result + Vector128.Create(1 << (roundBits - 1))) >> roundBits;
-        }
-
-        return Vector128.Max(Vector128<int>.Zero, Vector128.Min(Vector128.Create((int)byte.MaxValue), result));
-    }
-
-    /// <summary>
-    /// Gets the mask alpha for one plane sample, averaging its two or four luma samples when required.
-    /// </summary>
-    private static int GetSubsampledMaskAlpha(
-        ReadOnlySpan<byte> mask,
-        int maskStride,
-        int row,
-        int column,
-        int subX,
-        int subY)
-    {
-        int maskRow = row << subY;
-        int maskColumn = column << subX;
-        int alpha = mask[(maskRow * maskStride) + maskColumn];
-        if (subX != 0)
-        {
-            alpha += mask[(maskRow * maskStride) + maskColumn + 1];
-        }
-
-        if (subY != 0)
-        {
-            int lowerOffset = ((maskRow + 1) * maskStride) + maskColumn;
-            alpha += mask[lowerOffset];
-            if (subX != 0)
-            {
-                alpha += mask[lowerOffset + 1];
-            }
-        }
-
-        int sampleCountShift = subX + subY;
-        return sampleCountShift == 0
-            ? alpha
-            : RoundPowerOfTwo(alpha, sampleCountShift);
-    }
-
-    /// <summary>
-    /// Derives the bias and remaining fractional precision of a compound intermediate.
-    /// </summary>
-    private static void GetIntermediateRounding(int bitDepth, out int roundBits, out int roundOffset)
-    {
-        int intermediateRange = bitDepth + 7 - 3 + 2;
-        int round0 = 3 + Math.Max(intermediateRange - 16, 0);
-        int offsetBits = bitDepth + 14 - round0;
-        roundBits = 14 - round0 - Av1InterPredictor.CompoundRound1Bits;
-        roundOffset = (1 << (offsetBits - Av1InterPredictor.CompoundRound1Bits)) +
-            (1 << (offsetBits - Av1InterPredictor.CompoundRound1Bits - 1));
-    }
-
-    /// <summary>
-    /// Applies AV1's positive power-of-two rounding rule.
-    /// </summary>
-    private static int RoundPowerOfTwo(int value, int bits)
-        => bits == 0 ? value : (value + (1 << (bits - 1))) >> bits;
-}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.Masks.cs

@@ -1,294 +0,0 @@
-// Copyright (c) Six Labors.
-// Licensed under the Six Labors Split License.
-
-using System.Runtime.CompilerServices;
-using System.Runtime.InteropServices;
-using System.Runtime.Intrinsics;
-using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling;
-
-namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
-
-/// <content>
-/// Produces the smooth inter-intra and predictor-difference masks used by compound blending.
-/// </content>
-internal static partial class Av1CompoundInterPredictor
-{
-    /// <summary>
-    /// Gets libaom's one-dimensional inter-intra alpha curve.
-    /// </summary>
-    private static ReadOnlySpan<byte> InterIntraWeights =>
-    [
-        60, 58, 56, 54, 52, 50, 48, 47, 45, 44, 42, 41, 39, 38, 37, 35,
-        34, 33, 32, 31, 30, 29, 28, 27, 26, 25, 24, 23, 22, 22, 21, 20,
-        19, 19, 18, 18, 17, 16, 16, 15, 15, 14, 14, 13, 13, 12, 12, 12,
-        11, 11, 10, 10, 10, 9, 9, 9, 8, 8, 8, 8, 7, 7, 7, 7,
-        6, 6, 6, 6, 6, 5, 5, 5, 5, 5, 4, 4, 4, 4, 4, 4,
-        4, 4, 4, 4, 3, 3, 3, 3, 3, 3, 3, 3, 3, 2, 2, 2,
-        2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1,
-        1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
-    ];
-
-    /// <summary>
-    /// Fills a smooth inter-intra mask for one plane.
-    /// </summary>
-    public static void FillInterIntraMask(
-        Span<byte> mask,
-        int maskStride,
-        int width,
-        int height,
-        Av1InterIntraMode mode,
-        bool invert)
-    {
-        int sizeScale = 128 / Math.Max(width, height);
-        for (int row = 0; row < height; row++)
-        {
-            Span<byte> maskRow = mask.Slice(row * maskStride, width);
-            for (int column = 0; column < width; column++)
-            {
-                int alpha = mode switch
-                {
-                    Av1InterIntraMode.Vertical => InterIntraWeights[row * sizeScale],
-                    Av1InterIntraMode.Horizontal => InterIntraWeights[column * sizeScale],
-                    Av1InterIntraMode.Smooth => InterIntraWeights[Math.Min(row, column) * sizeScale],
-                    _ => 32,
-                };
-
-                maskRow[column] = (byte)(invert ? MaximumMaskAlpha - alpha : alpha);
-            }
-        }
-    }
-
-    /// <summary>
-    /// Fills an 8-bit difference-weighted compound mask.
-    /// </summary>
-    public static void FillDifferenceWeightedMask(
-        Span<byte> mask,
-        int maskStride,
-        ReadOnlySpan<byte> first,
-        int firstStride,
-        ReadOnlySpan<byte> second,
-        int secondStride,
-        int width,
-        int height,
-        Av1DifferenceWeightedMaskType maskType)
-    {
-        bool invert = maskType == Av1DifferenceWeightedMaskType.Type38Inverse;
-        for (int row = 0; row < height; row++)
-        {
-            Span<byte> maskRow = mask.Slice(row * maskStride, width);
-            ReadOnlySpan<byte> firstRow = first.Slice(row * firstStride, width);
-            ReadOnlySpan<byte> secondRow = second.Slice(row * secondStride, width);
-            ref byte maskReference = ref MemoryMarshal.GetReference(maskRow);
-            ref byte firstReference = ref MemoryMarshal.GetReference(firstRow);
-            ref byte secondReference = ref MemoryMarshal.GetReference(secondRow);
-            int column = 0;
-
-            if (Vector512.IsHardwareAccelerated)
-            {
-                int vectorEnd = width - Vector512<byte>.Count;
-                for (; column <= vectorEnd; column += Vector512<byte>.Count)
-                {
-                    Vector512<byte> firstVector = Vector512.LoadUnsafe(ref firstReference, (nuint)column);
-                    Vector512<byte> secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column);
-                    DifferenceWeighted(firstVector, secondVector, 4, invert).StoreUnsafe(ref maskReference, (nuint)column);
-                }
-            }
-
-            if (Vector256.IsHardwareAccelerated)
-            {
-                int vectorEnd = width - Vector256<byte>.Count;
-                for (; column <= vectorEnd; column += Vector256<byte>.Count)
-                {
-                    Vector256<byte> firstVector = Vector256.LoadUnsafe(ref firstReference, (nuint)column);
-                    Vector256<byte> secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column);
-                    DifferenceWeighted(firstVector, secondVector, 4, invert).StoreUnsafe(ref maskReference, (nuint)column);
-                }
-            }
-
-            if (Vector128.IsHardwareAccelerated)
-            {
-                int vectorEnd = width - Vector128<byte>.Count;
-                for (; column <= vectorEnd; column += Vector128<byte>.Count)
-                {
-                    Vector128<byte> firstVector = Vector128.LoadUnsafe(ref firstReference, (nuint)column);
-                    Vector128<byte> secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column);
-                    DifferenceWeighted(firstVector, secondVector, 4, invert).StoreUnsafe(ref maskReference, (nuint)column);
-                }
-            }
-
-            for (; column < width; column++)
-            {
-                int difference = Math.Abs(firstRow[column] - secondRow[column]) >> 4;
-                int alpha = Math.Min(MaximumMaskAlpha, 38 + difference);
-                maskRow[column] = (byte)(invert ? MaximumMaskAlpha - alpha : alpha);
-            }
-        }
-    }
-
-    /// <summary>
-    /// Fills a high-bit-depth difference-weighted compound mask.
-    /// </summary>
-    public static void FillDifferenceWeightedMask(
-        Span<byte> mask,
-        int maskStride,
-        ReadOnlySpan<ushort> first,
-        int firstStride,
-        ReadOnlySpan<ushort> second,
-        int secondStride,
-        int width,
-        int height,
-        int bitDepth,
-        Av1DifferenceWeightedMaskType maskType)
-    {
-        bool invert = maskType == Av1DifferenceWeightedMaskType.Type38Inverse;
-        int differenceShift = bitDepth - 8 + 4;
-        for (int row = 0; row < height; row++)
-        {
-            Span<byte> maskRow = mask.Slice(row * maskStride, width);
-            ReadOnlySpan<ushort> firstRow = first.Slice(row * firstStride, width);
-            ReadOnlySpan<ushort> secondRow = second.Slice(row * secondStride, width);
-            ref byte maskReference = ref MemoryMarshal.GetReference(maskRow);
-            ref ushort firstReference = ref MemoryMarshal.GetReference(firstRow);
-            ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow);
-            int column = 0;
-
-            // Two input vectors narrow to one packed byte mask. This keeps mask construction contiguous and avoids
-            // temporary buffers before the following vector blend consumes the complete plane block.
-            if (Vector512.IsHardwareAccelerated)
-            {
-                int vectorEnd = width - Vector512<byte>.Count;
-                for (; column <= vectorEnd; column += Vector512<byte>.Count)
-                {
-                    Vector512<ushort> first0 = Vector512.LoadUnsafe(ref firstReference, (nuint)column);
-                    Vector512<ushort> first1 = Vector512.LoadUnsafe(ref firstReference, (nuint)(column + Vector512<ushort>.Count));
-                    Vector512<ushort> second0 = Vector512.LoadUnsafe(ref secondReference, (nuint)column);
-                    Vector512<ushort> second1 = Vector512.LoadUnsafe(ref secondReference, (nuint)(column + Vector512<ushort>.Count));
-                    DifferenceWeighted(first0, first1, second0, second1, differenceShift, invert)
-                        .StoreUnsafe(ref maskReference, (nuint)column);
-                }
-            }
-
-            if (Vector256.IsHardwareAccelerated)
-            {
-                int vectorEnd = width - Vector256<byte>.Count;
-                for (; column <= vectorEnd; column += Vector256<byte>.Count)
-                {
-                    Vector256<ushort> first0 = Vector256.LoadUnsafe(ref firstReference, (nuint)column);
-                    Vector256<ushort> first1 = Vector256.LoadUnsafe(ref firstReference, (nuint)(column + Vector256<ushort>.Count));
-                    Vector256<ushort> second0 = Vector256.LoadUnsafe(ref secondReference, (nuint)column);
-                    Vector256<ushort> second1 = Vector256.LoadUnsafe(ref secondReference, (nuint)(column + Vector256<ushort>.Count));
-                    DifferenceWeighted(first0, first1, second0, second1, differenceShift, invert)
-                        .StoreUnsafe(ref maskReference, (nuint)column);
-                }
-            }
-
-            if (Vector128.IsHardwareAccelerated)
-            {
-                int vectorEnd = width - Vector128<byte>.Count;
-                for (; column <= vectorEnd; column += Vector128<byte>.Count)
-                {
-                    Vector128<ushort> first0 = Vector128.LoadUnsafe(ref firstReference, (nuint)column);
-                    Vector128<ushort> first1 = Vector128.LoadUnsafe(ref firstReference, (nuint)(column + Vector128<ushort>.Count));
-                    Vector128<ushort> second0 = Vector128.LoadUnsafe(ref secondReference, (nuint)column);
-                    Vector128<ushort> second1 = Vector128.LoadUnsafe(ref secondReference, (nuint)(column + Vector128<ushort>.Count));
-                    DifferenceWeighted(first0, first1, second0, second1, differenceShift, invert)
-                        .StoreUnsafe(ref maskReference, (nuint)column);
-                }
-            }
-
-            for (; column < width; column++)
-            {
-                int difference = Math.Abs(firstRow[column] - secondRow[column]) >> differenceShift;
-                int alpha = Math.Min(MaximumMaskAlpha, 38 + difference);
-                maskRow[column] = (byte)(invert ? MaximumMaskAlpha - alpha : alpha);
-            }
-        }
-    }
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector128<byte> DifferenceWeighted(Vector128<byte> first, Vector128<byte> second, int shift, bool invert)
-    {
-        Vector128<byte> difference = Vector128.Max(first, second) - Vector128.Min(first, second);
-        Vector128<ushort> lower = DifferenceWeightedAlpha(Vector128.WidenLower(difference), shift, invert);
-        Vector128<ushort> upper = DifferenceWeightedAlpha(Vector128.WidenUpper(difference), shift, invert);
-        return Vector128.Narrow(lower, upper);
-    }
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector256<byte> DifferenceWeighted(Vector256<byte> first, Vector256<byte> second, int shift, bool invert)
-    {
-        Vector256<byte> difference = Vector256.Max(first, second) - Vector256.Min(first, second);
-        Vector256<ushort> lower = DifferenceWeightedAlpha(Vector256.WidenLower(difference), shift, invert);
-        Vector256<ushort> upper = DifferenceWeightedAlpha(Vector256.WidenUpper(difference), shift, invert);
-        return Vector256.Narrow(lower, upper);
-    }
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector512<byte> DifferenceWeighted(Vector512<byte> first, Vector512<byte> second, int shift, bool invert)
-    {
-        Vector512<byte> difference = Vector512.Max(first, second) - Vector512.Min(first, second);
-        Vector512<ushort> lower = DifferenceWeightedAlpha(Vector512.WidenLower(difference), shift, invert);
-        Vector512<ushort> upper = DifferenceWeightedAlpha(Vector512.WidenUpper(difference), shift, invert);
-        return Vector512.Narrow(lower, upper);
-    }
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector128<byte> DifferenceWeighted(
-        Vector128<ushort> first0,
-        Vector128<ushort> first1,
-        Vector128<ushort> second0,
-        Vector128<ushort> second1,
-        int shift,
-        bool invert)
-        => Vector128.Narrow(
-            DifferenceWeightedAlpha(Vector128.Max(first0, second0) - Vector128.Min(first0, second0), shift, invert),
-            DifferenceWeightedAlpha(Vector128.Max(first1, second1) - Vector128.Min(first1, second1), shift, invert));
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector256<byte> DifferenceWeighted(
-        Vector256<ushort> first0,
-        Vector256<ushort> first1,
-        Vector256<ushort> second0,
-        Vector256<ushort> second1,
-        int shift,
-        bool invert)
-        => Vector256.Narrow(
-            DifferenceWeightedAlpha(Vector256.Max(first0, second0) - Vector256.Min(first0, second0), shift, invert),
-            DifferenceWeightedAlpha(Vector256.Max(first1, second1) - Vector256.Min(first1, second1), shift, invert));
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector512<byte> DifferenceWeighted(
-        Vector512<ushort> first0,
-        Vector512<ushort> first1,
-        Vector512<ushort> second0,
-        Vector512<ushort> second1,
-        int shift,
-        bool invert)
-        => Vector512.Narrow(
-            DifferenceWeightedAlpha(Vector512.Max(first0, second0) - Vector512.Min(first0, second0), shift, invert),
-            DifferenceWeightedAlpha(Vector512.Max(first1, second1) - Vector512.Min(first1, second1), shift, invert));
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector128<ushort> DifferenceWeightedAlpha(Vector128<ushort> difference, int shift, bool invert)
-    {
-        Vector128<ushort> maximum = Vector128.Create((ushort)MaximumMaskAlpha);
-        Vector128<ushort> alpha = Vector128.Min(maximum, (difference >> shift) + Vector128.Create((ushort)38));
-        return invert ? maximum - alpha : alpha;
-    }
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector256<ushort> DifferenceWeightedAlpha(Vector256<ushort> difference, int shift, bool invert)
-    {
-        Vector256<ushort> maximum = Vector256.Create((ushort)MaximumMaskAlpha);
-        Vector256<ushort> alpha = Vector256.Min(maximum, (difference >> shift) + Vector256.Create((ushort)38));
-        return invert ? maximum - alpha : alpha;
-    }
-
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector512<ushort> DifferenceWeightedAlpha(Vector512<ushort> difference, int shift, bool invert)
-    {
-        Vector512<ushort> maximum = Vector512.Create((ushort)MaximumMaskAlpha);
-        Vector512<ushort> alpha = Vector512.Min(maximum, (difference >> shift) + Vector512.Create((ushort)38));
-        return invert ? maximum - alpha : alpha;
-    }
-}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.Operator.cs

@@ -0,0 +1,344 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.CompilerServices;
+using System.Runtime.Intrinsics;
+
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
+
+/// <content>
+/// Defines biased compound-prediction conversion arithmetic.
+/// </content>
+internal static partial class Av1CompoundInterPredictor
+{
+    /// <summary>
+    /// Defines biased compound-prediction conversion for scalar and SIMD lane groups.
+    /// </summary>
+    private interface IAv1CompoundPredictionOperator
+    {
+        /// <summary>
+        /// Converts one integer-position sample to the compound intermediate representation.
+        /// </summary>
+        /// <param name="sample">The source sample.</param>
+        /// <param name="roundBits">The final reconstruction shift.</param>
+        /// <param name="roundOffset">The compound intermediate bias.</param>
+        /// <returns>The biased compound intermediate.</returns>
+        public static abstract ushort Copy(byte sample, int roundBits, int roundOffset);
+
+        /// <summary>
+        /// Converts 128 bits of integer-position samples to compound intermediates.
+        /// </summary>
+        /// <param name="samples">The source samples.</param>
+        /// <param name="roundBits">The final reconstruction shift.</param>
+        /// <param name="roundOffset">The compound intermediate bias.</param>
+        /// <param name="lower">Receives the lower widened intermediates.</param>
+        /// <param name="upper">Receives the upper widened intermediates.</param>
+        public static abstract void Copy(
+            Vector128<byte> samples,
+            int roundBits,
+            int roundOffset,
+            out Vector128<ushort> lower,
+            out Vector128<ushort> upper);
+
+        /// <summary>
+        /// Converts 256 bits of integer-position samples to compound intermediates.
+        /// </summary>
+        /// <param name="samples">The source samples.</param>
+        /// <param name="roundBits">The final reconstruction shift.</param>
+        /// <param name="roundOffset">The compound intermediate bias.</param>
+        /// <param name="lower">Receives the lower widened intermediates.</param>
+        /// <param name="upper">Receives the upper widened intermediates.</param>
+        public static abstract void Copy(
+            Vector256<byte> samples,
+            int roundBits,
+            int roundOffset,
+            out Vector256<ushort> lower,
+            out Vector256<ushort> upper);
+
+        /// <summary>
+        /// Converts 512 bits of integer-position samples to compound intermediates.
+        /// </summary>
+        /// <param name="samples">The source samples.</param>
+        /// <param name="roundBits">The final reconstruction shift.</param>
+        /// <param name="roundOffset">The compound intermediate bias.</param>
+        /// <param name="lower">Receives the lower widened intermediates.</param>
+        /// <param name="upper">Receives the upper widened intermediates.</param>
+        public static abstract void Copy(
+            Vector512<byte> samples,
+            int roundBits,
+            int roundOffset,
+            out Vector512<ushort> lower,
+            out Vector512<ushort> upper);
+
+        /// <summary>
+        /// Applies direct-filter rounding and bias to one convolution result.
+        /// </summary>
+        /// <param name="result">The convolution result.</param>
+        /// <param name="preShift">The shift applied before rounding.</param>
+        /// <param name="round">The rounding shift.</param>
+        /// <param name="roundOffset">The compound intermediate bias.</param>
+        /// <returns>The biased compound intermediate.</returns>
+        public static abstract ushort PrepareDirect(int result, int preShift, int round, int roundOffset);
+
+        /// <summary>
+        /// Applies direct-filter rounding and bias to 128-bit widened convolution results.
+        /// </summary>
+        /// <param name="lower">The lower convolution results.</param>
+        /// <param name="upper">The upper convolution results.</param>
+        /// <param name="preShift">The shift applied before rounding.</param>
+        /// <param name="round">The rounding shift.</param>
+        /// <param name="roundOffset">The compound intermediate bias.</param>
+        /// <returns>The biased compound intermediates.</returns>
+        public static abstract Vector128<ushort> PrepareDirect(
+            Vector128<int> lower,
+            Vector128<int> upper,
+            int preShift,
+            int round,
+            int roundOffset);
+
+        /// <summary>
+        /// Applies direct-filter rounding and bias to 256-bit widened convolution results.
+        /// </summary>
+        /// <param name="lower">The lower convolution results.</param>
+        /// <param name="upper">The upper convolution results.</param>
+        /// <param name="preShift">The shift applied before rounding.</param>
+        /// <param name="round">The rounding shift.</param>
+        /// <param name="roundOffset">The compound intermediate bias.</param>
+        /// <returns>The biased compound intermediates.</returns>
+        public static abstract Vector256<ushort> PrepareDirect(
+            Vector256<int> lower,
+            Vector256<int> upper,
+            int preShift,
+            int round,
+            int roundOffset);
+
+        /// <summary>
+        /// Applies direct-filter rounding and bias to 512-bit widened convolution results.
+        /// </summary>
+        /// <param name="lower">The lower convolution results.</param>
+        /// <param name="upper">The upper convolution results.</param>
+        /// <param name="preShift">The shift applied before rounding.</param>
+        /// <param name="round">The rounding shift.</param>
+        /// <param name="roundOffset">The compound intermediate bias.</param>
+        /// <returns>The biased compound intermediates.</returns>
+        public static abstract Vector512<ushort> PrepareDirect(
+            Vector512<int> lower,
+            Vector512<int> upper,
+            int preShift,
+            int round,
+            int roundOffset);
+
+        /// <summary>
+        /// Applies first-pass compound rounding to one biased horizontal convolution result.
+        /// </summary>
+        /// <param name="result">The biased horizontal convolution result.</param>
+        /// <returns>The rounded intermediate.</returns>
+        public static abstract short PrepareHorizontal(int result);
+
+        /// <summary>
+        /// Applies first-pass compound rounding to 128-bit widened horizontal convolution results.
+        /// </summary>
+        /// <param name="lower">The lower convolution results.</param>
+        /// <param name="upper">The upper convolution results.</param>
+        /// <returns>The rounded intermediates.</returns>
+        public static abstract Vector128<short> PrepareHorizontal(Vector128<int> lower, Vector128<int> upper);
+
+        /// <summary>
+        /// Applies first-pass compound rounding to 256-bit widened horizontal convolution results.
+        /// </summary>
+        /// <param name="lower">The lower convolution results.</param>
+        /// <param name="upper">The upper convolution results.</param>
+        /// <returns>The rounded intermediates.</returns>
+        public static abstract Vector256<short> PrepareHorizontal(Vector256<int> lower, Vector256<int> upper);
+
+        /// <summary>
+        /// Applies first-pass compound rounding to 512-bit widened horizontal convolution results.
+        /// </summary>
+        /// <param name="lower">The lower convolution results.</param>
+        /// <param name="upper">The upper convolution results.</param>
+        /// <returns>The rounded intermediates.</returns>
+        public static abstract Vector512<short> PrepareHorizontal(Vector512<int> lower, Vector512<int> upper);
+
+        /// <summary>
+        /// Applies second-pass compound rounding to one biased vertical convolution result.
+        /// </summary>
+        /// <param name="result">The biased vertical convolution result.</param>
+        /// <returns>The compound intermediate.</returns>
+        public static abstract ushort PrepareVertical(int result);
+
+        /// <summary>
+        /// Applies second-pass compound rounding to 128-bit widened vertical convolution results.
+        /// </summary>
+        /// <param name="lower">The lower convolution results.</param>
+        /// <param name="upper">The upper convolution results.</param>
+        /// <returns>The compound intermediates.</returns>
+        public static abstract Vector128<ushort> PrepareVertical(Vector128<int> lower, Vector128<int> upper);
+
+        /// <summary>
+        /// Applies second-pass compound rounding to 256-bit widened vertical convolution results.
+        /// </summary>
+        /// <param name="lower">The lower convolution results.</param>
+        /// <param name="upper">The upper convolution results.</param>
+        /// <returns>The compound intermediates.</returns>
+        public static abstract Vector256<ushort> PrepareVertical(Vector256<int> lower, Vector256<int> upper);
+
+        /// <summary>
+        /// Applies second-pass compound rounding to 512-bit widened vertical convolution results.
+        /// </summary>
+        /// <param name="lower">The lower convolution results.</param>
+        /// <param name="upper">The upper convolution results.</param>
+        /// <returns>The compound intermediates.</returns>
+        public static abstract Vector512<ushort> PrepareVertical(Vector512<int> lower, Vector512<int> upper);
+    }
+
+    /// <summary>
+    /// Implements AV1 compound-prediction conversion for scalar and SIMD lane groups.
+    /// </summary>
+    private readonly struct CompoundPredictionOperator : IAv1CompoundPredictionOperator
+    {
+        private const int HorizontalBias = 1 << (8 + FilterBits - 1);
+        private const int VerticalBias = 1 << (8 + (2 * FilterBits) - Round0Bits);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static ushort Copy(byte sample, int roundBits, int roundOffset)
+            => (ushort)((sample << roundBits) + roundOffset);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static void Copy(
+            Vector128<byte> samples,
+            int roundBits,
+            int roundOffset,
+            out Vector128<ushort> lower,
+            out Vector128<ushort> upper)
+        {
+            Vector128<ushort> offset = Vector128.Create((ushort)roundOffset);
+            lower = (Vector128.WidenLower(samples) << roundBits) + offset;
+            upper = (Vector128.WidenUpper(samples) << roundBits) + offset;
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static void Copy(
+            Vector256<byte> samples,
+            int roundBits,
+            int roundOffset,
+            out Vector256<ushort> lower,
+            out Vector256<ushort> upper)
+        {
+            Vector256<ushort> offset = Vector256.Create((ushort)roundOffset);
+            lower = (Vector256.WidenLower(samples) << roundBits) + offset;
+            upper = (Vector256.WidenUpper(samples) << roundBits) + offset;
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static void Copy(
+            Vector512<byte> samples,
+            int roundBits,
+            int roundOffset,
+            out Vector512<ushort> lower,
+            out Vector512<ushort> upper)
+        {
+            Vector512<ushort> offset = Vector512.Create((ushort)roundOffset);
+            lower = (Vector512.WidenLower(samples) << roundBits) + offset;
+            upper = (Vector512.WidenUpper(samples) << roundBits) + offset;
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static ushort PrepareDirect(int result, int preShift, int round, int roundOffset)
+            => (ushort)(RoundPowerOfTwo(result << preShift, round) + roundOffset);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector128<ushort> PrepareDirect(
+            Vector128<int> lower,
+            Vector128<int> upper,
+            int preShift,
+            int round,
+            int roundOffset)
+            => Av1IntraPredictorBase.Narrow(
+                RoundPowerOfTwo(lower << preShift, round) + Vector128.Create(roundOffset),
+                RoundPowerOfTwo(upper << preShift, round) + Vector128.Create(roundOffset)).AsUInt16();
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector256<ushort> PrepareDirect(
+            Vector256<int> lower,
+            Vector256<int> upper,
+            int preShift,
+            int round,
+            int roundOffset)
+            => Av1IntraPredictorBase.Narrow(
+                RoundPowerOfTwo(lower << preShift, round) + Vector256.Create(roundOffset),
+                RoundPowerOfTwo(upper << preShift, round) + Vector256.Create(roundOffset)).AsUInt16();
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector512<ushort> PrepareDirect(
+            Vector512<int> lower,
+            Vector512<int> upper,
+            int preShift,
+            int round,
+            int roundOffset)
+            => Av1IntraPredictorBase.Narrow(
+                RoundPowerOfTwo(lower << preShift, round) + Vector512.Create(roundOffset),
+                RoundPowerOfTwo(upper << preShift, round) + Vector512.Create(roundOffset)).AsUInt16();
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static short PrepareHorizontal(int result)
+            => (short)RoundPowerOfTwo(HorizontalBias + result, Round0Bits);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector128<short> PrepareHorizontal(Vector128<int> lower, Vector128<int> upper)
+            => Av1IntraPredictorBase.Narrow(
+                RoundPowerOfTwo(lower + Vector128.Create(HorizontalBias), Round0Bits),
+                RoundPowerOfTwo(upper + Vector128.Create(HorizontalBias), Round0Bits));
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector256<short> PrepareHorizontal(Vector256<int> lower, Vector256<int> upper)
+            => Av1IntraPredictorBase.Narrow(
+                RoundPowerOfTwo(lower + Vector256.Create(HorizontalBias), Round0Bits),
+                RoundPowerOfTwo(upper + Vector256.Create(HorizontalBias), Round0Bits));
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector512<short> PrepareHorizontal(Vector512<int> lower, Vector512<int> upper)
+            => Av1IntraPredictorBase.Narrow(
+                RoundPowerOfTwo(lower + Vector512.Create(HorizontalBias), Round0Bits),
+                RoundPowerOfTwo(upper + Vector512.Create(HorizontalBias), Round0Bits));
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static ushort PrepareVertical(int result)
+            => (ushort)RoundPowerOfTwo(VerticalBias + result, CompoundRound1Bits);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector128<ushort> PrepareVertical(Vector128<int> lower, Vector128<int> upper)
+            => Av1IntraPredictorBase.Narrow(
+                RoundPowerOfTwo(lower + Vector128.Create(VerticalBias), CompoundRound1Bits),
+                RoundPowerOfTwo(upper + Vector128.Create(VerticalBias), CompoundRound1Bits)).AsUInt16();
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector256<ushort> PrepareVertical(Vector256<int> lower, Vector256<int> upper)
+            => Av1IntraPredictorBase.Narrow(
+                RoundPowerOfTwo(lower + Vector256.Create(VerticalBias), CompoundRound1Bits),
+                RoundPowerOfTwo(upper + Vector256.Create(VerticalBias), CompoundRound1Bits)).AsUInt16();
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector512<ushort> PrepareVertical(Vector512<int> lower, Vector512<int> upper)
+            => Av1IntraPredictorBase.Narrow(
+                RoundPowerOfTwo(lower + Vector512.Create(VerticalBias), CompoundRound1Bits),
+                RoundPowerOfTwo(upper + Vector512.Create(VerticalBias), CompoundRound1Bits)).AsUInt16();
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.cs

@@ -5,241 +5,606 @@ using System.Runtime.CompilerServices;
 using System.Runtime.InteropServices;
 using System.Runtime.Intrinsics;
 
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor;
+
 namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
 
-/// <summary>
-/// Combines two AV1 inter predictors with equal-weight rounded averaging.
-/// </summary>
+/// <content>
+/// Produces the biased high-precision intermediates required by compound inter prediction.
+/// </content>
 internal static partial class Av1CompoundInterPredictor
 {
     /// <summary>
-    /// Averages an 8-bit predictor into an existing prediction block.
+    /// The second-round shift retained by every compound convolution path.
+    /// </summary>
+    internal const int CompoundRound1Bits = 7;
+
+    /// <summary>
+    /// The fixed-point precision used by AV1 distance weights.
+    /// </summary>
+    internal const int DistanceWeightBits = 4;
+
+    /// <summary>
+    /// The fixed-point precision used by AV1 compound masks.
+    /// </summary>
+    internal const int MaskWeightBits = 6;
+
+    /// <summary>
+    /// The inclusive upper bound for an AV1 compound-mask alpha value.
+    /// </summary>
+    internal const int MaximumMaskAlpha = 1 << MaskWeightBits;
+
+    /// <summary>
+    /// Reconstructs one 8-bit translational reference into AV1's unsigned compound intermediate format.
+    /// </summary>
+    public static void PredictCompound(
+        ReadOnlySpan<byte> source,
+        int sourceStride,
+        int sourceOrigin,
+        Span<ushort> destination,
+        int destinationStride,
+        int width,
+        int height,
+        Av1InterpolationFilter horizontalFilter,
+        Av1InterpolationFilter verticalFilter,
+        int horizontalPhase,
+        int verticalPhase,
+        Span<short> scratch)
+        => PredictCompound<CompoundPredictionOperator>(
+            source,
+            sourceStride,
+            sourceOrigin,
+            destination,
+            destinationStride,
+            width,
+            height,
+            horizontalFilter,
+            verticalFilter,
+            horizontalPhase,
+            verticalPhase,
+            scratch,
+            useSimd: true);
+
+    /// <summary>
+    /// Executes one closed compound-prediction conversion operator.
+    /// </summary>
+    /// <typeparam name="TOperator">The compound-prediction conversion operator.</typeparam>
+    private static void PredictCompound<TOperator>(
+        ReadOnlySpan<byte> source,
+        int sourceStride,
+        int sourceOrigin,
+        Span<ushort> destination,
+        int destinationStride,
+        int width,
+        int height,
+        Av1InterpolationFilter horizontalFilter,
+        Av1InterpolationFilter verticalFilter,
+        int horizontalPhase,
+        int verticalPhase,
+        Span<short> scratch,
+        bool useSimd)
+        where TOperator : struct, IAv1CompoundPredictionOperator
+    {
+        ReadOnlySpan<short> horizontalCoefficients = GetCompoundCoefficients(horizontalFilter, horizontalPhase, width <= 4);
+        ReadOnlySpan<short> verticalCoefficients = GetCompoundCoefficients(verticalFilter, verticalPhase, height <= 4);
+        int roundBits = (2 * FilterBits) - Round0Bits - CompoundRound1Bits;
+        int offsetBits = 8 + (2 * FilterBits) - Round0Bits;
+        int roundOffset = (1 << (offsetBits - CompoundRound1Bits)) +
+            (1 << (offsetBits - CompoundRound1Bits - 1));
+
+        if (horizontalPhase == 0 && verticalPhase == 0)
+        {
+            CopyCompound<TOperator>(
+                source,
+                sourceStride,
+                sourceOrigin,
+                destination,
+                destinationStride,
+                width,
+                height,
+                roundBits,
+                roundOffset,
+                useSimd);
+
+            return;
+        }
+
+        if (verticalPhase == 0)
+        {
+            GetEffectiveKernel(horizontalCoefficients, out int firstCoefficient, out int tapCount);
+            FilterCompoundDirect<TOperator>(
+                source,
+                sourceStride,
+                sourceOrigin,
+                destination,
+                destinationStride,
+                width,
+                height,
+                horizontalCoefficients[firstCoefficient..],
+                tapCount,
+                firstCoefficient - 3,
+                tapStride: 1,
+                preShift: 0,
+                round: Round0Bits,
+                roundOffset,
+                useSimd);
+
+            return;
+        }
+
+        if (horizontalPhase == 0)
+        {
+            GetEffectiveKernel(verticalCoefficients, out int firstCoefficient, out int tapCount);
+            FilterCompoundDirect<TOperator>(
+                source,
+                sourceStride,
+                sourceOrigin,
+                destination,
+                destinationStride,
+                width,
+                height,
+                verticalCoefficients[firstCoefficient..],
+                tapCount,
+                (firstCoefficient - 3) * sourceStride,
+                sourceStride,
+                FilterBits - Round0Bits,
+                CompoundRound1Bits,
+                roundOffset,
+                useSimd);
+
+            return;
+        }
+
+        GetEffectiveKernel(horizontalCoefficients, out int firstHorizontalCoefficient, out int horizontalTapCount);
+        GetEffectiveKernel(verticalCoefficients, out int firstVerticalCoefficient, out int verticalTapCount);
+        FilterCompound2D<TOperator>(
+            source,
+            sourceStride,
+            sourceOrigin,
+            destination,
+            destinationStride,
+            width,
+            height,
+            horizontalCoefficients[firstHorizontalCoefficient..],
+            horizontalTapCount,
+            firstHorizontalCoefficient - 3,
+            verticalCoefficients[firstVerticalCoefficient..],
+            verticalTapCount,
+            firstVerticalCoefficient - 3,
+            scratch,
+            useSimd);
+    }
+
+    /// <summary>
+    /// Reconstructs one compound intermediate without explicit hardware intrinsics.
     /// </summary>
-    /// <param name="destination">The first predictor and combined output.</param>
-    /// <param name="destinationStride">The distance between destination rows in samples.</param>
-    /// <param name="second">The second predictor.</param>
-    /// <param name="secondStride">The distance between second-predictor rows in samples.</param>
-    /// <param name="width">The active block width.</param>
-    /// <param name="height">The active block height.</param>
-    public static void Average(
-        Span<byte> destination,
+    public static void PredictCompoundScalar(
+        ReadOnlySpan<byte> source,
+        int sourceStride,
+        int sourceOrigin,
+        Span<ushort> destination,
         int destinationStride,
-        ReadOnlySpan<byte> second,
-        int secondStride,
         int width,
-        int height)
+        int height,
+        Av1InterpolationFilter horizontalFilter,
+        Av1InterpolationFilter verticalFilter,
+        int horizontalPhase,
+        int verticalPhase,
+        Span<short> scratch)
+        => PredictCompound<CompoundPredictionOperator>(
+            source,
+            sourceStride,
+            sourceOrigin,
+            destination,
+            destinationStride,
+            width,
+            height,
+            horizontalFilter,
+            verticalFilter,
+            horizontalPhase,
+            verticalPhase,
+            scratch,
+            useSimd: false);
+
+    /// <summary>
+    /// Copies integer-position samples through one closed compound-prediction operator.
+    /// </summary>
+    /// <typeparam name="TOperator">The compound-prediction conversion operator.</typeparam>
+    private static void CopyCompound<TOperator>(
+        ReadOnlySpan<byte> source,
+        int sourceStride,
+        int sourceOrigin,
+        Span<ushort> destination,
+        int destinationStride,
+        int width,
+        int height,
+        int roundBits,
+        int roundOffset,
+        bool useSimd)
+        where TOperator : struct, IAv1CompoundPredictionOperator
     {
+        ref byte sourceBase = ref Unsafe.Add(ref MemoryMarshal.GetReference(source), sourceOrigin);
+        ref ushort destinationBase = ref MemoryMarshal.GetReference(destination);
+
         for (int row = 0; row < height; row++)
         {
-            Span<byte> destinationRow = destination.Slice(row * destinationStride, width);
-            ReadOnlySpan<byte> secondRow = second.Slice(row * secondStride, width);
-            ref byte destinationReference = ref MemoryMarshal.GetReference(destinationRow);
-            ref byte secondReference = ref MemoryMarshal.GetReference(secondRow);
+            ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride);
+            ref ushort destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
             int column = 0;
 
-            if (Vector512.IsHardwareAccelerated)
+            if (useSimd && Vector512.IsHardwareAccelerated)
             {
                 int vectorEnd = width - Vector512<byte>.Count;
                 for (; column <= vectorEnd; column += Vector512<byte>.Count)
                 {
-                    Vector512<byte> firstVector = Vector512.LoadUnsafe(ref destinationReference, (nuint)column);
-                    Vector512<byte> secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column);
-                    Average(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column);
+                    Vector512<byte> samples = Vector512.LoadUnsafe(ref sourceRow, (nuint)column);
+                    TOperator.Copy(samples, roundBits, roundOffset, out Vector512<ushort> lower, out Vector512<ushort> upper);
+                    lower.StoreUnsafe(ref destinationRow, (nuint)column);
+                    upper.StoreUnsafe(ref destinationRow, (nuint)(column + Vector512<ushort>.Count));
                 }
             }
 
-            if (Vector256.IsHardwareAccelerated)
+            if (useSimd && Vector256.IsHardwareAccelerated)
             {
                 int vectorEnd = width - Vector256<byte>.Count;
                 for (; column <= vectorEnd; column += Vector256<byte>.Count)
                 {
-                    Vector256<byte> firstVector = Vector256.LoadUnsafe(ref destinationReference, (nuint)column);
-                    Vector256<byte> secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column);
-                    Average(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column);
+                    Vector256<byte> samples = Vector256.LoadUnsafe(ref sourceRow, (nuint)column);
+                    TOperator.Copy(samples, roundBits, roundOffset, out Vector256<ushort> lower, out Vector256<ushort> upper);
+                    lower.StoreUnsafe(ref destinationRow, (nuint)column);
+                    upper.StoreUnsafe(ref destinationRow, (nuint)(column + Vector256<ushort>.Count));
                 }
             }
 
-            if (Vector128.IsHardwareAccelerated)
+            if (useSimd && Vector128.IsHardwareAccelerated)
             {
                 int vectorEnd = width - Vector128<byte>.Count;
                 for (; column <= vectorEnd; column += Vector128<byte>.Count)
                 {
-                    Vector128<byte> firstVector = Vector128.LoadUnsafe(ref destinationReference, (nuint)column);
-                    Vector128<byte> secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column);
-                    Average(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column);
+                    Vector128<byte> samples = Vector128.LoadUnsafe(ref sourceRow, (nuint)column);
+                    TOperator.Copy(samples, roundBits, roundOffset, out Vector128<ushort> lower, out Vector128<ushort> upper);
+                    lower.StoreUnsafe(ref destinationRow, (nuint)column);
+                    upper.StoreUnsafe(ref destinationRow, (nuint)(column + Vector128<ushort>.Count));
                 }
             }
 
             for (; column < width; column++)
             {
-                destinationRow[column] = (byte)((destinationRow[column] + secondRow[column] + 1) >> 1);
+                Unsafe.Add(ref destinationRow, column) = TOperator.Copy(Unsafe.Add(ref sourceRow, column), roundBits, roundOffset);
             }
         }
     }
 
     /// <summary>
-    /// Averages a high-bit-depth predictor into an existing prediction block.
+    /// Applies one compound convolution direction through one closed conversion operator.
     /// </summary>
-    /// <param name="destination">The first predictor and combined output.</param>
-    /// <param name="destinationStride">The distance between destination rows in samples.</param>
-    /// <param name="second">The second predictor.</param>
-    /// <param name="secondStride">The distance between second-predictor rows in samples.</param>
-    /// <param name="width">The active block width.</param>
-    /// <param name="height">The active block height.</param>
-    public static void Average(
+    /// <typeparam name="TOperator">The compound-prediction conversion operator.</typeparam>
+    private static void FilterCompoundDirect<TOperator>(
+        ReadOnlySpan<byte> source,
+        int sourceStride,
+        int sourceOrigin,
         Span<ushort> destination,
         int destinationStride,
-        ReadOnlySpan<ushort> second,
-        int secondStride,
         int width,
-        int height)
+        int height,
+        ReadOnlySpan<short> coefficients,
+        int tapCount,
+        int sourceOffset,
+        int tapStride,
+        int preShift,
+        int round,
+        int roundOffset,
+        bool useSimd)
+        where TOperator : struct, IAv1CompoundPredictionOperator
     {
+        ref byte sourceBase = ref Unsafe.Add(ref MemoryMarshal.GetReference(source), sourceOrigin);
+        ref ushort destinationBase = ref MemoryMarshal.GetReference(destination);
+        ref short coefficientBase = ref MemoryMarshal.GetReference(coefficients);
+
         for (int row = 0; row < height; row++)
         {
-            Span<ushort> destinationRow = destination.Slice(row * destinationStride, width);
-            ReadOnlySpan<ushort> secondRow = second.Slice(row * secondStride, width);
-            ref ushort destinationReference = ref MemoryMarshal.GetReference(destinationRow);
-            ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow);
+            ref byte sourceRow = ref Unsafe.Add(ref sourceBase, (row * sourceStride) + sourceOffset);
+            ref ushort destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
             int column = 0;
 
-            if (Vector512.IsHardwareAccelerated)
+            if (useSimd && Vector512.IsHardwareAccelerated)
             {
-                int vectorEnd = width - Vector512<ushort>.Count;
-                for (; column <= vectorEnd; column += Vector512<ushort>.Count)
+                int vectorEnd = width - Vector512<byte>.Count;
+                for (; column <= vectorEnd; column += Vector512<byte>.Count)
                 {
-                    Vector512<ushort> firstVector = Vector512.LoadUnsafe(ref destinationReference, (nuint)column);
-                    Vector512<ushort> secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column);
-                    Average(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column);
+                    Convolve(
+                        ref sourceRow,
+                        tapStride,
+                        (nuint)column,
+                        ref coefficientBase,
+                        tapCount,
+                        Vector512<int>.Zero,
+                        out Vector512<int> result0,
+                        out Vector512<int> result1,
+                        out Vector512<int> result2,
+                        out Vector512<int> result3);
+
+                    TOperator.PrepareDirect(result0, result1, preShift, round, roundOffset)
+                        .StoreUnsafe(ref destinationRow, (nuint)column);
+
+                    TOperator.PrepareDirect(result2, result3, preShift, round, roundOffset)
+                        .StoreUnsafe(ref destinationRow, (nuint)(column + Vector512<ushort>.Count));
                 }
             }
 
-            if (Vector256.IsHardwareAccelerated)
+            if (useSimd && Vector256.IsHardwareAccelerated)
             {
-                int vectorEnd = width - Vector256<ushort>.Count;
-                for (; column <= vectorEnd; column += Vector256<ushort>.Count)
+                int vectorEnd = width - Vector256<byte>.Count;
+                for (; column <= vectorEnd; column += Vector256<byte>.Count)
                 {
-                    Vector256<ushort> firstVector = Vector256.LoadUnsafe(ref destinationReference, (nuint)column);
-                    Vector256<ushort> secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column);
-                    Average(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column);
+                    Convolve(
+                        ref sourceRow,
+                        tapStride,
+                        (nuint)column,
+                        ref coefficientBase,
+                        tapCount,
+                        Vector256<int>.Zero,
+                        out Vector256<int> result0,
+                        out Vector256<int> result1,
+                        out Vector256<int> result2,
+                        out Vector256<int> result3);
+
+                    TOperator.PrepareDirect(result0, result1, preShift, round, roundOffset)
+                        .StoreUnsafe(ref destinationRow, (nuint)column);
+
+                    TOperator.PrepareDirect(result2, result3, preShift, round, roundOffset)
+                        .StoreUnsafe(ref destinationRow, (nuint)(column + Vector256<ushort>.Count));
                 }
             }
 
-            if (Vector128.IsHardwareAccelerated)
+            if (useSimd && Vector128.IsHardwareAccelerated)
             {
-                int vectorEnd = width - Vector128<ushort>.Count;
-                for (; column <= vectorEnd; column += Vector128<ushort>.Count)
+                int vectorEnd = width - Vector128<byte>.Count;
+                for (; column <= vectorEnd; column += Vector128<byte>.Count)
                 {
-                    Vector128<ushort> firstVector = Vector128.LoadUnsafe(ref destinationReference, (nuint)column);
-                    Vector128<ushort> secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column);
-                    Average(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column);
+                    Convolve(
+                        ref sourceRow,
+                        tapStride,
+                        (nuint)column,
+                        ref coefficientBase,
+                        tapCount,
+                        Vector128<int>.Zero,
+                        out Vector128<int> result0,
+                        out Vector128<int> result1,
+                        out Vector128<int> result2,
+                        out Vector128<int> result3);
+
+                    TOperator.PrepareDirect(result0, result1, preShift, round, roundOffset)
+                        .StoreUnsafe(ref destinationRow, (nuint)column);
+
+                    TOperator.PrepareDirect(result2, result3, preShift, round, roundOffset)
+                        .StoreUnsafe(ref destinationRow, (nuint)(column + Vector128<ushort>.Count));
                 }
             }
 
             for (; column < width; column++)
             {
-                destinationRow[column] = (ushort)((destinationRow[column] + secondRow[column] + 1) >> 1);
+                int result = ConvolveScalar(
+                    ref Unsafe.Add(ref sourceRow, column),
+                    tapStride,
+                    ref coefficientBase,
+                    tapCount);
+
+                Unsafe.Add(ref destinationRow, column) = TOperator.PrepareDirect(result, preShift, round, roundOffset);
             }
         }
     }
 
     /// <summary>
-    /// Averages an 8-bit predictor without explicit hardware intrinsics.
+    /// Applies separable compound convolution through caller-owned signed scratch.
     /// </summary>
-    /// <param name="destination">The first predictor and combined output.</param>
-    /// <param name="destinationStride">The distance between destination rows in samples.</param>
-    /// <param name="second">The second predictor.</param>
-    /// <param name="secondStride">The distance between second-predictor rows in samples.</param>
-    /// <param name="width">The active block width.</param>
-    /// <param name="height">The active block height.</param>
-    public static void AverageScalar(
-        Span<byte> destination,
+    /// <typeparam name="TOperator">The compound-prediction conversion operator.</typeparam>
+    private static void FilterCompound2D<TOperator>(
+        ReadOnlySpan<byte> source,
+        int sourceStride,
+        int sourceOrigin,
+        Span<ushort> destination,
         int destinationStride,
-        ReadOnlySpan<byte> second,
-        int secondStride,
         int width,
-        int height)
+        int height,
+        ReadOnlySpan<short> horizontalCoefficients,
+        int horizontalTapCount,
+        int horizontalSourceOffset,
+        ReadOnlySpan<short> verticalCoefficients,
+        int verticalTapCount,
+        int verticalSourceOffset,
+        Span<short> scratch,
+        bool useSimd)
+        where TOperator : struct, IAv1CompoundPredictionOperator
     {
-        for (int row = 0; row < height; row++)
+        ref byte sourceBase = ref Unsafe.Add(ref MemoryMarshal.GetReference(source), sourceOrigin);
+        ref ushort destinationBase = ref MemoryMarshal.GetReference(destination);
+        ref short scratchBase = ref MemoryMarshal.GetReference(scratch);
+        ref short horizontalCoefficientBase = ref MemoryMarshal.GetReference(horizontalCoefficients);
+        ref short verticalCoefficientBase = ref MemoryMarshal.GetReference(verticalCoefficients);
+        int scratchStride = Math.Max(width, MinimumScratchStride);
+        int intermediateHeight = height + verticalTapCount - 1;
+
+        // The horizontal pass retains Q7 precision in signed scratch. Each SIMD stage continues at the shared
+        // column offset so mixed-width rows need no padding stores and never cross the logical block edge.
+        for (int row = 0; row < intermediateHeight; row++)
         {
-            Span<byte> destinationRow = destination.Slice(row * destinationStride, width);
-            ReadOnlySpan<byte> secondRow = second.Slice(row * secondStride, width);
-            for (int column = 0; column < width; column++)
+            ref byte sourceRow = ref Unsafe.Add(
+                ref sourceBase,
+                ((row + verticalSourceOffset) * sourceStride) + horizontalSourceOffset);
+
+            ref short scratchRow = ref Unsafe.Add(ref scratchBase, row * scratchStride);
+            int column = 0;
+
+            if (useSimd && Vector512.IsHardwareAccelerated)
             {
-                destinationRow[column] = (byte)((destinationRow[column] + secondRow[column] + 1) >> 1);
+                int vectorEnd = width - Vector512<byte>.Count;
+                for (; column <= vectorEnd; column += Vector512<byte>.Count)
+                {
+                    Convolve(
+                        ref sourceRow,
+                        1,
+                        (nuint)column,
+                        ref horizontalCoefficientBase,
+                        horizontalTapCount,
+                        Vector512<int>.Zero,
+                        out Vector512<int> result0,
+                        out Vector512<int> result1,
+                        out Vector512<int> result2,
+                        out Vector512<int> result3);
+
+                    TOperator.PrepareHorizontal(result0, result1).StoreUnsafe(ref scratchRow, (nuint)column);
+                    TOperator.PrepareHorizontal(result2, result3)
+                        .StoreUnsafe(ref scratchRow, (nuint)(column + Vector512<short>.Count));
+                }
+            }
+
+            if (useSimd && Vector256.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector256<byte>.Count;
+                for (; column <= vectorEnd; column += Vector256<byte>.Count)
+                {
+                    Convolve(
+                        ref sourceRow,
+                        1,
+                        (nuint)column,
+                        ref horizontalCoefficientBase,
+                        horizontalTapCount,
+                        Vector256<int>.Zero,
+                        out Vector256<int> result0,
+                        out Vector256<int> result1,
+                        out Vector256<int> result2,
+                        out Vector256<int> result3);
+
+                    TOperator.PrepareHorizontal(result0, result1).StoreUnsafe(ref scratchRow, (nuint)column);
+                    TOperator.PrepareHorizontal(result2, result3)
+                        .StoreUnsafe(ref scratchRow, (nuint)(column + Vector256<short>.Count));
+                }
+            }
+
+            if (useSimd && Vector128.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector128<byte>.Count;
+                for (; column <= vectorEnd; column += Vector128<byte>.Count)
+                {
+                    Convolve(
+                        ref sourceRow,
+                        1,
+                        (nuint)column,
+                        ref horizontalCoefficientBase,
+                        horizontalTapCount,
+                        Vector128<int>.Zero,
+                        out Vector128<int> result0,
+                        out Vector128<int> result1,
+                        out Vector128<int> result2,
+                        out Vector128<int> result3);
+
+                    TOperator.PrepareHorizontal(result0, result1).StoreUnsafe(ref scratchRow, (nuint)column);
+                    TOperator.PrepareHorizontal(result2, result3)
+                        .StoreUnsafe(ref scratchRow, (nuint)(column + Vector128<short>.Count));
+                }
+            }
+
+            for (; column < width; column++)
+            {
+                int result = ConvolveScalar(
+                    ref Unsafe.Add(ref sourceRow, column),
+                    1,
+                    ref horizontalCoefficientBase,
+                    horizontalTapCount);
+
+                Unsafe.Add(ref scratchRow, column) = TOperator.PrepareHorizontal(result);
             }
         }
-    }
 
-    /// <summary>
-    /// Averages a high-bit-depth predictor without explicit hardware intrinsics.
-    /// </summary>
-    /// <param name="destination">The first predictor and combined output.</param>
-    /// <param name="destinationStride">The distance between destination rows in samples.</param>
-    /// <param name="second">The second predictor.</param>
-    /// <param name="secondStride">The distance between second-predictor rows in samples.</param>
-    /// <param name="width">The active block width.</param>
-    /// <param name="height">The active block height.</param>
-    public static void AverageScalar(
-        Span<ushort> destination,
-        int destinationStride,
-        ReadOnlySpan<ushort> second,
-        int secondStride,
-        int width,
-        int height)
-    {
         for (int row = 0; row < height; row++)
         {
-            Span<ushort> destinationRow = destination.Slice(row * destinationStride, width);
-            ReadOnlySpan<ushort> secondRow = second.Slice(row * secondStride, width);
-            for (int column = 0; column < width; column++)
+            ref short scratchRow = ref Unsafe.Add(ref scratchBase, row * scratchStride);
+            ref ushort destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
+            int column = 0;
+
+            if (useSimd && Vector512.IsHardwareAccelerated)
             {
-                destinationRow[column] = (ushort)((destinationRow[column] + secondRow[column] + 1) >> 1);
+                int vectorEnd = width - Vector512<short>.Count;
+                for (; column <= vectorEnd; column += Vector512<short>.Count)
+                {
+                    Convolve(
+                        ref scratchRow,
+                        scratchStride,
+                        (nuint)column,
+                        ref verticalCoefficientBase,
+                        verticalTapCount,
+                        Vector512<int>.Zero,
+                        out Vector512<int> lower,
+                        out Vector512<int> upper);
+
+                    TOperator.PrepareVertical(lower, upper).StoreUnsafe(ref destinationRow, (nuint)column);
+                }
             }
-        }
-    }
 
-    /// <summary>
-    /// Computes rounded unsigned averages without widening either input vector.
-    /// </summary>
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector512<byte> Average(Vector512<byte> first, Vector512<byte> second)
-    {
-        // (a | b) - ((a ^ b) >> 1) is exactly (a + b + 1) >> 1 and cannot overflow an unsigned lane.
-        return (first | second) - ((first ^ second) >> 1);
-    }
+            if (useSimd && Vector256.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector256<short>.Count;
+                for (; column <= vectorEnd; column += Vector256<short>.Count)
+                {
+                    Convolve(
+                        ref scratchRow,
+                        scratchStride,
+                        (nuint)column,
+                        ref verticalCoefficientBase,
+                        verticalTapCount,
+                        Vector256<int>.Zero,
+                        out Vector256<int> lower,
+                        out Vector256<int> upper);
 
-    /// <summary>
-    /// Computes rounded unsigned averages without widening either input vector.
-    /// </summary>
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector256<byte> Average(Vector256<byte> first, Vector256<byte> second)
-        => (first | second) - ((first ^ second) >> 1);
+                    TOperator.PrepareVertical(lower, upper).StoreUnsafe(ref destinationRow, (nuint)column);
+                }
+            }
 
-    /// <summary>
-    /// Computes rounded unsigned averages without widening either input vector.
-    /// </summary>
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector128<byte> Average(Vector128<byte> first, Vector128<byte> second)
-        => (first | second) - ((first ^ second) >> 1);
+            if (useSimd && Vector128.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector128<short>.Count;
+                for (; column <= vectorEnd; column += Vector128<short>.Count)
+                {
+                    Convolve(
+                        ref scratchRow,
+                        scratchStride,
+                        (nuint)column,
+                        ref verticalCoefficientBase,
+                        verticalTapCount,
+                        Vector128<int>.Zero,
+                        out Vector128<int> lower,
+                        out Vector128<int> upper);
 
-    /// <summary>
-    /// Computes rounded unsigned averages without widening either input vector.
-    /// </summary>
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector512<ushort> Average(Vector512<ushort> first, Vector512<ushort> second)
-        => (first | second) - ((first ^ second) >> 1);
+                    TOperator.PrepareVertical(lower, upper).StoreUnsafe(ref destinationRow, (nuint)column);
+                }
+            }
 
-    /// <summary>
-    /// Computes rounded unsigned averages without widening either input vector.
-    /// </summary>
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector256<ushort> Average(Vector256<ushort> first, Vector256<ushort> second)
-        => (first | second) - ((first ^ second) >> 1);
+            for (; column < width; column++)
+            {
+                int result = ConvolveScalar(
+                    ref Unsafe.Add(ref scratchRow, column),
+                    scratchStride,
+                    ref verticalCoefficientBase,
+                    verticalTapCount);
+
+                Unsafe.Add(ref destinationRow, column) = TOperator.PrepareVertical(result);
+            }
+        }
+    }
 
     /// <summary>
-    /// Computes rounded unsigned averages without widening either input vector.
+    /// Gets the selected interpolation kernel for compound traversal.
     /// </summary>
-    [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector128<ushort> Average(Vector128<ushort> first, Vector128<ushort> second)
-        => (first | second) - ((first ^ second) >> 1);
+    private static ReadOnlySpan<short> GetCompoundCoefficients(
+        Av1InterpolationFilter filter,
+        int phase,
+        bool useReducedFilter)
+        => filter switch
+        {
+            Av1InterpolationFilter.Regular => RegularOperator.GetCoefficients(phase, useReducedFilter),
+            Av1InterpolationFilter.Smooth => SmoothOperator.GetCoefficients(phase, useReducedFilter),
+            Av1InterpolationFilter.Sharp => SharpOperator.GetCoefficients(phase, useReducedFilter),
+            _ => BilinearOperator.GetCoefficients(phase, useReducedFilter),
+        };
 }

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateAveragePredictor.Operator.cs

@@ -0,0 +1,140 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.CompilerServices;
+using System.Runtime.Intrinsics;
+
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor;
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
+
+/// <content>
+/// Defines final equal-average compound-intermediate reconstruction.
+/// </content>
+internal static partial class Av1CompoundIntermediateAveragePredictor
+{
+    /// <summary>
+    /// Defines equal-average finalization for scalar and SIMD lane groups.
+    /// </summary>
+    private interface IAv1CompoundIntermediateAverageOperator
+    {
+        /// <summary>
+        /// Equal-averages and finalizes one pair of compound intermediate samples.
+        /// </summary>
+        /// <param name="first">The first compound intermediate.</param>
+        /// <param name="second">The second compound intermediate.</param>
+        /// <param name="roundBits">The final reconstruction shift.</param>
+        /// <param name="roundOffset">The compound intermediate bias.</param>
+        /// <returns>The reconstructed sample.</returns>
+        public static abstract byte Average(ushort first, ushort second, int roundBits, int roundOffset);
+
+        /// <summary>
+        /// Equal-averages and finalizes 128 bits of compound intermediate samples.
+        /// </summary>
+        /// <param name="first0">The lower first-predictor intermediates.</param>
+        /// <param name="first1">The upper first-predictor intermediates.</param>
+        /// <param name="second0">The lower second-predictor intermediates.</param>
+        /// <param name="second1">The upper second-predictor intermediates.</param>
+        /// <param name="roundBits">The final reconstruction shift.</param>
+        /// <param name="roundOffset">The compound intermediate bias.</param>
+        /// <returns>The reconstructed samples.</returns>
+        public static abstract Vector128<byte> Average(
+            Vector128<ushort> first0,
+            Vector128<ushort> first1,
+            Vector128<ushort> second0,
+            Vector128<ushort> second1,
+            int roundBits,
+            int roundOffset);
+
+        /// <summary>
+        /// Equal-averages and finalizes 256 bits of compound intermediate samples.
+        /// </summary>
+        /// <param name="first0">The lower first-predictor intermediates.</param>
+        /// <param name="first1">The upper first-predictor intermediates.</param>
+        /// <param name="second0">The lower second-predictor intermediates.</param>
+        /// <param name="second1">The upper second-predictor intermediates.</param>
+        /// <param name="roundBits">The final reconstruction shift.</param>
+        /// <param name="roundOffset">The compound intermediate bias.</param>
+        /// <returns>The reconstructed samples.</returns>
+        public static abstract Vector256<byte> Average(
+            Vector256<ushort> first0,
+            Vector256<ushort> first1,
+            Vector256<ushort> second0,
+            Vector256<ushort> second1,
+            int roundBits,
+            int roundOffset);
+
+        /// <summary>
+        /// Equal-averages and finalizes 512 bits of compound intermediate samples.
+        /// </summary>
+        /// <param name="first0">The lower first-predictor intermediates.</param>
+        /// <param name="first1">The upper first-predictor intermediates.</param>
+        /// <param name="second0">The lower second-predictor intermediates.</param>
+        /// <param name="second1">The upper second-predictor intermediates.</param>
+        /// <param name="roundBits">The final reconstruction shift.</param>
+        /// <param name="roundOffset">The compound intermediate bias.</param>
+        /// <returns>The reconstructed samples.</returns>
+        public static abstract Vector512<byte> Average(
+            Vector512<ushort> first0,
+            Vector512<ushort> first1,
+            Vector512<ushort> second0,
+            Vector512<ushort> second1,
+            int roundBits,
+            int roundOffset);
+    }
+
+    /// <summary>
+    /// Implements equal-average finalization for scalar and SIMD lane groups.
+    /// </summary>
+    private readonly struct CompoundIntermediateAverageOperator : IAv1CompoundIntermediateAverageOperator
+    {
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static byte Average(ushort first, ushort second, int roundBits, int roundOffset)
+        {
+            // The reference average deliberately truncates here. Finalization performs the sole rounding step.
+            int result = ((first + second) >> 1) - roundOffset;
+            return (byte)Math.Clamp(RoundPowerOfTwo(result, roundBits), 0, byte.MaxValue);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector128<byte> Average(
+            Vector128<ushort> first0,
+            Vector128<ushort> first1,
+            Vector128<ushort> second0,
+            Vector128<ushort> second1,
+            int roundBits,
+            int roundOffset)
+            => Vector128.Narrow(
+                FinalizeIntermediate((first0 & second0) + ((first0 ^ second0) >> 1), roundBits, roundOffset),
+                FinalizeIntermediate((first1 & second1) + ((first1 ^ second1) >> 1), roundBits, roundOffset));
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector256<byte> Average(
+            Vector256<ushort> first0,
+            Vector256<ushort> first1,
+            Vector256<ushort> second0,
+            Vector256<ushort> second1,
+            int roundBits,
+            int roundOffset)
+            => Vector256.Narrow(
+                FinalizeIntermediate((first0 & second0) + ((first0 ^ second0) >> 1), roundBits, roundOffset),
+                FinalizeIntermediate((first1 & second1) + ((first1 ^ second1) >> 1), roundBits, roundOffset));
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector512<byte> Average(
+            Vector512<ushort> first0,
+            Vector512<ushort> first1,
+            Vector512<ushort> second0,
+            Vector512<ushort> second1,
+            int roundBits,
+            int roundOffset)
+            => Vector512.Narrow(
+                FinalizeIntermediate((first0 & second0) + ((first0 ^ second0) >> 1), roundBits, roundOffset),
+                FinalizeIntermediate((first1 & second1) + ((first1 ^ second1) >> 1), roundBits, roundOffset));
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateAveragePredictor.cs

@@ -0,0 +1,124 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.CompilerServices;
+using System.Runtime.InteropServices;
+using System.Runtime.Intrinsics;
+
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor;
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
+
+/// <content>
+/// Reconstructs final samples by equal-averaging compound intermediates.
+/// </content>
+internal static partial class Av1CompoundIntermediateAveragePredictor
+{
+    /// <summary>
+    /// Combines two compound intermediates by equal averaging.
+    /// </summary>
+    public static void AverageIntermediate(
+        Span<byte> destination,
+        int destinationStride,
+        ReadOnlySpan<ushort> first,
+        int firstStride,
+        ReadOnlySpan<ushort> second,
+        int secondStride,
+        int width,
+        int height,
+        int bitDepth)
+        => AverageIntermediate<CompoundIntermediateAverageOperator>(
+            destination,
+            destinationStride,
+            first,
+            firstStride,
+            second,
+            secondStride,
+            width,
+            height,
+            bitDepth);
+
+    /// <summary>
+    /// Executes one closed equal-average compound-intermediate operator.
+    /// </summary>
+    /// <typeparam name="TOperator">The compound-intermediate operator.</typeparam>
+    private static void AverageIntermediate<TOperator>(
+        Span<byte> destination,
+        int destinationStride,
+        ReadOnlySpan<ushort> first,
+        int firstStride,
+        ReadOnlySpan<ushort> second,
+        int secondStride,
+        int width,
+        int height,
+        int bitDepth)
+        where TOperator : struct, IAv1CompoundIntermediateAverageOperator
+    {
+        GetIntermediateRounding(bitDepth, out int roundBits, out int roundOffset);
+        for (int row = 0; row < height; row++)
+        {
+            Span<byte> destinationRow = destination.Slice(row * destinationStride, width);
+            ReadOnlySpan<ushort> firstRow = first.Slice(row * firstStride, width);
+            ReadOnlySpan<ushort> secondRow = second.Slice(row * secondStride, width);
+            ref byte destinationReference = ref MemoryMarshal.GetReference(destinationRow);
+            ref ushort firstReference = ref MemoryMarshal.GetReference(firstRow);
+            ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow);
+            int column = 0;
+
+            if (Vector512.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector512<byte>.Count;
+                for (; column <= vectorEnd; column += Vector512<byte>.Count)
+                {
+                    Vector512<ushort> first0 = Vector512.LoadUnsafe(ref firstReference, (nuint)column);
+                    Vector512<ushort> first1 = Vector512.LoadUnsafe(ref firstReference, (nuint)(column + Vector512<ushort>.Count));
+                    Vector512<ushort> second0 = Vector512.LoadUnsafe(ref secondReference, (nuint)column);
+                    Vector512<ushort> second1 = Vector512.LoadUnsafe(ref secondReference, (nuint)(column + Vector512<ushort>.Count));
+                    TOperator.Average(first0, first1, second0, second1, roundBits, roundOffset)
+                        .StoreUnsafe(ref destinationReference, (nuint)column);
+                }
+            }
+
+            if (Vector256.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector256<byte>.Count;
+                for (; column <= vectorEnd; column += Vector256<byte>.Count)
+                {
+                    Vector256<ushort> first0 = Vector256.LoadUnsafe(ref firstReference, (nuint)column);
+                    Vector256<ushort> first1 = Vector256.LoadUnsafe(ref firstReference, (nuint)(column + Vector256<ushort>.Count));
+                    Vector256<ushort> second0 = Vector256.LoadUnsafe(ref secondReference, (nuint)column);
+                    Vector256<ushort> second1 = Vector256.LoadUnsafe(ref secondReference, (nuint)(column + Vector256<ushort>.Count));
+                    TOperator.Average(first0, first1, second0, second1, roundBits, roundOffset)
+                        .StoreUnsafe(ref destinationReference, (nuint)column);
+                }
+            }
+
+            if (Vector128.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector128<byte>.Count;
+                for (; column <= vectorEnd; column += Vector128<byte>.Count)
+                {
+                    Vector128<ushort> first0 = Vector128.LoadUnsafe(ref firstReference, (nuint)column);
+                    Vector128<ushort> first1 = Vector128.LoadUnsafe(
+                        ref firstReference,
+                        (nuint)(column + Vector128<ushort>.Count));
+
+                    Vector128<ushort> second0 = Vector128.LoadUnsafe(ref secondReference, (nuint)column);
+                    Vector128<ushort> second1 = Vector128.LoadUnsafe(
+                        ref secondReference,
+                        (nuint)(column + Vector128<ushort>.Count));
+
+                    TOperator.Average(first0, first1, second0, second1, roundBits, roundOffset).StoreUnsafe(
+                        ref destinationReference,
+                        (nuint)column);
+                }
+            }
+
+            for (; column < width; column++)
+            {
+                destinationRow[column] = TOperator.Average(firstRow[column], secondRow[column], roundBits, roundOffset);
+            }
+        }
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateDifferenceWeightedMaskBuilder.Operator.cs

@@ -0,0 +1,218 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.CompilerServices;
+using System.Runtime.Intrinsics;
+
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor;
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
+
+/// <content>
+/// Defines difference-weighted mask generation from compound intermediates.
+/// </content>
+internal static partial class Av1CompoundIntermediateDifferenceWeightedMaskBuilder
+{
+    /// <summary>
+    /// Defines difference-weighted mask generation for scalar and SIMD lane groups.
+    /// </summary>
+    private interface IAv1CompoundIntermediateDifferenceWeightedMaskOperator
+    {
+        /// <summary>
+        /// Creates one difference-weighted mask value from compound intermediate samples.
+        /// </summary>
+        /// <param name="first">The first compound intermediate.</param>
+        /// <param name="second">The second compound intermediate.</param>
+        /// <param name="differenceRound">The difference scaling shift.</param>
+        /// <param name="invert">Whether to invert the selected predictor.</param>
+        /// <returns>The AV1 mask value.</returns>
+        public static abstract byte CreateMask(ushort first, ushort second, int differenceRound, bool invert);
+
+        /// <summary>
+        /// Creates 128 bits of difference-weighted mask values from compound intermediate samples.
+        /// </summary>
+        /// <param name="first0">The lower first-predictor intermediates.</param>
+        /// <param name="first1">The upper first-predictor intermediates.</param>
+        /// <param name="second0">The lower second-predictor intermediates.</param>
+        /// <param name="second1">The upper second-predictor intermediates.</param>
+        /// <param name="differenceRound">The difference scaling shift.</param>
+        /// <param name="invert">Whether to invert the selected predictor.</param>
+        /// <returns>The packed AV1 mask values.</returns>
+        public static abstract Vector128<byte> CreateMask(
+            Vector128<ushort> first0,
+            Vector128<ushort> first1,
+            Vector128<ushort> second0,
+            Vector128<ushort> second1,
+            int differenceRound,
+            bool invert);
+
+        /// <summary>
+        /// Creates 256 bits of difference-weighted mask values from compound intermediate samples.
+        /// </summary>
+        /// <param name="first0">The lower first-predictor intermediates.</param>
+        /// <param name="first1">The upper first-predictor intermediates.</param>
+        /// <param name="second0">The lower second-predictor intermediates.</param>
+        /// <param name="second1">The upper second-predictor intermediates.</param>
+        /// <param name="differenceRound">The difference scaling shift.</param>
+        /// <param name="invert">Whether to invert the selected predictor.</param>
+        /// <returns>The packed AV1 mask values.</returns>
+        public static abstract Vector256<byte> CreateMask(
+            Vector256<ushort> first0,
+            Vector256<ushort> first1,
+            Vector256<ushort> second0,
+            Vector256<ushort> second1,
+            int differenceRound,
+            bool invert);
+
+        /// <summary>
+        /// Creates 512 bits of difference-weighted mask values from compound intermediate samples.
+        /// </summary>
+        /// <param name="first0">The lower first-predictor intermediates.</param>
+        /// <param name="first1">The upper first-predictor intermediates.</param>
+        /// <param name="second0">The lower second-predictor intermediates.</param>
+        /// <param name="second1">The upper second-predictor intermediates.</param>
+        /// <param name="differenceRound">The difference scaling shift.</param>
+        /// <param name="invert">Whether to invert the selected predictor.</param>
+        /// <returns>The packed AV1 mask values.</returns>
+        public static abstract Vector512<byte> CreateMask(
+            Vector512<ushort> first0,
+            Vector512<ushort> first1,
+            Vector512<ushort> second0,
+            Vector512<ushort> second1,
+            int differenceRound,
+            bool invert);
+    }
+
+    /// <summary>
+    /// Implements difference-weighted mask generation for scalar and SIMD lane groups.
+    /// </summary>
+    private readonly struct CompoundIntermediateDifferenceWeightedMaskOperator : IAv1CompoundIntermediateDifferenceWeightedMaskOperator
+    {
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static byte CreateMask(ushort first, ushort second, int differenceRound, bool invert)
+        {
+            int difference = RoundPowerOfTwo(Math.Abs(first - second), differenceRound);
+            int alpha = Math.Min(MaximumMaskAlpha, 38 + (difference >> 4));
+            return (byte)(invert ? MaximumMaskAlpha - alpha : alpha);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector128<byte> CreateMask(
+            Vector128<ushort> first0,
+            Vector128<ushort> first1,
+            Vector128<ushort> second0,
+            Vector128<ushort> second1,
+            int differenceRound,
+            bool invert)
+            => Vector128.Narrow(
+                CreateMask(first0, second0, differenceRound, invert),
+                CreateMask(first1, second1, differenceRound, invert));
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector256<byte> CreateMask(
+            Vector256<ushort> first0,
+            Vector256<ushort> first1,
+            Vector256<ushort> second0,
+            Vector256<ushort> second1,
+            int differenceRound,
+            bool invert)
+            => Vector256.Narrow(
+                CreateMask(first0, second0, differenceRound, invert),
+                CreateMask(first1, second1, differenceRound, invert));
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector512<byte> CreateMask(
+            Vector512<ushort> first0,
+            Vector512<ushort> first1,
+            Vector512<ushort> second0,
+            Vector512<ushort> second1,
+            int differenceRound,
+            bool invert)
+            => Vector512.Narrow(
+                CreateMask(first0, second0, differenceRound, invert),
+                CreateMask(first1, second1, differenceRound, invert));
+
+        /// <summary>
+        /// Creates 128-bit unpacked mask values without losing the required pre-alpha rounding.
+        /// </summary>
+        private static Vector128<ushort> CreateMask(Vector128<ushort> first, Vector128<ushort> second, int differenceRound, bool invert)
+        {
+            Vector128<ushort> difference = Vector128.Max(first, second) - Vector128.Min(first, second);
+            Vector128<int> lower = CreateMaskAlpha(Vector128.WidenLower(difference).AsInt32(), differenceRound, invert);
+            Vector128<int> upper = CreateMaskAlpha(Vector128.WidenUpper(difference).AsInt32(), differenceRound, invert);
+            return Vector128.Narrow(lower, upper).AsUInt16();
+        }
+
+        /// <summary>
+        /// Creates 256-bit unpacked mask values without losing the required pre-alpha rounding.
+        /// </summary>
+        private static Vector256<ushort> CreateMask(Vector256<ushort> first, Vector256<ushort> second, int differenceRound, bool invert)
+        {
+            Vector256<ushort> difference = Vector256.Max(first, second) - Vector256.Min(first, second);
+            Vector256<int> lower = CreateMaskAlpha(Vector256.WidenLower(difference).AsInt32(), differenceRound, invert);
+            Vector256<int> upper = CreateMaskAlpha(Vector256.WidenUpper(difference).AsInt32(), differenceRound, invert);
+            return Vector256.Narrow(lower, upper).AsUInt16();
+        }
+
+        /// <summary>
+        /// Creates 512-bit unpacked mask values without losing the required pre-alpha rounding.
+        /// </summary>
+        private static Vector512<ushort> CreateMask(Vector512<ushort> first, Vector512<ushort> second, int differenceRound, bool invert)
+        {
+            Vector512<ushort> difference = Vector512.Max(first, second) - Vector512.Min(first, second);
+            Vector512<int> lower = CreateMaskAlpha(Vector512.WidenLower(difference).AsInt32(), differenceRound, invert);
+            Vector512<int> upper = CreateMaskAlpha(Vector512.WidenUpper(difference).AsInt32(), differenceRound, invert);
+            return Vector512.Narrow(lower, upper).AsUInt16();
+        }
+
+        /// <summary>
+        /// Converts 128-bit intermediate differences to the decoded type-38 mask range.
+        /// </summary>
+        private static Vector128<int> CreateMaskAlpha(Vector128<int> difference, int differenceRound, bool invert)
+        {
+            if (differenceRound != 0)
+            {
+                difference = (difference + Vector128.Create(1 << (differenceRound - 1))) >> differenceRound;
+            }
+
+            Vector128<int> maximum = Vector128.Create(MaximumMaskAlpha);
+            Vector128<int> alpha = Vector128.Min(maximum, (difference >> 4) + Vector128.Create(38));
+            return invert ? maximum - alpha : alpha;
+        }
+
+        /// <summary>
+        /// Converts 256-bit intermediate differences to the decoded type-38 mask range.
+        /// </summary>
+        private static Vector256<int> CreateMaskAlpha(Vector256<int> difference, int differenceRound, bool invert)
+        {
+            if (differenceRound != 0)
+            {
+                difference = (difference + Vector256.Create(1 << (differenceRound - 1))) >> differenceRound;
+            }
+
+            Vector256<int> maximum = Vector256.Create(MaximumMaskAlpha);
+            Vector256<int> alpha = Vector256.Min(maximum, (difference >> 4) + Vector256.Create(38));
+            return invert ? maximum - alpha : alpha;
+        }
+
+        /// <summary>
+        /// Converts 512-bit intermediate differences to the decoded type-38 mask range.
+        /// </summary>
+        private static Vector512<int> CreateMaskAlpha(Vector512<int> difference, int differenceRound, bool invert)
+        {
+            if (differenceRound != 0)
+            {
+                difference = (difference + Vector512.Create(1 << (differenceRound - 1))) >> differenceRound;
+            }
+
+            Vector512<int> maximum = Vector512.Create(MaximumMaskAlpha);
+            Vector512<int> alpha = Vector512.Min(maximum, (difference >> 4) + Vector512.Create(38));
+            return invert ? maximum - alpha : alpha;
+        }
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateDifferenceWeightedMaskBuilder.cs

@@ -0,0 +1,134 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.CompilerServices;
+using System.Runtime.InteropServices;
+using System.Runtime.Intrinsics;
+using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling;
+
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor;
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
+
+/// <content>
+/// Builds difference-weighted masks from compound intermediates.
+/// </content>
+internal static partial class Av1CompoundIntermediateDifferenceWeightedMaskBuilder
+{
+    /// <summary>
+    /// Fills a luma-resolution difference-weighted mask from compound intermediates.
+    /// </summary>
+    public static void FillDifferenceWeightedIntermediateMask(
+        Span<byte> mask,
+        int maskStride,
+        ReadOnlySpan<ushort> first,
+        int firstStride,
+        ReadOnlySpan<ushort> second,
+        int secondStride,
+        int width,
+        int height,
+        int bitDepth,
+        Av1DifferenceWeightedMaskType maskType)
+        => FillDifferenceWeightedIntermediateMask<CompoundIntermediateDifferenceWeightedMaskOperator>(
+            mask,
+            maskStride,
+            first,
+            firstStride,
+            second,
+            secondStride,
+            width,
+            height,
+            bitDepth,
+            maskType);
+
+    /// <summary>
+    /// Executes one closed difference-mask compound-intermediate operator.
+    /// </summary>
+    /// <typeparam name="TOperator">The compound-intermediate operator.</typeparam>
+    private static void FillDifferenceWeightedIntermediateMask<TOperator>(
+        Span<byte> mask,
+        int maskStride,
+        ReadOnlySpan<ushort> first,
+        int firstStride,
+        ReadOnlySpan<ushort> second,
+        int secondStride,
+        int width,
+        int height,
+        int bitDepth,
+        Av1DifferenceWeightedMaskType maskType)
+        where TOperator : struct, IAv1CompoundIntermediateDifferenceWeightedMaskOperator
+    {
+        bool invert = maskType == Av1DifferenceWeightedMaskType.Type38Inverse;
+        GetIntermediateRounding(bitDepth, out int roundBits, out _);
+        int differenceRound = roundBits + bitDepth - 8;
+        for (int row = 0; row < height; row++)
+        {
+            Span<byte> maskRow = mask.Slice(row * maskStride, width);
+            ReadOnlySpan<ushort> firstRow = first.Slice(row * firstStride, width);
+            ReadOnlySpan<ushort> secondRow = second.Slice(row * secondStride, width);
+            ref byte maskReference = ref MemoryMarshal.GetReference(maskRow);
+            ref ushort firstReference = ref MemoryMarshal.GetReference(firstRow);
+            ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow);
+            int column = 0;
+
+            if (Vector512.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector512<byte>.Count;
+                for (; column <= vectorEnd; column += Vector512<byte>.Count)
+                {
+                    Vector512<ushort> first0 = Vector512.LoadUnsafe(ref firstReference, (nuint)column);
+                    Vector512<ushort> first1 = Vector512.LoadUnsafe(ref firstReference, (nuint)(column + Vector512<ushort>.Count));
+                    Vector512<ushort> second0 = Vector512.LoadUnsafe(ref secondReference, (nuint)column);
+                    Vector512<ushort> second1 = Vector512.LoadUnsafe(ref secondReference, (nuint)(column + Vector512<ushort>.Count));
+                    TOperator.CreateMask(first0, first1, second0, second1, differenceRound, invert)
+                        .StoreUnsafe(ref maskReference, (nuint)column);
+                }
+            }
+
+            if (Vector256.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector256<byte>.Count;
+                for (; column <= vectorEnd; column += Vector256<byte>.Count)
+                {
+                    Vector256<ushort> first0 = Vector256.LoadUnsafe(ref firstReference, (nuint)column);
+                    Vector256<ushort> first1 = Vector256.LoadUnsafe(ref firstReference, (nuint)(column + Vector256<ushort>.Count));
+                    Vector256<ushort> second0 = Vector256.LoadUnsafe(ref secondReference, (nuint)column);
+                    Vector256<ushort> second1 = Vector256.LoadUnsafe(ref secondReference, (nuint)(column + Vector256<ushort>.Count));
+                    TOperator.CreateMask(first0, first1, second0, second1, differenceRound, invert)
+                        .StoreUnsafe(ref maskReference, (nuint)column);
+                }
+            }
+
+            if (Vector128.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector128<byte>.Count;
+                for (; column <= vectorEnd; column += Vector128<byte>.Count)
+                {
+                    Vector128<ushort> first0 = Vector128.LoadUnsafe(ref firstReference, (nuint)column);
+                    Vector128<ushort> first1 = Vector128.LoadUnsafe(
+                        ref firstReference,
+                        (nuint)(column + Vector128<ushort>.Count));
+
+                    Vector128<ushort> second0 = Vector128.LoadUnsafe(ref secondReference, (nuint)column);
+                    Vector128<ushort> second1 = Vector128.LoadUnsafe(
+                        ref secondReference,
+                        (nuint)(column + Vector128<ushort>.Count));
+
+                    TOperator.CreateMask(
+                        first0,
+                        first1,
+                        second0,
+                        second1,
+                        differenceRound,
+                        invert).StoreUnsafe(ref maskReference, (nuint)column);
+                }
+            }
+
+            for (; column < width; column++)
+            {
+                maskRow[column] = TOperator.CreateMask(firstRow[column], secondRow[column], differenceRound, invert);
+            }
+        }
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateDistanceWeightedPredictor.Operator.cs

@@ -0,0 +1,232 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.CompilerServices;
+using System.Runtime.Intrinsics;
+
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor;
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
+
+/// <content>
+/// Defines final distance-weighted compound-intermediate reconstruction.
+/// </content>
+internal static partial class Av1CompoundIntermediateDistanceWeightedPredictor
+{
+    /// <summary>
+    /// Defines distance-weighted finalization for scalar and SIMD lane groups.
+    /// </summary>
+    private interface IAv1CompoundIntermediateDistanceWeightedOperator
+    {
+        /// <summary>
+        /// Distance-weights and finalizes one pair of compound intermediate samples.
+        /// </summary>
+        /// <param name="first">The first compound intermediate.</param>
+        /// <param name="second">The second compound intermediate.</param>
+        /// <param name="firstWeight">The first predictor weight.</param>
+        /// <param name="secondWeight">The second predictor weight.</param>
+        /// <param name="roundBits">The final reconstruction shift.</param>
+        /// <param name="roundOffset">The compound intermediate bias.</param>
+        /// <returns>The reconstructed sample.</returns>
+        public static abstract byte DistanceWeighted(
+            ushort first,
+            ushort second,
+            int firstWeight,
+            int secondWeight,
+            int roundBits,
+            int roundOffset);
+
+        /// <summary>
+        /// Distance-weights and finalizes 128 bits of compound intermediate samples.
+        /// </summary>
+        /// <param name="first0">The lower first-predictor intermediates.</param>
+        /// <param name="first1">The upper first-predictor intermediates.</param>
+        /// <param name="second0">The lower second-predictor intermediates.</param>
+        /// <param name="second1">The upper second-predictor intermediates.</param>
+        /// <param name="firstWeight">The first predictor weight.</param>
+        /// <param name="secondWeight">The second predictor weight.</param>
+        /// <param name="roundBits">The final reconstruction shift.</param>
+        /// <param name="roundOffset">The compound intermediate bias.</param>
+        /// <returns>The reconstructed samples.</returns>
+        public static abstract Vector128<byte> DistanceWeighted(
+            Vector128<ushort> first0,
+            Vector128<ushort> first1,
+            Vector128<ushort> second0,
+            Vector128<ushort> second1,
+            int firstWeight,
+            int secondWeight,
+            int roundBits,
+            int roundOffset);
+
+        /// <summary>
+        /// Distance-weights and finalizes 256 bits of compound intermediate samples.
+        /// </summary>
+        /// <param name="first0">The lower first-predictor intermediates.</param>
+        /// <param name="first1">The upper first-predictor intermediates.</param>
+        /// <param name="second0">The lower second-predictor intermediates.</param>
+        /// <param name="second1">The upper second-predictor intermediates.</param>
+        /// <param name="firstWeight">The first predictor weight.</param>
+        /// <param name="secondWeight">The second predictor weight.</param>
+        /// <param name="roundBits">The final reconstruction shift.</param>
+        /// <param name="roundOffset">The compound intermediate bias.</param>
+        /// <returns>The reconstructed samples.</returns>
+        public static abstract Vector256<byte> DistanceWeighted(
+            Vector256<ushort> first0,
+            Vector256<ushort> first1,
+            Vector256<ushort> second0,
+            Vector256<ushort> second1,
+            int firstWeight,
+            int secondWeight,
+            int roundBits,
+            int roundOffset);
+
+        /// <summary>
+        /// Distance-weights and finalizes 512 bits of compound intermediate samples.
+        /// </summary>
+        /// <param name="first0">The lower first-predictor intermediates.</param>
+        /// <param name="first1">The upper first-predictor intermediates.</param>
+        /// <param name="second0">The lower second-predictor intermediates.</param>
+        /// <param name="second1">The upper second-predictor intermediates.</param>
+        /// <param name="firstWeight">The first predictor weight.</param>
+        /// <param name="secondWeight">The second predictor weight.</param>
+        /// <param name="roundBits">The final reconstruction shift.</param>
+        /// <param name="roundOffset">The compound intermediate bias.</param>
+        /// <returns>The reconstructed samples.</returns>
+        public static abstract Vector512<byte> DistanceWeighted(
+            Vector512<ushort> first0,
+            Vector512<ushort> first1,
+            Vector512<ushort> second0,
+            Vector512<ushort> second1,
+            int firstWeight,
+            int secondWeight,
+            int roundBits,
+            int roundOffset);
+    }
+
+    /// <summary>
+    /// Implements distance-weighted finalization for scalar and SIMD lane groups.
+    /// </summary>
+    private readonly struct CompoundIntermediateDistanceWeightedOperator : IAv1CompoundIntermediateDistanceWeightedOperator
+    {
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static byte DistanceWeighted(
+            ushort first,
+            ushort second,
+            int firstWeight,
+            int secondWeight,
+            int roundBits,
+            int roundOffset)
+        {
+            int result = ((first * firstWeight) + (second * secondWeight)) >> DistanceWeightBits;
+            result -= roundOffset;
+            return (byte)Math.Clamp(RoundPowerOfTwo(result, roundBits), 0, byte.MaxValue);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector128<byte> DistanceWeighted(
+            Vector128<ushort> first0,
+            Vector128<ushort> first1,
+            Vector128<ushort> second0,
+            Vector128<ushort> second1,
+            int firstWeight,
+            int secondWeight,
+            int roundBits,
+            int roundOffset)
+            => Vector128.Narrow(
+                DistanceWeighted(first0, second0, firstWeight, secondWeight, roundBits, roundOffset),
+                DistanceWeighted(first1, second1, firstWeight, secondWeight, roundBits, roundOffset));
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector256<byte> DistanceWeighted(
+            Vector256<ushort> first0,
+            Vector256<ushort> first1,
+            Vector256<ushort> second0,
+            Vector256<ushort> second1,
+            int firstWeight,
+            int secondWeight,
+            int roundBits,
+            int roundOffset)
+            => Vector256.Narrow(
+                DistanceWeighted(first0, second0, firstWeight, secondWeight, roundBits, roundOffset),
+                DistanceWeighted(first1, second1, firstWeight, secondWeight, roundBits, roundOffset));
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector512<byte> DistanceWeighted(
+            Vector512<ushort> first0,
+            Vector512<ushort> first1,
+            Vector512<ushort> second0,
+            Vector512<ushort> second1,
+            int firstWeight,
+            int secondWeight,
+            int roundBits,
+            int roundOffset)
+            => Vector512.Narrow(
+                DistanceWeighted(first0, second0, firstWeight, secondWeight, roundBits, roundOffset),
+                DistanceWeighted(first1, second1, firstWeight, secondWeight, roundBits, roundOffset));
+
+        /// <summary>
+        /// Distance-weights 128-bit lanes without overflowing the unsigned intermediate range.
+        /// </summary>
+        private static Vector128<ushort> DistanceWeighted(
+            Vector128<ushort> first,
+            Vector128<ushort> second,
+            int firstWeight,
+            int secondWeight,
+            int roundBits,
+            int roundOffset)
+        {
+            Vector128<int> firstLower = Vector128.WidenLower(first).AsInt32();
+            Vector128<int> firstUpper = Vector128.WidenUpper(first).AsInt32();
+            Vector128<int> secondLower = Vector128.WidenLower(second).AsInt32();
+            Vector128<int> secondUpper = Vector128.WidenUpper(second).AsInt32();
+            Vector128<int> lower = ((firstLower * firstWeight) + (secondLower * secondWeight)) >> DistanceWeightBits;
+            Vector128<int> upper = ((firstUpper * firstWeight) + (secondUpper * secondWeight)) >> DistanceWeightBits;
+            return Vector128.Narrow(FinalizeIntermediate(lower, roundBits, roundOffset), FinalizeIntermediate(upper, roundBits, roundOffset)).AsUInt16();
+        }
+
+        /// <summary>
+        /// Distance-weights 256-bit lanes without overflowing the unsigned intermediate range.
+        /// </summary>
+        private static Vector256<ushort> DistanceWeighted(
+            Vector256<ushort> first,
+            Vector256<ushort> second,
+            int firstWeight,
+            int secondWeight,
+            int roundBits,
+            int roundOffset)
+        {
+            Vector256<int> firstLower = Vector256.WidenLower(first).AsInt32();
+            Vector256<int> firstUpper = Vector256.WidenUpper(first).AsInt32();
+            Vector256<int> secondLower = Vector256.WidenLower(second).AsInt32();
+            Vector256<int> secondUpper = Vector256.WidenUpper(second).AsInt32();
+            Vector256<int> lower = ((firstLower * firstWeight) + (secondLower * secondWeight)) >> DistanceWeightBits;
+            Vector256<int> upper = ((firstUpper * firstWeight) + (secondUpper * secondWeight)) >> DistanceWeightBits;
+            return Vector256.Narrow(FinalizeIntermediate(lower, roundBits, roundOffset), FinalizeIntermediate(upper, roundBits, roundOffset)).AsUInt16();
+        }
+
+        /// <summary>
+        /// Distance-weights 512-bit lanes without overflowing the unsigned intermediate range.
+        /// </summary>
+        private static Vector512<ushort> DistanceWeighted(
+            Vector512<ushort> first,
+            Vector512<ushort> second,
+            int firstWeight,
+            int secondWeight,
+            int roundBits,
+            int roundOffset)
+        {
+            Vector512<int> firstLower = Vector512.WidenLower(first).AsInt32();
+            Vector512<int> firstUpper = Vector512.WidenUpper(first).AsInt32();
+            Vector512<int> secondLower = Vector512.WidenLower(second).AsInt32();
+            Vector512<int> secondUpper = Vector512.WidenUpper(second).AsInt32();
+            Vector512<int> lower = ((firstLower * firstWeight) + (secondLower * secondWeight)) >> DistanceWeightBits;
+            Vector512<int> upper = ((firstUpper * firstWeight) + (secondUpper * secondWeight)) >> DistanceWeightBits;
+            return Vector512.Narrow(FinalizeIntermediate(lower, roundBits, roundOffset), FinalizeIntermediate(upper, roundBits, roundOffset)).AsUInt16();
+        }
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateDistanceWeightedPredictor.cs

@@ -0,0 +1,156 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.CompilerServices;
+using System.Runtime.InteropServices;
+using System.Runtime.Intrinsics;
+
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor;
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
+
+/// <content>
+/// Reconstructs final samples by distance-weighting compound intermediates.
+/// </content>
+internal static partial class Av1CompoundIntermediateDistanceWeightedPredictor
+{
+    /// <summary>
+    /// Combines two compound intermediates using the decoded temporal-distance weights.
+    /// </summary>
+    public static void DistanceWeightedIntermediate(
+        Span<byte> destination,
+        int destinationStride,
+        ReadOnlySpan<ushort> first,
+        int firstStride,
+        ReadOnlySpan<ushort> second,
+        int secondStride,
+        int width,
+        int height,
+        int firstWeight,
+        int secondWeight,
+        int bitDepth)
+        => DistanceWeightedIntermediate<CompoundIntermediateDistanceWeightedOperator>(
+            destination,
+            destinationStride,
+            first,
+            firstStride,
+            second,
+            secondStride,
+            width,
+            height,
+            firstWeight,
+            secondWeight,
+            bitDepth);
+
+    /// <summary>
+    /// Executes one closed distance-weighted compound-intermediate operator.
+    /// </summary>
+    /// <typeparam name="TOperator">The compound-intermediate operator.</typeparam>
+    private static void DistanceWeightedIntermediate<TOperator>(
+        Span<byte> destination,
+        int destinationStride,
+        ReadOnlySpan<ushort> first,
+        int firstStride,
+        ReadOnlySpan<ushort> second,
+        int secondStride,
+        int width,
+        int height,
+        int firstWeight,
+        int secondWeight,
+        int bitDepth)
+        where TOperator : struct, IAv1CompoundIntermediateDistanceWeightedOperator
+    {
+        GetIntermediateRounding(bitDepth, out int roundBits, out int roundOffset);
+        for (int row = 0; row < height; row++)
+        {
+            Span<byte> destinationRow = destination.Slice(row * destinationStride, width);
+            ReadOnlySpan<ushort> firstRow = first.Slice(row * firstStride, width);
+            ReadOnlySpan<ushort> secondRow = second.Slice(row * secondStride, width);
+            ref byte destinationReference = ref MemoryMarshal.GetReference(destinationRow);
+            ref ushort firstReference = ref MemoryMarshal.GetReference(firstRow);
+            ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow);
+            int column = 0;
+
+            if (Vector512.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector512<byte>.Count;
+                for (; column <= vectorEnd; column += Vector512<byte>.Count)
+                {
+                    Vector512<ushort> first0 = Vector512.LoadUnsafe(ref firstReference, (nuint)column);
+                    Vector512<ushort> first1 = Vector512.LoadUnsafe(ref firstReference, (nuint)(column + Vector512<ushort>.Count));
+                    Vector512<ushort> second0 = Vector512.LoadUnsafe(ref secondReference, (nuint)column);
+                    Vector512<ushort> second1 = Vector512.LoadUnsafe(ref secondReference, (nuint)(column + Vector512<ushort>.Count));
+                    TOperator.DistanceWeighted(
+                        first0,
+                        first1,
+                        second0,
+                        second1,
+                        firstWeight,
+                        secondWeight,
+                        roundBits,
+                        roundOffset).StoreUnsafe(ref destinationReference, (nuint)column);
+                }
+            }
+
+            if (Vector256.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector256<byte>.Count;
+                for (; column <= vectorEnd; column += Vector256<byte>.Count)
+                {
+                    Vector256<ushort> first0 = Vector256.LoadUnsafe(ref firstReference, (nuint)column);
+                    Vector256<ushort> first1 = Vector256.LoadUnsafe(ref firstReference, (nuint)(column + Vector256<ushort>.Count));
+                    Vector256<ushort> second0 = Vector256.LoadUnsafe(ref secondReference, (nuint)column);
+                    Vector256<ushort> second1 = Vector256.LoadUnsafe(ref secondReference, (nuint)(column + Vector256<ushort>.Count));
+                    TOperator.DistanceWeighted(
+                        first0,
+                        first1,
+                        second0,
+                        second1,
+                        firstWeight,
+                        secondWeight,
+                        roundBits,
+                        roundOffset).StoreUnsafe(ref destinationReference, (nuint)column);
+                }
+            }
+
+            if (Vector128.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector128<byte>.Count;
+                for (; column <= vectorEnd; column += Vector128<byte>.Count)
+                {
+                    Vector128<ushort> first0 = Vector128.LoadUnsafe(ref firstReference, (nuint)column);
+                    Vector128<ushort> first1 = Vector128.LoadUnsafe(
+                        ref firstReference,
+                        (nuint)(column + Vector128<ushort>.Count));
+
+                    Vector128<ushort> second0 = Vector128.LoadUnsafe(ref secondReference, (nuint)column);
+                    Vector128<ushort> second1 = Vector128.LoadUnsafe(
+                        ref secondReference,
+                        (nuint)(column + Vector128<ushort>.Count));
+
+                    TOperator.DistanceWeighted(
+                        first0,
+                        first1,
+                        second0,
+                        second1,
+                        firstWeight,
+                        secondWeight,
+                        roundBits,
+                        roundOffset).StoreUnsafe(ref destinationReference, (nuint)column);
+                }
+            }
+
+            for (; column < width; column++)
+            {
+                destinationRow[column] = TOperator.DistanceWeighted(
+                    firstRow[column],
+                    secondRow[column],
+                    firstWeight,
+                    secondWeight,
+                    roundBits,
+                    roundOffset);
+            }
+        }
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateMaskBlendPredictor.Operator.cs

@@ -0,0 +1,217 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.CompilerServices;
+using System.Runtime.Intrinsics;
+
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor;
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
+
+/// <content>
+/// Defines final masked compound-intermediate reconstruction.
+/// </content>
+internal static partial class Av1CompoundIntermediateMaskBlendPredictor
+{
+    /// <summary>
+    /// Defines masked compound-intermediate finalization for scalar and SIMD lane groups.
+    /// </summary>
+    private interface IAv1CompoundIntermediateMaskBlendOperator
+    {
+        /// <summary>
+        /// Alpha-blends and finalizes one pair of compound intermediate samples.
+        /// </summary>
+        /// <param name="first">The first compound intermediate.</param>
+        /// <param name="second">The second compound intermediate.</param>
+        /// <param name="alpha">The first-predictor weight in the AV1 mask range.</param>
+        /// <param name="roundBits">The final reconstruction shift.</param>
+        /// <param name="roundOffset">The compound intermediate bias.</param>
+        /// <returns>The reconstructed sample.</returns>
+        public static abstract byte Blend(ushort first, ushort second, byte alpha, int roundBits, int roundOffset);
+
+        /// <summary>
+        /// Alpha-blends and finalizes 128 bits of compound intermediate samples.
+        /// </summary>
+        /// <param name="first0">The lower first-predictor intermediates.</param>
+        /// <param name="first1">The upper first-predictor intermediates.</param>
+        /// <param name="second0">The lower second-predictor intermediates.</param>
+        /// <param name="second1">The upper second-predictor intermediates.</param>
+        /// <param name="alpha">The first-predictor weights in the AV1 mask range.</param>
+        /// <param name="roundBits">The final reconstruction shift.</param>
+        /// <param name="roundOffset">The compound intermediate bias.</param>
+        /// <returns>The reconstructed samples.</returns>
+        public static abstract Vector128<byte> Blend(
+            Vector128<ushort> first0,
+            Vector128<ushort> first1,
+            Vector128<ushort> second0,
+            Vector128<ushort> second1,
+            Vector128<byte> alpha,
+            int roundBits,
+            int roundOffset);
+
+        /// <summary>
+        /// Alpha-blends and finalizes 256 bits of compound intermediate samples.
+        /// </summary>
+        /// <param name="first0">The lower first-predictor intermediates.</param>
+        /// <param name="first1">The upper first-predictor intermediates.</param>
+        /// <param name="second0">The lower second-predictor intermediates.</param>
+        /// <param name="second1">The upper second-predictor intermediates.</param>
+        /// <param name="alpha">The first-predictor weights in the AV1 mask range.</param>
+        /// <param name="roundBits">The final reconstruction shift.</param>
+        /// <param name="roundOffset">The compound intermediate bias.</param>
+        /// <returns>The reconstructed samples.</returns>
+        public static abstract Vector256<byte> Blend(
+            Vector256<ushort> first0,
+            Vector256<ushort> first1,
+            Vector256<ushort> second0,
+            Vector256<ushort> second1,
+            Vector256<byte> alpha,
+            int roundBits,
+            int roundOffset);
+
+        /// <summary>
+        /// Alpha-blends and finalizes 512 bits of compound intermediate samples.
+        /// </summary>
+        /// <param name="first0">The lower first-predictor intermediates.</param>
+        /// <param name="first1">The upper first-predictor intermediates.</param>
+        /// <param name="second0">The lower second-predictor intermediates.</param>
+        /// <param name="second1">The upper second-predictor intermediates.</param>
+        /// <param name="alpha">The first-predictor weights in the AV1 mask range.</param>
+        /// <param name="roundBits">The final reconstruction shift.</param>
+        /// <param name="roundOffset">The compound intermediate bias.</param>
+        /// <returns>The reconstructed samples.</returns>
+        public static abstract Vector512<byte> Blend(
+            Vector512<ushort> first0,
+            Vector512<ushort> first1,
+            Vector512<ushort> second0,
+            Vector512<ushort> second1,
+            Vector512<byte> alpha,
+            int roundBits,
+            int roundOffset);
+    }
+
+    /// <summary>
+    /// Implements masked compound-intermediate finalization for scalar and SIMD lane groups.
+    /// </summary>
+    private readonly struct CompoundIntermediateMaskBlendOperator : IAv1CompoundIntermediateMaskBlendOperator
+    {
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static byte Blend(ushort first, ushort second, byte alpha, int roundBits, int roundOffset)
+        {
+            // The Q6 blend truncates because final pixel rounding is still pending after bias removal.
+            int result = ((alpha * first) + ((MaximumMaskAlpha - alpha) * second)) >> MaskWeightBits;
+            result -= roundOffset;
+            return (byte)Math.Clamp(RoundPowerOfTwo(result, roundBits), 0, byte.MaxValue);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector128<byte> Blend(
+            Vector128<ushort> first0,
+            Vector128<ushort> first1,
+            Vector128<ushort> second0,
+            Vector128<ushort> second1,
+            Vector128<byte> alpha,
+            int roundBits,
+            int roundOffset)
+            => Vector128.Narrow(
+                Blend(first0, second0, Vector128.WidenLower(alpha), roundBits, roundOffset),
+                Blend(first1, second1, Vector128.WidenUpper(alpha), roundBits, roundOffset));
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector256<byte> Blend(
+            Vector256<ushort> first0,
+            Vector256<ushort> first1,
+            Vector256<ushort> second0,
+            Vector256<ushort> second1,
+            Vector256<byte> alpha,
+            int roundBits,
+            int roundOffset)
+            => Vector256.Narrow(
+                Blend(first0, second0, Vector256.WidenLower(alpha), roundBits, roundOffset),
+                Blend(first1, second1, Vector256.WidenUpper(alpha), roundBits, roundOffset));
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector512<byte> Blend(
+            Vector512<ushort> first0,
+            Vector512<ushort> first1,
+            Vector512<ushort> second0,
+            Vector512<ushort> second1,
+            Vector512<byte> alpha,
+            int roundBits,
+            int roundOffset)
+            => Vector512.Narrow(
+                Blend(first0, second0, Vector512.WidenLower(alpha), roundBits, roundOffset),
+                Blend(first1, second1, Vector512.WidenUpper(alpha), roundBits, roundOffset));
+
+        /// <summary>
+        /// Alpha-blends 128-bit lanes after widening every product to signed 32-bit precision.
+        /// </summary>
+        private static Vector128<ushort> Blend(
+            Vector128<ushort> first,
+            Vector128<ushort> second,
+            Vector128<ushort> alpha,
+            int roundBits,
+            int roundOffset)
+        {
+            Vector128<int> firstLower = Vector128.WidenLower(first).AsInt32();
+            Vector128<int> firstUpper = Vector128.WidenUpper(first).AsInt32();
+            Vector128<int> secondLower = Vector128.WidenLower(second).AsInt32();
+            Vector128<int> secondUpper = Vector128.WidenUpper(second).AsInt32();
+            Vector128<int> alphaLower = Vector128.WidenLower(alpha).AsInt32();
+            Vector128<int> alphaUpper = Vector128.WidenUpper(alpha).AsInt32();
+            Vector128<int> maximum = Vector128.Create(MaximumMaskAlpha);
+            Vector128<int> lower = ((alphaLower * firstLower) + ((maximum - alphaLower) * secondLower)) >> MaskWeightBits;
+            Vector128<int> upper = ((alphaUpper * firstUpper) + ((maximum - alphaUpper) * secondUpper)) >> MaskWeightBits;
+            return Vector128.Narrow(FinalizeIntermediate(lower, roundBits, roundOffset), FinalizeIntermediate(upper, roundBits, roundOffset)).AsUInt16();
+        }
+
+        /// <summary>
+        /// Alpha-blends 256-bit lanes after widening every product to signed 32-bit precision.
+        /// </summary>
+        private static Vector256<ushort> Blend(
+            Vector256<ushort> first,
+            Vector256<ushort> second,
+            Vector256<ushort> alpha,
+            int roundBits,
+            int roundOffset)
+        {
+            Vector256<int> firstLower = Vector256.WidenLower(first).AsInt32();
+            Vector256<int> firstUpper = Vector256.WidenUpper(first).AsInt32();
+            Vector256<int> secondLower = Vector256.WidenLower(second).AsInt32();
+            Vector256<int> secondUpper = Vector256.WidenUpper(second).AsInt32();
+            Vector256<int> alphaLower = Vector256.WidenLower(alpha).AsInt32();
+            Vector256<int> alphaUpper = Vector256.WidenUpper(alpha).AsInt32();
+            Vector256<int> maximum = Vector256.Create(MaximumMaskAlpha);
+            Vector256<int> lower = ((alphaLower * firstLower) + ((maximum - alphaLower) * secondLower)) >> MaskWeightBits;
+            Vector256<int> upper = ((alphaUpper * firstUpper) + ((maximum - alphaUpper) * secondUpper)) >> MaskWeightBits;
+            return Vector256.Narrow(FinalizeIntermediate(lower, roundBits, roundOffset), FinalizeIntermediate(upper, roundBits, roundOffset)).AsUInt16();
+        }
+
+        /// <summary>
+        /// Alpha-blends 512-bit lanes after widening every product to signed 32-bit precision.
+        /// </summary>
+        private static Vector512<ushort> Blend(
+            Vector512<ushort> first,
+            Vector512<ushort> second,
+            Vector512<ushort> alpha,
+            int roundBits,
+            int roundOffset)
+        {
+            Vector512<int> firstLower = Vector512.WidenLower(first).AsInt32();
+            Vector512<int> firstUpper = Vector512.WidenUpper(first).AsInt32();
+            Vector512<int> secondLower = Vector512.WidenLower(second).AsInt32();
+            Vector512<int> secondUpper = Vector512.WidenUpper(second).AsInt32();
+            Vector512<int> alphaLower = Vector512.WidenLower(alpha).AsInt32();
+            Vector512<int> alphaUpper = Vector512.WidenUpper(alpha).AsInt32();
+            Vector512<int> maximum = Vector512.Create(MaximumMaskAlpha);
+            Vector512<int> lower = ((alphaLower * firstLower) + ((maximum - alphaLower) * secondLower)) >> MaskWeightBits;
+            Vector512<int> upper = ((alphaUpper * firstUpper) + ((maximum - alphaUpper) * secondUpper)) >> MaskWeightBits;
+            return Vector512.Narrow(FinalizeIntermediate(lower, roundBits, roundOffset), FinalizeIntermediate(upper, roundBits, roundOffset)).AsUInt16();
+        }
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateMaskBlendPredictor.cs

@@ -0,0 +1,189 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.CompilerServices;
+using System.Runtime.InteropServices;
+using System.Runtime.Intrinsics;
+
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor;
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
+
+/// <content>
+/// Reconstructs final samples by alpha-blending compound intermediates.
+/// </content>
+internal static partial class Av1CompoundIntermediateMaskBlendPredictor
+{
+    /// <summary>
+    /// Blends two compound intermediates through a luma-resolution mask.
+    /// </summary>
+    public static void BlendIntermediate(
+        Span<byte> destination,
+        int destinationStride,
+        ReadOnlySpan<ushort> first,
+        int firstStride,
+        ReadOnlySpan<ushort> second,
+        int secondStride,
+        ReadOnlySpan<byte> mask,
+        int maskStride,
+        int width,
+        int height,
+        int subX,
+        int subY,
+        int bitDepth)
+        => BlendIntermediate<CompoundIntermediateMaskBlendOperator>(
+            destination,
+            destinationStride,
+            first,
+            firstStride,
+            second,
+            secondStride,
+            mask,
+            maskStride,
+            width,
+            height,
+            subX,
+            subY,
+            bitDepth);
+
+    /// <summary>
+    /// Executes one closed alpha-blend compound-intermediate operator.
+    /// </summary>
+    /// <typeparam name="TOperator">The compound-intermediate operator.</typeparam>
+    private static void BlendIntermediate<TOperator>(
+        Span<byte> destination,
+        int destinationStride,
+        ReadOnlySpan<ushort> first,
+        int firstStride,
+        ReadOnlySpan<ushort> second,
+        int secondStride,
+        ReadOnlySpan<byte> mask,
+        int maskStride,
+        int width,
+        int height,
+        int subX,
+        int subY,
+        int bitDepth)
+        where TOperator : struct, IAv1CompoundIntermediateMaskBlendOperator
+    {
+        GetIntermediateRounding(bitDepth, out int roundBits, out int roundOffset);
+        for (int row = 0; row < height; row++)
+        {
+            Span<byte> destinationRow = destination.Slice(row * destinationStride, width);
+            ReadOnlySpan<ushort> firstRow = first.Slice(row * firstStride, width);
+            ReadOnlySpan<ushort> secondRow = second.Slice(row * secondStride, width);
+            ref byte destinationReference = ref MemoryMarshal.GetReference(destinationRow);
+            ref ushort firstReference = ref MemoryMarshal.GetReference(firstRow);
+            ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow);
+            int column = 0;
+
+            if (Vector512.IsHardwareAccelerated && subX == 0 && subY == 0)
+            {
+                ref byte maskReference = ref MemoryMarshal.GetReference(mask);
+                int maskRowOffset = row * maskStride;
+                int vectorEnd = width - Vector512<byte>.Count;
+                for (; column <= vectorEnd; column += Vector512<byte>.Count)
+                {
+                    Vector512<ushort> first0 = Vector512.LoadUnsafe(ref firstReference, (nuint)column);
+                    Vector512<ushort> first1 = Vector512.LoadUnsafe(ref firstReference, (nuint)(column + Vector512<ushort>.Count));
+                    Vector512<ushort> second0 = Vector512.LoadUnsafe(ref secondReference, (nuint)column);
+                    Vector512<ushort> second1 = Vector512.LoadUnsafe(ref secondReference, (nuint)(column + Vector512<ushort>.Count));
+                    Vector512<byte> alpha = Vector512.LoadUnsafe(ref maskReference, (nuint)(maskRowOffset + column));
+                    TOperator.Blend(first0, first1, second0, second1, alpha, roundBits, roundOffset)
+                        .StoreUnsafe(ref destinationReference, (nuint)column);
+                }
+            }
+
+            if (Vector256.IsHardwareAccelerated && subX == 0 && subY == 0)
+            {
+                ref byte maskReference = ref MemoryMarshal.GetReference(mask);
+                int maskRowOffset = row * maskStride;
+                int vectorEnd = width - Vector256<byte>.Count;
+                for (; column <= vectorEnd; column += Vector256<byte>.Count)
+                {
+                    Vector256<ushort> first0 = Vector256.LoadUnsafe(ref firstReference, (nuint)column);
+                    Vector256<ushort> first1 = Vector256.LoadUnsafe(ref firstReference, (nuint)(column + Vector256<ushort>.Count));
+                    Vector256<ushort> second0 = Vector256.LoadUnsafe(ref secondReference, (nuint)column);
+                    Vector256<ushort> second1 = Vector256.LoadUnsafe(ref secondReference, (nuint)(column + Vector256<ushort>.Count));
+                    Vector256<byte> alpha = Vector256.LoadUnsafe(ref maskReference, (nuint)(maskRowOffset + column));
+                    TOperator.Blend(first0, first1, second0, second1, alpha, roundBits, roundOffset)
+                        .StoreUnsafe(ref destinationReference, (nuint)column);
+                }
+            }
+
+            if (Vector128.IsHardwareAccelerated && subX == 0 && subY == 0)
+            {
+                ref byte maskReference = ref MemoryMarshal.GetReference(mask);
+                int maskRowOffset = row * maskStride;
+                int vectorEnd = width - Vector128<byte>.Count;
+                for (; column <= vectorEnd; column += Vector128<byte>.Count)
+                {
+                    Vector128<ushort> first0 = Vector128.LoadUnsafe(ref firstReference, (nuint)column);
+                    Vector128<ushort> first1 = Vector128.LoadUnsafe(
+                        ref firstReference,
+                        (nuint)(column + Vector128<ushort>.Count));
+
+                    Vector128<ushort> second0 = Vector128.LoadUnsafe(ref secondReference, (nuint)column);
+                    Vector128<ushort> second1 = Vector128.LoadUnsafe(
+                        ref secondReference,
+                        (nuint)(column + Vector128<ushort>.Count));
+
+                    Vector128<byte> alpha = Vector128.LoadUnsafe(
+                        ref maskReference,
+                        (nuint)(maskRowOffset + column));
+
+                    TOperator.Blend(
+                        first0,
+                        first1,
+                        second0,
+                        second1,
+                        alpha,
+                        roundBits,
+                        roundOffset).StoreUnsafe(ref destinationReference, (nuint)column);
+                }
+            }
+
+            for (; column < width; column++)
+            {
+                byte alpha = (byte)GetSubsampledMaskAlpha(mask, maskStride, row, column, subX, subY);
+                destinationRow[column] = TOperator.Blend(firstRow[column], secondRow[column], alpha, roundBits, roundOffset);
+            }
+        }
+    }
+
+    /// <summary>
+    /// Gets the mask alpha for one plane sample, averaging its two or four luma samples when required.
+    /// </summary>
+    private static int GetSubsampledMaskAlpha(
+        ReadOnlySpan<byte> mask,
+        int maskStride,
+        int row,
+        int column,
+        int subX,
+        int subY)
+    {
+        int maskRow = row << subY;
+        int maskColumn = column << subX;
+        int alpha = mask[(maskRow * maskStride) + maskColumn];
+        if (subX != 0)
+        {
+            alpha += mask[(maskRow * maskStride) + maskColumn + 1];
+        }
+
+        if (subY != 0)
+        {
+            int lowerOffset = ((maskRow + 1) * maskStride) + maskColumn;
+            alpha += mask[lowerOffset];
+            if (subX != 0)
+            {
+                alpha += mask[lowerOffset + 1];
+            }
+        }
+
+        int sampleCountShift = subX + subY;
+        return sampleCountShift == 0
+            ? alpha
+            : RoundPowerOfTwo(alpha, sampleCountShift);
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundMaskBlendPredictor.Operator.cs

@@ -0,0 +1,203 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.CompilerServices;
+using System.Runtime.Intrinsics;
+
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor;
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
+
+/// <content>
+/// Defines alpha-masked compound prediction arithmetic.
+/// </content>
+internal static partial class Av1CompoundMaskBlendPredictor
+{
+    /// <summary>
+    /// Defines alpha-masked compound blending for scalar and SIMD lane groups.
+    /// </summary>
+    private interface IAv1CompoundMaskBlendOperator
+    {
+        /// <summary>
+        /// Blends two 8-bit samples through an AV1 alpha value.
+        /// </summary>
+        /// <param name="first">The first sample.</param>
+        /// <param name="second">The second sample.</param>
+        /// <param name="alpha">The first-sample weight in the AV1 mask range.</param>
+        /// <returns>The blended sample.</returns>
+        public static abstract byte Blend(byte first, byte second, byte alpha);
+
+        /// <summary>
+        /// Blends two high-bit-depth samples through an AV1 alpha value.
+        /// </summary>
+        /// <param name="first">The first sample.</param>
+        /// <param name="second">The second sample.</param>
+        /// <param name="alpha">The first-sample weight in the AV1 mask range.</param>
+        /// <returns>The blended sample.</returns>
+        public static abstract ushort Blend(ushort first, ushort second, byte alpha);
+
+        /// <summary>
+        /// Blends 128-bit vectors of 8-bit samples through AV1 alpha values.
+        /// </summary>
+        /// <param name="first">The first samples.</param>
+        /// <param name="second">The second samples.</param>
+        /// <param name="alpha">The first-sample weights in the AV1 mask range.</param>
+        /// <returns>The blended samples.</returns>
+        public static abstract Vector128<byte> Blend(Vector128<byte> first, Vector128<byte> second, Vector128<byte> alpha);
+
+        /// <summary>
+        /// Blends 256-bit vectors of 8-bit samples through AV1 alpha values.
+        /// </summary>
+        /// <param name="first">The first samples.</param>
+        /// <param name="second">The second samples.</param>
+        /// <param name="alpha">The first-sample weights in the AV1 mask range.</param>
+        /// <returns>The blended samples.</returns>
+        public static abstract Vector256<byte> Blend(Vector256<byte> first, Vector256<byte> second, Vector256<byte> alpha);
+
+        /// <summary>
+        /// Blends 512-bit vectors of 8-bit samples through AV1 alpha values.
+        /// </summary>
+        /// <param name="first">The first samples.</param>
+        /// <param name="second">The second samples.</param>
+        /// <param name="alpha">The first-sample weights in the AV1 mask range.</param>
+        /// <returns>The blended samples.</returns>
+        public static abstract Vector512<byte> Blend(Vector512<byte> first, Vector512<byte> second, Vector512<byte> alpha);
+
+        /// <summary>
+        /// Blends 128-bit vectors of high-bit-depth samples through AV1 alpha values.
+        /// </summary>
+        /// <param name="first">The first samples.</param>
+        /// <param name="second">The second samples.</param>
+        /// <param name="alpha">The first-sample weights in the AV1 mask range.</param>
+        /// <returns>The blended samples.</returns>
+        public static abstract Vector128<ushort> Blend(Vector128<ushort> first, Vector128<ushort> second, Vector128<ushort> alpha);
+
+        /// <summary>
+        /// Blends 256-bit vectors of high-bit-depth samples through AV1 alpha values.
+        /// </summary>
+        /// <param name="first">The first samples.</param>
+        /// <param name="second">The second samples.</param>
+        /// <param name="alpha">The first-sample weights in the AV1 mask range.</param>
+        /// <returns>The blended samples.</returns>
+        public static abstract Vector256<ushort> Blend(Vector256<ushort> first, Vector256<ushort> second, Vector256<ushort> alpha);
+
+        /// <summary>
+        /// Blends 512-bit vectors of high-bit-depth samples through AV1 alpha values.
+        /// </summary>
+        /// <param name="first">The first samples.</param>
+        /// <param name="second">The second samples.</param>
+        /// <param name="alpha">The first-sample weights in the AV1 mask range.</param>
+        /// <returns>The blended samples.</returns>
+        public static abstract Vector512<ushort> Blend(Vector512<ushort> first, Vector512<ushort> second, Vector512<ushort> alpha);
+    }
+
+    /// <summary>
+    /// Implements AV1 alpha-mask blending for scalar and SIMD lane groups.
+    /// </summary>
+    private readonly struct CompoundMaskBlendOperator : IAv1CompoundMaskBlendOperator
+    {
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static byte Blend(byte first, byte second, byte alpha)
+            => (byte)(((alpha * first) + ((MaximumMaskAlpha - alpha) * second) + 32) >> MaskWeightBits);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static ushort Blend(ushort first, ushort second, byte alpha)
+            => (ushort)(((alpha * first) + ((MaximumMaskAlpha - alpha) * second) + 32) >> MaskWeightBits);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector128<byte> Blend(Vector128<byte> first, Vector128<byte> second, Vector128<byte> alpha)
+        {
+            Av1IntraPredictorBase.Widen(first, out Vector128<int> first0, out Vector128<int> first1, out Vector128<int> first2, out Vector128<int> first3);
+            Av1IntraPredictorBase.Widen(second, out Vector128<int> second0, out Vector128<int> second1, out Vector128<int> second2, out Vector128<int> second3);
+            Av1IntraPredictorBase.Widen(alpha, out Vector128<int> alpha0, out Vector128<int> alpha1, out Vector128<int> alpha2, out Vector128<int> alpha3);
+            return Av1IntraPredictorBase.Narrow(
+                Blend(first0, second0, alpha0),
+                Blend(first1, second1, alpha1),
+                Blend(first2, second2, alpha2),
+                Blend(first3, second3, alpha3));
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector256<byte> Blend(Vector256<byte> first, Vector256<byte> second, Vector256<byte> alpha)
+        {
+            Av1IntraPredictorBase.Widen(first, out Vector256<int> first0, out Vector256<int> first1, out Vector256<int> first2, out Vector256<int> first3);
+            Av1IntraPredictorBase.Widen(second, out Vector256<int> second0, out Vector256<int> second1, out Vector256<int> second2, out Vector256<int> second3);
+            Av1IntraPredictorBase.Widen(alpha, out Vector256<int> alpha0, out Vector256<int> alpha1, out Vector256<int> alpha2, out Vector256<int> alpha3);
+            return Av1IntraPredictorBase.Narrow(
+                Blend(first0, second0, alpha0),
+                Blend(first1, second1, alpha1),
+                Blend(first2, second2, alpha2),
+                Blend(first3, second3, alpha3));
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector512<byte> Blend(Vector512<byte> first, Vector512<byte> second, Vector512<byte> alpha)
+        {
+            Av1IntraPredictorBase.Widen(first, out Vector512<int> first0, out Vector512<int> first1, out Vector512<int> first2, out Vector512<int> first3);
+            Av1IntraPredictorBase.Widen(second, out Vector512<int> second0, out Vector512<int> second1, out Vector512<int> second2, out Vector512<int> second3);
+            Av1IntraPredictorBase.Widen(alpha, out Vector512<int> alpha0, out Vector512<int> alpha1, out Vector512<int> alpha2, out Vector512<int> alpha3);
+            return Av1IntraPredictorBase.Narrow(
+                Blend(first0, second0, alpha0),
+                Blend(first1, second1, alpha1),
+                Blend(first2, second2, alpha2),
+                Blend(first3, second3, alpha3));
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector128<ushort> Blend(Vector128<ushort> first, Vector128<ushort> second, Vector128<ushort> alpha)
+        {
+            Av1IntraPredictorBase.Widen(first.AsInt16(), out Vector128<int> first0, out Vector128<int> first1);
+            Av1IntraPredictorBase.Widen(second.AsInt16(), out Vector128<int> second0, out Vector128<int> second1);
+            Av1IntraPredictorBase.Widen(alpha.AsInt16(), out Vector128<int> alpha0, out Vector128<int> alpha1);
+            return Av1IntraPredictorBase.Narrow(Blend(first0, second0, alpha0), Blend(first1, second1, alpha1)).AsUInt16();
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector256<ushort> Blend(Vector256<ushort> first, Vector256<ushort> second, Vector256<ushort> alpha)
+        {
+            Av1IntraPredictorBase.Widen(first.AsInt16(), out Vector256<int> first0, out Vector256<int> first1);
+            Av1IntraPredictorBase.Widen(second.AsInt16(), out Vector256<int> second0, out Vector256<int> second1);
+            Av1IntraPredictorBase.Widen(alpha.AsInt16(), out Vector256<int> alpha0, out Vector256<int> alpha1);
+            return Av1IntraPredictorBase.Narrow(Blend(first0, second0, alpha0), Blend(first1, second1, alpha1)).AsUInt16();
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector512<ushort> Blend(Vector512<ushort> first, Vector512<ushort> second, Vector512<ushort> alpha)
+        {
+            Av1IntraPredictorBase.Widen(first.AsInt16(), out Vector512<int> first0, out Vector512<int> first1);
+            Av1IntraPredictorBase.Widen(second.AsInt16(), out Vector512<int> second0, out Vector512<int> second1);
+            Av1IntraPredictorBase.Widen(alpha.AsInt16(), out Vector512<int> alpha0, out Vector512<int> alpha1);
+            return Av1IntraPredictorBase.Narrow(Blend(first0, second0, alpha0), Blend(first1, second1, alpha1)).AsUInt16();
+        }
+
+        /// <summary>
+        /// Applies alpha-mask blending to 128-bit vectors of widened samples.
+        /// </summary>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        private static Vector128<int> Blend(Vector128<int> first, Vector128<int> second, Vector128<int> alpha)
+            => ((alpha * first) + ((Vector128.Create(MaximumMaskAlpha) - alpha) * second) + Vector128.Create(32)) >> MaskWeightBits;
+
+        /// <summary>
+        /// Applies alpha-mask blending to 256-bit vectors of widened samples.
+        /// </summary>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        private static Vector256<int> Blend(Vector256<int> first, Vector256<int> second, Vector256<int> alpha)
+            => ((alpha * first) + ((Vector256.Create(MaximumMaskAlpha) - alpha) * second) + Vector256.Create(32)) >> MaskWeightBits;
+
+        /// <summary>
+        /// Applies alpha-mask blending to 512-bit vectors of widened samples.
+        /// </summary>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        private static Vector512<int> Blend(Vector512<int> first, Vector512<int> second, Vector512<int> alpha)
+            => ((alpha * first) + ((Vector512.Create(MaximumMaskAlpha) - alpha) * second) + Vector512.Create(32)) >> MaskWeightBits;
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundMaskBlendPredictor.cs

@@ -0,0 +1,267 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.CompilerServices;
+using System.Runtime.InteropServices;
+using System.Runtime.Intrinsics;
+
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor;
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
+
+/// <content>
+/// Reconstructs alpha-masked compound prediction.
+/// </content>
+internal static partial class Av1CompoundMaskBlendPredictor
+{
+    /// <summary>
+    /// Blends two 8-bit predictors through a contiguous AV1 alpha mask.
+    /// </summary>
+    public static void Blend(
+        Span<byte> destination,
+        int destinationStride,
+        ReadOnlySpan<byte> second,
+        int secondStride,
+        ReadOnlySpan<byte> mask,
+        int maskStride,
+        int width,
+        int height)
+        => Blend<CompoundMaskBlendOperator>(
+            destination,
+            destinationStride,
+            second,
+            secondStride,
+            mask,
+            maskStride,
+            width,
+            height);
+
+    /// <summary>
+    /// Executes one closed 8-bit masked compound operator.
+    /// </summary>
+    /// <typeparam name="TOperator">The compound arithmetic operator.</typeparam>
+    private static void Blend<TOperator>(
+        Span<byte> destination,
+        int destinationStride,
+        ReadOnlySpan<byte> second,
+        int secondStride,
+        ReadOnlySpan<byte> mask,
+        int maskStride,
+        int width,
+        int height)
+        where TOperator : struct, IAv1CompoundMaskBlendOperator
+    {
+        for (int row = 0; row < height; row++)
+        {
+            Span<byte> destinationRow = destination.Slice(row * destinationStride, width);
+            ReadOnlySpan<byte> secondRow = second.Slice(row * secondStride, width);
+            ReadOnlySpan<byte> maskRow = mask.Slice(row * maskStride, width);
+            ref byte destinationReference = ref MemoryMarshal.GetReference(destinationRow);
+            ref byte secondReference = ref MemoryMarshal.GetReference(secondRow);
+            ref byte maskReference = ref MemoryMarshal.GetReference(maskRow);
+            int column = 0;
+
+            if (Vector512.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector512<byte>.Count;
+                for (; column <= vectorEnd; column += Vector512<byte>.Count)
+                {
+                    Vector512<byte> firstVector = Vector512.LoadUnsafe(ref destinationReference, (nuint)column);
+                    Vector512<byte> secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column);
+                    Vector512<byte> maskVector = Vector512.LoadUnsafe(ref maskReference, (nuint)column);
+                    TOperator.Blend(firstVector, secondVector, maskVector).StoreUnsafe(ref destinationReference, (nuint)column);
+                }
+            }
+
+            if (Vector256.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector256<byte>.Count;
+                for (; column <= vectorEnd; column += Vector256<byte>.Count)
+                {
+                    Vector256<byte> firstVector = Vector256.LoadUnsafe(ref destinationReference, (nuint)column);
+                    Vector256<byte> secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column);
+                    Vector256<byte> maskVector = Vector256.LoadUnsafe(ref maskReference, (nuint)column);
+                    TOperator.Blend(firstVector, secondVector, maskVector).StoreUnsafe(ref destinationReference, (nuint)column);
+                }
+            }
+
+            if (Vector128.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector128<byte>.Count;
+                for (; column <= vectorEnd; column += Vector128<byte>.Count)
+                {
+                    Vector128<byte> firstVector = Vector128.LoadUnsafe(ref destinationReference, (nuint)column);
+                    Vector128<byte> secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column);
+                    Vector128<byte> maskVector = Vector128.LoadUnsafe(ref maskReference, (nuint)column);
+                    TOperator.Blend(firstVector, secondVector, maskVector).StoreUnsafe(ref destinationReference, (nuint)column);
+                }
+            }
+
+            for (; column < width; column++)
+            {
+                destinationRow[column] = TOperator.Blend(destinationRow[column], secondRow[column], maskRow[column]);
+            }
+        }
+    }
+
+    /// <summary>
+    /// Blends two high-bit-depth predictors through a contiguous AV1 alpha mask.
+    /// </summary>
+    public static void Blend(
+        Span<ushort> destination,
+        int destinationStride,
+        ReadOnlySpan<ushort> second,
+        int secondStride,
+        ReadOnlySpan<byte> mask,
+        int maskStride,
+        int width,
+        int height)
+        => Blend<CompoundMaskBlendOperator>(
+            destination,
+            destinationStride,
+            second,
+            secondStride,
+            mask,
+            maskStride,
+            width,
+            height);
+
+    /// <summary>
+    /// Executes one closed high-bit-depth masked compound operator.
+    /// </summary>
+    /// <typeparam name="TOperator">The compound arithmetic operator.</typeparam>
+    private static void Blend<TOperator>(
+        Span<ushort> destination,
+        int destinationStride,
+        ReadOnlySpan<ushort> second,
+        int secondStride,
+        ReadOnlySpan<byte> mask,
+        int maskStride,
+        int width,
+        int height)
+        where TOperator : struct, IAv1CompoundMaskBlendOperator
+    {
+        for (int row = 0; row < height; row++)
+        {
+            Span<ushort> destinationRow = destination.Slice(row * destinationStride, width);
+            ReadOnlySpan<ushort> secondRow = second.Slice(row * secondStride, width);
+            ReadOnlySpan<byte> maskRow = mask.Slice(row * maskStride, width);
+            ref ushort destinationReference = ref MemoryMarshal.GetReference(destinationRow);
+            ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow);
+            ref byte maskReference = ref MemoryMarshal.GetReference(maskRow);
+            int column = 0;
+
+            if (Vector512.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector512<ushort>.Count;
+                for (; column <= vectorEnd; column += Vector512<ushort>.Count)
+                {
+                    Vector512<ushort> firstVector = Vector512.LoadUnsafe(ref destinationReference, (nuint)column);
+                    Vector512<ushort> secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column);
+                    Vector512<ushort> maskVector = LoadMask512(ref maskReference, column);
+                    TOperator.Blend(firstVector, secondVector, maskVector).StoreUnsafe(ref destinationReference, (nuint)column);
+                }
+            }
+
+            if (Vector256.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector256<ushort>.Count;
+                for (; column <= vectorEnd; column += Vector256<ushort>.Count)
+                {
+                    Vector256<ushort> firstVector = Vector256.LoadUnsafe(ref destinationReference, (nuint)column);
+                    Vector256<ushort> secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column);
+                    Vector256<ushort> maskVector = LoadMask256(ref maskReference, column);
+                    TOperator.Blend(firstVector, secondVector, maskVector).StoreUnsafe(ref destinationReference, (nuint)column);
+                }
+            }
+
+            if (Vector128.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector128<ushort>.Count;
+                for (; column <= vectorEnd; column += Vector128<ushort>.Count)
+                {
+                    Vector128<ushort> firstVector = Vector128.LoadUnsafe(ref destinationReference, (nuint)column);
+                    Vector128<ushort> secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column);
+                    Vector128<ushort> maskVector = LoadMask128(ref maskReference, column);
+                    TOperator.Blend(firstVector, secondVector, maskVector).StoreUnsafe(ref destinationReference, (nuint)column);
+                }
+            }
+
+            for (; column < width; column++)
+            {
+                destinationRow[column] = TOperator.Blend(destinationRow[column], secondRow[column], maskRow[column]);
+            }
+        }
+    }
+
+    /// <summary>
+    /// Blends two 8-bit predictors through an alpha mask without explicit hardware intrinsics.
+    /// </summary>
+    public static void BlendScalar(
+        Span<byte> destination,
+        int destinationStride,
+        ReadOnlySpan<byte> second,
+        int secondStride,
+        ReadOnlySpan<byte> mask,
+        int maskStride,
+        int width,
+        int height)
+        => BlendScalar<CompoundMaskBlendOperator>(
+            destination,
+            destinationStride,
+            second,
+            secondStride,
+            mask,
+            maskStride,
+            width,
+            height);
+
+    /// <summary>
+    /// Executes one closed 8-bit masked compound operator without explicit hardware intrinsics.
+    /// </summary>
+    /// <typeparam name="TOperator">The compound arithmetic operator.</typeparam>
+    private static void BlendScalar<TOperator>(
+        Span<byte> destination,
+        int destinationStride,
+        ReadOnlySpan<byte> second,
+        int secondStride,
+        ReadOnlySpan<byte> mask,
+        int maskStride,
+        int width,
+        int height)
+        where TOperator : struct, IAv1CompoundMaskBlendOperator
+    {
+        for (int row = 0; row < height; row++)
+        {
+            Span<byte> destinationRow = destination.Slice(row * destinationStride, width);
+            ReadOnlySpan<byte> secondRow = second.Slice(row * secondStride, width);
+            ReadOnlySpan<byte> maskRow = mask.Slice(row * maskStride, width);
+            for (int column = 0; column < width; column++)
+            {
+                destinationRow[column] = TOperator.Blend(destinationRow[column], secondRow[column], maskRow[column]);
+            }
+        }
+    }
+
+    [MethodImpl(MethodImplOptions.AggressiveInlining)]
+    private static Vector128<ushort> LoadMask128(ref byte source, int offset)
+    {
+        Vector64<byte> packed = Unsafe.As<byte, Vector64<byte>>(ref Unsafe.Add(ref source, offset));
+        return Vector128.WidenLower(Vector128.Create(packed, Vector64<byte>.Zero));
+    }
+
+    [MethodImpl(MethodImplOptions.AggressiveInlining)]
+    private static Vector256<ushort> LoadMask256(ref byte source, int offset)
+    {
+        Vector128<byte> packed = Vector128.LoadUnsafe(ref source, (nuint)offset);
+        return Vector256.WidenLower(Vector256.Create(packed, Vector128<byte>.Zero));
+    }
+
+    [MethodImpl(MethodImplOptions.AggressiveInlining)]
+    private static Vector512<ushort> LoadMask512(ref byte source, int offset)
+    {
+        Vector256<byte> packed = Vector256.LoadUnsafe(ref source, (nuint)offset);
+        return Vector512.WidenLower(Vector512.Create(packed, Vector256<byte>.Zero));
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1DifferenceWeightedMaskBuilder.Operator.cs

@@ -0,0 +1,254 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.CompilerServices;
+using System.Runtime.Intrinsics;
+
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor;
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
+
+/// <content>
+/// Defines difference-weighted compound mask arithmetic.
+/// </content>
+internal static partial class Av1DifferenceWeightedMaskBuilder
+{
+    /// <summary>
+    /// Defines difference-weighted compound mask generation for scalar and SIMD lane groups.
+    /// </summary>
+    private interface IAv1DifferenceWeightedMaskOperator
+    {
+        /// <summary>
+        /// Creates one mask value from two 8-bit predictor samples.
+        /// </summary>
+        /// <param name="first">The first predictor sample.</param>
+        /// <param name="second">The second predictor sample.</param>
+        /// <param name="shift">The difference scaling shift.</param>
+        /// <param name="invert">Whether to invert the selected predictor.</param>
+        /// <returns>The AV1 mask value.</returns>
+        public static abstract byte Create(byte first, byte second, int shift, bool invert);
+
+        /// <summary>
+        /// Creates one mask value from two high-bit-depth predictor samples.
+        /// </summary>
+        /// <param name="first">The first predictor sample.</param>
+        /// <param name="second">The second predictor sample.</param>
+        /// <param name="shift">The difference scaling shift.</param>
+        /// <param name="invert">Whether to invert the selected predictor.</param>
+        /// <returns>The AV1 mask value.</returns>
+        public static abstract byte Create(ushort first, ushort second, int shift, bool invert);
+
+        /// <summary>
+        /// Creates 128 bits of mask values from 8-bit predictor samples.
+        /// </summary>
+        /// <param name="first">The first predictor samples.</param>
+        /// <param name="second">The second predictor samples.</param>
+        /// <param name="shift">The difference scaling shift.</param>
+        /// <param name="invert">Whether to invert the selected predictor.</param>
+        /// <returns>The AV1 mask values.</returns>
+        public static abstract Vector128<byte> Create(Vector128<byte> first, Vector128<byte> second, int shift, bool invert);
+
+        /// <summary>
+        /// Creates 256 bits of mask values from 8-bit predictor samples.
+        /// </summary>
+        /// <param name="first">The first predictor samples.</param>
+        /// <param name="second">The second predictor samples.</param>
+        /// <param name="shift">The difference scaling shift.</param>
+        /// <param name="invert">Whether to invert the selected predictor.</param>
+        /// <returns>The AV1 mask values.</returns>
+        public static abstract Vector256<byte> Create(Vector256<byte> first, Vector256<byte> second, int shift, bool invert);
+
+        /// <summary>
+        /// Creates 512 bits of mask values from 8-bit predictor samples.
+        /// </summary>
+        /// <param name="first">The first predictor samples.</param>
+        /// <param name="second">The second predictor samples.</param>
+        /// <param name="shift">The difference scaling shift.</param>
+        /// <param name="invert">Whether to invert the selected predictor.</param>
+        /// <returns>The AV1 mask values.</returns>
+        public static abstract Vector512<byte> Create(Vector512<byte> first, Vector512<byte> second, int shift, bool invert);
+
+        /// <summary>
+        /// Creates 128 bits of packed mask values from high-bit-depth predictor samples.
+        /// </summary>
+        /// <param name="first0">The lower first-predictor samples.</param>
+        /// <param name="first1">The upper first-predictor samples.</param>
+        /// <param name="second0">The lower second-predictor samples.</param>
+        /// <param name="second1">The upper second-predictor samples.</param>
+        /// <param name="shift">The difference scaling shift.</param>
+        /// <param name="invert">Whether to invert the selected predictor.</param>
+        /// <returns>The packed AV1 mask values.</returns>
+        public static abstract Vector128<byte> Create(
+            Vector128<ushort> first0,
+            Vector128<ushort> first1,
+            Vector128<ushort> second0,
+            Vector128<ushort> second1,
+            int shift,
+            bool invert);
+
+        /// <summary>
+        /// Creates 256 bits of packed mask values from high-bit-depth predictor samples.
+        /// </summary>
+        /// <param name="first0">The lower first-predictor samples.</param>
+        /// <param name="first1">The upper first-predictor samples.</param>
+        /// <param name="second0">The lower second-predictor samples.</param>
+        /// <param name="second1">The upper second-predictor samples.</param>
+        /// <param name="shift">The difference scaling shift.</param>
+        /// <param name="invert">Whether to invert the selected predictor.</param>
+        /// <returns>The packed AV1 mask values.</returns>
+        public static abstract Vector256<byte> Create(
+            Vector256<ushort> first0,
+            Vector256<ushort> first1,
+            Vector256<ushort> second0,
+            Vector256<ushort> second1,
+            int shift,
+            bool invert);
+
+        /// <summary>
+        /// Creates 512 bits of packed mask values from high-bit-depth predictor samples.
+        /// </summary>
+        /// <param name="first0">The lower first-predictor samples.</param>
+        /// <param name="first1">The upper first-predictor samples.</param>
+        /// <param name="second0">The lower second-predictor samples.</param>
+        /// <param name="second1">The upper second-predictor samples.</param>
+        /// <param name="shift">The difference scaling shift.</param>
+        /// <param name="invert">Whether to invert the selected predictor.</param>
+        /// <returns>The packed AV1 mask values.</returns>
+        public static abstract Vector512<byte> Create(
+            Vector512<ushort> first0,
+            Vector512<ushort> first1,
+            Vector512<ushort> second0,
+            Vector512<ushort> second1,
+            int shift,
+            bool invert);
+    }
+
+    /// <summary>
+    /// Implements AV1 difference-weighted mask generation for scalar and SIMD lane groups.
+    /// </summary>
+    private readonly struct DifferenceWeightedMaskOperator : IAv1DifferenceWeightedMaskOperator
+    {
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static byte Create(byte first, byte second, int shift, bool invert)
+            => Create((ushort)Math.Abs(first - second), shift, invert);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static byte Create(ushort first, ushort second, int shift, bool invert)
+            => Create((ushort)Math.Abs(first - second), shift, invert);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector128<byte> Create(Vector128<byte> first, Vector128<byte> second, int shift, bool invert)
+        {
+            Vector128<byte> difference = Vector128.Max(first, second) - Vector128.Min(first, second);
+            Vector128<ushort> lower = CreateAlpha(Vector128.WidenLower(difference), shift, invert);
+            Vector128<ushort> upper = CreateAlpha(Vector128.WidenUpper(difference), shift, invert);
+            return Vector128.Narrow(lower, upper);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector256<byte> Create(Vector256<byte> first, Vector256<byte> second, int shift, bool invert)
+        {
+            Vector256<byte> difference = Vector256.Max(first, second) - Vector256.Min(first, second);
+            Vector256<ushort> lower = CreateAlpha(Vector256.WidenLower(difference), shift, invert);
+            Vector256<ushort> upper = CreateAlpha(Vector256.WidenUpper(difference), shift, invert);
+            return Vector256.Narrow(lower, upper);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector512<byte> Create(Vector512<byte> first, Vector512<byte> second, int shift, bool invert)
+        {
+            Vector512<byte> difference = Vector512.Max(first, second) - Vector512.Min(first, second);
+            Vector512<ushort> lower = CreateAlpha(Vector512.WidenLower(difference), shift, invert);
+            Vector512<ushort> upper = CreateAlpha(Vector512.WidenUpper(difference), shift, invert);
+            return Vector512.Narrow(lower, upper);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector128<byte> Create(
+            Vector128<ushort> first0,
+            Vector128<ushort> first1,
+            Vector128<ushort> second0,
+            Vector128<ushort> second1,
+            int shift,
+            bool invert)
+            => Vector128.Narrow(
+                CreateAlpha(Vector128.Max(first0, second0) - Vector128.Min(first0, second0), shift, invert),
+                CreateAlpha(Vector128.Max(first1, second1) - Vector128.Min(first1, second1), shift, invert));
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector256<byte> Create(
+            Vector256<ushort> first0,
+            Vector256<ushort> first1,
+            Vector256<ushort> second0,
+            Vector256<ushort> second1,
+            int shift,
+            bool invert)
+            => Vector256.Narrow(
+                CreateAlpha(Vector256.Max(first0, second0) - Vector256.Min(first0, second0), shift, invert),
+                CreateAlpha(Vector256.Max(first1, second1) - Vector256.Min(first1, second1), shift, invert));
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector512<byte> Create(
+            Vector512<ushort> first0,
+            Vector512<ushort> first1,
+            Vector512<ushort> second0,
+            Vector512<ushort> second1,
+            int shift,
+            bool invert)
+            => Vector512.Narrow(
+                CreateAlpha(Vector512.Max(first0, second0) - Vector512.Min(first0, second0), shift, invert),
+                CreateAlpha(Vector512.Max(first1, second1) - Vector512.Min(first1, second1), shift, invert));
+
+        /// <summary>
+        /// Creates one mask value from an absolute predictor difference.
+        /// </summary>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        private static byte Create(ushort difference, int shift, bool invert)
+        {
+            int alpha = Math.Min(MaximumMaskAlpha, 38 + (difference >> shift));
+            return (byte)(invert ? MaximumMaskAlpha - alpha : alpha);
+        }
+
+        /// <summary>
+        /// Creates 128-bit vectors of unpacked mask values from absolute predictor differences.
+        /// </summary>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        private static Vector128<ushort> CreateAlpha(Vector128<ushort> difference, int shift, bool invert)
+        {
+            Vector128<ushort> maximum = Vector128.Create((ushort)MaximumMaskAlpha);
+            Vector128<ushort> alpha = Vector128.Min(maximum, (difference >> shift) + Vector128.Create((ushort)38));
+            return invert ? maximum - alpha : alpha;
+        }
+
+        /// <summary>
+        /// Creates 256-bit vectors of unpacked mask values from absolute predictor differences.
+        /// </summary>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        private static Vector256<ushort> CreateAlpha(Vector256<ushort> difference, int shift, bool invert)
+        {
+            Vector256<ushort> maximum = Vector256.Create((ushort)MaximumMaskAlpha);
+            Vector256<ushort> alpha = Vector256.Min(maximum, (difference >> shift) + Vector256.Create((ushort)38));
+            return invert ? maximum - alpha : alpha;
+        }
+
+        /// <summary>
+        /// Creates 512-bit vectors of unpacked mask values from absolute predictor differences.
+        /// </summary>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        private static Vector512<ushort> CreateAlpha(Vector512<ushort> difference, int shift, bool invert)
+        {
+            Vector512<ushort> maximum = Vector512.Create((ushort)MaximumMaskAlpha);
+            Vector512<ushort> alpha = Vector512.Min(maximum, (difference >> shift) + Vector512.Create((ushort)38));
+            return invert ? maximum - alpha : alpha;
+        }
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1DifferenceWeightedMaskBuilder.cs

@@ -0,0 +1,214 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.InteropServices;
+using System.Runtime.Intrinsics;
+using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling;
+
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor;
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
+
+/// <content>
+/// Builds AV1 difference-weighted compound masks.
+/// </content>
+internal static partial class Av1DifferenceWeightedMaskBuilder
+{
+    /// <summary>
+    /// Fills an 8-bit difference-weighted compound mask.
+    /// </summary>
+    public static void FillDifferenceWeightedMask(
+        Span<byte> mask,
+        int maskStride,
+        ReadOnlySpan<byte> first,
+        int firstStride,
+        ReadOnlySpan<byte> second,
+        int secondStride,
+        int width,
+        int height,
+        Av1DifferenceWeightedMaskType maskType)
+        => FillDifferenceWeightedMask<DifferenceWeightedMaskOperator>(
+            mask,
+            maskStride,
+            first,
+            firstStride,
+            second,
+            secondStride,
+            width,
+            height,
+            maskType);
+
+    /// <summary>
+    /// Executes one closed 8-bit difference-weighted mask operator.
+    /// </summary>
+    /// <typeparam name="TOperator">The difference-weighted mask operator.</typeparam>
+    private static void FillDifferenceWeightedMask<TOperator>(
+        Span<byte> mask,
+        int maskStride,
+        ReadOnlySpan<byte> first,
+        int firstStride,
+        ReadOnlySpan<byte> second,
+        int secondStride,
+        int width,
+        int height,
+        Av1DifferenceWeightedMaskType maskType)
+        where TOperator : struct, IAv1DifferenceWeightedMaskOperator
+    {
+        bool invert = maskType == Av1DifferenceWeightedMaskType.Type38Inverse;
+        for (int row = 0; row < height; row++)
+        {
+            Span<byte> maskRow = mask.Slice(row * maskStride, width);
+            ReadOnlySpan<byte> firstRow = first.Slice(row * firstStride, width);
+            ReadOnlySpan<byte> secondRow = second.Slice(row * secondStride, width);
+            ref byte maskReference = ref MemoryMarshal.GetReference(maskRow);
+            ref byte firstReference = ref MemoryMarshal.GetReference(firstRow);
+            ref byte secondReference = ref MemoryMarshal.GetReference(secondRow);
+            int column = 0;
+
+            if (Vector512.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector512<byte>.Count;
+                for (; column <= vectorEnd; column += Vector512<byte>.Count)
+                {
+                    Vector512<byte> firstVector = Vector512.LoadUnsafe(ref firstReference, (nuint)column);
+                    Vector512<byte> secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column);
+                    TOperator.Create(firstVector, secondVector, 4, invert).StoreUnsafe(ref maskReference, (nuint)column);
+                }
+            }
+
+            if (Vector256.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector256<byte>.Count;
+                for (; column <= vectorEnd; column += Vector256<byte>.Count)
+                {
+                    Vector256<byte> firstVector = Vector256.LoadUnsafe(ref firstReference, (nuint)column);
+                    Vector256<byte> secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column);
+                    TOperator.Create(firstVector, secondVector, 4, invert).StoreUnsafe(ref maskReference, (nuint)column);
+                }
+            }
+
+            if (Vector128.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector128<byte>.Count;
+                for (; column <= vectorEnd; column += Vector128<byte>.Count)
+                {
+                    Vector128<byte> firstVector = Vector128.LoadUnsafe(ref firstReference, (nuint)column);
+                    Vector128<byte> secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column);
+                    TOperator.Create(firstVector, secondVector, 4, invert).StoreUnsafe(ref maskReference, (nuint)column);
+                }
+            }
+
+            for (; column < width; column++)
+            {
+                maskRow[column] = TOperator.Create(firstRow[column], secondRow[column], 4, invert);
+            }
+        }
+    }
+
+    /// <summary>
+    /// Fills a high-bit-depth difference-weighted compound mask.
+    /// </summary>
+    public static void FillDifferenceWeightedMask(
+        Span<byte> mask,
+        int maskStride,
+        ReadOnlySpan<ushort> first,
+        int firstStride,
+        ReadOnlySpan<ushort> second,
+        int secondStride,
+        int width,
+        int height,
+        int bitDepth,
+        Av1DifferenceWeightedMaskType maskType)
+        => FillDifferenceWeightedMask<DifferenceWeightedMaskOperator>(
+            mask,
+            maskStride,
+            first,
+            firstStride,
+            second,
+            secondStride,
+            width,
+            height,
+            bitDepth,
+            maskType);
+
+    /// <summary>
+    /// Executes one closed high-bit-depth difference-weighted mask operator.
+    /// </summary>
+    /// <typeparam name="TOperator">The difference-weighted mask operator.</typeparam>
+    private static void FillDifferenceWeightedMask<TOperator>(
+        Span<byte> mask,
+        int maskStride,
+        ReadOnlySpan<ushort> first,
+        int firstStride,
+        ReadOnlySpan<ushort> second,
+        int secondStride,
+        int width,
+        int height,
+        int bitDepth,
+        Av1DifferenceWeightedMaskType maskType)
+        where TOperator : struct, IAv1DifferenceWeightedMaskOperator
+    {
+        bool invert = maskType == Av1DifferenceWeightedMaskType.Type38Inverse;
+        int differenceShift = bitDepth - 8 + 4;
+        for (int row = 0; row < height; row++)
+        {
+            Span<byte> maskRow = mask.Slice(row * maskStride, width);
+            ReadOnlySpan<ushort> firstRow = first.Slice(row * firstStride, width);
+            ReadOnlySpan<ushort> secondRow = second.Slice(row * secondStride, width);
+            ref byte maskReference = ref MemoryMarshal.GetReference(maskRow);
+            ref ushort firstReference = ref MemoryMarshal.GetReference(firstRow);
+            ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow);
+            int column = 0;
+
+            // Two input vectors narrow to one packed byte mask. This keeps mask construction contiguous and avoids
+            // temporary buffers before the following vector blend consumes the complete plane block.
+            if (Vector512.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector512<byte>.Count;
+                for (; column <= vectorEnd; column += Vector512<byte>.Count)
+                {
+                    Vector512<ushort> first0 = Vector512.LoadUnsafe(ref firstReference, (nuint)column);
+                    Vector512<ushort> first1 = Vector512.LoadUnsafe(ref firstReference, (nuint)(column + Vector512<ushort>.Count));
+                    Vector512<ushort> second0 = Vector512.LoadUnsafe(ref secondReference, (nuint)column);
+                    Vector512<ushort> second1 = Vector512.LoadUnsafe(ref secondReference, (nuint)(column + Vector512<ushort>.Count));
+                    TOperator.Create(first0, first1, second0, second1, differenceShift, invert)
+                        .StoreUnsafe(ref maskReference, (nuint)column);
+                }
+            }
+
+            if (Vector256.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector256<byte>.Count;
+                for (; column <= vectorEnd; column += Vector256<byte>.Count)
+                {
+                    Vector256<ushort> first0 = Vector256.LoadUnsafe(ref firstReference, (nuint)column);
+                    Vector256<ushort> first1 = Vector256.LoadUnsafe(ref firstReference, (nuint)(column + Vector256<ushort>.Count));
+                    Vector256<ushort> second0 = Vector256.LoadUnsafe(ref secondReference, (nuint)column);
+                    Vector256<ushort> second1 = Vector256.LoadUnsafe(ref secondReference, (nuint)(column + Vector256<ushort>.Count));
+                    TOperator.Create(first0, first1, second0, second1, differenceShift, invert)
+                        .StoreUnsafe(ref maskReference, (nuint)column);
+                }
+            }
+
+            if (Vector128.IsHardwareAccelerated)
+            {
+                int vectorEnd = width - Vector128<byte>.Count;
+                for (; column <= vectorEnd; column += Vector128<byte>.Count)
+                {
+                    Vector128<ushort> first0 = Vector128.LoadUnsafe(ref firstReference, (nuint)column);
+                    Vector128<ushort> first1 = Vector128.LoadUnsafe(ref firstReference, (nuint)(column + Vector128<ushort>.Count));
+                    Vector128<ushort> second0 = Vector128.LoadUnsafe(ref secondReference, (nuint)column);
+                    Vector128<ushort> second1 = Vector128.LoadUnsafe(ref secondReference, (nuint)(column + Vector128<ushort>.Count));
+                    TOperator.Create(first0, first1, second0, second1, differenceShift, invert)
+                        .StoreUnsafe(ref maskReference, (nuint)column);
+                }
+            }
+
+            for (; column < width; column++)
+            {
+                maskRow[column] = TOperator.Create(firstRow[column], secondRow[column], differenceShift, invert);
+            }
+        }
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterIntraMaskBuilder.cs

@@ -0,0 +1,62 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.InteropServices;
+using System.Runtime.Intrinsics;
+using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling;
+
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor;
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
+
+/// <content>
+/// Builds AV1 inter-intra prediction masks.
+/// </content>
+internal static partial class Av1InterIntraMaskBuilder
+{
+    /// <summary>
+    /// Gets libaom's one-dimensional inter-intra alpha curve.
+    /// </summary>
+    private static ReadOnlySpan<byte> InterIntraWeights =>
+    [
+        60, 58, 56, 54, 52, 50, 48, 47, 45, 44, 42, 41, 39, 38, 37, 35,
+        34, 33, 32, 31, 30, 29, 28, 27, 26, 25, 24, 23, 22, 22, 21, 20,
+        19, 19, 18, 18, 17, 16, 16, 15, 15, 14, 14, 13, 13, 12, 12, 12,
+        11, 11, 10, 10, 10, 9, 9, 9, 8, 8, 8, 8, 7, 7, 7, 7,
+        6, 6, 6, 6, 6, 5, 5, 5, 5, 5, 4, 4, 4, 4, 4, 4,
+        4, 4, 3, 3, 3, 3, 3, 3, 3, 3, 3, 2, 2, 2, 2, 2,
+        2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1,
+        1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
+    ];
+
+    /// <summary>
+    /// Fills a smooth inter-intra mask for one plane.
+    /// </summary>
+    public static void FillInterIntraMask(
+        Span<byte> mask,
+        int maskStride,
+        int width,
+        int height,
+        Av1InterIntraMode mode,
+        bool invert)
+    {
+        int sizeScale = 128 / Math.Max(width, height);
+        for (int row = 0; row < height; row++)
+        {
+            Span<byte> maskRow = mask.Slice(row * maskStride, width);
+            for (int column = 0; column < width; column++)
+            {
+                int alpha = mode switch
+                {
+                    Av1InterIntraMode.Vertical => InterIntraWeights[row * sizeScale],
+                    Av1InterIntraMode.Horizontal => InterIntraWeights[column * sizeScale],
+                    Av1InterIntraMode.Smooth => InterIntraWeights[Math.Min(row, column) * sizeScale],
+                    _ => 32,
+                };
+
+                maskRow[column] = (byte)(invert ? MaximumMaskAlpha - alpha : alpha);
+            }
+        }
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Arithmetic.cs

@@ -15,7 +15,7 @@ internal static partial class Av1InterPredictor
     /// Convolves sixteen adjacent 8-bit samples into four signed 32-bit accumulator vectors.
     /// </summary>
     [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static void Convolve(
+    internal static void Convolve(
         ref byte source,
         int tapStride,
         nuint column,
@@ -51,7 +51,7 @@ internal static partial class Av1InterPredictor
     /// Convolves thirty-two adjacent 8-bit samples into four signed 32-bit accumulator vectors.
     /// </summary>
     [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static void Convolve(
+    internal static void Convolve(
         ref byte source,
         int tapStride,
         nuint column,
@@ -85,7 +85,7 @@ internal static partial class Av1InterPredictor
     /// Convolves sixty-four adjacent 8-bit samples into four signed 32-bit accumulator vectors.
     /// </summary>
     [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static void Convolve(
+    internal static void Convolve(
         ref byte source,
         int tapStride,
         nuint column,
@@ -119,7 +119,7 @@ internal static partial class Av1InterPredictor
     /// Convolves eight adjacent nonnegative 16-bit samples into two signed 32-bit accumulator vectors.
     /// </summary>
     [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static void Convolve(
+    internal static void Convolve(
         ref short source,
         int tapStride,
         nuint column,
@@ -149,7 +149,7 @@ internal static partial class Av1InterPredictor
     /// Convolves sixteen adjacent nonnegative 16-bit samples into two signed 32-bit accumulator vectors.
     /// </summary>
     [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static void Convolve(
+    internal static void Convolve(
         ref short source,
         int tapStride,
         nuint column,
@@ -176,7 +176,7 @@ internal static partial class Av1InterPredictor
     /// Convolves thirty-two adjacent nonnegative 16-bit samples into two signed 32-bit accumulator vectors.
     /// </summary>
     [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static void Convolve(
+    internal static void Convolve(
         ref short source,
         int tapStride,
         nuint column,
@@ -203,28 +203,28 @@ internal static partial class Av1InterPredictor
     /// Applies AV1 power-of-two rounding to four-lane signed accumulators.
     /// </summary>
     [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector128<int> RoundPowerOfTwo(Vector128<int> value, int bits)
+    internal static Vector128<int> RoundPowerOfTwo(Vector128<int> value, int bits)
         => bits == 0 ? value : (value + Vector128.Create(1 << (bits - 1))) >> bits;
 
     /// <summary>
     /// Applies AV1 power-of-two rounding to eight-lane signed accumulators.
     /// </summary>
     [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector256<int> RoundPowerOfTwo(Vector256<int> value, int bits)
+    internal static Vector256<int> RoundPowerOfTwo(Vector256<int> value, int bits)
         => bits == 0 ? value : (value + Vector256.Create(1 << (bits - 1))) >> bits;
 
     /// <summary>
     /// Applies AV1 power-of-two rounding to sixteen-lane signed accumulators.
     /// </summary>
     [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector512<int> RoundPowerOfTwo(Vector512<int> value, int bits)
+    internal static Vector512<int> RoundPowerOfTwo(Vector512<int> value, int bits)
         => bits == 0 ? value : (value + Vector512.Create(1 << (bits - 1))) >> bits;
 
     /// <summary>
     /// Clips and packs sixteen signed accumulators into 8-bit samples.
     /// </summary>
     [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector128<byte> PackBytes(Vector128<int> result0, Vector128<int> result1, Vector128<int> result2, Vector128<int> result3)
+    internal static Vector128<byte> PackBytes(Vector128<int> result0, Vector128<int> result1, Vector128<int> result2, Vector128<int> result3)
     {
         Vector128<int> maximum = Vector128.Create((int)byte.MaxValue);
         result0 = Vector128.Clamp(result0, Vector128<int>.Zero, maximum);
@@ -238,7 +238,7 @@ internal static partial class Av1InterPredictor
     /// Clips and packs thirty-two signed accumulators into 8-bit samples.
     /// </summary>
     [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector256<byte> PackBytes(Vector256<int> result0, Vector256<int> result1, Vector256<int> result2, Vector256<int> result3)
+    internal static Vector256<byte> PackBytes(Vector256<int> result0, Vector256<int> result1, Vector256<int> result2, Vector256<int> result3)
     {
         Vector256<int> maximum = Vector256.Create((int)byte.MaxValue);
         result0 = Vector256.Clamp(result0, Vector256<int>.Zero, maximum);
@@ -252,7 +252,7 @@ internal static partial class Av1InterPredictor
     /// Clips and packs sixty-four signed accumulators into 8-bit samples.
     /// </summary>
     [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector512<byte> PackBytes(Vector512<int> result0, Vector512<int> result1, Vector512<int> result2, Vector512<int> result3)
+    internal static Vector512<byte> PackBytes(Vector512<int> result0, Vector512<int> result1, Vector512<int> result2, Vector512<int> result3)
     {
         Vector512<int> maximum = Vector512.Create((int)byte.MaxValue);
         result0 = Vector512.Clamp(result0, Vector512<int>.Zero, maximum);
@@ -266,7 +266,7 @@ internal static partial class Av1InterPredictor
     /// Clips and packs eight signed accumulators into high-bit-depth samples.
     /// </summary>
     [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector128<ushort> PackHighBitDepth(Vector128<int> result0, Vector128<int> result1, int maximumValue)
+    internal static Vector128<ushort> PackHighBitDepth(Vector128<int> result0, Vector128<int> result1, int maximumValue)
     {
         Vector128<int> maximum = Vector128.Create(maximumValue);
         result0 = Vector128.Clamp(result0, Vector128<int>.Zero, maximum);
@@ -278,7 +278,7 @@ internal static partial class Av1InterPredictor
     /// Clips and packs sixteen signed accumulators into high-bit-depth samples.
     /// </summary>
     [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector256<ushort> PackHighBitDepth(Vector256<int> result0, Vector256<int> result1, int maximumValue)
+    internal static Vector256<ushort> PackHighBitDepth(Vector256<int> result0, Vector256<int> result1, int maximumValue)
     {
         Vector256<int> maximum = Vector256.Create(maximumValue);
         result0 = Vector256.Clamp(result0, Vector256<int>.Zero, maximum);
@@ -290,7 +290,7 @@ internal static partial class Av1InterPredictor
     /// Clips and packs thirty-two signed accumulators into high-bit-depth samples.
     /// </summary>
     [MethodImpl(MethodImplOptions.AggressiveInlining)]
-    private static Vector512<ushort> PackHighBitDepth(Vector512<int> result0, Vector512<int> result1, int maximumValue)
+    internal static Vector512<ushort> PackHighBitDepth(Vector512<int> result0, Vector512<int> result1, int maximumValue)
     {
         Vector512<int> maximum = Vector512.Create(maximumValue);
         result0 = Vector512.Clamp(result0, Vector512<int>.Zero, maximum);
@@ -301,7 +301,7 @@ internal static partial class Av1InterPredictor
     /// <summary>
     /// Computes one signed Q7 convolution sum from 8-bit samples.
     /// </summary>
-    private static int ConvolveScalar(ref byte source, int sourceStride, ref short coefficients, int tapCount)
+    internal static int ConvolveScalar(ref byte source, int sourceStride, ref short coefficients, int tapCount)
     {
         int sum = 0;
         for (int tap = 0; tap < tapCount; tap++)
@@ -315,7 +315,7 @@ internal static partial class Av1InterPredictor
     /// <summary>
     /// Computes one signed Q7 convolution sum from high-bit-depth samples.
     /// </summary>
-    private static int ConvolveScalar(ref ushort source, int sourceStride, ref short coefficients, int tapCount)
+    internal static int ConvolveScalar(ref ushort source, int sourceStride, ref short coefficients, int tapCount)
     {
         int sum = 0;
         for (int tap = 0; tap < tapCount; tap++)
@@ -329,7 +329,7 @@ internal static partial class Av1InterPredictor
     /// <summary>
     /// Computes one signed Q7 convolution sum from biased intermediate samples.
     /// </summary>
-    private static int ConvolveScalar(ref short source, int sourceStride, ref short coefficients, int tapCount)
+    internal static int ConvolveScalar(ref short source, int sourceStride, ref short coefficients, int tapCount)
     {
         int sum = 0;
         for (int tap = 0; tap < tapCount; tap++)
@@ -343,5 +343,5 @@ internal static partial class Av1InterPredictor
     /// <summary>
     /// Rounds an integer after division by a power of two using AV1's unsigned-bias rule.
     /// </summary>
-    private static int RoundPowerOfTwo(int value, int bits) => bits == 0 ? value : (value + (1 << (bits - 1))) >> bits;
+    internal static int RoundPowerOfTwo(int value, int bits) => bits == 0 ? value : (value + (1 << (bits - 1))) >> bits;
 }

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Compound.cs

@@ -1,732 +0,0 @@
-// Copyright (c) Six Labors.
-// Licensed under the Six Labors Split License.
-
-using System.Runtime.CompilerServices;
-using System.Runtime.InteropServices;
-using System.Runtime.Intrinsics;
-
-namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
-
-/// <content>
-/// Produces the biased high-precision intermediates required by compound inter prediction.
-/// </content>
-internal static partial class Av1InterPredictor
-{
-    /// <summary>
-    /// The second-round shift retained by every compound convolution path.
-    /// </summary>
-    internal const int CompoundRound1Bits = 7;
-
-    /// <summary>
-    /// Reconstructs one 8-bit translational reference into AV1's unsigned compound intermediate format.
-    /// </summary>
-    public static void PredictCompound(
-        ReadOnlySpan<byte> source,
-        int sourceStride,
-        int sourceOrigin,
-        Span<ushort> destination,
-        int destinationStride,
-        int width,
-        int height,
-        Av1InterpolationFilter horizontalFilter,
-        Av1InterpolationFilter verticalFilter,
-        int horizontalPhase,
-        int verticalPhase,
-        Span<short> scratch)
-    {
-        if (Vector128.IsHardwareAccelerated)
-        {
-            PredictCompoundVector128(
-                source,
-                sourceStride,
-                sourceOrigin,
-                destination,
-                destinationStride,
-                width,
-                height,
-                horizontalFilter,
-                verticalFilter,
-                horizontalPhase,
-                verticalPhase,
-                scratch);
-
-            return;
-        }
-
-        PredictCompoundScalar(
-            source,
-            sourceStride,
-            sourceOrigin,
-            destination,
-            destinationStride,
-            width,
-            height,
-            horizontalFilter,
-            verticalFilter,
-            horizontalPhase,
-            verticalPhase,
-            scratch);
-    }
-
-    /// <summary>
-    /// Reconstructs one compound intermediate without explicit hardware intrinsics.
-    /// </summary>
-    public static void PredictCompoundScalar(
-        ReadOnlySpan<byte> source,
-        int sourceStride,
-        int sourceOrigin,
-        Span<ushort> destination,
-        int destinationStride,
-        int width,
-        int height,
-        Av1InterpolationFilter horizontalFilter,
-        Av1InterpolationFilter verticalFilter,
-        int horizontalPhase,
-        int verticalPhase,
-        Span<short> scratch)
-    {
-        ReadOnlySpan<short> horizontalCoefficients = GetCompoundCoefficients(horizontalFilter, horizontalPhase, width <= 4);
-        ReadOnlySpan<short> verticalCoefficients = GetCompoundCoefficients(verticalFilter, verticalPhase, height <= 4);
-        int roundBits = (2 * FilterBits) - Round0Bits - CompoundRound1Bits;
-        int offsetBits = 8 + (2 * FilterBits) - Round0Bits;
-        int roundOffset = (1 << (offsetBits - CompoundRound1Bits)) +
-            (1 << (offsetBits - CompoundRound1Bits - 1));
-
-        ref byte sourceBase = ref Unsafe.Add(ref MemoryMarshal.GetReference(source), sourceOrigin);
-        ref ushort destinationBase = ref MemoryMarshal.GetReference(destination);
-
-        if (horizontalPhase == 0 && verticalPhase == 0)
-        {
-            for (int row = 0; row < height; row++)
-            {
-                ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride);
-                ref ushort destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
-                for (int column = 0; column < width; column++)
-                {
-                    Unsafe.Add(ref destinationRow, column) =
-                        (ushort)((Unsafe.Add(ref sourceRow, column) << roundBits) + roundOffset);
-                }
-            }
-
-            return;
-        }
-
-        if (verticalPhase == 0)
-        {
-            GetEffectiveKernel(horizontalCoefficients, out int firstCoefficient, out int tapCount);
-            ref short coefficientBase = ref Unsafe.Add(
-                ref MemoryMarshal.GetReference(horizontalCoefficients),
-                firstCoefficient);
-
-            int sourceOffset = firstCoefficient - 3;
-            for (int row = 0; row < height; row++)
-            {
-                ref byte sourceRow = ref Unsafe.Add(ref sourceBase, (row * sourceStride) + sourceOffset);
-                ref ushort destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
-                for (int column = 0; column < width; column++)
-                {
-                    int sum = ConvolveScalar(ref Unsafe.Add(ref sourceRow, column), 1, ref coefficientBase, tapCount);
-                    int result = (RoundPowerOfTwo(sum, Round0Bits) << (FilterBits - CompoundRound1Bits)) + roundOffset;
-                    Unsafe.Add(ref destinationRow, column) = (ushort)result;
-                }
-            }
-
-            return;
-        }
-
-        if (horizontalPhase == 0)
-        {
-            GetEffectiveKernel(verticalCoefficients, out int firstCoefficient, out int tapCount);
-            ref short coefficientBase = ref Unsafe.Add(
-                ref MemoryMarshal.GetReference(verticalCoefficients),
-                firstCoefficient);
-
-            int sourceOffset = firstCoefficient - 3;
-            int firstPassBits = FilterBits - Round0Bits;
-            for (int row = 0; row < height; row++)
-            {
-                ref byte sourceRow = ref Unsafe.Add(ref sourceBase, (row + sourceOffset) * sourceStride);
-                ref ushort destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
-                for (int column = 0; column < width; column++)
-                {
-                    int sum = ConvolveScalar(
-                        ref Unsafe.Add(ref sourceRow, column),
-                        sourceStride,
-                        ref coefficientBase,
-                        tapCount);
-
-                    int result = RoundPowerOfTwo(sum << firstPassBits, CompoundRound1Bits) + roundOffset;
-                    Unsafe.Add(ref destinationRow, column) = (ushort)result;
-                }
-            }
-
-            return;
-        }
-
-        GetEffectiveKernel(horizontalCoefficients, out int firstHorizontalCoefficient, out int horizontalTapCount);
-        GetEffectiveKernel(verticalCoefficients, out int firstVerticalCoefficient, out int verticalTapCount);
-        ref short horizontalCoefficientBase = ref Unsafe.Add(
-            ref MemoryMarshal.GetReference(horizontalCoefficients),
-            firstHorizontalCoefficient);
-
-        ref short verticalCoefficientBase = ref Unsafe.Add(
-            ref MemoryMarshal.GetReference(verticalCoefficients),
-            firstVerticalCoefficient);
-
-        int scratchStride = Math.Max(width, MinimumScratchStride);
-        int intermediateHeight = height + verticalTapCount - 1;
-        int horizontalSourceOffset = firstHorizontalCoefficient - 3;
-        int verticalSourceOffset = firstVerticalCoefficient - 3;
-        int horizontalBias = 1 << (8 + FilterBits - 1);
-        ref short scratchBase = ref MemoryMarshal.GetReference(scratch);
-
-        // The Q7 horizontal pass keeps enough precision for the vertical pass while the positive bias makes every
-        // intermediate representable by signed 16-bit scratch. This is the same no-round compound shape as libaom.
-        for (int row = 0; row < intermediateHeight; row++)
-        {
-            ref byte sourceRow = ref Unsafe.Add(
-                ref sourceBase,
-                ((row + verticalSourceOffset) * sourceStride) + horizontalSourceOffset);
-
-            ref short scratchRow = ref Unsafe.Add(ref scratchBase, row * scratchStride);
-            for (int column = 0; column < width; column++)
-            {
-                int sum = horizontalBias + ConvolveScalar(
-                    ref Unsafe.Add(ref sourceRow, column),
-                    1,
-                    ref horizontalCoefficientBase,
-                    horizontalTapCount);
-
-                Unsafe.Add(ref scratchRow, column) = (short)RoundPowerOfTwo(sum, Round0Bits);
-            }
-        }
-
-        int verticalBias = 1 << offsetBits;
-        for (int row = 0; row < height; row++)
-        {
-            ref short scratchRow = ref Unsafe.Add(ref scratchBase, row * scratchStride);
-            ref ushort destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
-            for (int column = 0; column < width; column++)
-            {
-                int sum = verticalBias + ConvolveScalar(
-                    ref Unsafe.Add(ref scratchRow, column),
-                    scratchStride,
-                    ref verticalCoefficientBase,
-                    verticalTapCount);
-
-                Unsafe.Add(ref destinationRow, column) = (ushort)RoundPowerOfTwo(sum, CompoundRound1Bits);
-            }
-        }
-    }
-
-    /// <summary>
-    /// Reconstructs one compound intermediate through the 128-bit convolution tier.
-    /// </summary>
-    private static void PredictCompoundVector128(
-        ReadOnlySpan<byte> source,
-        int sourceStride,
-        int sourceOrigin,
-        Span<ushort> destination,
-        int destinationStride,
-        int width,
-        int height,
-        Av1InterpolationFilter horizontalFilter,
-        Av1InterpolationFilter verticalFilter,
-        int horizontalPhase,
-        int verticalPhase,
-        Span<short> scratch)
-    {
-        ReadOnlySpan<short> horizontalCoefficients = GetCompoundCoefficients(horizontalFilter, horizontalPhase, width <= 4);
-        ReadOnlySpan<short> verticalCoefficients = GetCompoundCoefficients(verticalFilter, verticalPhase, height <= 4);
-        int roundBits = (2 * FilterBits) - Round0Bits - CompoundRound1Bits;
-        int offsetBits = 8 + (2 * FilterBits) - Round0Bits;
-        int roundOffset = (1 << (offsetBits - CompoundRound1Bits)) +
-            (1 << (offsetBits - CompoundRound1Bits - 1));
-
-        if (horizontalPhase == 0 && verticalPhase == 0)
-        {
-            CopyCompoundVector128(
-                source,
-                sourceStride,
-                sourceOrigin,
-                destination,
-                destinationStride,
-                width,
-                height,
-                roundBits,
-                roundOffset);
-
-            return;
-        }
-
-        if (verticalPhase == 0)
-        {
-            GetEffectiveKernel(horizontalCoefficients, out int firstCoefficient, out int tapCount);
-            FilterCompoundDirectVector128(
-                source,
-                sourceStride,
-                sourceOrigin,
-                destination,
-                destinationStride,
-                width,
-                height,
-                horizontalCoefficients[firstCoefficient..],
-                tapCount,
-                firstCoefficient - 3,
-                tapStride: 1,
-                preShift: 0,
-                round: Round0Bits,
-                roundOffset);
-
-            return;
-        }
-
-        if (horizontalPhase == 0)
-        {
-            GetEffectiveKernel(verticalCoefficients, out int firstCoefficient, out int tapCount);
-            FilterCompoundDirectVector128(
-                source,
-                sourceStride,
-                sourceOrigin,
-                destination,
-                destinationStride,
-                width,
-                height,
-                verticalCoefficients[firstCoefficient..],
-                tapCount,
-                (firstCoefficient - 3) * sourceStride,
-                sourceStride,
-                FilterBits - Round0Bits,
-                CompoundRound1Bits,
-                roundOffset);
-
-            return;
-        }
-
-        GetEffectiveKernel(horizontalCoefficients, out int firstHorizontalCoefficient, out int horizontalTapCount);
-        GetEffectiveKernel(verticalCoefficients, out int firstVerticalCoefficient, out int verticalTapCount);
-        FilterCompound2DVector128(
-            source,
-            sourceStride,
-            sourceOrigin,
-            destination,
-            destinationStride,
-            width,
-            height,
-            horizontalCoefficients[firstHorizontalCoefficient..],
-            horizontalTapCount,
-            firstHorizontalCoefficient - 3,
-            verticalCoefficients[firstVerticalCoefficient..],
-            verticalTapCount,
-            firstVerticalCoefficient - 3,
-            scratch);
-    }
-
-    /// <summary>
-    /// Copies integer-position samples into biased compound intermediates in sixteen-sample groups.
-    /// </summary>
-    private static void CopyCompoundVector128(
-        ReadOnlySpan<byte> source,
-        int sourceStride,
-        int sourceOrigin,
-        Span<ushort> destination,
-        int destinationStride,
-        int width,
-        int height,
-        int roundBits,
-        int roundOffset)
-    {
-        ref byte sourceBase = ref Unsafe.Add(ref MemoryMarshal.GetReference(source), sourceOrigin);
-        ref ushort destinationBase = ref MemoryMarshal.GetReference(destination);
-        Vector128<ushort> offset = Vector128.Create((ushort)roundOffset);
-
-        for (int row = 0; row < height; row++)
-        {
-            ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride);
-            ref ushort destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
-            int column = 0;
-            int vectorEnd = width - Vector128<byte>.Count;
-            for (; column <= vectorEnd; column += Vector128<byte>.Count)
-            {
-                Vector128<byte> samples = Vector128.LoadUnsafe(ref sourceRow, (nuint)column);
-                ((Vector128.WidenLower(samples) << roundBits) + offset).StoreUnsafe(
-                    ref destinationRow,
-                    (nuint)column);
-
-                ((Vector128.WidenUpper(samples) << roundBits) + offset).StoreUnsafe(
-                    ref destinationRow,
-                    (nuint)(column + Vector128<ushort>.Count));
-            }
-
-            if (column == 0)
-            {
-                // Narrow AV1 blocks still use the SIMD load; the width-specific stores preserve the adjacent block.
-                Vector128<byte> samples = Vector128.LoadUnsafe(ref sourceRow);
-                StoreCompoundVectors(
-                    (Vector128.WidenLower(samples) << roundBits) + offset,
-                    (Vector128.WidenUpper(samples) << roundBits) + offset,
-                    ref destinationRow,
-                    width);
-
-                continue;
-            }
-
-            for (; column < width; column++)
-            {
-                Unsafe.Add(ref destinationRow, column) =
-                    (ushort)((Unsafe.Add(ref sourceRow, column) << roundBits) + roundOffset);
-            }
-        }
-    }
-
-    /// <summary>
-    /// Applies one compound convolution direction in sixteen-sample groups.
-    /// </summary>
-    private static void FilterCompoundDirectVector128(
-        ReadOnlySpan<byte> source,
-        int sourceStride,
-        int sourceOrigin,
-        Span<ushort> destination,
-        int destinationStride,
-        int width,
-        int height,
-        ReadOnlySpan<short> coefficients,
-        int tapCount,
-        int sourceOffset,
-        int tapStride,
-        int preShift,
-        int round,
-        int roundOffset)
-    {
-        ref byte sourceBase = ref Unsafe.Add(ref MemoryMarshal.GetReference(source), sourceOrigin);
-        ref ushort destinationBase = ref MemoryMarshal.GetReference(destination);
-        ref short coefficientBase = ref MemoryMarshal.GetReference(coefficients);
-        Vector128<int> offset = Vector128.Create(roundOffset);
-
-        for (int row = 0; row < height; row++)
-        {
-            ref byte sourceRow = ref Unsafe.Add(ref sourceBase, (row * sourceStride) + sourceOffset);
-            ref ushort destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
-            int column = 0;
-            int vectorEnd = width - Vector128<byte>.Count;
-            for (; column <= vectorEnd; column += Vector128<byte>.Count)
-            {
-                Convolve(
-                    ref sourceRow,
-                    tapStride,
-                    (nuint)column,
-                    ref coefficientBase,
-                    tapCount,
-                    Vector128<int>.Zero,
-                    out Vector128<int> result0,
-                    out Vector128<int> result1,
-                    out Vector128<int> result2,
-                    out Vector128<int> result3);
-
-                PrepareCompoundResults(
-                    ref result0,
-                    ref result1,
-                    ref result2,
-                    ref result3,
-                    preShift,
-                    round,
-                    offset);
-
-                StoreCompoundVectors(result0, result1, result2, result3, ref destinationRow, column, Vector128<byte>.Count);
-            }
-
-            if (column == 0)
-            {
-                Convolve(
-                    ref sourceRow,
-                    tapStride,
-                    0,
-                    ref coefficientBase,
-                    tapCount,
-                    Vector128<int>.Zero,
-                    out Vector128<int> result0,
-                    out Vector128<int> result1,
-                    out Vector128<int> result2,
-                    out Vector128<int> result3);
-
-                PrepareCompoundResults(
-                    ref result0,
-                    ref result1,
-                    ref result2,
-                    ref result3,
-                    preShift,
-                    round,
-                    offset);
-
-                StoreCompoundVectors(result0, result1, result2, result3, ref destinationRow, 0, width);
-                continue;
-            }
-
-            for (; column < width; column++)
-            {
-                int sum = ConvolveScalar(
-                    ref Unsafe.Add(ref sourceRow, column),
-                    tapStride,
-                    ref coefficientBase,
-                    tapCount);
-
-                sum = RoundPowerOfTwo(sum << preShift, round) + roundOffset;
-                Unsafe.Add(ref destinationRow, column) = (ushort)sum;
-            }
-        }
-    }
-
-    /// <summary>
-    /// Applies separable compound convolution through caller-owned signed scratch.
-    /// </summary>
-    private static void FilterCompound2DVector128(
-        ReadOnlySpan<byte> source,
-        int sourceStride,
-        int sourceOrigin,
-        Span<ushort> destination,
-        int destinationStride,
-        int width,
-        int height,
-        ReadOnlySpan<short> horizontalCoefficients,
-        int horizontalTapCount,
-        int horizontalSourceOffset,
-        ReadOnlySpan<short> verticalCoefficients,
-        int verticalTapCount,
-        int verticalSourceOffset,
-        Span<short> scratch)
-    {
-        ref byte sourceBase = ref Unsafe.Add(ref MemoryMarshal.GetReference(source), sourceOrigin);
-        ref ushort destinationBase = ref MemoryMarshal.GetReference(destination);
-        ref short scratchBase = ref MemoryMarshal.GetReference(scratch);
-        ref short horizontalCoefficientBase = ref MemoryMarshal.GetReference(horizontalCoefficients);
-        ref short verticalCoefficientBase = ref MemoryMarshal.GetReference(verticalCoefficients);
-        int scratchStride = Math.Max(width, MinimumScratchStride);
-        int intermediateHeight = height + verticalTapCount - 1;
-        Vector128<int> horizontalBias = Vector128.Create(1 << (8 + FilterBits - 1));
-
-        // The complete narrow-block vector is retained in scratch because the vertical pass consumes the same lanes.
-        // Wider blocks use one vector per sixteen output samples and finish any nonstandard tail scalarly.
-        for (int row = 0; row < intermediateHeight; row++)
-        {
-            ref byte sourceRow = ref Unsafe.Add(
-                ref sourceBase,
-                ((row + verticalSourceOffset) * sourceStride) + horizontalSourceOffset);
-
-            ref short scratchRow = ref Unsafe.Add(ref scratchBase, row * scratchStride);
-            int column = 0;
-            int vectorEnd = width - Vector128<byte>.Count;
-            for (; column <= vectorEnd; column += Vector128<byte>.Count)
-            {
-                Convolve(
-                    ref sourceRow,
-                    1,
-                    (nuint)column,
-                    ref horizontalCoefficientBase,
-                    horizontalTapCount,
-                    horizontalBias,
-                    out Vector128<int> result0,
-                    out Vector128<int> result1,
-                    out Vector128<int> result2,
-                    out Vector128<int> result3);
-
-                Av1IntraPredictorBase.Narrow(
-                    RoundPowerOfTwo(result0, Round0Bits),
-                    RoundPowerOfTwo(result1, Round0Bits)).StoreUnsafe(ref scratchRow, (nuint)column);
-
-                Av1IntraPredictorBase.Narrow(
-                    RoundPowerOfTwo(result2, Round0Bits),
-                    RoundPowerOfTwo(result3, Round0Bits)).StoreUnsafe(
-                        ref scratchRow,
-                        (nuint)(column + Vector128<short>.Count));
-            }
-
-            if (column == 0)
-            {
-                Convolve(
-                    ref sourceRow,
-                    1,
-                    0,
-                    ref horizontalCoefficientBase,
-                    horizontalTapCount,
-                    horizontalBias,
-                    out Vector128<int> result0,
-                    out Vector128<int> result1,
-                    out Vector128<int> result2,
-                    out Vector128<int> result3);
-
-                Av1IntraPredictorBase.Narrow(
-                    RoundPowerOfTwo(result0, Round0Bits),
-                    RoundPowerOfTwo(result1, Round0Bits)).StoreUnsafe(ref scratchRow);
-
-                Av1IntraPredictorBase.Narrow(
-                    RoundPowerOfTwo(result2, Round0Bits),
-                    RoundPowerOfTwo(result3, Round0Bits)).StoreUnsafe(
-                        ref scratchRow,
-                        (nuint)Vector128<short>.Count);
-
-                continue;
-            }
-
-            for (; column < width; column++)
-            {
-                int sum = (1 << (8 + FilterBits - 1)) + ConvolveScalar(
-                    ref Unsafe.Add(ref sourceRow, column),
-                    1,
-                    ref horizontalCoefficientBase,
-                    horizontalTapCount);
-
-                Unsafe.Add(ref scratchRow, column) = (short)RoundPowerOfTwo(sum, Round0Bits);
-            }
-        }
-
-        Vector128<int> verticalBias = Vector128.Create(1 << (8 + (2 * FilterBits) - Round0Bits));
-        for (int row = 0; row < height; row++)
-        {
-            ref short scratchRow = ref Unsafe.Add(ref scratchBase, row * scratchStride);
-            ref ushort destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
-            int column = 0;
-            int vectorEnd = width - Vector128<byte>.Count;
-            for (; column <= vectorEnd; column += Vector128<byte>.Count)
-            {
-                Convolve(
-                    ref scratchRow,
-                    scratchStride,
-                    (nuint)column,
-                    ref verticalCoefficientBase,
-                    verticalTapCount,
-                    verticalBias,
-                    out Vector128<int> result0,
-                    out Vector128<int> result1);
-
-                Convolve(
-                    ref scratchRow,
-                    scratchStride,
-                    (nuint)(column + Vector128<short>.Count),
-                    ref verticalCoefficientBase,
-                    verticalTapCount,
-                    verticalBias,
-                    out Vector128<int> result2,
-                    out Vector128<int> result3);
-
-                result0 = RoundPowerOfTwo(result0, CompoundRound1Bits);
-                result1 = RoundPowerOfTwo(result1, CompoundRound1Bits);
-                result2 = RoundPowerOfTwo(result2, CompoundRound1Bits);
-                result3 = RoundPowerOfTwo(result3, CompoundRound1Bits);
-                StoreCompoundVectors(result0, result1, result2, result3, ref destinationRow, column, Vector128<byte>.Count);
-            }
-
-            if (column == 0)
-            {
-                Convolve(
-                    ref scratchRow,
-                    scratchStride,
-                    0,
-                    ref verticalCoefficientBase,
-                    verticalTapCount,
-                    verticalBias,
-                    out Vector128<int> result0,
-                    out Vector128<int> result1);
-
-                Convolve(
-                    ref Unsafe.Add(ref scratchRow, Vector128<short>.Count),
-                    scratchStride,
-                    0,
-                    ref verticalCoefficientBase,
-                    verticalTapCount,
-                    verticalBias,
-                    out Vector128<int> result2,
-                    out Vector128<int> result3);
-
-                result0 = RoundPowerOfTwo(result0, CompoundRound1Bits);
-                result1 = RoundPowerOfTwo(result1, CompoundRound1Bits);
-                result2 = RoundPowerOfTwo(result2, CompoundRound1Bits);
-                result3 = RoundPowerOfTwo(result3, CompoundRound1Bits);
-                StoreCompoundVectors(result0, result1, result2, result3, ref destinationRow, 0, width);
-                continue;
-            }
-
-            for (; column < width; column++)
-            {
-                int sum = (1 << (8 + (2 * FilterBits) - Round0Bits)) + ConvolveScalar(
-                    ref Unsafe.Add(ref scratchRow, column),
-                    scratchStride,
-                    ref verticalCoefficientBase,
-                    verticalTapCount);
-
-                Unsafe.Add(ref destinationRow, column) =
-                    (ushort)RoundPowerOfTwo(sum, CompoundRound1Bits);
-            }
-        }
-    }
-
-    /// <summary>
-    /// Applies the compound direct-filter shifts and bias to sixteen convolution results.
-    /// </summary>
-    private static void PrepareCompoundResults(
-        ref Vector128<int> result0,
-        ref Vector128<int> result1,
-        ref Vector128<int> result2,
-        ref Vector128<int> result3,
-        int preShift,
-        int round,
-        Vector128<int> offset)
-    {
-        result0 = RoundPowerOfTwo(result0 << preShift, round) + offset;
-        result1 = RoundPowerOfTwo(result1 << preShift, round) + offset;
-        result2 = RoundPowerOfTwo(result2 << preShift, round) + offset;
-        result3 = RoundPowerOfTwo(result3 << preShift, round) + offset;
-    }
-
-    /// <summary>
-    /// Packs and stores up to sixteen unsigned compound results.
-    /// </summary>
-    private static void StoreCompoundVectors(
-        Vector128<int> result0,
-        Vector128<int> result1,
-        Vector128<int> result2,
-        Vector128<int> result3,
-        ref ushort destination,
-        int destinationOffset,
-        int width)
-    {
-        Vector128<ushort> lower = Av1IntraPredictorBase.Narrow(result0, result1).AsUInt16();
-        Vector128<ushort> upper = Av1IntraPredictorBase.Narrow(result2, result3).AsUInt16();
-        ref ushort destinationStart = ref Unsafe.Add(ref destination, destinationOffset);
-        StoreCompoundVectors(lower, upper, ref destinationStart, width);
-    }
-
-    /// <summary>
-    /// Stores up to sixteen packed compound results without crossing the logical block edge.
-    /// </summary>
-    private static void StoreCompoundVectors(
-        Vector128<ushort> lower,
-        Vector128<ushort> upper,
-        ref ushort destination,
-        int width)
-    {
-        int lowerWidth = Math.Min(width, Vector128<ushort>.Count);
-        StorePartial(lower, ref destination, lowerWidth);
-        if (width > Vector128<ushort>.Count)
-        {
-            StorePartial(
-                upper,
-                ref Unsafe.Add(ref destination, Vector128<ushort>.Count),
-                width - Vector128<ushort>.Count);
-        }
-    }
-
-    /// <summary>
-    /// Gets the selected interpolation kernel for compound traversal.
-    /// </summary>
-    private static ReadOnlySpan<short> GetCompoundCoefficients(
-        Av1InterpolationFilter filter,
-        int phase,
-        bool useReducedFilter)
-        => filter switch
-        {
-            Av1InterpolationFilter.Regular => RegularOperator.GetCoefficients(phase, useReducedFilter),
-            Av1InterpolationFilter.Smooth => SmoothOperator.GetCoefficients(phase, useReducedFilter),
-            Av1InterpolationFilter.Sharp => SharpOperator.GetCoefficients(phase, useReducedFilter),
-            _ => BilinearOperator.GetCoefficients(phase, useReducedFilter),
-        };
-}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Filters.cs

@@ -11,7 +11,7 @@ internal static partial class Av1InterPredictor
     /// <summary>
     /// The number of stored coefficient positions in every decoder interpolation kernel.
     /// </summary>
-    private const int FilterCoefficientCount = 8;
+    internal const int FilterCoefficientCount = 8;
 
     /// <summary>
     /// Gets the regular eight-tap kernels for the sixteen subpixel phases.

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Operator.cs

@@ -1,6 +1,8 @@
 // Copyright (c) Six Labors.
 // Licensed under the Six Labors Split License.
 
+using System.Runtime.Intrinsics;
+
 namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
 
 /// <content>
@@ -753,7 +755,7 @@ internal static partial class Av1InterPredictor
     /// <param name="coefficients">The selected Q7 phase kernel.</param>
     /// <param name="firstCoefficient">Receives the first coefficient used by the effective kernel.</param>
     /// <param name="tapCount">Receives the effective two-, four-, six-, or eight-tap length.</param>
-    private static void GetEffectiveKernel(ReadOnlySpan<short> coefficients, out int firstCoefficient, out int tapCount)
+    internal static void GetEffectiveKernel(ReadOnlySpan<short> coefficients, out int firstCoefficient, out int tapCount)
     {
         // This matches libaom's get_filter_tap decision. Reducing symmetric zero endpoints avoids source loads and
         // multiply-adds while retaining the original tap-to-source alignment through firstCoefficient.

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.ScaledOperator.cs

@@ -1,105 +0,0 @@
-// Copyright (c) Six Labors.
-// Licensed under the Six Labors Split License.
-
-using System.Runtime.CompilerServices;
-using System.Runtime.Intrinsics;
-
-namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
-
-/// <content>
-/// Defines sample-storage operators for reference-scaled prediction.
-/// </content>
-internal static partial class Av1InterPredictor
-{
-    /// <summary>
-    /// Supplies sample loading, clipping, and storage for a scaled predictor pipeline.
-    /// </summary>
-    /// <typeparam name="T">The native sample storage type.</typeparam>
-    private interface IScaledSampleOperator<T>
-        where T : unmanaged
-    {
-        /// <summary>
-        /// Loads one source sample as a signed accumulator value.
-        /// </summary>
-        /// <param name="source">The first source sample.</param>
-        /// <param name="index">The sample offset.</param>
-        /// <returns>The widened sample value.</returns>
-        public static abstract int Load(ref T source, int index);
-
-        /// <summary>
-        /// Clips and stores eight completed vector lanes.
-        /// </summary>
-        /// <param name="destination">The first destination sample.</param>
-        /// <param name="index">The output offset.</param>
-        /// <param name="result0">The first four completed lanes.</param>
-        /// <param name="result1">The second four completed lanes.</param>
-        /// <param name="bitDepth">The decoded sample precision.</param>
-        public static abstract void StoreVector(
-            ref T destination,
-            int index,
-            Vector128<int> result0,
-            Vector128<int> result1,
-            int bitDepth);
-
-        /// <summary>
-        /// Clips and stores one completed scalar value.
-        /// </summary>
-        /// <param name="destination">The first destination sample.</param>
-        /// <param name="index">The output offset.</param>
-        /// <param name="value">The completed sample value.</param>
-        /// <param name="bitDepth">The decoded sample precision.</param>
-        public static abstract void StoreScalar(ref T destination, int index, int value, int bitDepth);
-    }
-
-    /// <summary>
-    /// Implements scaled prediction storage for 8-bit samples.
-    /// </summary>
-    private readonly struct ScaledByteOperator : IScaledSampleOperator<byte>
-    {
-        /// <inheritdoc/>
-        [MethodImpl(MethodImplOptions.AggressiveInlining)]
-        public static int Load(ref byte source, int index) => Unsafe.Add(ref source, index);
-
-        /// <inheritdoc/>
-        [MethodImpl(MethodImplOptions.AggressiveInlining)]
-        public static void StoreVector(
-            ref byte destination,
-            int index,
-            Vector128<int> result0,
-            Vector128<int> result1,
-            int bitDepth)
-            => PackBytes(result0, result1, Vector128<int>.Zero, Vector128<int>.Zero)
-                .GetLower()
-                .StoreUnsafe(ref destination, (nuint)index);
-
-        /// <inheritdoc/>
-        [MethodImpl(MethodImplOptions.AggressiveInlining)]
-        public static void StoreScalar(ref byte destination, int index, int value, int bitDepth)
-            => Unsafe.Add(ref destination, index) = (byte)Math.Clamp(value, byte.MinValue, byte.MaxValue);
-    }
-
-    /// <summary>
-    /// Implements scaled prediction storage for 8-, 10-, and 12-bit samples.
-    /// </summary>
-    private readonly struct ScaledUInt16Operator : IScaledSampleOperator<ushort>
-    {
-        /// <inheritdoc/>
-        [MethodImpl(MethodImplOptions.AggressiveInlining)]
-        public static int Load(ref ushort source, int index) => Unsafe.Add(ref source, index);
-
-        /// <inheritdoc/>
-        [MethodImpl(MethodImplOptions.AggressiveInlining)]
-        public static void StoreVector(
-            ref ushort destination,
-            int index,
-            Vector128<int> result0,
-            Vector128<int> result1,
-            int bitDepth)
-            => PackHighBitDepth(result0, result1, (1 << bitDepth) - 1).StoreUnsafe(ref destination, (nuint)index);
-
-        /// <inheritdoc/>
-        [MethodImpl(MethodImplOptions.AggressiveInlining)]
-        public static void StoreScalar(ref ushort destination, int index, int value, int bitDepth)
-            => Unsafe.Add(ref destination, index) = (ushort)Math.Clamp(value, 0, (1 << bitDepth) - 1);
-    }
-}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Warped.cs

@@ -1,730 +0,0 @@
-// Copyright (c) Six Labors.
-// Licensed under the Six Labors Split License.
-
-using System.Runtime.CompilerServices;
-using System.Runtime.InteropServices;
-using System.Runtime.Intrinsics;
-using SixLabors.ImageSharp.Formats.Heif.Av1.Motion;
-
-namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
-
-/// <content>
-/// Reconstructs local and global affine warped-motion prediction blocks.
-/// </content>
-internal static partial class Av1InterPredictor
-{
-    /// <summary>
-    /// The number of rows in one warped filter's horizontal intermediate tile.
-    /// </summary>
-    private const int WarpedIntermediateRows = 15;
-
-    /// <summary>
-    /// The number of columns in one warped filter tile.
-    /// </summary>
-    private const int WarpedTileSize = 8;
-
-    /// <summary>
-    /// The number of low model bits removed when addressing the warped filter table.
-    /// </summary>
-    private const int WarpedDifferencePrecisionBits = 10;
-
-    /// <summary>
-    /// The number of fractional positions in one warped pixel.
-    /// </summary>
-    private const int WarpedPixelPrecisionShifts = 64;
-
-    /// <summary>
-    /// Gets the number of signed 16-bit elements required by warped prediction.
-    /// </summary>
-    public const int WarpedScratchLength = WarpedIntermediateRows * WarpedTileSize;
-
-    /// <summary>
-    /// Supplies the horizontal and vertical eight-tap dot products for one execution width.
-    /// </summary>
-    private interface IWarpedConvolution
-    {
-        /// <summary>
-        /// Convolves eight adjacent unsigned byte samples.
-        /// </summary>
-        /// <param name="source">The first source sample.</param>
-        /// <param name="coefficients">The first signed Q7 coefficient.</param>
-        /// <returns>The sum of the eight sample-coefficient products.</returns>
-        public static abstract int Convolve(ref byte source, ref short coefficients);
-
-        /// <summary>
-        /// Convolves eight adjacent unsigned high-bit-depth samples.
-        /// </summary>
-        /// <param name="source">The first source sample.</param>
-        /// <param name="coefficients">The first signed Q7 coefficient.</param>
-        /// <returns>The sum of the eight sample-coefficient products.</returns>
-        public static abstract int Convolve(ref ushort source, ref short coefficients);
-
-        /// <summary>
-        /// Convolves eight vertically strided unsigned intermediate samples.
-        /// </summary>
-        /// <param name="source">The first intermediate sample.</param>
-        /// <param name="stride">The distance between intermediate rows.</param>
-        /// <param name="coefficients">The first signed Q7 coefficient.</param>
-        /// <returns>The sum of the eight sample-coefficient products.</returns>
-        public static abstract int ConvolveVertical(ref ushort source, int stride, ref short coefficients);
-    }
-
-    /// <summary>
-    /// Reconstructs an 8-bit affine warped prediction using the widest supported convolution operator.
-    /// </summary>
-    public static void PredictWarped(
-        ReadOnlySpan<byte> source,
-        int sourceStride,
-        Point sourceOrigin,
-        int sourceWidth,
-        int sourceHeight,
-        Span<byte> destination,
-        int destinationStride,
-        Point destinationPosition,
-        int width,
-        int height,
-        int subsamplingX,
-        int subsamplingY,
-        Av1GlobalMotionParameters parameters,
-        Span<short> scratch)
-    {
-        if (Vector128.IsHardwareAccelerated)
-        {
-            PredictWarped<WarpedVector128Convolution>(
-                source,
-                sourceStride,
-                sourceOrigin,
-                sourceWidth,
-                sourceHeight,
-                destination,
-                destinationStride,
-                destinationPosition,
-                width,
-                height,
-                subsamplingX,
-                subsamplingY,
-                parameters,
-                scratch);
-
-            return;
-        }
-
-        PredictWarped<WarpedScalarConvolution>(
-            source,
-            sourceStride,
-            sourceOrigin,
-            sourceWidth,
-            sourceHeight,
-            destination,
-            destinationStride,
-            destinationPosition,
-            width,
-            height,
-            subsamplingX,
-            subsamplingY,
-            parameters,
-            scratch);
-    }
-
-    /// <summary>
-    /// Reconstructs an 8-bit affine warped reference into AV1's unsigned compound intermediate format.
-    /// </summary>
-    public static void PredictWarpedCompound(
-        ReadOnlySpan<byte> source,
-        int sourceStride,
-        Point sourceOrigin,
-        int sourceWidth,
-        int sourceHeight,
-        Span<ushort> destination,
-        int destinationStride,
-        Point destinationPosition,
-        int width,
-        int height,
-        int subsamplingX,
-        int subsamplingY,
-        Av1GlobalMotionParameters parameters,
-        Span<short> scratch)
-    {
-        if (Vector128.IsHardwareAccelerated)
-        {
-            PredictWarpedCompound<WarpedVector128Convolution>(
-                source,
-                sourceStride,
-                sourceOrigin,
-                sourceWidth,
-                sourceHeight,
-                destination,
-                destinationStride,
-                destinationPosition,
-                width,
-                height,
-                subsamplingX,
-                subsamplingY,
-                parameters,
-                scratch);
-
-            return;
-        }
-
-        PredictWarpedCompound<WarpedScalarConvolution>(
-            source,
-            sourceStride,
-            sourceOrigin,
-            sourceWidth,
-            sourceHeight,
-            destination,
-            destinationStride,
-            destinationPosition,
-            width,
-            height,
-            subsamplingX,
-            subsamplingY,
-            parameters,
-            scratch);
-    }
-
-    /// <summary>
-    /// Reconstructs a high-bit-depth affine warped prediction using the widest supported convolution operator.
-    /// </summary>
-    public static void PredictWarped(
-        ReadOnlySpan<ushort> source,
-        int sourceStride,
-        Point sourceOrigin,
-        int sourceWidth,
-        int sourceHeight,
-        Span<ushort> destination,
-        int destinationStride,
-        Point destinationPosition,
-        int width,
-        int height,
-        int subsamplingX,
-        int subsamplingY,
-        int bitDepth,
-        Av1GlobalMotionParameters parameters,
-        Span<short> scratch)
-    {
-        if (Vector128.IsHardwareAccelerated)
-        {
-            PredictWarped<WarpedVector128Convolution>(
-                source,
-                sourceStride,
-                sourceOrigin,
-                sourceWidth,
-                sourceHeight,
-                destination,
-                destinationStride,
-                destinationPosition,
-                width,
-                height,
-                subsamplingX,
-                subsamplingY,
-                bitDepth,
-                parameters,
-                scratch);
-
-            return;
-        }
-
-        PredictWarped<WarpedScalarConvolution>(
-            source,
-            sourceStride,
-            sourceOrigin,
-            sourceWidth,
-            sourceHeight,
-            destination,
-            destinationStride,
-            destinationPosition,
-            width,
-            height,
-            subsamplingX,
-            subsamplingY,
-            bitDepth,
-            parameters,
-            scratch);
-    }
-
-    /// <summary>
-    /// Reconstructs an 8-bit affine warped prediction without explicit hardware intrinsics.
-    /// </summary>
-    public static void PredictWarpedScalar(
-        ReadOnlySpan<byte> source,
-        int sourceStride,
-        Point sourceOrigin,
-        int sourceWidth,
-        int sourceHeight,
-        Span<byte> destination,
-        int destinationStride,
-        Point destinationPosition,
-        int width,
-        int height,
-        int subsamplingX,
-        int subsamplingY,
-        Av1GlobalMotionParameters parameters,
-        Span<short> scratch)
-        => PredictWarped<WarpedScalarConvolution>(
-            source,
-            sourceStride,
-            sourceOrigin,
-            sourceWidth,
-            sourceHeight,
-            destination,
-            destinationStride,
-            destinationPosition,
-            width,
-            height,
-            subsamplingX,
-            subsamplingY,
-            parameters,
-            scratch);
-
-    /// <summary>
-    /// Reconstructs an 8-bit affine warped reference into compound intermediates without explicit hardware intrinsics.
-    /// </summary>
-    public static void PredictWarpedCompoundScalar(
-        ReadOnlySpan<byte> source,
-        int sourceStride,
-        Point sourceOrigin,
-        int sourceWidth,
-        int sourceHeight,
-        Span<ushort> destination,
-        int destinationStride,
-        Point destinationPosition,
-        int width,
-        int height,
-        int subsamplingX,
-        int subsamplingY,
-        Av1GlobalMotionParameters parameters,
-        Span<short> scratch)
-        => PredictWarpedCompound<WarpedScalarConvolution>(
-            source,
-            sourceStride,
-            sourceOrigin,
-            sourceWidth,
-            sourceHeight,
-            destination,
-            destinationStride,
-            destinationPosition,
-            width,
-            height,
-            subsamplingX,
-            subsamplingY,
-            parameters,
-            scratch);
-
-    /// <summary>
-    /// Reconstructs a high-bit-depth affine warped prediction without explicit hardware intrinsics.
-    /// </summary>
-    public static void PredictWarpedScalar(
-        ReadOnlySpan<ushort> source,
-        int sourceStride,
-        Point sourceOrigin,
-        int sourceWidth,
-        int sourceHeight,
-        Span<ushort> destination,
-        int destinationStride,
-        Point destinationPosition,
-        int width,
-        int height,
-        int subsamplingX,
-        int subsamplingY,
-        int bitDepth,
-        Av1GlobalMotionParameters parameters,
-        Span<short> scratch)
-        => PredictWarped<WarpedScalarConvolution>(
-            source,
-            sourceStride,
-            sourceOrigin,
-            sourceWidth,
-            sourceHeight,
-            destination,
-            destinationStride,
-            destinationPosition,
-            width,
-            height,
-            subsamplingX,
-            subsamplingY,
-            bitDepth,
-            parameters,
-            scratch);
-
-    /// <summary>
-    /// Reconstructs one 8-bit warped block through a closed convolution operator.
-    /// </summary>
-    private static void PredictWarped<TConvolution>(
-        ReadOnlySpan<byte> source,
-        int sourceStride,
-        Point sourceOrigin,
-        int sourceWidth,
-        int sourceHeight,
-        Span<byte> destination,
-        int destinationStride,
-        Point destinationPosition,
-        int width,
-        int height,
-        int subsamplingX,
-        int subsamplingY,
-        Av1GlobalMotionParameters parameters,
-        Span<short> scratch)
-        where TConvolution : struct, IWarpedConvolution
-    {
-        ref byte sourceBase = ref MemoryMarshal.GetReference(source);
-        ref byte destinationBase = ref MemoryMarshal.GetReference(destination);
-        Span<ushort> intermediate = MemoryMarshal.Cast<short, ushort>(scratch)[..WarpedScratchLength];
-        int horizontalBias = 1 << (8 + FilterBits - 1);
-        int verticalBias = 1 << (8 + (2 * FilterBits) - Round0Bits);
-        int verticalRound = (2 * FilterBits) - Round0Bits;
-
-        for (int tileRow = destinationPosition.Y; tileRow < destinationPosition.Y + height; tileRow += WarpedTileSize)
-        {
-            for (int tileColumn = destinationPosition.X; tileColumn < destinationPosition.X + width; tileColumn += WarpedTileSize)
-            {
-                DeriveWarpedTilePosition(
-                    parameters,
-                    tileColumn,
-                    tileRow,
-                    subsamplingX,
-                    subsamplingY,
-                    out int integerX,
-                    out int integerY,
-                    out int phaseX,
-                    out int phaseY);
-
-                for (int row = -7; row < 8; row++)
-                {
-                    int sourceY = Math.Clamp(integerY + row, 0, sourceHeight - 1);
-                    int phase = phaseX + (parameters.Beta * (row + 4));
-                    for (int column = -4; column < 4; column++)
-                    {
-                        int sourceX = integerX + column - 3;
-                        int sourceIndex = ((sourceOrigin.Y + sourceY) * sourceStride) + sourceOrigin.X + sourceX;
-                        ref short coefficients = ref GetWarpedFilterReference(phase);
-                        int sum = horizontalBias + TConvolution.Convolve(ref Unsafe.Add(ref sourceBase, sourceIndex), ref coefficients);
-                        intermediate[((row + 7) * WarpedTileSize) + column + 4] = (ushort)RoundPowerOfTwoScalar(sum, Round0Bits);
-                        phase += parameters.Alpha;
-                    }
-                }
-
-                int tileHeight = Math.Min(WarpedTileSize, destinationPosition.Y + height - tileRow);
-                int tileWidth = Math.Min(WarpedTileSize, destinationPosition.X + width - tileColumn);
-                for (int row = 0; row < tileHeight; row++)
-                {
-                    int phase = phaseY + (parameters.Delta * row);
-                    int destinationRowOffset = (tileRow - destinationPosition.Y + row) * destinationStride;
-                    for (int column = 0; column < tileWidth; column++)
-                    {
-                        ref ushort intermediateSource = ref intermediate[(row * WarpedTileSize) + column];
-                        ref short coefficients = ref GetWarpedFilterReference(phase);
-                        int sum = verticalBias + TConvolution.ConvolveVertical(ref intermediateSource, WarpedTileSize, ref coefficients);
-                        int value = RoundPowerOfTwoScalar(sum, verticalRound) - (1 << 7) - (1 << 8);
-                        Unsafe.Add(ref destinationBase, destinationRowOffset + tileColumn - destinationPosition.X + column) =
-                            (byte)Math.Clamp(value, byte.MinValue, byte.MaxValue);
-
-                        phase += parameters.Gamma;
-                    }
-                }
-            }
-        }
-    }
-
-    /// <summary>
-    /// Reconstructs one 8-bit warped reference without discarding the compound convolution precision.
-    /// </summary>
-    private static void PredictWarpedCompound<TConvolution>(
-        ReadOnlySpan<byte> source,
-        int sourceStride,
-        Point sourceOrigin,
-        int sourceWidth,
-        int sourceHeight,
-        Span<ushort> destination,
-        int destinationStride,
-        Point destinationPosition,
-        int width,
-        int height,
-        int subsamplingX,
-        int subsamplingY,
-        Av1GlobalMotionParameters parameters,
-        Span<short> scratch)
-        where TConvolution : struct, IWarpedConvolution
-    {
-        ref byte sourceBase = ref MemoryMarshal.GetReference(source);
-        ref ushort destinationBase = ref MemoryMarshal.GetReference(destination);
-        Span<ushort> intermediate = MemoryMarshal.Cast<short, ushort>(scratch)[..WarpedScratchLength];
-        int horizontalBias = 1 << (8 + FilterBits - 1);
-        int verticalBias = 1 << (8 + (2 * FilterBits) - Round0Bits);
-
-        for (int tileRow = destinationPosition.Y; tileRow < destinationPosition.Y + height; tileRow += WarpedTileSize)
-        {
-            for (int tileColumn = destinationPosition.X; tileColumn < destinationPosition.X + width; tileColumn += WarpedTileSize)
-            {
-                DeriveWarpedTilePosition(
-                    parameters,
-                    tileColumn,
-                    tileRow,
-                    subsamplingX,
-                    subsamplingY,
-                    out int integerX,
-                    out int integerY,
-                    out int phaseX,
-                    out int phaseY);
-
-                for (int row = -7; row < 8; row++)
-                {
-                    int sourceY = Math.Clamp(integerY + row, 0, sourceHeight - 1);
-                    int phase = phaseX + (parameters.Beta * (row + 4));
-                    for (int column = -4; column < 4; column++)
-                    {
-                        int sourceX = integerX + column - 3;
-                        int sourceIndex = ((sourceOrigin.Y + sourceY) * sourceStride) + sourceOrigin.X + sourceX;
-                        ref short coefficients = ref GetWarpedFilterReference(phase);
-                        int sum = horizontalBias + TConvolution.Convolve(ref Unsafe.Add(ref sourceBase, sourceIndex), ref coefficients);
-                        intermediate[((row + 7) * WarpedTileSize) + column + 4] =
-                            (ushort)RoundPowerOfTwoScalar(sum, Round0Bits);
-
-                        phase += parameters.Alpha;
-                    }
-                }
-
-                int tileHeight = Math.Min(WarpedTileSize, destinationPosition.Y + height - tileRow);
-                int tileWidth = Math.Min(WarpedTileSize, destinationPosition.X + width - tileColumn);
-                for (int row = 0; row < tileHeight; row++)
-                {
-                    int phase = phaseY + (parameters.Delta * row);
-                    int destinationRowOffset = (tileRow - destinationPosition.Y + row) * destinationStride;
-                    for (int column = 0; column < tileWidth; column++)
-                    {
-                        ref ushort intermediateSource = ref intermediate[(row * WarpedTileSize) + column];
-                        ref short coefficients = ref GetWarpedFilterReference(phase);
-                        int sum = verticalBias + TConvolution.ConvolveVertical(ref intermediateSource, WarpedTileSize, ref coefficients);
-                        Unsafe.Add(ref destinationBase, destinationRowOffset + tileColumn - destinationPosition.X + column) =
-                            (ushort)RoundPowerOfTwoScalar(sum, CompoundRound1Bits);
-
-                        phase += parameters.Gamma;
-                    }
-                }
-            }
-        }
-    }
-
-    /// <summary>
-    /// Reconstructs one high-bit-depth warped block through a closed convolution operator.
-    /// </summary>
-    private static void PredictWarped<TConvolution>(
-        ReadOnlySpan<ushort> source,
-        int sourceStride,
-        Point sourceOrigin,
-        int sourceWidth,
-        int sourceHeight,
-        Span<ushort> destination,
-        int destinationStride,
-        Point destinationPosition,
-        int width,
-        int height,
-        int subsamplingX,
-        int subsamplingY,
-        int bitDepth,
-        Av1GlobalMotionParameters parameters,
-        Span<short> scratch)
-        where TConvolution : struct, IWarpedConvolution
-    {
-        ref ushort sourceBase = ref MemoryMarshal.GetReference(source);
-        ref ushort destinationBase = ref MemoryMarshal.GetReference(destination);
-        Span<ushort> intermediate = MemoryMarshal.Cast<short, ushort>(scratch)[..WarpedScratchLength];
-
-        // Twelve-bit prediction increases round0 by two so the biased horizontal intermediate remains representable
-        // in sixteen bits. Reducing round1 by the same amount preserves the complete normative Q14 shift.
-        int intermediateRange = bitDepth + FilterBits - Round0Bits + 2;
-        int round0 = Round0Bits + Math.Max(intermediateRange - 16, 0);
-        int verticalRound = (2 * FilterBits) - round0;
-        int horizontalBias = 1 << (bitDepth + FilterBits - 1);
-        int verticalBias = 1 << (bitDepth + (2 * FilterBits) - round0);
-        int maximum = (1 << bitDepth) - 1;
-
-        for (int tileRow = destinationPosition.Y; tileRow < destinationPosition.Y + height; tileRow += WarpedTileSize)
-        {
-            for (int tileColumn = destinationPosition.X; tileColumn < destinationPosition.X + width; tileColumn += WarpedTileSize)
-            {
-                DeriveWarpedTilePosition(
-                    parameters,
-                    tileColumn,
-                    tileRow,
-                    subsamplingX,
-                    subsamplingY,
-                    out int integerX,
-                    out int integerY,
-                    out int phaseX,
-                    out int phaseY);
-
-                for (int row = -7; row < 8; row++)
-                {
-                    int sourceY = Math.Clamp(integerY + row, 0, sourceHeight - 1);
-                    int phase = phaseX + (parameters.Beta * (row + 4));
-                    for (int column = -4; column < 4; column++)
-                    {
-                        int sourceX = integerX + column - 3;
-                        int sourceIndex = ((sourceOrigin.Y + sourceY) * sourceStride) + sourceOrigin.X + sourceX;
-                        ref short coefficients = ref GetWarpedFilterReference(phase);
-                        int sum = horizontalBias + TConvolution.Convolve(ref Unsafe.Add(ref sourceBase, sourceIndex), ref coefficients);
-                        intermediate[((row + 7) * WarpedTileSize) + column + 4] = (ushort)RoundPowerOfTwoScalar(sum, round0);
-                        phase += parameters.Alpha;
-                    }
-                }
-
-                int tileHeight = Math.Min(WarpedTileSize, destinationPosition.Y + height - tileRow);
-                int tileWidth = Math.Min(WarpedTileSize, destinationPosition.X + width - tileColumn);
-                for (int row = 0; row < tileHeight; row++)
-                {
-                    int phase = phaseY + (parameters.Delta * row);
-                    int destinationRowOffset = (tileRow - destinationPosition.Y + row) * destinationStride;
-                    for (int column = 0; column < tileWidth; column++)
-                    {
-                        ref ushort intermediateSource = ref intermediate[(row * WarpedTileSize) + column];
-                        ref short coefficients = ref GetWarpedFilterReference(phase);
-                        int sum = verticalBias + TConvolution.ConvolveVertical(ref intermediateSource, WarpedTileSize, ref coefficients);
-                        int value = RoundPowerOfTwoScalar(sum, verticalRound) - (1 << (bitDepth - 1)) - (1 << bitDepth);
-                        Unsafe.Add(ref destinationBase, destinationRowOffset + tileColumn - destinationPosition.X + column) =
-                            (ushort)Math.Clamp(value, 0, maximum);
-
-                        phase += parameters.Gamma;
-                    }
-                }
-            }
-        }
-    }
-
-    /// <summary>
-    /// Projects the center of one 8x8 output tile and derives its integer source position and reduced phases.
-    /// </summary>
-    private static void DeriveWarpedTilePosition(
-        Av1GlobalMotionParameters parameters,
-        int tileColumn,
-        int tileRow,
-        int subsamplingX,
-        int subsamplingY,
-        out int integerX,
-        out int integerY,
-        out int phaseX,
-        out int phaseY)
-    {
-        int sourceX = (tileColumn + 4) << subsamplingX;
-        int sourceY = (tileRow + 4) << subsamplingY;
-        long projectedX = ((long)parameters[2] * sourceX) + ((long)parameters[3] * sourceY) + parameters[0];
-        long projectedY = ((long)parameters[4] * sourceX) + ((long)parameters[5] * sourceY) + parameters[1];
-        long planeX = projectedX >> subsamplingX;
-        long planeY = projectedY >> subsamplingY;
-        integerX = (int)(planeX >> Av1GlobalMotionParameters.ModelPrecisionBits);
-        integerY = (int)(planeY >> Av1GlobalMotionParameters.ModelPrecisionBits);
-        phaseX = (int)planeX & (Av1GlobalMotionParameters.ModelScale - 1);
-        phaseY = (int)planeY & (Av1GlobalMotionParameters.ModelScale - 1);
-        phaseX += (-4 * parameters.Alpha) + (-4 * parameters.Beta);
-        phaseY += (-4 * parameters.Gamma) + (-4 * parameters.Delta);
-
-        // Shear parameters are quantized to 64-model-unit steps. Clearing the same low bits after the tile-center
-        // projection keeps negative and positive phases on the exact filter-table grid used by the bitstream model.
-        phaseX &= -1 << 6;
-        phaseY &= -1 << 6;
-    }
-
-    /// <summary>
-    /// Gets a reference to the first coefficient for one reduced warped-filter phase.
-    /// </summary>
-    private static ref short GetWarpedFilterReference(int phase)
-    {
-        int filterIndex = ((phase + (1 << (WarpedDifferencePrecisionBits - 1))) >> WarpedDifferencePrecisionBits) +
-            WarpedPixelPrecisionShifts;
-
-        return ref Unsafe.Add(ref MemoryMarshal.GetReference(WarpedFilter), filterIndex * FilterCoefficientCount);
-    }
-
-    /// <summary>
-    /// Divides a nonnegative value by a power of two with nearest-integer rounding.
-    /// </summary>
-    private static int RoundPowerOfTwoScalar(int value, int bitCount)
-        => (value + (1 << (bitCount - 1))) >> bitCount;
-
-    /// <summary>
-    /// Executes warped dot products with portable 128-bit SIMD.
-    /// </summary>
-    private readonly struct WarpedVector128Convolution : IWarpedConvolution
-    {
-        /// <inheritdoc/>
-        public static int Convolve(ref byte source, ref short coefficients)
-        {
-            Vector128<byte> packed = Vector128.LoadUnsafe(ref source);
-            (Vector128<ushort> samples, _) = Vector128.Widen(packed);
-            return MultiplyAndSum(samples, Vector128.LoadUnsafe(ref coefficients));
-        }
-
-        /// <inheritdoc/>
-        public static int Convolve(ref ushort source, ref short coefficients)
-            => MultiplyAndSum(Vector128.LoadUnsafe(ref source), Vector128.LoadUnsafe(ref coefficients));
-
-        /// <inheritdoc/>
-        public static int ConvolveVertical(ref ushort source, int stride, ref short coefficients)
-        {
-            Vector128<ushort> samples = Vector128.Create(
-                source,
-                Unsafe.Add(ref source, stride),
-                Unsafe.Add(ref source, stride * 2),
-                Unsafe.Add(ref source, stride * 3),
-                Unsafe.Add(ref source, stride * 4),
-                Unsafe.Add(ref source, stride * 5),
-                Unsafe.Add(ref source, stride * 6),
-                Unsafe.Add(ref source, stride * 7));
-
-            return MultiplyAndSum(samples, Vector128.LoadUnsafe(ref coefficients));
-        }
-
-        /// <summary>
-        /// Widens unsigned samples and signed coefficients before accumulating their exact 32-bit products.
-        /// </summary>
-        private static int MultiplyAndSum(Vector128<ushort> samples, Vector128<short> coefficients)
-        {
-            (Vector128<uint> sampleLower, Vector128<uint> sampleUpper) = Vector128.Widen(samples);
-            (Vector128<int> coefficientLower, Vector128<int> coefficientUpper) = Vector128.Widen(coefficients);
-            return Vector128.Sum(sampleLower.AsInt32() * coefficientLower) +
-                Vector128.Sum(sampleUpper.AsInt32() * coefficientUpper);
-        }
-    }
-
-    /// <summary>
-    /// Executes warped dot products without explicit hardware intrinsics.
-    /// </summary>
-    private readonly struct WarpedScalarConvolution : IWarpedConvolution
-    {
-        /// <inheritdoc/>
-        public static int Convolve(ref byte source, ref short coefficients)
-        {
-            int sum = 0;
-            for (nuint index = 0; index < FilterCoefficientCount; index++)
-            {
-                sum += Unsafe.Add(ref source, index) * Unsafe.Add(ref coefficients, index);
-            }
-
-            return sum;
-        }
-
-        /// <inheritdoc/>
-        public static int Convolve(ref ushort source, ref short coefficients)
-        {
-            int sum = 0;
-            for (nuint index = 0; index < FilterCoefficientCount; index++)
-            {
-                sum += Unsafe.Add(ref source, index) * Unsafe.Add(ref coefficients, index);
-            }
-
-            return sum;
-        }
-
-        /// <inheritdoc/>
-        public static int ConvolveVertical(ref ushort source, int stride, ref short coefficients)
-        {
-            int sum = 0;
-            for (int index = 0; index < FilterCoefficientCount; index++)
-            {
-                sum += Unsafe.Add(ref source, index * stride) * Unsafe.Add(ref coefficients, index);
-            }
-
-            return sum;
-        }
-    }
-}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.cs

@@ -25,12 +25,12 @@ internal static partial class Av1InterPredictor
     /// <summary>
     /// The number of fractional bits in each interpolation coefficient.
     /// </summary>
-    private const int FilterBits = 7;
+    internal const int FilterBits = 7;
 
     /// <summary>
     /// The normal first-round shift used by libaom single-reference convolution.
     /// </summary>
-    private const int Round0Bits = 3;
+    internal const int Round0Bits = 3;
 
     /// <summary>
     /// The maximum number of source rows added by an eight-tap vertical filter.
@@ -40,7 +40,7 @@ internal static partial class Av1InterPredictor
     /// <summary>
     /// The minimum scratch stride that lets a 128-bit byte kernel handle four- and eight-sample blocks.
     /// </summary>
-    private const int MinimumScratchStride = 16;
+    internal const int MinimumScratchStride = 16;
 
     /// <summary>
     /// Gets the maximum number of signed 16-bit elements required for one two-dimensional prediction block.

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1ScaledInterPredictor.Operator.cs

@@ -0,0 +1,364 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.CompilerServices;
+using System.Runtime.Intrinsics;
+
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
+
+/// <content>
+/// Defines variable-phase reference-scaled prediction arithmetic.
+/// </content>
+internal static partial class Av1ScaledInterPredictor
+{
+    /// <summary>
+    /// Defines variable-phase scaled prediction for one native sample storage type.
+    /// </summary>
+    private interface IAv1ScaledPredictionOperator
+    {
+        /// <summary>
+        /// Loads one native source sample as a signed accumulator value.
+        /// </summary>
+        /// <typeparam name="T">The native sample storage type.</typeparam>
+        /// <param name="source">The first native source sample.</param>
+        /// <param name="index">The source sample offset.</param>
+        /// <returns>The widened sample.</returns>
+        public static abstract int Load<T>(ref T source, int index)
+            where T : unmanaged;
+
+        /// <summary>
+        /// Accumulates one sample-coefficient product.
+        /// </summary>
+        /// <param name="accumulator">The current convolution sum.</param>
+        /// <param name="sample">The source sample.</param>
+        /// <param name="coefficient">The signed Q7 coefficient.</param>
+        /// <returns>The updated convolution sum.</returns>
+        public static abstract int MultiplyAdd(int accumulator, int sample, int coefficient);
+
+        /// <summary>
+        /// Accumulates four independent sample-coefficient products.
+        /// </summary>
+        /// <param name="accumulator">The current convolution sums.</param>
+        /// <param name="samples">The source samples.</param>
+        /// <param name="coefficients">The signed Q7 coefficients.</param>
+        /// <returns>The updated convolution sums.</returns>
+        public static abstract Vector128<int> MultiplyAdd(Vector128<int> accumulator, Vector128<int> samples, Vector128<int> coefficients);
+
+        /// <summary>
+        /// Accumulates eight independent sample-coefficient products.
+        /// </summary>
+        /// <param name="accumulator">The current convolution sums.</param>
+        /// <param name="samples">The source samples.</param>
+        /// <param name="coefficients">The signed Q7 coefficients.</param>
+        /// <returns>The updated convolution sums.</returns>
+        public static abstract Vector256<int> MultiplyAdd(Vector256<int> accumulator, Vector256<int> samples, Vector256<int> coefficients);
+
+        /// <summary>
+        /// Accumulates sixteen independent sample-coefficient products.
+        /// </summary>
+        /// <param name="accumulator">The current convolution sums.</param>
+        /// <param name="samples">The source samples.</param>
+        /// <param name="coefficients">The signed Q7 coefficients.</param>
+        /// <returns>The updated convolution sums.</returns>
+        public static abstract Vector512<int> MultiplyAdd(Vector512<int> accumulator, Vector512<int> samples, Vector512<int> coefficients);
+
+        /// <summary>
+        /// Convolves one intermediate sample column without hardware intrinsics.
+        /// </summary>
+        /// <param name="source">The first intermediate sample.</param>
+        /// <param name="sourceStride">The distance between intermediate rows.</param>
+        /// <param name="coefficients">The first signed Q7 coefficient.</param>
+        /// <param name="coefficientCount">The number of active coefficients.</param>
+        /// <returns>The exact convolution sum.</returns>
+        public static abstract int Convolve(ref short source, int sourceStride, ref short coefficients, int coefficientCount);
+
+        /// <summary>
+        /// Convolves eight adjacent intermediate samples through a 128-bit lane group.
+        /// </summary>
+        /// <param name="source">The first intermediate sample.</param>
+        /// <param name="sourceStride">The distance between intermediate rows.</param>
+        /// <param name="sourceOffset">The first column offset.</param>
+        /// <param name="coefficients">The first signed Q7 coefficient.</param>
+        /// <param name="coefficientCount">The number of active coefficients.</param>
+        /// <param name="initial">The initial convolution bias.</param>
+        /// <param name="result0">Receives the first four completed sums.</param>
+        /// <param name="result1">Receives the next four completed sums.</param>
+        public static abstract void Convolve(
+            ref short source,
+            int sourceStride,
+            nuint sourceOffset,
+            ref short coefficients,
+            int coefficientCount,
+            Vector128<int> initial,
+            out Vector128<int> result0,
+            out Vector128<int> result1);
+
+        /// <summary>
+        /// Convolves sixteen adjacent intermediate samples through a 256-bit lane group.
+        /// </summary>
+        /// <param name="source">The first intermediate sample.</param>
+        /// <param name="sourceStride">The distance between intermediate rows.</param>
+        /// <param name="sourceOffset">The first column offset.</param>
+        /// <param name="coefficients">The first signed Q7 coefficient.</param>
+        /// <param name="coefficientCount">The number of active coefficients.</param>
+        /// <param name="initial">The initial convolution bias.</param>
+        /// <param name="result0">Receives the first eight completed sums.</param>
+        /// <param name="result1">Receives the next eight completed sums.</param>
+        public static abstract void Convolve(
+            ref short source,
+            int sourceStride,
+            nuint sourceOffset,
+            ref short coefficients,
+            int coefficientCount,
+            Vector256<int> initial,
+            out Vector256<int> result0,
+            out Vector256<int> result1);
+
+        /// <summary>
+        /// Convolves thirty-two adjacent intermediate samples through a 512-bit lane group.
+        /// </summary>
+        /// <param name="source">The first intermediate sample.</param>
+        /// <param name="sourceStride">The distance between intermediate rows.</param>
+        /// <param name="sourceOffset">The first column offset.</param>
+        /// <param name="coefficients">The first signed Q7 coefficient.</param>
+        /// <param name="coefficientCount">The number of active coefficients.</param>
+        /// <param name="initial">The initial convolution bias.</param>
+        /// <param name="result0">Receives the first sixteen completed sums.</param>
+        /// <param name="result1">Receives the next sixteen completed sums.</param>
+        public static abstract void Convolve(
+            ref short source,
+            int sourceStride,
+            nuint sourceOffset,
+            ref short coefficients,
+            int coefficientCount,
+            Vector512<int> initial,
+            out Vector512<int> result0,
+            out Vector512<int> result1);
+
+        /// <summary>
+        /// Clips and stores one completed prediction.
+        /// </summary>
+        /// <typeparam name="T">The native sample storage type.</typeparam>
+        /// <param name="destination">The first destination sample.</param>
+        /// <param name="index">The destination offset.</param>
+        /// <param name="value">The completed prediction.</param>
+        /// <param name="bitDepth">The decoded sample precision.</param>
+        public static abstract void Store<T>(ref T destination, int index, int value, int bitDepth)
+            where T : unmanaged;
+
+        /// <summary>
+        /// Clips and stores eight completed predictions.
+        /// </summary>
+        /// <typeparam name="T">The native sample storage type.</typeparam>
+        /// <param name="destination">The first destination sample.</param>
+        /// <param name="index">The destination offset.</param>
+        /// <param name="result0">The first four completed predictions.</param>
+        /// <param name="result1">The next four completed predictions.</param>
+        /// <param name="bitDepth">The decoded sample precision.</param>
+        public static abstract void Store<T>(ref T destination, int index, Vector128<int> result0, Vector128<int> result1, int bitDepth)
+            where T : unmanaged;
+
+        /// <summary>
+        /// Clips and stores sixteen completed predictions.
+        /// </summary>
+        /// <typeparam name="T">The native sample storage type.</typeparam>
+        /// <param name="destination">The first destination sample.</param>
+        /// <param name="index">The destination offset.</param>
+        /// <param name="result0">The first eight completed predictions.</param>
+        /// <param name="result1">The next eight completed predictions.</param>
+        /// <param name="bitDepth">The decoded sample precision.</param>
+        public static abstract void Store<T>(ref T destination, int index, Vector256<int> result0, Vector256<int> result1, int bitDepth)
+            where T : unmanaged;
+
+        /// <summary>
+        /// Clips and stores thirty-two completed predictions.
+        /// </summary>
+        /// <typeparam name="T">The native sample storage type.</typeparam>
+        /// <param name="destination">The first destination sample.</param>
+        /// <param name="index">The destination offset.</param>
+        /// <param name="result0">The first sixteen completed predictions.</param>
+        /// <param name="result1">The next sixteen completed predictions.</param>
+        /// <param name="bitDepth">The decoded sample precision.</param>
+        public static abstract void Store<T>(ref T destination, int index, Vector512<int> result0, Vector512<int> result1, int bitDepth)
+            where T : unmanaged;
+    }
+
+    /// <summary>
+    /// Implements variable-phase scaled prediction for scalar and SIMD lane groups.
+    /// </summary>
+    private readonly struct ScaledOperator : IAv1ScaledPredictionOperator
+    {
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static int Load<T>(ref T source, int index)
+            where T : unmanaged
+        {
+            // The only closed forms are byte and ushort. The JIT removes this storage choice from each specialization,
+            // leaving the shared variable-phase traversal free of duplicate 8-bit and high-bit-depth implementations.
+            if (typeof(T) == typeof(byte))
+            {
+                return Unsafe.Add(ref Unsafe.As<T, byte>(ref source), index);
+            }
+
+            return Unsafe.Add(ref Unsafe.As<T, ushort>(ref source), index);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static int MultiplyAdd(int accumulator, int sample, int coefficient)
+            => accumulator + (sample * coefficient);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector128<int> MultiplyAdd(Vector128<int> accumulator, Vector128<int> samples, Vector128<int> coefficients)
+            => accumulator + (samples * coefficients);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector256<int> MultiplyAdd(Vector256<int> accumulator, Vector256<int> samples, Vector256<int> coefficients)
+            => accumulator + (samples * coefficients);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector512<int> MultiplyAdd(Vector512<int> accumulator, Vector512<int> samples, Vector512<int> coefficients)
+            => accumulator + (samples * coefficients);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static int Convolve(ref short source, int sourceStride, ref short coefficients, int coefficientCount)
+            => ConvolveScalar(ref source, sourceStride, ref coefficients, coefficientCount);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static void Convolve(
+            ref short source,
+            int sourceStride,
+            nuint sourceOffset,
+            ref short coefficients,
+            int coefficientCount,
+            Vector128<int> initial,
+            out Vector128<int> result0,
+            out Vector128<int> result1)
+            => Av1InterPredictor.Convolve(
+                ref source,
+                sourceStride,
+                sourceOffset,
+                ref coefficients,
+                coefficientCount,
+                initial,
+                out result0,
+                out result1);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static void Convolve(
+            ref short source,
+            int sourceStride,
+            nuint sourceOffset,
+            ref short coefficients,
+            int coefficientCount,
+            Vector256<int> initial,
+            out Vector256<int> result0,
+            out Vector256<int> result1)
+            => Av1InterPredictor.Convolve(
+                ref source,
+                sourceStride,
+                sourceOffset,
+                ref coefficients,
+                coefficientCount,
+                initial,
+                out result0,
+                out result1);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static void Convolve(
+            ref short source,
+            int sourceStride,
+            nuint sourceOffset,
+            ref short coefficients,
+            int coefficientCount,
+            Vector512<int> initial,
+            out Vector512<int> result0,
+            out Vector512<int> result1)
+            => Av1InterPredictor.Convolve(
+                ref source,
+                sourceStride,
+                sourceOffset,
+                ref coefficients,
+                coefficientCount,
+                initial,
+                out result0,
+                out result1);
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static void Store<T>(ref T destination, int index, int value, int bitDepth)
+            where T : unmanaged
+        {
+            if (typeof(T) == typeof(byte))
+            {
+                Unsafe.Add(ref Unsafe.As<T, byte>(ref destination), index) = (byte)Math.Clamp(value, byte.MinValue, byte.MaxValue);
+                return;
+            }
+
+            Unsafe.Add(ref Unsafe.As<T, ushort>(ref destination), index) = (ushort)Math.Clamp(value, 0, (1 << bitDepth) - 1);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static void Store<T>(ref T destination, int index, Vector128<int> result0, Vector128<int> result1, int bitDepth)
+            where T : unmanaged
+        {
+            if (typeof(T) == typeof(byte))
+            {
+                PackBytes(result0, result1, Vector128<int>.Zero, Vector128<int>.Zero)
+                    .GetLower()
+                    .StoreUnsafe(ref Unsafe.As<T, byte>(ref destination), (nuint)index);
+
+                return;
+            }
+
+            PackHighBitDepth(result0, result1, (1 << bitDepth) - 1)
+                .StoreUnsafe(ref Unsafe.As<T, ushort>(ref destination), (nuint)index);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static void Store<T>(ref T destination, int index, Vector256<int> result0, Vector256<int> result1, int bitDepth)
+            where T : unmanaged
+        {
+            if (typeof(T) == typeof(byte))
+            {
+                PackBytes(result0, result1, Vector256<int>.Zero, Vector256<int>.Zero)
+                    .GetLower()
+                    .StoreUnsafe(ref Unsafe.As<T, byte>(ref destination), (nuint)index);
+
+                return;
+            }
+
+            PackHighBitDepth(result0, result1, (1 << bitDepth) - 1)
+                .StoreUnsafe(ref Unsafe.As<T, ushort>(ref destination), (nuint)index);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static void Store<T>(ref T destination, int index, Vector512<int> result0, Vector512<int> result1, int bitDepth)
+            where T : unmanaged
+        {
+            if (typeof(T) == typeof(byte))
+            {
+                PackBytes(result0, result1, Vector512<int>.Zero, Vector512<int>.Zero)
+                    .GetLower()
+                    .StoreUnsafe(ref Unsafe.As<T, byte>(ref destination), (nuint)index);
+
+                return;
+            }
+
+            PackHighBitDepth(result0, result1, (1 << bitDepth) - 1)
+                .StoreUnsafe(ref Unsafe.As<T, ushort>(ref destination), (nuint)index);
+        }
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Scaled.cs → src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1ScaledInterPredictor.cs

@@ -5,12 +5,14 @@ using System.Runtime.CompilerServices;
 using System.Runtime.InteropServices;
 using System.Runtime.Intrinsics;
 
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor;
+
 namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
 
 /// <content>
 /// Reconstructs reference-scaled inter prediction through variable-phase separable convolution.
 /// </content>
-internal static partial class Av1InterPredictor
+internal static partial class Av1ScaledInterPredictor
 {
     /// <summary>
     /// Gets the scratch capacity required by one scaled prediction block.
@@ -45,7 +47,7 @@ internal static partial class Av1InterPredictor
         int verticalPhase,
         int verticalStep,
         Span<short> scratch)
-        => DispatchScaled<byte, ScaledByteOperator>(
+        => DispatchScaled<byte, ScaledOperator>(
             source,
             sourceStride,
             sourceOrigin,
@@ -81,7 +83,7 @@ internal static partial class Av1InterPredictor
         int verticalStep,
         int bitDepth,
         Span<short> scratch)
-        => DispatchScaled<ushort, ScaledUInt16Operator>(
+        => DispatchScaled<ushort, ScaledOperator>(
             source,
             sourceStride,
             sourceOrigin,
@@ -118,7 +120,7 @@ internal static partial class Av1InterPredictor
         int bitDepth,
         Span<short> scratch)
         where T : unmanaged
-        where TSample : struct, IScaledSampleOperator<T>
+        where TSample : struct, IAv1ScaledPredictionOperator
     {
         switch (horizontalFilter)
         {
@@ -216,7 +218,7 @@ internal static partial class Av1InterPredictor
         int bitDepth,
         Span<short> scratch)
         where T : unmanaged
-        where TSample : struct, IScaledSampleOperator<T>
+        where TSample : struct, IAv1ScaledPredictionOperator
         where THorizontal : struct, IAv1InterPredictorOperator
     {
         switch (verticalFilter)
@@ -310,7 +312,7 @@ internal static partial class Av1InterPredictor
         int bitDepth,
         Span<short> scratch)
         where T : unmanaged
-        where TSample : struct, IScaledSampleOperator<T>
+        where TSample : struct, IAv1ScaledPredictionOperator
         where THorizontal : struct, IAv1InterPredictorOperator
         where TVertical : struct, IAv1InterPredictorOperator
     {
@@ -332,45 +334,63 @@ internal static partial class Av1InterPredictor
             ref T sourceRow = ref Unsafe.Add(ref sourceBase, (row - 3) * sourceStride);
             ref short scratchRow = ref Unsafe.Add(ref scratchBase, row * scratchStride);
             int column = 0;
+
+            if (Vector512.IsHardwareAccelerated)
+            {
+                int oneVectorFromEnd = width - Vector512<int>.Count;
+                for (; column <= oneVectorFromEnd; column += Vector512<int>.Count)
+                {
+                    Vector512<int> result = FilterScaledHorizontalVector512<T, TSample, THorizontal>(
+                        ref sourceRow,
+                        horizontalPhase,
+                        horizontalStep,
+                        column,
+                        useReducedHorizontalFilter,
+                        horizontalBias,
+                        round0);
+
+                    Av1IntraPredictorBase.Narrow(result, Vector512<int>.Zero)
+                        .GetLower()
+                        .StoreUnsafe(ref scratchRow, (nuint)column);
+                }
+            }
+
+            if (Vector256.IsHardwareAccelerated)
+            {
+                int oneVectorFromEnd = width - Vector256<int>.Count;
+                for (; column <= oneVectorFromEnd; column += Vector256<int>.Count)
+                {
+                    Vector256<int> result = FilterScaledHorizontalVector256<T, TSample, THorizontal>(
+                        ref sourceRow,
+                        horizontalPhase,
+                        horizontalStep,
+                        column,
+                        useReducedHorizontalFilter,
+                        horizontalBias,
+                        round0);
+
+                    Av1IntraPredictorBase.Narrow(result, Vector256<int>.Zero)
+                        .GetLower()
+                        .StoreUnsafe(ref scratchRow, (nuint)column);
+                }
+            }
+
             if (Vector128.IsHardwareAccelerated)
             {
                 for (; column <= width - Vector128<int>.Count; column += Vector128<int>.Count)
                 {
-                    int position0 = horizontalPhase + (column * horizontalStep);
-                    int position1 = position0 + horizontalStep;
-                    int position2 = position1 + horizontalStep;
-                    int position3 = position2 + horizontalStep;
-                    int source0 = (position0 >> Av1ReferenceScale.SubpixelBits) - 3;
-                    int source1 = (position1 >> Av1ReferenceScale.SubpixelBits) - 3;
-                    int source2 = (position2 >> Av1ReferenceScale.SubpixelBits) - 3;
-                    int source3 = (position3 >> Av1ReferenceScale.SubpixelBits) - 3;
-                    ReadOnlySpan<short> coefficients0 = THorizontal.GetCoefficients((position0 & Av1ReferenceScale.SubpixelMask) >> 6, useReducedHorizontalFilter);
-                    ReadOnlySpan<short> coefficients1 = THorizontal.GetCoefficients((position1 & Av1ReferenceScale.SubpixelMask) >> 6, useReducedHorizontalFilter);
-                    ReadOnlySpan<short> coefficients2 = THorizontal.GetCoefficients((position2 & Av1ReferenceScale.SubpixelMask) >> 6, useReducedHorizontalFilter);
-                    ReadOnlySpan<short> coefficients3 = THorizontal.GetCoefficients((position3 & Av1ReferenceScale.SubpixelMask) >> 6, useReducedHorizontalFilter);
-                    Vector128<int> result = Vector128.Create(horizontalBias);
-                    for (int tap = 0; tap < FilterCoefficientCount; tap++)
-                    {
-                        Vector128<int> samples = Vector128.Create(
-                            TSample.Load(ref sourceRow, source0 + tap),
-                            TSample.Load(ref sourceRow, source1 + tap),
-                            TSample.Load(ref sourceRow, source2 + tap),
-                            TSample.Load(ref sourceRow, source3 + tap));
-
-                        Vector128<int> coefficients = Vector128.Create(
-                            (int)coefficients0[tap],
-                            coefficients1[tap],
-                            coefficients2[tap],
-                            coefficients3[tap]);
-
-                        result += samples * coefficients;
-                    }
-
-                    Vector64<short> intermediate = Av1IntraPredictorBase.Narrow(
-                        RoundPowerOfTwo(result, round0),
-                        Vector128<int>.Zero).GetLower();
-
-                    intermediate.StoreUnsafe(ref scratchRow, (nuint)column);
+                    Vector128<int> result = FilterScaledHorizontalVector128<T, TSample, THorizontal>(
+                        ref sourceRow,
+                        horizontalPhase,
+                        horizontalStep,
+                        column,
+                        useReducedHorizontalFilter,
+                        horizontalBias,
+                        round0);
+
+                    Av1IntraPredictorBase.Narrow(result, Vector128<int>.Zero)
+                        .GetLower()
+                        .StoreUnsafe(ref scratchRow, (nuint)column);
                 }
             }
 
@@ -385,7 +405,7 @@ internal static partial class Av1InterPredictor
                 int sum = horizontalBias;
                 for (int tap = 0; tap < FilterCoefficientCount; tap++)
                 {
-                    sum += coefficients[tap] * TSample.Load(ref sourceRow, sourceColumn + tap);
+                    sum = TSample.MultiplyAdd(sum, TSample.Load(ref sourceRow, sourceColumn + tap), coefficients[tap]);
                 }
 
                 Unsafe.Add(ref scratchRow, column) = (short)RoundPowerOfTwo(sum, round0);
@@ -408,13 +428,61 @@ internal static partial class Av1InterPredictor
             ref short coefficientBase = ref MemoryMarshal.GetReference(coefficients);
             ref T destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
             int column = 0;
+
+            if (Vector512.IsHardwareAccelerated)
+            {
+                Vector512<int> initial = Vector512.Create(verticalBias);
+                Vector512<int> offset = Vector512.Create(roundOffset);
+                int oneVectorFromEnd = width - (Vector512<int>.Count * 2);
+                for (; column <= oneVectorFromEnd; column += Vector512<int>.Count * 2)
+                {
+                    TSample.Convolve(
+                        ref scratchRow,
+                        scratchStride,
+                        (nuint)column,
+                        ref coefficientBase,
+                        FilterCoefficientCount,
+                        initial,
+                        out Vector512<int> result0,
+                        out Vector512<int> result1);
+
+                    result0 = RoundPowerOfTwo(result0, round1) - offset;
+                    result1 = RoundPowerOfTwo(result1, round1) - offset;
+                    TSample.Store(ref destinationRow, column, result0, result1, bitDepth);
+                }
+            }
+
+            if (Vector256.IsHardwareAccelerated)
+            {
+                Vector256<int> initial = Vector256.Create(verticalBias);
+                Vector256<int> offset = Vector256.Create(roundOffset);
+                int oneVectorFromEnd = width - (Vector256<int>.Count * 2);
+                for (; column <= oneVectorFromEnd; column += Vector256<int>.Count * 2)
+                {
+                    TSample.Convolve(
+                        ref scratchRow,
+                        scratchStride,
+                        (nuint)column,
+                        ref coefficientBase,
+                        FilterCoefficientCount,
+                        initial,
+                        out Vector256<int> result0,
+                        out Vector256<int> result1);
+
+                    result0 = RoundPowerOfTwo(result0, round1) - offset;
+                    result1 = RoundPowerOfTwo(result1, round1) - offset;
+                    TSample.Store(ref destinationRow, column, result0, result1, bitDepth);
+                }
+            }
+
             if (Vector128.IsHardwareAccelerated)
             {
                 Vector128<int> initial = Vector128.Create(verticalBias);
                 Vector128<int> offset = Vector128.Create(roundOffset);
-                for (; column <= width - Vector128<short>.Count; column += Vector128<short>.Count)
+                int oneVectorFromEnd = width - (Vector128<int>.Count * 2);
+                for (; column <= oneVectorFromEnd; column += Vector128<int>.Count * 2)
                 {
-                    Convolve(
+                    TSample.Convolve(
                         ref scratchRow,
                         scratchStride,
                         (nuint)column,
@@ -426,19 +494,19 @@ internal static partial class Av1InterPredictor
 
                     result0 = RoundPowerOfTwo(result0, round1) - offset;
                     result1 = RoundPowerOfTwo(result1, round1) - offset;
-                    TSample.StoreVector(ref destinationRow, column, result0, result1, bitDepth);
+                    TSample.Store(ref destinationRow, column, result0, result1, bitDepth);
                 }
             }
 
             for (; column < width; column++)
             {
-                int sum = verticalBias + ConvolveScalar(
+                int sum = verticalBias + TSample.Convolve(
                     ref Unsafe.Add(ref scratchRow, column),
                     scratchStride,
                     ref coefficientBase,
                     FilterCoefficientCount);
 
-                TSample.StoreScalar(
+                TSample.Store(
                     ref destinationRow,
                     column,
                     RoundPowerOfTwo(sum, round1) - roundOffset,
@@ -446,4 +514,163 @@ internal static partial class Av1InterPredictor
             }
         }
     }
+
+    /// <summary>
+    /// Filters four independently positioned horizontal samples through a closed scaled-prediction operator.
+    /// </summary>
+    private static Vector128<int> FilterScaledHorizontalVector128<T, TOperator, TFilter>(
+        ref T source,
+        int phase,
+        int step,
+        int column,
+        bool useReducedFilter,
+        int bias,
+        int round)
+        where T : unmanaged
+        where TOperator : struct, IAv1ScaledPredictionOperator
+        where TFilter : struct, IAv1InterPredictorOperator
+    {
+        Vector128<int> result = Vector128.Create(bias);
+        for (int tap = 0; tap < FilterCoefficientCount; tap++)
+        {
+            result = TOperator.MultiplyAdd(
+                result,
+                LoadScaledSamplesVector128<T, TOperator>(ref source, phase, step, column, tap),
+                LoadScaledCoefficientsVector128<TFilter>(phase, step, column, tap, useReducedFilter));
+        }
+
+        return RoundPowerOfTwo(result, round);
+    }
+
+    /// <summary>
+    /// Filters eight independently positioned horizontal samples through a closed scaled-prediction operator.
+    /// </summary>
+    private static Vector256<int> FilterScaledHorizontalVector256<T, TOperator, TFilter>(
+        ref T source,
+        int phase,
+        int step,
+        int column,
+        bool useReducedFilter,
+        int bias,
+        int round)
+        where T : unmanaged
+        where TOperator : struct, IAv1ScaledPredictionOperator
+        where TFilter : struct, IAv1InterPredictorOperator
+    {
+        Vector256<int> result = Vector256.Create(bias);
+        for (int tap = 0; tap < FilterCoefficientCount; tap++)
+        {
+            Vector256<int> samples = Vector256.Create(
+                LoadScaledSamplesVector128<T, TOperator>(ref source, phase, step, column, tap),
+                LoadScaledSamplesVector128<T, TOperator>(ref source, phase, step, column + Vector128<int>.Count, tap));
+
+            Vector256<int> coefficients = Vector256.Create(
+                LoadScaledCoefficientsVector128<TFilter>(phase, step, column, tap, useReducedFilter),
+                LoadScaledCoefficientsVector128<TFilter>(phase, step, column + Vector128<int>.Count, tap, useReducedFilter));
+
+            result = TOperator.MultiplyAdd(result, samples, coefficients);
+        }
+
+        return RoundPowerOfTwo(result, round);
+    }
+
+    /// <summary>
+    /// Filters sixteen independently positioned horizontal samples through a closed scaled-prediction operator.
+    /// </summary>
+    private static Vector512<int> FilterScaledHorizontalVector512<T, TOperator, TFilter>(
+        ref T source,
+        int phase,
+        int step,
+        int column,
+        bool useReducedFilter,
+        int bias,
+        int round)
+        where T : unmanaged
+        where TOperator : struct, IAv1ScaledPredictionOperator
+        where TFilter : struct, IAv1InterPredictorOperator
+    {
+        Vector512<int> result = Vector512.Create(bias);
+        for (int tap = 0; tap < FilterCoefficientCount; tap++)
+        {
+            Vector256<int> sampleLower = Vector256.Create(
+                LoadScaledSamplesVector128<T, TOperator>(ref source, phase, step, column, tap),
+                LoadScaledSamplesVector128<T, TOperator>(ref source, phase, step, column + Vector128<int>.Count, tap));
+
+            Vector256<int> sampleUpper = Vector256.Create(
+                LoadScaledSamplesVector128<T, TOperator>(ref source, phase, step, column + Vector256<int>.Count, tap),
+                LoadScaledSamplesVector128<T, TOperator>(ref source, phase, step, column + Vector256<int>.Count + Vector128<int>.Count, tap));
+
+            Vector256<int> coefficientLower = Vector256.Create(
+                LoadScaledCoefficientsVector128<TFilter>(phase, step, column, tap, useReducedFilter),
+                LoadScaledCoefficientsVector128<TFilter>(phase, step, column + Vector128<int>.Count, tap, useReducedFilter));
+
+            Vector256<int> coefficientUpper = Vector256.Create(
+                LoadScaledCoefficientsVector128<TFilter>(phase, step, column + Vector256<int>.Count, tap, useReducedFilter),
+                LoadScaledCoefficientsVector128<TFilter>(phase, step, column + Vector256<int>.Count + Vector128<int>.Count, tap, useReducedFilter));
+
+            result = TOperator.MultiplyAdd(
+                result,
+                Vector512.Create(sampleLower, sampleUpper),
+                Vector512.Create(coefficientLower, coefficientUpper));
+        }
+
+        return RoundPowerOfTwo(result, round);
+    }
+
+    /// <summary>
+    /// Gathers four variable-position source samples for one horizontal filter tap.
+    /// </summary>
+    private static Vector128<int> LoadScaledSamplesVector128<T, TOperator>(
+        ref T source,
+        int phase,
+        int step,
+        int column,
+        int tap)
+        where T : unmanaged
+        where TOperator : struct, IAv1ScaledPredictionOperator
+        => Vector128.Create(
+            LoadScaledSample<T, TOperator>(ref source, phase, step, column, tap),
+            LoadScaledSample<T, TOperator>(ref source, phase, step, column + 1, tap),
+            LoadScaledSample<T, TOperator>(ref source, phase, step, column + 2, tap),
+            LoadScaledSample<T, TOperator>(ref source, phase, step, column + 3, tap));
+
+    /// <summary>
+    /// Gathers four variable-phase coefficients for one horizontal filter tap.
+    /// </summary>
+    private static Vector128<int> LoadScaledCoefficientsVector128<TFilter>(
+        int phase,
+        int step,
+        int column,
+        int tap,
+        bool useReducedFilter)
+        where TFilter : struct, IAv1InterPredictorOperator
+        => Vector128.Create(
+            LoadScaledCoefficient<TFilter>(phase, step, column, tap, useReducedFilter),
+            LoadScaledCoefficient<TFilter>(phase, step, column + 1, tap, useReducedFilter),
+            LoadScaledCoefficient<TFilter>(phase, step, column + 2, tap, useReducedFilter),
+            LoadScaledCoefficient<TFilter>(phase, step, column + 3, tap, useReducedFilter));
+
+    /// <summary>
+    /// Loads one variable-position source sample for a horizontal filter tap.
+    /// </summary>
+    private static int LoadScaledSample<T, TOperator>(ref T source, int phase, int step, int column, int tap)
+        where T : unmanaged
+        where TOperator : struct, IAv1ScaledPredictionOperator
+    {
+        int position = phase + (column * step);
+        int sourceColumn = (position >> Av1ReferenceScale.SubpixelBits) - 3;
+        return TOperator.Load(ref source, sourceColumn + tap);
+    }
+
+    /// <summary>
+    /// Loads one variable-phase horizontal filter coefficient.
+    /// </summary>
+    private static int LoadScaledCoefficient<TFilter>(int phase, int step, int column, int tap, bool useReducedFilter)
+        where TFilter : struct, IAv1InterPredictorOperator
+    {
+        int position = phase + (column * step);
+        return TFilter.GetCoefficients(
+            (position & Av1ReferenceScale.SubpixelMask) >> 6,
+            useReducedFilter)[tap];
+    }
 }

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.WarpedFilters.cs → src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1WarpedInterPredictor.Filters.cs

@@ -1,12 +1,14 @@
 // Copyright (c) Six Labors.
 // Licensed under the Six Labors Split License.
 
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor;
+
 namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
 
 /// <content>
 /// Provides the normative Q7 filter kernels for affine warped-motion prediction.
 /// </content>
-internal static partial class Av1InterPredictor
+internal static partial class Av1WarpedInterPredictor
 {
     /// <summary>
     /// Gets the 193 consecutive eight-tap warped-filter phases spanning fractional positions [-1, 2].

src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1WarpedInterPredictor.Operator.cs

@@ -0,0 +1,141 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.CompilerServices;
+using System.Runtime.Intrinsics;
+
+using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter;
+
+/// <content>
+/// Defines affine warped-motion prediction arithmetic.
+/// </content>
+internal static partial class Av1WarpedInterPredictor
+{
+    /// <summary>
+    /// Defines the affine warped-motion dot product for scalar and SIMD lane groups.
+    /// </summary>
+    /// <remarks>
+    /// Each SIMD overload contains consecutive independent eight-tap filters. The generic warped traversal
+    /// gathers source windows and coefficient phases, while the closed operator owns the exact multiply-and-sum arithmetic.
+    /// </remarks>
+    private interface IAv1WarpedPredictionOperator
+    {
+        /// <summary>
+        /// Convolves one eight-sample window without hardware intrinsics.
+        /// </summary>
+        /// <param name="source">The first source sample.</param>
+        /// <param name="sourceStride">The distance between source samples.</param>
+        /// <param name="coefficients">The first signed Q7 coefficient.</param>
+        /// <returns>The exact dot product.</returns>
+        public static abstract int Convolve(ref byte source, int sourceStride, ref short coefficients);
+
+        /// <summary>
+        /// Convolves one high-bit-depth eight-sample window without hardware intrinsics.
+        /// </summary>
+        /// <param name="source">The first source sample.</param>
+        /// <param name="sourceStride">The distance between source samples.</param>
+        /// <param name="coefficients">The first signed Q7 coefficient.</param>
+        /// <returns>The exact dot product.</returns>
+        public static abstract int Convolve(ref ushort source, int sourceStride, ref short coefficients);
+
+        /// <summary>
+        /// Convolves one packed eight-sample window.
+        /// </summary>
+        /// <param name="samples">The unsigned samples.</param>
+        /// <param name="coefficients">The signed Q7 coefficients.</param>
+        /// <returns>The exact dot product.</returns>
+        public static abstract int Convolve(Vector128<ushort> samples, Vector128<short> coefficients);
+
+        /// <summary>
+        /// Convolves two packed eight-sample windows.
+        /// </summary>
+        /// <param name="samples">The two unsigned sample windows.</param>
+        /// <param name="coefficients">The two signed Q7 coefficient windows.</param>
+        /// <returns>The two exact dot products in the low lanes.</returns>
+        public static abstract Vector128<int> Convolve(Vector256<ushort> samples, Vector256<short> coefficients);
+
+        /// <summary>
+        /// Convolves four packed eight-sample windows.
+        /// </summary>
+        /// <param name="samples">The four unsigned sample windows.</param>
+        /// <param name="coefficients">The four signed Q7 coefficient windows.</param>
+        /// <returns>The four exact dot products.</returns>
+        public static abstract Vector128<int> Convolve(Vector512<ushort> samples, Vector512<short> coefficients);
+    }
+
+    /// <summary>
+    /// Implements the affine warped-motion dot product for scalar and SIMD lane groups.
+    /// </summary>
+    private readonly struct WarpedOperator : IAv1WarpedPredictionOperator
+    {
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static int Convolve(ref byte source, int sourceStride, ref short coefficients)
+        {
+            int sum = 0;
+            for (int index = 0; index < FilterCoefficientCount; index++)
+            {
+                sum += Unsafe.Add(ref source, index * sourceStride) * Unsafe.Add(ref coefficients, index);
+            }
+
+            return sum;
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static int Convolve(ref ushort source, int sourceStride, ref short coefficients)
+        {
+            int sum = 0;
+            for (int index = 0; index < FilterCoefficientCount; index++)
+            {
+                sum += Unsafe.Add(ref source, index * sourceStride) * Unsafe.Add(ref coefficients, index);
+            }
+
+            return sum;
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static int Convolve(Vector128<ushort> samples, Vector128<short> coefficients)
+        {
+            (Vector128<uint> sampleLower, Vector128<uint> sampleUpper) = Vector128.Widen(samples);
+            (Vector128<int> coefficientLower, Vector128<int> coefficientUpper) = Vector128.Widen(coefficients);
+            return Vector128.Sum(sampleLower.AsInt32() * coefficientLower) +
+                Vector128.Sum(sampleUpper.AsInt32() * coefficientUpper);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector128<int> Convolve(Vector256<ushort> samples, Vector256<short> coefficients)
+        {
+            // Widen preserves the two eight-tap windows as separate 256-bit results. Reducing each product vector
+            // therefore produces the two independent predictions without horizontal lane shuffles.
+            (Vector256<uint> sample0, Vector256<uint> sample1) = Vector256.Widen(samples);
+            (Vector256<int> coefficient0, Vector256<int> coefficient1) = Vector256.Widen(coefficients);
+            return Vector128.Create(
+                Vector256.Sum(sample0.AsInt32() * coefficient0),
+                Vector256.Sum(sample1.AsInt32() * coefficient1),
+                0,
+                0);
+        }
+
+        /// <inheritdoc/>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Vector128<int> Convolve(Vector512<ushort> samples, Vector512<short> coefficients)
+        {
+            // The four eight-tap windows occupy four consecutive 256-bit quarters after widening. Multiplication
+            // remains 512-bit; quarter reductions recover the four independent scalar dot products in output order.
+            (Vector512<uint> sampleLower, Vector512<uint> sampleUpper) = Vector512.Widen(samples);
+            (Vector512<int> coefficientLower, Vector512<int> coefficientUpper) = Vector512.Widen(coefficients);
+            Vector512<int> productLower = sampleLower.AsInt32() * coefficientLower;
+            Vector512<int> productUpper = sampleUpper.AsInt32() * coefficientUpper;
+            return Vector128.Create(
+                Vector256.Sum(productLower.GetLower()),
+                Vector256.Sum(productLower.GetUpper()),
+                Vector256.Sum(productUpper.GetLower()),
+                Vector256.Sum(productUpper.GetUpper()));
+        }
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyPredictor.BilinearOperator.cs → src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyBilinearPredictor.Operator.cs

@@ -5,8 +5,121 @@ using System.Runtime.Intrinsics;
 
 namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.IntraBlockCopy;
 
-internal static partial class Av1IntraBlockCopyPredictor
+/// <content>
+/// Defines the closed bilinear intra-block-copy interpolation operator.
+/// </content>
+internal static partial class Av1IntraBlockCopyBilinearPredictor
 {
+    /// <summary>
+    /// Defines bilinear intra-block-copy filtering for scalar and SIMD lane groups.
+    /// </summary>
+    private interface IAv1IntraBlockCopyBilinearOperator
+    {
+        /// <summary>
+        /// Filters one 8-bit sample.
+        /// </summary>
+        /// <param name="topLeft">The integer-position source sample.</param>
+        /// <param name="topRight">The source sample one column to the right.</param>
+        /// <param name="bottomLeft">The source sample one row below.</param>
+        /// <param name="bottomRight">The source sample one row below and one column to the right.</param>
+        /// <returns>The filtered 8-bit sample.</returns>
+        public static abstract byte Filter(byte topLeft, byte topRight, byte bottomLeft, byte bottomRight);
+
+        /// <summary>
+        /// Filters sixteen 8-bit samples in parallel.
+        /// </summary>
+        /// <param name="topLeft">The integer-position source samples.</param>
+        /// <param name="topRight">The source samples one column to the right.</param>
+        /// <param name="bottomLeft">The source samples one row below.</param>
+        /// <param name="bottomRight">The source samples one row below and one column to the right.</param>
+        /// <returns>The filtered 8-bit samples.</returns>
+        public static abstract Vector128<byte> Filter(
+            Vector128<byte> topLeft,
+            Vector128<byte> topRight,
+            Vector128<byte> bottomLeft,
+            Vector128<byte> bottomRight);
+
+        /// <summary>
+        /// Filters thirty-two 8-bit samples in parallel.
+        /// </summary>
+        /// <param name="topLeft">The integer-position source samples.</param>
+        /// <param name="topRight">The source samples one column to the right.</param>
+        /// <param name="bottomLeft">The source samples one row below.</param>
+        /// <param name="bottomRight">The source samples one row below and one column to the right.</param>
+        /// <returns>The filtered 8-bit samples.</returns>
+        public static abstract Vector256<byte> Filter(
+            Vector256<byte> topLeft,
+            Vector256<byte> topRight,
+            Vector256<byte> bottomLeft,
+            Vector256<byte> bottomRight);
+
+        /// <summary>
+        /// Filters sixty-four 8-bit samples in parallel.
+        /// </summary>
+        /// <param name="topLeft">The integer-position source samples.</param>
+        /// <param name="topRight">The source samples one column to the right.</param>
+        /// <param name="bottomLeft">The source samples one row below.</param>
+        /// <param name="bottomRight">The source samples one row below and one column to the right.</param>
+        /// <returns>The filtered 8-bit samples.</returns>
+        public static abstract Vector512<byte> Filter(
+            Vector512<byte> topLeft,
+            Vector512<byte> topRight,
+            Vector512<byte> bottomLeft,
+            Vector512<byte> bottomRight);
+
+        /// <summary>
+        /// Filters one high-bit-depth sample.
+        /// </summary>
+        /// <param name="topLeft">The integer-position source sample.</param>
+        /// <param name="topRight">The source sample one column to the right.</param>
+        /// <param name="bottomLeft">The source sample one row below.</param>
+        /// <param name="bottomRight">The source sample one row below and one column to the right.</param>
+        /// <returns>The filtered high-bit-depth sample.</returns>
+        public static abstract short Filter(short topLeft, short topRight, short bottomLeft, short bottomRight);
+
+        /// <summary>
+        /// Filters eight high-bit-depth samples in parallel.
+        /// </summary>
+        /// <param name="topLeft">The integer-position source samples.</param>
+        /// <param name="topRight">The source samples one column to the right.</param>
+        /// <param name="bottomLeft">The source samples one row below.</param>
+        /// <param name="bottomRight">The source samples one row below and one column to the right.</param>
+        /// <returns>The filtered high-bit-depth samples.</returns>
+        public static abstract Vector128<short> Filter(
+            Vector128<short> topLeft,
+            Vector128<short> topRight,
+            Vector128<short> bottomLeft,
+            Vector128<short> bottomRight);
+
+        /// <summary>
+        /// Filters sixteen high-bit-depth samples in parallel.
+        /// </summary>
+        /// <param name="topLeft">The integer-position source samples.</param>
+        /// <param name="topRight">The source samples one column to the right.</param>
+        /// <param name="bottomLeft">The source samples one row below.</param>
+        /// <param name="bottomRight">The source samples one row below and one column to the right.</param>
+        /// <returns>The filtered high-bit-depth samples.</returns>
+        public static abstract Vector256<short> Filter(
+            Vector256<short> topLeft,
+            Vector256<short> topRight,
+            Vector256<short> bottomLeft,
+            Vector256<short> bottomRight);
+
+        /// <summary>
+        /// Filters thirty-two high-bit-depth samples in parallel.
+        /// </summary>
+        /// <param name="topLeft">The integer-position source samples.</param>
+        /// <param name="topRight">The source samples one column to the right.</param>
+        /// <param name="bottomLeft">The source samples one row below.</param>
+        /// <param name="bottomRight">The source samples one row below and one column to the right.</param>
+        /// <returns>The filtered high-bit-depth samples.</returns>
+        public static abstract Vector512<short> Filter(
+            Vector512<short> topLeft,
+            Vector512<short> topRight,
+            Vector512<short> bottomLeft,
+            Vector512<short> bottomRight);
+    }
+
     /// <summary>
     /// Applies the separable two-dimensional interpolation required when both source axes have a half-sample phase.
     /// </summary>
@@ -19,14 +132,8 @@ internal static partial class Av1IntraBlockCopyPredictor
     /// unsigned 32-bit halves. Narrowing recombines those halves in source-column order after the rounded result has
     /// returned to the original sample range.
     /// </remarks>
-    private readonly struct BilinearOperator : IAv1IntraBlockCopyOperator
+    private readonly struct IntraBlockCopyBilinearOperator : IAv1IntraBlockCopyBilinearOperator
     {
-        /// <inheritdoc/>
-        public static bool UsesRight => true;
-
-        /// <inheritdoc/>
-        public static bool UsesBottom => true;
-
         /// <inheritdoc/>
         public static byte Filter(byte topLeft, byte topRight, byte bottomLeft, byte bottomRight)
             => (byte)((topLeft + topRight + bottomLeft + bottomRight + 2) >> 2);

src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyPredictor.Operator.cs → src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyBilinearPredictor.cs

@@ -8,134 +8,57 @@ using System.Runtime.Intrinsics;
 namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.IntraBlockCopy;
 
 /// <content>
-/// Defines the scalar and SIMD contract for closed intra-block-copy filter operators, and provides their shared
-/// width-progressive SIMD traversal.
+/// Provides the family-owned scalar and width-progressive SIMD traversal for bilinear intra-block-copy prediction.
 /// </content>
-internal static partial class Av1IntraBlockCopyPredictor
+internal static partial class Av1IntraBlockCopyBilinearPredictor
 {
     /// <summary>
-    /// Defines lane-wise arithmetic for one intra-block-copy filter phase.
+    /// Reconstructs an 8-bit filtered intra-block-copy prediction.
     /// </summary>
-    /// <remarks>
-    /// Every SIMD lane corresponds to one output column. The generic traversal supplies the integer source sample and
-    /// its right, lower, and lower-right neighbors; closed operator types allow the JIT to remove unused source loads.
-    /// </remarks>
-    private interface IAv1IntraBlockCopyOperator
-    {
-        /// <summary>
-        /// Gets a value indicating whether the operator consumes the source sample to the right.
-        /// </summary>
-        public static abstract bool UsesRight { get; }
-
-        /// <summary>
-        /// Gets a value indicating whether the operator consumes the source sample on the following row.
-        /// </summary>
-        public static abstract bool UsesBottom { get; }
-
-        /// <summary>
-        /// Filters one 8-bit sample.
-        /// </summary>
-        /// <param name="topLeft">The integer-position source sample.</param>
-        /// <param name="topRight">The source sample one column to the right.</param>
-        /// <param name="bottomLeft">The source sample one row below.</param>
-        /// <param name="bottomRight">The source sample one row below and one column to the right.</param>
-        /// <returns>The filtered 8-bit sample.</returns>
-        public static abstract byte Filter(byte topLeft, byte topRight, byte bottomLeft, byte bottomRight);
-
-        /// <summary>
-        /// Filters sixteen 8-bit samples in parallel.
-        /// </summary>
-        /// <param name="topLeft">The integer-position source samples.</param>
-        /// <param name="topRight">The source samples one column to the right.</param>
-        /// <param name="bottomLeft">The source samples one row below.</param>
-        /// <param name="bottomRight">The source samples one row below and one column to the right.</param>
-        /// <returns>The filtered 8-bit samples.</returns>
-        public static abstract Vector128<byte> Filter(
-            Vector128<byte> topLeft,
-            Vector128<byte> topRight,
-            Vector128<byte> bottomLeft,
-            Vector128<byte> bottomRight);
-
-        /// <summary>
-        /// Filters thirty-two 8-bit samples in parallel.
-        /// </summary>
-        /// <param name="topLeft">The integer-position source samples.</param>
-        /// <param name="topRight">The source samples one column to the right.</param>
-        /// <param name="bottomLeft">The source samples one row below.</param>
-        /// <param name="bottomRight">The source samples one row below and one column to the right.</param>
-        /// <returns>The filtered 8-bit samples.</returns>
-        public static abstract Vector256<byte> Filter(
-            Vector256<byte> topLeft,
-            Vector256<byte> topRight,
-            Vector256<byte> bottomLeft,
-            Vector256<byte> bottomRight);
-
-        /// <summary>
-        /// Filters sixty-four 8-bit samples in parallel.
-        /// </summary>
-        /// <param name="topLeft">The integer-position source samples.</param>
-        /// <param name="topRight">The source samples one column to the right.</param>
-        /// <param name="bottomLeft">The source samples one row below.</param>
-        /// <param name="bottomRight">The source samples one row below and one column to the right.</param>
-        /// <returns>The filtered 8-bit samples.</returns>
-        public static abstract Vector512<byte> Filter(
-            Vector512<byte> topLeft,
-            Vector512<byte> topRight,
-            Vector512<byte> bottomLeft,
-            Vector512<byte> bottomRight);
-
-        /// <summary>
-        /// Filters one high-bit-depth sample.
-        /// </summary>
-        /// <param name="topLeft">The integer-position source sample.</param>
-        /// <param name="topRight">The source sample one column to the right.</param>
-        /// <param name="bottomLeft">The source sample one row below.</param>
-        /// <param name="bottomRight">The source sample one row below and one column to the right.</param>
-        /// <returns>The filtered high-bit-depth sample.</returns>
-        public static abstract short Filter(short topLeft, short topRight, short bottomLeft, short bottomRight);
-
-        /// <summary>
-        /// Filters eight high-bit-depth samples in parallel.
-        /// </summary>
-        /// <param name="topLeft">The integer-position source samples.</param>
-        /// <param name="topRight">The source samples one column to the right.</param>
-        /// <param name="bottomLeft">The source samples one row below.</param>
-        /// <param name="bottomRight">The source samples one row below and one column to the right.</param>
-        /// <returns>The filtered high-bit-depth samples.</returns>
-        public static abstract Vector128<short> Filter(
-            Vector128<short> topLeft,
-            Vector128<short> topRight,
-            Vector128<short> bottomLeft,
-            Vector128<short> bottomRight);
-
-        /// <summary>
-        /// Filters sixteen high-bit-depth samples in parallel.
-        /// </summary>
-        /// <param name="topLeft">The integer-position source samples.</param>
-        /// <param name="topRight">The source samples one column to the right.</param>
-        /// <param name="bottomLeft">The source samples one row below.</param>
-        /// <param name="bottomRight">The source samples one row below and one column to the right.</param>
-        /// <returns>The filtered high-bit-depth samples.</returns>
-        public static abstract Vector256<short> Filter(
-            Vector256<short> topLeft,
-            Vector256<short> topRight,
-            Vector256<short> bottomLeft,
-            Vector256<short> bottomRight);
-
-        /// <summary>
-        /// Filters thirty-two high-bit-depth samples in parallel.
-        /// </summary>
-        /// <param name="topLeft">The integer-position source samples.</param>
-        /// <param name="topRight">The source samples one column to the right.</param>
-        /// <param name="bottomLeft">The source samples one row below.</param>
-        /// <param name="bottomRight">The source samples one row below and one column to the right.</param>
-        /// <returns>The filtered high-bit-depth samples.</returns>
-        public static abstract Vector512<short> Filter(
-            Vector512<short> topLeft,
-            Vector512<short> topRight,
-            Vector512<short> bottomLeft,
-            Vector512<short> bottomRight);
-    }
+    public static void Predict(
+        ReadOnlySpan<byte> source,
+        int sourceStride,
+        Span<byte> destination,
+        int destinationStride,
+        int width,
+        int height)
+        => Predict<IntraBlockCopyBilinearOperator>(source, sourceStride, destination, destinationStride, width, height);
+
+    /// <summary>
+    /// Reconstructs a high-bit-depth filtered intra-block-copy prediction.
+    /// </summary>
+    public static void Predict(
+        ReadOnlySpan<short> source,
+        int sourceStride,
+        Span<short> destination,
+        int destinationStride,
+        int width,
+        int height)
+        => Predict<IntraBlockCopyBilinearOperator>(source, sourceStride, destination, destinationStride, width, height);
+
+    /// <summary>
+    /// Reconstructs an 8-bit filtered intra-block-copy prediction without explicit hardware intrinsics.
+    /// </summary>
+    public static void PredictScalar(
+        ReadOnlySpan<byte> source,
+        int sourceStride,
+        Span<byte> destination,
+        int destinationStride,
+        int width,
+        int height)
+        => PredictScalar<IntraBlockCopyBilinearOperator>(source, sourceStride, destination, destinationStride, width, height);
+
+    /// <summary>
+    /// Reconstructs a high-bit-depth filtered intra-block-copy prediction without explicit hardware intrinsics.
+    /// </summary>
+    public static void PredictScalar(
+        ReadOnlySpan<short> source,
+        int sourceStride,
+        Span<short> destination,
+        int destinationStride,
+        int width,
+        int height)
+        => PredictScalar<IntraBlockCopyBilinearOperator>(source, sourceStride, destination, destinationStride, width, height);
 
     /// <summary>
     /// Applies one closed interpolation operator to an 8-bit source block.
@@ -148,7 +71,7 @@ internal static partial class Av1IntraBlockCopyPredictor
         int destinationStride,
         int width,
         int height)
-        where TOperator : struct, IAv1IntraBlockCopyOperator
+        where TOperator : struct, IAv1IntraBlockCopyBilinearOperator
     {
         ref byte sourceBase = ref MemoryMarshal.GetReference(source);
         ref byte destinationBase = ref MemoryMarshal.GetReference(destination);
@@ -163,11 +86,9 @@ internal static partial class Av1IntraBlockCopyPredictor
                 ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride);
                 ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
                 Vector128<byte> topLeft = Vector128.LoadUnsafe(ref sourceRow);
-                Vector128<byte> topRight = TOperator.UsesRight ? Vector128.LoadUnsafe(ref sourceRow, 1) : default;
-                Vector128<byte> bottomLeft = TOperator.UsesBottom ? Vector128.LoadUnsafe(ref sourceRow, (nuint)sourceStride) : default;
-                Vector128<byte> bottomRight = TOperator.UsesRight && TOperator.UsesBottom
-                    ? Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + 1))
-                    : default;
+                Vector128<byte> topRight = Vector128.LoadUnsafe(ref sourceRow, 1);
+                Vector128<byte> bottomLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)sourceStride);
+                Vector128<byte> bottomRight = Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + 1));
 
                 Vector128<byte> prediction = TOperator.Filter(topLeft, topRight, bottomLeft, bottomRight);
                 if (width == 8)
@@ -200,13 +121,9 @@ internal static partial class Av1IntraBlockCopyPredictor
                     for (int column = 0; column < vectorizedColumns; column += Vector512<byte>.Count)
                     {
                         Vector512<byte> topLeft = Vector512.LoadUnsafe(ref sourceRow, (nuint)column);
-                        Vector512<byte> topRight = TOperator.UsesRight ? Vector512.LoadUnsafe(ref sourceRow, (nuint)(column + 1)) : default;
-                        Vector512<byte> bottomLeft = TOperator.UsesBottom
-                            ? Vector512.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column))
-                            : default;
-                        Vector512<byte> bottomRight = TOperator.UsesRight && TOperator.UsesBottom
-                            ? Vector512.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column + 1))
-                            : default;
+                        Vector512<byte> topRight = Vector512.LoadUnsafe(ref sourceRow, (nuint)(column + 1));
+                        Vector512<byte> bottomLeft = Vector512.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column));
+                        Vector512<byte> bottomRight = Vector512.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column + 1));
 
                         TOperator.Filter(topLeft, topRight, bottomLeft, bottomRight).StoreUnsafe(ref destinationRow, (nuint)column);
                     }
@@ -230,13 +147,9 @@ internal static partial class Av1IntraBlockCopyPredictor
                 for (int column = processedColumns; column < endColumn; column += Vector256<byte>.Count)
                 {
                     Vector256<byte> topLeft = Vector256.LoadUnsafe(ref sourceRow, (nuint)column);
-                    Vector256<byte> topRight = TOperator.UsesRight ? Vector256.LoadUnsafe(ref sourceRow, (nuint)(column + 1)) : default;
-                    Vector256<byte> bottomLeft = TOperator.UsesBottom
-                        ? Vector256.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column))
-                        : default;
-                    Vector256<byte> bottomRight = TOperator.UsesRight && TOperator.UsesBottom
-                        ? Vector256.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column + 1))
-                        : default;
+                    Vector256<byte> topRight = Vector256.LoadUnsafe(ref sourceRow, (nuint)(column + 1));
+                    Vector256<byte> bottomLeft = Vector256.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column));
+                    Vector256<byte> bottomRight = Vector256.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column + 1));
 
                     TOperator.Filter(topLeft, topRight, bottomLeft, bottomRight).StoreUnsafe(ref destinationRow, (nuint)column);
                 }
@@ -259,13 +172,9 @@ internal static partial class Av1IntraBlockCopyPredictor
                 for (int column = processedColumns; column < endColumn; column += Vector128<byte>.Count)
                 {
                     Vector128<byte> topLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)column);
-                    Vector128<byte> topRight = TOperator.UsesRight ? Vector128.LoadUnsafe(ref sourceRow, (nuint)(column + 1)) : default;
-                    Vector128<byte> bottomLeft = TOperator.UsesBottom
-                        ? Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column))
-                        : default;
-                    Vector128<byte> bottomRight = TOperator.UsesRight && TOperator.UsesBottom
-                        ? Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column + 1))
-                        : default;
+                    Vector128<byte> topRight = Vector128.LoadUnsafe(ref sourceRow, (nuint)(column + 1));
+                    Vector128<byte> bottomLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column));
+                    Vector128<byte> bottomRight = Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column + 1));
 
                     TOperator.Filter(topLeft, topRight, bottomLeft, bottomRight).StoreUnsafe(ref destinationRow, (nuint)column);
                 }
@@ -284,11 +193,9 @@ internal static partial class Av1IntraBlockCopyPredictor
             for (int column = processedColumns; column < width; column++)
             {
                 byte topLeft = Unsafe.Add(ref sourceRow, column);
-                byte topRight = TOperator.UsesRight ? Unsafe.Add(ref sourceRow, column + 1) : default;
-                byte bottomLeft = TOperator.UsesBottom ? Unsafe.Add(ref sourceRow, sourceStride + column) : default;
-                byte bottomRight = TOperator.UsesRight && TOperator.UsesBottom
-                    ? Unsafe.Add(ref sourceRow, sourceStride + column + 1)
-                    : default;
+                byte topRight = Unsafe.Add(ref sourceRow, column + 1);
+                byte bottomLeft = Unsafe.Add(ref sourceRow, sourceStride + column);
+                byte bottomRight = Unsafe.Add(ref sourceRow, sourceStride + column + 1);
 
                 Unsafe.Add(ref destinationRow, column) = TOperator.Filter(topLeft, topRight, bottomLeft, bottomRight);
             }
@@ -306,7 +213,7 @@ internal static partial class Av1IntraBlockCopyPredictor
         int destinationStride,
         int width,
         int height)
-        where TOperator : struct, IAv1IntraBlockCopyOperator
+        where TOperator : struct, IAv1IntraBlockCopyBilinearOperator
     {
         ref short sourceBase = ref MemoryMarshal.GetReference(source);
         ref short destinationBase = ref MemoryMarshal.GetReference(destination);
@@ -321,11 +228,9 @@ internal static partial class Av1IntraBlockCopyPredictor
                 ref short sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride);
                 ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
                 Vector128<short> topLeft = Vector128.LoadUnsafe(ref sourceRow);
-                Vector128<short> topRight = TOperator.UsesRight ? Vector128.LoadUnsafe(ref sourceRow, 1) : default;
-                Vector128<short> bottomLeft = TOperator.UsesBottom ? Vector128.LoadUnsafe(ref sourceRow, (nuint)sourceStride) : default;
-                Vector128<short> bottomRight = TOperator.UsesRight && TOperator.UsesBottom
-                    ? Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + 1))
-                    : default;
+                Vector128<short> topRight = Vector128.LoadUnsafe(ref sourceRow, 1);
+                Vector128<short> bottomLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)sourceStride);
+                Vector128<short> bottomRight = Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + 1));
 
                 TOperator.Filter(topLeft, topRight, bottomLeft, bottomRight).GetLower().StoreUnsafe(ref destinationRow);
             }
@@ -350,13 +255,9 @@ internal static partial class Av1IntraBlockCopyPredictor
                     for (int column = 0; column < vectorizedColumns; column += Vector512<short>.Count)
                     {
                         Vector512<short> topLeft = Vector512.LoadUnsafe(ref sourceRow, (nuint)column);
-                        Vector512<short> topRight = TOperator.UsesRight ? Vector512.LoadUnsafe(ref sourceRow, (nuint)(column + 1)) : default;
-                        Vector512<short> bottomLeft = TOperator.UsesBottom
-                            ? Vector512.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column))
-                            : default;
-                        Vector512<short> bottomRight = TOperator.UsesRight && TOperator.UsesBottom
-                            ? Vector512.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column + 1))
-                            : default;
+                        Vector512<short> topRight = Vector512.LoadUnsafe(ref sourceRow, (nuint)(column + 1));
+                        Vector512<short> bottomLeft = Vector512.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column));
+                        Vector512<short> bottomRight = Vector512.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column + 1));
 
                         TOperator.Filter(topLeft, topRight, bottomLeft, bottomRight).StoreUnsafe(ref destinationRow, (nuint)column);
                     }
@@ -380,13 +281,9 @@ internal static partial class Av1IntraBlockCopyPredictor
                 for (int column = processedColumns; column < endColumn; column += Vector256<short>.Count)
                 {
                     Vector256<short> topLeft = Vector256.LoadUnsafe(ref sourceRow, (nuint)column);
-                    Vector256<short> topRight = TOperator.UsesRight ? Vector256.LoadUnsafe(ref sourceRow, (nuint)(column + 1)) : default;
-                    Vector256<short> bottomLeft = TOperator.UsesBottom
-                        ? Vector256.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column))
-                        : default;
-                    Vector256<short> bottomRight = TOperator.UsesRight && TOperator.UsesBottom
-                        ? Vector256.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column + 1))
-                        : default;
+                    Vector256<short> topRight = Vector256.LoadUnsafe(ref sourceRow, (nuint)(column + 1));
+                    Vector256<short> bottomLeft = Vector256.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column));
+                    Vector256<short> bottomRight = Vector256.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column + 1));
 
                     TOperator.Filter(topLeft, topRight, bottomLeft, bottomRight).StoreUnsafe(ref destinationRow, (nuint)column);
                 }
@@ -409,13 +306,9 @@ internal static partial class Av1IntraBlockCopyPredictor
                 for (int column = processedColumns; column < endColumn; column += Vector128<short>.Count)
                 {
                     Vector128<short> topLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)column);
-                    Vector128<short> topRight = TOperator.UsesRight ? Vector128.LoadUnsafe(ref sourceRow, (nuint)(column + 1)) : default;
-                    Vector128<short> bottomLeft = TOperator.UsesBottom
-                        ? Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column))
-                        : default;
-                    Vector128<short> bottomRight = TOperator.UsesRight && TOperator.UsesBottom
-                        ? Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column + 1))
-                        : default;
+                    Vector128<short> topRight = Vector128.LoadUnsafe(ref sourceRow, (nuint)(column + 1));
+                    Vector128<short> bottomLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column));
+                    Vector128<short> bottomRight = Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column + 1));
 
                     TOperator.Filter(topLeft, topRight, bottomLeft, bottomRight).StoreUnsafe(ref destinationRow, (nuint)column);
                 }
@@ -432,11 +325,9 @@ internal static partial class Av1IntraBlockCopyPredictor
             for (int column = processedColumns; column < width; column++)
             {
                 short topLeft = Unsafe.Add(ref sourceRow, column);
-                short topRight = TOperator.UsesRight ? Unsafe.Add(ref sourceRow, column + 1) : default;
-                short bottomLeft = TOperator.UsesBottom ? Unsafe.Add(ref sourceRow, sourceStride + column) : default;
-                short bottomRight = TOperator.UsesRight && TOperator.UsesBottom
-                    ? Unsafe.Add(ref sourceRow, sourceStride + column + 1)
-                    : default;
+                short topRight = Unsafe.Add(ref sourceRow, column + 1);
+                short bottomLeft = Unsafe.Add(ref sourceRow, sourceStride + column);
+                short bottomRight = Unsafe.Add(ref sourceRow, sourceStride + column + 1);
 
                 Unsafe.Add(ref destinationRow, column) = TOperator.Filter(topLeft, topRight, bottomLeft, bottomRight);
             }
@@ -454,7 +345,7 @@ internal static partial class Av1IntraBlockCopyPredictor
         int destinationStride,
         int width,
         int height)
-        where TOperator : struct, IAv1IntraBlockCopyOperator
+        where TOperator : struct, IAv1IntraBlockCopyBilinearOperator
     {
         for (int row = 0; row < height; row++)
         {
@@ -464,9 +355,9 @@ internal static partial class Av1IntraBlockCopyPredictor
             for (int column = 0; column < width; column++)
             {
                 byte topLeft = source[sourceRow + column];
-                byte topRight = TOperator.UsesRight ? source[sourceRow + column + 1] : default;
-                byte bottomLeft = TOperator.UsesBottom ? source[sourceRow + sourceStride + column] : default;
-                byte bottomRight = TOperator.UsesRight && TOperator.UsesBottom ? source[sourceRow + sourceStride + column + 1] : default;
+                byte topRight = source[sourceRow + column + 1];
+                byte bottomLeft = source[sourceRow + sourceStride + column];
+                byte bottomRight = source[sourceRow + sourceStride + column + 1];
                 destination[destinationRow + column] = TOperator.Filter(topLeft, topRight, bottomLeft, bottomRight);
             }
         }
@@ -483,7 +374,7 @@ internal static partial class Av1IntraBlockCopyPredictor
         int destinationStride,
         int width,
         int height)
-        where TOperator : struct, IAv1IntraBlockCopyOperator
+        where TOperator : struct, IAv1IntraBlockCopyBilinearOperator
     {
         for (int row = 0; row < height; row++)
         {
@@ -493,9 +384,9 @@ internal static partial class Av1IntraBlockCopyPredictor
             for (int column = 0; column < width; column++)
             {
                 short topLeft = source[sourceRow + column];
-                short topRight = TOperator.UsesRight ? source[sourceRow + column + 1] : default;
-                short bottomLeft = TOperator.UsesBottom ? source[sourceRow + sourceStride + column] : default;
-                short bottomRight = TOperator.UsesRight && TOperator.UsesBottom ? source[sourceRow + sourceStride + column + 1] : default;
+                short topRight = source[sourceRow + column + 1];
+                short bottomLeft = source[sourceRow + sourceStride + column];
+                short bottomRight = source[sourceRow + sourceStride + column + 1];
                 destination[destinationRow + column] = TOperator.Filter(topLeft, topRight, bottomLeft, bottomRight);
             }
         }

src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyHorizontalPredictor.Operator.cs

@@ -0,0 +1,180 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.Intrinsics;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.IntraBlockCopy;
+
+/// <content>
+/// Defines the closed horizontal intra-block-copy interpolation operator.
+/// </content>
+internal static partial class Av1IntraBlockCopyHorizontalPredictor
+{
+    /// <summary>
+    /// Defines horizontal intra-block-copy filtering for scalar and SIMD lane groups.
+    /// </summary>
+    private interface IAv1IntraBlockCopyHorizontalOperator
+    {
+        /// <summary>
+        /// Filters one 8-bit sample.
+        /// </summary>
+        /// <param name="left">The integer-position source sample.</param>
+        /// <param name="right">The source sample one column to the right.</param>
+        /// <returns>The filtered 8-bit sample.</returns>
+        public static abstract byte Filter(byte left, byte right);
+
+        /// <summary>
+        /// Filters sixteen 8-bit samples in parallel.
+        /// </summary>
+        /// <param name="left">The integer-position source samples.</param>
+        /// <param name="right">The source samples one column to the right.</param>
+        /// <returns>The filtered 8-bit samples.</returns>
+        public static abstract Vector128<byte> Filter(
+            Vector128<byte> left,
+            Vector128<byte> right);
+
+        /// <summary>
+        /// Filters thirty-two 8-bit samples in parallel.
+        /// </summary>
+        /// <param name="left">The integer-position source samples.</param>
+        /// <param name="right">The source samples one column to the right.</param>
+        /// <returns>The filtered 8-bit samples.</returns>
+        public static abstract Vector256<byte> Filter(
+            Vector256<byte> left,
+            Vector256<byte> right);
+
+        /// <summary>
+        /// Filters sixty-four 8-bit samples in parallel.
+        /// </summary>
+        /// <param name="left">The integer-position source samples.</param>
+        /// <param name="right">The source samples one column to the right.</param>
+        /// <returns>The filtered 8-bit samples.</returns>
+        public static abstract Vector512<byte> Filter(
+            Vector512<byte> left,
+            Vector512<byte> right);
+
+        /// <summary>
+        /// Filters one high-bit-depth sample.
+        /// </summary>
+        /// <param name="left">The integer-position source sample.</param>
+        /// <param name="right">The source sample one column to the right.</param>
+        /// <returns>The filtered high-bit-depth sample.</returns>
+        public static abstract short Filter(short left, short right);
+
+        /// <summary>
+        /// Filters eight high-bit-depth samples in parallel.
+        /// </summary>
+        /// <param name="left">The integer-position source samples.</param>
+        /// <param name="right">The source samples one column to the right.</param>
+        /// <returns>The filtered high-bit-depth samples.</returns>
+        public static abstract Vector128<short> Filter(
+            Vector128<short> left,
+            Vector128<short> right);
+
+        /// <summary>
+        /// Filters sixteen high-bit-depth samples in parallel.
+        /// </summary>
+        /// <param name="left">The integer-position source samples.</param>
+        /// <param name="right">The source samples one column to the right.</param>
+        /// <returns>The filtered high-bit-depth samples.</returns>
+        public static abstract Vector256<short> Filter(
+            Vector256<short> left,
+            Vector256<short> right);
+
+        /// <summary>
+        /// Filters thirty-two high-bit-depth samples in parallel.
+        /// </summary>
+        /// <param name="left">The integer-position source samples.</param>
+        /// <param name="right">The source samples one column to the right.</param>
+        /// <returns>The filtered high-bit-depth samples.</returns>
+        public static abstract Vector512<short> Filter(
+            Vector512<short> left,
+            Vector512<short> right);
+    }
+
+    /// <summary>
+    /// Averages horizontally adjacent source samples for a half-sample horizontal phase.
+    /// </summary>
+    private readonly struct IntraBlockCopyHorizontalOperator : IAv1IntraBlockCopyHorizontalOperator
+    {
+        /// <inheritdoc/>
+        public static byte Filter(byte left, byte right) => (byte)((left + right + 1) >> 1);
+
+        /// <inheritdoc/>
+        public static Vector128<byte> Filter(Vector128<byte> left, Vector128<byte> right)
+            => AverageRounded(left, right);
+
+        /// <inheritdoc/>
+        public static Vector256<byte> Filter(Vector256<byte> left, Vector256<byte> right)
+            => AverageRounded(left, right);
+
+        /// <inheritdoc/>
+        public static Vector512<byte> Filter(Vector512<byte> left, Vector512<byte> right)
+            => AverageRounded(left, right);
+
+        /// <inheritdoc/>
+        public static short Filter(short left, short right) => (short)((left + right + 1) >> 1);
+
+        /// <inheritdoc/>
+        public static Vector128<short> Filter(Vector128<short> left, Vector128<short> right)
+            => AverageRounded(left, right);
+
+        /// <inheritdoc/>
+        public static Vector256<short> Filter(Vector256<short> left, Vector256<short> right)
+            => AverageRounded(left, right);
+
+        /// <inheritdoc/>
+        public static Vector512<short> Filter(Vector512<short> left, Vector512<short> right)
+            => AverageRounded(left, right);
+
+        /// <summary>
+        /// Computes a rounded average without overflowing unsigned byte lanes.
+        /// </summary>
+        private static Vector128<byte> AverageRounded(Vector128<byte> left, Vector128<byte> right)
+            => (left | right) - ((left ^ right) >> 1);
+
+        /// <summary>
+        /// Computes a rounded average without overflowing unsigned byte lanes.
+        /// </summary>
+        private static Vector256<byte> AverageRounded(Vector256<byte> left, Vector256<byte> right)
+            => (left | right) - ((left ^ right) >> 1);
+
+        /// <summary>
+        /// Computes a rounded average without overflowing unsigned byte lanes.
+        /// </summary>
+        private static Vector512<byte> AverageRounded(Vector512<byte> left, Vector512<byte> right)
+            => (left | right) - ((left ^ right) >> 1);
+
+        /// <summary>
+        /// Computes a rounded average without overflowing nonnegative high-bit-depth lanes.
+        /// </summary>
+        private static Vector128<short> AverageRounded(Vector128<short> left, Vector128<short> right)
+        {
+            Vector128<ushort> leftUnsigned = left.AsUInt16();
+            Vector128<ushort> rightUnsigned = right.AsUInt16();
+
+            // This identity computes ceil((a + b) / 2) without an overflowing lane-wise addition.
+            return ((leftUnsigned | rightUnsigned) - ((leftUnsigned ^ rightUnsigned) >> 1)).AsInt16();
+        }
+
+        /// <summary>
+        /// Computes a rounded average without overflowing nonnegative high-bit-depth lanes.
+        /// </summary>
+        private static Vector256<short> AverageRounded(Vector256<short> left, Vector256<short> right)
+        {
+            Vector256<ushort> leftUnsigned = left.AsUInt16();
+            Vector256<ushort> rightUnsigned = right.AsUInt16();
+            return ((leftUnsigned | rightUnsigned) - ((leftUnsigned ^ rightUnsigned) >> 1)).AsInt16();
+        }
+
+        /// <summary>
+        /// Computes a rounded average without overflowing nonnegative high-bit-depth lanes.
+        /// </summary>
+        private static Vector512<short> AverageRounded(Vector512<short> left, Vector512<short> right)
+        {
+            Vector512<ushort> leftUnsigned = left.AsUInt16();
+            Vector512<ushort> rightUnsigned = right.AsUInt16();
+            return ((leftUnsigned | rightUnsigned) - ((leftUnsigned ^ rightUnsigned) >> 1)).AsInt16();
+        }
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyHorizontalPredictor.cs

@@ -0,0 +1,370 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.CompilerServices;
+using System.Runtime.InteropServices;
+using System.Runtime.Intrinsics;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.IntraBlockCopy;
+
+/// <content>
+/// Provides the family-owned scalar and width-progressive SIMD traversal for horizontal intra-block-copy prediction.
+/// </content>
+internal static partial class Av1IntraBlockCopyHorizontalPredictor
+{
+    /// <summary>
+    /// Reconstructs an 8-bit filtered intra-block-copy prediction.
+    /// </summary>
+    public static void Predict(
+        ReadOnlySpan<byte> source,
+        int sourceStride,
+        Span<byte> destination,
+        int destinationStride,
+        int width,
+        int height)
+        => Predict<IntraBlockCopyHorizontalOperator>(source, sourceStride, destination, destinationStride, width, height);
+
+    /// <summary>
+    /// Reconstructs a high-bit-depth filtered intra-block-copy prediction.
+    /// </summary>
+    public static void Predict(
+        ReadOnlySpan<short> source,
+        int sourceStride,
+        Span<short> destination,
+        int destinationStride,
+        int width,
+        int height)
+        => Predict<IntraBlockCopyHorizontalOperator>(source, sourceStride, destination, destinationStride, width, height);
+
+    /// <summary>
+    /// Reconstructs an 8-bit filtered intra-block-copy prediction without explicit hardware intrinsics.
+    /// </summary>
+    public static void PredictScalar(
+        ReadOnlySpan<byte> source,
+        int sourceStride,
+        Span<byte> destination,
+        int destinationStride,
+        int width,
+        int height)
+        => PredictScalar<IntraBlockCopyHorizontalOperator>(source, sourceStride, destination, destinationStride, width, height);
+
+    /// <summary>
+    /// Reconstructs a high-bit-depth filtered intra-block-copy prediction without explicit hardware intrinsics.
+    /// </summary>
+    public static void PredictScalar(
+        ReadOnlySpan<short> source,
+        int sourceStride,
+        Span<short> destination,
+        int destinationStride,
+        int width,
+        int height)
+        => PredictScalar<IntraBlockCopyHorizontalOperator>(source, sourceStride, destination, destinationStride, width, height);
+
+    /// <summary>
+    /// Applies one closed interpolation operator to an 8-bit source block.
+    /// </summary>
+    /// <typeparam name="TOperator">The source-phase-specific interpolation arithmetic.</typeparam>
+    private static void Predict<TOperator>(
+        ReadOnlySpan<byte> source,
+        int sourceStride,
+        Span<byte> destination,
+        int destinationStride,
+        int width,
+        int height)
+        where TOperator : struct, IAv1IntraBlockCopyHorizontalOperator
+    {
+        ref byte sourceBase = ref MemoryMarshal.GetReference(source);
+        ref byte destinationBase = ref MemoryMarshal.GetReference(destination);
+
+        if (Vector128.IsHardwareAccelerated && width is 4 or 8)
+        {
+            // AV1 permits 4- and 8-sample transform widths, both smaller than a byte Vector128. The frame allocation's
+            // 72-sample prediction border makes each full source load readable; exact-width stores avoid touching
+            // destination padding.
+            for (int row = 0; row < height; row++)
+            {
+                ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride);
+                ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
+                Vector128<byte> topLeft = Vector128.LoadUnsafe(ref sourceRow);
+                Vector128<byte> topRight = Vector128.LoadUnsafe(ref sourceRow, 1);
+
+                Vector128<byte> prediction = TOperator.Filter(topLeft, topRight);
+                if (width == 8)
+                {
+                    prediction.GetLower().StoreUnsafe(ref destinationRow);
+                }
+                else
+                {
+                    Unsafe.As<byte, uint>(ref destinationRow) = prediction.AsUInt32().GetElement(0);
+                }
+            }
+
+            return;
+        }
+
+        int processedColumns = 0;
+
+        // AV1 transform widths are powers of two. The widest supported tier normally consumes the complete row; the
+        // cumulative narrower tiers preserve the same contract for future legal widths without over-reading a tail.
+        if (Vector512.IsHardwareAccelerated)
+        {
+            int vectorizedColumns = width - (width % Vector512<byte>.Count);
+            if (vectorizedColumns > 0)
+            {
+                for (int row = 0; row < height; row++)
+                {
+                    ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride);
+                    ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
+
+                    for (int column = 0; column < vectorizedColumns; column += Vector512<byte>.Count)
+                    {
+                        Vector512<byte> topLeft = Vector512.LoadUnsafe(ref sourceRow, (nuint)column);
+                        Vector512<byte> topRight = Vector512.LoadUnsafe(ref sourceRow, (nuint)(column + 1));
+
+                        TOperator.Filter(topLeft, topRight).StoreUnsafe(ref destinationRow, (nuint)column);
+                    }
+                }
+
+                processedColumns = vectorizedColumns;
+            }
+        }
+
+        if (Vector256.IsHardwareAccelerated)
+        {
+            int remainingColumns = width - processedColumns;
+            int vectorizedColumns = remainingColumns - (remainingColumns % Vector256<byte>.Count);
+            int endColumn = processedColumns + vectorizedColumns;
+
+            for (int row = 0; row < height; row++)
+            {
+                ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride);
+                ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
+
+                for (int column = processedColumns; column < endColumn; column += Vector256<byte>.Count)
+                {
+                    Vector256<byte> topLeft = Vector256.LoadUnsafe(ref sourceRow, (nuint)column);
+                    Vector256<byte> topRight = Vector256.LoadUnsafe(ref sourceRow, (nuint)(column + 1));
+
+                    TOperator.Filter(topLeft, topRight).StoreUnsafe(ref destinationRow, (nuint)column);
+                }
+            }
+
+            processedColumns = endColumn;
+        }
+
+        if (Vector128.IsHardwareAccelerated)
+        {
+            int remainingColumns = width - processedColumns;
+            int vectorizedColumns = remainingColumns - (remainingColumns % Vector128<byte>.Count);
+            int endColumn = processedColumns + vectorizedColumns;
+
+            for (int row = 0; row < height; row++)
+            {
+                ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride);
+                ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
+
+                for (int column = processedColumns; column < endColumn; column += Vector128<byte>.Count)
+                {
+                    Vector128<byte> topLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)column);
+                    Vector128<byte> topRight = Vector128.LoadUnsafe(ref sourceRow, (nuint)(column + 1));
+
+                    TOperator.Filter(topLeft, topRight).StoreUnsafe(ref destinationRow, (nuint)column);
+                }
+            }
+
+            processedColumns = endColumn;
+        }
+
+        // FeatureTestRunner can disable every intrinsic tier. Keeping the scalar continuation in the same traversal
+        // proves the fallback without changing source addressing or the normative rounding performed by the operator.
+        for (int row = 0; row < height; row++)
+        {
+            ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride);
+            ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
+
+            for (int column = processedColumns; column < width; column++)
+            {
+                byte topLeft = Unsafe.Add(ref sourceRow, column);
+                byte topRight = Unsafe.Add(ref sourceRow, column + 1);
+
+                Unsafe.Add(ref destinationRow, column) = TOperator.Filter(topLeft, topRight);
+            }
+        }
+    }
+
+    /// <summary>
+    /// Applies one closed interpolation operator to a high-bit-depth source block.
+    /// </summary>
+    /// <typeparam name="TOperator">The source-phase-specific interpolation arithmetic.</typeparam>
+    private static void Predict<TOperator>(
+        ReadOnlySpan<short> source,
+        int sourceStride,
+        Span<short> destination,
+        int destinationStride,
+        int width,
+        int height)
+        where TOperator : struct, IAv1IntraBlockCopyHorizontalOperator
+    {
+        ref short sourceBase = ref MemoryMarshal.GetReference(source);
+        ref short destinationBase = ref MemoryMarshal.GetReference(destination);
+
+        if (Vector128.IsHardwareAccelerated && width == 4)
+        {
+            // Four high-bit-depth samples occupy the lower half of a Vector128. The frame allocation's prediction
+            // border makes the full source load readable; storing only the lower four lanes avoids relying on writable
+            // samples beyond the transform boundary.
+            for (int row = 0; row < height; row++)
+            {
+                ref short sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride);
+                ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
+                Vector128<short> topLeft = Vector128.LoadUnsafe(ref sourceRow);
+                Vector128<short> topRight = Vector128.LoadUnsafe(ref sourceRow, 1);
+
+                TOperator.Filter(topLeft, topRight).GetLower().StoreUnsafe(ref destinationRow);
+            }
+
+            return;
+        }
+
+        int processedColumns = 0;
+
+        // High-bit-depth lanes hold half as many samples, but retain the same descending-width traversal and one scalar
+        // continuation as the byte path.
+        if (Vector512.IsHardwareAccelerated)
+        {
+            int vectorizedColumns = width - (width % Vector512<short>.Count);
+            if (vectorizedColumns > 0)
+            {
+                for (int row = 0; row < height; row++)
+                {
+                    ref short sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride);
+                    ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
+
+                    for (int column = 0; column < vectorizedColumns; column += Vector512<short>.Count)
+                    {
+                        Vector512<short> topLeft = Vector512.LoadUnsafe(ref sourceRow, (nuint)column);
+                        Vector512<short> topRight = Vector512.LoadUnsafe(ref sourceRow, (nuint)(column + 1));
+
+                        TOperator.Filter(topLeft, topRight).StoreUnsafe(ref destinationRow, (nuint)column);
+                    }
+                }
+
+                processedColumns = vectorizedColumns;
+            }
+        }
+
+        if (Vector256.IsHardwareAccelerated)
+        {
+            int remainingColumns = width - processedColumns;
+            int vectorizedColumns = remainingColumns - (remainingColumns % Vector256<short>.Count);
+            int endColumn = processedColumns + vectorizedColumns;
+
+            for (int row = 0; row < height; row++)
+            {
+                ref short sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride);
+                ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
+
+                for (int column = processedColumns; column < endColumn; column += Vector256<short>.Count)
+                {
+                    Vector256<short> topLeft = Vector256.LoadUnsafe(ref sourceRow, (nuint)column);
+                    Vector256<short> topRight = Vector256.LoadUnsafe(ref sourceRow, (nuint)(column + 1));
+
+                    TOperator.Filter(topLeft, topRight).StoreUnsafe(ref destinationRow, (nuint)column);
+                }
+            }
+
+            processedColumns = endColumn;
+        }
+
+        if (Vector128.IsHardwareAccelerated)
+        {
+            int remainingColumns = width - processedColumns;
+            int vectorizedColumns = remainingColumns - (remainingColumns % Vector128<short>.Count);
+            int endColumn = processedColumns + vectorizedColumns;
+
+            for (int row = 0; row < height; row++)
+            {
+                ref short sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride);
+                ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
+
+                for (int column = processedColumns; column < endColumn; column += Vector128<short>.Count)
+                {
+                    Vector128<short> topLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)column);
+                    Vector128<short> topRight = Vector128.LoadUnsafe(ref sourceRow, (nuint)(column + 1));
+
+                    TOperator.Filter(topLeft, topRight).StoreUnsafe(ref destinationRow, (nuint)column);
+                }
+            }
+
+            processedColumns = endColumn;
+        }
+
+        for (int row = 0; row < height; row++)
+        {
+            ref short sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride);
+            ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
+
+            for (int column = processedColumns; column < width; column++)
+            {
+                short topLeft = Unsafe.Add(ref sourceRow, column);
+                short topRight = Unsafe.Add(ref sourceRow, column + 1);
+
+                Unsafe.Add(ref destinationRow, column) = TOperator.Filter(topLeft, topRight);
+            }
+        }
+    }
+
+    /// <summary>
+    /// Applies one closed interpolation operator to an 8-bit source block without explicit hardware intrinsics.
+    /// </summary>
+    /// <typeparam name="TOperator">The source-phase-specific interpolation arithmetic.</typeparam>
+    private static void PredictScalar<TOperator>(
+        ReadOnlySpan<byte> source,
+        int sourceStride,
+        Span<byte> destination,
+        int destinationStride,
+        int width,
+        int height)
+        where TOperator : struct, IAv1IntraBlockCopyHorizontalOperator
+    {
+        for (int row = 0; row < height; row++)
+        {
+            int sourceRow = row * sourceStride;
+            int destinationRow = row * destinationStride;
+
+            for (int column = 0; column < width; column++)
+            {
+                byte topLeft = source[sourceRow + column];
+                byte topRight = source[sourceRow + column + 1];
+                destination[destinationRow + column] = TOperator.Filter(topLeft, topRight);
+            }
+        }
+    }
+
+    /// <summary>
+    /// Applies one closed interpolation operator to a high-bit-depth source block without explicit hardware intrinsics.
+    /// </summary>
+    /// <typeparam name="TOperator">The source-phase-specific interpolation arithmetic.</typeparam>
+    private static void PredictScalar<TOperator>(
+        ReadOnlySpan<short> source,
+        int sourceStride,
+        Span<short> destination,
+        int destinationStride,
+        int width,
+        int height)
+        where TOperator : struct, IAv1IntraBlockCopyHorizontalOperator
+    {
+        for (int row = 0; row < height; row++)
+        {
+            int sourceRow = row * sourceStride;
+            int destinationRow = row * destinationStride;
+
+            for (int column = 0; column < width; column++)
+            {
+                short topLeft = source[sourceRow + column];
+                short topRight = source[sourceRow + column + 1];
+                destination[destinationRow + column] = TOperator.Filter(topLeft, topRight);
+            }
+        }
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyPredictor.Arithmetic.cs

@@ -1,80 +0,0 @@
-// Copyright (c) Six Labors.
-// Licensed under the Six Labors Split License.
-
-using System.Runtime.Intrinsics;
-
-namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.IntraBlockCopy;
-
-/// <content>
-/// Provides the overflow-free rounded-average arithmetic shared by the interpolation operators.
-/// </content>
-internal static partial class Av1IntraBlockCopyPredictor
-{
-    /// <summary>
-    /// Computes the AV1 rounded average of two unsigned 8-bit vectors without widening their lanes.
-    /// </summary>
-    /// <param name="left">The first source vector.</param>
-    /// <param name="right">The second source vector.</param>
-    /// <returns>The lane-wise rounded averages.</returns>
-    private static Vector128<byte> AverageRounded(Vector128<byte> left, Vector128<byte> right)
-        => (left | right) - ((left ^ right) >> 1);
-
-    /// <summary>
-    /// Computes the AV1 rounded average of two unsigned 8-bit vectors without widening their lanes.
-    /// </summary>
-    /// <param name="left">The first source vector.</param>
-    /// <param name="right">The second source vector.</param>
-    /// <returns>The lane-wise rounded averages.</returns>
-    private static Vector256<byte> AverageRounded(Vector256<byte> left, Vector256<byte> right)
-        => (left | right) - ((left ^ right) >> 1);
-
-    /// <summary>
-    /// Computes the AV1 rounded average of two unsigned 8-bit vectors without widening their lanes.
-    /// </summary>
-    /// <param name="left">The first source vector.</param>
-    /// <param name="right">The second source vector.</param>
-    /// <returns>The lane-wise rounded averages.</returns>
-    private static Vector512<byte> AverageRounded(Vector512<byte> left, Vector512<byte> right)
-        => (left | right) - ((left ^ right) >> 1);
-
-    /// <summary>
-    /// Computes the AV1 rounded average of two nonnegative high-bit-depth vectors without widening their lanes.
-    /// </summary>
-    /// <param name="left">The first source vector.</param>
-    /// <param name="right">The second source vector.</param>
-    /// <returns>The lane-wise rounded averages.</returns>
-    private static Vector128<short> AverageRounded(Vector128<short> left, Vector128<short> right)
-    {
-        Vector128<ushort> leftUnsigned = left.AsUInt16();
-        Vector128<ushort> rightUnsigned = right.AsUInt16();
-
-        // (a | b) - ((a ^ b) >> 1) is ceil((a + b) / 2) without an overflowing lane-wise addition.
-        return ((leftUnsigned | rightUnsigned) - ((leftUnsigned ^ rightUnsigned) >> 1)).AsInt16();
-    }
-
-    /// <summary>
-    /// Computes the AV1 rounded average of two nonnegative high-bit-depth vectors without widening their lanes.
-    /// </summary>
-    /// <param name="left">The first source vector.</param>
-    /// <param name="right">The second source vector.</param>
-    /// <returns>The lane-wise rounded averages.</returns>
-    private static Vector256<short> AverageRounded(Vector256<short> left, Vector256<short> right)
-    {
-        Vector256<ushort> leftUnsigned = left.AsUInt16();
-        Vector256<ushort> rightUnsigned = right.AsUInt16();
-        return ((leftUnsigned | rightUnsigned) - ((leftUnsigned ^ rightUnsigned) >> 1)).AsInt16();
-    }
-
-    /// <summary>
-    /// Computes the AV1 rounded average of two nonnegative high-bit-depth vectors without widening their lanes.
-    /// </summary>
-    /// <param name="left">The first source vector.</param>
-    /// <param name="right">The second source vector.</param>
-    /// <returns>The lane-wise rounded averages.</returns>
-    private static Vector512<short> AverageRounded(Vector512<short> left, Vector512<short> right)
-    {
-        Vector512<ushort> leftUnsigned = left.AsUInt16();
-        Vector512<ushort> rightUnsigned = right.AsUInt16();
-        return ((leftUnsigned | rightUnsigned) - ((leftUnsigned ^ rightUnsigned) >> 1)).AsInt16();
-    }
-}

src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyPredictor.HorizontalOperator.cs

@@ -1,78 +0,0 @@
-// Copyright (c) Six Labors.
-// Licensed under the Six Labors Split License.
-
-using System.Runtime.Intrinsics;
-
-namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.IntraBlockCopy;
-
-/// <content>
-/// Defines the closed interpolation operators used by intra-block-copy prediction.
-/// </content>
-internal static partial class Av1IntraBlockCopyPredictor
-{
-    /// <summary>
-    /// Averages horizontally adjacent source samples for a half-sample horizontal phase.
-    /// </summary>
-    private readonly struct HorizontalOperator : IAv1IntraBlockCopyOperator
-    {
-        /// <inheritdoc/>
-        public static bool UsesRight => true;
-
-        /// <inheritdoc/>
-        public static bool UsesBottom => false;
-
-        /// <inheritdoc/>
-        public static byte Filter(byte topLeft, byte topRight, byte bottomLeft, byte bottomRight) => (byte)((topLeft + topRight + 1) >> 1);
-
-        /// <inheritdoc/>
-        public static Vector128<byte> Filter(
-            Vector128<byte> topLeft,
-            Vector128<byte> topRight,
-            Vector128<byte> bottomLeft,
-            Vector128<byte> bottomRight)
-            => AverageRounded(topLeft, topRight);
-
-        /// <inheritdoc/>
-        public static Vector256<byte> Filter(
-            Vector256<byte> topLeft,
-            Vector256<byte> topRight,
-            Vector256<byte> bottomLeft,
-            Vector256<byte> bottomRight)
-            => AverageRounded(topLeft, topRight);
-
-        /// <inheritdoc/>
-        public static Vector512<byte> Filter(
-            Vector512<byte> topLeft,
-            Vector512<byte> topRight,
-            Vector512<byte> bottomLeft,
-            Vector512<byte> bottomRight)
-            => AverageRounded(topLeft, topRight);
-
-        /// <inheritdoc/>
-        public static short Filter(short topLeft, short topRight, short bottomLeft, short bottomRight) => (short)((topLeft + topRight + 1) >> 1);
-
-        /// <inheritdoc/>
-        public static Vector128<short> Filter(
-            Vector128<short> topLeft,
-            Vector128<short> topRight,
-            Vector128<short> bottomLeft,
-            Vector128<short> bottomRight)
-            => AverageRounded(topLeft, topRight);
-
-        /// <inheritdoc/>
-        public static Vector256<short> Filter(
-            Vector256<short> topLeft,
-            Vector256<short> topRight,
-            Vector256<short> bottomLeft,
-            Vector256<short> bottomRight)
-            => AverageRounded(topLeft, topRight);
-
-        /// <inheritdoc/>
-        public static Vector512<short> Filter(
-            Vector512<short> topLeft,
-            Vector512<short> topRight,
-            Vector512<short> bottomLeft,
-            Vector512<short> bottomRight)
-            => AverageRounded(topLeft, topRight);
-    }
-}

src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyPredictor.VerticalOperator.cs

@@ -1,75 +0,0 @@
-// Copyright (c) Six Labors.
-// Licensed under the Six Labors Split License.
-
-using System.Runtime.Intrinsics;
-
-namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.IntraBlockCopy;
-
-internal static partial class Av1IntraBlockCopyPredictor
-{
-    /// <summary>
-    /// Averages vertically adjacent source samples for a half-sample vertical phase.
-    /// </summary>
-    private readonly struct VerticalOperator : IAv1IntraBlockCopyOperator
-    {
-        /// <inheritdoc/>
-        public static bool UsesRight => false;
-
-        /// <inheritdoc/>
-        public static bool UsesBottom => true;
-
-        /// <inheritdoc/>
-        public static byte Filter(byte topLeft, byte topRight, byte bottomLeft, byte bottomRight) => (byte)((topLeft + bottomLeft + 1) >> 1);
-
-        /// <inheritdoc/>
-        public static Vector128<byte> Filter(
-            Vector128<byte> topLeft,
-            Vector128<byte> topRight,
-            Vector128<byte> bottomLeft,
-            Vector128<byte> bottomRight)
-            => AverageRounded(topLeft, bottomLeft);
-
-        /// <inheritdoc/>
-        public static Vector256<byte> Filter(
-            Vector256<byte> topLeft,
-            Vector256<byte> topRight,
-            Vector256<byte> bottomLeft,
-            Vector256<byte> bottomRight)
-            => AverageRounded(topLeft, bottomLeft);
-
-        /// <inheritdoc/>
-        public static Vector512<byte> Filter(
-            Vector512<byte> topLeft,
-            Vector512<byte> topRight,
-            Vector512<byte> bottomLeft,
-            Vector512<byte> bottomRight)
-            => AverageRounded(topLeft, bottomLeft);
-
-        /// <inheritdoc/>
-        public static short Filter(short topLeft, short topRight, short bottomLeft, short bottomRight) => (short)((topLeft + bottomLeft + 1) >> 1);
-
-        /// <inheritdoc/>
-        public static Vector128<short> Filter(
-            Vector128<short> topLeft,
-            Vector128<short> topRight,
-            Vector128<short> bottomLeft,
-            Vector128<short> bottomRight)
-            => AverageRounded(topLeft, bottomLeft);
-
-        /// <inheritdoc/>
-        public static Vector256<short> Filter(
-            Vector256<short> topLeft,
-            Vector256<short> topRight,
-            Vector256<short> bottomLeft,
-            Vector256<short> bottomRight)
-            => AverageRounded(topLeft, bottomLeft);
-
-        /// <inheritdoc/>
-        public static Vector512<short> Filter(
-            Vector512<short> topLeft,
-            Vector512<short> topRight,
-            Vector512<short> bottomLeft,
-            Vector512<short> bottomRight)
-            => AverageRounded(topLeft, bottomLeft);
-    }
-}

src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyPredictor.cs

@@ -42,15 +42,15 @@ internal static partial class Av1IntraBlockCopyPredictor
         }
         else if (halfX && halfY)
         {
-            Predict<BilinearOperator>(source, sourceStride, destination, destinationStride, width, height);
+            Av1IntraBlockCopyBilinearPredictor.Predict(source, sourceStride, destination, destinationStride, width, height);
         }
         else if (halfX)
         {
-            Predict<HorizontalOperator>(source, sourceStride, destination, destinationStride, width, height);
+            Av1IntraBlockCopyHorizontalPredictor.Predict(source, sourceStride, destination, destinationStride, width, height);
         }
         else
         {
-            Predict<VerticalOperator>(source, sourceStride, destination, destinationStride, width, height);
+            Av1IntraBlockCopyVerticalPredictor.Predict(source, sourceStride, destination, destinationStride, width, height);
         }
     }
 
@@ -81,15 +81,15 @@ internal static partial class Av1IntraBlockCopyPredictor
         }
         else if (halfX && halfY)
         {
-            Predict<BilinearOperator>(source, sourceStride, destination, destinationStride, width, height);
+            Av1IntraBlockCopyBilinearPredictor.Predict(source, sourceStride, destination, destinationStride, width, height);
         }
         else if (halfX)
         {
-            Predict<HorizontalOperator>(source, sourceStride, destination, destinationStride, width, height);
+            Av1IntraBlockCopyHorizontalPredictor.Predict(source, sourceStride, destination, destinationStride, width, height);
         }
         else
         {
-            Predict<VerticalOperator>(source, sourceStride, destination, destinationStride, width, height);
+            Av1IntraBlockCopyVerticalPredictor.Predict(source, sourceStride, destination, destinationStride, width, height);
         }
     }
 
@@ -120,15 +120,15 @@ internal static partial class Av1IntraBlockCopyPredictor
         }
         else if (halfX && halfY)
         {
-            PredictScalar<BilinearOperator>(source, sourceStride, destination, destinationStride, width, height);
+            Av1IntraBlockCopyBilinearPredictor.PredictScalar(source, sourceStride, destination, destinationStride, width, height);
         }
         else if (halfX)
         {
-            PredictScalar<HorizontalOperator>(source, sourceStride, destination, destinationStride, width, height);
+            Av1IntraBlockCopyHorizontalPredictor.PredictScalar(source, sourceStride, destination, destinationStride, width, height);
         }
         else
         {
-            PredictScalar<VerticalOperator>(source, sourceStride, destination, destinationStride, width, height);
+            Av1IntraBlockCopyVerticalPredictor.PredictScalar(source, sourceStride, destination, destinationStride, width, height);
         }
     }
 
@@ -159,15 +159,15 @@ internal static partial class Av1IntraBlockCopyPredictor
         }
         else if (halfX && halfY)
         {
-            PredictScalar<BilinearOperator>(source, sourceStride, destination, destinationStride, width, height);
+            Av1IntraBlockCopyBilinearPredictor.PredictScalar(source, sourceStride, destination, destinationStride, width, height);
         }
         else if (halfX)
         {
-            PredictScalar<HorizontalOperator>(source, sourceStride, destination, destinationStride, width, height);
+            Av1IntraBlockCopyHorizontalPredictor.PredictScalar(source, sourceStride, destination, destinationStride, width, height);
         }
         else
         {
-            PredictScalar<VerticalOperator>(source, sourceStride, destination, destinationStride, width, height);
+            Av1IntraBlockCopyVerticalPredictor.PredictScalar(source, sourceStride, destination, destinationStride, width, height);
         }
     }
 

src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyVerticalPredictor.Operator.cs

@@ -0,0 +1,180 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.Intrinsics;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.IntraBlockCopy;
+
+/// <content>
+/// Defines the closed vertical intra-block-copy interpolation operator.
+/// </content>
+internal static partial class Av1IntraBlockCopyVerticalPredictor
+{
+    /// <summary>
+    /// Defines vertical intra-block-copy filtering for scalar and SIMD lane groups.
+    /// </summary>
+    private interface IAv1IntraBlockCopyVerticalOperator
+    {
+        /// <summary>
+        /// Filters one 8-bit sample.
+        /// </summary>
+        /// <param name="top">The integer-position source sample.</param>
+        /// <param name="bottom">The source sample one row below.</param>
+        /// <returns>The filtered 8-bit sample.</returns>
+        public static abstract byte Filter(byte top, byte bottom);
+
+        /// <summary>
+        /// Filters sixteen 8-bit samples in parallel.
+        /// </summary>
+        /// <param name="top">The integer-position source samples.</param>
+        /// <param name="bottom">The source samples one row below.</param>
+        /// <returns>The filtered 8-bit samples.</returns>
+        public static abstract Vector128<byte> Filter(
+            Vector128<byte> top,
+            Vector128<byte> bottom);
+
+        /// <summary>
+        /// Filters thirty-two 8-bit samples in parallel.
+        /// </summary>
+        /// <param name="top">The integer-position source samples.</param>
+        /// <param name="bottom">The source samples one row below.</param>
+        /// <returns>The filtered 8-bit samples.</returns>
+        public static abstract Vector256<byte> Filter(
+            Vector256<byte> top,
+            Vector256<byte> bottom);
+
+        /// <summary>
+        /// Filters sixty-four 8-bit samples in parallel.
+        /// </summary>
+        /// <param name="top">The integer-position source samples.</param>
+        /// <param name="bottom">The source samples one row below.</param>
+        /// <returns>The filtered 8-bit samples.</returns>
+        public static abstract Vector512<byte> Filter(
+            Vector512<byte> top,
+            Vector512<byte> bottom);
+
+        /// <summary>
+        /// Filters one high-bit-depth sample.
+        /// </summary>
+        /// <param name="top">The integer-position source sample.</param>
+        /// <param name="bottom">The source sample one row below.</param>
+        /// <returns>The filtered high-bit-depth sample.</returns>
+        public static abstract short Filter(short top, short bottom);
+
+        /// <summary>
+        /// Filters eight high-bit-depth samples in parallel.
+        /// </summary>
+        /// <param name="top">The integer-position source samples.</param>
+        /// <param name="bottom">The source samples one row below.</param>
+        /// <returns>The filtered high-bit-depth samples.</returns>
+        public static abstract Vector128<short> Filter(
+            Vector128<short> top,
+            Vector128<short> bottom);
+
+        /// <summary>
+        /// Filters sixteen high-bit-depth samples in parallel.
+        /// </summary>
+        /// <param name="top">The integer-position source samples.</param>
+        /// <param name="bottom">The source samples one row below.</param>
+        /// <returns>The filtered high-bit-depth samples.</returns>
+        public static abstract Vector256<short> Filter(
+            Vector256<short> top,
+            Vector256<short> bottom);
+
+        /// <summary>
+        /// Filters thirty-two high-bit-depth samples in parallel.
+        /// </summary>
+        /// <param name="top">The integer-position source samples.</param>
+        /// <param name="bottom">The source samples one row below.</param>
+        /// <returns>The filtered high-bit-depth samples.</returns>
+        public static abstract Vector512<short> Filter(
+            Vector512<short> top,
+            Vector512<short> bottom);
+    }
+
+    /// <summary>
+    /// Averages vertically adjacent source samples for a half-sample vertical phase.
+    /// </summary>
+    private readonly struct IntraBlockCopyVerticalOperator : IAv1IntraBlockCopyVerticalOperator
+    {
+        /// <inheritdoc/>
+        public static byte Filter(byte top, byte bottom) => (byte)((top + bottom + 1) >> 1);
+
+        /// <inheritdoc/>
+        public static Vector128<byte> Filter(Vector128<byte> top, Vector128<byte> bottom)
+            => AverageRounded(top, bottom);
+
+        /// <inheritdoc/>
+        public static Vector256<byte> Filter(Vector256<byte> top, Vector256<byte> bottom)
+            => AverageRounded(top, bottom);
+
+        /// <inheritdoc/>
+        public static Vector512<byte> Filter(Vector512<byte> top, Vector512<byte> bottom)
+            => AverageRounded(top, bottom);
+
+        /// <inheritdoc/>
+        public static short Filter(short top, short bottom) => (short)((top + bottom + 1) >> 1);
+
+        /// <inheritdoc/>
+        public static Vector128<short> Filter(Vector128<short> top, Vector128<short> bottom)
+            => AverageRounded(top, bottom);
+
+        /// <inheritdoc/>
+        public static Vector256<short> Filter(Vector256<short> top, Vector256<short> bottom)
+            => AverageRounded(top, bottom);
+
+        /// <inheritdoc/>
+        public static Vector512<short> Filter(Vector512<short> top, Vector512<short> bottom)
+            => AverageRounded(top, bottom);
+
+        /// <summary>
+        /// Computes a rounded average without overflowing unsigned byte lanes.
+        /// </summary>
+        private static Vector128<byte> AverageRounded(Vector128<byte> left, Vector128<byte> right)
+            => (left | right) - ((left ^ right) >> 1);
+
+        /// <summary>
+        /// Computes a rounded average without overflowing unsigned byte lanes.
+        /// </summary>
+        private static Vector256<byte> AverageRounded(Vector256<byte> left, Vector256<byte> right)
+            => (left | right) - ((left ^ right) >> 1);
+
+        /// <summary>
+        /// Computes a rounded average without overflowing unsigned byte lanes.
+        /// </summary>
+        private static Vector512<byte> AverageRounded(Vector512<byte> left, Vector512<byte> right)
+            => (left | right) - ((left ^ right) >> 1);
+
+        /// <summary>
+        /// Computes a rounded average without overflowing nonnegative high-bit-depth lanes.
+        /// </summary>
+        private static Vector128<short> AverageRounded(Vector128<short> left, Vector128<short> right)
+        {
+            Vector128<ushort> leftUnsigned = left.AsUInt16();
+            Vector128<ushort> rightUnsigned = right.AsUInt16();
+
+            // This identity computes ceil((a + b) / 2) without an overflowing lane-wise addition.
+            return ((leftUnsigned | rightUnsigned) - ((leftUnsigned ^ rightUnsigned) >> 1)).AsInt16();
+        }
+
+        /// <summary>
+        /// Computes a rounded average without overflowing nonnegative high-bit-depth lanes.
+        /// </summary>
+        private static Vector256<short> AverageRounded(Vector256<short> left, Vector256<short> right)
+        {
+            Vector256<ushort> leftUnsigned = left.AsUInt16();
+            Vector256<ushort> rightUnsigned = right.AsUInt16();
+            return ((leftUnsigned | rightUnsigned) - ((leftUnsigned ^ rightUnsigned) >> 1)).AsInt16();
+        }
+
+        /// <summary>
+        /// Computes a rounded average without overflowing nonnegative high-bit-depth lanes.
+        /// </summary>
+        private static Vector512<short> AverageRounded(Vector512<short> left, Vector512<short> right)
+        {
+            Vector512<ushort> leftUnsigned = left.AsUInt16();
+            Vector512<ushort> rightUnsigned = right.AsUInt16();
+            return ((leftUnsigned | rightUnsigned) - ((leftUnsigned ^ rightUnsigned) >> 1)).AsInt16();
+        }
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyVerticalPredictor.cs

@@ -0,0 +1,370 @@
+// Copyright (c) Six Labors.
+// Licensed under the Six Labors Split License.
+
+using System.Runtime.CompilerServices;
+using System.Runtime.InteropServices;
+using System.Runtime.Intrinsics;
+
+namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.IntraBlockCopy;
+
+/// <content>
+/// Provides the family-owned scalar and width-progressive SIMD traversal for vertical intra-block-copy prediction.
+/// </content>
+internal static partial class Av1IntraBlockCopyVerticalPredictor
+{
+    /// <summary>
+    /// Reconstructs an 8-bit filtered intra-block-copy prediction.
+    /// </summary>
+    public static void Predict(
+        ReadOnlySpan<byte> source,
+        int sourceStride,
+        Span<byte> destination,
+        int destinationStride,
+        int width,
+        int height)
+        => Predict<IntraBlockCopyVerticalOperator>(source, sourceStride, destination, destinationStride, width, height);
+
+    /// <summary>
+    /// Reconstructs a high-bit-depth filtered intra-block-copy prediction.
+    /// </summary>
+    public static void Predict(
+        ReadOnlySpan<short> source,
+        int sourceStride,
+        Span<short> destination,
+        int destinationStride,
+        int width,
+        int height)
+        => Predict<IntraBlockCopyVerticalOperator>(source, sourceStride, destination, destinationStride, width, height);
+
+    /// <summary>
+    /// Reconstructs an 8-bit filtered intra-block-copy prediction without explicit hardware intrinsics.
+    /// </summary>
+    public static void PredictScalar(
+        ReadOnlySpan<byte> source,
+        int sourceStride,
+        Span<byte> destination,
+        int destinationStride,
+        int width,
+        int height)
+        => PredictScalar<IntraBlockCopyVerticalOperator>(source, sourceStride, destination, destinationStride, width, height);
+
+    /// <summary>
+    /// Reconstructs a high-bit-depth filtered intra-block-copy prediction without explicit hardware intrinsics.
+    /// </summary>
+    public static void PredictScalar(
+        ReadOnlySpan<short> source,
+        int sourceStride,
+        Span<short> destination,
+        int destinationStride,
+        int width,
+        int height)
+        => PredictScalar<IntraBlockCopyVerticalOperator>(source, sourceStride, destination, destinationStride, width, height);
+
+    /// <summary>
+    /// Applies one closed interpolation operator to an 8-bit source block.
+    /// </summary>
+    /// <typeparam name="TOperator">The source-phase-specific interpolation arithmetic.</typeparam>
+    private static void Predict<TOperator>(
+        ReadOnlySpan<byte> source,
+        int sourceStride,
+        Span<byte> destination,
+        int destinationStride,
+        int width,
+        int height)
+        where TOperator : struct, IAv1IntraBlockCopyVerticalOperator
+    {
+        ref byte sourceBase = ref MemoryMarshal.GetReference(source);
+        ref byte destinationBase = ref MemoryMarshal.GetReference(destination);
+
+        if (Vector128.IsHardwareAccelerated && width is 4 or 8)
+        {
+            // AV1 permits 4- and 8-sample transform widths, both smaller than a byte Vector128. The frame allocation's
+            // 72-sample prediction border makes each full source load readable; exact-width stores avoid touching
+            // destination padding.
+            for (int row = 0; row < height; row++)
+            {
+                ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride);
+                ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
+                Vector128<byte> topLeft = Vector128.LoadUnsafe(ref sourceRow);
+                Vector128<byte> bottomLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)sourceStride);
+
+                Vector128<byte> prediction = TOperator.Filter(topLeft, bottomLeft);
+                if (width == 8)
+                {
+                    prediction.GetLower().StoreUnsafe(ref destinationRow);
+                }
+                else
+                {
+                    Unsafe.As<byte, uint>(ref destinationRow) = prediction.AsUInt32().GetElement(0);
+                }
+            }
+
+            return;
+        }
+
+        int processedColumns = 0;
+
+        // AV1 transform widths are powers of two. The widest supported tier normally consumes the complete row; the
+        // cumulative narrower tiers preserve the same contract for future legal widths without over-reading a tail.
+        if (Vector512.IsHardwareAccelerated)
+        {
+            int vectorizedColumns = width - (width % Vector512<byte>.Count);
+            if (vectorizedColumns > 0)
+            {
+                for (int row = 0; row < height; row++)
+                {
+                    ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride);
+                    ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
+
+                    for (int column = 0; column < vectorizedColumns; column += Vector512<byte>.Count)
+                    {
+                        Vector512<byte> topLeft = Vector512.LoadUnsafe(ref sourceRow, (nuint)column);
+                        Vector512<byte> bottomLeft = Vector512.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column));
+
+                        TOperator.Filter(topLeft, bottomLeft).StoreUnsafe(ref destinationRow, (nuint)column);
+                    }
+                }
+
+                processedColumns = vectorizedColumns;
+            }
+        }
+
+        if (Vector256.IsHardwareAccelerated)
+        {
+            int remainingColumns = width - processedColumns;
+            int vectorizedColumns = remainingColumns - (remainingColumns % Vector256<byte>.Count);
+            int endColumn = processedColumns + vectorizedColumns;
+
+            for (int row = 0; row < height; row++)
+            {
+                ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride);
+                ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
+
+                for (int column = processedColumns; column < endColumn; column += Vector256<byte>.Count)
+                {
+                    Vector256<byte> topLeft = Vector256.LoadUnsafe(ref sourceRow, (nuint)column);
+                    Vector256<byte> bottomLeft = Vector256.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column));
+
+                    TOperator.Filter(topLeft, bottomLeft).StoreUnsafe(ref destinationRow, (nuint)column);
+                }
+            }
+
+            processedColumns = endColumn;
+        }
+
+        if (Vector128.IsHardwareAccelerated)
+        {
+            int remainingColumns = width - processedColumns;
+            int vectorizedColumns = remainingColumns - (remainingColumns % Vector128<byte>.Count);
+            int endColumn = processedColumns + vectorizedColumns;
+
+            for (int row = 0; row < height; row++)
+            {
+                ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride);
+                ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
+
+                for (int column = processedColumns; column < endColumn; column += Vector128<byte>.Count)
+                {
+                    Vector128<byte> topLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)column);
+                    Vector128<byte> bottomLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column));
+
+                    TOperator.Filter(topLeft, bottomLeft).StoreUnsafe(ref destinationRow, (nuint)column);
+                }
+            }
+
+            processedColumns = endColumn;
+        }
+
+        // FeatureTestRunner can disable every intrinsic tier. Keeping the scalar continuation in the same traversal
+        // proves the fallback without changing source addressing or the normative rounding performed by the operator.
+        for (int row = 0; row < height; row++)
+        {
+            ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride);
+            ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
+
+            for (int column = processedColumns; column < width; column++)
+            {
+                byte topLeft = Unsafe.Add(ref sourceRow, column);
+                byte bottomLeft = Unsafe.Add(ref sourceRow, sourceStride + column);
+
+                Unsafe.Add(ref destinationRow, column) = TOperator.Filter(topLeft, bottomLeft);
+            }
+        }
+    }
+
+    /// <summary>
+    /// Applies one closed interpolation operator to a high-bit-depth source block.
+    /// </summary>
+    /// <typeparam name="TOperator">The source-phase-specific interpolation arithmetic.</typeparam>
+    private static void Predict<TOperator>(
+        ReadOnlySpan<short> source,
+        int sourceStride,
+        Span<short> destination,
+        int destinationStride,
+        int width,
+        int height)
+        where TOperator : struct, IAv1IntraBlockCopyVerticalOperator
+    {
+        ref short sourceBase = ref MemoryMarshal.GetReference(source);
+        ref short destinationBase = ref MemoryMarshal.GetReference(destination);
+
+        if (Vector128.IsHardwareAccelerated && width == 4)
+        {
+            // Four high-bit-depth samples occupy the lower half of a Vector128. The frame allocation's prediction
+            // border makes the full source load readable; storing only the lower four lanes avoids relying on writable
+            // samples beyond the transform boundary.
+            for (int row = 0; row < height; row++)
+            {
+                ref short sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride);
+                ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
+                Vector128<short> topLeft = Vector128.LoadUnsafe(ref sourceRow);
+                Vector128<short> bottomLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)sourceStride);
+
+                TOperator.Filter(topLeft, bottomLeft).GetLower().StoreUnsafe(ref destinationRow);
+            }
+
+            return;
+        }
+
+        int processedColumns = 0;
+
+        // High-bit-depth lanes hold half as many samples, but retain the same descending-width traversal and one scalar
+        // continuation as the byte path.
+        if (Vector512.IsHardwareAccelerated)
+        {
+            int vectorizedColumns = width - (width % Vector512<short>.Count);
+            if (vectorizedColumns > 0)
+            {
+                for (int row = 0; row < height; row++)
+                {
+                    ref short sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride);
+                    ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
+
+                    for (int column = 0; column < vectorizedColumns; column += Vector512<short>.Count)
+                    {
+                        Vector512<short> topLeft = Vector512.LoadUnsafe(ref sourceRow, (nuint)column);
+                        Vector512<short> bottomLeft = Vector512.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column));
+
+                        TOperator.Filter(topLeft, bottomLeft).StoreUnsafe(ref destinationRow, (nuint)column);
+                    }
+                }
+
+                processedColumns = vectorizedColumns;
+            }
+        }
+
+        if (Vector256.IsHardwareAccelerated)
+        {
+            int remainingColumns = width - processedColumns;
+            int vectorizedColumns = remainingColumns - (remainingColumns % Vector256<short>.Count);
+            int endColumn = processedColumns + vectorizedColumns;
+
+            for (int row = 0; row < height; row++)
+            {
+                ref short sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride);
+                ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
+
+                for (int column = processedColumns; column < endColumn; column += Vector256<short>.Count)
+                {
+                    Vector256<short> topLeft = Vector256.LoadUnsafe(ref sourceRow, (nuint)column);
+                    Vector256<short> bottomLeft = Vector256.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column));
+
+                    TOperator.Filter(topLeft, bottomLeft).StoreUnsafe(ref destinationRow, (nuint)column);
+                }
+            }
+
+            processedColumns = endColumn;
+        }
+
+        if (Vector128.IsHardwareAccelerated)
+        {
+            int remainingColumns = width - processedColumns;
+            int vectorizedColumns = remainingColumns - (remainingColumns % Vector128<short>.Count);
+            int endColumn = processedColumns + vectorizedColumns;
+
+            for (int row = 0; row < height; row++)
+            {
+                ref short sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride);
+                ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
+
+                for (int column = processedColumns; column < endColumn; column += Vector128<short>.Count)
+                {
+                    Vector128<short> topLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)column);
+                    Vector128<short> bottomLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column));
+
+                    TOperator.Filter(topLeft, bottomLeft).StoreUnsafe(ref destinationRow, (nuint)column);
+                }
+            }
+
+            processedColumns = endColumn;
+        }
+
+        for (int row = 0; row < height; row++)
+        {
+            ref short sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride);
+            ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride);
+
+            for (int column = processedColumns; column < width; column++)
+            {
+                short topLeft = Unsafe.Add(ref sourceRow, column);
+                short bottomLeft = Unsafe.Add(ref sourceRow, sourceStride + column);
+
+                Unsafe.Add(ref destinationRow, column) = TOperator.Filter(topLeft, bottomLeft);
+            }
+        }
+    }
+
+    /// <summary>
+    /// Applies one closed interpolation operator to an 8-bit source block without explicit hardware intrinsics.
+    /// </summary>
+    /// <typeparam name="TOperator">The source-phase-specific interpolation arithmetic.</typeparam>
+    private static void PredictScalar<TOperator>(
+        ReadOnlySpan<byte> source,
+        int sourceStride,
+        Span<byte> destination,
+        int destinationStride,
+        int width,
+        int height)
+        where TOperator : struct, IAv1IntraBlockCopyVerticalOperator
+    {
+        for (int row = 0; row < height; row++)
+        {
+            int sourceRow = row * sourceStride;
+            int destinationRow = row * destinationStride;
+
+            for (int column = 0; column < width; column++)
+            {
+                byte topLeft = source[sourceRow + column];
+                byte bottomLeft = source[sourceRow + sourceStride + column];
+                destination[destinationRow + column] = TOperator.Filter(topLeft, bottomLeft);
+            }
+        }
+    }
+
+    /// <summary>
+    /// Applies one closed interpolation operator to a high-bit-depth source block without explicit hardware intrinsics.
+    /// </summary>
+    /// <typeparam name="TOperator">The source-phase-specific interpolation arithmetic.</typeparam>
+    private static void PredictScalar<TOperator>(
+        ReadOnlySpan<short> source,
+        int sourceStride,
+        Span<short> destination,
+        int destinationStride,
+        int width,
+        int height)
+        where TOperator : struct, IAv1IntraBlockCopyVerticalOperator
+    {
+        for (int row = 0; row < height; row++)
+        {
+            int sourceRow = row * sourceStride;
+            int destinationRow = row * destinationStride;
+
+            for (int column = 0; column < width; column++)
+            {
+                short topLeft = source[sourceRow + column];
+                short bottomLeft = source[sourceRow + sourceStride + column];
+                destination[destinationRow + column] = TOperator.Filter(topLeft, bottomLeft);
+            }
+        }
+    }
+}

src/ImageSharp/Formats/Heif/Av1/Transform/Av1BlockDecoder.cs

@@ -131,7 +131,7 @@ internal sealed class Av1BlockDecoder : IDisposable
                 Av1PredictionDecoder.ScratchLength,
                 Math.Max(
                     Av1InterPredictor.GetScratchLength(maximumBlockLength, maximumBlockLength),
-                    Av1InterPredictor.GetMaximumScaledScratchLength(maximumBlockLength, maximumBlockLength)));
+                    Av1ScaledInterPredictor.GetMaximumScaledScratchLength(maximumBlockLength, maximumBlockLength)));
 
             int compoundMaskLength = (maximumBlockArea + 1) >> 1;
 
@@ -444,7 +444,7 @@ internal sealed class Av1BlockDecoder : IDisposable
                                     ? highBitDepthBlockReconstructionBuffer[reconstructionStride..]
                                     : secondPredictionStorage);
 
-                            Av1InterPredictor.PredictWarped(
+                            Av1WarpedInterPredictor.PredictWarped(
                                 source,
                                 sourceStride,
                                 sourceOrigin,
@@ -476,7 +476,7 @@ internal sealed class Av1BlockDecoder : IDisposable
                                     ? firstCompoundPrediction
                                     : highBitDepthSecondPrediction;
 
-                                Av1InterPredictor.PredictWarpedCompound(
+                                Av1WarpedInterPredictor.PredictWarpedCompound(
                                     source,
                                     sourceStride,
                                     sourceOrigin,
@@ -498,7 +498,7 @@ internal sealed class Av1BlockDecoder : IDisposable
                                     ? blockReconstructionBuffer[reconstructionStride..]
                                     : secondPrediction;
 
-                                Av1InterPredictor.PredictWarped(
+                                Av1WarpedInterPredictor.PredictWarped(
                                     source,
                                     sourceStride,
                                     sourceOrigin,
@@ -635,7 +635,7 @@ internal sealed class Av1BlockDecoder : IDisposable
                                 ? firstCompoundPrediction
                                 : highBitDepthSecondPrediction;
 
-                            Av1InterPredictor.PredictCompound(
+                            Av1CompoundInterPredictor.PredictCompound(
                                 source,
                                 sourceStride,
                                 sourceIndex,
@@ -681,7 +681,7 @@ internal sealed class Av1BlockDecoder : IDisposable
                         switch (modeInfo.CompoundType)
                         {
                             case Av1CompoundType.Average:
-                                Av1CompoundInterPredictor.AverageIntermediate(
+                                Av1CompoundIntermediateAveragePredictor.AverageIntermediate(
                                     destination,
                                     reconstructionStride,
                                     first,
@@ -694,7 +694,7 @@ internal sealed class Av1BlockDecoder : IDisposable
 
                                 break;
                             case Av1CompoundType.DistanceWeighted:
-                                Av1CompoundInterPredictor.DistanceWeightedIntermediate(
+                                Av1CompoundIntermediateDistanceWeightedPredictor.DistanceWeightedIntermediate(
                                     destination,
                                     reconstructionStride,
                                     first,
@@ -719,7 +719,7 @@ internal sealed class Av1BlockDecoder : IDisposable
                                     subY,
                                     invert: false);
 
-                                Av1CompoundInterPredictor.BlendIntermediate(
+                                Av1CompoundIntermediateMaskBlendPredictor.BlendIntermediate(
                                     destination,
                                     reconstructionStride,
                                     first,
@@ -739,7 +739,7 @@ internal sealed class Av1BlockDecoder : IDisposable
                                 int lumaWidth = blockSize.GetWidth();
                                 if (plane == 0)
                                 {
-                                    Av1CompoundInterPredictor.FillDifferenceWeightedIntermediateMask(
+                                    Av1CompoundIntermediateDifferenceWeightedMaskBuilder.FillDifferenceWeightedIntermediateMask(
                                         compoundMask,
                                         lumaWidth,
                                         first,
@@ -752,7 +752,7 @@ internal sealed class Av1BlockDecoder : IDisposable
                                         modeInfo.DifferenceWeightedMaskType);
                                 }
 
-                                Av1CompoundInterPredictor.BlendIntermediate(
+                                Av1CompoundIntermediateMaskBlendPredictor.BlendIntermediate(
                                     destination,
                                     reconstructionStride,
                                     first,
@@ -778,7 +778,7 @@ internal sealed class Av1BlockDecoder : IDisposable
                         switch (modeInfo.CompoundType)
                         {
                             case Av1CompoundType.Average:
-                                Av1CompoundInterPredictor.Average(
+                                Av1CompoundAveragePredictor.Average(
                                     destination,
                                     reconstructionStride,
                                     highBitDepthSecondPrediction,
@@ -788,7 +788,7 @@ internal sealed class Av1BlockDecoder : IDisposable
 
                                 break;
                             case Av1CompoundType.DistanceWeighted:
-                                Av1CompoundInterPredictor.DistanceWeighted(
+                                Av1CompoundDistanceWeightedPredictor.DistanceWeighted(
                                     destination,
                                     reconstructionStride,
                                     highBitDepthSecondPrediction,
@@ -810,7 +810,7 @@ internal sealed class Av1BlockDecoder : IDisposable
                                     subY,
                                     invert: false);
 
-                                Av1CompoundInterPredictor.Blend(
+                                Av1CompoundMaskBlendPredictor.Blend(
                                     destination,
                                     reconstructionStride,
                                     highBitDepthSecondPrediction,
@@ -822,7 +822,7 @@ internal sealed class Av1BlockDecoder : IDisposable
 
                                 break;
                             default:
-                                Av1CompoundInterPredictor.FillDifferenceWeightedMask(
+                                Av1DifferenceWeightedMaskBuilder.FillDifferenceWeightedMask(
                                     compoundMask,
                                     predictionWidth,
                                     destination,
@@ -834,7 +834,7 @@ internal sealed class Av1BlockDecoder : IDisposable
                                     this.frameBuffer.BitDepth.GetBitCount(),
                                     modeInfo.DifferenceWeightedMaskType);
 
-                                Av1CompoundInterPredictor.Blend(
+                                Av1CompoundMaskBlendPredictor.Blend(
                                     destination,
                                     reconstructionStride,
                                     highBitDepthSecondPrediction,
@@ -853,7 +853,7 @@ internal sealed class Av1BlockDecoder : IDisposable
                         switch (modeInfo.CompoundType)
                         {
                             case Av1CompoundType.Average:
-                                Av1CompoundInterPredictor.Average(
+                                Av1CompoundAveragePredictor.Average(
                                     destination,
                                     reconstructionStride,
                                     secondPrediction,
@@ -863,7 +863,7 @@ internal sealed class Av1BlockDecoder : IDisposable
 
                                 break;
                             case Av1CompoundType.DistanceWeighted:
-                                Av1CompoundInterPredictor.DistanceWeighted(
+                                Av1CompoundDistanceWeightedPredictor.DistanceWeighted(
                                     destination,
                                     reconstructionStride,
                                     secondPrediction,
@@ -885,7 +885,7 @@ internal sealed class Av1BlockDecoder : IDisposable
                                     subY,
                                     invert: false);
 
-                                Av1CompoundInterPredictor.Blend(
+                                Av1CompoundMaskBlendPredictor.Blend(
                                     destination,
                                     reconstructionStride,
                                     secondPrediction,
@@ -897,7 +897,7 @@ internal sealed class Av1BlockDecoder : IDisposable
 
                                 break;
                             default:
-                                Av1CompoundInterPredictor.FillDifferenceWeightedMask(
+                                Av1DifferenceWeightedMaskBuilder.FillDifferenceWeightedMask(
                                     compoundMask,
                                     predictionWidth,
                                     destination,
@@ -908,7 +908,7 @@ internal sealed class Av1BlockDecoder : IDisposable
                                     predictionHeight,
                                     modeInfo.DifferenceWeightedMaskType);
 
-                                Av1CompoundInterPredictor.Blend(
+                                Av1CompoundMaskBlendPredictor.Blend(
                                     destination,
                                     reconstructionStride,
                                     secondPrediction,
@@ -950,7 +950,7 @@ internal sealed class Av1BlockDecoder : IDisposable
                         }
                         else
                         {
-                            Av1CompoundInterPredictor.FillInterIntraMask(
+                            Av1InterIntraMaskBuilder.FillInterIntraMask(
                                 compoundMask,
                                 predictionWidth,
                                 predictionWidth,
@@ -959,7 +959,7 @@ internal sealed class Av1BlockDecoder : IDisposable
                                 invert: true);
                         }
 
-                        Av1CompoundInterPredictor.Blend(
+                        Av1CompoundMaskBlendPredictor.Blend(
                             MemoryMarshal.Cast<short, ushort>(highBitDepthBlockReconstructionBuffer[reconstructionStride..]),
                             reconstructionStride,
                             highBitDepthSecondPrediction,
@@ -995,7 +995,7 @@ internal sealed class Av1BlockDecoder : IDisposable
                         }
                         else
                         {
-                            Av1CompoundInterPredictor.FillInterIntraMask(
+                            Av1InterIntraMaskBuilder.FillInterIntraMask(
                                 compoundMask,
                                 predictionWidth,
                                 predictionWidth,
@@ -1004,7 +1004,7 @@ internal sealed class Av1BlockDecoder : IDisposable
                                 invert: true);
                         }
 
-                        Av1CompoundInterPredictor.Blend(
+                        Av1CompoundMaskBlendPredictor.Blend(
                             blockReconstructionBuffer[reconstructionStride..],
                             reconstructionStride,
                             secondPrediction,
@@ -1528,7 +1528,7 @@ internal sealed class Av1BlockDecoder : IDisposable
                 sourceOrigin.X +
                 (sourceColumnQ10 >> Av1ReferenceScale.SubpixelBits);
 
-            Av1InterPredictor.PredictScaled(
+            Av1ScaledInterPredictor.PredictScaled(
                 source,
                 sourceStride,
                 sourceIndex,
@@ -1559,7 +1559,7 @@ internal sealed class Av1BlockDecoder : IDisposable
                 sourceOrigin.X +
                 (sourceColumnQ10 >> Av1ReferenceScale.SubpixelBits);
 
-            Av1InterPredictor.PredictScaled(
+            Av1ScaledInterPredictor.PredictScaled(
                 source,
                 sourceStride,
                 sourceIndex,
@@ -1664,7 +1664,7 @@ internal sealed class Av1BlockDecoder : IDisposable
 
                     if (highBitDepth)
                     {
-                        Av1CompoundInterPredictor.Blend(
+                        Av1CompoundMaskBlendPredictor.Blend(
                             MemoryMarshal.Cast<short, ushort>(highBitDepthBlockReconstructionBuffer[reconstructionStride..])[destinationColumn..],
                             reconstructionStride,
                             highBitDepthNeighborPrediction,
@@ -1676,7 +1676,7 @@ internal sealed class Av1BlockDecoder : IDisposable
                     }
                     else
                     {
-                        Av1CompoundInterPredictor.Blend(
+                        Av1CompoundMaskBlendPredictor.Blend(
                             blockReconstructionBuffer[reconstructionStride..][destinationColumn..],
                             reconstructionStride,
                             neighborPrediction,
@@ -1742,7 +1742,7 @@ internal sealed class Av1BlockDecoder : IDisposable
                     ReadOnlySpan<byte> horizontalMask = Av1ObmcMask.Get(overlapWidth);
                     if (highBitDepth)
                     {
-                        Av1CompoundInterPredictor.Blend(
+                        Av1CompoundMaskBlendPredictor.Blend(
                             MemoryMarshal.Cast<short, ushort>(highBitDepthBlockReconstructionBuffer[reconstructionStride..])[(destinationRow * reconstructionStride)..],
                             reconstructionStride,
                             highBitDepthNeighborPrediction,
@@ -1754,7 +1754,7 @@ internal sealed class Av1BlockDecoder : IDisposable
                     }
                     else
                     {
-                        Av1CompoundInterPredictor.Blend(
+                        Av1CompoundMaskBlendPredictor.Blend(
                             blockReconstructionBuffer[reconstructionStride..][(destinationRow * reconstructionStride)..],
                             reconstructionStride,
                             neighborPrediction,

src/ImageSharp/Formats/Heif/Av1/Transform/Av1Transform2dOperations.cs

@@ -202,9 +202,9 @@ internal static class Av1Transform2dOperations
         Vector128<int> columns01 = Vector128_.UnpackLow(pair0.AsInt32(), pair1.AsInt32());
         Vector128<int> columns23 = Vector128_.UnpackHigh(pair0.AsInt32(), pair1.AsInt32());
 
-        Store4Int16(columns01.AsUInt64().GetElement(0), ref destination);
+        Store4Int16(columns01.AsUInt64().ToScalar(), ref destination);
         Store4Int16(columns01.AsUInt64().GetElement(1), ref Unsafe.Add(ref destination, destinationStride));
-        Store4Int16(columns23.AsUInt64().GetElement(0), ref Unsafe.Add(ref destination, 2 * destinationStride));
+        Store4Int16(columns23.AsUInt64().ToScalar(), ref Unsafe.Add(ref destination, 2 * destinationStride));
         Store4Int16(columns23.AsUInt64().GetElement(1), ref Unsafe.Add(ref destination, 3 * destinationStride));
     }
 

src/ImageSharp/Formats/Heif/Components/ColorConverters/HeifSampleConversion.cs

@@ -884,5 +884,4 @@ internal static class HeifSampleConversion
         Unsafe.As<Rgba64, Vector128<uint>>(ref destination) = lower;
         Unsafe.As<Rgba64, Vector128<uint>>(ref Unsafe.Add(ref destination, 2)) = upper;
     }
-
 }

src/ImageSharp/Formats/Heif/Components/ColorConverters/HeifYuv420ToRgb8Converter.Parameters.cs

@@ -2,6 +2,7 @@
 // Licensed under the Six Labors Split License.
 
 using System.Runtime.Intrinsics;
+
 namespace SixLabors.ImageSharp.Formats.Heif.Components;
 
 /// <content>

tests/ImageSharp.Benchmarks/Codecs/Heif/Av1ChromaFromLumaBenchmarks.cs

@@ -5,6 +5,7 @@ using BenchmarkDotNet.Attributes;
 using BenchmarkDotNet.Columns;
 using BenchmarkDotNet.Configs;
 using BenchmarkDotNet.Jobs;
+using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction;
 using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.ChromaFromLuma;
 
 namespace SixLabors.ImageSharp.Benchmarks.Codecs.Heif;

tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundBlockDecoderTests.cs

@@ -671,8 +671,8 @@ public class Av1CompoundBlockDecoderTests
         Av1FrameBuffer<byte> secondReference = referenceFrames.Resolve(1)!.FrameBuffer;
         ushort[] firstHighBitDepthPrediction = new ushort[blockSize * blockSize];
         ushort[] secondHighBitDepthPrediction = new ushort[blockSize * blockSize];
-        short[] firstScratch = new short[Av1InterPredictor.WarpedScratchLength];
-        short[] secondScratch = new short[Av1InterPredictor.WarpedScratchLength];
+        short[] firstScratch = new short[Av1WarpedInterPredictor.WarpedScratchLength];
+        short[] secondScratch = new short[Av1WarpedInterPredictor.WarpedScratchLength];
         Point blockPosition = new(blockOrigin, blockOrigin);
         if (bitDepth == Av1BitDepth.EightBit)
         {
@@ -690,7 +690,7 @@ public class Av1CompoundBlockDecoderTests
                 out int secondStride,
                 out Point secondOrigin);
 
-            Av1InterPredictor.PredictWarpedCompoundScalar(
+            Av1WarpedInterPredictor.PredictWarpedCompoundScalar(
                 firstSource,
                 firstStride,
                 firstOrigin,
@@ -706,7 +706,7 @@ public class Av1CompoundBlockDecoderTests
                 globalMotionParameters,
                 firstScratch);
 
-            Av1InterPredictor.PredictWarpedCompoundScalar(
+            Av1WarpedInterPredictor.PredictWarpedCompoundScalar(
                 secondSource,
                 secondStride,
                 secondOrigin,
@@ -739,7 +739,7 @@ public class Av1CompoundBlockDecoderTests
                 out Point secondOrigin);
 
             int bitDepthValue = bitDepth.GetBitCount();
-            Av1InterPredictor.PredictWarpedScalar(
+            Av1WarpedInterPredictor.PredictWarpedScalar(
                 firstSource,
                 firstStride,
                 firstOrigin,
@@ -756,7 +756,7 @@ public class Av1CompoundBlockDecoderTests
                 globalMotionParameters,
                 firstScratch);
 
-            Av1InterPredictor.PredictWarpedScalar(
+            Av1WarpedInterPredictor.PredictWarpedScalar(
                 secondSource,
                 secondStride,
                 secondOrigin,

tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundInterPredictorTests.cs

@@ -8,7 +8,7 @@ using SixLabors.ImageSharp.Tests.TestUtilities;
 namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1;
 
 /// <summary>
-/// Verifies equal-weight AV1 compound prediction across every hardware-intrinsic tier.
+/// Verifies AV1 compound prediction and blending across every hardware-intrinsic tier.
 /// </summary>
 [Trait("Format", "Avif")]
 public class Av1CompoundInterPredictorTests
@@ -65,8 +65,8 @@ public class Av1CompoundInterPredictorTests
             mask.AsSpan().Fill(0xA5);
             inverted.AsSpan().Fill(0xA5);
 
-            Av1CompoundInterPredictor.FillInterIntraMask(mask, stride, width, height, mode, invert: false);
-            Av1CompoundInterPredictor.FillInterIntraMask(inverted, stride, width, height, mode, invert: true);
+            Av1InterIntraMaskBuilder.FillInterIntraMask(mask, stride, width, height, mode, invert: false);
+            Av1InterIntraMaskBuilder.FillInterIntraMask(inverted, stride, width, height, mode, invert: true);
 
             for (int row = 0; row < height; row++)
             {
@@ -93,6 +93,24 @@ public class Av1CompoundInterPredictorTests
         }
     }
 
+    /// <summary>
+    /// Verifies the pinned horizontal curve at the index exercised by a 32-by-16 inter-intra block.
+    /// </summary>
+    [Fact]
+    public void HorizontalInterIntraMaskMatchesPinnedThirtyTwoWideCurve()
+    {
+        const int width = 32;
+        const int height = 16;
+        byte[] mask = new byte[width * height];
+        byte[] inverted = new byte[width * height];
+
+        Av1InterIntraMaskBuilder.FillInterIntraMask(mask, width, width, height, Av1InterIntraMode.Horizontal, invert: false);
+        Av1InterIntraMaskBuilder.FillInterIntraMask(inverted, width, width, height, Av1InterIntraMode.Horizontal, invert: true);
+
+        Assert.Equal(2, mask[23]);
+        Assert.Equal(62, inverted[23]);
+    }
+
     /// <summary>
     /// Verifies both difference-mask orientations at each supported bit depth.
     /// </summary>
@@ -136,7 +154,7 @@ public class Av1CompoundInterPredictorTests
                     {
                         byte[] firstByte = Array.ConvertAll(first, value => (byte)value);
                         byte[] secondByte = Array.ConvertAll(second, value => (byte)value);
-                        Av1CompoundInterPredictor.FillDifferenceWeightedMask(
+                        Av1DifferenceWeightedMaskBuilder.FillDifferenceWeightedMask(
                             actual,
                             maskStride,
                             firstByte,
@@ -149,7 +167,7 @@ public class Av1CompoundInterPredictorTests
                     }
                     else
                     {
-                        Av1CompoundInterPredictor.FillDifferenceWeightedMask(
+                        Av1DifferenceWeightedMaskBuilder.FillDifferenceWeightedMask(
                             actual,
                             maskStride,
                             first,
@@ -214,8 +232,8 @@ public class Av1CompoundInterPredictorTests
                 }
             }
 
-            Av1CompoundInterPredictor.Average(actual, destinationStride, second, secondStride, width, height);
-            Av1CompoundInterPredictor.AverageScalar(scalar, destinationStride, second, secondStride, width, height);
+            Av1CompoundAveragePredictor.Average(actual, destinationStride, second, secondStride, width, height);
+            Av1CompoundAveragePredictor.AverageScalar(scalar, destinationStride, second, secondStride, width, height);
 
             Assert.Equal(expected, actual);
             Assert.Equal(expected, scalar);
@@ -255,8 +273,8 @@ public class Av1CompoundInterPredictorTests
                     }
                 }
 
-                Av1CompoundInterPredictor.Average(actual, destinationStride, second, secondStride, width, height);
-                Av1CompoundInterPredictor.AverageScalar(scalar, destinationStride, second, secondStride, width, height);
+                Av1CompoundAveragePredictor.Average(actual, destinationStride, second, secondStride, width, height);
+                Av1CompoundAveragePredictor.AverageScalar(scalar, destinationStride, second, secondStride, width, height);
 
                 Assert.Equal(expected, actual);
                 Assert.Equal(expected, scalar);
@@ -303,7 +321,7 @@ public class Av1CompoundInterPredictorTests
                     }
                 }
 
-                Av1CompoundInterPredictor.DistanceWeighted(
+                Av1CompoundDistanceWeightedPredictor.DistanceWeighted(
                     actual,
                     destinationStride,
                     second,
@@ -330,7 +348,7 @@ public class Av1CompoundInterPredictorTests
                 }
             }
 
-            Av1CompoundInterPredictor.Blend(
+            Av1CompoundMaskBlendPredictor.Blend(
                 maskedActual,
                 destinationStride,
                 second,
@@ -385,7 +403,7 @@ public class Av1CompoundInterPredictorTests
                         }
                     }
 
-                    Av1CompoundInterPredictor.DistanceWeighted(
+                    Av1CompoundDistanceWeightedPredictor.DistanceWeighted(
                         actual,
                         destinationStride,
                         second,
@@ -412,7 +430,7 @@ public class Av1CompoundInterPredictorTests
                     }
                 }
 
-                Av1CompoundInterPredictor.Blend(
+                Av1CompoundMaskBlendPredictor.Blend(
                     maskedActual,
                     destinationStride,
                     second,

tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterPredictorTests.cs

@@ -186,7 +186,7 @@ public class Av1InterPredictorTests
             short[] simdScratch = CreateScratch(testCase);
             short[] scalarScratch = CreateScratch(testCase);
 
-            Av1InterPredictor.PredictCompoundScalar(
+            Av1CompoundInterPredictor.PredictCompoundScalar(
                 source,
                 sourceStride,
                 sourceOrigin,
@@ -200,7 +200,7 @@ public class Av1InterPredictorTests
                 testCase.VerticalPhase,
                 scalarScratch);
 
-            Av1InterPredictor.PredictCompound(
+            Av1CompoundInterPredictor.PredictCompound(
                 source,
                 sourceStride,
                 sourceOrigin,

tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReconstructionConformanceTests.cs

@@ -2,6 +2,7 @@
 // Licensed under the Six Labors Split License.
 
 using System.Buffers.Binary;
+using System.Globalization;
 using System.Text;
 using SixLabors.ImageSharp.Formats;
 using SixLabors.ImageSharp.Formats.Heif;
@@ -155,6 +156,53 @@ public class Av1ReconstructionConformanceTests
     /// </summary>
     private const int AverageCompoundFixtureFrameCount = 19;
 
+    /// <summary>
+    /// The number of shown frames in the official libaom all-intra sequence.
+    /// </summary>
+    private const int OfficialAllIntraFixtureFrameCount = 39;
+
+    /// <summary>
+    /// The number of shown frames in the official libaom CDF-update sequence.
+    /// </summary>
+    private const int OfficialCdfUpdateFixtureFrameCount = 2;
+
+    /// <summary>
+    /// The number of shown frames in the official libaom temporal motion-field sequence.
+    /// </summary>
+    private const int OfficialMotionFieldFixtureFrameCount = 4;
+
+    /// <summary>
+    /// The coverage bit representing tile-local adaptive CDF updates.
+    /// </summary>
+    private const int TileCdfUpdateCoverage = 1 << 0;
+
+    /// <summary>
+    /// The coverage bit representing publication of the selected frame-end CDF.
+    /// </summary>
+    private const int FrameEndCdfUpdateCoverage = 1 << 1;
+
+    /// <summary>
+    /// The coverage bit representing temporal reference-motion-vector projection.
+    /// </summary>
+    private const int ReferenceFrameMotionVectorCoverage = 1 << 2;
+
+    /// <summary>
+    /// The bit mask containing every intra prediction mode.
+    /// </summary>
+    private const int RequiredIntraModeCoverage = (1 << (int)Av1PredictionMode.IntraModes) - 1;
+
+    /// <summary>
+    /// The transform types selected by the official all-intra conformance sequence.
+    /// </summary>
+    private const int RequiredAllIntraTransformTypeCoverage =
+        (1 << (int)Av1TransformType.DctDct) |
+        (1 << (int)Av1TransformType.AdstDct) |
+        (1 << (int)Av1TransformType.DctAdst) |
+        (1 << (int)Av1TransformType.AdstAdst) |
+        (1 << (int)Av1TransformType.Identity) |
+        (1 << (int)Av1TransformType.VerticalDct) |
+        (1 << (int)Av1TransformType.HorizontalDct);
+
     /// <summary>
     /// The displayed width of the official libaom motion-vector sequence.
     /// </summary>
@@ -898,6 +946,255 @@ public class Av1ReconstructionConformanceTests
             fixtureSize: 256,
             visibleFrameCount: 2);
 
+    /// <summary>
+    /// Verifies every intra prediction mode and the fixture's seven transform types against the official
+    /// pinned-libaom all-intra conformance sequence and its exact native output.
+    /// </summary>
+    [Fact]
+    public void DecodeOfficialAllIntraSequenceMatchesPinnedLibaomReference() => ValidateOfficialAllIntraFixture();
+
+    /// <summary>
+    /// Decodes every all-intra IVF sample in one session, compares each frame exactly, and records the syntax
+    /// selections that make the fixture authoritative for prediction and transform coverage.
+    /// </summary>
+    private static void ValidateOfficialAllIntraFixture()
+    {
+        byte[] ivf = TestFile.Create(TestImages.Heif.Av1OfficialAllIntraSequence).Bytes;
+        byte[] nativeReference = TestFile.Create(TestImages.Heif.Av1OfficialAllIntraSequenceNativeReference).Bytes;
+        ReadOnlySpan<byte> y4mFileHeader = "YUV4MPEG2 W352 H288 F3:1 Ip C420jpeg\n"u8;
+        ReadOnlySpan<byte> y4mFrameHeader = "FRAME\n"u8;
+
+        Assert.True(ivf.AsSpan(0, 4).SequenceEqual("DKIF"u8));
+        Assert.Equal(0, BinaryPrimitives.ReadUInt16LittleEndian(ivf.AsSpan(4, 2)));
+        Assert.Equal(32, BinaryPrimitives.ReadUInt16LittleEndian(ivf.AsSpan(6, 2)));
+        Assert.True(ivf.AsSpan(8, 4).SequenceEqual("AV01"u8));
+        Assert.Equal(OfficialMotionVectorFixtureWidth, BinaryPrimitives.ReadUInt16LittleEndian(ivf.AsSpan(12, 2)));
+        Assert.Equal(OfficialMotionVectorFixtureHeight, BinaryPrimitives.ReadUInt16LittleEndian(ivf.AsSpan(14, 2)));
+        Assert.Equal(
+            OfficialAllIntraFixtureFrameCount,
+            checked((int)BinaryPrimitives.ReadUInt32LittleEndian(ivf.AsSpan(24, 4))));
+
+        Assert.True(nativeReference.AsSpan().StartsWith(y4mFileHeader));
+
+        int ivfOffset = 32;
+        int nativeOffset = y4mFileHeader.Length;
+        int nativeFrameLength =
+            (OfficialMotionVectorFixtureWidth * OfficialMotionVectorFixtureHeight) +
+            (2 * (OfficialMotionVectorFixtureWidth >> 1) * (OfficialMotionVectorFixtureHeight >> 1));
+
+        int intraModeCoverage = 0;
+        int transformTypeCoverage = 0;
+        using Av1Decoder decoder = new(Configuration.Default);
+        for (int frameIndex = 0; frameIndex < OfficialAllIntraFixtureFrameCount; frameIndex++)
+        {
+            int payloadLength = checked((int)BinaryPrimitives.ReadUInt32LittleEndian(ivf.AsSpan(ivfOffset, 4)));
+            ivfOffset += 12;
+            using ImageFrame<Rgba32> frame = decoder.DecodeSequenceFrame<Rgba32>(
+                ivf.AsSpan(ivfOffset, payloadLength),
+                null,
+                null);
+
+            ivfOffset += payloadLength;
+            Assert.Equal(OfficialMotionVectorFixtureWidth, frame.Width);
+            Assert.Equal(OfficialMotionVectorFixtureHeight, frame.Height);
+            Assert.True(nativeReference.AsSpan(nativeOffset).StartsWith(y4mFrameHeader));
+            nativeOffset += y4mFrameHeader.Length;
+
+            Av1FrameBuffer<byte> frameBuffer = Assert.IsType<Av1FrameBuffer<byte>>(decoder.FrameBuffer);
+            Assert.Equal(OfficialMotionVectorFixtureWidth, frameBuffer.Width);
+            Assert.Equal(OfficialMotionVectorFixtureHeight, frameBuffer.Height);
+            Assert.Equal(Av1BitDepth.EightBit, frameBuffer.BitDepth);
+            Assert.Equal(Av1ColorFormat.Yuv420, frameBuffer.ColorFormat);
+            AssertNativePlanesEqual(
+                decoder,
+                frameBuffer,
+                nativeReference.AsSpan(nativeOffset, nativeFrameLength));
+
+            nativeOffset += nativeFrameLength;
+
+            ObuSequenceHeader sequenceHeader = Assert.IsType<ObuSequenceHeader>(decoder.SequenceHeader);
+            Av1FrameInfo frameInfo = Assert.IsType<Av1FrameInfo>(decoder.FrameInfo);
+            int superblockColumnCount = Av1Math.AlignPowerOf2(sequenceHeader.MaxFrameWidth, sequenceHeader.SuperblockSizeLog2)
+                >> sequenceHeader.SuperblockSizeLog2;
+            int superblockRowCount = Av1Math.AlignPowerOf2(sequenceHeader.MaxFrameHeight, sequenceHeader.SuperblockSizeLog2)
+                >> sequenceHeader.SuperblockSizeLog2;
+
+            for (int superblockRow = 0; superblockRow < superblockRowCount; superblockRow++)
+            {
+                for (int superblockColumn = 0; superblockColumn < superblockColumnCount; superblockColumn++)
+                {
+                    Av1SuperblockInfo superblockInfo = frameInfo.GetSuperblock(new Point(superblockColumn, superblockRow));
+                    foreach (Av1BlockModeInfo modeInfo in superblockInfo.GetModeInfos())
+                    {
+                        if (modeInfo.YMode is >= Av1PredictionMode.IntraModeStart and < Av1PredictionMode.IntraModeEnd)
+                        {
+                            intraModeCoverage |= 1 << ((int)modeInfo.YMode - (int)Av1PredictionMode.IntraModeStart);
+                        }
+
+                        int firstTransformLocation = modeInfo.GetFirstTransformLocation(Av1Plane.Y);
+                        int transformUnitCount = modeInfo.GetTransformUnitCount(Av1Plane.Y);
+                        foreach (Av1TransformInfo transformInfo in
+                            superblockInfo.GetTransformInfoY().Slice(firstTransformLocation, transformUnitCount))
+                        {
+                            transformTypeCoverage |= 1 << (int)transformInfo.Type;
+                        }
+                    }
+                }
+            }
+        }
+
+        Assert.Equal(ivf.Length, ivfOffset);
+        Assert.Equal(nativeReference.Length, nativeOffset);
+        Assert.Equal(RequiredIntraModeCoverage, intraModeCoverage);
+        Assert.Equal(RequiredAllIntraTransformTypeCoverage, transformTypeCoverage);
+    }
+
+    /// <summary>
+    /// Verifies adaptive tile and frame-end CDF updates against the official pinned-libaom sequence and exact native
+    /// output under normal and scalar dispatch.
+    /// </summary>
+    [Fact]
+    public void DecodeOfficialCdfUpdateSequenceMatchesPinnedLibaomReference()
+        => FeatureTestRunner.RunWithHwIntrinsicsFeature(
+            ValidateOfficialCdfUpdateFixture,
+            ReconstructionConfigurations);
+
+    /// <summary>
+    /// Verifies temporal reference-motion-vector projection against the official pinned-libaom sequence and exact
+    /// native output under normal and scalar dispatch.
+    /// </summary>
+    [Fact]
+    public void DecodeOfficialMotionFieldSequenceMatchesPinnedLibaomReference()
+        => FeatureTestRunner.RunWithHwIntrinsicsFeature(
+            ValidateOfficialMotionFieldFixture,
+            ReconstructionConfigurations);
+
+    /// <summary>
+    /// Verifies exact temporal motion-field reconstruction and balanced ownership with a constrained allocator.
+    /// </summary>
+    [Fact]
+    [ValidateDisposedMemoryAllocations]
+    public void DecodeOfficialMotionFieldSequenceWithConstrainedAllocator()
+    {
+        TestMemoryAllocator allocator = new() { BufferCapacityInBytes = 2_048 };
+        allocator.EnableNonThreadSafeLogging();
+        Configuration configuration = Configuration.Default.Clone();
+        configuration.MemoryAllocator = allocator;
+
+        int coverage = ValidateOfficialCompactSequence(
+            configuration,
+            TestImages.Heif.Av1OfficialMotionFieldSequence,
+            TestImages.Heif.Av1OfficialMotionFieldSequenceNativeReference,
+            OfficialMotionFieldFixtureFrameCount);
+
+        Assert.NotEqual(0, coverage & ReferenceFrameMotionVectorCoverage);
+        Assert.Contains(allocator.AllocationLog, request => request.ElementType.Name == "RetainedMotionFieldEntry");
+        Assert.Contains(allocator.AllocationLog, request => request.ElementType.Name == "TemporalMotionFieldEntry");
+        Assert.Equal(allocator.AllocationLog.Count, allocator.ReturnLog.Count);
+        Assert.All(
+            allocator.AllocationLog,
+            allocation => Assert.Single(
+                allocator.ReturnLog,
+                returned => returned.AllocationId == allocation.AllocationId));
+    }
+
+    /// <summary>
+    /// Validates that the official CDF-update fixture selects both adaptive update boundaries.
+    /// </summary>
+    private static void ValidateOfficialCdfUpdateFixture()
+    {
+        int coverage = ValidateOfficialCompactSequence(
+            Configuration.Default,
+            TestImages.Heif.Av1OfficialCdfUpdateSequence,
+            TestImages.Heif.Av1OfficialCdfUpdateSequenceNativeReference,
+            OfficialCdfUpdateFixtureFrameCount);
+
+        Assert.Equal(TileCdfUpdateCoverage | FrameEndCdfUpdateCoverage, coverage & 3);
+    }
+
+    /// <summary>
+    /// Validates that the official temporal motion-field fixture enables projected reference motion vectors.
+    /// </summary>
+    private static void ValidateOfficialMotionFieldFixture()
+    {
+        int coverage = ValidateOfficialCompactSequence(
+            Configuration.Default,
+            TestImages.Heif.Av1OfficialMotionFieldSequence,
+            TestImages.Heif.Av1OfficialMotionFieldSequenceNativeReference,
+            OfficialMotionFieldFixtureFrameCount);
+
+        Assert.NotEqual(0, coverage & ReferenceFrameMotionVectorCoverage);
+    }
+
+    /// <summary>
+    /// Decodes one compact official IVF sequence, compares every native sample, and returns its active frame-state
+    /// coverage mask.
+    /// </summary>
+    private static int ValidateOfficialCompactSequence(
+        Configuration configuration,
+        string fixturePath,
+        string nativeReferencePath,
+        int expectedFrameCount)
+    {
+        byte[] ivf = TestFile.Create(fixturePath).Bytes;
+        byte[] nativeReference = TestFile.Create(nativeReferencePath).Bytes;
+        ReadOnlySpan<byte> y4mFileHeader = "YUV4MPEG2 W352 H288 F30:1 Ip C420jpeg\n"u8;
+        ReadOnlySpan<byte> y4mFrameHeader = "FRAME\n"u8;
+
+        Assert.True(ivf.AsSpan(0, 4).SequenceEqual("DKIF"u8));
+        Assert.Equal(0, BinaryPrimitives.ReadUInt16LittleEndian(ivf.AsSpan(4, 2)));
+        Assert.Equal(32, BinaryPrimitives.ReadUInt16LittleEndian(ivf.AsSpan(6, 2)));
+        Assert.True(ivf.AsSpan(8, 4).SequenceEqual("AV01"u8));
+        Assert.Equal(OfficialMotionVectorFixtureWidth, BinaryPrimitives.ReadUInt16LittleEndian(ivf.AsSpan(12, 2)));
+        Assert.Equal(OfficialMotionVectorFixtureHeight, BinaryPrimitives.ReadUInt16LittleEndian(ivf.AsSpan(14, 2)));
+        Assert.Equal(
+            expectedFrameCount,
+            checked((int)BinaryPrimitives.ReadUInt32LittleEndian(ivf.AsSpan(24, 4))));
+
+        Assert.True(nativeReference.AsSpan().StartsWith(y4mFileHeader));
+
+        int ivfOffset = 32;
+        int nativeOffset = y4mFileHeader.Length;
+        int nativeFrameLength =
+            (OfficialMotionVectorFixtureWidth * OfficialMotionVectorFixtureHeight) +
+            (2 * (OfficialMotionVectorFixtureWidth >> 1) * (OfficialMotionVectorFixtureHeight >> 1));
+
+        int coverage = 0;
+        using Av1Decoder decoder = new(configuration);
+        for (int frameIndex = 0; frameIndex < expectedFrameCount; frameIndex++)
+        {
+            int payloadLength = checked((int)BinaryPrimitives.ReadUInt32LittleEndian(ivf.AsSpan(ivfOffset, 4)));
+            ivfOffset += 12;
+            using ImageFrame<Rgba32> frame = decoder.DecodeSequenceFrame<Rgba32>(
+                ivf.AsSpan(ivfOffset, payloadLength),
+                null,
+                null);
+
+            ivfOffset += payloadLength;
+            Assert.Equal(OfficialMotionVectorFixtureWidth, frame.Width);
+            Assert.Equal(OfficialMotionVectorFixtureHeight, frame.Height);
+            Assert.True(nativeReference.AsSpan(nativeOffset).StartsWith(y4mFrameHeader));
+            nativeOffset += y4mFrameHeader.Length;
+
+            Av1FrameBuffer<byte> frameBuffer = Assert.IsType<Av1FrameBuffer<byte>>(decoder.FrameBuffer);
+            AssertNativePlanesEqual(
+                decoder,
+                frameBuffer,
+                nativeReference.AsSpan(nativeOffset, nativeFrameLength));
+
+            nativeOffset += nativeFrameLength;
+
+            ObuFrameHeader frameHeader = Assert.IsType<ObuFrameHeader>(decoder.FrameHeader);
+            coverage |= frameHeader.DisableCdfUpdate ? 0 : TileCdfUpdateCoverage;
+            coverage |= frameHeader.DisableFrameEndUpdateCdf ? 0 : FrameEndCdfUpdateCoverage;
+            coverage |= frameHeader.UseReferenceFrameMotionVectors ? ReferenceFrameMotionVectorCoverage : 0;
+        }
+
+        Assert.Equal(ivf.Length, ivfOffset);
+        Assert.Equal(nativeReference.Length, nativeOffset);
+        return coverage;
+    }
+
     /// <summary>
     /// Verifies every ordinary inter mode, motion mode, and switchable dual-filter pair against the official
     /// pinned-libaom motion-vector conformance sequence and its exact native output.
@@ -3219,7 +3516,7 @@ public class Av1ReconstructionConformanceTests
                             if (mismatchCount < 16)
                             {
                                 mismatchDescription ??= new StringBuilder();
-                                mismatchDescription.Append($" {plane}({x},{y})={expectedRow[x]}/{actualRow[x]}");
+                                mismatchDescription.Append(CultureInfo.InvariantCulture, $" {plane}({x},{y})={expectedRow[x]}/{actualRow[x]}");
                             }
 
                             if (mismatchCount == 0 || Math.Abs(expectedRow[x] - actualRow[x]) > Math.Abs(largestExpected - largestActual))
@@ -3252,7 +3549,7 @@ public class Av1ReconstructionConformanceTests
                             if (mismatchCount < 16)
                             {
                                 mismatchDescription ??= new StringBuilder();
-                                mismatchDescription.Append($" {plane}({x},{y})={expected}/{actualRow[x]}");
+                                mismatchDescription.Append(CultureInfo.InvariantCulture, $" {plane}({x},{y})={expected}/{actualRow[x]}");
                             }
 
                             if (mismatchCount == 0 || Math.Abs(expected - actualRow[x]) > Math.Abs(largestExpected - largestActual))
@@ -3374,7 +3671,7 @@ public class Av1ReconstructionConformanceTests
             if (containingTransform.CodeBlockFlag)
             {
                 int coefficientCount = superblockCoefficients[coefficientOffset];
-                coefficientDescription.Append($", quantized-coefficients={coefficientCount}:[");
+                coefficientDescription.Append(CultureInfo.InvariantCulture, $", quantized-coefficients={coefficientCount}:[");
                 for (int coefficientIndex = 0; coefficientIndex < coefficientCount; coefficientIndex++)
                 {
                     if (coefficientIndex != 0)

tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ScaledInterPredictorTests.cs

@@ -118,7 +118,7 @@ public class Av1ScaledInterPredictorTests
             byte[] expected = CreateByteDestination(testCase, destinationStride);
             byte[] actual = (byte[])expected.Clone();
             short[] scratch = new short[
-                Av1InterPredictor.GetScaledScratchLength(
+                Av1ScaledInterPredictor.GetScaledScratchLength(
                     testCase.Width,
                     testCase.Height,
                     testCase.VerticalPhase,
@@ -126,7 +126,7 @@ public class Av1ScaledInterPredictorTests
 
             ApplyReference(source, sourceStride, sourceOrigin, expected, destinationStride, testCase, 8);
 
-            Av1InterPredictor.PredictScaled(
+            Av1ScaledInterPredictor.PredictScaled(
                 source,
                 sourceStride,
                 sourceOrigin,
@@ -161,7 +161,7 @@ public class Av1ScaledInterPredictorTests
                 ushort[] expected = CreateUInt16Destination(testCase, destinationStride);
                 ushort[] actual = (ushort[])expected.Clone();
                 short[] scratch = new short[
-                    Av1InterPredictor.GetScaledScratchLength(
+                    Av1ScaledInterPredictor.GetScaledScratchLength(
                         testCase.Width,
                         testCase.Height,
                         testCase.VerticalPhase,
@@ -169,7 +169,7 @@ public class Av1ScaledInterPredictorTests
 
                 ApplyReference(source, sourceStride, sourceOrigin, expected, destinationStride, testCase, bitDepth);
 
-                Av1InterPredictor.PredictScaled(
+                Av1ScaledInterPredictor.PredictScaled(
                     source,
                     sourceStride,
                     sourceOrigin,
@@ -201,6 +201,7 @@ public class Av1ScaledInterPredictorTests
         new("fixture-regular-8x4", 8, 4, Av1InterpolationFilter.Regular, Av1InterpolationFilter.Regular, 800, 512, 800, 512),
         new("bilinear-variable-phase", 13, 9, Av1InterpolationFilter.Bilinear, Av1InterpolationFilter.Bilinear, 192, 1536, 512, 640),
         new("regular-smooth-wide", 20, 8, Av1InterpolationFilter.Regular, Av1InterpolationFilter.Smooth, 64, 2048, 448, 2048),
+        new("regular-sharp-all-widths", 37, 7, Av1InterpolationFilter.Regular, Av1InterpolationFilter.Sharp, 64, 2048, 512, 2048),
         new("sharp-bilinear-tail", 12, 5, Av1InterpolationFilter.Sharp, Av1InterpolationFilter.Bilinear, 512, 2048, 192, 2048),
         new("reduced-regular", 4, 8, Av1InterpolationFilter.Regular, Av1InterpolationFilter.Smooth, 192, 2048, 448, 2048),
         new("reduced-sharp-maps-to-regular", 4, 8, Av1InterpolationFilter.Sharp, Av1InterpolationFilter.Smooth, 192, 2048, 448, 2048),

tests/ImageSharp.Tests/Formats/Heif/Av1/Av1WarpedInterPredictorTests.cs

@@ -59,11 +59,11 @@ public class Av1WarpedInterPredictorTests
             byte[] actual = new byte[destinationStride * height];
             Array.Fill(expected, (byte)0xD3);
             Array.Fill(actual, (byte)0xD3);
-            short[] expectedScratch = new short[Av1InterPredictor.WarpedScratchLength];
-            short[] actualScratch = new short[Av1InterPredictor.WarpedScratchLength];
+            short[] expectedScratch = new short[Av1WarpedInterPredictor.WarpedScratchLength];
+            short[] actualScratch = new short[Av1WarpedInterPredictor.WarpedScratchLength];
             Point destinationPosition = subsampling == 0 ? new Point(32, 24) : new Point(16, 12);
 
-            Av1InterPredictor.PredictWarpedScalar(
+            Av1WarpedInterPredictor.PredictWarpedScalar(
                 source,
                 sourceStride,
                 new Point(padding, padding),
@@ -79,7 +79,7 @@ public class Av1WarpedInterPredictorTests
                 parameters,
                 expectedScratch);
 
-            Av1InterPredictor.PredictWarped(
+            Av1WarpedInterPredictor.PredictWarped(
                 source,
                 sourceStride,
                 new Point(padding, padding),
@@ -131,11 +131,11 @@ public class Av1WarpedInterPredictorTests
                 ushort[] actual = new ushort[destinationStride * height];
                 Array.Fill(expected, (ushort)0xDEAD);
                 Array.Fill(actual, (ushort)0xDEAD);
-                short[] expectedScratch = new short[Av1InterPredictor.WarpedScratchLength];
-                short[] actualScratch = new short[Av1InterPredictor.WarpedScratchLength];
+                short[] expectedScratch = new short[Av1WarpedInterPredictor.WarpedScratchLength];
+                short[] actualScratch = new short[Av1WarpedInterPredictor.WarpedScratchLength];
                 Point destinationPosition = subsampling == 0 ? new Point(32, 24) : new Point(16, 12);
 
-                Av1InterPredictor.PredictWarpedScalar(
+                Av1WarpedInterPredictor.PredictWarpedScalar(
                     source,
                     sourceStride,
                     new Point(padding, padding),
@@ -152,7 +152,7 @@ public class Av1WarpedInterPredictorTests
                     parameters,
                     expectedScratch);
 
-                Av1InterPredictor.PredictWarped(
+                Av1WarpedInterPredictor.PredictWarped(
                     source,
                     sourceStride,
                     new Point(padding, padding),

tests/ImageSharp.Tests/TestImages.cs

@@ -1388,8 +1388,14 @@ public static class TestImages
         public const string Av1GlobalWarpSequenceAvif = "Heif/Av1/Conformance/libavif-rotating-grid-global-warp.avif";
         public const string Av1GlobalWarpSequenceNativeReference = "Heif/Av1/Conformance/libavif-rotating-grid-global-warp-libaom.y4m";
         public const string Av1GlobalWarpSequencePresentationReference = "Heif/Av1/Conformance/libavif-rotating-grid-global-warp-libavif.png";
+        public const string Av1OfficialAllIntraSequence = "Heif/Av1/Conformance/libaom-av1-1-b8-02-allintra.ivf";
+        public const string Av1OfficialAllIntraSequenceNativeReference = "Heif/Av1/Conformance/libaom-av1-1-b8-02-allintra-libaom.y4m";
+        public const string Av1OfficialCdfUpdateSequence = "Heif/Av1/Conformance/libaom-av1-1-b8-04-cdfupdate.ivf";
+        public const string Av1OfficialCdfUpdateSequenceNativeReference = "Heif/Av1/Conformance/libaom-av1-1-b8-04-cdfupdate-libaom.y4m";
         public const string Av1OfficialMotionVectorSequence = "Heif/Av1/Conformance/libaom-av1-1-b8-05-mv.ivf";
         public const string Av1OfficialMotionVectorSequenceNativeReference = "Heif/Av1/Conformance/libaom-av1-1-b8-05-mv-libaom.y4m";
+        public const string Av1OfficialMotionFieldSequence = "Heif/Av1/Conformance/libaom-av1-1-b8-06-mfmv.ivf";
+        public const string Av1OfficialMotionFieldSequenceNativeReference = "Heif/Av1/Conformance/libaom-av1-1-b8-06-mfmv-libaom.y4m";
         public const string Av1Cdef8BitPayload = "Heif/Av1/Conformance/libaom-cdef-kodim23-8b.bit";
         public const string Av1Cdef8BitReference = "Heif/Av1/Conformance/libaom-cdef-kodim23-8b-libaom.yuv";
         public const string Av1Cdef8BitAvif = "Heif/Av1/Conformance/libavif-cdef-kodim23-8b.avif";

tests/Images/Input/Heif/Av1/Conformance/libaom-av1-1-b8-02-allintra-libaom.y4m

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1211ebefbc9ccef9ed19be4cce3f807d69fffe338e95cca1b5f4ca8023482175
+size 5930767

tests/Images/Input/Heif/Av1/Conformance/libaom-av1-1-b8-02-allintra.ivf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5fcd265fd9f9bdd0d3179340b4c4532f1422ca5e5d97741c7481b84cb5dc122f
+size 1488725

tests/Images/Input/Heif/Av1/Conformance/libaom-av1-1-b8-04-cdfupdate-libaom.y4m

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4fbff73ff0de2d9084dae557d1d4bd677b0486516525bf4d327d2d795d5a7779
+size 304178

tests/Images/Input/Heif/Av1/Conformance/libaom-av1-1-b8-04-cdfupdate.ivf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:14a3dbf537b6bf15efc003182d9916d61438c93624a8bd26e6e3ae7eaf33ea82
+size 36088

tests/Images/Input/Heif/Av1/Conformance/libaom-av1-1-b8-06-mfmv-libaom.y4m

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f7db607694818c19e62fd9a27f53e1a3e2d00b72c39c0430c1b26399cc76777d
+size 608318

tests/Images/Input/Heif/Av1/Conformance/libaom-av1-1-b8-06-mfmv.ivf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b59bf9586d8546dfda81dfec4ee4e32ceb502c9d22412ab0b63a2abb534a1f14
+size 44964