diff --git a/HEIF_IMPLEMENTATION_PLAN.md b/HEIF_IMPLEMENTATION_PLAN.md index 7c892f09f..673384cc6 100644 --- a/HEIF_IMPLEMENTATION_PLAN.md +++ b/HEIF_IMPLEMENTATION_PLAN.md @@ -30,9 +30,8 @@ Reference checkout evidence on 2026-08-31: Reconciled with the worktree on 2026-08-31. - [~] The bounded container reader, still-image path, sequence parser, AV1 decoder, color pipeline, presentation pipeline, and broad AV1 test suite exist locally. -- [~] The inter-frame decoder has verified checkpoints through OBMC. Scaled references, local warped - motion, and global motion exist locally but remain open until their ordered checkpoints below are - completed. +- [~] The inter-frame decoder has verified checkpoints through scaled-reference prediction. Local warped + motion and global motion exist locally but remain open until their ordered checkpoints below are completed. - [~] Loop filtering, CDEF, super-resolution, restoration, film grain, layered presentation, alpha composition, and color conversion exist locally. Shared-source cleanup changed the current tree, so final production-path verification is open. - [~] AV1 writer primitives, forward transforms, symbol encoding, and tile-writing source exist locally, but they are not connected to the public encoder. - [ ] The public AV1 encoder is not implemented. HeifEncoderCore.Encode throws NotSupportedException when AV1 is selected. @@ -184,8 +183,8 @@ The single-reference syntax, buffer, reconstruction, and ownership foundation is - [x] Wedge compound prediction. - [x] Difference-weighted compound prediction. - [x] OBMC. -- [~] Scaled-reference prediction. Next item. -- [~] Local warped prediction. +- [x] Scaled-reference prediction. +- [~] Local warped prediction. Next item. - [~] Non-translational global prediction. - [~] Inter deblocking decisions and reference/mode deltas. @@ -394,6 +393,42 @@ Verified OBMC checkpoint evidence on 2026-08-31: - [x] The focused Release checkpoint set passes 18/18 on net10.0 and 18/18 on net11.0, with zero failures or skips. Scoped analyzer and whitespace verification pass for every changed C# file. Roslynk reports zero compiler errors, `git diff --check` passes, and `.gitattributes` is unchanged. +- [x] The completed checkpoint was committed as `7e7e3cbe6438d63926b31d966795d2652e221939` + with author and committer `James Jackson-South `. + +Verified scaled-reference checkpoint evidence on 2026-08-31: + +- [x] Audited reference-size validation and variable-scale coordinates, filters, edge extension, convolution + rounding, and compound intermediates against current libaom `av1/common/scale.c`, + `av1/decoder/decodeframe.c`, and `av1/common/convolve.c`. The frame boundary accepts the same + half-to-sixteen-times dimension range and requires at least one compatible selected reference. +- [x] Corrected the production scaled-compound branch. It previously rounded each scaled reference into + native pixels before blending; current libaom retains both `CONV_BUF_TYPE` values with + `COMPOUND_ROUND1_BITS` equal to seven and performs one final rounding after the selected compound blend. +- [x] Kept native-pixel and compound output in the existing `Av1ScaledInterPredictor` traversal with + semantic `NativeOperator` and `CompoundOperator` output contracts. The closed generic traversal shares + variable-phase arithmetic across byte and ushort sources, dispatches Vector512, Vector256, Vector128, + then scalar, and adds no per-block allocation or copy. +- [x] Added independent FeatureTestRunner oracles for native and no-round compound output across 8, 10, + and 12 bits, variable phases, all interpolation families, reduced kernels, vector tails, and destination + padding. A complete `Av1BlockDecoder.DecodeBlock()` regression covers scaled compound prediction across + all, AVX-512-disabled, AVX-disabled, and scalar configurations and proves the vector differs from an + incorrectly early-rounded blend. +- [x] Decoded the 2,195-byte layered payload with refreshed current libaom `aomdec`, using one thread, + row threading disabled, all layers selected, and raw 8-bit output. The 40x40 YUV444 base and 80x80 + YUV444 dependent frames total 24,000 samples with SHA-256 + `DD219E41B52C6C9343A92CD0A2D451DF57B73B25F10124811675B4CB2F8D666F`; both match their retained + native references with zero differing samples. +- [x] The production tests compare both native frames exactly, compare selected-layer and final RGBA + presentation through ImageSharp's established reference-output API, and repeat both paths with a + 1,024-byte constrained tracked allocator whose allocations have balanced exactly-once returns. +- [x] Renamed the two stale pinned-reference tests and their contract-derived PNGs together. Their Git blob + identifiers remain unchanged, and their SHA-256 values remain + `DC4C6DBE6BD92C5FCE1E3E23700AFA603EF04ED02EDD336213EBBA1E3BD84BA0` and + `678C5E5D4650EA6F0C590302E7DB9E3C6608851BC577453DA4A6837BDB4D3AF3`. +- [x] The focused Release checkpoint set passes 10/10 on net10.0 and 10/10 on net11.0, with zero failures + or skips. Scoped analyzer and whitespace verification pass for every changed C# file. Roslynk reports + zero compiler errors, `git diff --check` passes, and `.gitattributes` is unchanged. For every item: diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1ScaledInterPredictor.Operator.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1ScaledInterPredictor.Operator.cs index 9215ae9bd..b4fa5aac5 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1ScaledInterPredictor.Operator.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1ScaledInterPredictor.Operator.cs @@ -14,9 +14,9 @@ namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; internal static partial class Av1ScaledInterPredictor { /// - /// Defines variable-phase scaled prediction for one native sample storage type. + /// Defines source and convolution arithmetic shared by variable-phase scaled prediction. /// - private interface IAv1ScaledPredictionOperator + private interface IAv1ScaledArithmeticOperator { /// /// Loads one native source sample as a signed accumulator value. @@ -136,9 +136,30 @@ internal static partial class Av1ScaledInterPredictor Vector512 initial, out Vector512 result0, out Vector512 result1); + } + /// + /// Defines the output domain produced by variable-phase scaled prediction. + /// + private interface IAv1ScaledPredictionOperator + { /// - /// Clips and stores one completed prediction. + /// Gets the vertical convolution shift for the selected output domain. + /// + /// The horizontal convolution shift. + /// The vertical convolution shift. + public static abstract int GetVerticalRound(int horizontalRound); + + /// + /// Gets the bias removed after vertical convolution for the selected output domain. + /// + /// The biased intermediate precision. + /// The vertical convolution shift. + /// The bias removed before storing the result. + public static abstract int GetRoundOffset(int offsetBits, int verticalRound); + + /// + /// Stores one completed prediction in the selected output domain. /// /// The native sample storage type. /// The first destination sample. @@ -149,7 +170,7 @@ internal static partial class Av1ScaledInterPredictor where T : unmanaged; /// - /// Clips and stores eight completed predictions. + /// Stores eight completed predictions in the selected output domain. /// /// The native sample storage type. /// The first destination sample. @@ -161,7 +182,7 @@ internal static partial class Av1ScaledInterPredictor where T : unmanaged; /// - /// Clips and stores sixteen completed predictions. + /// Stores sixteen completed predictions in the selected output domain. /// /// The native sample storage type. /// The first destination sample. @@ -173,7 +194,7 @@ internal static partial class Av1ScaledInterPredictor where T : unmanaged; /// - /// Clips and stores thirty-two completed predictions. + /// Stores thirty-two completed predictions in the selected output domain. /// /// The native sample storage type. /// The first destination sample. @@ -186,10 +207,20 @@ internal static partial class Av1ScaledInterPredictor } /// - /// Implements variable-phase scaled prediction for scalar and SIMD lane groups. + /// Produces native-pixel scaled prediction for scalar and SIMD lane groups. /// - private readonly struct ScaledOperator : IAv1ScaledPredictionOperator + private readonly struct NativeOperator : IAv1ScaledArithmeticOperator, IAv1ScaledPredictionOperator { + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int GetVerticalRound(int horizontalRound) + => (2 * FilterBits) - horizontalRound; + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int GetRoundOffset(int offsetBits, int verticalRound) + => (1 << (offsetBits - verticalRound)) + (1 << (offsetBits - verticalRound - 1)); + /// [MethodImpl(MethodImplOptions.AggressiveInlining)] public static int Load(ref T source, int index) @@ -361,4 +392,68 @@ internal static partial class Av1ScaledInterPredictor .StoreUnsafe(ref Unsafe.As(ref destination), (nuint)index); } } + + /// + /// Produces no-round compound intermediates for scalar and SIMD lane groups. + /// + private readonly struct CompoundOperator : IAv1ScaledPredictionOperator + { + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int GetVerticalRound(int horizontalRound) + => Av1CompoundInterPredictor.CompoundRound1Bits; + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int GetRoundOffset(int offsetBits, int verticalRound) => 0; + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Store(ref T destination, int index, int value, int bitDepth) + where T : unmanaged + { + // Compound entry points close T as ushort. Their no-round values retain the positive convolution bias, + // so storing the normative unsigned intermediate needs neither pixel clipping nor a storage-type branch. + Unsafe.Add(ref Unsafe.As(ref destination), index) = (ushort)value; + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Store( + ref T destination, + int index, + Vector128 result0, + Vector128 result1, + int bitDepth) + where T : unmanaged + => Av1IntraPredictorBase.Narrow(result0, result1) + .AsUInt16() + .StoreUnsafe(ref Unsafe.As(ref destination), (nuint)index); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Store( + ref T destination, + int index, + Vector256 result0, + Vector256 result1, + int bitDepth) + where T : unmanaged + => Av1IntraPredictorBase.Narrow(result0, result1) + .AsUInt16() + .StoreUnsafe(ref Unsafe.As(ref destination), (nuint)index); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Store( + ref T destination, + int index, + Vector512 result0, + Vector512 result1, + int bitDepth) + where T : unmanaged + => Av1IntraPredictorBase.Narrow(result0, result1) + .AsUInt16() + .StoreUnsafe(ref Unsafe.As(ref destination), (nuint)index); + } } diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1ScaledInterPredictor.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1ScaledInterPredictor.cs index 0c6142f04..4a8a40d2a 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1ScaledInterPredictor.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1ScaledInterPredictor.cs @@ -47,7 +47,7 @@ internal static partial class Av1ScaledInterPredictor int verticalPhase, int verticalStep, Span scratch) - => DispatchScaled( + => DispatchScaled( source, sourceStride, sourceOrigin, @@ -83,7 +83,78 @@ internal static partial class Av1ScaledInterPredictor int verticalStep, int bitDepth, Span scratch) - => DispatchScaled( + => DispatchScaled( + source, + sourceStride, + sourceOrigin, + destination, + destinationStride, + width, + height, + horizontalFilter, + verticalFilter, + horizontalPhase, + horizontalStep, + verticalPhase, + verticalStep, + bitDepth, + scratch); + + /// + /// Reconstructs an 8-bit scaled predictor into the no-round compound intermediate domain. + /// + public static void PredictScaledCompound( + ReadOnlySpan source, + int sourceStride, + int sourceOrigin, + Span destination, + int destinationStride, + int width, + int height, + Av1InterpolationFilter horizontalFilter, + Av1InterpolationFilter verticalFilter, + int horizontalPhase, + int horizontalStep, + int verticalPhase, + int verticalStep, + Span scratch) + => DispatchScaled( + source, + sourceStride, + sourceOrigin, + destination, + destinationStride, + width, + height, + horizontalFilter, + verticalFilter, + horizontalPhase, + horizontalStep, + verticalPhase, + verticalStep, + 8, + scratch); + + /// + /// Reconstructs an 8-, 10-, or 12-bit scaled predictor into the no-round compound intermediate domain. + /// + public static void PredictScaledCompound( + ReadOnlySpan source, + int sourceStride, + int sourceOrigin, + Span destination, + int destinationStride, + int width, + int height, + Av1InterpolationFilter horizontalFilter, + Av1InterpolationFilter verticalFilter, + int horizontalPhase, + int horizontalStep, + int verticalPhase, + int verticalStep, + int bitDepth, + Span scratch) + => DispatchScaled( source, sourceStride, sourceOrigin, @@ -103,11 +174,11 @@ internal static partial class Av1ScaledInterPredictor /// /// Selects the horizontal filter family for scaled prediction. /// - private static void DispatchScaled( - ReadOnlySpan source, + private static void DispatchScaled( + ReadOnlySpan source, int sourceStride, int sourceOrigin, - Span destination, + Span destination, int destinationStride, int width, int height, @@ -119,13 +190,14 @@ internal static partial class Av1ScaledInterPredictor int verticalStep, int bitDepth, Span scratch) - where T : unmanaged - where TSample : struct, IAv1ScaledPredictionOperator + where TSource : unmanaged + where TDestination : unmanaged + where TOperator : struct, IAv1ScaledPredictionOperator { switch (horizontalFilter) { case Av1InterpolationFilter.Regular: - DispatchScaledVertical( + DispatchScaledVertical( source, sourceStride, sourceOrigin, @@ -143,7 +215,7 @@ internal static partial class Av1ScaledInterPredictor break; case Av1InterpolationFilter.Smooth: - DispatchScaledVertical( + DispatchScaledVertical( source, sourceStride, sourceOrigin, @@ -161,7 +233,7 @@ internal static partial class Av1ScaledInterPredictor break; case Av1InterpolationFilter.Sharp: - DispatchScaledVertical( + DispatchScaledVertical( source, sourceStride, sourceOrigin, @@ -179,7 +251,7 @@ internal static partial class Av1ScaledInterPredictor break; default: - DispatchScaledVertical( + DispatchScaledVertical( source, sourceStride, sourceOrigin, @@ -202,11 +274,11 @@ internal static partial class Av1ScaledInterPredictor /// /// Selects the vertical filter family for a closed horizontal scaled-prediction operator. /// - private static void DispatchScaledVertical( - ReadOnlySpan source, + private static void DispatchScaledVertical( + ReadOnlySpan source, int sourceStride, int sourceOrigin, - Span destination, + Span destination, int destinationStride, int width, int height, @@ -217,14 +289,15 @@ internal static partial class Av1ScaledInterPredictor int verticalStep, int bitDepth, Span scratch) - where T : unmanaged - where TSample : struct, IAv1ScaledPredictionOperator + where TSource : unmanaged + where TDestination : unmanaged + where TOperator : struct, IAv1ScaledPredictionOperator where THorizontal : struct, IAv1InterPredictorOperator { switch (verticalFilter) { case Av1InterpolationFilter.Regular: - PredictScaled( + PredictScaled( source, sourceStride, sourceOrigin, @@ -241,7 +314,7 @@ internal static partial class Av1ScaledInterPredictor break; case Av1InterpolationFilter.Smooth: - PredictScaled( + PredictScaled( source, sourceStride, sourceOrigin, @@ -258,7 +331,7 @@ internal static partial class Av1ScaledInterPredictor break; case Av1InterpolationFilter.Sharp: - PredictScaled( + PredictScaled( source, sourceStride, sourceOrigin, @@ -275,7 +348,7 @@ internal static partial class Av1ScaledInterPredictor break; default: - PredictScaled( + PredictScaled( source, sourceStride, sourceOrigin, @@ -297,11 +370,11 @@ internal static partial class Av1ScaledInterPredictor /// /// Applies variable-phase horizontal filtering followed by variable-phase vertical filtering. /// - private static void PredictScaled( - ReadOnlySpan source, + private static void PredictScaled( + ReadOnlySpan source, int sourceStride, int sourceOrigin, - Span destination, + Span destination, int destinationStride, int width, int height, @@ -311,13 +384,14 @@ internal static partial class Av1ScaledInterPredictor int verticalStep, int bitDepth, Span scratch) - where T : unmanaged - where TSample : struct, IAv1ScaledPredictionOperator + where TSource : unmanaged + where TDestination : unmanaged + where TOperator : struct, IAv1ScaledPredictionOperator where THorizontal : struct, IAv1InterPredictorOperator where TVertical : struct, IAv1InterPredictorOperator { - ref T sourceBase = ref Unsafe.Add(ref MemoryMarshal.GetReference(source), sourceOrigin); - ref T destinationBase = ref MemoryMarshal.GetReference(destination); + ref TSource sourceBase = ref Unsafe.Add(ref MemoryMarshal.GetReference(source), sourceOrigin); + ref TDestination destinationBase = ref MemoryMarshal.GetReference(destination); ref short scratchBase = ref MemoryMarshal.GetReference(scratch); int scratchStride = Math.Max(width, Vector128.Count); int intermediateHeight = ((((height - 1) * verticalStep) + verticalPhase) >> Av1ReferenceScale.SubpixelBits) + FilterCoefficientCount; @@ -331,7 +405,7 @@ internal static partial class Av1ScaledInterPredictor // vectors gather those independent positions into one multiply-accumulate chain without allocating an index map. for (int row = 0; row < intermediateHeight; row++) { - ref T sourceRow = ref Unsafe.Add(ref sourceBase, (row - 3) * sourceStride); + ref TSource sourceRow = ref Unsafe.Add(ref sourceBase, (row - 3) * sourceStride); ref short scratchRow = ref Unsafe.Add(ref scratchBase, row * scratchStride); int column = 0; @@ -340,7 +414,7 @@ internal static partial class Av1ScaledInterPredictor int oneVectorFromEnd = width - Vector512.Count; for (; column <= oneVectorFromEnd; column += Vector512.Count) { - Vector512 result = FilterScaledHorizontalVector512( + Vector512 result = FilterScaledHorizontalVector512( ref sourceRow, horizontalPhase, horizontalStep, @@ -360,7 +434,7 @@ internal static partial class Av1ScaledInterPredictor int oneVectorFromEnd = width - Vector256.Count; for (; column <= oneVectorFromEnd; column += Vector256.Count) { - Vector256 result = FilterScaledHorizontalVector256( + Vector256 result = FilterScaledHorizontalVector256( ref sourceRow, horizontalPhase, horizontalStep, @@ -379,7 +453,7 @@ internal static partial class Av1ScaledInterPredictor { for (; column <= width - Vector128.Count; column += Vector128.Count) { - Vector128 result = FilterScaledHorizontalVector128( + Vector128 result = FilterScaledHorizontalVector128( ref sourceRow, horizontalPhase, horizontalStep, @@ -405,17 +479,17 @@ internal static partial class Av1ScaledInterPredictor int sum = horizontalBias; for (int tap = 0; tap < FilterCoefficientCount; tap++) { - sum = TSample.MultiplyAdd(sum, TSample.Load(ref sourceRow, sourceColumn + tap), coefficients[tap]); + sum = NativeOperator.MultiplyAdd(sum, NativeOperator.Load(ref sourceRow, sourceColumn + tap), coefficients[tap]); } Unsafe.Add(ref scratchRow, column) = (short)RoundPowerOfTwo(sum, round0); } } - int round1 = (2 * FilterBits) - round0; + int round1 = TOperator.GetVerticalRound(round0); int offsetBits = bitDepth + (2 * FilterBits) - round0; int verticalBias = 1 << offsetBits; - int roundOffset = (1 << (offsetBits - round1)) + (1 << (offsetBits - round1 - 1)); + int roundOffset = TOperator.GetRoundOffset(offsetBits, round1); for (int row = 0; row < height; row++) { int position = verticalPhase + (row * verticalStep); @@ -426,7 +500,7 @@ internal static partial class Av1ScaledInterPredictor ref short scratchRow = ref Unsafe.Add(ref scratchBase, sourceRowIndex * scratchStride); ref short coefficientBase = ref MemoryMarshal.GetReference(coefficients); - ref T destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); + ref TDestination destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); int column = 0; if (Vector512.IsHardwareAccelerated) @@ -436,7 +510,7 @@ internal static partial class Av1ScaledInterPredictor int oneVectorFromEnd = width - (Vector512.Count * 2); for (; column <= oneVectorFromEnd; column += Vector512.Count * 2) { - TSample.Convolve( + NativeOperator.Convolve( ref scratchRow, scratchStride, (nuint)column, @@ -448,7 +522,7 @@ internal static partial class Av1ScaledInterPredictor result0 = RoundPowerOfTwo(result0, round1) - offset; result1 = RoundPowerOfTwo(result1, round1) - offset; - TSample.Store(ref destinationRow, column, result0, result1, bitDepth); + TOperator.Store(ref destinationRow, column, result0, result1, bitDepth); } } @@ -459,7 +533,7 @@ internal static partial class Av1ScaledInterPredictor int oneVectorFromEnd = width - (Vector256.Count * 2); for (; column <= oneVectorFromEnd; column += Vector256.Count * 2) { - TSample.Convolve( + NativeOperator.Convolve( ref scratchRow, scratchStride, (nuint)column, @@ -471,7 +545,7 @@ internal static partial class Av1ScaledInterPredictor result0 = RoundPowerOfTwo(result0, round1) - offset; result1 = RoundPowerOfTwo(result1, round1) - offset; - TSample.Store(ref destinationRow, column, result0, result1, bitDepth); + TOperator.Store(ref destinationRow, column, result0, result1, bitDepth); } } @@ -482,7 +556,7 @@ internal static partial class Av1ScaledInterPredictor int oneVectorFromEnd = width - (Vector128.Count * 2); for (; column <= oneVectorFromEnd; column += Vector128.Count * 2) { - TSample.Convolve( + NativeOperator.Convolve( ref scratchRow, scratchStride, (nuint)column, @@ -494,19 +568,19 @@ internal static partial class Av1ScaledInterPredictor result0 = RoundPowerOfTwo(result0, round1) - offset; result1 = RoundPowerOfTwo(result1, round1) - offset; - TSample.Store(ref destinationRow, column, result0, result1, bitDepth); + TOperator.Store(ref destinationRow, column, result0, result1, bitDepth); } } for (; column < width; column++) { - int sum = verticalBias + TSample.Convolve( + int sum = verticalBias + NativeOperator.Convolve( ref Unsafe.Add(ref scratchRow, column), scratchStride, ref coefficientBase, FilterCoefficientCount); - TSample.Store( + TOperator.Store( ref destinationRow, column, RoundPowerOfTwo(sum, round1) - roundOffset, @@ -518,7 +592,7 @@ internal static partial class Av1ScaledInterPredictor /// /// Filters four independently positioned horizontal samples through a closed scaled-prediction operator. /// - private static Vector128 FilterScaledHorizontalVector128( + private static Vector128 FilterScaledHorizontalVector128( ref T source, int phase, int step, @@ -527,15 +601,14 @@ internal static partial class Av1ScaledInterPredictor int bias, int round) where T : unmanaged - where TOperator : struct, IAv1ScaledPredictionOperator where TFilter : struct, IAv1InterPredictorOperator { Vector128 result = Vector128.Create(bias); for (int tap = 0; tap < FilterCoefficientCount; tap++) { - result = TOperator.MultiplyAdd( + result = NativeOperator.MultiplyAdd( result, - LoadScaledSamplesVector128(ref source, phase, step, column, tap), + LoadScaledSamplesVector128(ref source, phase, step, column, tap), LoadScaledCoefficientsVector128(phase, step, column, tap, useReducedFilter)); } @@ -545,7 +618,7 @@ internal static partial class Av1ScaledInterPredictor /// /// Filters eight independently positioned horizontal samples through a closed scaled-prediction operator. /// - private static Vector256 FilterScaledHorizontalVector256( + private static Vector256 FilterScaledHorizontalVector256( ref T source, int phase, int step, @@ -554,21 +627,20 @@ internal static partial class Av1ScaledInterPredictor int bias, int round) where T : unmanaged - where TOperator : struct, IAv1ScaledPredictionOperator where TFilter : struct, IAv1InterPredictorOperator { Vector256 result = Vector256.Create(bias); for (int tap = 0; tap < FilterCoefficientCount; tap++) { Vector256 samples = Vector256.Create( - LoadScaledSamplesVector128(ref source, phase, step, column, tap), - LoadScaledSamplesVector128(ref source, phase, step, column + Vector128.Count, tap)); + LoadScaledSamplesVector128(ref source, phase, step, column, tap), + LoadScaledSamplesVector128(ref source, phase, step, column + Vector128.Count, tap)); Vector256 coefficients = Vector256.Create( LoadScaledCoefficientsVector128(phase, step, column, tap, useReducedFilter), LoadScaledCoefficientsVector128(phase, step, column + Vector128.Count, tap, useReducedFilter)); - result = TOperator.MultiplyAdd(result, samples, coefficients); + result = NativeOperator.MultiplyAdd(result, samples, coefficients); } return RoundPowerOfTwo(result, round); @@ -577,7 +649,7 @@ internal static partial class Av1ScaledInterPredictor /// /// Filters sixteen independently positioned horizontal samples through a closed scaled-prediction operator. /// - private static Vector512 FilterScaledHorizontalVector512( + private static Vector512 FilterScaledHorizontalVector512( ref T source, int phase, int step, @@ -586,19 +658,18 @@ internal static partial class Av1ScaledInterPredictor int bias, int round) where T : unmanaged - where TOperator : struct, IAv1ScaledPredictionOperator where TFilter : struct, IAv1InterPredictorOperator { Vector512 result = Vector512.Create(bias); for (int tap = 0; tap < FilterCoefficientCount; tap++) { Vector256 sampleLower = Vector256.Create( - LoadScaledSamplesVector128(ref source, phase, step, column, tap), - LoadScaledSamplesVector128(ref source, phase, step, column + Vector128.Count, tap)); + LoadScaledSamplesVector128(ref source, phase, step, column, tap), + LoadScaledSamplesVector128(ref source, phase, step, column + Vector128.Count, tap)); Vector256 sampleUpper = Vector256.Create( - LoadScaledSamplesVector128(ref source, phase, step, column + Vector256.Count, tap), - LoadScaledSamplesVector128(ref source, phase, step, column + Vector256.Count + Vector128.Count, tap)); + LoadScaledSamplesVector128(ref source, phase, step, column + Vector256.Count, tap), + LoadScaledSamplesVector128(ref source, phase, step, column + Vector256.Count + Vector128.Count, tap)); Vector256 coefficientLower = Vector256.Create( LoadScaledCoefficientsVector128(phase, step, column, tap, useReducedFilter), @@ -608,7 +679,7 @@ internal static partial class Av1ScaledInterPredictor LoadScaledCoefficientsVector128(phase, step, column + Vector256.Count, tap, useReducedFilter), LoadScaledCoefficientsVector128(phase, step, column + Vector256.Count + Vector128.Count, tap, useReducedFilter)); - result = TOperator.MultiplyAdd( + result = NativeOperator.MultiplyAdd( result, Vector512.Create(sampleLower, sampleUpper), Vector512.Create(coefficientLower, coefficientUpper)); @@ -620,19 +691,18 @@ internal static partial class Av1ScaledInterPredictor /// /// Gathers four variable-position source samples for one horizontal filter tap. /// - private static Vector128 LoadScaledSamplesVector128( + private static Vector128 LoadScaledSamplesVector128( ref T source, int phase, int step, int column, int tap) where T : unmanaged - where TOperator : struct, IAv1ScaledPredictionOperator => Vector128.Create( - LoadScaledSample(ref source, phase, step, column, tap), - LoadScaledSample(ref source, phase, step, column + 1, tap), - LoadScaledSample(ref source, phase, step, column + 2, tap), - LoadScaledSample(ref source, phase, step, column + 3, tap)); + LoadScaledSample(ref source, phase, step, column, tap), + LoadScaledSample(ref source, phase, step, column + 1, tap), + LoadScaledSample(ref source, phase, step, column + 2, tap), + LoadScaledSample(ref source, phase, step, column + 3, tap)); /// /// Gathers four variable-phase coefficients for one horizontal filter tap. @@ -653,13 +723,12 @@ internal static partial class Av1ScaledInterPredictor /// /// Loads one variable-position source sample for a horizontal filter tap. /// - private static int LoadScaledSample(ref T source, int phase, int step, int column, int tap) + private static int LoadScaledSample(ref T source, int phase, int step, int column, int tap) where T : unmanaged - where TOperator : struct, IAv1ScaledPredictionOperator { int position = phase + (column * step); int sourceColumn = (position >> Av1ReferenceScale.SubpixelBits) - 3; - return TOperator.Load(ref source, sourceColumn + tap); + return NativeOperator.Load(ref source, sourceColumn + tap); } /// diff --git a/src/ImageSharp/Formats/Heif/Av1/Transform/Av1BlockDecoder.cs b/src/ImageSharp/Formats/Heif/Av1/Transform/Av1BlockDecoder.cs index 0f9c9338a..95ed03966 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Transform/Av1BlockDecoder.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Transform/Av1BlockDecoder.cs @@ -362,14 +362,9 @@ internal sealed class Av1BlockDecoder : IDisposable predictionScratch); int referenceCount = usesSub8x8ChromaPrediction ? 0 : isCompound ? 2 : 1; - bool hasScaledCompoundReference = isCompound && - (referenceFrameBuffer!.Width != this.frameHeader.FrameSize.FrameWidth || - referenceFrameBuffer.Height != this.frameHeader.FrameSize.FrameHeight || - secondaryReferenceFrameBuffer!.Width != this.frameHeader.FrameSize.FrameWidth || - secondaryReferenceFrameBuffer.Height != this.frameHeader.FrameSize.FrameHeight); - - // Compound convolution is combined before its final rounding step. Scaled references and high-bit-depth - // warped/global models have separate kernels and remain with their owning later prediction checkpoints. + + // Compound convolution is combined before its final rounding step. Scaled and ordinary translational + // predictors share that no-round domain; high-bit-depth warped/global models retain their own kernels. bool useHighBitDepthCompoundIntermediates = highBitDepth && modeInfo.CompoundType is @@ -382,7 +377,6 @@ internal sealed class Av1BlockDecoder : IDisposable bool useCompoundIntermediates = isCompound && - !hasScaledCompoundReference && (!highBitDepth || useHighBitDepthCompoundIntermediates); for (int referenceIndex = 0; referenceIndex < referenceCount; referenceIndex++) @@ -535,7 +529,14 @@ internal sealed class Av1BlockDecoder : IDisposable { Span scaledDestination = default; Span scaledHighBitDepthDestination = default; - if (highBitDepth) + Span scaledCompoundDestination = default; + if (useCompoundIntermediates) + { + scaledCompoundDestination = referenceIndex == 0 + ? firstCompoundPrediction + : highBitDepthSecondPrediction; + } + else if (highBitDepth) { scaledHighBitDepthDestination = referenceIndex == 0 ? MemoryMarshal.Cast(highBitDepthBlockReconstructionBuffer[reconstructionStride..]) @@ -561,6 +562,7 @@ internal sealed class Av1BlockDecoder : IDisposable modeInfo.InterpolationFilters[0], scaledDestination, scaledHighBitDepthDestination, + scaledCompoundDestination, destinationStride, predictionScratch); @@ -1493,6 +1495,7 @@ internal sealed class Av1BlockDecoder : IDisposable candidate.InterpolationFilters[0], scaledDestination, scaledHighBitDepthDestination, + default, reconstructionStride, predictionScratch); } @@ -1623,6 +1626,7 @@ internal sealed class Av1BlockDecoder : IDisposable Av1InterpolationFilter verticalFilter, Span destination, Span highBitDepthDestination, + Span compoundDestination, int destinationStride, Span predictionScratch) { @@ -1669,22 +1673,44 @@ internal sealed class Av1BlockDecoder : IDisposable sourceOrigin.X + (sourceColumnQ10 >> Av1ReferenceScale.SubpixelBits); - Av1ScaledInterPredictor.PredictScaled( - source, - sourceStride, - sourceIndex, - highBitDepthDestination, - destinationStride, - predictionWidth, - predictionHeight, - horizontalFilter, - verticalFilter, - horizontalPhase, - scale.HorizontalStep, - verticalPhase, - scale.VerticalStep, - this.frameBuffer.BitDepth.GetBitCount(), - predictionScratch); + if (compoundDestination.IsEmpty) + { + Av1ScaledInterPredictor.PredictScaled( + source, + sourceStride, + sourceIndex, + highBitDepthDestination, + destinationStride, + predictionWidth, + predictionHeight, + horizontalFilter, + verticalFilter, + horizontalPhase, + scale.HorizontalStep, + verticalPhase, + scale.VerticalStep, + this.frameBuffer.BitDepth.GetBitCount(), + predictionScratch); + } + else + { + Av1ScaledInterPredictor.PredictScaledCompound( + source, + sourceStride, + sourceIndex, + compoundDestination, + destinationStride, + predictionWidth, + predictionHeight, + horizontalFilter, + verticalFilter, + horizontalPhase, + scale.HorizontalStep, + verticalPhase, + scale.VerticalStep, + this.frameBuffer.BitDepth.GetBitCount(), + predictionScratch); + } } else { @@ -1700,21 +1726,42 @@ internal sealed class Av1BlockDecoder : IDisposable sourceOrigin.X + (sourceColumnQ10 >> Av1ReferenceScale.SubpixelBits); - Av1ScaledInterPredictor.PredictScaled( - source, - sourceStride, - sourceIndex, - destination, - destinationStride, - predictionWidth, - predictionHeight, - horizontalFilter, - verticalFilter, - horizontalPhase, - scale.HorizontalStep, - verticalPhase, - scale.VerticalStep, - predictionScratch); + if (compoundDestination.IsEmpty) + { + Av1ScaledInterPredictor.PredictScaled( + source, + sourceStride, + sourceIndex, + destination, + destinationStride, + predictionWidth, + predictionHeight, + horizontalFilter, + verticalFilter, + horizontalPhase, + scale.HorizontalStep, + verticalPhase, + scale.VerticalStep, + predictionScratch); + } + else + { + Av1ScaledInterPredictor.PredictScaledCompound( + source, + sourceStride, + sourceIndex, + compoundDestination, + destinationStride, + predictionWidth, + predictionHeight, + horizontalFilter, + verticalFilter, + horizontalPhase, + scale.HorizontalStep, + verticalPhase, + scale.VerticalStep, + predictionScratch); + } } } @@ -1949,6 +1996,7 @@ internal sealed class Av1BlockDecoder : IDisposable neighbor.InterpolationFilters[0], destination, highBitDepthDestination, + default, predictionWidth, predictionScratch); diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundBlockDecoderTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundBlockDecoderTests.cs index 868af1d6a..6c87e95ed 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundBlockDecoderTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundBlockDecoderTests.cs @@ -239,6 +239,15 @@ public class Av1CompoundBlockDecoderTests } } + /// + /// Verifies that scaled predictors retain their no-round precision until compound averaging. + /// + [Fact] + public void DecodeBlockReconstructsScaledCompoundPrediction() + => FeatureTestRunner.RunWithHwIntrinsicsFeature( + ValidateScaledCompoundPrediction, + CompoundPredictionConfigurations); + /// /// Verifies selectable compound reconstruction through the production block branch at every supported bit depth. /// @@ -658,6 +667,262 @@ public class Av1CompoundBlockDecoderTests return modeInfo; } + /// + /// Reconstructs the scaled compound regression at every supported source precision. + /// + private static void ValidateScaledCompoundPrediction() + { + foreach (Av1BitDepth bitDepth in new[] { Av1BitDepth.EightBit, Av1BitDepth.TenBit, Av1BitDepth.TwelveBit }) + { + ValidateScaledCompoundPredictionAtBitDepth(bitDepth); + } + } + + /// + /// Reconstructs one scaled compound block and compares the production branch with the no-round pipeline. + /// + /// The native sample depth. + private static void ValidateScaledCompoundPredictionAtBitDepth(Av1BitDepth bitDepth) + { + const int currentSize = 8; + const int referenceSize = 16; + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(bitDepth, referenceSize); + ObuFrameHeader frameHeader = CreateFrameHeader(currentSize); + frameHeader.GetReferenceFrameIndices()[0] = 0; + frameHeader.GetReferenceFrameIndices()[1] = 1; + + using Av1ReferenceFrameStore referenceFrames = new(); + Assert.True(referenceFrames.Commit( + 1, + CreateScaledPatternReferenceFrame(sequenceHeader), + showFrame: false)); + + Assert.True(referenceFrames.Commit( + 2, + CreateScaledPatternReferenceFrame( + sequenceHeader, + horizontalScale: 3, + verticalScale: 7, + nonlinearScale: 3), + showFrame: false)); + + Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block8x8, Point.Empty) + { + Skip = true, + YMode = Av1PredictionMode.NearestNearestMotionVector, + CompoundIndex = true, + CompoundType = Av1CompoundType.Average, + }; + + modeInfo.ReferenceFrames[0] = Av1ReferenceFrameType.Last; + modeInfo.ReferenceFrames[1] = Av1ReferenceFrameType.Last2; + modeInfo.InterpolationFilters.Fill(Av1InterpolationFilter.Bilinear); + modeInfo.SetTransformUnitCount(Av1PlaneType.Y, 1); + + Av1ReferenceScale scale = new(referenceSize, referenceSize, currentSize, currentSize); + int sourceColumnQ10 = scale.ScaleHorizontal(0) + Av1ReferenceScale.ExtraOffset; + int sourceRowQ10 = scale.ScaleVertical(0) + Av1ReferenceScale.ExtraOffset; + int horizontalPhase = sourceColumnQ10 & Av1ReferenceScale.SubpixelMask; + int verticalPhase = sourceRowQ10 & Av1ReferenceScale.SubpixelMask; + ushort[] firstIntermediate = new ushort[currentSize * currentSize]; + ushort[] secondIntermediate = new ushort[currentSize * currentSize]; + byte[] firstRounded8 = new byte[currentSize * currentSize]; + byte[] secondRounded8 = new byte[currentSize * currentSize]; + ushort[] firstRoundedHigh = new ushort[currentSize * currentSize]; + ushort[] secondRoundedHigh = new ushort[currentSize * currentSize]; + short[] predictionScratch = new short[ + Av1ScaledInterPredictor.GetScaledScratchLength( + currentSize, + currentSize, + verticalPhase, + scale.VerticalStep)]; + + for (int referenceIndex = 0; referenceIndex < 2; referenceIndex++) + { + Av1FrameBuffer reference = referenceFrames.Resolve(referenceIndex)!.FrameBuffer; + Span intermediate = referenceIndex == 0 ? firstIntermediate : secondIntermediate; + int sourceIndex; + if (bitDepth == Av1BitDepth.EightBit) + { + Span source = reference.GetPaddedPlaneSpan( + Av1Plane.Y, + 0, + 0, + out int sourceStride, + out Point sourceOrigin); + + sourceIndex = + ((sourceOrigin.Y + (sourceRowQ10 >> Av1ReferenceScale.SubpixelBits)) * sourceStride) + + sourceOrigin.X + + (sourceColumnQ10 >> Av1ReferenceScale.SubpixelBits); + + Av1ScaledInterPredictor.PredictScaledCompound( + source, + sourceStride, + sourceIndex, + intermediate, + currentSize, + currentSize, + currentSize, + Av1InterpolationFilter.Bilinear, + Av1InterpolationFilter.Bilinear, + horizontalPhase, + scale.HorizontalStep, + verticalPhase, + scale.VerticalStep, + predictionScratch); + + Av1ScaledInterPredictor.PredictScaled( + source, + sourceStride, + sourceIndex, + referenceIndex == 0 ? firstRounded8 : secondRounded8, + currentSize, + currentSize, + currentSize, + Av1InterpolationFilter.Bilinear, + Av1InterpolationFilter.Bilinear, + horizontalPhase, + scale.HorizontalStep, + verticalPhase, + scale.VerticalStep, + predictionScratch); + } + else + { + Span source = reference.GetPaddedPlaneSpan16( + Av1Plane.Y, + 0, + 0, + out int sourceStride, + out Point sourceOrigin); + + sourceIndex = + ((sourceOrigin.Y + (sourceRowQ10 >> Av1ReferenceScale.SubpixelBits)) * sourceStride) + + sourceOrigin.X + + (sourceColumnQ10 >> Av1ReferenceScale.SubpixelBits); + + Av1ScaledInterPredictor.PredictScaledCompound( + source, + sourceStride, + sourceIndex, + intermediate, + currentSize, + currentSize, + currentSize, + Av1InterpolationFilter.Bilinear, + Av1InterpolationFilter.Bilinear, + horizontalPhase, + scale.HorizontalStep, + verticalPhase, + scale.VerticalStep, + bitDepth.GetBitCount(), + predictionScratch); + + Av1ScaledInterPredictor.PredictScaled( + source, + sourceStride, + sourceIndex, + referenceIndex == 0 ? firstRoundedHigh : secondRoundedHigh, + currentSize, + currentSize, + currentSize, + Av1InterpolationFilter.Bilinear, + Av1InterpolationFilter.Bilinear, + horizontalPhase, + scale.HorizontalStep, + verticalPhase, + scale.VerticalStep, + bitDepth.GetBitCount(), + predictionScratch); + } + } + + using Av1FrameBuffer frameBuffer = new( + Configuration.Default, + sequenceHeader, + Av1ColorFormat.Yuv400, + false); + + frameBuffer.Width = currentSize; + frameBuffer.Height = currentSize; + using Av1FrameInfo frameInfo = new(sequenceHeader); + Av1SuperblockInfo superblockInfo = frameInfo.GetSuperblock(Point.Empty); + superblockInfo.GetTransformInfoY()[0] = new Av1TransformInfo(Av1TransformSize.Size8x8, 0, 0); + Av1LoopFilterContext loopFilterContext = new(sequenceHeader); + Av1InverseQuantizer inverseQuantizer = new(sequenceHeader, frameHeader); + using Av1BlockDecoder decoder = new( + sequenceHeader, + frameHeader, + frameBuffer, + loopFilterContext, + inverseQuantizer, + referenceFrames); + + decoder.UpdateSuperblock(superblockInfo); + decoder.DecodeBlock( + modeInfo, + Point.Empty, + Av1BlockSize.Block8x8, + superblockInfo, + new Av1TileInfo(0, 0, frameHeader)); + + if (bitDepth == Av1BitDepth.EightBit) + { + byte[] expected = new byte[currentSize * currentSize]; + Av1CompoundIntermediateAveragePredictor.AverageIntermediate( + expected, + currentSize, + firstIntermediate, + currentSize, + secondIntermediate, + currentSize, + currentSize, + currentSize, + bitDepth.GetBitCount()); + + byte[] prematurelyRounded = new byte[currentSize * currentSize]; + for (int index = 0; index < prematurelyRounded.Length; index++) + { + prematurelyRounded[index] = (byte)((firstRounded8[index] + secondRounded8[index] + 1) >> 1); + } + + Assert.False(expected.AsSpan().SequenceEqual(prematurelyRounded)); + for (int row = 0; row < currentSize; row++) + { + Span actual = frameBuffer.DeriveBlockPointer(Av1Plane.Y, 0, 0).DangerousGetRowSpan(row); + Assert.Equal(expected.AsSpan(row * currentSize, currentSize), actual[..currentSize]); + } + } + else + { + ushort[] expected = new ushort[currentSize * currentSize]; + Av1CompoundIntermediateAveragePredictor.AverageIntermediate( + expected, + currentSize, + firstIntermediate, + currentSize, + secondIntermediate, + currentSize, + currentSize, + currentSize, + bitDepth.GetBitCount()); + + ushort[] prematurelyRounded = new ushort[currentSize * currentSize]; + for (int index = 0; index < prematurelyRounded.Length; index++) + { + prematurelyRounded[index] = (ushort)((firstRoundedHigh[index] + secondRoundedHigh[index] + 1) >> 1); + } + + Assert.False(expected.AsSpan().SequenceEqual(prematurelyRounded)); + for (int row = 0; row < currentSize; row++) + { + Span actual = frameBuffer.GetHighBitDepthRowSpan(Av1Plane.Y, row, 0, 0); + Assert.Equal(expected.AsSpan(row * currentSize, currentSize), actual[..currentSize]); + } + } + } + /// /// Reconstructs a compound global-warp block at every supported native sample depth. /// @@ -1273,9 +1538,17 @@ public class Av1CompoundBlockDecoderTests } /// - /// Creates a 16x16 retained frame whose linear pattern has an exact half-sample bilinear result. + /// Creates a 16x16 retained frame whose deterministic pattern exposes scaled bilinear precision. /// - private static Av1ReferenceFrame CreateScaledPatternReferenceFrame(ObuSequenceHeader sequenceHeader) + /// The sequence dimensions and sample precision. + /// The horizontal linear contribution. + /// The vertical linear contribution. + /// The contribution that makes neighboring interpolation errors differ. + private static Av1ReferenceFrame CreateScaledPatternReferenceFrame( + ObuSequenceHeader sequenceHeader, + int horizontalScale = 2, + int verticalScale = 8, + int nonlinearScale = 0) { Av1FrameBuffer frameBuffer = new( Configuration.Default, @@ -1292,7 +1565,10 @@ public class Av1CompoundBlockDecoderTests Span samples = frameBuffer.DeriveBlockPointer(Av1Plane.Y, 0, 0).DangerousGetRowSpan(row); for (int column = 0; column < 16; column++) { - samples[column] = (byte)((column * 2) + (row * 8)); + samples[column] = (byte)( + (column * horizontalScale) + + (row * verticalScale) + + (((column * row) & 7) * nonlinearScale)); } } else @@ -1300,7 +1576,10 @@ public class Av1CompoundBlockDecoderTests Span samples = frameBuffer.GetHighBitDepthRowSpan(Av1Plane.Y, row, 0, 0); for (int column = 0; column < 16; column++) { - samples[column] = (ushort)((column * 2) + (row * 8)); + samples[column] = (ushort)( + (column * horizontalScale) + + (row * verticalScale) + + (((column * row) & 7) * nonlinearScale)); } } } diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReconstructionConformanceTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReconstructionConformanceTests.cs index 9eada06d4..689346079 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReconstructionConformanceTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReconstructionConformanceTests.cs @@ -717,12 +717,12 @@ public class Av1ReconstructionConformanceTests /// /// Verifies that an essential lsel property returns the selected base spatial layer rather than the final - /// progressive layer, with exact pinned-libaom native planes and pinned-libavif presentation. + /// progressive layer, with exact current-libaom native planes and the retained exact presentation reference. /// /// The selected-layer AVIF input and matching reference-output naming context. [Theory] [WithFile(TestImages.Heif.Av1ScaledReferenceSelectedLayerAvif, PixelTypes.Rgba32)] - public void DecodeSelectedProgressiveSpatialLayerMatchesPinnedReferences( + public void DecodeSelectedProgressiveSpatialLayerMatchesCurrentLibaomReferences( TestImageProvider provider) => FeatureTestRunner.RunWithHwIntrinsicsFeature( ValidateSelectedProgressiveSpatialLayerWithDefaultConfiguration, @@ -753,12 +753,12 @@ public class Av1ReconstructionConformanceTests /// /// Verifies an independently encoded 40x40 retained layer scaled into an 80x80 dependent layer against exact - /// pinned-libaom native planes and pinned-libavif presentation. + /// current-libaom native planes and the retained exact presentation reference. /// /// The AVIF input and matching reference-output naming context. [Theory] [WithFile(TestImages.Heif.Av1ScaledReferenceAvif, PixelTypes.Rgba32)] - public void DecodeScaledReferenceMatchesPinnedReferences(TestImageProvider provider) + public void DecodeScaledReferenceMatchesCurrentLibaomReferences(TestImageProvider provider) => FeatureTestRunner.RunWithHwIntrinsicsFeature( ValidateScaledReferenceFixtureWithDefaultConfiguration, ReconstructionConfigurations, diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ScaledInterPredictorTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ScaledInterPredictorTests.cs index 04f2c4ddc..31ccf0613 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ScaledInterPredictorTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ScaledInterPredictorTests.cs @@ -7,7 +7,7 @@ using SixLabors.ImageSharp.Tests.TestUtilities; namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; /// -/// Verifies AV1 reference scaling and variable-phase inter convolution against an independent libaom-shaped oracle. +/// Verifies AV1 reference scaling and variable-phase inter convolution against current libaom arithmetic. /// [Trait("Format", "Avif")] public class Av1ScaledInterPredictorTests @@ -22,6 +22,11 @@ public class Av1ScaledInterPredictorTests /// private const int Round0Bits = 3; + /// + /// The vertical shift that preserves AV1's compound intermediate precision. + /// + private const int CompoundRound1Bits = 7; + /// /// The number of samples in every stored interpolation row. /// @@ -63,10 +68,10 @@ public class Av1ScaledInterPredictorTests private const HwIntrinsics PredictorConfigurations = HwIntrinsics.AllowAll | HwIntrinsics.DisableHWIntrinsic; /// - /// Verifies the pinned Q14 scale factors, Q10 steps, and signed coordinate rounding. + /// Verifies current libaom's Q14 scale factors, Q10 steps, and signed coordinate rounding. /// [Fact] - public void ReferenceScaleMatchesPinnedLibaomFixedPointRules() + public void ReferenceScaleMatchesCurrentLibaomFixedPointRules() { Av1ReferenceScale downscaledReference = new(40, 24, 64, 48); @@ -96,16 +101,30 @@ public class Av1ScaledInterPredictorTests /// Verifies exact scaled 8-bit output, variable filter phases, vector tails, and untouched destination padding. /// [Fact] - public void BytePredictionMatchesLibaomOracleAcrossIntrinsicConfigurations() + public void BytePredictionMatchesCurrentLibaomOracleAcrossIntrinsicConfigurations() => FeatureTestRunner.RunWithHwIntrinsicsFeature(ValidateBytePredictions, PredictorConfigurations); /// /// Verifies exact scaled 8-, 10-, and 12-bit output under the native vector and scalar configurations. /// [Fact] - public void HighBitDepthPredictionMatchesLibaomOracleAcrossIntrinsicConfigurations() + public void HighBitDepthPredictionMatchesCurrentLibaomOracleAcrossIntrinsicConfigurations() => FeatureTestRunner.RunWithHwIntrinsicsFeature(ValidateHighBitDepthPredictions, PredictorConfigurations); + /// + /// Verifies exact scaled 8-bit no-round compound intermediates under native vector and scalar configurations. + /// + [Fact] + public void ByteCompoundPredictionMatchesCurrentLibaomOracleAcrossIntrinsicConfigurations() + => FeatureTestRunner.RunWithHwIntrinsicsFeature(ValidateByteCompoundPredictions, PredictorConfigurations); + + /// + /// Verifies exact scaled 8-, 10-, and 12-bit no-round compound intermediates under native vector and scalar configurations. + /// + [Fact] + public void HighBitDepthCompoundPredictionMatchesCurrentLibaomOracleAcrossIntrinsicConfigurations() + => FeatureTestRunner.RunWithHwIntrinsicsFeature(ValidateHighBitDepthCompoundPredictions, PredictorConfigurations); + /// /// Applies each scaled-prediction scenario to byte storage. /// @@ -191,6 +210,91 @@ public class Av1ScaledInterPredictorTests } } + /// + /// Applies each scaled compound-prediction scenario to byte source storage. + /// + private static void ValidateByteCompoundPredictions() + { + foreach (ScaledPredictionCase testCase in CreatePredictionCases()) + { + byte[] source = CreateByteSource(testCase, out int sourceStride, out int sourceOrigin); + int destinationStride = testCase.Width + DestinationRowPadding; + ushort[] expected = CreateUInt16Destination(testCase, destinationStride); + ushort[] actual = (ushort[])expected.Clone(); + short[] scratch = new short[ + Av1ScaledInterPredictor.GetScaledScratchLength( + testCase.Width, + testCase.Height, + testCase.VerticalPhase, + testCase.VerticalStep)]; + + ApplyCompoundReference(source, sourceStride, sourceOrigin, expected, destinationStride, testCase, 8); + + Av1ScaledInterPredictor.PredictScaledCompound( + source, + sourceStride, + sourceOrigin, + actual.AsSpan(DestinationPrefix), + destinationStride, + testCase.Width, + testCase.Height, + testCase.HorizontalFilter, + testCase.VerticalFilter, + testCase.HorizontalPhase, + testCase.HorizontalStep, + testCase.VerticalPhase, + testCase.VerticalStep, + scratch); + + Assert.Equal(expected, actual); + } + } + + /// + /// Applies each scaled compound-prediction scenario to every supported high-bit-depth precision. + /// + private static void ValidateHighBitDepthCompoundPredictions() + { + int[] bitDepths = [8, 10, 12]; + foreach (int bitDepth in bitDepths) + { + foreach (ScaledPredictionCase testCase in CreatePredictionCases()) + { + ushort[] source = CreateUInt16Source(testCase, bitDepth, out int sourceStride, out int sourceOrigin); + int destinationStride = testCase.Width + DestinationRowPadding; + ushort[] expected = CreateUInt16Destination(testCase, destinationStride); + ushort[] actual = (ushort[])expected.Clone(); + short[] scratch = new short[ + Av1ScaledInterPredictor.GetScaledScratchLength( + testCase.Width, + testCase.Height, + testCase.VerticalPhase, + testCase.VerticalStep)]; + + ApplyCompoundReference(source, sourceStride, sourceOrigin, expected, destinationStride, testCase, bitDepth); + + Av1ScaledInterPredictor.PredictScaledCompound( + source, + sourceStride, + sourceOrigin, + actual.AsSpan(DestinationPrefix), + destinationStride, + testCase.Width, + testCase.Height, + testCase.HorizontalFilter, + testCase.VerticalFilter, + testCase.HorizontalPhase, + testCase.HorizontalStep, + testCase.VerticalPhase, + testCase.VerticalStep, + bitDepth, + scratch); + + Assert.Equal(expected, actual); + } + } + } + /// /// Creates cases covering variable phases, every filter family, reduced kernels, and vector tails. /// @@ -348,6 +452,74 @@ public class Av1ScaledInterPredictorTests WriteReference(intermediate, intermediateStride, destination, destinationStride, testCase, bitDepth); } + /// + /// Applies independent scaled compound convolution to byte source storage. + /// + private static void ApplyCompoundReference( + byte[] source, + int sourceStride, + int sourceOrigin, + ushort[] destination, + int destinationStride, + ScaledPredictionCase testCase, + int bitDepth) + { + short[] intermediate = CreateIntermediate(testCase); + int intermediateStride = testCase.Width; + int round0 = GetRound0Bits(bitDepth); + int horizontalBias = 1 << (bitDepth + FilterBits - 1); + Span coefficients = stackalloc short[FilterTapCount]; + + for (int row = 0; row < intermediate.Length / intermediateStride; row++) + { + for (int column = 0; column < testCase.Width; column++) + { + int position = testCase.HorizontalPhase + (column * testCase.HorizontalStep); + int sourceColumn = (position >> Av1ReferenceScale.SubpixelBits) - 3; + FillCoefficients(testCase.HorizontalFilter, (position & Av1ReferenceScale.SubpixelMask) >> 6, testCase.Width <= 4, coefficients); + int sourceIndex = sourceOrigin + ((row - 3) * sourceStride) + sourceColumn; + int sum = horizontalBias + Convolve(source, sourceIndex, coefficients); + intermediate[(row * intermediateStride) + column] = (short)RoundPowerOfTwo(sum, round0); + } + } + + WriteCompoundReference(intermediate, intermediateStride, destination, destinationStride, testCase, bitDepth); + } + + /// + /// Applies independent scaled compound convolution to ushort source storage. + /// + private static void ApplyCompoundReference( + ushort[] source, + int sourceStride, + int sourceOrigin, + ushort[] destination, + int destinationStride, + ScaledPredictionCase testCase, + int bitDepth) + { + short[] intermediate = CreateIntermediate(testCase); + int intermediateStride = testCase.Width; + int round0 = GetRound0Bits(bitDepth); + int horizontalBias = 1 << (bitDepth + FilterBits - 1); + Span coefficients = stackalloc short[FilterTapCount]; + + for (int row = 0; row < intermediate.Length / intermediateStride; row++) + { + for (int column = 0; column < testCase.Width; column++) + { + int position = testCase.HorizontalPhase + (column * testCase.HorizontalStep); + int sourceColumn = (position >> Av1ReferenceScale.SubpixelBits) - 3; + FillCoefficients(testCase.HorizontalFilter, (position & Av1ReferenceScale.SubpixelMask) >> 6, testCase.Width <= 4, coefficients); + int sourceIndex = sourceOrigin + ((row - 3) * sourceStride) + sourceColumn; + int sum = horizontalBias + Convolve(source, sourceIndex, coefficients); + intermediate[(row * intermediateStride) + column] = (short)RoundPowerOfTwo(sum, round0); + } + } + + WriteCompoundReference(intermediate, intermediateStride, destination, destinationStride, testCase, bitDepth); + } + /// /// Allocates the oracle's independently shaped intermediate block. /// @@ -409,6 +581,40 @@ public class Av1ScaledInterPredictorTests } } + /// + /// Writes the biased no-round compound output from the horizontally filtered intermediate block. + /// + private static void WriteCompoundReference( + short[] intermediate, + int intermediateStride, + ushort[] destination, + int destinationStride, + ScaledPredictionCase testCase, + int bitDepth) + { + int round0 = GetRound0Bits(bitDepth); + int offsetBits = bitDepth + (2 * FilterBits) - round0; + int verticalBias = 1 << offsetBits; + Span coefficients = stackalloc short[FilterTapCount]; + for (int row = 0; row < testCase.Height; row++) + { + int position = testCase.VerticalPhase + (row * testCase.VerticalStep); + int sourceRow = position >> Av1ReferenceScale.SubpixelBits; + FillCoefficients(testCase.VerticalFilter, (position & Av1ReferenceScale.SubpixelMask) >> 6, testCase.Height <= 4, coefficients); + for (int column = 0; column < testCase.Width; column++) + { + int sum = verticalBias + Convolve( + intermediate, + (sourceRow * intermediateStride) + column, + intermediateStride, + coefficients); + + destination[DestinationPrefix + (row * destinationStride) + column] = + (ushort)RoundPowerOfTwo(sum, CompoundRound1Bits); + } + } + } + /// /// Removes both normative convolution biases after the vertical pass. /// @@ -471,7 +677,7 @@ public class Av1ScaledInterPredictorTests } /// - /// Selects one pinned coefficient row without reading production filter storage. + /// Selects one current-libaom coefficient row without reading production filter storage. /// private static void FillCoefficients(Av1InterpolationFilter filter, int phase, bool reduced, Span destination) { diff --git a/tests/Images/External/ReferenceOutput/Av1ReconstructionConformanceTests/DecodeScaledReferenceMatchesPinnedReferences_Rgba32_libavif-webp-logo-scaled-reference.png b/tests/Images/External/ReferenceOutput/Av1ReconstructionConformanceTests/DecodeScaledReferenceMatchesCurrentLibaomReferences_Rgba32_libavif-webp-logo-scaled-reference.png similarity index 100% rename from tests/Images/External/ReferenceOutput/Av1ReconstructionConformanceTests/DecodeScaledReferenceMatchesPinnedReferences_Rgba32_libavif-webp-logo-scaled-reference.png rename to tests/Images/External/ReferenceOutput/Av1ReconstructionConformanceTests/DecodeScaledReferenceMatchesCurrentLibaomReferences_Rgba32_libavif-webp-logo-scaled-reference.png diff --git a/tests/Images/External/ReferenceOutput/Av1ReconstructionConformanceTests/DecodeSelectedProgressiveSpatialLayerMatchesPinnedReferences_Rgba32_libavif-webp-logo-scaled-reference-lsel0.png b/tests/Images/External/ReferenceOutput/Av1ReconstructionConformanceTests/DecodeSelectedProgressiveSpatialLayerMatchesCurrentLibaomReferences_Rgba32_libavif-webp-logo-scaled-reference-lsel0.png similarity index 100% rename from tests/Images/External/ReferenceOutput/Av1ReconstructionConformanceTests/DecodeSelectedProgressiveSpatialLayerMatchesPinnedReferences_Rgba32_libavif-webp-logo-scaled-reference-lsel0.png rename to tests/Images/External/ReferenceOutput/Av1ReconstructionConformanceTests/DecodeSelectedProgressiveSpatialLayerMatchesCurrentLibaomReferences_Rgba32_libavif-webp-logo-scaled-reference-lsel0.png diff --git a/tests/Images/Input/Heif/Av1/Conformance/README.md b/tests/Images/Input/Heif/Av1/Conformance/README.md index 8e00ce989..464bd8c66 100644 --- a/tests/Images/Input/Heif/Av1/Conformance/README.md +++ b/tests/Images/Input/Heif/Av1/Conformance/README.md @@ -46,6 +46,7 @@ The reference builds use `AOM_TARGET_CPU=generic` and disable libyuv. Native rec | `libavif-webp-logo-difference-weighted-compound` | Difference-weighted compound prediction with both mask orientations | | `libavif-webp-logo-inter-intra` | Smooth and wedge inter-intra prediction | | `libavif-webp-logo-obmc` | Overlapping motion compensation through a 19-frame dependent sequence | +| `libavif-webp-logo-scaled-reference` | A 40x40 retained reference scaled into an 80x80 dependent frame | | `libavif-rotating-grid-local-warp` | Multi-sample local affine projection and warped prediction through a two-frame dependent sequence | | `libavif-rotating-grid-global-warp` | Non-translational rotation/zoom GLOBALMV prediction through a two-frame dependent sequence | @@ -194,6 +195,38 @@ PNG's SHA-256 is `D2CB388C9092EF17C4F0382C0150DD30D6F9D0EE247FF45AB5D7D4D312CEB23C`; its pixels were not changed when its contract-derived filename was updated with the test name. +## Scaled-reference fixture + +The `libavif-webp-logo-scaled-reference.avif` and +`libavif-webp-logo-scaled-reference-lsel0.avif` files are retained solely as interoperability inputs. +Their shared 2,195-byte `.bit` payload contains a 701-byte independent base layer followed by the +dependent layer; no container implementation is used as an AV1 arithmetic reference. + +On 2026-08-31 the payload was decoded with `aomdec` from the freshly updated clean checkout of current +official libaom `main`, observed at `441c439b9916474cac15d2822af47a9ad70674a8`: + +```text +aomdec --codec=av1 --threads=1 --row-mt=0 --all-layers --rawvideo --output-bit-depth=8 -o current-libaom-scaled-reference-all.yuv libavif-webp-logo-scaled-reference.bit +``` + +Current libaom produced a 40x40 YUV444 base frame and an 80x80 YUV444 dependent frame. The combined +24,000 native samples have SHA-256 +`DD219E41B52C6C9343A92CD0A2D451DF57B73B25F10124811675B4CB2F8D666F`. The base frame matches +`libavif-webp-logo-scaled-reference-base-libaom.yuv` exactly, and the dependent frame matches the native +planes in `libavif-webp-logo-scaled-reference-libaom.y4m` exactly, with zero differing samples. + +The production tests require the retained 40x40 base and the 80x80 dependent reconstruction, compare both +native frames exactly, compare the selected base layer and final RGBA presentation through ImageSharp's +established reference-output API, and repeat both layer selections with constrained tracked allocation. +The scaled-prediction FeatureTestRunner tests cover native and no-round compound output at 8, 10, and +12 bits, and a complete `Av1BlockDecoder.DecodeBlock()` test proves that scaled compound references remain +in the no-round intermediate domain until the final blend. + +The two presentation PNGs were renamed with their current-libaom test contracts without changing their +bytes. Their SHA-256 values remain +`DC4C6DBE6BD92C5FCE1E3E23700AFA603EF04ED02EDD336213EBBA1E3BD84BA0` and +`678C5E5D4650EA6F0C590302E7DB9E3C6608851BC577453DA4A6837BDB4D3AF3`. + ## Local warped-motion fixture The `libavif-rotating-grid-local-warp.avif` fixture was encoded from a deterministic two-frame 256x256 limited-range YUV444 source. The source combines checkerboard, ring, and chroma-gradient detail; its second frame rotates the first by 2.5 degrees with nearest-neighbor sampling and edge clamping. The two-frame source Y4M SHA-256 is `82C1468C95C996B05165590417184F59373D67896F8C7B0EB398582C29C9C7A7`. Pinned scalar libavif and libaom generated the fixture and references with: