From fb4c64474e1ced4067a42731384f3b5ad4212a2f Mon Sep 17 00:00:00 2001 From: James Jackson-South Date: Mon, 31 Aug 2026 03:35:53 +1000 Subject: [PATCH] Complete AV1 difference-weighted compound decoding checkpoint --- HEIF_IMPLEMENTATION_PLAN.md | 42 +++++++++- .../Heif/Av1/Transform/Av1BlockDecoder.cs | 40 +++++++++- .../Heif/Av1/Av1CompoundBlockDecoderTests.cs | 79 ++++++++++++++++++- .../Av1/Av1CompoundInterPredictorTests.cs | 48 ++++++++++- .../Input/Heif/Av1/Conformance/README.md | 17 ++-- 5 files changed, 212 insertions(+), 14 deletions(-) diff --git a/HEIF_IMPLEMENTATION_PLAN.md b/HEIF_IMPLEMENTATION_PLAN.md index 8f2a955f0..7b7a3ac81 100644 --- a/HEIF_IMPLEMENTATION_PLAN.md +++ b/HEIF_IMPLEMENTATION_PLAN.md @@ -30,7 +30,7 @@ Reference checkout evidence on 2026-08-31: Reconciled with the worktree on 2026-08-31. - [~] The bounded container reader, still-image path, sequence parser, AV1 decoder, color pipeline, presentation pipeline, and broad AV1 test suite exist locally. -- [~] The inter-frame decoder has verified checkpoints through wedge compound prediction. Difference-weighted compound prediction, OBMC, scaled references, local warped motion, and global motion exist locally but remain open until their ordered checkpoints below are completed. +- [~] The inter-frame decoder has verified checkpoints through difference-weighted compound prediction. OBMC, scaled references, local warped motion, and global motion exist locally but remain open until their ordered checkpoints below are completed. - [~] Loop filtering, CDEF, super-resolution, restoration, film grain, layered presentation, alpha composition, and color conversion exist locally. Shared-source cleanup changed the current tree, so final production-path verification is open. - [~] AV1 writer primitives, forward transforms, symbol encoding, and tile-writing source exist locally, but they are not connected to the public encoder. - [ ] The public AV1 encoder is not implemented. HeifEncoderCore.Encode throws NotSupportedException when AV1 is selected. @@ -180,8 +180,8 @@ The single-reference syntax, buffer, reconstruction, and ownership foundation is - [x] Inter-intra prediction. - [x] Distance-weighted compound prediction. - [x] Wedge compound prediction. -- [~] Difference-weighted compound prediction. Current item. -- [~] OBMC. +- [x] Difference-weighted compound prediction. +- [~] OBMC. Current item. - [~] Scaled-reference prediction. - [~] Local warped prediction. - [~] Non-translational global prediction. @@ -322,6 +322,42 @@ Verified wedge compound checkpoint evidence on 2026-08-31: - [x] The focused Release checkpoint set passes 35/35 on net10.0 and 35/35 on net11.0, with zero failures or skips. Scoped analyzer and whitespace verification pass for every changed C# file. Roslynk reports zero compiler errors, `git diff --check` passes, and `.gitattributes` is unchanged. +- [x] The completed checkpoint was committed as `9883a24dc319e16b471f68be632d4f62f2c1cd5e` + with author and committer `James Jackson-South `. + +Verified difference-weighted compound checkpoint evidence on 2026-08-31: + +- [x] Audited syntax against current libaom `av1/decoder/decodemv.c`. ImageSharp applies the same + masked-compound enable and block-size gates, selects difference-weighted compound directly when wedge + is unavailable, and reads the same one-bit type-38 mask orientation. +- [x] Audited mask generation and reconstruction against current libaom `av1/common/reconinter.c` and + `aom_dsp/blend_a64_mask.c`. The d16 path rounds the absolute intermediate difference by the + convolution and bit-depth shift, scales it by 1/16, adds the type-38 base, clamps or inverts the mask, + and then blends the original no-round intermediates before final rounding and clipping. Chroma reuses + the luma-derived mask through rounded subsampling. +- [x] Corrected the production 10/12-bit subpixel eligibility gate, which previously rounded both + references before difference-mask construction and blending. Difference-weighted blocks now use the + existing semantic intermediate mask-builder and mask-blend predictor/operator families through the + sole final rounding step. No new operator family, per-block allocation, or copy was introduced. +- [x] Renamed the stale “pinned formula” test and extended FeatureTestRunner's independent oracle across + current-libaom regular and d16 mask arithmetic, both mask orientations, 8/10/12-bit samples, widths + that cross every Vector512, Vector256, Vector128, and scalar boundary, subpixel phases, and row-padding + sentinels. +- [x] Added a complete `Av1BlockDecoder.DecodeBlock` regression for 10/12-bit half-sample prediction + and both type-38 orientations. Its expected mask and reconstruction are calculated directly from the + current-libaom equations, independently of the production mask builder and finalizer. +- [x] Extracted the fixture's 5,358-byte AV1 `mdat` payload and decoded it with refreshed current + libaom `aomdec`, using one thread with row threading disabled. All 19 frames decoded. The final + 19,200 YUV444 samples have SHA-256 + `E8CAA650F1571C5B9CACAF8C06E1DDF5F5D2ED35F65F1C34377076C573425899` and match the retained native + reference with zero differing samples. +- [x] The real 19-frame production sequence requires both difference-mask orientations, compares final + native Y, Cb, and Cr planes exactly, compares final RGBA presentation through ImageSharp's established + reference-output API, and repeats the complete decode with a 1,024-byte constrained tracked allocator + and exactly-once return checks. +- [x] The focused Release checkpoint set passes 37/37 on net10.0 and 37/37 on net11.0, with zero + failures or skips. Scoped analyzer and whitespace verification pass for every changed C# file. + Roslynk reports zero compiler errors, `git diff --check` passes, and `.gitattributes` is unchanged. For every item: diff --git a/src/ImageSharp/Formats/Heif/Av1/Transform/Av1BlockDecoder.cs b/src/ImageSharp/Formats/Heif/Av1/Transform/Av1BlockDecoder.cs index eeb8d6d78..0e01de0b9 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Transform/Av1BlockDecoder.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Transform/Av1BlockDecoder.cs @@ -375,7 +375,8 @@ internal sealed class Av1BlockDecoder : IDisposable modeInfo.CompoundType is Av1CompoundType.Average or Av1CompoundType.DistanceWeighted or - Av1CompoundType.Wedge && + Av1CompoundType.Wedge or + Av1CompoundType.DifferenceWeighted && modeInfo.MotionMode != Av1MotionMode.Warped && modeInfo.YMode != Av1PredictionMode.GlobalGlobalMotionVector; @@ -763,6 +764,43 @@ internal sealed class Av1BlockDecoder : IDisposable subY: 0, this.frameBuffer.BitDepth.GetBitCount()); } + else if (modeInfo.CompoundType == Av1CompoundType.DifferenceWeighted) + { + int lumaWidth = blockSize.GetWidth(); + if (plane == 0) + { + // Difference-weighted chroma reuses the luma-derived segment mask. Building it + // only for plane zero preserves that decoded contract before chroma subsampling. + Av1CompoundIntermediateDifferenceWeightedMaskBuilder.FillDifferenceWeightedIntermediateMask( + compoundMask, + lumaWidth, + first, + predictionWidth, + highBitDepthSecondPrediction, + predictionWidth, + predictionWidth, + predictionHeight, + this.frameBuffer.BitDepth.GetBitCount(), + modeInfo.DifferenceWeightedMaskType); + } + + // The d16 mask and final blend consume the same no-round intermediates. Rounding + // either reference first changes both the derived mask and the reconstructed sample. + Av1CompoundIntermediateMaskBlendPredictor.BlendIntermediate( + highBitDepthDestination, + reconstructionStride, + first, + predictionWidth, + highBitDepthSecondPrediction, + predictionWidth, + compoundMask, + lumaWidth, + predictionWidth, + predictionHeight, + subX, + subY, + this.frameBuffer.BitDepth.GetBitCount()); + } else { Av1CompoundIntermediateAveragePredictor.AverageIntermediate( diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundBlockDecoderTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundBlockDecoderTests.cs index 0c0e92079..868af1d6a 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundBlockDecoderTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundBlockDecoderTests.cs @@ -150,6 +150,15 @@ public class Av1CompoundBlockDecoderTests ValidateSubpixelHighBitDepthWedgeCompoundPrediction, CompoundPredictionConfigurations); + /// + /// Verifies that high-bit-depth subpixel predictors retain no-round precision through difference masking and blending. + /// + [Fact] + public void DecodeBlockReconstructsSubpixelHighBitDepthDifferenceWeightedCompoundPrediction() + => FeatureTestRunner.RunWithHwIntrinsicsFeature( + ValidateSubpixelHighBitDepthDifferenceWeightedCompoundPrediction, + CompoundPredictionConfigurations); + /// /// Verifies that both references of a GLOBAL_GLOBALMV block use their complete matrix before compound averaging. /// @@ -693,14 +702,33 @@ public class Av1CompoundBlockDecoderTests } } + /// + /// Reconstructs both high-bit-depth subpixel difference-mask orientations at every supported source precision. + /// + private static void ValidateSubpixelHighBitDepthDifferenceWeightedCompoundPrediction() + { + foreach (Av1BitDepth bitDepth in new[] { Av1BitDepth.TenBit, Av1BitDepth.TwelveBit }) + { + foreach (Av1DifferenceWeightedMaskType maskType in Enum.GetValues()) + { + ValidateSubpixelHighBitDepthCompoundPredictionAtBitDepth( + bitDepth, + Av1CompoundType.DifferenceWeighted, + maskType); + } + } + } + /// /// Reconstructs one high-bit-depth half-sample compound block and compares it with the scalar no-round pipeline. /// /// The native sample depth. /// The final compound operation. + /// The difference-mask orientation when that compound operation is selected. private static void ValidateSubpixelHighBitDepthCompoundPredictionAtBitDepth( Av1BitDepth bitDepth, - Av1CompoundType compoundType) + Av1CompoundType compoundType, + Av1DifferenceWeightedMaskType differenceWeightedMaskType = Av1DifferenceWeightedMaskType.Type38) { const int frameSize = 32; const int blockOrigin = 8; @@ -734,6 +762,7 @@ public class Av1CompoundBlockDecoderTests CompoundType = compoundType, CompoundWedgeIndex = 0, CompoundWedgeSign = true, + DifferenceWeightedMaskType = differenceWeightedMaskType, }; modeInfo.ReferenceFrames[0] = Av1ReferenceFrameType.Last; @@ -832,6 +861,54 @@ public class Av1CompoundBlockDecoderTests Assert.NotEqual((ushort)60, expected[0]); } + else if (compoundType == Av1CompoundType.DifferenceWeighted) + { + int bitCount = bitDepth.GetBitCount(); + int intermediateRange = bitCount + 7 - 3 + 2; + int round0 = 3 + Math.Max(intermediateRange - 16, 0); + int roundBits = 14 - round0 - 7; + int offsetBits = bitCount + 14 - round0; + int roundOffset = (1 << (offsetBits - 7)) + (1 << (offsetBits - 8)); + int differenceRound = roundBits + bitCount - 8; + int maximum = (1 << bitCount) - 1; + byte[] expectedMask = new byte[blockSize * blockSize]; + byte[] actualMask = new byte[blockSize * blockSize]; + + for (int index = 0; index < expected.Length; index++) + { + int difference = Math.Abs(expectedFirst[index] - expectedSecond[index]); + difference = (difference + (1 << (differenceRound - 1))) >> differenceRound; + int alpha = Math.Min(64, 38 + (difference >> 4)); + if (differenceWeightedMaskType == Av1DifferenceWeightedMaskType.Type38Inverse) + { + alpha = 64 - alpha; + } + + expectedMask[index] = (byte)alpha; + int result = ((alpha * expectedFirst[index]) + ((64 - alpha) * expectedSecond[index])) >> 6; + result -= roundOffset; + if (roundBits != 0) + { + result = (result + (1 << (roundBits - 1))) >> roundBits; + } + + expected[index] = (ushort)Math.Clamp(result, 0, maximum); + } + + Av1CompoundIntermediateDifferenceWeightedMaskBuilder.FillDifferenceWeightedIntermediateMask( + actualMask, + blockSize, + expectedFirst, + blockSize, + expectedSecond, + blockSize, + blockSize, + blockSize, + bitCount, + differenceWeightedMaskType); + + Assert.Equal(expectedMask, actualMask); + } else if (compoundType == Av1CompoundType.Wedge) { ReadOnlySpan wedgeMask = diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundInterPredictorTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundInterPredictorTests.cs index cde1731cf..8990a73da 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundInterPredictorTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundInterPredictorTests.cs @@ -170,14 +170,14 @@ public class Av1CompoundInterPredictorTests } /// - /// Verifies both difference-mask orientations at each supported bit depth. + /// Verifies the current libaom difference-mask formula in both orientations at each supported bit depth. /// [Fact] - public void DifferenceWeightedMasksMatchPinnedFormula() + public void DifferenceWeightedMasksMatchCurrentLibaomFormula() => FeatureTestRunner.RunWithHwIntrinsicsFeature(ValidateDifferenceWeightedMasks, PredictorConfigurations); /// - /// Applies the pinned difference-mask formula at every bit depth and intrinsic width. + /// Applies the current libaom difference-mask formula at every bit depth and intrinsic width. /// private static void ValidateDifferenceWeightedMasks() { @@ -634,6 +634,48 @@ public class Av1CompoundInterPredictorTests bitDepth); Assert.Equal(expectedMasked, actualMasked); + + int differenceRound = roundBits + bitDepth - 8; + foreach (Av1DifferenceWeightedMaskType maskType in Enum.GetValues()) + { + byte[] expectedDifferenceMask = new byte[maskStride * height]; + byte[] actualDifferenceMask = new byte[maskStride * height]; + expectedDifferenceMask.AsSpan().Fill(0xA5); + actualDifferenceMask.AsSpan().Fill(0xA5); + + for (int row = 0; row < height; row++) + { + for (int column = 0; column < width; column++) + { + int intermediateIndex = (row * intermediateStride) + column; + int difference = Math.Abs( + expectedFirst[intermediateIndex] - expectedSecond[intermediateIndex]); + + difference = (difference + (1 << (differenceRound - 1))) >> differenceRound; + int alpha = Math.Min(64, 38 + (difference >> 4)); + if (maskType == Av1DifferenceWeightedMaskType.Type38Inverse) + { + alpha = 64 - alpha; + } + + expectedDifferenceMask[(row * maskStride) + column] = (byte)alpha; + } + } + + Av1CompoundIntermediateDifferenceWeightedMaskBuilder.FillDifferenceWeightedIntermediateMask( + actualDifferenceMask, + maskStride, + actualFirst, + intermediateStride, + actualSecond, + intermediateStride, + width, + height, + bitDepth, + maskType); + + Assert.Equal(expectedDifferenceMask, actualDifferenceMask); + } } } } diff --git a/tests/Images/Input/Heif/Av1/Conformance/README.md b/tests/Images/Input/Heif/Av1/Conformance/README.md index eb7af8ab6..7b2d0f42b 100644 --- a/tests/Images/Input/Heif/Av1/Conformance/README.md +++ b/tests/Images/Input/Heif/Av1/Conformance/README.md @@ -154,13 +154,18 @@ Current `aomdec` produced all 19 YUV444 frames. The final frame's 19,200 native `E8CAA650F1571C5B9CACAF8C06E1DDF5F5D2ED35F65F1C34377076C573425899` and match the retained Y4M with zero differing samples. +The difference-weighted fixture's 5,358-byte AV1 `mdat` payload was decoded under the same current-libaom +conditions. Current `aomdec` produced all 19 YUV444 frames. The final frame's 19,200 native samples have +SHA-256 `E8CAA650F1571C5B9CACAF8C06E1DDF5F5D2ED35F65F1C34377076C573425899` and match the retained +Y4M with zero differing samples. + The production tests independently require their decoded mode states. The distance-weighted input must -exercise distance-weighted compound prediction, the wedge input must exercise both mask orientations, -and the inter-intra input must exercise both smooth and wedge inter-intra prediction. The tests decode all -preceding samples, compare final native Y, Cb, and Cr planes exactly, compare final RGBA presentation -through ImageSharp's established reference-output API, and repeat reconstruction with constrained tracked -allocation. The retained PNG files are presentation evidence only and are not AV1 reconstruction -references. +exercise distance-weighted compound prediction, the wedge and difference-weighted inputs must each +exercise both mask orientations, and the inter-intra input must exercise both smooth and wedge inter-intra +prediction. The tests decode all preceding samples, compare final native Y, Cb, and Cr planes exactly, +compare final RGBA presentation through ImageSharp's established reference-output API, and repeat +reconstruction with constrained tracked allocation. The retained PNG files are presentation evidence only +and are not AV1 reconstruction references. ## Overlapping motion-compensation fixture