From c06fabf3040f55ce92c1c5ccc542785895b1f471 Mon Sep 17 00:00:00 2001 From: James Jackson-South Date: Sat, 5 Sep 2026 20:21:33 +1000 Subject: [PATCH] Dequantize AV1 coefficients during entropy parsing --- HEIF_IMPLEMENTATION_PLAN.md | 51 +++++++ .../Heif/Av1/Entropy/Av1SymbolDecoder.cs | 41 +++-- .../Heif/Av1/Pipeline/Av1FrameDecoder.cs | 18 +-- .../Quantizers/Av1InverseQuantizer.cs | 143 +++++++----------- .../Formats/Heif/Av1/Tiling/Av1FrameInfo.cs | 4 +- .../Formats/Heif/Av1/Tiling/Av1TileReader.cs | 34 +++-- .../Heif/Av1/Tiling/Av1TransformInfo.cs | 14 +- .../Heif/Av1/Transform/Av1BlockDecoder.cs | 131 +++++----------- .../Heif/Av1/Av1CoefficientsEntropyTests.cs | 40 +++-- .../Heif/Av1/Av1CompoundBlockDecoderTests.cs | 28 ++-- .../Formats/Heif/Av1/Av1FrameBufferTests.cs | 15 +- .../Heif/Av1/Av1InverseQuantizationTests.cs | 91 +++++++++++ 12 files changed, 336 insertions(+), 274 deletions(-) diff --git a/HEIF_IMPLEMENTATION_PLAN.md b/HEIF_IMPLEMENTATION_PLAN.md index 4d83fd1e93..116bb50eb3 100644 --- a/HEIF_IMPLEMENTATION_PLAN.md +++ b/HEIF_IMPLEMENTATION_PLAN.md @@ -363,6 +363,57 @@ Retained-state and cost-policy follow-up after `ef8b1a823`: and initializes frame costs; `encodeframe_utils.c:1629-1689` suppresses block refresh when CDF updates are disabled. No new managed effort mapping or isolated cost-refresh threshold was introduced. +Decoder coefficient-stage correction after `b305e6e89`, verified on 2026-09-05: + +- The source trace established an architectural deviation: `Av1SymbolDecoder.cs:1415-1459` published a + count prefix and scan-ordered quantized levels; `Av1TileReader.cs:1148-1157` packed those variable-length + groups. `Av1BlockDecoder.cs:126-153,1349-1399` then used another superblock-sized, all-plane workspace + to dequantize and reorder every transform during reconstruction. Both buffers were cleared separately. + The native entropy traversal dequantizes each signed level directly into its coefficient region + (`av1/decoder/decodetxb.c:116-165,279-312`); EOB belongs to separate metadata + (`av1/common/blockd.h:452-461`). Native region cursors advance by nominal transform area + (`av1/decoder/decodeframe.c:274-279`), independently of EOB. +- Parsing now publishes dequantized coefficients directly and records EOB in `Av1TransformInfo`. Each plane's + parser/reconstruction cursor advances by nominal transform area, including skipped transforms. Frame state + reserves 16 coefficient slots per 4x4 unit, with no count prefix. Reconstruction consumes that storage + directly; its second coefficient workspace and inverse-quantization pass are removed. These changes span + the production parser, transform descriptors, frame storage, and reconstruction caller, rather than adding + a disconnected native primitive. +- Quantization state moves to the parser. Mode syntax establishes delta-Q before `Residual` updates the + segment/plane values, matching `decodeframe.c:1172-1221`. Transform-local parameters preserve matrix + bypass, weighted-quantizer rounding, the 24-bit product mask, transform scaling before sign, and signed + precision clipping (`Av1InverseQuantizer.cs:92-135`, `decodetxb.c:52-58,298-312`). The entropy context still + uses the masked quantized magnitude and original DC sign (`Av1SymbolDecoder.cs:1424-1476`). No additional + allocator-owned buffer or native production dependency was added. +- The coefficient capacities decrease by 25.5 KiB for a 64x64 4:2:0 superblock configuration and 102 KiB for + 128x128 4:2:0: this combines removal of the second workspace with removal of count-prefix capacity. + These are source-derived coefficient-buffer sizes, excluding descriptor/object overhead, not measured + total memory or a timing improvement. No benchmark was run. +- Existing entropy tests now check the published dequantized raster values, including sparse and beyond-EOB + zeros, against fixed reference qindex-23 DC/AC values. Four added matrix/arithmetic cases use explicit + 8/10/12-bit reference values, matrix bypass for identity/one-dimensional transforms, lossless bypass, + asymmetric precision limits, product-mask wraparound, and sign-after-scaling rounding. The old matrix test + checked lengths only. These component cases do not establish complete signaled-matrix bitstream coverage. +- After the production edit, serialized Release .NET 11 Visual Studio VSTest passed **9,371/9,371** AV1 and + public HEIF encoder cases in 2.6933 minutes (`coefficient-stage-final.trx`). After adding the fixed-value + tests, a focused set passed **130/130** in 3.0217 seconds (`coefficient-stage-last-edit.trx`). Following final + whitespace cleanup, the checkpoint set passed **159/159** in 5.5189 seconds (`coefficient-stage-checkpoint.trx`). + The final build had zero errors and the existing 1,009 warnings; Roslynk reported zero compiler errors. + No production behavior changed after the broad run, and no retained reference samples were altered. +- Optimized current-reference redecoding matched the retained restoration and film-grain references across + **8,500,087** samples, maximum error **0**, zero exceeding one; those references also passed the managed + conformance tests. Twelve regenerated color sequences matched another **21,348** native samples exactly. + Reports are `restoration-comparison.json`, `film-grain-comparison.json`, and `decoder-comparison.json` in + `D:\GitHub\ynse01\av1-takeover-20260905`, outside the repository. This is bounded same-bitstream evidence, + not separately encoded output parity, complete decoder conformance, or a performance acceptance result. +- Remaining architecture differences are explicit: the managed reader still parses a complete superblock + before reconstruction (`Av1TileReader.ReadTile`, `Av1FrameDecoder.DecodePartition`) and clears its complete + coefficient regions before reuse. Native single-thread decoding interleaves parsing/reconstruction through + visitors (`decodeframe.c:935-958,2746-2765,2792-2801`), clears only through the maximum populated raster + position after inverse transform (`:154-164`), and separates parsing/reconstruction for row workers with + different buffer lifetimes (`:3244-3277`). Those traversal, clearing, and worker-lifetime differences remain + open; this checkpoint does not claim that changing coefficient representation completes them. + Film-grain decoder source comparison after `ef8b1a823`: - The complete template generation, random state, autoregression, scaling interpolation, overlap traversal, diff --git a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolDecoder.cs b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolDecoder.cs index ef5423167d..eb4223e3a3 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolDecoder.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolDecoder.cs @@ -2,6 +2,7 @@ // Licensed under the Six Labors Split License. using SixLabors.ImageSharp.Formats.Heif.Av1.Motion; +using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.Quantizers; using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.ChromaFromLuma; using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; @@ -1177,11 +1178,12 @@ internal ref struct Av1SymbolDecoder /// Indicates whether the active segment is lossless. /// Indicates whether the frame restricts transform choices. /// The luma transform type shared by inter-predicted chroma. - /// The transform descriptor updated with the decoded type and coded-block flag. + /// The transform descriptor updated with the decoded type and end-of-block position. /// The signed distance from the mode block to the right frame edge. /// The signed distance from the mode block to the bottom frame edge. /// Reusable padded coefficient-context storage owned by the tile reader. - /// The destination receiving the coefficient count followed by scan-ordered signed levels. + /// The zero-initialized destination receiving dequantized raster coefficients. + /// The quantizer containing the active segment and superblock delta-Q values. /// The one-based end-of-block position, or zero for an empty transform block. public int ReadCoefficients( Av1BlockModeInfo modeInfo, @@ -1202,7 +1204,8 @@ internal ref struct Av1SymbolDecoder int modeBlocksToRightEdge, int modeBlocksToBottomEdge, Av1LevelBuffer levels, - Span coefficientBuffer) + Span coefficientBuffer, + Av1InverseQuantizer inverseQuantizer) { Av1TransformSize adjustedTransformSize = transformSize.GetAdjusted(); int width = adjustedTransformSize.GetWidth(); @@ -1219,10 +1222,10 @@ internal ref struct Av1SymbolDecoder int endOfBlock; if (allZero) { + transformInfo.EndOfBlock = 0; if (plane == 0) { transformInfo.Type = Av1TransformType.DctDct; - transformInfo.CodeBlockFlag = false; } UpdateCoefficientContext(aboveContexts, leftContexts, blocksWide, blocksHigh, transformSize, blockPosition, aboveOffset, leftOffset, culLevel, modeBlocksToRightEdge, modeBlocksToBottomEdge); @@ -1278,10 +1281,15 @@ internal ref struct Av1SymbolDecoder } DebugGuard.MustBeGreaterThan(scan.Length, 0, nameof(scan)); - culLevel = this.ReadCoefficientsSign(coefficientBuffer, endOfBlock, scan, levels, transformBlockContext.DcSignContext, planeType); + Av1InverseQuantizer.TransformParameters quantization = new( + inverseQuantizer, modeInfo, transformInfo.Type, transformSize, (Av1Plane)plane); + + culLevel = this.ReadCoefficientsSign( + coefficientBuffer, endOfBlock, scan, levels, transformBlockContext.DcSignContext, planeType, quantization); + UpdateCoefficientContext(aboveContexts, leftContexts, blocksWide, blocksHigh, transformSize, blockPosition, aboveOffset, leftOffset, culLevel, modeBlocksToRightEdge, modeBlocksToBottomEdge); - transformInfo.CodeBlockFlag = true; + transformInfo.EndOfBlock = (ushort)endOfBlock; return endOfBlock; } @@ -1403,21 +1411,28 @@ internal ref struct Av1SymbolDecoder } /// - /// Reads coefficient signs and Golomb extensions, then writes scan-ordered signed levels. + /// Reads coefficient signs and Golomb extensions, then writes dequantized raster coefficients. /// - /// The destination receiving the coefficient count followed by signed levels. + /// The zero-initialized destination receiving dequantized coefficients. /// The one-based end-of-block position and coefficient count. /// The transform's scan-to-raster mapping. /// The decoded absolute-coefficient level plane. /// The neighboring DC sign context. /// The luma or chroma plane category. + /// The segment, plane, matrix, scale, and clipping parameters for this transform. /// The packed coefficient context used by adjacent transform blocks. - public int ReadCoefficientsSign(Span coefficientBuffer, int endOfBlock, ReadOnlySpan scan, Av1LevelBuffer levels, int dcSignContext, Av1PlaneType planeType) + private int ReadCoefficientsSign( + Span coefficientBuffer, + int endOfBlock, + ReadOnlySpan scan, + Av1LevelBuffer levels, + int dcSignContext, + Av1PlaneType planeType, + Av1InverseQuantizer.TransformParameters quantization) { ref Av1SymbolReader r = ref this.reader; int culLevel = 0; int dcValue = 0; - coefficientBuffer[0] = endOfBlock; for (int c = 0; c < endOfBlock; c++) { int sign = 0; @@ -1447,9 +1462,11 @@ internal ref struct Av1SymbolDecoder level &= 0xfffff; culLevel += level; - } - coefficientBuffer[c + 1] = sign != 0 ? -level : level; + // The entropy context uses the masked quantized magnitude, while reconstruction consumes the + // dequantized raster coefficient. Write it directly into the current superblock's zeroed region. + coefficientBuffer[pos] = quantization.Dequantize(level, pos, sign != 0); + } } culLevel = Math.Min(Av1Constants.CoefficientContextMask, culLevel); diff --git a/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1FrameDecoder.cs b/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1FrameDecoder.cs index 64e200a00b..d814b1fe61 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1FrameDecoder.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1FrameDecoder.cs @@ -5,12 +5,10 @@ using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit; using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.Cdef; using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.LoopFilter; using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.LoopRestoration; -using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.Quantizers; using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.SuperResolution; using SixLabors.ImageSharp.Formats.Heif.Av1.ReferenceFrames; using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; using SixLabors.ImageSharp.Formats.Heif.Av1.Transform; -using SixLabors.ImageSharp.Memory; namespace SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline; @@ -44,16 +42,6 @@ internal sealed class Av1FrameDecoder : IAv1FrameDecoder, IDisposable /// private readonly Av1ReferenceFrameStore referenceFrames; - /// - /// The coefficient inverse-quantization stage shared across superblocks. - /// - private readonly Av1InverseQuantizer inverseQuantizer; - - /// - /// The frame's base per-segment and per-plane dequantization values. - /// - private readonly Av1DeQuantizationContext deQuants; - /// /// The transform-size map populated during reconstruction and consumed by deblocking. /// @@ -86,8 +74,6 @@ internal sealed class Av1FrameDecoder : IAv1FrameDecoder, IDisposable this.frameInfo = frameInfo; this.frameBuffer = frameBuffer; this.referenceFrames = referenceFrames; - this.inverseQuantizer = new(sequenceHeader, frameHeader); - this.deQuants = new(sequenceHeader, frameHeader); this.loopFilterContext = new(frameBuffer.MemoryAllocator, sequenceHeader, frameHeader); try { @@ -96,7 +82,6 @@ internal sealed class Av1FrameDecoder : IAv1FrameDecoder, IDisposable this.frameHeader, this.frameBuffer, this.loopFilterContext, - this.inverseQuantizer, this.referenceFrames, paletteColorIndexMaps); } @@ -165,7 +150,7 @@ internal sealed class Av1FrameDecoder : IAv1FrameDecoder, IDisposable } /// - /// Reconstructs one superblock after applying its block state and delta-Q context. + /// Reconstructs one superblock from its parsed block state and dequantized coefficients. /// /// The superblock's top-left position in 4x4 mode-info units. /// The decoded syntax and block modes for the superblock. @@ -173,7 +158,6 @@ internal sealed class Av1FrameDecoder : IAv1FrameDecoder, IDisposable public void DecodeSuperblock(Point modeInfoPosition, Av1SuperblockInfo superblockInfo, Av1TileInfo tileInfo) { this.blockDecoder.UpdateSuperblock(superblockInfo); - this.inverseQuantizer.UpdateDequant(this.deQuants, superblockInfo); this.DecodePartition(modeInfoPosition, superblockInfo, tileInfo); } diff --git a/src/ImageSharp/Formats/Heif/Av1/Pipeline/Quantizers/Av1InverseQuantizer.cs b/src/ImageSharp/Formats/Heif/Av1/Pipeline/Quantizers/Av1InverseQuantizer.cs index cb2eb63828..3d8d1fa57f 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Pipeline/Quantizers/Av1InverseQuantizer.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Pipeline/Quantizers/Av1InverseQuantizer.cs @@ -70,99 +70,68 @@ internal sealed class Av1InverseQuantizer } /// - /// Converts scan-ordered quantized levels into clamped, raster-ordered transform coefficients. + /// Applies the active segment, plane, matrix, and transform scale to decoded coefficient magnitudes. /// - /// The block mode information containing the active segment identifier. - /// The packed coefficient buffer: the first element is the coefficient count and the remaining elements are scan-ordered levels. - /// The destination for raster-ordered dequantized coefficients. - /// The transform type that selects the coefficient scan and matrix class. - /// The transform dimensions and scale. - /// The color plane whose quantizer and matrix are used. - /// The number of coefficient levels consumed. - public int InverseQuantize(Av1BlockModeInfo mode, Span level, Span qCoefficients, Av1TransformType transformType, Av1TransformSize transformSize, Av1Plane plane) + public readonly ref struct TransformParameters { - Av1ScanOrder scanOrder = Av1ScanOrderConstants.GetScanOrder(transformSize, transformType); - ReadOnlySpan scanIndices = scanOrder.Scan; - - // AV1 bounds reconstructed coefficients to a signed range with seven headroom bits beyond pixel precision. - int maxValue = (1 << (7 + this.sequenceHeader.ColorConfig.BitDepth.GetBitCount())) - 1; - int minValue = -(1 << (7 + this.sequenceHeader.ColorConfig.BitDepth.GetBitCount())); - bool usingQuantizationMatrix = this.frameHeader.QuantizationParameters.IsUsingQMatrix; - bool lossless = this.frameHeader.LosslessArray[mode.SegmentId]; - short dequantDc = this.deQuantsDeltaQ.GetDc(mode.SegmentId, plane); - short dequantAc = this.deQuantsDeltaQ.GetAc(mode.SegmentId, plane); - - // The final matrix level is flat. Lossless blocks and frames without matrices select it globally. AV1 also - // requires identity and one-dimensional transform types, which occupy the enum range from Identity onward, - // to bypass frequency weighting even when the frame signals quantization matrices. - int qmLevel = lossless || !usingQuantizationMatrix - ? Av1ScanOrderConstants.QuantizationMatrixLevelCount - 1 - : this.frameHeader.SegmentationParameters.QMLevel[(int)plane][mode.SegmentId]; - - ReadOnlySpan iqMatrix = transformType < Av1TransformType.Identity - ? Av1InverseQuantizationLookup.GetQuantizationMatrix(qmLevel, plane, transformSize) - : Av1InverseQuantizationLookup.GetQuantizationMatrix(Av1Constants.QuantificationMatrixLevelCount - 1, Av1Plane.Y, transformSize); - - int shift = transformSize.GetScale(); - - // Entropy decoding stores the populated coefficient count in the leading slot and the levels after it. - int coefficientCount = level[0]; - level = level[1..]; - int lev = level[0]; - int qCoefficient; - if (lev != 0) + private readonly short dc; + private readonly short ac; + private readonly int minimum; + private readonly int maximum; + private readonly int shift; + private readonly ReadOnlySpan inverseMatrix; + + /// + /// Initializes a new instance of the struct. + /// + /// The active frame and superblock quantization values. + /// The block mode selecting the segment. + /// The transform type selecting frequency weighting. + /// The transform dimensions and coefficient scale. + /// The color plane selecting DC, AC, and matrix values. + public TransformParameters( + Av1InverseQuantizer quantizer, + Av1BlockModeInfo mode, + Av1TransformType transformType, + Av1TransformSize transformSize, + Av1Plane plane) { - int pos = scanIndices[0]; - - // Preserve the AV1 24-bit dequantization intermediate before removing transform-size scaling. - qCoefficient = (int)(((long)Math.Abs(lev) * GetDeQuantizedValue(dequantDc, pos, iqMatrix)) & 0xffffff); - qCoefficient >>= shift; - - if (lev < 0) - { - qCoefficient = -qCoefficient; - } - - qCoefficients[0] = Av1Math.Clamp(qCoefficient, minValue, maxValue); + int bitCount = quantizer.sequenceHeader.ColorConfig.BitDepth.GetBitCount(); + this.minimum = -(1 << (7 + bitCount)); + this.maximum = (1 << (7 + bitCount)) - 1; + this.dc = quantizer.deQuantsDeltaQ.GetDc(mode.SegmentId, plane); + this.ac = quantizer.deQuantsDeltaQ.GetAc(mode.SegmentId, plane); + this.shift = transformSize.GetScale(); + + // Lossless segments and one-dimensional or identity transforms use the flat matrix. Matrix lookup + // happens once per transform, before the entropy loop supplies its nonzero magnitudes and signs. + int matrixLevel = quantizer.frameHeader.LosslessArray[mode.SegmentId] || + !quantizer.frameHeader.QuantizationParameters.IsUsingQMatrix || + transformType >= Av1TransformType.Identity + ? Av1ScanOrderConstants.QuantizationMatrixLevelCount - 1 + : quantizer.frameHeader.SegmentationParameters.QMLevel[(int)plane][mode.SegmentId]; + + this.inverseMatrix = Av1InverseQuantizationLookup.GetQuantizationMatrix(matrixLevel, plane, transformSize); } - for (int i = 1; i < coefficientCount; i++) + /// + /// Dequantizes one coefficient magnitude and applies its sign and precision bounds. + /// + /// The nonnegative coefficient magnitude masked to twenty bits. + /// The coefficient's raster position. + /// Whether the decoded coefficient sign is negative. + /// The signed, scaled, and clipped transform coefficient. + public int Dequantize(int magnitude, int coefficientIndex, bool negative) { - lev = level[i]; - if (lev != 0) - { - int pos = scanIndices[i]; - - // AC levels arrive in entropy scan order but the inverse transform consumes raster positions. - qCoefficient = (int)(((long)Math.Abs(lev) * GetDeQuantizedValue(dequantAc, pos, iqMatrix)) & 0xffffff); - qCoefficient >>= shift; - - if (lev < 0) - { - qCoefficient = -qCoefficient; - } - - qCoefficients[pos] = Av1Math.Clamp(qCoefficient, minValue, maxValue); - } + int dequant = coefficientIndex == 0 ? this.dc : this.ac; + + // Matrix weights have five fractional bits. Round the weighted quantizer first, then retain the + // normative 24-bit product before removing transform-size scaling. Sign and clipping follow the shift. + const int bias = 1 << (Av1Constants.QuantizationMatrixElementBitCount - 1); + dequant = ((this.inverseMatrix[coefficientIndex] * dequant) + bias) >> Av1Constants.QuantizationMatrixElementBitCount; + int coefficient = (int)(((long)magnitude * dequant) & 0xffffff) >> this.shift; + coefficient = negative ? -coefficient : coefficient; + return Av1Math.Clamp(coefficient, this.minimum, this.maximum); } - - return coefficientCount; - } - - /// - /// Applies an inverse quantization-matrix weight to a plane dequantization value. - /// - /// The unweighted DC or AC dequantization value. - /// The raster coefficient index into the inverse matrix. - /// The inverse quantization matrix for the current level, plane, and transform size. - /// The matrix-weighted dequantization value. - private static int GetDeQuantizedValue(short dequant, int coefficientIndex, ReadOnlySpan iqMatrix) - { - // Matrix elements use fixed-point precision; adding half a unit produces nearest-integer rounding on shift. - const int bias = 1 << (Av1Constants.QuantizationMatrixElementBitCount - 1); - int deQuantifiedValue = dequant; - - deQuantifiedValue = ((iqMatrix[coefficientIndex] * deQuantifiedValue) + bias) >> Av1Constants.QuantizationMatrixElementBitCount; - return deQuantifiedValue; } } diff --git a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.cs b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.cs index 2c85573f71..11958d5eac 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.cs @@ -83,9 +83,9 @@ internal sealed partial class Av1FrameInfo : IDisposable private readonly MemoryAllocator memoryAllocator; /// - /// The coefficient slots reserved for one 4x4 mode-information unit: one end index followed by 16 coefficients. + /// The raster coefficient slots reserved for one 4x4 mode-information unit. /// - public const int CoefficientCountPerModeInfo = 1 + 16; + public const int CoefficientCountPerModeInfo = 16; /// /// Owns the luma and chroma coefficient scratch for the superblock currently being decoded. diff --git a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileReader.cs b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileReader.cs index f4052b7b53..4eadc10365 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileReader.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileReader.cs @@ -7,6 +7,7 @@ using SixLabors.ImageSharp.Formats.Heif.Av1.Entropy; using SixLabors.ImageSharp.Formats.Heif.Av1.Motion; using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit; using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline; +using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.Quantizers; using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; using SixLabors.ImageSharp.Formats.Heif.Av1.ReferenceFrames; @@ -76,6 +77,16 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable /// private int currentQuantizerIndex; + /// + /// The coefficient quantizer updated before residual syntax consumes the active superblock delta-Q state. + /// + private readonly Av1InverseQuantizer inverseQuantizer; + + /// + /// The frame's base per-segment and per-plane dequantization values. + /// + private readonly Av1DeQuantizationContext deQuants; + /// /// Stores the loop-filter delta values carried between superblocks in the current tile. /// @@ -258,6 +269,8 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable this.referenceFrames = referenceFrames; this.ownsPaletteColorIndexMaps = sharedPaletteColorIndexMaps is null; this.entropyContexts.BeginFrame(frameHeader.QuantizationParameters.BaseQIndex, primaryReferenceContext); + this.inverseQuantizer = new(sequenceHeader, frameHeader); + this.deQuants = new(sequenceHeader, frameHeader); // FrameInfo owns traversal records for this coded frame and one superblock of coefficient scratch. this.FrameInfo = new(this.configuration, this.SequenceHeader, this.FrameHeader); @@ -1038,6 +1051,9 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable Av1TileInfo tileInfo, Av1BlockSize blockSize) { + // Mode syntax has established delta-Q before residual decoding. Keep dequantization at this parsing + // boundary so each signed level is published once in the form consumed by inverse reconstruction. + this.inverseQuantizer.UpdateDequant(this.deQuants, superblockInfo); int maxBlocksWide = partitionInfo.GetMaxBlockWide(blockSize, false); int maxBlocksHigh = partitionInfo.GetMaxBlockHigh(blockSize, false); Av1BlockSize maxUnitSize = Av1BlockSize.Block64x64; @@ -1145,17 +1161,10 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable subY != 0); } - if (endOfBlock != 0) - { - // Coefficients are stored as an end index followed by scan-order values, so the - // next transform begins after both the prefix and its decoded coefficient range. - this.coefficientIndex[plane] += endOfBlock + 1; - transformInfo.CodeBlockFlag = true; - } - else - { - transformInfo.CodeBlockFlag = false; - } + // Each transform reserves its nominal area even when its residual is empty. EOB belongs + // to the descriptor, so the raster coefficient region contains no packed metadata prefix. + this.coefficientIndex[plane] += transformInfo.Size.GetWidth() * transformInfo.Size.GetHeight(); + transformInfo.EndOfBlock = (ushort)endOfBlock; transformInfoIndex++; } @@ -1353,7 +1362,8 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable partitionInfo.ModeBlockToRightEdge, partitionInfo.ModeBlockToBottomEdge, this.coefficientLevels, - coefficientBuffer); + coefficientBuffer, + this.inverseQuantizer); } /// diff --git a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TransformInfo.cs b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TransformInfo.cs index 7e6722cd9a..872c9aec24 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TransformInfo.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TransformInfo.cs @@ -63,17 +63,7 @@ internal struct Av1TransformInfo public int OffsetY { get; set; } /// - /// Gets or sets a value indicating whether the transform block contains a coded residual. - /// - /// - /// false - /// The block has no residual. - /// - /// - /// true - /// The block has a residual. - /// - /// + /// Gets or sets the end position of the coded coefficients in entropy scan order; zero means no residual. /// - public bool CodeBlockFlag { get; set; } + public ushort EndOfBlock { get; set; } } diff --git a/src/ImageSharp/Formats/Heif/Av1/Transform/Av1BlockDecoder.cs b/src/ImageSharp/Formats/Heif/Av1/Transform/Av1BlockDecoder.cs index fb95ac0a63..a74d100d43 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Transform/Av1BlockDecoder.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Transform/Av1BlockDecoder.cs @@ -2,19 +2,16 @@ // Licensed under the Six Labors Split License. using System.Buffers; -using System.Runtime.CompilerServices; using System.Runtime.InteropServices; using SixLabors.ImageSharp.Formats.Heif.Av1.Motion; using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit; using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.LoopFilter; -using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.Quantizers; using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.ChromaFromLuma; using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.IntraBlockCopy; using SixLabors.ImageSharp.Formats.Heif.Av1.ReferenceFrames; using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; -using SixLabors.ImageSharp.Memory; namespace SixLabors.ImageSharp.Formats.Heif.Av1.Transform; @@ -43,31 +40,16 @@ internal sealed class Av1BlockDecoder : IDisposable /// private readonly Av1LoopFilterContext loopFilterContext; - /// - /// The frame-owned inverse quantizer carrying the active superblock delta-Q state. - /// - private readonly Av1InverseQuantizer inverseQuantizer; - /// /// The retained reconstructed frames addressable by inter prediction. /// private readonly Av1ReferenceFrameStore referenceFrames; /// - /// Owns all reusable inverse-quantization, transform, and prediction storage. + /// Owns the reusable inverse-transform and prediction storage. /// private readonly IMemoryOwner workspaceOwner; - /// - /// The inverse-quantization prefix length in signed-short storage elements. - /// - private readonly int inverseQuantizationStorageLength; - - /// - /// The inverse-transform workspace offset in signed-short storage elements. - /// - private readonly int transformWorkspaceOffset; - /// /// The prediction workspace offset in signed-short storage elements. /// @@ -89,7 +71,7 @@ internal sealed class Av1BlockDecoder : IDisposable private readonly bool isLoopFilterEnabled; /// - /// The next packed coefficient position for each plane in the current superblock. + /// The next raster coefficient region for each plane in the current superblock. /// private InlineArray4 currentCoefficientIndex; @@ -105,7 +87,6 @@ internal sealed class Av1BlockDecoder : IDisposable /// The decoded frame header. /// The frame buffer receiving reconstructed samples. /// The transform-size map populated while reconstructing blocks. - /// The inverse quantizer carrying the active superblock delta-Q state. /// The retained reconstructed frames selected by inter blocks. /// The complete decoder-session palette map state. public Av1BlockDecoder( @@ -113,7 +94,6 @@ internal sealed class Av1BlockDecoder : IDisposable ObuFrameHeader frameHeader, Av1FrameBuffer frameBuffer, Av1LoopFilterContext loopFilterContext, - Av1InverseQuantizer inverseQuantizer, Av1ReferenceFrameStore referenceFrames, Av1TileReader.PaletteColorIndexMaps? paletteColorIndexMaps = null) { @@ -121,16 +101,7 @@ internal sealed class Av1BlockDecoder : IDisposable this.frameHeader = frameHeader; this.frameBuffer = frameBuffer; this.loopFilterContext = loopFilterContext; - this.inverseQuantizer = inverseQuantizer; this.referenceFrames = referenceFrames; - int ySize = (1 << this.sequenceHeader.SuperblockSizeLog2) * (1 << this.sequenceHeader.SuperblockSizeLog2); - - // One scratch plane is reused for every transform unit. Its maximum size must cover a complete superblock - // across all coded planes, with chroma dimensions reduced independently by their subsampling axes. - ObuColorConfig colorConfig = this.sequenceHeader.ColorConfig; - int chromaSubsampling = (colorConfig.SubSamplingX ? 1 : 0) + (colorConfig.SubSamplingY ? 1 : 0); - int chromaSize = ySize >> chromaSubsampling; - int inverseQuantizationSize = colorConfig.IsMonochrome ? ySize : ySize + (2 * chromaSize); int maximumBlockLength = 1 << sequenceHeader.SuperblockSizeLog2; int maximumBlockArea = maximumBlockLength * maximumBlockLength; int predictorWorkingLength = Math.Max( @@ -143,9 +114,7 @@ internal sealed class Av1BlockDecoder : IDisposable int predictorWorkingOffset = (2 * maximumBlockArea) + compoundMaskLength; int chromaFromLumaOffset = predictorWorkingOffset + predictorWorkingLength; int predictionScratchLength = chromaFromLumaOffset + Av1ChromaFromLumaContext.BufferLength; - this.inverseQuantizationStorageLength = inverseQuantizationSize * 2; - this.transformWorkspaceOffset = this.inverseQuantizationStorageLength; - this.predictionScratchOffset = this.transformWorkspaceOffset + (Av1TransformWorkspace.MaximumLength * 2); + this.predictionScratchOffset = Av1TransformWorkspace.MaximumLength * 2; // Integer workspaces occupy even signed-short slices so one allocator owner can retain the complete block // lifetime while prediction still receives the Memory contract needed by its reusable context. @@ -167,25 +136,19 @@ internal sealed class Av1BlockDecoder : IDisposable predictionScratch.Slice(chromaFromLumaOffset, Av1ChromaFromLumaContext.BufferLength)); } - /// - /// Gets the reusable raster-order coefficient buffer populated by inverse quantization. - /// - public Span CurrentInverseQuantizationCoefficients - => MemoryMarshal.Cast(this.workspaceOwner.Memory.Span[..this.inverseQuantizationStorageLength]); - /// /// Releases the pooled reconstruction workspaces owned by this decoder. /// public void Dispose() => this.workspaceOwner.Dispose(); /// - /// Resets the per-plane packed coefficient cursors before reconstructing a superblock. + /// Resets the per-plane coefficient-region cursors before reconstructing a superblock. /// /// The superblock whose coefficient streams will be consumed. public void UpdateSuperblock(Av1SuperblockInfo superblockInfo) { - // Each superblock owns independent packed coefficient streams for Y, U, and V. The first value for each - // transform unit stores its coefficient count, so DecodeBlock advances a plane cursor as units are consumed. + // Each superblock owns independent coefficient regions for Y, U, and V. Every transform advances its + // plane cursor by its nominal area, including transforms with no coded residual. this.currentCoefficientIndex[0] = 0; this.currentCoefficientIndex[1] = 0; this.currentCoefficientIndex[2] = 0; @@ -202,9 +165,7 @@ internal sealed class Av1BlockDecoder : IDisposable public void DecodeBlock(Av1BlockModeInfo modeInfo, Point modeInfoPosition, Av1BlockSize blockSize, Av1SuperblockInfo superblockInfo, Av1TileInfo tileInfo) { Span transformWorkspace = MemoryMarshal.Cast( - this.workspaceOwner.Memory.Span.Slice( - this.transformWorkspaceOffset, - Av1TransformWorkspace.MaximumLength * 2)); + this.workspaceOwner.Memory.Span[..(Av1TransformWorkspace.MaximumLength * 2)]); ObuColorConfig colorConfig = this.sequenceHeader.ColorConfig; Av1TransformType transformType; @@ -1346,58 +1307,44 @@ internal sealed class Av1BlockDecoder : IDisposable } } - int numberOfCoefficients = 0; - - if (!modeInfo.Skip && transformInfo[0].CodeBlockFlag) + int endOfBlock = transformInfo[0].EndOfBlock; + if (endOfBlock != 0) { - Span quantizationCoefficients = this.CurrentInverseQuantizationCoefficients; - int inverseQuantizationSize = transformSize.GetWidth() * transformSize.GetHeight(); - quantizationCoefficients[..inverseQuantizationSize].Clear(); transformType = transformInfo[0].Type; - // Inverse quantization writes raster coefficients into the reusable superblock scratch plane. - numberOfCoefficients = this.inverseQuantizer.InverseQuantize( - modeInfo, coefficients, quantizationCoefficients, transformType, transformSize, (Av1Plane)plane); - if (numberOfCoefficients != 0) + // Entropy decoding has already applied quantization, scan placement, and coefficient clipping. + // Prediction includes a top-reference row; inverse reconstruction begins one stride after it. + if (highBitDepth) { - // The packed coefficient stream prefixes every transform unit with its decoded coefficient - // count. Advance past that prefix as well as the coefficient values before the next unit. - this.currentCoefficientIndex[plane] += numberOfCoefficients + 1; - - if (highBitDepth) - { - // Prediction receives a reference-prefixed span beginning on the previous row. Inverse - // reconstruction operates on the transform itself, so advance to the first destination row. - Av1InverseTransformer.ReconstructHighBitDepth( - quantizationCoefficients, - highBitDepthTransformBlockReconstructionBuffer[reconstructionStride..], - reconstructionStride, - transformSize, - transformType, - plane, - numberOfCoefficients, - isLossless, - this.frameBuffer.BitDepth, - transformWorkspace); - } - else - { - // Keep the reference-prefix convention local to prediction; residuals are added at the - // first reconstructed row rather than the top-neighbor row. - Av1InverseTransformer.Reconstruct8Bit( - quantizationCoefficients, - transformBlockReconstructionBuffer[reconstructionStride..], - reconstructionStride, - transformSize, - transformType, - plane, - numberOfCoefficients, - isLossless, - transformWorkspace); - } + Av1InverseTransformer.ReconstructHighBitDepth( + coefficients, + highBitDepthTransformBlockReconstructionBuffer[reconstructionStride..], + reconstructionStride, + transformSize, + transformType, + plane, + endOfBlock, + isLossless, + this.frameBuffer.BitDepth, + transformWorkspace); + } + else + { + Av1InverseTransformer.Reconstruct8Bit( + coefficients, + transformBlockReconstructionBuffer[reconstructionStride..], + reconstructionStride, + transformSize, + transformType, + plane, + endOfBlock, + isLossless, + transformWorkspace); } } + this.currentCoefficientIndex[plane] += transformSize.GetWidth() * transformSize.GetHeight(); + // Store Luma for CFL if required! if (plane == (int)Av1Plane.Y && StoreChromaFromLumaRequired(colorConfig, ref partitionInfo)) { @@ -1429,7 +1376,7 @@ internal sealed class Av1BlockDecoder : IDisposable } } - // Transform descriptors are stored in the same traversal order as their packed coefficient groups. + // Transform descriptors and their coefficient regions follow the same per-plane traversal order. transformInfo = transformInfo[1..]; } } diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CoefficientsEntropyTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CoefficientsEntropyTests.cs index 9af860135f..f304fe83cd 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CoefficientsEntropyTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CoefficientsEntropyTests.cs @@ -7,6 +7,7 @@ using SixLabors.ImageSharp.Formats.Heif.Av1; using SixLabors.ImageSharp.Formats.Heif.Av1.Entropy; using SixLabors.ImageSharp.Formats.Heif.Av1.Motion; using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit; +using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.Quantizers; using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; @@ -1287,10 +1288,11 @@ public class Av1CoefficientsEntropyTests 0, 0, levels, - actuals); + actuals, + CreateInverseQuantizer()); // Assert - Assert.Equal(endOfBlock, actuals[0]); + Assert.Equal(endOfBlock, transformInfo.EndOfBlock); Assert.Equal(expected, actuals); } @@ -1334,7 +1336,7 @@ public class Av1CoefficientsEntropyTests coefficientsBuffer[scan[scanIndex]] = 0; } - Span actuals = new int[16 + 1]; + Span actuals = new int[16]; // Act encoder.WriteCoefficients(transformSize, transformType, intraDirection, coefficientsBuffer, componentType, transformBlockContext, endOfBlock, true, filterIntraMode, usesInterTransformSet: false); @@ -1363,12 +1365,13 @@ public class Av1CoefficientsEntropyTests 0, 0, levels, - actuals); + actuals, + CreateInverseQuantizer()); decoder.ValidateTrailingBits(); // Assert - Assert.Equal(endOfBlock, actuals[0]); + Assert.Equal(endOfBlock, transformInfo.EndOfBlock); } [Theory] @@ -1420,6 +1423,18 @@ public class Av1CoefficientsEntropyTests RoundTripCoefficientsCore(endOfBlock, componentType, blockSize, transformSize, transformType, intraDirection, filterIntraMode, false, true); } + private static Av1InverseQuantizer CreateInverseQuantizer() + { + ObuSequenceHeader sequenceHeader = new() + { + ColorConfig = new ObuColorConfig { BitDepth = Av1BitDepth.EightBit } + }; + + ObuFrameHeader frameHeader = new(); + frameHeader.QuantizationParameters.BaseQIndex = BaseQIndex; + return new Av1InverseQuantizer(sequenceHeader, frameHeader); + } + private static void RoundTripCoefficientsCore( ushort endOfBlock, Av1ComponentType componentType, @@ -1453,7 +1468,7 @@ public class Av1CoefficientsEntropyTests } } - Span actuals = new int[coefficientCount + 1]; + Span actuals = new int[coefficientCount]; // Act encoder.WriteCoefficients( @@ -1492,17 +1507,20 @@ public class Av1CoefficientsEntropyTests 0, 0, levels, - actuals); + actuals, + CreateInverseQuantizer()); decoder.ValidateTrailingBits(); // Assert - Assert.Equal(endOfBlock, actuals[0]); + Assert.Equal(endOfBlock, transformInfo.EndOfBlock); - // The parser retains quantized levels in entropy scan order; inverse quantization maps them back to raster positions. - for (int coefficientIndex = 0; coefficientIndex < endOfBlock; coefficientIndex++) + // Reference quant_common.c defines 8-bit qindex 23 as DC=26 and AC=30. Entropy output now publishes + // dequantized raster values, including zero runs and positions beyond EOB, rather than packed raw levels. + for (int coefficientIndex = 0; coefficientIndex < coefficientCount; coefficientIndex++) { - Assert.Equal(coefficientsBuffer[scan[coefficientIndex]], actuals[coefficientIndex + 1]); + int dequant = coefficientIndex == 0 ? 26 : 30; + Assert.Equal(coefficientsBuffer[coefficientIndex] * dequant, actuals[coefficientIndex]); } } diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundBlockDecoderTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundBlockDecoderTests.cs index 8551aacd6d..2771176831 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundBlockDecoderTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundBlockDecoderTests.cs @@ -5,7 +5,6 @@ using SixLabors.ImageSharp.Formats.Heif.Av1; using SixLabors.ImageSharp.Formats.Heif.Av1.Motion; using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit; using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.LoopFilter; -using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.Quantizers; using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; using SixLabors.ImageSharp.Formats.Heif.Av1.ReferenceFrames; @@ -86,13 +85,12 @@ public class Av1CompoundBlockDecoderTests using Av1LoopFilterContext loopFilterContext = new(frameBuffer.MemoryAllocator, sequenceHeader, frameHeader); - Av1InverseQuantizer inverseQuantizer = new(sequenceHeader, frameHeader); + using Av1BlockDecoder decoder = new( sequenceHeader, frameHeader, frameBuffer, loopFilterContext, - inverseQuantizer, referenceFrames); decoder.UpdateSuperblock(superblockInfo); @@ -205,13 +203,12 @@ public class Av1CompoundBlockDecoderTests using Av1LoopFilterContext loopFilterContext = new(frameBuffer.MemoryAllocator, sequenceHeader, frameHeader); - Av1InverseQuantizer inverseQuantizer = new(sequenceHeader, frameHeader); + using Av1BlockDecoder decoder = new( sequenceHeader, frameHeader, frameBuffer, loopFilterContext, - inverseQuantizer, referenceFrames); decoder.UpdateSuperblock(superblockInfo); @@ -331,13 +328,12 @@ public class Av1CompoundBlockDecoderTests using Av1LoopFilterContext loopFilterContext = new(frameBuffer.MemoryAllocator, sequenceHeader, frameHeader); - Av1InverseQuantizer inverseQuantizer = new(sequenceHeader, frameHeader); + using Av1BlockDecoder decoder = new( sequenceHeader, frameHeader, frameBuffer, loopFilterContext, - inverseQuantizer, referenceFrames); decoder.UpdateSuperblock(superblockInfo); @@ -422,13 +418,12 @@ public class Av1CompoundBlockDecoderTests using Av1LoopFilterContext loopFilterContext = new(frameBuffer.MemoryAllocator, sequenceHeader, frameHeader); - Av1InverseQuantizer inverseQuantizer = new(sequenceHeader, frameHeader); + using Av1BlockDecoder decoder = new( sequenceHeader, frameHeader, frameBuffer, loopFilterContext, - inverseQuantizer, referenceFrames); decoder.UpdateSuperblock(superblockInfo); @@ -505,13 +500,12 @@ public class Av1CompoundBlockDecoderTests using Av1LoopFilterContext loopFilterContext = new(frameBuffer.MemoryAllocator, sequenceHeader, frameHeader); - Av1InverseQuantizer inverseQuantizer = new(sequenceHeader, frameHeader); + using Av1BlockDecoder decoder = new( sequenceHeader, frameHeader, frameBuffer, loopFilterContext, - inverseQuantizer, referenceFrames); decoder.UpdateSuperblock(superblockInfo); @@ -601,13 +595,12 @@ public class Av1CompoundBlockDecoderTests using Av1LoopFilterContext loopFilterContext = new(frameBuffer.MemoryAllocator, sequenceHeader, frameHeader); - Av1InverseQuantizer inverseQuantizer = new(sequenceHeader, frameHeader); + using Av1BlockDecoder decoder = new( sequenceHeader, frameHeader, frameBuffer, loopFilterContext, - inverseQuantizer, referenceFrames); decoder.UpdateSuperblock(superblockInfo); @@ -857,13 +850,12 @@ public class Av1CompoundBlockDecoderTests superblockInfo.GetTransformInfoY()[0] = new Av1TransformInfo(Av1TransformSize.Size8x8, 0, 0); using Av1LoopFilterContext loopFilterContext = new(frameBuffer.MemoryAllocator, sequenceHeader, frameHeader); - Av1InverseQuantizer inverseQuantizer = new(sequenceHeader, frameHeader); + using Av1BlockDecoder decoder = new( sequenceHeader, frameHeader, frameBuffer, loopFilterContext, - inverseQuantizer, referenceFrames); decoder.UpdateSuperblock(superblockInfo); @@ -1252,13 +1244,12 @@ public class Av1CompoundBlockDecoderTests superblockInfo.GetTransformInfoY()[0] = new Av1TransformInfo(Av1TransformSize.Size8x8, 0, 0); using Av1LoopFilterContext loopFilterContext = new(frameBuffer.MemoryAllocator, sequenceHeader, frameHeader); - Av1InverseQuantizer inverseQuantizer = new(sequenceHeader, frameHeader); + using Av1BlockDecoder decoder = new( sequenceHeader, frameHeader, frameBuffer, loopFilterContext, - inverseQuantizer, referenceFrames); decoder.UpdateSuperblock(superblockInfo); @@ -1461,13 +1452,12 @@ public class Av1CompoundBlockDecoderTests using Av1LoopFilterContext loopFilterContext = new(frameBuffer.MemoryAllocator, sequenceHeader, frameHeader); - Av1InverseQuantizer inverseQuantizer = new(sequenceHeader, frameHeader); + using Av1BlockDecoder decoder = new( sequenceHeader, frameHeader, frameBuffer, loopFilterContext, - inverseQuantizer, referenceFrames); decoder.UpdateSuperblock(superblockInfo); diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1FrameBufferTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1FrameBufferTests.cs index 443e2f8688..188ff5e400 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1FrameBufferTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1FrameBufferTests.cs @@ -127,15 +127,14 @@ public class Av1FrameBufferTests /// Verifies that block reconstruction uses one exact-size owner across monochrome and chroma plane layouts. /// [Theory] - [InlineData(true, false, false, 4096)] - [InlineData(false, true, true, 6144)] - [InlineData(false, true, false, 8192)] - [InlineData(false, false, false, 12288)] + [InlineData(true, false, false)] + [InlineData(false, true, true)] + [InlineData(false, true, false)] + [InlineData(false, false, false)] public void BlockDecoderUsesOneContiguousWorkspaceOwner( bool isMonochrome, bool subsamplingX, - bool subsamplingY, - int expectedInverseQuantizationSize) + bool subsamplingY) { TestMemoryAllocator allocator = new(); Configuration configuration = Configuration.Default.Clone(); @@ -170,7 +169,6 @@ public class Av1FrameBufferTests using Av1LoopFilterContext loopFilterContext = new(Configuration.Default.MemoryAllocator, sequenceHeader, frameHeader); - Av1InverseQuantizer inverseQuantizer = new(sequenceHeader, frameHeader); using Av1ReferenceFrameStore referenceFrames = new(); // Reset the frame-plane logs so the following assertions describe only the block decoder's scratch owner. @@ -191,7 +189,6 @@ public class Av1FrameBufferTests Av1ChromaFromLumaContext.BufferLength; int expectedWorkspaceLength = - (expectedInverseQuantizationSize * 2) + (Av1TransformWorkspace.MaximumLength * 2) + predictionScratchLength; @@ -201,14 +198,12 @@ public class Av1FrameBufferTests frameHeader, frameBuffer, loopFilterContext, - inverseQuantizer, referenceFrames)) { workspaceAllocation = Assert.Single(allocator.AllocationLog); Assert.Empty(allocator.ReturnLog); Assert.Equal(typeof(short), workspaceAllocation.ElementType); Assert.Equal(expectedWorkspaceLength, workspaceAllocation.Length); - Assert.Equal(expectedInverseQuantizationSize, blockDecoder.CurrentInverseQuantizationCoefficients.Length); } TestMemoryAllocator.ReturnRequest returned = Assert.Single(allocator.ReturnLog); diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InverseQuantizationTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InverseQuantizationTests.cs index cf6d448790..cd46dabeb3 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InverseQuantizationTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InverseQuantizationTests.cs @@ -2,6 +2,7 @@ // Licensed under the Six Labors Split License. using SixLabors.ImageSharp.Formats.Heif.Av1; +using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit; using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.Quantizers; using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; using SixLabors.ImageSharp.Formats.Heif.Av1.Transform; @@ -11,6 +12,96 @@ namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; [Trait("Format", "Avif")] public class Av1InverseQuantizationTests { + [Theory] + [InlineData((int)Av1BitDepth.EightBit, 26, 30, 40, 32767, -32768)] + [InlineData((int)Av1BitDepth.TenBit, 75, 83, 112, 131071, -131072)] + [InlineData((int)Av1BitDepth.TwelveBit, 266, 297, 399, 524287, -524288)] + public void DequantizationMatchesReferenceMatrixAndPrecisionValues( + int bitDepthValue, + int dc, + int ac, + int weightedAc, + int maximum, + int minimum) + { + ObuSequenceHeader sequenceHeader = new() + { + ColorConfig = new ObuColorConfig { BitDepth = (Av1BitDepth)bitDepthValue } + }; + + ObuFrameHeader frameHeader = new(); + frameHeader.QuantizationParameters.BaseQIndex = 23; + frameHeader.QuantizationParameters.IsUsingQMatrix = true; + frameHeader.SegmentationParameters.QMLevel[0][0] = 0; + Av1InverseQuantizer quantizer = new(sequenceHeader, frameHeader); + Av1BlockModeInfo mode = new(Av1BlockSize.Block4x4, Point.Empty); + Av1InverseQuantizer.TransformParameters matrix = new( + quantizer, mode, Av1TransformType.DctDct, Av1TransformSize.Size4x4, Av1Plane.Y); + + // quant_common.c's qindex-23 tables supply the three DC/AC pairs above. Its level-zero luma matrix + // begins with weights 32 and 43; the rounded AC values are independently fixed in the theory data. + Assert.Equal(7 * dc, matrix.Dequantize(7, 0, false)); + Assert.Equal(11 * weightedAc, matrix.Dequantize(11, 1, false)); + Assert.Equal(-11 * weightedAc, matrix.Dequantize(11, 1, true)); + Assert.Equal(maximum, matrix.Dequantize(0xfffff, 1, false)); + Assert.Equal(minimum, matrix.Dequantize(0xfffff, 1, true)); + + // Identity and one-dimensional transforms bypass matrix weighting even when the frame enables it. + Av1InverseQuantizer.TransformParameters identity = new( + quantizer, mode, Av1TransformType.Identity, Av1TransformSize.Size4x4, Av1Plane.Y); + + Av1InverseQuantizer.TransformParameters horizontal = new( + quantizer, mode, Av1TransformType.HorizontalAdst, Av1TransformSize.Size4x4, Av1Plane.Y); + + Assert.Equal(11 * ac, identity.Dequantize(11, 1, false)); + Assert.Equal(11 * ac, horizontal.Dequantize(11, 1, false)); + } + + [Fact] + public void DequantizationPreservesProductMaskAndTransformRounding() + { + ObuSequenceHeader sequenceHeader = new() + { + ColorConfig = new ObuColorConfig { BitDepth = Av1BitDepth.EightBit } + }; + + ObuFrameHeader frameHeader = new(); + frameHeader.QuantizationParameters.BaseQIndex = 23; + frameHeader.QuantizationParameters.IsUsingQMatrix = true; + frameHeader.SegmentationParameters.QMLevel[0][0] = 0; + Av1InverseQuantizer quantizer = new(sequenceHeader, frameHeader); + Av1BlockModeInfo mode = new(Av1BlockSize.Block64x64, Point.Empty); + Av1InverseQuantizer.TransformParameters matrix = new( + quantizer, mode, Av1TransformType.DctDct, Av1TransformSize.Size4x4, Av1Plane.Y); + + // The final 4x4 matrix weight is 200, giving AC=188. Its product with 89241 is 2^24 + 92: + // retaining the 24-bit intermediate must produce 92, rather than saturating the unmasked product. + Assert.Equal(92, matrix.Dequantize(89241, 15, false)); + Assert.Equal(-92, matrix.Dequantize(89241, 15, true)); + + frameHeader.QuantizationParameters.IsUsingQMatrix = false; + Av1InverseQuantizer.TransformParameters scaled32 = new( + quantizer, mode, Av1TransformType.DctDct, Av1TransformSize.Size32x32, Av1Plane.Y); + + Av1InverseQuantizer.TransformParameters scaled64 = new( + quantizer, mode, Av1TransformType.DctDct, Av1TransformSize.Size64x64, Av1Plane.Y); + + // A magnitude of 11 with AC=30 gives 330 before scaling. Sign follows truncation of the positive + // magnitude, so the negative 64x64 result is -82 rather than the arithmetic-right-shift result -83. + Assert.Equal(165, scaled32.Dequantize(11, 1, false)); + Assert.Equal(82, scaled64.Dequantize(11, 1, false)); + Assert.Equal(-82, scaled64.Dequantize(11, 1, true)); + + frameHeader.QuantizationParameters.BaseQIndex = 0; + frameHeader.QuantizationParameters.IsUsingQMatrix = true; + frameHeader.LosslessArray[0] = true; + Av1InverseQuantizer losslessQuantizer = new(sequenceHeader, frameHeader); + Av1InverseQuantizer.TransformParameters lossless = new( + losslessQuantizer, mode, Av1TransformType.DctDct, Av1TransformSize.Size4x4, Av1Plane.Y); + + Assert.Equal(44, lossless.Dequantize(11, 1, false)); + } + [Fact] public void MatricesCoverAllLevelsPlanesAndTransformSizes() {