diff --git a/HEIF_IMPLEMENTATION_PLAN.md b/HEIF_IMPLEMENTATION_PLAN.md index 43ea1e813..5c4eb4b57 100644 --- a/HEIF_IMPLEMENTATION_PLAN.md +++ b/HEIF_IMPLEMENTATION_PLAN.md @@ -72,6 +72,7 @@ This snapshot pins or classifies the available references and failures; it does | `Av1PredictionDecoder`, `Av1HighBitDepthPredictor`, `Av1ChromaFromLumaContext`, `Av1PartitionInfo`, and the scalar DC, directional, Paeth, smooth, filter-intra, and chroma-from-luma predictors | AV1 sections 7.11.2 and 7.11.2.3 intra prediction | libaom `aom_dsp/intrapred.c`, `av1/common/reconintra.c`, `av1/common/av1_common_int.h`, `av1/common/blockd.h`, `av1/common/cfl.c`, and `av1/common/cfl.h` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Behavioral oracle for luma/chroma mode-neighbor addressing, directional upsampling, Paeth selection, smooth normalization, filter-intra taps, high-bit-depth clipping, chroma-from-luma storage/subsampling, and chroma-from-luma row strides. Existing managed scalar tables and predictors remain the implementation base. The WIP rectangular byte-pipeline smooth digest expectations encode width/height-swapped weights and must be replaced only from an independently generated oracle, not regenerated from this implementation. | | `Av1TileReader` palette mode/color-map parsing, `Av1SymbolDecoder` palette distributions, `Av1BlockModeInfo` palette state, and `Av1PredictionDecoder` palette reconstruction | AV1 sections 5.11.46, 5.11.49, and 7.11.2 palette prediction | libaom `av1/decoder/decodemv.c`, `av1/decoder/detokenize.c`, `av1/decoder/decoder.h`, `av1/common/pred_common.c`, `av1/common/pred_common.h`, and `av1/common/entropymode.c` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Reuse the existing tile range decoder and frame-owned mode map while porting the normative palette cache merge, high-bit-depth color deltas, tile-adaptive mode/size/index distributions, diagonal color-map traversal, edge padding, and direct palette-sample reconstruction. This is AV1 still-image compression syntax and does not add retained video reference state or any ISO BMFF surface. | | `Av1LoopFilterKernels`, `Av1LoopFilterContext`, and `Av1LoopFilterDecoder` | AV1 section 7.14 deblocking loop filter | libaom `aom_dsp/loopfilter.c` and `av1/common/av1_loopfilter.c` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Port the scalar 4-, 6-, 8-, and 14-tap low/high-bit-depth filters, sharpness thresholds, still-frame intra filter-level derivation, transform-edge selection, and plane traversal before enabling the stage. Later SIMD must preserve the scalar result. This is normative AV1 image reconstruction and adds neither generic ISO BMFF models nor retained video reference state. | +| `Av1CdefDecoder`, `Av1CdefKernels`, and CDEF-unit strength storage | AV1 sections 7.15.2 through 7.15.4 constrained directional enhancement filtering | libaom `av1/common/cdef.c`, `av1/common/cdef_block.c`, `av1/common/cdef.h`, and `av1/common/cdef_block.h` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Port the scalar direction search, variance adjustment, constrained primary/secondary taps, subsampling direction conversion, skipped-8x8 selection, and frame-edge sentinel behavior. Use a frame-owned source snapshot so filtering never consumes already modified samples. This is normative AV1 still-image reconstruction and introduces no ISO BMFF, track, timing, or sequence-playback surface. | | `Av1FrameInfo`, `Av1TileReader`, and `Av1BlockDecoder` transform/coefficient storage | AV1 section 5.11.39 coefficient syntax and section 7.11.2 reconstruction | libaom `av1/decoder/decodetxb.c` and `av1/decoder/decoder.h` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Preserve separate luma and chroma transform coefficients at monotonically advancing per-plane offsets within each superblock so reconstruction consumes the same transform-block order produced by tile parsing. | | `Av1InverseQuantizer` and `Av1InverseQuantizationLookup` | AV1 section 7.12.3 inverse quantization | libaom `aom_dsp/aom_dsp_common.h`, `av1/common/quant_common.c`, and `av1/decoder/decodetxb.c` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Select the per-segment matrix level, alias 64-pixel transform dimensions to their adjusted matrices, retain a flat level-15 matrix, and apply the five-bit inverse-matrix weight scale. The large managed lookup remains a single process-wide table. | | `Av1Inverse2dTransformer` and `Av1InverseTransformerFactory` | AV1 section 7.11.2 inverse transform and reconstruction | libaom `av1/common/av1_inv_txfm1d.c`, `av1/common/av1_inv_txfm2d.c`, and `av1/common/idct.c` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Scalar transform oracle for coefficient-row traversal, intermediate layout, stage ranges, clipping, and high-bit-depth sample addition. The managed 16-bit overload is also used as a parity oracle for the byte overload. | @@ -119,7 +120,7 @@ This assessment is based on the current source after the upstream ImageSharp mer ### AV1 decoder - The single-still `Av1Decoder` path now parses tile state before allocating and reconstructing one independently decodable frame, and it disposes the reconstruction planes after pixel conversion. It deliberately does not retain animation/video reference frames or implement `show_existing_frame` playback state. -- The reconstruction pipeline now records plane-relative transform geometry, preserves tile-local delta-Q and delta-LF predictors, derives segmentation and reference-adjusted filter levels, and runs the exact scalar low/high-bit-depth AV1 4-, 6-, 8-, and 14-tap deblocking kernels in normative vertical-then-horizontal order. The available full-decode AV1 fixture has no internal edge, so independent low/high-bit-depth AVIF vectors with filtered transform boundaries remain required before deblocking has external pixel-level verification. CDEF, super-resolution, loop restoration, and padding remain disabled even though they are normative when signaled, not optional quality improvements. +- The reconstruction pipeline now records plane-relative transform geometry, preserves tile-local delta-Q and delta-LF predictors, derives segmentation and reference-adjusted filter levels, and runs the exact scalar low/high-bit-depth AV1 4-, 6-, 8-, and 14-tap deblocking kernels in normative vertical-then-horizontal order. It then applies scalar CDEF direction search, luma variance adjustment, primary and secondary constrained taps, chroma direction conversion, high-bit-depth scaling, skipped-block selection, and frame-edge sentinel handling from immutable per-plane snapshots. The available full-decode AV1 fixture does not exercise filtered transform edges or active CDEF strengths, so independent 8-, 10-, and 12-bit AVIF vectors remain required before either stage has external pixel-level verification. Super-resolution, loop restoration, and padding remain disabled even though they are normative when signaled, not optional quality improvements. - Palette mode now reads the normative luma/chroma mode and size CDFs, neighbor color caches, high-bit-depth color syntax, diagonal color-index maps, clipped-edge padding, and direct sample prediction through the existing reconstruction pipeline. The scalar implementation matches the pinned libaom source, but an independently encoded palette AVIF fixture is still required before this path is independently verified. - Loop restoration and other normative independently decodable still-image syntax paths still contain `NotImplementedException` or equivalent unsupported branches. Tile-local palette CDF adaptation is now present; the remaining still-image frame-context behavior requires a separate source audit without introducing sequence playback state. - The frame buffer now establishes two-byte native sample storage, logical plane rows, and sample-unit block strides for 10/12-bit frames. The active intra-prediction, inverse-transform, and block-reconstruction path selects native 16-bit samples for 10/12-bit frames and has focused pipeline wiring coverage. Independently encoded high-bit-depth AVIF conformance files are still required; chroma-from-luma storage and the other normative reconstruction stages listed below remain incomplete. diff --git a/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1FrameDecoder.cs b/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1FrameDecoder.cs index a9e66f84a..707d95cf8 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1FrameDecoder.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1FrameDecoder.cs @@ -2,6 +2,7 @@ // Licensed under the Six Labors Split License. using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit; +using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.Cdef; using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.LoopFilter; using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.Quantification; using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; @@ -101,7 +102,9 @@ internal class Av1FrameDecoder : IAv1FrameDecoder // LoopRestorationSaveBoundaryLines(false); } - // DecodeCdef(); + Av1CdefDecoder cdefDecoder = new(this.sequenceHeader, this.frameHeader, this.frameInfo, this.frameBuffer); + cdefDecoder.DecodeFrame(); + // SuperResolutionUpscaling(doUpscale); if (doLoopRestoration && doUpscale) { diff --git a/src/ImageSharp/Formats/Heif/Av1/Pipeline/Cdef/Av1CdefDecoder.cs b/src/ImageSharp/Formats/Heif/Av1/Pipeline/Cdef/Av1CdefDecoder.cs new file mode 100644 index 000000000..a178d3d81 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Pipeline/Cdef/Av1CdefDecoder.cs @@ -0,0 +1,352 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.InteropServices; +using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit; +using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.Cdef; + +/// +/// Applies AV1 constrained directional enhancement filtering to a reconstructed still-image frame. +/// +internal class Av1CdefDecoder +{ + /// + /// The width and height of a CDEF unit in 4x4 luma mode-information units. + /// + private const int CdefUnitModeInfoSize = 16; + + /// + /// The number of unavailable samples reserved on each source-plane edge. + /// + private const int SourceBorder = 2; + + /// + /// The sequence-level superblock, bit-depth, and color configuration. + /// + private readonly ObuSequenceHeader sequenceHeader; + + /// + /// The frame dimensions and CDEF strength table. + /// + private readonly ObuFrameHeader frameHeader; + + /// + /// The decoded block skip state and CDEF-unit strength selections. + /// + private readonly Av1FrameInfo frameInfo; + + /// + /// The reconstructed plane samples modified by CDEF. + /// + private readonly Av1FrameBuffer frameBuffer; + + /// + /// Initializes a new instance of the class. + /// + /// The sequence header defining CDEF availability and the color layout. + /// The frame header defining dimensions and CDEF strengths. + /// The decoded block skip state and per-unit strength selections. + /// The deblocked frame samples to filter. + public Av1CdefDecoder( + ObuSequenceHeader sequenceHeader, + ObuFrameHeader frameHeader, + Av1FrameInfo frameInfo, + Av1FrameBuffer frameBuffer) + { + this.sequenceHeader = sequenceHeader; + this.frameHeader = frameHeader; + this.frameInfo = frameInfo; + this.frameBuffer = frameBuffer; + } + + /// + /// Filters every enabled color plane using directions derived from the deblocked luma plane. + /// + public void DecodeFrame() + { + if (!this.sequenceHeader.EnableCdef || this.frameHeader.CodedLossless || this.frameHeader.AllowIntraBlockCopy) + { + return; + } + + ObuConstraintDirectionalEnhancementFilterParameters parameters = this.frameHeader.CdefParameters; + int strengthCount = 1 << parameters.BitCount; + bool hasNonZeroStrength = false; + for (int i = 0; i < strengthCount; i++) + { + if (parameters.YStrength[i] != 0 || + (this.sequenceHeader.ColorConfig.PlaneCount > 1 && parameters.UvStrength[i] != 0)) + { + hasNonZeroStrength = true; + break; + } + } + + if (!hasNonZeroStrength) + { + return; + } + + int lumaBlockColumnCount = this.frameHeader.ModeInfoColumnCount >> 1; + int lumaBlockRowCount = this.frameHeader.ModeInfoRowCount >> 1; + int[] directions = new int[lumaBlockColumnCount * lumaBlockRowCount]; + int[] variances = new int[directions.Length]; + ObuColorConfig colorConfig = this.sequenceHeader.ColorConfig; + + // Luma must be processed first even when its strengths are zero because chroma CDEF + // consumes directions derived from the immutable, deblocked luma source. + for (int planeIndex = 0; planeIndex < colorConfig.PlaneCount; planeIndex++) + { + Av1Plane plane = (Av1Plane)planeIndex; + int subsamplingX = plane != Av1Plane.Y && colorConfig.SubSamplingX ? 1 : 0; + int subsamplingY = plane != Av1Plane.Y && colorConfig.SubSamplingY ? 1 : 0; + this.FilterPlane(plane, subsamplingX, subsamplingY, directions, variances, lumaBlockColumnCount); + } + } + + /// + /// Filters one color plane from an immutable snapshot of its deblocked samples. + /// + /// The color plane to filter. + /// The horizontal chroma subsampling shift. + /// The vertical chroma subsampling shift. + /// The frame-wide luma direction map in 8x8 block order. + /// The frame-wide luma directional-variance map in 8x8 block order. + /// The number of 8x8 blocks in an aligned luma row. + private void FilterPlane( + Av1Plane plane, + int subsamplingX, + int subsamplingY, + int[] directions, + int[] variances, + int lumaBlockColumnCount) + { + int planeWidth = this.frameHeader.ModeInfoColumnCount << (Av1Constants.ModeInfoSizeLog2 - subsamplingX); + int planeHeight = this.frameHeader.ModeInfoRowCount << (Av1Constants.ModeInfoSizeLog2 - subsamplingY); + int sourceStride = planeWidth + (SourceBorder * 2); + + // CDEF output must never become input to a later block. The sentinel border also makes + // frame-edge taps follow AV1 without exposing the frame buffer's prediction padding. + ushort[] source = new ushort[(planeHeight + (SourceBorder * 2)) * sourceStride]; + Array.Fill(source, Av1CdefKernels.VeryLarge); + + Span lowBitDepthDestination = default; + Span highBitDepthDestination = default; + int destinationStride; + if (this.frameBuffer.BytesPerSample == 2) + { + Span signedDestination = this.frameBuffer.DeriveBlockPointer16( + plane, + Point.Empty, + subsamplingX, + subsamplingY, + out destinationStride); + + highBitDepthDestination = MemoryMarshal.Cast(signedDestination); + } + else + { + lowBitDepthDestination = this.frameBuffer.DeriveBlockPointer( + plane, + Point.Empty, + subsamplingX, + subsamplingY, + out destinationStride); + } + + for (int row = 0; row < planeHeight; row++) + { + Span sourceRow = source.AsSpan( + ((row + SourceBorder) * sourceStride) + SourceBorder, + planeWidth); + + int destinationOffset = destinationStride + (row * destinationStride); + if (this.frameBuffer.BytesPerSample == 2) + { + highBitDepthDestination.Slice(destinationOffset, planeWidth).CopyTo(sourceRow); + } + else + { + Span destinationRow = lowBitDepthDestination.Slice(destinationOffset, planeWidth); + for (int column = 0; column < planeWidth; column++) + { + sourceRow[column] = destinationRow[column]; + } + } + } + + ObuConstraintDirectionalEnhancementFilterParameters parameters = this.frameHeader.CdefParameters; + int coefficientShift = Math.Max(this.frameBuffer.BitDepth.GetBitCount() - 8, 0); + int blockWidth = 8 >> subsamplingX; + int blockHeight = 8 >> subsamplingY; + int unitColumnCount = (this.frameHeader.ModeInfoColumnCount + CdefUnitModeInfoSize - 1) / CdefUnitModeInfoSize; + int unitRowCount = (this.frameHeader.ModeInfoRowCount + CdefUnitModeInfoSize - 1) / CdefUnitModeInfoSize; + Span filteredBlock = stackalloc ushort[8 * 8]; + + for (int unitRow = 0; unitRow < unitRowCount; unitRow++) + { + for (int unitColumn = 0; unitColumn < unitColumnCount; unitColumn++) + { + int unitModeInfoRow = unitRow * CdefUnitModeInfoSize; + int unitModeInfoColumn = unitColumn * CdefUnitModeInfoSize; + int strengthIndex = this.GetStrengthIndex(unitModeInfoColumn, unitModeInfoRow); + if (strengthIndex < 0) + { + continue; + } + + int yStrength = parameters.YStrength[strengthIndex]; + int uvStrength = parameters.UvStrength[strengthIndex]; + bool unitNeedsDirections = yStrength != 0 || + (this.sequenceHeader.ColorConfig.PlaneCount > 1 && uvStrength != 0); + + if ((plane == Av1Plane.Y && !unitNeedsDirections) || (plane != Av1Plane.Y && uvStrength == 0)) + { + continue; + } + + int codedStrength = plane == Av1Plane.Y ? yStrength : uvStrength; + int primaryStrength = (codedStrength / 4) << coefficientShift; + int secondaryStrength = codedStrength % 4; + + // The two-bit secondary field leaves value three unused and represents strength four instead. + secondaryStrength += secondaryStrength == 3 ? 1 : 0; + secondaryStrength <<= coefficientShift; + int damping = parameters.Damping + coefficientShift - (plane == Av1Plane.Y ? 0 : 1); + int unitModeInfoRowEnd = Math.Min(unitModeInfoRow + CdefUnitModeInfoSize, this.frameHeader.ModeInfoRowCount); + int unitModeInfoColumnEnd = Math.Min(unitModeInfoColumn + CdefUnitModeInfoSize, this.frameHeader.ModeInfoColumnCount); + + for (int blockModeInfoRow = unitModeInfoRow; blockModeInfoRow < unitModeInfoRowEnd; blockModeInfoRow += 2) + { + for (int blockModeInfoColumn = unitModeInfoColumn; blockModeInfoColumn < unitModeInfoColumnEnd; blockModeInfoColumn += 2) + { + if (this.IsBlockSkipped(blockModeInfoColumn, blockModeInfoRow)) + { + continue; + } + + int lumaBlockRow = blockModeInfoRow >> 1; + int lumaBlockColumn = blockModeInfoColumn >> 1; + int directionIndex = (lumaBlockRow * lumaBlockColumnCount) + lumaBlockColumn; + int planeColumn = (blockModeInfoColumn << Av1Constants.ModeInfoSizeLog2) >> subsamplingX; + int planeRow = (blockModeInfoRow << Av1Constants.ModeInfoSizeLog2) >> subsamplingY; + int sourceOffset = ((planeRow + SourceBorder) * sourceStride) + planeColumn + SourceBorder; + + if (plane == Av1Plane.Y) + { + directions[directionIndex] = Av1CdefKernels.FindDirection( + source, + sourceOffset, + sourceStride, + coefficientShift, + out variances[directionIndex]); + } + + if (codedStrength == 0) + { + continue; + } + + int filteredPrimaryStrength = plane == Av1Plane.Y + ? Av1CdefKernels.AdjustStrength(primaryStrength, variances[directionIndex]) + : primaryStrength; + + if (filteredPrimaryStrength == 0 && secondaryStrength == 0) + { + continue; + } + + // Secondary-only filtering uses direction zero; otherwise chroma remaps the + // luma direction into its asymmetrically subsampled sample grid when required. + int direction = primaryStrength != 0 + ? Av1CdefKernels.ConvertDirection(directions[directionIndex], subsamplingX, subsamplingY) + : 0; + + Av1CdefKernels.FilterBlock( + source, + sourceOffset, + sourceStride, + filteredBlock, + 0, + blockWidth, + filteredPrimaryStrength, + secondaryStrength, + direction, + damping, + damping, + coefficientShift, + blockWidth, + blockHeight); + + int destinationOffset = destinationStride + (planeRow * destinationStride) + planeColumn; + for (int row = 0; row < blockHeight; row++) + { + ReadOnlySpan filteredRow = filteredBlock.Slice(row * blockWidth, blockWidth); + if (this.frameBuffer.BytesPerSample == 2) + { + filteredRow.CopyTo(highBitDepthDestination.Slice(destinationOffset + (row * destinationStride), blockWidth)); + } + else + { + Span destinationRow = lowBitDepthDestination.Slice( + destinationOffset + (row * destinationStride), + blockWidth); + + for (int column = 0; column < blockWidth; column++) + { + destinationRow[column] = (byte)filteredRow[column]; + } + } + } + } + } + } + } + } + + /// + /// Gets the strength-table selection assigned to a 64x64 CDEF unit. + /// + /// The unit's frame-relative column in 4x4 luma units. + /// The unit's frame-relative row in 4x4 luma units. + /// The strength-table index, or minus one when every block in the unit is skipped. + private int GetStrengthIndex(int modeInfoColumn, int modeInfoRow) + { + int superblockModeInfoSize = this.frameInfo.SuperblockModeInfoSize; + Point superblockPosition = new( + modeInfoColumn / superblockModeInfoSize, + modeInfoRow / superblockModeInfoSize); + + int unitColumn = (modeInfoColumn % superblockModeInfoSize) / CdefUnitModeInfoSize; + int unitRow = (modeInfoRow % superblockModeInfoSize) / CdefUnitModeInfoSize; + + // A 128x128 superblock stores four raster-ordered 64x64 selections; the same + // expression naturally resolves to index zero for a 64x64 superblock. + int unitIndex = unitColumn + (unitRow << 1); + return this.frameInfo.GetCdefStrength(superblockPosition)[unitIndex]; + } + + /// + /// Determines whether every 4x4 mode-information block covered by an 8x8 CDEF block is skipped. + /// + /// The block's frame-relative column in 4x4 luma units. + /// The block's frame-relative row in 4x4 luma units. + /// when the complete 8x8 block is skipped; otherwise, . + private bool IsBlockSkipped(int modeInfoColumn, int modeInfoRow) + { + for (int row = 0; row < 2; row++) + { + for (int column = 0; column < 2; column++) + { + if (!this.frameInfo.GetModeInfoAt(new Point(modeInfoColumn + column, modeInfoRow + row)).Skip) + { + return false; + } + } + } + + return true; + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Pipeline/Cdef/Av1CdefKernels.cs b/src/ImageSharp/Formats/Heif/Av1/Pipeline/Cdef/Av1CdefKernels.cs new file mode 100644 index 000000000..d0cbf9ef2 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Pipeline/Cdef/Av1CdefKernels.cs @@ -0,0 +1,321 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.Cdef; + +/// +/// Provides the scalar constrained directional enhancement filter operations defined by AV1. +/// +internal static class Av1CdefKernels +{ + /// + /// The sample value used for neighbors outside the coded frame. + /// + public const ushort VeryLarge = 0x4000; + + /// + /// The horizontal offsets for the nearest primary or secondary taps in each direction. + /// + private static readonly int[] DirectionX1 = [1, 1, 1, 1, 1, 0, 0, 0]; + + /// + /// The vertical offsets for the nearest primary or secondary taps in each direction. + /// + private static readonly int[] DirectionY1 = [-1, 0, 0, 0, 1, 1, 1, 1]; + + /// + /// The horizontal offsets for the furthest primary or secondary taps in each direction. + /// + private static readonly int[] DirectionX2 = [2, 2, 2, 2, 2, 1, 0, -1]; + + /// + /// The vertical offsets for the furthest primary or secondary taps in each direction. + /// + private static readonly int[] DirectionY2 = [-2, -1, 0, 1, 2, 2, 2, 2]; + + /// + /// The primary-tap weights selected by the parity of the unscaled primary strength. + /// + private static readonly int[,] PrimaryTaps = + { + { 4, 2 }, + { 3, 3 } + }; + + /// + /// The secondary-tap weights for the nearest and furthest samples. + /// + private static readonly int[] SecondaryTaps = [2, 1]; + + /// + /// The common multiples used to compare line variance without division. + /// + private static readonly int[] DivisionTable = [0, 840, 420, 280, 210, 168, 140, 120, 105]; + + /// + /// The direction mapping for horizontally subsampled, vertically full-resolution chroma. + /// + private static readonly int[] DirectionMap422 = [7, 0, 2, 4, 5, 6, 6, 6]; + + /// + /// The direction mapping for horizontally full-resolution, vertically subsampled chroma. + /// + private static readonly int[] DirectionMap440 = [1, 2, 2, 2, 3, 4, 6, 0]; + + /// + /// Finds the dominant direction of an 8x8 luma block and its directional variance. + /// + /// The bordered, deblocked source plane. + /// The offset of the block's top-left sample. + /// The number of samples between adjacent source rows. + /// The number of bits above the eight-bit analysis precision. + /// Receives the variance difference between the selected and orthogonal directions. + /// The zero-based AV1 direction index. + public static int FindDirection( + ReadOnlySpan source, + int sourceOffset, + int sourceStride, + int coefficientShift, + out int variance) + { + Span partial = stackalloc int[8 * 15]; + Span cost = stackalloc int[8]; + partial.Clear(); + cost.Clear(); + + for (int row = 0; row < 8; row++) + { + for (int column = 0; column < 8; column++) + { + // Direction analysis deliberately reduces every source to eight-bit precision so its + // strength selection is identical for 8-, 10-, and 12-bit coded images. + int value = (source[sourceOffset + (row * sourceStride) + column] >> coefficientShift) - 128; + partial[(0 * 15) + row + column] += value; + partial[(1 * 15) + row + (column / 2)] += value; + partial[(2 * 15) + row] += value; + partial[(3 * 15) + 3 + row - (column / 2)] += value; + partial[(4 * 15) + 7 + row - column] += value; + partial[(5 * 15) + 3 - (row / 2) + column] += value; + partial[(6 * 15) + column] += value; + partial[(7 * 15) + (row / 2) + column] += value; + } + } + + for (int i = 0; i < 8; i++) + { + cost[2] += partial[(2 * 15) + i] * partial[(2 * 15) + i]; + cost[6] += partial[(6 * 15) + i] * partial[(6 * 15) + i]; + } + + cost[2] *= DivisionTable[8]; + cost[6] *= DivisionTable[8]; + for (int i = 0; i < 7; i++) + { + cost[0] += ((partial[(0 * 15) + i] * partial[(0 * 15) + i]) + + (partial[(0 * 15) + 14 - i] * partial[(0 * 15) + 14 - i])) * DivisionTable[i + 1]; + cost[4] += ((partial[(4 * 15) + i] * partial[(4 * 15) + i]) + + (partial[(4 * 15) + 14 - i] * partial[(4 * 15) + 14 - i])) * DivisionTable[i + 1]; + } + + cost[0] += partial[(0 * 15) + 7] * partial[(0 * 15) + 7] * DivisionTable[8]; + cost[4] += partial[(4 * 15) + 7] * partial[(4 * 15) + 7] * DivisionTable[8]; + for (int direction = 1; direction < 8; direction += 2) + { + for (int i = 0; i < 5; i++) + { + cost[direction] += partial[(direction * 15) + 3 + i] * partial[(direction * 15) + 3 + i]; + } + + cost[direction] *= DivisionTable[8]; + for (int i = 0; i < 3; i++) + { + cost[direction] += ((partial[(direction * 15) + i] * partial[(direction * 15) + i]) + + (partial[(direction * 15) + 10 - i] * partial[(direction * 15) + 10 - i])) * DivisionTable[(2 * i) + 2]; + } + } + + int bestCost = 0; + int bestDirection = 0; + for (int direction = 0; direction < 8; direction++) + { + if (cost[direction] > bestCost) + { + bestCost = cost[direction]; + bestDirection = direction; + } + } + + // Both costs omit the same sum-of-squares term. Their scaled difference is the + // directional variance consumed by AV1's luma strength adjustment. + variance = (bestCost - cost[(bestDirection + 4) & 7]) >> 10; + return bestDirection; + } + + /// + /// Adjusts a luma primary strength according to the directional variance of its 8x8 block. + /// + /// The bit-depth-scaled primary strength. + /// The directional variance returned by . + /// The variance-adjusted primary strength. + public static int AdjustStrength(int strength, int variance) + { + int varianceClass = variance >> 6; + int adjustment = varianceClass != 0 ? Math.Min(Av1Math.MostSignificantBit((uint)varianceClass), 12) : 0; + return variance != 0 ? ((strength * (4 + adjustment)) + 8) >> 4 : 0; + } + + /// + /// Converts a luma direction to the matching chroma direction for asymmetric subsampling. + /// + /// The zero-based luma direction index. + /// The horizontal chroma subsampling shift. + /// The vertical chroma subsampling shift. + /// The direction index in the chroma sample grid. + public static int ConvertDirection(int direction, int subsamplingX, int subsamplingY) + { + if (subsamplingX == subsamplingY) + { + return direction; + } + + return subsamplingX != 0 ? DirectionMap422[direction] : DirectionMap440[direction]; + } + + /// + /// Filters one luma or chroma block from an immutable bordered source plane. + /// + /// The bordered, deblocked source plane. + /// The offset of the block's top-left source sample. + /// The number of samples between adjacent source rows. + /// The unbordered filtered destination plane. + /// The offset of the block's top-left destination sample. + /// The number of samples between adjacent destination rows. + /// The bit-depth-scaled primary strength. + /// The bit-depth-scaled secondary strength. + /// The zero-based AV1 direction index. + /// The damping value applied to primary taps. + /// The damping value applied to secondary taps. + /// The number of bits above eight-bit sample precision. + /// The block width in plane samples. + /// The block height in plane samples. + public static void FilterBlock( + ReadOnlySpan source, + int sourceOffset, + int sourceStride, + Span destination, + int destinationOffset, + int destinationStride, + int primaryStrength, + int secondaryStrength, + int direction, + int primaryDamping, + int secondaryDamping, + int coefficientShift, + int blockWidth, + int blockHeight) + { + bool enablePrimary = primaryStrength != 0; + bool enableSecondary = secondaryStrength != 0; + bool clippingRequired = enablePrimary && enableSecondary; + int primaryTapSet = (primaryStrength >> coefficientShift) & 1; + + for (int row = 0; row < blockHeight; row++) + { + for (int column = 0; column < blockWidth; column++) + { + int sourceIndex = sourceOffset + (row * sourceStride) + column; + int sample = source[sourceIndex]; + int sum = 0; + int minimum = sample; + int maximum = sample; + + for (int tap = 0; tap < 2; tap++) + { + if (enablePrimary) + { + int primaryDirectionOffset = GetDirectionOffset(direction, tap, sourceStride); + int neighbor0 = source[sourceIndex + primaryDirectionOffset]; + int neighbor1 = source[sourceIndex - primaryDirectionOffset]; + int weight = PrimaryTaps[primaryTapSet, tap]; + sum += weight * Constrain(neighbor0 - sample, primaryStrength, primaryDamping); + sum += weight * Constrain(neighbor1 - sample, primaryStrength, primaryDamping); + + if (clippingRequired) + { + maximum = neighbor0 != VeryLarge ? Math.Max(maximum, neighbor0) : maximum; + maximum = neighbor1 != VeryLarge ? Math.Max(maximum, neighbor1) : maximum; + minimum = Math.Min(minimum, neighbor0); + minimum = Math.Min(minimum, neighbor1); + } + } + + if (enableSecondary) + { + int secondaryDirection0 = GetDirectionOffset((direction + 2) & 7, tap, sourceStride); + int secondaryDirection1 = GetDirectionOffset((direction + 6) & 7, tap, sourceStride); + int neighbor0 = source[sourceIndex + secondaryDirection0]; + int neighbor1 = source[sourceIndex - secondaryDirection0]; + int neighbor2 = source[sourceIndex + secondaryDirection1]; + int neighbor3 = source[sourceIndex - secondaryDirection1]; + int weight = SecondaryTaps[tap]; + + if (clippingRequired) + { + maximum = neighbor0 != VeryLarge ? Math.Max(maximum, neighbor0) : maximum; + maximum = neighbor1 != VeryLarge ? Math.Max(maximum, neighbor1) : maximum; + maximum = neighbor2 != VeryLarge ? Math.Max(maximum, neighbor2) : maximum; + maximum = neighbor3 != VeryLarge ? Math.Max(maximum, neighbor3) : maximum; + minimum = Math.Min(minimum, neighbor0); + minimum = Math.Min(minimum, neighbor1); + minimum = Math.Min(minimum, neighbor2); + minimum = Math.Min(minimum, neighbor3); + } + + sum += weight * Constrain(neighbor0 - sample, secondaryStrength, secondaryDamping); + sum += weight * Constrain(neighbor1 - sample, secondaryStrength, secondaryDamping); + sum += weight * Constrain(neighbor2 - sample, secondaryStrength, secondaryDamping); + sum += weight * Constrain(neighbor3 - sample, secondaryStrength, secondaryDamping); + } + } + + // The negative-sum correction preserves AV1's asymmetric signed rounding exactly. + int filtered = sample + ((8 + sum - (sum < 0 ? 1 : 0)) >> 4); + destination[destinationOffset + (row * destinationStride) + column] = + (ushort)(clippingRequired ? Av1Math.Clip3(minimum, maximum, filtered) : filtered); + } + } + } + + /// + /// Converts a direction and tap number to a signed plane-buffer offset. + /// + /// The zero-based AV1 direction index. + /// The zero-based distance index. + /// The number of samples between adjacent rows. + /// The signed sample offset. + private static int GetDirectionOffset(int direction, int tap, int stride) + => tap == 0 + ? (DirectionY1[direction] * stride) + DirectionX1[direction] + : (DirectionY2[direction] * stride) + DirectionX2[direction]; + + /// + /// Limits a neighbor difference according to a filter strength and damping value. + /// + /// The signed difference from the current sample. + /// The bit-depth-scaled filter strength. + /// The damping value. + /// The signed constrained difference. + private static int Constrain(int difference, int threshold, int damping) + { + if (threshold == 0) + { + return 0; + } + + // Stronger thresholds reduce the effective damping shift, matching the AV1 constrain function. + int shift = Math.Max(0, damping - Av1Math.MostSignificantBit((uint)threshold)); + int magnitude = Math.Abs(difference); + int constrained = Av1Math.Clip3(0, magnitude, threshold - (magnitude >> shift)); + return difference < 0 ? -constrained : constrained; + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.cs b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.cs index 29d7777cc..0a3f6276a 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.cs @@ -157,7 +157,7 @@ internal partial class Av1FrameInfo this.quantizerIndices = new int[superblockCount]; // A 128x128 superblock contains four 64x64 CDEF filter blocks; a 64x64 superblock contains one. - this.cdefStrengthFactorLog2 = (superblockSizeLog2 - 6) << 2; + this.cdefStrengthFactorLog2 = (superblockSizeLog2 - 6) << 1; this.cdefStrength = new int[superblockCount << this.cdefStrengthFactorLog2]; Array.Fill(this.cdefStrength, -1); this.deltaLoopFilter = new int[superblockCount << this.deltaLoopFactorLog2]; diff --git a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1PartitionInfo.cs b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1PartitionInfo.cs index bb6862e3c..53b5da77e 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1PartitionInfo.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1PartitionInfo.cs @@ -24,7 +24,6 @@ internal class Av1PartitionInfo this.SuperblockInfo = superblockInfo; this.IsChroma = isChroma; this.Type = partitionType; - this.CdefStrength = []; this.ReferenceFrame = [-1, -1]; this.WidthInPixels = new int[3]; this.HeightInPixels = new int[3]; @@ -100,11 +99,6 @@ internal class Av1PartitionInfo /// public Av1BlockModeInfo? LeftModeInfoForChroma { get; set; } - /// - /// Gets or sets the constrained directional enhancement filter strengths associated with the block. - /// - public int[] CdefStrength { get; set; } - /// /// Gets or sets the reference-frame identifiers selected for the block. /// diff --git a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileReader.cs b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileReader.cs index 706b4626e..0ddaf212d 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileReader.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileReader.cs @@ -1947,7 +1947,7 @@ internal class Av1TileReader : IAv1TileReader /// /// The tile symbol decoder. /// The current coding block. - /// Implements AV1 section 5.11.56 and corresponds to read_cdef in SVT-AV1. + /// Implements AV1 section 5.11.56 and corresponds to read_cdef in libaom. private void ReadCdef(ref Av1SymbolDecoder reader, Av1PartitionInfo partitionInfo) { if (partitionInfo.ModeInfo.Skip || this.FrameHeader.CodedLossless || !this.SequenceHeader.EnableCdef || this.FrameHeader.AllowIntraBlockCopy) @@ -1956,26 +1956,32 @@ internal class Av1TileReader : IAv1TileReader } int cdefSize4 = Av1BlockSize.Block64x64.Get4x4WideCount(); - int row = partitionInfo.RowIndex & cdefSize4; - int col = partitionInfo.ColumnIndex & cdefSize4; - int index = this.SequenceHeader.SuperblockSize == Av1BlockSize.Block128x128 ? Math.Max(1, col) + (Math.Max(1, row) << 1) : 0; - if (partitionInfo.CdefStrength[index] == -1) - { - int cdfStrength = reader.ReadCdfStrength(this.FrameHeader.CdefParameters.BitCount); - partitionInfo.CdefStrength[index] = cdfStrength; - - // A block in a 128x128 superblock can cover multiple 64x64 CDEF units. Replicate the - // first decoded strength so subsequent blocks in every covered unit observe it as assigned. - if (this.SequenceHeader.SuperblockSize == Av1BlockSize.Block128x128) + int superblockMask = this.SequenceHeader.SuperblockModeInfoSize - 1; + int rowInSuperblock = partitionInfo.RowIndex & superblockMask; + int columnInSuperblock = partitionInfo.ColumnIndex & superblockMask; + int unitRow = rowInSuperblock / cdefSize4; + int unitColumn = columnInSuperblock / cdefSize4; + int index = this.SequenceHeader.SuperblockSize == Av1BlockSize.Block128x128 ? unitColumn + (unitRow << 1) : 0; + Span cdefStrength = partitionInfo.SuperblockInfo.CdefStrength; + if (cdefStrength[index] == -1) + { + int cdefStrengthIndex = reader.ReadCdfStrength(this.FrameHeader.CdefParameters.BitCount); + int blockWidth4 = partitionInfo.ModeInfo.BlockSize.Get4x4WideCount(); + int blockHeight4 = partitionInfo.ModeInfo.BlockSize.Get4x4HighCount(); + int lastUnitRow = (rowInSuperblock + blockHeight4 - 1) / cdefSize4; + int lastUnitColumn = (columnInSuperblock + blockWidth4 - 1) / cdefSize4; + + // A coding block can cover the top-left cell of more than one 64x64 CDEF unit. libaom + // stores the index on shared mode information, so the frame-owned unit map must mirror it. + for (int coveredUnitRow = unitRow; coveredUnitRow <= lastUnitRow; coveredUnitRow++) { - int w4 = partitionInfo.ModeInfo.BlockSize.Get4x4WideCount(); - int h4 = partitionInfo.ModeInfo.BlockSize.Get4x4HighCount(); - for (int i = row; i < row + h4; i += cdefSize4) + for (int coveredUnitColumn = unitColumn; coveredUnitColumn <= lastUnitColumn; coveredUnitColumn++) { - for (int j = col; j < col + w4; j += cdefSize4) - { - partitionInfo.CdefStrength[Math.Max(1, j & cdefSize4) + (Math.Max(1, i & cdefSize4) << 1)] = cdfStrength; - } + int coveredIndex = this.SequenceHeader.SuperblockSize == Av1BlockSize.Block128x128 + ? coveredUnitColumn + (coveredUnitRow << 1) + : 0; + + cdefStrength[coveredIndex] = cdefStrengthIndex; } } }