Browse Source

Use implicit transforms for AV1 chroma

pull/2633/head
James Jackson-South 1 month ago
parent
commit
06352cc5dc
  1. 3
      HEIF_IMPLEMENTATION_PLAN.md
  2. 22
      src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolContextHelper.cs
  3. 5
      src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolDecoder.cs
  4. 11
      src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1IntraSuperblockEncoder.ChromaModeDecision.cs
  5. 1
      src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1IntraSuperblockEncoder.ModeDecision.cs
  6. 8
      src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1IntraSuperblockEncoder.Operator.cs
  7. 37
      tests/ImageSharp.Tests/Formats/Heif/Av1/Av1IntraSuperblockEncoderTests.cs
  8. 31
      tests/ImageSharp.Tests/Formats/Heif/Av1/Av1SymbolContextTests.cs

3
HEIF_IMPLEMENTATION_PLAN.md

@ -843,7 +843,8 @@ Encoder verification contract:
- [~] The combined-frame writer now completes the byte-counted uncompressed frame header before starting the optional multi-tile tile-group flag, matching current libaom's separate frame-header and tile-group writers. A non-uniform two-tile round trip verifies the explicit boundaries, both tile payloads, and complete stream consumption through direct net11 VSTest in Release.
- [~] The first internal frame-to-OBU operation encodes 8-, 10-, and 12-bit monochrome reduced still pictures through the production tile writer and production decoder. Coefficient context initialization now stores `min(abs(level), 127)`, matching current libaom; the previous signed clamp converted every negative transform coefficient to zero and selected invalid nonzero-map distributions. Signed dense and sparse entropy round trips, direct level-buffer saturation coverage, and eight constant/gradient frame cases pass 52 of 52 direct net11 VSTest cases in Release. Current-main `aomdec` accepts all eight emitted payloads. Their decoded-frame MD5 values are `d09ea148582b9c93fa78e59426193bbc` (16x16 8-bit constant), `700698094fb922d38361e04d56e73aa6` (16x16 8-bit gradient), `f949f7422913e83dff07ee5e0a5087d3` (8x8 8-bit constant), `ae7233a94558978934469dcc4da764dd` (8x8 8-bit gradient), `09223b227f3abc3134d0a3ea15f70c0a` (8x8 10-bit constant), `539aab0e6e14bcaec271febfa8e25444` (8x8 10-bit gradient), `73117a8fc102e5d028f82444fc4d15ab` (8x8 12-bit constant), and `3cbecfd4eb5b8aeb7520e57927d6b6ba` (8x8 12-bit gradient). This is an independently decodable baseline, not completion evidence for chroma, alpha, options, containers, or the public encoder.
- [x] The exact net11 Release rebuild completed at the established 1,005-warning repository baseline with zero errors. The complete HEIF/AV1 namespace passes 8,838 of 8,838 direct VSTest cases with zero failures or skips. Roslynk reports zero compiler errors and no diagnostics in the five changed C# files; `git diff --check` passes and `.gitattributes` is unchanged.
- [~] The same internal frame operation now produces 4:2:0, 4:2:2, and 4:4:4 payloads at 8, 10, and 12 bits. Twenty-one color cases cover constant and spatially varying input at aligned dimensions plus odd 13x11 visible dimensions for every chroma geometry. The production decoder consumes every payload, the decoded output retains non-neutral chroma, and current-main `aomdec` accepts all 29 monochrome and color outputs. After live spatial chroma mode selection, the odd-dimension decoded-frame MD5 values are `24743606d07dd9c932df8a459b146ea2` (4:2:0), `335d523be6f3ca0cfcfc16a90917fa96` (4:2:2), and `0343234d3f2774fc2f881300c8f013d5` (4:4:4). This proves legal current-libaom payload syntax across native plane geometries; it does not yet prove target quality or native-plane equality with an independently encoded reference.
- [~] The same internal frame operation now produces 4:2:0, 4:2:2, and 4:4:4 payloads at 8, 10, and 12 bits. Twenty-one color cases cover constant and spatially varying input at aligned dimensions plus odd 13x11 visible dimensions for every chroma geometry. The production decoder consumes every payload, the decoded output retains non-neutral chroma, and current-main `aomdec` accepts all 29 monochrome and color outputs. After live spatial chroma mode selection and implicit chroma-transform correction, the odd-dimension decoded-frame MD5 values are `1721dc6a018892e7fca5d4e003be1058` (4:2:0), `90dfa2d705410c86bff3ba750efec0ac` (4:2:2), and `dbd15340e2cb7d2f731c74eb1b885e0d` (4:4:4). This proves legal current-libaom payload syntax across native plane geometries; it does not yet prove target quality or native-plane equality with an independently encoded reference.
- [x] Spatial chroma candidates now use the implicit transform derived from the selected UV mode and active transform set, matching current libaom's `intra_mode_to_tx_type` and `av1_get_tx_type` behavior. The same shared derivation is consumed by the decoder, so coefficient scan order, entropy contexts, inverse reconstruction, and encoder rate estimates cannot drift between the two paths. The previous DCT-DCT candidate transform could produce syntactically accepted streams whose non-DC chroma coefficients were interpreted under a different implicit transform. Six production mode-decision cases retain nonzero U and V coefficients and assert the selected transform state across 4:2:0, 4:2:2, and 4:4:4; fifteen exact mapping cases cover every intra mode, reduced sets, and the 32x32 DCT-only fallback. The focused contract passes 21 of 21 direct net11 VSTest cases, the complete HEIF/AV1 namespace passes 8,947 of 8,947, the exact Release rebuild remains at 1,005 warnings and zero errors, and current-main `aomdec` accepts all 29 regenerated payloads.
- [x] The expanded checkpoint exposed a pre-existing transform-block test that asserted uninitialized pooled padding was zero. The test now initializes the complete physical luma plane with a sentinel and proves the block operation leaves both adjacent padding samples unchanged. The exact net11 Release rebuild remains at 1,005 baseline warnings and zero errors, the focused allocator-order set passes 30 of 30 cases, and the complete HEIF/AV1 namespace passes 8,859 of 8,859 direct VSTest cases with zero failures or skips.
- [x] Combined-frame OBU output now counts the byte-aligned frame and tile-group headers, non-final tile-size fields, and owned tile payloads before emitting the OBU size. It retains only the small allocator-owned header scratch and writes each entropy-coded tile span directly from its detached owner, removing the second file-sized allocator rent and complete-payload copy. A 64 KiB regression proves exactly one sub-payload-sized byte rent with a balanced return and verifies the exact streamed tile tail; the existing two-tile round trip proves size-prefix and ordering parity. The focused writer and production-frame set passes 32 of 32 direct net11 VSTest cases, current-main `aomdec` accepts all 29 generated native-format payloads, and the complete HEIF/AV1 namespace passes 8,860 of 8,860 cases with zero failures or skips.
- [x] Finalized fixed-block decisions now set the block-level transform-skip flag only when every retained luma and coded chroma transform has zero EOB, matching current libaom's conjunction of per-plane skip state. The previous always-false flag produced legal but redundant non-skip and zero-coefficient syntax. Monochrome and 4:2:0 regressions prove both branches from actual coefficient state; the focused decision and production-frame set passes 32 of 32 direct net11 VSTest cases. Current-main `aomdec` accepts all 29 regenerated payloads, the recorded decoded-frame MD5s are unchanged, and affected 16x16 constant 8-bit and 10-bit payloads are one byte smaller. The complete HEIF/AV1 namespace passes 8,862 of 8,862 cases with zero failures or skips.

22
src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolContextHelper.cs

@ -543,18 +543,22 @@ internal static class Av1SymbolContextHelper
}
/// <summary>
/// Maps an intra prediction mode to its default transform type.
/// Gets the implicit intra transform type after applying the active transform-set restriction.
/// </summary>
/// <param name="modeInfo">The block prediction modes.</param>
/// <param name="planeType">The luma or chroma plane category.</param>
/// <returns>The transform type associated with the selected prediction mode.</returns>
public static Av1TransformType ConvertIntraModeToTransformType(Av1BlockModeInfo modeInfo, Av1PlaneType planeType)
/// <param name="mode">The intra prediction mode.</param>
/// <param name="transformSize">The coded transform size.</param>
/// <param name="useReducedSet">Indicates whether the frame restricts transform choices.</param>
/// <returns>The implicit transform type used to encode and decode the block.</returns>
public static Av1TransformType GetImplicitIntraTransformType(
Av1PredictionMode mode,
Av1TransformSize transformSize,
bool useReducedSet)
{
// the reference decoder's get_uv_mode() is the explicit boundary between the distinct UV and luma prediction domains. CfL maps
// to DC because the chroma AC contribution is applied to a DC predictor before coefficient reconstruction.
Av1PredictionMode mode = planeType == Av1PlaneType.Y ? modeInfo.YMode : modeInfo.UvMode.ToLumaMode();
Av1TransformType transformType = mode.ToTransformType();
Av1TransformSetType transformSetType = GetExtendedTransformSetType(transformSize, useReducedSet);
return mode.ToTransformType();
// An implicit mode-derived transform falls back to DCT-DCT when its transform set omits that type.
return transformType.IsExtendedSetUsed(transformSetType) ? transformType : Av1TransformType.DctDct;
}
/// <summary>

5
src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolDecoder.cs

@ -1458,7 +1458,10 @@ internal ref struct Av1SymbolDecoder
else
{
// Chroma has its own intra mode, so its implicit transform must be derived independently of luma.
transformType = Av1SymbolContextHelper.ConvertIntraModeToTransformType(modeInfo, Av1PlaneType.Uv);
transformType = Av1SymbolContextHelper.GetImplicitIntraTransformType(
modeInfo.UvMode.ToLumaMode(),
transformSize,
useReducedTransformSet);
}
}

11
src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1IntraSuperblockEncoder.ChromaModeDecision.cs

@ -262,6 +262,11 @@ internal static partial class Av1IntraSuperblockEncoder
{
const Av1BlockSize BlockSize = Av1BlockSize.Block8x8;
Av1PredictionMode predictionMode = chromaMode.ToLumaMode();
Av1TransformType transformType = Av1SymbolContextHelper.GetImplicitIntraTransformType(
predictionMode,
transformSize,
this.picture.Parent.FrameHeader.UseReducedTransformSet);
long distortion = TOperator.EncodeCandidate(
this.blockWorkspace,
blueSource,
@ -275,6 +280,7 @@ internal static partial class Av1IntraSuperblockEncoder
angleDelta,
candidateBlueCoefficients,
transformSize,
transformType,
Av1Plane.U,
this.quantization.QIndex[0],
this.quantization.DeltaQDc[(int)Av1Plane.U],
@ -295,6 +301,7 @@ internal static partial class Av1IntraSuperblockEncoder
angleDelta,
candidateRedCoefficients,
transformSize,
transformType,
Av1Plane.V,
this.quantization.QIndex[0],
this.quantization.DeltaQDc[(int)Av1Plane.V],
@ -314,7 +321,7 @@ internal static partial class Av1IntraSuperblockEncoder
rate += writer.GetCoefficientCost(
transformSize,
Av1TransformType.DctDct,
transformType,
lumaMode,
candidateBlueCoefficients,
Av1ComponentType.Chroma,
@ -325,7 +332,7 @@ internal static partial class Av1IntraSuperblockEncoder
rate += writer.GetCoefficientCost(
transformSize,
Av1TransformType.DctDct,
transformType,
lumaMode,
candidateRedCoefficients,
Av1ComponentType.Chroma,

1
src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1IntraSuperblockEncoder.ModeDecision.cs

@ -464,6 +464,7 @@ internal static partial class Av1IntraSuperblockEncoder
angleDelta,
candidateCoefficients,
TransformSize,
Av1TransformType.DctDct,
Av1Plane.Y,
this.quantization.QIndex[0],
this.quantization.DeltaQDc[(int)Av1Plane.Y],

8
src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1IntraSuperblockEncoder.Operator.cs

@ -88,6 +88,7 @@ internal static partial class Av1IntraSuperblockEncoder
/// <param name="angleDelta">The signed directional-angle adjustment.</param>
/// <param name="quantizedCoefficients">The candidate entropy-coding coefficients.</param>
/// <param name="transformSize">The transform dimensions.</param>
/// <param name="transformType">The compound transform applied to the residual.</param>
/// <param name="plane">The component plane containing the block.</param>
/// <param name="qIndex">The effective segment quantizer index.</param>
/// <param name="dcDeltaQ">The plane DC quantizer adjustment.</param>
@ -108,6 +109,7 @@ internal static partial class Av1IntraSuperblockEncoder
int angleDelta,
Span<int> quantizedCoefficients,
Av1TransformSize transformSize,
Av1TransformType transformType,
Av1Plane plane,
int qIndex,
int dcDeltaQ,
@ -178,6 +180,7 @@ internal static partial class Av1IntraSuperblockEncoder
int angleDelta,
Span<int> quantizedCoefficients,
Av1TransformSize transformSize,
Av1TransformType transformType,
Av1Plane plane,
int qIndex,
int dcDeltaQ,
@ -197,7 +200,7 @@ internal static partial class Av1IntraSuperblockEncoder
angleDelta,
quantizedCoefficients,
transformSize,
Av1TransformType.DctDct,
transformType,
qIndex,
dcDeltaQ,
acDeltaQ,
@ -268,6 +271,7 @@ internal static partial class Av1IntraSuperblockEncoder
int angleDelta,
Span<int> quantizedCoefficients,
Av1TransformSize transformSize,
Av1TransformType transformType,
Av1Plane plane,
int qIndex,
int dcDeltaQ,
@ -287,7 +291,7 @@ internal static partial class Av1IntraSuperblockEncoder
angleDelta,
quantizedCoefficients,
transformSize,
Av1TransformType.DctDct,
transformType,
qIndex,
dcDeltaQ,
acDeltaQ,

37
tests/ImageSharp.Tests/Formats/Heif/Av1/Av1IntraSuperblockEncoderTests.cs

@ -797,21 +797,23 @@ public class Av1IntraSuperblockEncoderTests
}
[Theory]
[InlineData((int)Av1ChromaPredictionMode.Vertical, 0, (int)Av1ColorFormat.Yuv444)]
[InlineData((int)Av1ChromaPredictionMode.Horizontal, 0, (int)Av1ColorFormat.Yuv420)]
[InlineData((int)Av1ChromaPredictionMode.Paeth, 0, (int)Av1ColorFormat.Yuv422)]
[InlineData((int)Av1ChromaPredictionMode.Directional45Degrees, -3, (int)Av1ColorFormat.Yuv420)]
[InlineData((int)Av1ChromaPredictionMode.Directional135Degrees, 3, (int)Av1ColorFormat.Yuv422)]
[InlineData((int)Av1ChromaPredictionMode.Directional203Degrees, -3, (int)Av1ColorFormat.Yuv444)]
[InlineData((int)Av1ChromaPredictionMode.Vertical, 0, (int)Av1TransformType.AdstDct, (int)Av1ColorFormat.Yuv444)]
[InlineData((int)Av1ChromaPredictionMode.Horizontal, 0, (int)Av1TransformType.DctAdst, (int)Av1ColorFormat.Yuv420)]
[InlineData((int)Av1ChromaPredictionMode.Paeth, 0, (int)Av1TransformType.AdstAdst, (int)Av1ColorFormat.Yuv422)]
[InlineData((int)Av1ChromaPredictionMode.Directional45Degrees, -3, (int)Av1TransformType.DctDct, (int)Av1ColorFormat.Yuv420)]
[InlineData((int)Av1ChromaPredictionMode.Directional135Degrees, 3, (int)Av1TransformType.AdstAdst, (int)Av1ColorFormat.Yuv422)]
[InlineData((int)Av1ChromaPredictionMode.Directional203Degrees, -3, (int)Av1TransformType.DctAdst, (int)Av1ColorFormat.Yuv444)]
public void ProductionTileSelectsChromaModeFromCurrentReconstruction(
int expectedModeValue,
int expectedAngleDelta,
int expectedTransformTypeValue,
int colorFormatValue)
{
const int Width = 16;
const int Height = 16;
const int QIndex = 1;
Av1ChromaPredictionMode expectedMode = (Av1ChromaPredictionMode)expectedModeValue;
Av1TransformType expectedTransformType = (Av1TransformType)expectedTransformTypeValue;
Av1ColorFormat colorFormat = (Av1ColorFormat)colorFormatValue;
bool subsamplingX = colorFormat is Av1ColorFormat.Yuv420 or Av1ColorFormat.Yuv422;
bool subsamplingY = colorFormat == Av1ColorFormat.Yuv420;
@ -896,6 +898,19 @@ public class Av1IntraSuperblockEncoderTests
expectedAngleDelta,
superblockWorkspace.FinalBlocks[3].PredictionUnit.AngleDelta[(int)Av1PlaneType.Uv]);
int targetTransformIndex = (3 * transformSize.GetSize2d()) /
Av1EncoderCoefficientBuffer.TransformBlockUnitCoefficientCount;
Av1EncoderTransformBlockState blueState =
coefficients.GetTransformBlockSpan(0, Av1Plane.U)[targetTransformIndex];
Av1EncoderTransformBlockState redState =
coefficients.GetTransformBlockSpan(0, Av1Plane.V)[targetTransformIndex];
Assert.NotEqual((ushort)0, blueState.EndOfBlock);
Assert.NotEqual((ushort)0, redState.EndOfBlock);
Assert.Equal(expectedTransformType, blueState.TransformType);
Assert.Equal(expectedTransformType, redState.TransformType);
Assert.NotEqual(0, tileWriter.GetTileData(0).Length);
}
@ -1251,8 +1266,8 @@ public class Av1IntraSuperblockEncoderTests
}
}
// The first three transform-sized quadrants establish the top, left, and corner reconstruction
// consumed by the bottom-right target during the real tile traversal.
// The first three transform-sized quadrants establish the references consumed by the bottom-right
// target. Its checkerboard offset keeps coefficients nonzero so the implicit transform affects the stream.
for (int row = 0; row < plane.Height; row++)
{
Span<byte> destination = plane.DangerousGetRowSpan(row);
@ -1262,7 +1277,11 @@ public class Av1IntraSuperblockEncoderTests
? column < width ? (byte)128 : above[column - width]
: column < width
? left[row - height]
: target[((row - height) * width) + column - width];
: (byte)Math.Clamp(
target[((row - height) * width) + column - width] +
((((row - height) + column - width) & 1) == 0 ? 5 : -5),
0,
255);
}
}
}

31
tests/ImageSharp.Tests/Formats/Heif/Av1/Av1SymbolContextTests.cs

@ -4,6 +4,7 @@
using Microsoft.Diagnostics.Symbols;
using SixLabors.ImageSharp.Formats.Heif.Av1;
using SixLabors.ImageSharp.Formats.Heif.Av1.Entropy;
using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction;
using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling;
using SixLabors.ImageSharp.Formats.Heif.Av1.Transform;
@ -45,6 +46,36 @@ public class Av1SymbolContextTests
Assert.Equal(actualIndex, index);
}
[Theory]
[InlineData((int)Av1PredictionMode.DC, (int)Av1TransformSize.Size8x8, false, (int)Av1TransformType.DctDct)]
[InlineData((int)Av1PredictionMode.Vertical, (int)Av1TransformSize.Size8x8, false, (int)Av1TransformType.AdstDct)]
[InlineData((int)Av1PredictionMode.Horizontal, (int)Av1TransformSize.Size8x8, false, (int)Av1TransformType.DctAdst)]
[InlineData((int)Av1PredictionMode.Directional45Degrees, (int)Av1TransformSize.Size8x8, false, (int)Av1TransformType.DctDct)]
[InlineData((int)Av1PredictionMode.Directional135Degrees, (int)Av1TransformSize.Size8x8, false, (int)Av1TransformType.AdstAdst)]
[InlineData((int)Av1PredictionMode.Directional113Degrees, (int)Av1TransformSize.Size8x8, false, (int)Av1TransformType.AdstDct)]
[InlineData((int)Av1PredictionMode.Directional157Degrees, (int)Av1TransformSize.Size8x8, false, (int)Av1TransformType.DctAdst)]
[InlineData((int)Av1PredictionMode.Directional203Degrees, (int)Av1TransformSize.Size8x8, false, (int)Av1TransformType.DctAdst)]
[InlineData((int)Av1PredictionMode.Directional67Degrees, (int)Av1TransformSize.Size8x8, false, (int)Av1TransformType.AdstDct)]
[InlineData((int)Av1PredictionMode.Smooth, (int)Av1TransformSize.Size8x8, false, (int)Av1TransformType.AdstAdst)]
[InlineData((int)Av1PredictionMode.SmoothVertical, (int)Av1TransformSize.Size8x8, false, (int)Av1TransformType.AdstDct)]
[InlineData((int)Av1PredictionMode.SmoothHorizontal, (int)Av1TransformSize.Size8x8, false, (int)Av1TransformType.DctAdst)]
[InlineData((int)Av1PredictionMode.Paeth, (int)Av1TransformSize.Size8x8, false, (int)Av1TransformType.AdstAdst)]
[InlineData((int)Av1PredictionMode.Directional135Degrees, (int)Av1TransformSize.Size8x8, true, (int)Av1TransformType.AdstAdst)]
[InlineData((int)Av1PredictionMode.Directional135Degrees, (int)Av1TransformSize.Size32x32, false, (int)Av1TransformType.DctDct)]
public void ImplicitIntraTransformTypeMatchesCurrentLibaom(
int modeValue,
int transformSizeValue,
bool useReducedSet,
int expectedValue)
{
Av1TransformType actual = Av1SymbolContextHelper.GetImplicitIntraTransformType(
(Av1PredictionMode)modeValue,
(Av1TransformSize)transformSizeValue,
useReducedSet);
Assert.Equal((Av1TransformType)expectedValue, actual);
}
public static TheoryData<int, int, int> GetLowLevelContextEndOfBlockData()
{
TheoryData<int, int, int> result = [];

Loading…
Cancel
Save