Browse Source

Fix lossless AV1 chroma mode coding

pull/2633/head
James Jackson-South 1 month ago
parent
commit
1afa18feb9
  1. 2
      HEIF_IMPLEMENTATION_PLAN.md
  2. 23
      src/ImageSharp/Formats/Heif/Av1/Av1BlockSizeExtensions.cs
  3. 34
      src/ImageSharp/Formats/Heif/Av1/Prediction/Av1PredictionDecoder.cs
  4. 20
      src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileWriter.cs
  5. 68
      tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CoefficientsEntropyTests.cs

2
HEIF_IMPLEMENTATION_PLAN.md

@ -831,7 +831,7 @@ Encoder verification contract:
- [~] Encoder mode information now uses a frame-owned integer alias grid over a packed 8-byte value allocation, matching current libaom's `mi_grid_base` and `mi_alloc` relationship without a managed object or reference per 4x4 entry. Every coded 4x4 cell covered by square, rectangular, or clipped edge blocks maps to its owning allocation entry before context-dependent symbols are written. At 4K, mode values occupy about 4.0 MiB and the alias grid about 2.0 MiB. Packed syntax, relative neighbor lookup, full block mapping, writer traversal, entropy, and OBU coverage pass 1,947 of 1,947 direct net11 VSTest cases in Release; complete mode decision still remains. - [~] Encoder mode information now uses a frame-owned integer alias grid over a packed 8-byte value allocation, matching current libaom's `mi_grid_base` and `mi_alloc` relationship without a managed object or reference per 4x4 entry. Every coded 4x4 cell covered by square, rectangular, or clipped edge blocks maps to its owning allocation entry before context-dependent symbols are written. At 4K, mode values occupy about 4.0 MiB and the alias grid about 2.0 MiB. Packed syntax, relative neighbor lookup, full block mapping, writer traversal, entropy, and OBU coverage pass 1,947 of 1,947 direct net11 VSTest cases in Release; complete mode decision still remains.
- [~] The final-block decision workspace uses one reusable 10.3 KiB ImageSharp allocator owner. It contains 1,024 explicitly packed 10-byte final-block entries and the 341 preorder partition bytes required by a complete 128x128-through-8x8 quadtree, replacing separate managed arrays. Initial rental is clean, and the explicit per-superblock reset clears the complete owner so pooled palette, quantizer, prediction, and partition bytes cannot leak into the next decision pass. Exact allocation, size, reset, return, repeated-run, writer, entropy, and OBU coverage pass 1,948 of 1,948 direct net11 VSTest cases in Release; complete mode decision still remains. - [~] The final-block decision workspace uses one reusable 10.3 KiB ImageSharp allocator owner. It contains 1,024 explicitly packed 10-byte final-block entries and the 341 preorder partition bytes required by a complete 128x128-through-8x8 quadtree, replacing separate managed arrays. Initial rental is clean, and the explicit per-superblock reset clears the complete owner so pooled palette, quantizer, prediction, and partition bytes cannot leak into the next decision pass. Exact allocation, size, reset, return, repeated-run, writer, entropy, and OBU coverage pass 1,948 of 1,948 direct net11 VSTest cases in Release; complete mode decision still remains.
- [~] Finalized transform coefficients and packed EOB/type state now use raster-ordered, per-superblock plane segments matching current libaom's coefficient-pool geometry. One ImageSharp allocator owner replaces libaom's separate coefficient, EOB, and entropy-context allocations while preserving the full 1024 luma and 256-per-chroma 4x4 state capacity of a 128x128 4:2:0 superblock. The transform-block boundary can populate the owner's quantized coefficient and state slices directly; mode-decision traversal still needs to select and invoke it. - [~] Finalized transform coefficients and packed EOB/type state now use raster-ordered, per-superblock plane segments matching current libaom's coefficient-pool geometry. One ImageSharp allocator owner replaces libaom's separate coefficient, EOB, and entropy-context allocations while preserving the full 1024 luma and 256-per-chroma 4x4 state capacity of a 128x128 4:2:0 superblock. The transform-block boundary can populate the owner's quantized coefficient and state slices directly; mode-decision traversal still needs to select and invoke it.
- [~] Tile partition writing now follows current libaom's recursive `write_modes_sb` preorder traversal and `update_ext_partition_context` edge updates directly. Bottom-edge blocks use the horizontal-alike partition CDF and right-edge blocks use the vertical-alike CDF; byte-exact regressions cover both paths after the previous calls were found reversed. The obsolete SVT-derived global geometry catalog and its unimplemented lookup are removed; transform geometry is derived in libaom's bounded 64x64 residual order, fixed intra transform-size symbols use the reference depth and neighbor contexts, frame-edge and segmentation syntax use mode-information units, and 128x128 CDEF units use libaom's 0-to-3 indexing and first-block strength ownership. Writer, entropy, and OBU coverage passes 1,950 of 1,950 direct net11 VSTest cases in Release. Partition and mode analysis still need to populate these retained decisions; variable inter-transform syntax remains part of later inter-frame support. - [~] Tile partition writing now follows current libaom's recursive `write_modes_sb` preorder traversal and `update_ext_partition_context` edge updates directly. Bottom-edge blocks use the horizontal-alike partition CDF and right-edge blocks use the vertical-alike CDF; byte-exact regressions cover both paths after the previous calls were found reversed. Lossless chroma-from-luma availability now uses the subsampled plane block size shared with the decoder instead of the lossy 32x32 limit, preserving the correct UV-mode alphabet for each segment. The obsolete SVT-derived global geometry catalog and its unimplemented lookup are removed; transform geometry is derived in libaom's bounded 64x64 residual order, fixed intra transform-size symbols use the reference depth and neighbor contexts, frame-edge and segmentation syntax use mode-information units, and 128x128 CDEF units use libaom's 0-to-3 indexing and first-block strength ownership. Writer, entropy, and OBU coverage passes 1,957 of 1,957 direct net11 VSTest cases in Release, with 20 of 20 focused encoder and decoder chroma-from-luma cases. Partition and mode analysis still need to populate these retained decisions; variable inter-transform syntax remains part of later inter-frame support.
- [ ] Implement legal deblocking, CDEF, restoration, super-resolution, and film-grain signaling decisions. - [ ] Implement legal deblocking, CDEF, restoration, super-resolution, and film-grain signaling decisions.
- [~] The coefficient symbol encoder now reuses tile-lifetime level and context workspaces instead of allocating per transform, defers both coefficient rents until the first nonzero transform block, and disposes all tile scratch independently from the detached encoded bytes. Its range coder matches current libaom's 64-bit coding window, bulk big-endian byte flush, and backward carry propagation while using one byte of allocator scratch per estimated output byte instead of the former 16-bit pre-carry storage. The reference-type symbol encoder is passed normally through tile traversal, and the operation boundary owns the allocator-backed item payload stream for exactly one synchronous encode. Every remaining encoder fragment must be audited before it becomes active. - [~] The coefficient symbol encoder now reuses tile-lifetime level and context workspaces instead of allocating per transform, defers both coefficient rents until the first nonzero transform block, and disposes all tile scratch independently from the detached encoded bytes. Its range coder matches current libaom's 64-bit coding window, bulk big-endian byte flush, and backward carry propagation while using one byte of allocator scratch per estimated output byte instead of the former 16-bit pre-carry storage. The reference-type symbol encoder is passed normally through tile traversal, and the operation boundary owns the allocator-backed item payload stream for exactly one synchronous encode. Every remaining encoder fragment must be audited before it becomes active.
- [~] The planar conversion, residual construction, forward transform, and forward quantizer use descending SIMD dispatch: Vector512, Vector256, Vector128, then scalar. Residual construction matches current libaom's exact source-minus-prediction arithmetic for 8-bit and high-bit-depth planes, preserves independent row strides and unaligned starts, and writes directly into caller-owned signed-short storage without allocation. Apply the same rule to every later hot-path family. - [~] The planar conversion, residual construction, forward transform, and forward quantizer use descending SIMD dispatch: Vector512, Vector256, Vector128, then scalar. Residual construction matches current libaom's exact source-minus-prediction arithmetic for 8-bit and high-bit-depth planes, preserves independent row strides and unaligned starts, and writes directly into caller-owned signed-short storage without allocation. Apply the same rule to every later hot-path family.

23
src/ImageSharp/Formats/Heif/Av1/Av1BlockSizeExtensions.cs

@ -179,6 +179,29 @@ internal static class Av1BlockSizeExtensions
return SubSampled[(int)blockSize][subX][subY]; return SubSampled[(int)blockSize][subX][subY];
} }
/// <summary>
/// Determines whether a luma block permits chroma-from-luma prediction.
/// </summary>
/// <param name="blockSize">The luma block size.</param>
/// <param name="isLossless">Indicates whether the block belongs to a lossless segment.</param>
/// <param name="subX">Indicates horizontal chroma subsampling.</param>
/// <param name="subY">Indicates vertical chroma subsampling.</param>
/// <returns><see langword="true"/> when chroma-from-luma prediction is permitted; otherwise, <see langword="false"/>.</returns>
public static bool AllowsChromaFromLuma(
this Av1BlockSize blockSize,
bool isLossless,
bool subX,
bool subY)
{
if (isLossless)
{
// Lossless coding fixes the transform to 4x4, so the subsampled chroma block must have the same dimensions.
return blockSize.GetSubsampled(subX, subY) == Av1BlockSize.Block4x4;
}
return blockSize.GetWidth() <= 32 && blockSize.GetHeight() <= 32;
}
/// <summary> /// <summary>
/// Gets the maximum chroma transform size after applying plane subsampling and AV1 chroma transform limits. /// Gets the maximum chroma transform size after applying plane subsampling and AV1 chroma transform limits.
/// </summary> /// </summary>

34
src/ImageSharp/Formats/Heif/Av1/Prediction/Av1PredictionDecoder.cs

@ -348,7 +348,13 @@ internal sealed class Av1PredictionDecoder
where T : unmanaged, IBinaryInteger<T> where T : unmanaged, IBinaryInteger<T>
{ {
Av1BlockModeInfo modeInfo = partitionInfo.ModeInfo; Av1BlockModeInfo modeInfo = partitionInfo.ModeInfo;
bool isChromaFromLumaAllowedFlag = IsChromaFromLumaAllowedWithFrameHeader(ref partitionInfo, this.sequenceHeader.ColorConfig, this.frameHeader); Av1BlockSize blockSize = modeInfo.BlockSize;
DebugGuard.MustBeLessThan((int)blockSize, (int)Av1BlockSize.AllSizes, nameof(blockSize));
bool isChromaFromLumaAllowedFlag = blockSize.AllowsChromaFromLuma(
this.frameHeader.LosslessArray[modeInfo.SegmentId],
this.sequenceHeader.ColorConfig.SubSamplingX,
this.sequenceHeader.ColorConfig.SubSamplingY);
DebugGuard.IsTrue(isChromaFromLumaAllowedFlag, "Chroma from Luma should be allowed then computing it."); DebugGuard.IsTrue(isChromaFromLumaAllowedFlag, "Chroma from Luma should be allowed then computing it.");
if (chromaFromLumaContext == null) if (chromaFromLumaContext == null)
@ -379,32 +385,6 @@ internal sealed class Av1PredictionDecoder
} }
} }
/// <summary>
/// Determines whether chroma-from-luma prediction is permitted for the current block and frame state.
/// </summary>
/// <param name="partitionInfo">The decoded partition and mode state for the containing block.</param>
/// <param name="colorConfig">The sequence color configuration.</param>
/// <param name="frameHeader">The decoded frame header.</param>
/// <returns><see langword="true"/> when the block may use chroma-from-luma prediction; otherwise, <see langword="false"/>.</returns>
private static bool IsChromaFromLumaAllowedWithFrameHeader(ref Av1PartitionInfo partitionInfo, ObuColorConfig colorConfig, ObuFrameHeader frameHeader)
{
Av1BlockModeInfo modeInfo = partitionInfo.ModeInfo;
Av1BlockSize blockSize = modeInfo.BlockSize;
DebugGuard.MustBeLessThan((int)blockSize, (int)Av1BlockSize.AllSizes, nameof(blockSize));
if (frameHeader.LosslessArray[modeInfo.SegmentId])
{
// In lossless, CfL is available when the partition size is equal to the
// transform size.
bool subX = colorConfig.SubSamplingX;
bool subY = colorConfig.SubSamplingY;
Av1BlockSize planeBlockSize = blockSize.GetSubsampled(subX, subY);
return planeBlockSize == Av1BlockSize.Block4x4;
}
// Outside lossless mode, AV1 limits CfL to luma partitions no larger than 32 by 32.
return blockSize.GetWidth() <= 32 && blockSize.GetHeight() <= 32;
}
/// <summary> /// <summary>
/// Converts the packed chroma-from-luma magnitude and joint sign into a signed Q3 scaling factor. /// Converts the packed chroma-from-luma magnitude and joint sign into a signed Q3 scaling factor.
/// </summary> /// </summary>

20
src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileWriter.cs

@ -708,12 +708,13 @@ internal partial class Av1TileWriter
{ {
EncodeIntraChromaMode( EncodeIntraChromaMode(
writer, writer,
frm_hdr,
scs.SequenceHeader.ColorConfig,
macroBlockModeInfo, macroBlockModeInfo,
ref blk_ptr, ref blk_ptr,
blockSize, blockSize,
intra_luma_mode, intra_luma_mode,
intra_chroma_mode, intra_chroma_mode);
blockSize.GetWidth() <= 32 && blockSize.GetHeight() <= 32);
} }
} }
@ -844,21 +845,28 @@ internal partial class Av1TileWriter
/// Writes the chroma intra mode, chroma-from-luma alpha values, and directional angle adjustment for a block. /// Writes the chroma intra mode, chroma-from-luma alpha values, and directional angle adjustment for a block.
/// </summary> /// </summary>
/// <param name="writer">The tile symbol encoder.</param> /// <param name="writer">The tile symbol encoder.</param>
/// <param name="frameHeader">The current frame syntax and segment lossless state.</param>
/// <param name="colorConfig">The sequence chroma subsampling configuration.</param>
/// <param name="macroBlockModeInfo">The selected block modes.</param> /// <param name="macroBlockModeInfo">The selected block modes.</param>
/// <param name="blk_ptr">The encoder prediction-unit state.</param> /// <param name="blk_ptr">The encoder prediction-unit state.</param>
/// <param name="blockSize">The luma block size.</param> /// <param name="blockSize">The luma block size.</param>
/// <param name="lumaMode">The selected luma prediction mode.</param> /// <param name="lumaMode">The selected luma prediction mode.</param>
/// <param name="chromaMode">The selected chroma prediction mode.</param> /// <param name="chromaMode">The selected chroma prediction mode.</param>
/// <param name="isChromaFromLumaAllowed">A value indicating whether chroma-from-luma mode is available.</param> public static void EncodeIntraChromaMode(
private static void EncodeIntraChromaMode(
Av1SymbolEncoder writer, Av1SymbolEncoder writer,
ObuFrameHeader frameHeader,
ObuColorConfig colorConfig,
Av1MacroBlockModeInfo macroBlockModeInfo, Av1MacroBlockModeInfo macroBlockModeInfo,
ref Av1EncoderBlockStruct blk_ptr, ref Av1EncoderBlockStruct blk_ptr,
Av1BlockSize blockSize, Av1BlockSize blockSize,
Av1PredictionMode lumaMode, Av1PredictionMode lumaMode,
Av1ChromaPredictionMode chromaMode, Av1ChromaPredictionMode chromaMode)
bool isChromaFromLumaAllowed)
{ {
bool isChromaFromLumaAllowed = blockSize.AllowsChromaFromLuma(
frameHeader.LosslessArray[macroBlockModeInfo.Block.SegmentId],
colorConfig.SubSamplingX,
colorConfig.SubSamplingY);
writer.WriteChromaMode(chromaMode, isChromaFromLumaAllowed, lumaMode); writer.WriteChromaMode(chromaMode, isChromaFromLumaAllowed, lumaMode);
if (chromaMode == Av1ChromaPredictionMode.ChromaFromLuma) if (chromaMode == Av1ChromaPredictionMode.ChromaFromLuma)

68
tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CoefficientsEntropyTests.cs

@ -813,6 +813,74 @@ public class Av1CoefficientsEntropyTests
Assert.True(expected.GetSpan().SequenceEqual(actual.GetSpan())); Assert.True(expected.GetSpan().SequenceEqual(actual.GetSpan()));
} }
[Theory]
[InlineData((int)Av1BlockSize.Block4x4, false, false, true, true)]
[InlineData((int)Av1BlockSize.Block8x8, true, true, true, true)]
[InlineData((int)Av1BlockSize.Block8x8, false, false, true, false)]
[InlineData((int)Av1BlockSize.Block16x16, true, true, true, false)]
[InlineData((int)Av1BlockSize.Block32x32, true, true, false, true)]
[InlineData((int)Av1BlockSize.Block64x64, true, true, false, false)]
public void ChromaFromLumaAvailabilityUsesLosslessPlaneGeometry(
int blockSize,
bool subSamplingX,
bool subSamplingY,
bool isLossless,
bool expected)
=> Assert.Equal(
expected,
((Av1BlockSize)blockSize).AllowsChromaFromLuma(isLossless, subSamplingX, subSamplingY));
[Fact]
public void LosslessChromaModeUsesPlaneSizedChromaFromLumaAlphabet()
{
ObuFrameHeader frameHeader = new();
frameHeader.LosslessArray[0] = true;
ObuColorConfig colorConfig = new()
{
SubSamplingX = true,
SubSamplingY = true
};
Av1MacroBlockModeInfo modeInfo = default;
modeInfo.Block.SegmentId = 0;
Av1EncoderBlockStruct block = default;
Av1BlockSize blockSize = Av1BlockSize.Block16x16;
ReadOnlySpan<Av1ChromaPredictionMode> decisions =
[
Av1ChromaPredictionMode.DC,
Av1ChromaPredictionMode.Smooth,
Av1ChromaPredictionMode.Paeth,
Av1ChromaPredictionMode.SmoothVertical,
Av1ChromaPredictionMode.DC,
Av1ChromaPredictionMode.SmoothHorizontal
];
using Av1SymbolEncoder actualWriter = new(Configuration.Default, 16, BaseQIndex);
using Av1SymbolEncoder expectedWriter = new(Configuration.Default, 16, BaseQIndex);
foreach (Av1ChromaPredictionMode decision in decisions)
{
Av1TileWriter.EncodeIntraChromaMode(
actualWriter,
frameHeader,
colorConfig,
modeInfo,
ref block,
blockSize,
Av1PredictionMode.DC,
decision);
expectedWriter.WriteChromaMode(
decision,
isChromaFromLumaAllowed: false,
Av1PredictionMode.DC);
}
using IMemoryOwner<byte> actual = actualWriter.Exit();
using IMemoryOwner<byte> expected = expectedWriter.Exit();
Assert.True(expected.GetSpan().SequenceEqual(actual.GetSpan()));
}
[Fact] [Fact]
public void RoundTripZeroEndOfBlock() public void RoundTripZeroEndOfBlock()
{ {

Loading…
Cancel
Save