Browse Source

Integrate AV1 blocks with frame storage

pull/2633/head
James Jackson-South 1 month ago
parent
commit
c407f7c864
  1. 2
      HEIF_IMPLEMENTATION_PLAN.md
  2. 13
      src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1EncoderFrame.cs
  3. 140
      src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1TransformBlockEncoder.cs
  4. 92
      tests/ImageSharp.Tests/Formats/Heif/Av1/Av1TransformBlockEncoderTests.cs

2
HEIF_IMPLEMENTATION_PLAN.md

@ -819,7 +819,7 @@ Encoder verification contract:
### 6. Build the complete AV1 frame encoder ### 6. Build the complete AV1 frame encoder
- [~] SIMD-first RGB-to-native-plane conversion now feeds eight-bit and high-bit-depth bordered AV1 source frames directly, preserving ImageSharp's arbitrary packed-pixel input contract without an intermediate full-frame native-plane copy. - [~] SIMD-first RGB-to-native-plane conversion now feeds eight-bit and high-bit-depth bordered AV1 source frames directly, preserving ImageSharp's arbitrary packed-pixel input contract without an intermediate full-frame native-plane copy.
- [~] Forward transform families, transform workspace, and an allocation-free DC intra block boundary exist locally. For eight-bit and high-bit-depth samples, the composed boundary now follows current libaom's encoder order: predict into the reconstruction plane, subtract prediction from source, transform, quantize into separate qcoeff and dqcoeff storage, retain EOB and transform type, and inverse-transform only when EOB is nonzero so later blocks consume decoder-identical references. Prediction and subtraction retain their SIMD-first operators, independent source and reconstruction strides are preserved, and no frame-sized or per-block buffer is introduced. One reusable 61 KiB allocator owner supplies tightly packed residual, aligned transform-coefficient, dequantized-coefficient, and transform scratch spans across transform blocks; quantized coefficients write directly to the retained frame coefficient owner instead of being duplicated. Stage-by-stage scalar-oracle, padding, retained-syntax, and zero-allocation coverage passes 6 of 6 through direct net11 VSTest in Release. Frame traversal still needs to select blocks, gather contiguous left references, and supply coefficient-owner slices. - [~] Forward transform families, transform workspace, and an allocation-free DC intra block boundary exist locally. For eight-bit and high-bit-depth samples, the composed boundary now follows current libaom's encoder order: predict into the reconstruction plane, subtract prediction from source, transform, quantize into separate qcoeff and dqcoeff storage, retain EOB and transform type, and inverse-transform only when EOB is nonzero so later blocks consume decoder-identical references. Prediction and subtraction retain their SIMD-first operators, independent source and reconstruction strides are preserved, and no frame-sized or per-block buffer is introduced. The block boundary consumes the real bordered encoder-plane regions and indexes their one-segment owner directly; this preserves physical row strides without a row copy and avoids the per-call enumerator allocation exposed by the initial array-only test. One reusable 61 KiB allocator owner supplies tightly packed residual, aligned transform-coefficient, dequantized-coefficient, and transform scratch spans across transform blocks; quantized coefficients write directly to the retained frame coefficient owner instead of being duplicated. Stage-by-stage scalar-oracle, physical-border, retained-syntax, and zero-allocation coverage passes 6 of 6 through direct net11 VSTest in Release. Frame traversal still needs to select blocks, gather contiguous left references, and supply coefficient-owner slices.
- [~] Symbol writer, coefficient writer, and tile writer fragments exist locally. - [~] Symbol writer, coefficient writer, and tile writer fragments exist locally.
- [~] A non-owning encoder-frame view now separates visible conversion regions from coded regions and performs complete left, top, right, bottom, and corner extension across each bordered plane. Current libaom uses 8-sample-aligned coded dimensions, a 32-sample-aligned luma stride with chroma stride derived from it, and a 64-pixel luma border for non-resized all-intra encoding. One operation-ready frame owner now rents the aligned Y, U, and V storage contiguously, exposes non-owning `Buffer2D` plane views, and returns the rent exactly once. A 4K 4:2:0 frame occupies about 13.0 MiB at 8-bit or 26.0 MiB at 10/12-bit; source and reconstruction therefore remain distinct frame owners rather than adding a full-frame copy. The corrected tests use this real ownership path and verify the exact 54 KiB 64x64 4:2:0 rent. The frame-encoder boundary converts packed pixels directly into the source owner before extension; the containing encode operation still needs to instantiate matching source and reconstruction owners with ordinary `using` lifetimes. - [~] A non-owning encoder-frame view now separates visible conversion regions from coded regions and performs complete left, top, right, bottom, and corner extension across each bordered plane. Current libaom uses 8-sample-aligned coded dimensions, a 32-sample-aligned luma stride with chroma stride derived from it, and a 64-pixel luma border for non-resized all-intra encoding. One operation-ready frame owner now rents the aligned Y, U, and V storage contiguously, exposes non-owning `Buffer2D` plane views, and returns the rent exactly once. A 4K 4:2:0 frame occupies about 13.0 MiB at 8-bit or 26.0 MiB at 10/12-bit; source and reconstruction therefore remain distinct frame owners rather than adding a full-frame copy. The corrected tests use this real ownership path and verify the exact 54 KiB 64x64 4:2:0 rent. The frame-encoder boundary converts packed pixels directly into the source owner before extension; the containing encode operation still needs to instantiate matching source and reconstruction owners with ordinary `using` lifetimes.
- [~] Temporal delimiter, sequence header, frame header, and combined-frame tile-group writing exist locally. The remaining required metadata, padding, and encoder-wide syntax paths are not complete. - [~] Temporal delimiter, sequence header, frame header, and combined-frame tile-group writing exist locally. The remaining required metadata, padding, and encoder-wide syntax paths are not complete.

13
src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1EncoderFrame.cs

@ -332,6 +332,19 @@ internal readonly struct Av1EncoderFrame<TSample>
} }
} }
/// <summary>
/// Gets a writable coded component plane.
/// </summary>
/// <param name="plane">The requested component plane.</param>
/// <returns>The complete coded plane region.</returns>
public Buffer2DRegion<TSample> GetPlane(Av1Plane plane)
=> plane switch
{
Av1Plane.Y => this.luma,
Av1Plane.U => this.chromaBlue,
_ => this.chromaRed
};
/// <inheritdoc/> /// <inheritdoc/>
public Span<TSample> GetLumaRowSpan(int row) => this.luma.DangerousGetRowSpan(row); public Span<TSample> GetLumaRowSpan(int row) => this.luma.DangerousGetRowSpan(row);

140
src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1TransformBlockEncoder.cs

@ -6,6 +6,7 @@ using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.Quantizers;
using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction;
using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling;
using SixLabors.ImageSharp.Formats.Heif.Av1.Transform; using SixLabors.ImageSharp.Formats.Heif.Av1.Transform;
using SixLabors.ImageSharp.Memory;
namespace SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline; namespace SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline;
@ -14,6 +15,129 @@ namespace SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline;
/// </summary> /// </summary>
internal static class Av1TransformBlockEncoder internal static class Av1TransformBlockEncoder
{ {
/// <summary>
/// Encodes and reconstructs one eight-bit lossy DC intra block in contiguous encoder planes.
/// </summary>
/// <param name="workspace">The reusable residual, coefficient, and transform storage.</param>
/// <param name="source">The coded source plane.</param>
/// <param name="reconstruction">The coded reconstruction plane.</param>
/// <param name="blockOrigin">The block origin in plane samples.</param>
/// <param name="above">The contiguous top reference samples.</param>
/// <param name="left">The contiguous left reference samples.</param>
/// <param name="hasLeft">Whether the left reference is available.</param>
/// <param name="hasAbove">Whether the top reference is available.</param>
/// <param name="quantizedCoefficients">The retained entropy-coding coefficients.</param>
/// <param name="transformSize">The selected transform dimensions.</param>
/// <param name="transformType">The selected compound transform type.</param>
/// <param name="qIndex">The segment quantizer index.</param>
/// <param name="dcDeltaQ">The plane DC quantizer adjustment.</param>
/// <param name="acDeltaQ">The plane AC quantizer adjustment.</param>
/// <param name="plane">The component plane containing the block.</param>
/// <param name="state">The retained transform type and end-of-block syntax.</param>
public static void EncodeIntraDcLossy(
Av1EncoderBlockWorkspace workspace,
Buffer2DRegion<byte> source,
Buffer2DRegion<byte> reconstruction,
Point blockOrigin,
ReadOnlySpan<byte> above,
ReadOnlySpan<byte> left,
bool hasLeft,
bool hasAbove,
Span<int> quantizedCoefficients,
Av1TransformSize transformSize,
Av1TransformType transformType,
int qIndex,
int dcDeltaQ,
int acDeltaQ,
Av1Plane plane,
ref Av1EncoderTransformBlockState state)
{
ReadOnlySpan<byte> sourceSamples = GetPlaneSpan(source, blockOrigin);
Span<byte> reconstructionSamples = GetPlaneSpan(reconstruction, blockOrigin);
EncodeIntraDcLossyContiguous(
workspace,
sourceSamples,
source.Stride,
reconstructionSamples,
reconstruction.Stride,
above,
left,
hasLeft,
hasAbove,
quantizedCoefficients,
transformSize,
transformType,
qIndex,
dcDeltaQ,
acDeltaQ,
plane,
ref state);
}
/// <summary>
/// Encodes and reconstructs one high-bit-depth lossy DC intra block in contiguous encoder planes.
/// </summary>
/// <param name="workspace">The reusable residual, coefficient, and transform storage.</param>
/// <param name="source">The coded source plane.</param>
/// <param name="reconstruction">The coded reconstruction plane.</param>
/// <param name="blockOrigin">The block origin in plane samples.</param>
/// <param name="above">The contiguous top reference samples.</param>
/// <param name="left">The contiguous left reference samples.</param>
/// <param name="hasLeft">Whether the left reference is available.</param>
/// <param name="hasAbove">Whether the top reference is available.</param>
/// <param name="quantizedCoefficients">The retained entropy-coding coefficients.</param>
/// <param name="transformSize">The selected transform dimensions.</param>
/// <param name="transformType">The selected compound transform type.</param>
/// <param name="qIndex">The segment quantizer index.</param>
/// <param name="dcDeltaQ">The plane DC quantizer adjustment.</param>
/// <param name="acDeltaQ">The plane AC quantizer adjustment.</param>
/// <param name="plane">The component plane containing the block.</param>
/// <param name="bitDepth">The coded sample bit depth.</param>
/// <param name="state">The retained transform type and end-of-block syntax.</param>
public static void EncodeIntraDcLossy(
Av1EncoderBlockWorkspace workspace,
Buffer2DRegion<ushort> source,
Buffer2DRegion<ushort> reconstruction,
Point blockOrigin,
ReadOnlySpan<ushort> above,
ReadOnlySpan<ushort> left,
bool hasLeft,
bool hasAbove,
Span<int> quantizedCoefficients,
Av1TransformSize transformSize,
Av1TransformType transformType,
int qIndex,
int dcDeltaQ,
int acDeltaQ,
Av1Plane plane,
Av1BitDepth bitDepth,
ref Av1EncoderTransformBlockState state)
{
ReadOnlySpan<ushort> sourceSamples = GetPlaneSpan(source, blockOrigin);
Span<ushort> reconstructionSamples = GetPlaneSpan(reconstruction, blockOrigin);
EncodeIntraDcLossyContiguous(
workspace,
sourceSamples,
source.Stride,
reconstructionSamples,
reconstruction.Stride,
above,
left,
hasLeft,
hasAbove,
quantizedCoefficients,
transformSize,
transformType,
qIndex,
dcDeltaQ,
acDeltaQ,
plane,
bitDepth,
ref state);
}
/// <summary> /// <summary>
/// Encodes and reconstructs one eight-bit lossy DC intra block. /// Encodes and reconstructs one eight-bit lossy DC intra block.
/// </summary> /// </summary>
@ -34,7 +158,7 @@ internal static class Av1TransformBlockEncoder
/// <param name="acDeltaQ">The plane AC quantizer adjustment.</param> /// <param name="acDeltaQ">The plane AC quantizer adjustment.</param>
/// <param name="plane">The component plane containing the block.</param> /// <param name="plane">The component plane containing the block.</param>
/// <param name="state">The retained transform type and end-of-block syntax.</param> /// <param name="state">The retained transform type and end-of-block syntax.</param>
public static void EncodeIntraDcLossy( private static void EncodeIntraDcLossyContiguous(
Av1EncoderBlockWorkspace workspace, Av1EncoderBlockWorkspace workspace,
ReadOnlySpan<byte> source, ReadOnlySpan<byte> source,
int sourceStride, int sourceStride,
@ -108,7 +232,7 @@ internal static class Av1TransformBlockEncoder
/// <param name="plane">The component plane containing the block.</param> /// <param name="plane">The component plane containing the block.</param>
/// <param name="bitDepth">The coded sample bit depth.</param> /// <param name="bitDepth">The coded sample bit depth.</param>
/// <param name="state">The retained transform type and end-of-block syntax.</param> /// <param name="state">The retained transform type and end-of-block syntax.</param>
public static void EncodeIntraDcLossy( private static void EncodeIntraDcLossyContiguous(
Av1EncoderBlockWorkspace workspace, Av1EncoderBlockWorkspace workspace,
ReadOnlySpan<ushort> source, ReadOnlySpan<ushort> source,
int sourceStride, int sourceStride,
@ -236,4 +360,16 @@ internal static class Av1TransformBlockEncoder
state.TransformType = transformType; state.TransformType = transformType;
} }
private static Span<TSample> GetPlaneSpan<TSample>(Buffer2DRegion<TSample> plane, Point blockOrigin)
where TSample : unmanaged
{
int offset =
((plane.Bounds.Y + blockOrigin.Y) * plane.Stride) +
plane.Bounds.X +
blockOrigin.X;
// Encoder planes wrap one contiguous frame owner, so direct segment access retains physical strides without an enumerator or row copy.
return plane.Buffer.FastMemoryGroup[0].Span[offset..];
}
} }

92
tests/ImageSharp.Tests/Formats/Heif/Av1/Av1TransformBlockEncoderTests.cs

@ -8,6 +8,7 @@ using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.Quantizers;
using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction;
using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling;
using SixLabors.ImageSharp.Formats.Heif.Av1.Transform; using SixLabors.ImageSharp.Formats.Heif.Av1.Transform;
using SixLabors.ImageSharp.Memory;
using SixLabors.ImageSharp.Tests.Memory; using SixLabors.ImageSharp.Tests.Memory;
namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1;
@ -49,6 +50,26 @@ public class Av1TransformBlockEncoderTests
byte[] left = new byte[height]; byte[] left = new byte[height];
int[] expectedQuantized = new int[coefficientCount + 7]; int[] expectedQuantized = new int[coefficientCount + 7];
int[] actualQuantized = new int[coefficientCount + 7]; int[] actualQuantized = new int[coefficientCount + 7];
using Av1EncoderFrameBuffer<byte> sourceFrame = new(
Configuration.Default,
width,
height,
8,
Av1ColorFormat.Yuv400,
0,
0);
using Av1EncoderFrameBuffer<byte> reconstructionFrame = new(
Configuration.Default,
width,
height,
8,
Av1ColorFormat.Yuv400,
0,
0);
Buffer2DRegion<byte> sourcePlane = sourceFrame.Frame.CodedView.GetPlane(Av1Plane.Y);
Buffer2DRegion<byte> reconstructionPlane = reconstructionFrame.Frame.CodedView.GetPlane(Av1Plane.Y);
using Av1EncoderBlockWorkspace expectedWorkspace = new(Configuration.Default); using Av1EncoderBlockWorkspace expectedWorkspace = new(Configuration.Default);
using Av1EncoderBlockWorkspace actualWorkspace = new(Configuration.Default); using Av1EncoderBlockWorkspace actualWorkspace = new(Configuration.Default);
FillSource(source, SourceStride, width, height, byte.MaxValue); FillSource(source, SourceStride, width, height, byte.MaxValue);
@ -57,6 +78,12 @@ public class Av1TransformBlockEncoderTests
Array.Fill(expectedQuantized, int.MinValue); Array.Fill(expectedQuantized, int.MinValue);
Array.Fill(actualQuantized, int.MinValue); Array.Fill(actualQuantized, int.MinValue);
for (int y = 0; y < height; y++)
{
source.AsSpan(y * SourceStride, width).CopyTo(sourcePlane.DangerousGetRowSpan(y));
reconstructionPlane.DangerousGetRowSpan(y).Fill(211);
}
for (int i = 0; i < above.Length; i++) for (int i = 0; i < above.Length; i++)
{ {
above[i] = (byte)(37 + (i * 11)); above[i] = (byte)(37 + (i * 11));
@ -115,10 +142,9 @@ public class Av1TransformBlockEncoderTests
Av1EncoderTransformBlockState actualState = default; Av1EncoderTransformBlockState actualState = default;
Av1TransformBlockEncoder.EncodeIntraDcLossy( Av1TransformBlockEncoder.EncodeIntraDcLossy(
actualWorkspace, actualWorkspace,
source, sourcePlane,
SourceStride, reconstructionPlane,
actualReconstruction, Point.Empty,
ReconstructionStride,
above, above,
left, left,
true, true,
@ -132,10 +158,21 @@ public class Av1TransformBlockEncoderTests
Av1Plane.Y, Av1Plane.Y,
ref actualState); ref actualState);
for (int y = 0; y < height; y++)
{
reconstructionPlane.DangerousGetRowSpan(y).CopyTo(
actualReconstruction.AsSpan(y * ReconstructionStride, width));
}
int physicalRow = reconstructionPlane.Bounds.Y;
int physicalColumn = reconstructionPlane.Bounds.X;
ReadOnlySpan<byte> completeRow = reconstructionFrame.Luma.DangerousGetRowSpan(physicalRow);
Assert.Equal(expectedReconstruction, actualReconstruction); Assert.Equal(expectedReconstruction, actualReconstruction);
Assert.Equal(expectedQuantized, actualQuantized); Assert.Equal(expectedQuantized, actualQuantized);
Assert.Equal(expectedState.EndOfBlock, actualState.EndOfBlock); Assert.Equal(expectedState.EndOfBlock, actualState.EndOfBlock);
Assert.Equal(expectedState.TransformType, actualState.TransformType); Assert.Equal(expectedState.TransformType, actualState.TransformType);
Assert.Equal(0, completeRow[physicalColumn - 1]);
Assert.Equal(0, completeRow[physicalColumn + width]);
} }
/// <summary> /// <summary>
@ -158,6 +195,10 @@ public class Av1TransformBlockEncoderTests
ushort[] left = new ushort[height]; ushort[] left = new ushort[height];
int[] expectedQuantized = new int[coefficientCount + 7]; int[] expectedQuantized = new int[coefficientCount + 7];
int[] actualQuantized = new int[coefficientCount + 7]; int[] actualQuantized = new int[coefficientCount + 7];
using Buffer2D<ushort> sourceBuffer = Buffer2D<ushort>.WrapMemory(source, SourceStride, height, SourceStride);
using Buffer2D<ushort> reconstructionBuffer =
Buffer2D<ushort>.WrapMemory(actualReconstruction, ReconstructionStride, height, ReconstructionStride);
using Av1EncoderBlockWorkspace expectedWorkspace = new(Configuration.Default); using Av1EncoderBlockWorkspace expectedWorkspace = new(Configuration.Default);
using Av1EncoderBlockWorkspace actualWorkspace = new(Configuration.Default); using Av1EncoderBlockWorkspace actualWorkspace = new(Configuration.Default);
FillSource(source, SourceStride, width, height, (1 << bitDepth.GetBitCount()) - 1); FillSource(source, SourceStride, width, height, (1 << bitDepth.GetBitCount()) - 1);
@ -227,10 +268,9 @@ public class Av1TransformBlockEncoderTests
Av1EncoderTransformBlockState actualState = default; Av1EncoderTransformBlockState actualState = default;
Av1TransformBlockEncoder.EncodeIntraDcLossy( Av1TransformBlockEncoder.EncodeIntraDcLossy(
actualWorkspace, actualWorkspace,
source, new Buffer2DRegion<ushort>(sourceBuffer),
SourceStride, new Buffer2DRegion<ushort>(reconstructionBuffer),
actualReconstruction, Point.Empty,
ReconstructionStride,
above, above,
left, left,
true, true,
@ -269,6 +309,14 @@ public class Av1TransformBlockEncoderTests
ushort[] above10 = new ushort[Stride]; ushort[] above10 = new ushort[Stride];
ushort[] left10 = new ushort[Stride]; ushort[] left10 = new ushort[Stride];
int[] quantized = new int[coefficientCount]; int[] quantized = new int[coefficientCount];
using Buffer2D<byte> sourceBuffer8 = Buffer2D<byte>.WrapMemory(source8, Stride, Stride);
using Buffer2D<byte> reconstructionBuffer8 = Buffer2D<byte>.WrapMemory(reconstruction8, Stride, Stride);
using Buffer2D<ushort> sourceBuffer10 = Buffer2D<ushort>.WrapMemory(source10, Stride, Stride);
using Buffer2D<ushort> reconstructionBuffer10 = Buffer2D<ushort>.WrapMemory(reconstruction10, Stride, Stride);
Buffer2DRegion<byte> sourcePlane8 = new(sourceBuffer8);
Buffer2DRegion<byte> reconstructionPlane8 = new(reconstructionBuffer8);
Buffer2DRegion<ushort> sourcePlane10 = new(sourceBuffer10);
Buffer2DRegion<ushort> reconstructionPlane10 = new(reconstructionBuffer10);
using Av1EncoderBlockWorkspace workspace = new(Configuration.Default); using Av1EncoderBlockWorkspace workspace = new(Configuration.Default);
FillSource(source8, Stride, Stride, Stride, byte.MaxValue); FillSource(source8, Stride, Stride, Stride, byte.MaxValue);
FillSource(source10, Stride, Stride, Stride, 1023); FillSource(source10, Stride, Stride, Stride, 1023);
@ -280,10 +328,9 @@ public class Av1TransformBlockEncoderTests
Av1TransformBlockEncoder.EncodeIntraDcLossy( Av1TransformBlockEncoder.EncodeIntraDcLossy(
workspace, workspace,
source8, sourcePlane8,
Stride, reconstructionPlane8,
reconstruction8, Point.Empty,
Stride,
above8, above8,
left8, left8,
true, true,
@ -299,10 +346,9 @@ public class Av1TransformBlockEncoderTests
Av1TransformBlockEncoder.EncodeIntraDcLossy( Av1TransformBlockEncoder.EncodeIntraDcLossy(
workspace, workspace,
source10, sourcePlane10,
Stride, reconstructionPlane10,
reconstruction10, Point.Empty,
Stride,
above10, above10,
left10, left10,
true, true,
@ -322,10 +368,9 @@ public class Av1TransformBlockEncoderTests
{ {
Av1TransformBlockEncoder.EncodeIntraDcLossy( Av1TransformBlockEncoder.EncodeIntraDcLossy(
workspace, workspace,
source8, sourcePlane8,
Stride, reconstructionPlane8,
reconstruction8, Point.Empty,
Stride,
above8, above8,
left8, left8,
true, true,
@ -341,10 +386,9 @@ public class Av1TransformBlockEncoderTests
Av1TransformBlockEncoder.EncodeIntraDcLossy( Av1TransformBlockEncoder.EncodeIntraDcLossy(
workspace, workspace,
source10, sourcePlane10,
Stride, reconstructionPlane10,
reconstruction10, Point.Empty,
Stride,
above10, above10,
left10, left10,
true, true,

Loading…
Cancel
Save