Browse Source

Retain AV1 intra-block-copy vectors

pull/2633/head
James Jackson-South 1 month ago
parent
commit
568cfa7c5a
  1. 2
      HEIF_IMPLEMENTATION_PLAN.md
  2. 252
      src/ImageSharp/Formats/Heif/Av1/Motion/Av1IntraBlockCopy.cs
  3. 20
      src/ImageSharp/Formats/Heif/Av1/Tiling/Av1EncoderDisplacementVector.cs
  4. 23
      src/ImageSharp/Formats/Heif/Av1/Tiling/Av1EncoderPictureBuffer.cs
  5. 45
      src/ImageSharp/Formats/Heif/Av1/Tiling/Av1PartitionInfo.cs
  6. 37
      src/ImageSharp/Formats/Heif/Av1/Tiling/Av1PictureControlSet.cs
  7. 38
      src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileWriter.cs
  8. 30
      tests/ImageSharp.Tests/Formats/Heif/Av1/Av1EncoderModeInfoBufferTests.cs
  9. 134
      tests/ImageSharp.Tests/Formats/Heif/Av1/Av1IntraBlockCopyTests.cs

2
HEIF_IMPLEMENTATION_PLAN.md

@ -859,7 +859,7 @@ Encoder verification contract:
- [~] Paired chroma palette clustering now preserves current libaom's squared two-component distance, first-centroid tie order, independently rounded U/V means, paired deterministic empty-cluster replacement, preceding-state retention on increased distortion, and 50-iteration limit. Keeping the source planes separate avoids interleave/deinterleave copies and improves on libaom's AVX2 ceiling with Vector512, Vector256, Vector128, then scalar dispatch through ImageSharp's shared vector-count helpers. Three independent tests cover exact paired convergence, midpoint initialization, 12-bit distance and index parity, untouched destination bounds, and every intrinsic tier. The exact Release test-project build reports 1,992 baseline warnings and zero errors; the focused three-case set, complete 8,934-case AVIF set, and complete 230-case HEIF set pass direct foreground net11 Release VSTest. Roslynk reports zero compiler errors and no touched-file analyzer warnings. Candidate integration and production activation remain in the open chroma-palette checkpoint.
- [~] Live paired chroma palette selection now follows current libaom's complete 2-through-8 color-size search, U-plane neighbor-cache snapping, stable U-ordered color pairs, shared U/V index map, implicit DCT-DCT transform, and strict rate-distortion winner replacement. It improves on speed-configured libaom by applying no early header-cost pruning, keeps planar U/V source data separate, and reuses the SIMD-first prediction, residual, transform, quantization, and reconstruction operators without allocator-backed candidate storage. The production tile regression proves both palette-mode probability branches, exact paired colors and indices, coefficient-free reconstruction, and nonempty syntax. The complete 58-case intra-superblock set, 8,935-case AVIF set, and 230-case HEIF set pass direct foreground net11 Release VSTest. The exact Release test-project build reports 1,992 baseline warnings and zero errors; Roslynk reports zero compiler errors and no touched-file analyzer warnings. Production frame activation remains the next checkpoint.
- [~] Production palette activation now matches current libaom's default good-quality screen detector: it scans only complete 16x16 luma blocks, normalizes high-bit-depth samples to eight bits, admits 2-through-4-color blocks, and uses the reference's strict greater-than-ten-percent frame-area threshold. A 256-bit stack bitset and a fifth-color early exit replace libaom's larger per-block histogram without changing the decision, allocation, or source precision. The adaptive sequence flag remains enabled, the frame flag is set before picture-state allocation, and intra-block copy remains disabled. Focused regressions prove strict-threshold equality, high-bit-depth normalization, five-color rejection, emitted frame-header activation, production decode, and generated payload retention. The exact Release test-project build reports 1,992 baseline warnings and zero errors; all 8,935 AVIF cases and all 230 HEIF cases pass direct foreground net11 Release VSTest. Current-main `aomdec` at `a40ed1ea9e4ecc3df58a5bccb76623f2c94ae727` accepts all 30 regenerated production payloads, including the 54-byte palette case. Roslynk reports zero compiler errors and no touched-file analyzer warnings.
- [~] Intra-block-copy rate accounting now uses the live frame-local flag and displacement-vector distributions without copying or adapting either context during candidate measurement. Displacement-vector costing and writing share one closed symbol operation over the exact current-libaom joint, sign, magnitude-class, class-zero, and integer-offset syntax; mode search applies libaom's 120/128 displacement-rate weight with nearest-integer rounding. Independent fixed costs cover all four joint states, both signs, class zero, and large offset classes before adaptive writes, followed by an encoder/decoder round trip through the same sequence. The exact net11 Release build reports 1,005 baseline warnings and zero errors; all 1,942 entropy cases and all 9,006 AV1 cases pass through direct foreground VSTest, and Roslynk reports zero compiler errors and no diagnostics in the touched files. Reference-vector derivation, legal source search, retained displacement state, production writing, and adaptive frame-flag clearing remain before intra-block copy can be activated.
- [~] Intra-block-copy rate accounting now uses the live frame-local flag and displacement-vector distributions without copying or adapting either context during candidate measurement. Displacement-vector costing and writing share one closed symbol operation over the exact current-libaom joint, sign, magnitude-class, class-zero, and integer-offset syntax; mode search applies libaom's 120/128 displacement-rate weight with nearest-integer rounding. Independent fixed costs cover all four joint states, both signs, class zero, and large offset classes before adaptive writes, followed by an encoder/decoder round trip through the same sequence. Encoder and decoder reference-vector derivation now share the exact eight-candidate spatial scan, independent nearest and outer-region ranking, top-right partition geometry, clamping, and tile-relative fallback. Selected vectors use a naturally aligned pair of signed 16-bit components packed into the existing picture-state owner only when intra-block copy is permitted; a 3840x2160 frame retains 130,560 vectors in 510 KiB while leaving the compact 8-byte mode allocation unchanged. The tile writer derives the same reference and emits the retained vector without another allocation or copy. The exact net11 Release build reports 1,005 baseline warnings and zero errors; all 13 focused ownership, mapping, reference, and syntax cases, all 9,009 AV1 cases, and all 206 non-AV1 HEIF cases pass through direct foreground VSTest, and Roslynk reports zero compiler errors. Legal source search, joint luma/chroma rate-distortion selection, production activation, and adaptive frame-flag clearing remain before intra-block copy can be enabled.
- [x] The expanded checkpoint exposed a pre-existing transform-block test that asserted uninitialized pooled padding was zero. The test now initializes the complete physical luma plane with a sentinel and proves the block operation leaves both adjacent padding samples unchanged. The exact net11 Release rebuild remains at 1,005 baseline warnings and zero errors, the focused allocator-order set passes 30 of 30 cases, and the complete HEIF/AV1 namespace passes 8,859 of 8,859 direct VSTest cases with zero failures or skips.
- [x] Combined-frame OBU output now counts the byte-aligned frame and tile-group headers, non-final tile-size fields, and owned tile payloads before emitting the OBU size. It retains only the small allocator-owned header scratch and writes each entropy-coded tile span directly from its detached owner, removing the second file-sized allocator rent and complete-payload copy. A 64 KiB regression proves exactly one sub-payload-sized byte rent with a balanced return and verifies the exact streamed tile tail; the existing two-tile round trip proves size-prefix and ordering parity. The focused writer and production-frame set passes 32 of 32 direct net11 VSTest cases, current-main `aomdec` accepts all 29 generated native-format payloads, and the complete HEIF/AV1 namespace passes 8,860 of 8,860 cases with zero failures or skips.
- [x] Finalized fixed-block decisions now set the block-level transform-skip flag only when every retained luma and coded chroma transform has zero EOB, matching current libaom's conjunction of per-plane skip state. The previous always-false flag produced legal but redundant non-skip and zero-coefficient syntax. Monochrome and 4:2:0 regressions prove both branches from actual coefficient state; the focused decision and production-frame set passes 32 of 32 direct net11 VSTest cases. Current-main `aomdec` accepts all 29 regenerated payloads, the recorded decoded-frame MD5s are unchanged, and affected 16x16 constant 8-bit and 10-bit payloads are one byte smaller. The complete HEIF/AV1 namespace passes 8,862 of 8,862 cases with zero failures or skips.

252
src/ImageSharp/Formats/Heif/Av1/Motion/Av1IntraBlockCopy.cs

@ -42,23 +42,74 @@ internal static class Av1IntraBlockCopy
Span<Av1MotionVector> candidates,
Span<int> weights)
{
Av1BlockSize blockSize = partitionInfo.ModeInfo.BlockSize;
ReferenceContext context = new(ref partitionInfo, superblockModeInfoSize);
return FindReference(ref context, tileInfo, superblockModeInfoSize, candidates, weights);
}
/// <summary>
/// Finds the spatial reference used to differentially encode an intra-block-copy displacement vector.
/// </summary>
/// <param name="picture">The encoded frame's mapped mode and displacement state.</param>
/// <param name="macroBlock">The current block's frame edges and tile availability.</param>
/// <param name="modeInfoPosition">The current block origin in 4x4 mode-information units.</param>
/// <param name="blockSize">The current block size.</param>
/// <param name="partitionType">The partition type that produced the block.</param>
/// <param name="candidates">Reusable storage for up to eight unique reference vectors.</param>
/// <param name="weights">Reusable storage for the corresponding spatial weights.</param>
/// <returns>The nearest nonzero spatial candidate, or the normative tile-relative fallback.</returns>
public static Av1MotionVector FindReference(
Av1PictureControlSet picture,
Av1MacroBlockD macroBlock,
Point modeInfoPosition,
Av1BlockSize blockSize,
Av1PartitionType partitionType,
Span<Av1MotionVector> candidates,
Span<int> weights)
{
int superblockModeInfoSize = picture.Sequence.SequenceHeader.SuperblockModeInfoSize;
ReferenceContext context = new(
picture,
macroBlock,
modeInfoPosition,
blockSize,
partitionType,
superblockModeInfoSize);
return FindReference(
ref context,
macroBlock.Tile,
superblockModeInfoSize,
candidates,
weights);
}
/// <summary>
/// Ranks the shared decoder or encoder reference context without allocating candidate state.
/// </summary>
private static Av1MotionVector FindReference(
ref ReferenceContext context,
Av1TileInfo tileInfo,
int superblockModeInfoSize,
Span<Av1MotionVector> candidates,
Span<int> weights)
{
Av1BlockSize blockSize = context.BlockSize;
int width = blockSize.Get4x4WideCount();
int height = blockSize.Get4x4HighCount();
int row = partitionInfo.RowIndex;
int column = partitionInfo.ColumnIndex;
int row = context.RowIndex;
int column = context.ColumnIndex;
int rowAdjustment = height < 2 && (row & 1) != 0 ? 1 : 0;
int columnAdjustment = width < 2 && (column & 1) != 0 ? 1 : 0;
int maximumRowOffset = 0;
int maximumColumnOffset = 0;
if (partitionInfo.AvailableAbove)
if (context.AvailableAbove)
{
maximumRowOffset = height < 2 ? -4 + rowAdjustment : -(ReferenceSearchDistance << 1) + rowAdjustment;
maximumRowOffset = Math.Clamp(maximumRowOffset, tileInfo.ModeInfoRowStart - row, tileInfo.ModeInfoRowEnd - row - 1);
}
if (partitionInfo.AvailableLeft)
if (context.AvailableLeft)
{
maximumColumnOffset = width < 2 ? -4 + columnAdjustment : -(ReferenceSearchDistance << 1) + columnAdjustment;
maximumColumnOffset = Math.Clamp(maximumColumnOffset, tileInfo.ModeInfoColumnStart - column, tileInfo.ModeInfoColumnEnd - column - 1);
@ -69,17 +120,17 @@ internal static class Av1IntraBlockCopy
int processedColumns = 0;
if (Math.Abs(maximumRowOffset) >= 1)
{
ScanRow(ref partitionInfo, -1, maximumRowOffset, candidates, weights, ref candidateCount, ref processedRows);
ScanRow(ref context, -1, maximumRowOffset, candidates, weights, ref candidateCount, ref processedRows);
}
if (Math.Abs(maximumColumnOffset) >= 1)
{
ScanColumn(ref partitionInfo, -1, maximumColumnOffset, candidates, weights, ref candidateCount, ref processedColumns);
ScanColumn(ref context, -1, maximumColumnOffset, candidates, weights, ref candidateCount, ref processedColumns);
}
if (partitionInfo.HasTopRight(superblockModeInfoSize))
if (context.HasTopRight)
{
AddBlock(ref partitionInfo, -1, width, tileInfo, candidates, weights, ref candidateCount);
AddBlock(ref context, -1, width, tileInfo, candidates, weights, ref candidateCount);
}
int nearestCandidateCount = candidateCount;
@ -90,19 +141,19 @@ internal static class Av1IntraBlockCopy
// The top-left sample begins the outer search region. Sorting the adjacent and outer regions independently
// preserves the reference decoder's nearest/near ordering while still accumulating repeated vectors across both regions.
AddBlock(ref partitionInfo, -1, -1, tileInfo, candidates, weights, ref candidateCount);
AddBlock(ref context, -1, -1, tileInfo, candidates, weights, ref candidateCount);
for (int index = 2; index <= ReferenceSearchDistance; index++)
{
int rowOffset = -(index << 1) + 1 + rowAdjustment;
int columnOffset = -(index << 1) + 1 + columnAdjustment;
if (Math.Abs(rowOffset) <= Math.Abs(maximumRowOffset) && Math.Abs(rowOffset) > processedRows)
{
ScanRow(ref partitionInfo, rowOffset, maximumRowOffset, candidates, weights, ref candidateCount, ref processedRows);
ScanRow(ref context, rowOffset, maximumRowOffset, candidates, weights, ref candidateCount, ref processedRows);
}
if (Math.Abs(columnOffset) <= Math.Abs(maximumColumnOffset) && Math.Abs(columnOffset) > processedColumns)
{
ScanColumn(ref partitionInfo, columnOffset, maximumColumnOffset, candidates, weights, ref candidateCount, ref processedColumns);
ScanColumn(ref context, columnOffset, maximumColumnOffset, candidates, weights, ref candidateCount, ref processedColumns);
}
}
@ -117,10 +168,10 @@ internal static class Av1IntraBlockCopy
candidates[index] = candidates[index].ClampReference(
blockSize.GetWidth(),
blockSize.GetHeight(),
partitionInfo.ModeBlockToLeftEdge,
partitionInfo.ModeBlockToRightEdge,
partitionInfo.ModeBlockToTopEdge,
partitionInfo.ModeBlockToBottomEdge);
context.ModeBlockToLeftEdge,
context.ModeBlockToRightEdge,
context.ModeBlockToTopEdge,
context.ModeBlockToBottomEdge);
}
Av1MotionVector reference = candidateCount > 0 ? candidates[0] : default;
@ -221,7 +272,7 @@ internal static class Av1IntraBlockCopy
/// Scans a mode-information row using AV1's block-size-dependent steps and weights.
/// </summary>
private static void ScanRow(
ref Av1PartitionInfo partitionInfo,
ref ReferenceContext context,
int rowOffset,
int maximumRowOffset,
Span<Av1MotionVector> candidates,
@ -229,13 +280,13 @@ internal static class Av1IntraBlockCopy
ref int candidateCount,
ref int processedRows)
{
int width = partitionInfo.ModeInfo.BlockSize.Get4x4WideCount();
int end = Math.Min(partitionInfo.GetMaxBlockWide(partitionInfo.ModeInfo.BlockSize, false), 16);
int width = context.BlockSize.Get4x4WideCount();
int end = Math.Min(context.GetMaxBlockWide(), 16);
int columnOffset = 0;
if (Math.Abs(rowOffset) > 1)
{
columnOffset = 1;
if ((partitionInfo.ColumnIndex & 1) != 0 && width < 2)
if ((context.ColumnIndex & 1) != 0 && width < 2)
{
columnOffset--;
}
@ -245,8 +296,8 @@ internal static class Av1IntraBlockCopy
bool useFourUnitStep = width >= 16;
for (int index = 0; index < end;)
{
Av1BlockModeInfo candidate = partitionInfo.SuperblockInfo.GetModeInfoAt(
new Point(partitionInfo.ColumnIndex + columnOffset + index, partitionInfo.RowIndex + rowOffset));
ReferenceBlock candidate = context.GetModeInfoAt(
new Point(context.ColumnIndex + columnOffset + index, context.RowIndex + rowOffset));
int candidateWidth = candidate.BlockSize.Get4x4WideCount();
int length = Math.Min(width, candidateWidth);
@ -276,7 +327,7 @@ internal static class Av1IntraBlockCopy
/// Scans a mode-information column using AV1's block-size-dependent steps and weights.
/// </summary>
private static void ScanColumn(
ref Av1PartitionInfo partitionInfo,
ref ReferenceContext context,
int columnOffset,
int maximumColumnOffset,
Span<Av1MotionVector> candidates,
@ -284,13 +335,13 @@ internal static class Av1IntraBlockCopy
ref int candidateCount,
ref int processedColumns)
{
int height = partitionInfo.ModeInfo.BlockSize.Get4x4HighCount();
int end = Math.Min(partitionInfo.GetMaxBlockHigh(partitionInfo.ModeInfo.BlockSize, false), 16);
int height = context.BlockSize.Get4x4HighCount();
int end = Math.Min(context.GetMaxBlockHigh(), 16);
int rowOffset = 0;
if (Math.Abs(columnOffset) > 1)
{
rowOffset = 1;
if ((partitionInfo.RowIndex & 1) != 0 && height < 2)
if ((context.RowIndex & 1) != 0 && height < 2)
{
rowOffset--;
}
@ -300,8 +351,8 @@ internal static class Av1IntraBlockCopy
bool useFourUnitStep = height >= 16;
for (int index = 0; index < end;)
{
Av1BlockModeInfo candidate = partitionInfo.SuperblockInfo.GetModeInfoAt(
new Point(partitionInfo.ColumnIndex + columnOffset, partitionInfo.RowIndex + rowOffset + index));
ReferenceBlock candidate = context.GetModeInfoAt(
new Point(context.ColumnIndex + columnOffset, context.RowIndex + rowOffset + index));
int candidateHeight = candidate.BlockSize.Get4x4HighCount();
int length = Math.Min(height, candidateHeight);
@ -331,7 +382,7 @@ internal static class Av1IntraBlockCopy
/// Adds the intra-block-copy vector at one tile-relative search position.
/// </summary>
private static void AddBlock(
ref Av1PartitionInfo partitionInfo,
ref ReferenceContext context,
int rowOffset,
int columnOffset,
Av1TileInfo tileInfo,
@ -339,15 +390,15 @@ internal static class Av1IntraBlockCopy
Span<int> weights,
ref int candidateCount)
{
int row = partitionInfo.RowIndex + rowOffset;
int column = partitionInfo.ColumnIndex + columnOffset;
int row = context.RowIndex + rowOffset;
int column = context.ColumnIndex + columnOffset;
if (row < tileInfo.ModeInfoRowStart || row >= tileInfo.ModeInfoRowEnd ||
column < tileInfo.ModeInfoColumnStart || column >= tileInfo.ModeInfoColumnEnd)
{
return;
}
Av1BlockModeInfo candidate = partitionInfo.SuperblockInfo.GetModeInfoAt(new Point(column, row));
ReferenceBlock candidate = context.GetModeInfoAt(new Point(column, row));
AddCandidate(candidate, 4, candidates, weights, ref candidateCount);
}
@ -355,7 +406,7 @@ internal static class Av1IntraBlockCopy
/// Accumulates one unique intra-block-copy candidate and its spatial weight.
/// </summary>
private static void AddCandidate(
Av1BlockModeInfo candidate,
ReferenceBlock candidate,
int weight,
Span<Av1MotionVector> candidates,
Span<int> weights,
@ -412,4 +463,139 @@ internal static class Av1IntraBlockCopy
length = lastSwap;
}
}
/// <summary>
/// Provides one allocation-free view over decoder or encoder mode-information storage.
/// </summary>
private readonly struct ReferenceContext
{
private readonly Av1SuperblockInfo decodedSuperblock;
private readonly Av1PictureControlSet? encodedPicture;
public ReferenceContext(ref Av1PartitionInfo partitionInfo, int superblockModeInfoSize)
{
this.decodedSuperblock = partitionInfo.SuperblockInfo;
this.encodedPicture = null;
this.BlockSize = partitionInfo.ModeInfo.BlockSize;
this.RowIndex = partitionInfo.RowIndex;
this.ColumnIndex = partitionInfo.ColumnIndex;
this.AvailableAbove = partitionInfo.AvailableAbove;
this.AvailableLeft = partitionInfo.AvailableLeft;
this.ModeBlockToLeftEdge = partitionInfo.ModeBlockToLeftEdge;
this.ModeBlockToRightEdge = partitionInfo.ModeBlockToRightEdge;
this.ModeBlockToTopEdge = partitionInfo.ModeBlockToTopEdge;
this.ModeBlockToBottomEdge = partitionInfo.ModeBlockToBottomEdge;
this.HasTopRight = partitionInfo.HasTopRight(superblockModeInfoSize);
}
public ReferenceContext(
Av1PictureControlSet picture,
Av1MacroBlockD macroBlock,
Point modeInfoPosition,
Av1BlockSize blockSize,
Av1PartitionType partitionType,
int superblockModeInfoSize)
{
this.decodedSuperblock = default;
this.encodedPicture = picture;
this.BlockSize = blockSize;
this.RowIndex = modeInfoPosition.Y;
this.ColumnIndex = modeInfoPosition.X;
this.AvailableAbove = macroBlock.IsUpAvailable;
this.AvailableLeft = macroBlock.IsLeftAvailable;
this.ModeBlockToLeftEdge = macroBlock.ToLeftEdge;
this.ModeBlockToRightEdge = macroBlock.ToRightEdge;
this.ModeBlockToTopEdge = macroBlock.ToTopEdge;
this.ModeBlockToBottomEdge = macroBlock.ToBottomEdge;
this.HasTopRight = Av1PartitionInfo.HasTopRight(
blockSize,
partitionType,
modeInfoPosition.Y,
modeInfoPosition.X,
superblockModeInfoSize);
}
public Av1BlockSize BlockSize { get; }
public int RowIndex { get; }
public int ColumnIndex { get; }
public bool AvailableAbove { get; }
public bool AvailableLeft { get; }
public int ModeBlockToLeftEdge { get; }
public int ModeBlockToRightEdge { get; }
public int ModeBlockToTopEdge { get; }
public int ModeBlockToBottomEdge { get; }
public bool HasTopRight { get; }
public int GetMaxBlockWide()
{
int width = this.BlockSize.GetWidth();
if (this.ModeBlockToRightEdge < 0)
{
width += this.ModeBlockToRightEdge >> 3;
}
return width >> Av1Constants.ModeInfoSizeLog2;
}
public int GetMaxBlockHigh()
{
int height = this.BlockSize.GetHeight();
if (this.ModeBlockToBottomEdge < 0)
{
height += this.ModeBlockToBottomEdge >> 3;
}
return height >> Av1Constants.ModeInfoSizeLog2;
}
public ReferenceBlock GetModeInfoAt(Point position)
{
Av1PictureControlSet? picture = this.encodedPicture;
if (picture is not null)
{
Av1MacroBlockModeInfo encodedModeInfo = picture.GetFromModeInfoGrid(position);
return new ReferenceBlock(
encodedModeInfo.Block.BlockSize,
encodedModeInfo.Block.UseIntraBlockCopy,
picture.GetDisplacementVector(position));
}
Av1BlockModeInfo decodedModeInfo = this.decodedSuperblock.GetModeInfoAt(position);
return new ReferenceBlock(
decodedModeInfo.BlockSize,
decodedModeInfo.UseIntraBlockCopy,
decodedModeInfo.DisplacementVector);
}
}
/// <summary>
/// Carries the three neighboring mode fields consumed by displacement-reference ranking.
/// </summary>
private readonly struct ReferenceBlock
{
public ReferenceBlock(
Av1BlockSize blockSize,
bool useIntraBlockCopy,
Av1MotionVector displacementVector)
{
this.BlockSize = blockSize;
this.UseIntraBlockCopy = useIntraBlockCopy;
this.DisplacementVector = displacementVector;
}
public Av1BlockSize BlockSize { get; }
public bool UseIntraBlockCopy { get; }
public Av1MotionVector DisplacementVector { get; }
}
}

20
src/ImageSharp/Formats/Heif/Av1/Tiling/Av1EncoderDisplacementVector.cs

@ -0,0 +1,20 @@
// Copyright (c) Six Labors.
// Licensed under the Six Labors Split License.
namespace SixLabors.ImageSharp.Formats.Heif.Av1.Tiling;
/// <summary>
/// Stores one encoder-selected integer displacement vector in the AV1 signed component domain.
/// </summary>
internal struct Av1EncoderDisplacementVector
{
/// <summary>
/// The vertical component in one-eighth-sample units.
/// </summary>
public short Row;
/// <summary>
/// The horizontal component in one-eighth-sample units.
/// </summary>
public short Column;
}

23
src/ImageSharp/Formats/Heif/Av1/Tiling/Av1EncoderPictureBuffer.cs

@ -84,7 +84,16 @@ internal sealed class Av1EncoderPictureBuffer : IDisposable
? checked(tileCount * paletteContextLength * Unsafe.SizeOf<Av1EncoderPaletteInfo>())
: 0;
int stateStorageLength = checked(paletteStorageOffset + paletteStorageLength);
int paletteStorageEnd = checked(paletteStorageOffset + paletteStorageLength);
int displacementVectorLength = frameHeader.AllowIntraBlockCopy ? this.modeInfo.Allocation.Length : 0;
int displacementVectorStorageOffset = frameHeader.AllowIntraBlockCopy
? Av1Math.AlignPowerOf2(paletteStorageEnd, 1)
: paletteStorageEnd;
int displacementVectorStorageLength = checked(
displacementVectorLength * Unsafe.SizeOf<Av1EncoderDisplacementVector>());
int stateStorageLength = checked(displacementVectorStorageOffset + displacementVectorStorageLength);
// Segmentation and every tile edge share one clean picture lifetime. The partition region begins at its
// native alignment, while typed views keep the entropy writer independent from the packed byte owner.
@ -118,6 +127,17 @@ internal sealed class Av1EncoderPictureBuffer : IDisposable
this.paletteContexts = [];
}
Memory<Av1EncoderDisplacementVector> displacementVectors = Memory<Av1EncoderDisplacementVector>.Empty;
if (frameHeader.AllowIntraBlockCopy)
{
// Each component lies strictly inside plus or minus 16384, so two signed 16-bit fields preserve the
// complete syntax domain without expanding every frame's compact mode-information allocation.
ByteMemoryManager<Av1EncoderDisplacementVector> displacementVectorMemory = new(
stateStorage.Slice(displacementVectorStorageOffset, displacementVectorStorageLength));
displacementVectors = displacementVectorMemory.Memory;
}
int[][] cdefPreset = new int[tileCount][];
int[] previousQIndex = new int[tileCount];
for (int tileIndex = 0; tileIndex < tileCount; tileIndex++)
@ -206,6 +226,7 @@ internal sealed class Av1EncoderPictureBuffer : IDisposable
SegmentationNeighborMap = stateStorage[..segmentationLength],
ModeInfoGrid = this.modeInfo.Grid,
ModeInfoAllocation = this.modeInfo.Allocation,
DisplacementVectors = displacementVectors,
ModeInfoStride = this.modeInfo.ModeInfoStride,
Disallow4x4AllFrames = this.modeInfo.Disallow4x4AllFrames,
CdefPreset = cdefPreset

45
src/ImageSharp/Formats/Heif/Av1/Tiling/Av1PartitionInfo.cs

@ -309,20 +309,41 @@ internal ref struct Av1PartitionInfo
/// </summary>
/// <param name="superblockModeInfoSize">The superblock width in 4x4 mode-information units.</param>
/// <returns><see langword="true"/> when the top-right block has already been decoded; otherwise, <see langword="false"/>.</returns>
public bool HasTopRight(int superblockModeInfoSize)
public bool HasTopRight(int superblockModeInfoSize) => HasTopRight(
this.ModeInfo.BlockSize,
this.Type,
this.RowIndex,
this.ColumnIndex,
superblockModeInfoSize);
/// <summary>
/// Determines whether a block can use its top-right search position from raster and partition geometry.
/// </summary>
/// <param name="blockSize">The current block size.</param>
/// <param name="partitionType">The partition type that produced the block.</param>
/// <param name="rowIndex">The block row in 4x4 mode-information units.</param>
/// <param name="columnIndex">The block column in 4x4 mode-information units.</param>
/// <param name="superblockModeInfoSize">The superblock width in 4x4 mode-information units.</param>
/// <returns><see langword="true"/> when the top-right block precedes the current block; otherwise, <see langword="false"/>.</returns>
internal static bool HasTopRight(
Av1BlockSize blockSize,
Av1PartitionType partitionType,
int rowIndex,
int columnIndex,
int superblockModeInfoSize)
{
int width = this.ModeInfo.BlockSize.Get4x4WideCount();
int height = this.ModeInfo.BlockSize.Get4x4HighCount();
int blockSize = Math.Max(width, height);
if (blockSize > 16)
int width = blockSize.Get4x4WideCount();
int height = blockSize.Get4x4HighCount();
int squareSize = Math.Max(width, height);
if (squareSize > 16)
{
return false;
}
int row = this.RowIndex & (superblockModeInfoSize - 1);
int column = this.ColumnIndex & (superblockModeInfoSize - 1);
bool hasTopRight = !((row & blockSize) != 0 && (column & blockSize) != 0);
int traversalSize = blockSize;
int row = rowIndex & (superblockModeInfoSize - 1);
int column = columnIndex & (superblockModeInfoSize - 1);
bool hasTopRight = !((row & squareSize) != 0 && (column & squareSize) != 0);
int traversalSize = squareSize;
// Split partitions decode three quadrants before the bottom-right quadrant. Walking the enclosing split levels
// excludes a right-hand block whenever traversal has not reached that block yet.
@ -344,18 +365,18 @@ internal ref struct Av1PartitionInfo
// Rectangular partitions override the square traversal rule because their sub-blocks are decoded along the
// long axis. Earlier vertical rectangles have a completed row above; later horizontal rectangles do not.
if (width < height && ((this.ColumnIndex + width) & (height - 1)) != 0)
if (width < height && ((columnIndex + width) & (height - 1)) != 0)
{
hasTopRight = true;
}
if (width > height && (this.RowIndex & (width - 1)) != 0)
if (width > height && (rowIndex & (width - 1)) != 0)
{
hasTopRight = false;
}
// The lower-left square of a vertical-A partition precedes its right-hand rectangle in bitstream order.
if (this.Type == Av1PartitionType.VerticalA && width == height && (row & traversalSize) != 0)
if (partitionType == Av1PartitionType.VerticalA && width == height && (row & traversalSize) != 0)
{
hasTopRight = false;
}

37
src/ImageSharp/Formats/Heif/Av1/Tiling/Av1PictureControlSet.cs

@ -1,6 +1,8 @@
// Copyright (c) Six Labors.
// Licensed under the Six Labors Split License.
using SixLabors.ImageSharp.Formats.Heif.Av1.Motion;
namespace SixLabors.ImageSharp.Formats.Heif.Av1.Tiling;
/// <summary>
@ -63,6 +65,11 @@ internal class Av1PictureControlSet
/// </summary>
public required Memory<Av1MacroBlockModeInfo> ModeInfoAllocation { get; set; }
/// <summary>
/// Gets or sets the packed displacement vectors present only when the frame permits intra-block copy.
/// </summary>
public Memory<Av1EncoderDisplacementVector> DisplacementVectors { get; set; }
/// <summary>
/// Gets or sets the row stride of <see cref="ModeInfoGrid"/> in 4x4 mode-information units.
/// </summary>
@ -90,6 +97,36 @@ internal class Av1PictureControlSet
return ref this.ModeInfoAllocation.Span[allocationOffset];
}
/// <summary>
/// Gets the displacement vector mapped to a frame position.
/// </summary>
/// <param name="position">The frame position in 4x4 mode-information units.</param>
/// <returns>The displacement vector retained for the covering block.</returns>
public Av1MotionVector GetDisplacementVector(Point position)
{
int gridOffset = (position.Y * this.ModeInfoStride) + position.X;
int allocationOffset = this.ModeInfoGrid.Span[gridOffset];
Av1EncoderDisplacementVector vector = this.DisplacementVectors.Span[allocationOffset];
return new Av1MotionVector(vector.Row, vector.Column);
}
/// <summary>
/// Stores the displacement vector selected at a block origin.
/// </summary>
/// <param name="modeInfoPosition">The block position in 4x4 mode-information units.</param>
/// <param name="vector">The selected integer displacement vector.</param>
public void SetDisplacementVector(Point modeInfoPosition, Av1MotionVector vector)
{
int modeInfoStride = this.ModeInfoStride;
int disallow4x4 = this.Disallow4x4AllFrames ? 1 : 0;
int allocationOffset = ((modeInfoPosition.Y >> disallow4x4) * (modeInfoStride >> disallow4x4)) + (modeInfoPosition.X >> disallow4x4);
this.DisplacementVectors.Span[allocationOffset] = new Av1EncoderDisplacementVector
{
Row = (short)vector.Row,
Column = (short)vector.Column
};
}
/// <summary>
/// Gets the macroblock mode information allocated at a block origin.
/// </summary>

38
src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileWriter.cs

@ -2,6 +2,7 @@
// Licensed under the Six Labors Split License.
using SixLabors.ImageSharp.Formats.Heif.Av1.Entropy;
using SixLabors.ImageSharp.Formats.Heif.Av1.Motion;
using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit;
using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction;
using SixLabors.ImageSharp.Formats.Heif.Av1.Transform;
@ -772,7 +773,12 @@ internal partial class Av1TileWriter
Av1ChromaPredictionMode intra_chroma_mode = macroBlockModeInfo.Block.UvMode;
if (IsIntraBlockCopyAllowed(pcs.Parent.FrameHeader/*, pcs.Parent.SliceType*/))
{
WriteIntraBlockCopyInfo(writer, macroBlockModeInfo);
WriteIntraBlockCopyInfo(
pcs,
writer,
macroBlock,
modeInfoPosition,
macroBlockModeInfo);
}
if (!macroBlockModeInfo.Block.UseIntraBlockCopy)
@ -1299,18 +1305,36 @@ internal partial class Av1TileWriter
/// <summary>
/// Writes the intra-block-copy selection and displacement-vector syntax for a block.
/// </summary>
/// <param name="picture">The frame-owned mode and displacement state.</param>
/// <param name="writer">The tile symbol encoder.</param>
/// <param name="macroBlock">The current block's frame edges and tile availability.</param>
/// <param name="modeInfoPosition">The block origin in 4x4 mode-information units.</param>
/// <param name="macroBlockModeInfo">The selected block modes.</param>
/// <exception cref="NotImplementedException">The displacement-vector syntax is not implemented when intra block copy is selected.</exception>
private static void WriteIntraBlockCopyInfo(
public static void WriteIntraBlockCopyInfo(
Av1PictureControlSet picture,
Av1SymbolEncoder writer,
Av1MacroBlockD macroBlock,
Point modeInfoPosition,
Av1MacroBlockModeInfo macroBlockModeInfo)
{
bool use_intrabc = macroBlockModeInfo.Block.UseIntraBlockCopy;
writer.WriteUseIntraBlockCopy(use_intrabc);
if (use_intrabc)
bool useIntraBlockCopy = macroBlockModeInfo.Block.UseIntraBlockCopy;
writer.WriteUseIntraBlockCopy(useIntraBlockCopy);
if (useIntraBlockCopy)
{
throw new NotImplementedException("Intra block code encoding not implemented.");
Span<Av1MotionVector> candidates = stackalloc Av1MotionVector[8];
Span<int> weights = stackalloc int[8];
Av1MotionVector reference = Av1IntraBlockCopy.FindReference(
picture,
macroBlock,
modeInfoPosition,
macroBlockModeInfo.Block.BlockSize,
macroBlockModeInfo.Block.PartitionType,
candidates,
weights);
writer.WriteDisplacementVector(
picture.GetDisplacementVector(modeInfoPosition),
reference);
}
}

30
tests/ImageSharp.Tests/Formats/Heif/Av1/Av1EncoderModeInfoBufferTests.cs

@ -2,6 +2,7 @@
// Licensed under the Six Labors Split License.
using SixLabors.ImageSharp.Formats.Heif.Av1;
using SixLabors.ImageSharp.Formats.Heif.Av1.Motion;
using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit;
using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction;
using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling;
@ -47,10 +48,12 @@ public class Av1EncoderModeInfoBufferTests
}
[Theory]
[InlineData(false, 336)]
[InlineData(true, 3_536)]
[InlineData(false, false, 336)]
[InlineData(true, false, 3_536)]
[InlineData(true, true, 4_560)]
public unsafe void PictureBufferPacksAllPictureStateIntoTwoAllocatorOwners(
bool allowScreenContentTools,
bool allowIntraBlockCopy,
int expectedStateStorageLength)
{
const int Width = 16;
@ -84,6 +87,7 @@ public class Av1EncoderModeInfoBufferTests
ObuFrameHeader frameHeader = new()
{
AllowScreenContentTools = allowScreenContentTools,
AllowIntraBlockCopy = allowIntraBlockCopy,
ModeInfoColumnCount = Width >> Av1Constants.ModeInfoSizeLog2,
ModeInfoRowCount = Height >> Av1Constants.ModeInfoSizeLog2,
TilesInfo = tiles
@ -139,6 +143,28 @@ public class Av1EncoderModeInfoBufferTests
{
Assert.Empty(picture.PaletteContexts);
}
if (allowIntraBlockCopy)
{
Assert.Equal(256, picture.DisplacementVectors.Length);
Assert.Equal(4, sizeof(Av1EncoderDisplacementVector));
// The packed vector region starts at its natural 16-bit alignment inside the shared byte owner.
fixed (Av1EncoderDisplacementVector* pointer = picture.DisplacementVectors.Span)
{
Assert.Equal((nuint)0, (nuint)pointer % (nuint)sizeof(short));
}
Point position = new(2, 2);
Av1MotionVector displacement = new(-16376, 16376);
picture.MapModeInfoBlock(position, Av1BlockSize.Block8x8);
picture.SetDisplacementVector(position, displacement);
Assert.Equal(displacement, picture.GetDisplacementVector(new Point(3, 3)));
}
else
{
Assert.Equal(0, picture.DisplacementVectors.Length);
}
}
Assert.Equal(2, allocator.ReturnLog.Count);

134
tests/ImageSharp.Tests/Formats/Heif/Av1/Av1IntraBlockCopyTests.cs

@ -2,9 +2,11 @@
// Licensed under the Six Labors Split License.
using SixLabors.ImageSharp.Formats.Heif.Av1;
using SixLabors.ImageSharp.Formats.Heif.Av1.Entropy;
using SixLabors.ImageSharp.Formats.Heif.Av1.Motion;
using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit;
using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling;
using SixLabors.ImageSharp.Memory;
namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1;
@ -79,6 +81,128 @@ public class Av1IntraBlockCopyTests
Assert.Equal(new Av1MotionVector(-512, 0), actual);
}
/// <summary>
/// Verifies that encoder mode aliases and packed displacement storage feed the shared spatial ranking.
/// </summary>
[Fact]
public void EncoderReferenceUsesMappedIntraBlockCopyNeighbor()
{
const int Width = 640;
const int Height = 256;
ObuSequenceHeader sequenceHeader = CreateSequenceHeader();
ObuFrameHeader frameHeader = CreateFrameHeader();
frameHeader.AllowScreenContentTools = true;
frameHeader.AllowIntraBlockCopy = true;
using Av1EncoderPictureBuffer buffer = new(
Configuration.Default,
sequenceHeader,
frameHeader,
Width,
Height);
Av1PictureControlSet picture = buffer.Picture;
Point candidatePosition = new(80, 12);
ref Av1MacroBlockModeInfo candidate = ref picture.GetMacroBlockModeInfo(candidatePosition);
candidate.Block = new Av1EncoderBlockModeInfo
{
BlockSize = Av1BlockSize.Block16x16,
PartitionType = Av1PartitionType.None,
UseIntraBlockCopy = true
};
Av1MotionVector displacement = new(0, -2560);
picture.MapModeInfoBlock(candidatePosition, candidate.Block.BlockSize);
picture.SetDisplacementVector(candidatePosition, displacement);
Point currentPosition = new(80, 16);
Av1TileInfo tileInfo = new(0, 0, frameHeader);
Av1MacroBlockD macroBlock = new() { Tile = tileInfo };
Av1TileWriter.SetModeInfoRowAndColumn(
picture,
macroBlock,
tileInfo,
currentPosition,
Av1BlockSize.Block16x16,
picture.ModeInfoStride,
picture.Parent.Common.ModeInfoRowCount,
picture.Parent.Common.ModeInfoColumnCount);
Span<Av1MotionVector> candidates = stackalloc Av1MotionVector[8];
Span<int> weights = stackalloc int[8];
Av1MotionVector actual = Av1IntraBlockCopy.FindReference(
picture,
macroBlock,
currentPosition,
Av1BlockSize.Block16x16,
Av1PartitionType.None,
candidates,
weights);
Assert.Equal(displacement, actual);
}
/// <summary>
/// Verifies that the tile writer derives the encoder reference and emits the retained displacement.
/// </summary>
[Fact]
public void TileWriterEmitsRetainedDisplacementAgainstDerivedReference()
{
const int Width = 640;
const int Height = 256;
ObuSequenceHeader sequenceHeader = CreateSequenceHeader();
ObuFrameHeader frameHeader = CreateFrameHeader();
frameHeader.AllowScreenContentTools = true;
frameHeader.AllowIntraBlockCopy = true;
using Av1EncoderPictureBuffer buffer = new(
Configuration.Default,
sequenceHeader,
frameHeader,
Width,
Height);
Av1PictureControlSet picture = buffer.Picture;
Point modeInfoPosition = new(80, 0);
Av1MacroBlockModeInfo modeInfo = default;
modeInfo.Block = new Av1EncoderBlockModeInfo
{
BlockSize = Av1BlockSize.Block16x16,
PartitionType = Av1PartitionType.None,
UseIntraBlockCopy = true
};
Av1MotionVector displacement = new(0, -3072);
picture.MapModeInfoBlock(modeInfoPosition, modeInfo.Block.BlockSize);
picture.SetDisplacementVector(modeInfoPosition, displacement);
Av1TileInfo tileInfo = new(0, 0, frameHeader);
Av1MacroBlockD macroBlock = new() { Tile = tileInfo };
Av1TileWriter.SetModeInfoRowAndColumn(
picture,
macroBlock,
tileInfo,
modeInfoPosition,
modeInfo.Block.BlockSize,
picture.ModeInfoStride,
picture.Parent.Common.ModeInfoRowCount,
picture.Parent.Common.ModeInfoColumnCount);
using Av1SymbolEncoder writer = new(Configuration.Default, 64, 0);
Av1TileWriter.WriteIntraBlockCopyInfo(
picture,
writer,
macroBlock,
modeInfoPosition,
modeInfo);
using var encoded = writer.Exit();
Av1SymbolDecoder decoder = new(Configuration.Default, encoded.GetSpan(), 0);
Assert.True(decoder.ReadUseIntraBlockCopy());
Assert.Equal(
displacement,
decoder.ReadDisplacementVector(new Av1MotionVector(0, -2560)));
}
/// <summary>
/// Verifies tile bounds, whole-sample precision, the four-block delay, and wavefront ordering.
/// </summary>
@ -129,6 +253,12 @@ public class Av1IntraBlockCopyTests
/// Creates one tile covering the complete test frame.
/// </summary>
private static Av1TileInfo CreateTileInfo()
=> new(0, 0, CreateFrameHeader());
/// <summary>
/// Creates the frame and tile geometry shared by reference and validity tests.
/// </summary>
private static ObuFrameHeader CreateFrameHeader()
{
ObuTileGroupHeader tilesInfo = new()
{
@ -139,13 +269,11 @@ public class Av1IntraBlockCopyTests
tilesInfo.TileColumnStartModeInfo[1] = 160;
tilesInfo.TileRowStartModeInfo[1] = 64;
ObuFrameHeader frameHeader = new()
return new ObuFrameHeader
{
ModeInfoColumnCount = 160,
ModeInfoRowCount = 64,
TilesInfo = tilesInfo,
};
return new Av1TileInfo(0, 0, frameHeader);
}
}

Loading…
Cancel
Save