Browse Source

Decode AV1 inter-frame intra blocks

pull/2633/head
James Jackson-South 6 days ago
parent
commit
288d8b6d00
  1. 10
      HEIF_IMPLEMENTATION_PLAN.md
  2. 15
      src/ImageSharp/Formats/Heif/Av1/Entropy/Av1DefaultDistributions.cs
  3. 27
      src/ImageSharp/Formats/Heif/Av1/Entropy/Av1FrameEntropyContext.cs
  4. 51
      src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolContextHelper.cs
  5. 36
      src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolDecoder.cs
  6. 23
      src/ImageSharp/Formats/Heif/Av1/OpenBitstreamUnit/ObuOrderHintInfo.cs
  7. 20
      src/ImageSharp/Formats/Heif/Av1/OpenBitstreamUnit/ObuReader.cs
  8. 95
      src/ImageSharp/Formats/Heif/Av1/OpenBitstreamUnit/ObuSkipModeParameters.cs
  9. 5
      src/ImageSharp/Formats/Heif/Av1/Tiling/Av1BlockModeInfo.cs
  10. 38
      src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.MotionField.cs
  11. 19
      src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.cs
  12. 268
      src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileReader.cs
  13. 257
      tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterFrameIntraEntropyTests.cs
  14. 133
      tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterFrameModeInfoTests.cs
  15. 275
      tests/ImageSharp.Tests/Formats/Heif/Av1/Av1TemporalSegmentationTests.cs
  16. 1
      tests/ImageSharp.Tests/Formats/Heif/Av1/ObuFrameLifecycleTests.cs
  17. 246
      tests/ImageSharp.Tests/Formats/Heif/Av1/ObuSkipModeParametersTests.cs

10
HEIF_IMPLEMENTATION_PLAN.md

@ -38,7 +38,7 @@ Status meanings:
- **Not started:** supporting primitives may exist, but the production format path is absent.
- **Current:** the only work item that should be advanced before taking the next queued item.
Current development stage: **Stage 3 — complete AV1 still-image decoding.** The transform checkpoint is closed: forward transforms use one libaom-shaped SIMD-first operator architecture across `Vector512`, `Vector256`, and `Vector128`, with scalar fallback; inverse production traversal uses the verified `Vector256` and `Vector128` tiers with scalar fallback; and implementation-mechanic type and file suffixes have been removed. Neither AV1 nor HEVC production encoding is implemented.
Current development stage: **Stage 3 — complete AV1 still-image decoding.** The transform checkpoint is closed. Layered decoding now retains reference/header/CDF/motion-field state, derives frame-level skip-mode references, consumes temporal segment prediction, and decodes intra-coded blocks inside inter frames. True inter-coded blocks still stop before reference/MV parsing and reconstruction. Neither AV1 nor HEVC production encoding is implemented.
Immediate checkpoint: **complete layered AV1 image-item decoding through the existing image-only container surface.** This includes `a1op`, `lsel`, and `a1lx` properties, operating-point selection, dependency-preserving layer consumption, and final or explicitly selected spatial-layer output for color, alpha, and grid items. The bounded decoder session now retains reference owners and the header, entropy, segmentation, loop-filter, global-motion, and temporal motion-field state required by dependent layers. Inter tile syntax and reconstruction still need to consume that state. This work must not be represented as animation or expanded into a general ISO BMFF/video model.
@ -46,7 +46,7 @@ Immediate checkpoint: **complete layered AV1 image-item decoding through the exi
| --- | --- | --- | --- | --- |
| 1 | Baseline, provenance, documentation, and public contract | In progress | Pinned codec references, a bounded image-only scope, encoder options, typed bit depth, decoder-option propagation, and extensive HEIF documentation exist. | Complete the all-file documentation audit, record a fresh Release baseline, finish distinct public HEIC/AVIF save boundaries, and close API review. |
| 2 | Bounded HEIF item and image-sequence container | In progress | Still-item parsing, grids, auxiliary alpha, metadata properties, bounded image-sequence tracks, Identify, and all-sync AV1 sequence presentation are connected. | Complete adversarial boundary coverage, remaining item/property behavior, reference-dependent sequence reconstruction, and the bounded sequence writer. |
| 3 | Still-image AV1 and HEVC decoding | **Current** | HEVC reconstruction reaches exact HM/libheif fixtures across the recorded 8/10/12-bit and chroma cases. AV1 includes bounded OBU framing, reconstruction, filters, grain, color, transforms, intra-block copy, an exact independent 12-profile bit-depth/chroma matrix through every dispatch tier, and retained layered reference/header/CDF state. | Complete inter tile syntax and reconstruction, including temporal segmentation consumption and motion/global/warped prediction, remove every other valid AV1 still-image unsupported branch with independent compression-tool vectors, then complete the remaining HEVC profile and Range Extensions matrix. |
| 3 | Still-image AV1 and HEVC decoding | **Current** | HEVC reconstruction reaches exact HM/libheif fixtures across the recorded 8/10/12-bit and chroma cases. AV1 includes bounded OBU framing, reconstruction, filters, grain, color, transforms, intra-block copy, an exact independent 12-profile bit-depth/chroma matrix through every dispatch tier, retained layered reference/header/CDF state, temporal segment prediction, and the inter-frame intra-coded-block branch. | Complete true inter tile syntax and reconstruction, including reference/MV, compound, global, and warped prediction; remove every other valid AV1 still-image unsupported branch with independent compression-tool vectors; then complete the remaining HEVC profile and Range Extensions matrix. |
| 4 | Complete decoded presentation and animation | In progress | Shared SIMD-first AV1/HEVC color conversion, ICC application, grids, transforms, direct planar alpha composition, frame metadata, repetition, and independently decodable AV1 sequence samples exist. | Close the full color/ICC cross-product, HEVC sequence decoding, AV1/HEVC reference-dependent samples, frame-local metadata/alpha behavior, and independent animated decode vectors. |
| 5 | AV1/AVIF encoding | Not started | RGB-to-planar conversion, forward transforms, OBU writer foundations, options, and container-writing infrastructure exist. | `HeifEncoderCore` still rejects AV1. Implement a real independently decodable lossy/lossless AV1 payload and the complete AVIF item/metadata matrix. |
| 6 | HEVC/HEIC encoding | Not started | Shared input color conversion, options, and HEIF writer infrastructure exist. | `HeifEncoderCore` still rejects HEVC. Implement a real independently decodable lossy/lossless HEVC payload and the complete HEIC item/metadata matrix. |
@ -68,10 +68,12 @@ Immediate checkpoint: **complete layered AV1 image-item decoding through the exi
- [x] Parse full and short reference signaling against retained-slot occupancy separately from frame-ID validity, resolve the primary-reference slot, implement `frame_size_with_refs`, and read high-precision-motion-vector, interpolation-filter, switchable-motion-mode, and reference-frame-motion-vector flags.
- [x] Initialize each frame's working CDF graph from the resolved primary snapshot, inherit segmentation feature data and an unchanged contiguous segmentation map, inherit loop-filter delta state, parse and inherit global-motion parameters, and initialize/project the retained per-8x8 temporal motion field.
- [x] Match libaom's bounded range-decoder consumed-bit accounting and tile trailing-bit validation so implicit zero refill cannot conceal truncated entropy data. Validate before publishing the selected CDF, reset decoder state after failure, and unwind every successful frame/block workspace rent when a later constructor allocation fails. A real truncated palette tile, decoder reuse, parser lifecycle, and allocator-identity tests cover these boundaries.
- [ ] Consume the retained segmentation map through `seg_id_predicted` when temporal segmentation is enabled. The current intra-only tile path does not read that inter-block decision.
- [x] Consume the retained segmentation map through `seg_id_predicted` when temporal segmentation is enabled, including all three neighbor CDF contexts, pre-skip/post-skip ordering, skipped-block spatial inference, compatible retained-map geometry, and current-map updates.
- [ ] Implement the complete inter-frame entropy, mode, motion-vector, compound-prediction, inter-prediction, and warped/global-motion paths permitted by the image profile.
- [x] Derive the frame-level skip-mode reference pair from mapped order hints, including modulo wraparound and the two-forward fallback, then decode the common inter-frame block prefix and intra-coded-block branch with retained CDF state and block-size luma contexts.
- [x] Implement allocation-free SIMD-first translational single-reference interpolation for regular, smooth, sharp, and bilinear filters across 8/10/12-bit samples. The predictor mirrors JPEG's closed static operator architecture, descends through `Vector512`, `Vector256`, and `Vector128` before scalar fallback, and passes the exact independent convolution oracle through `FeatureTestRunner`.
- [ ] Decode inter-block reference selection, motion-vector derivation, compound and inter-intra modes, masked blending, OBMC, and warped/global-motion prediction, then write predicted and residual samples through tile reconstruction.
- [ ] **Current:** decode single-reference inter selection, the spatial/temporal reference-MV stack, NEAREST/NEAR/NEW/GLOBAL motion modes, DRL and interpolation filters, then reconstruct the complete block once through the existing SIMD-first translational predictor before residual traversal.
- [ ] Decode compound and inter-intra modes, masked blending, OBMC, scaled references, and warped/global-motion prediction without changing the single-reference predictor contract or rounding model.
- [ ] Verify every connected mode and filter with independently encoded dependent-layer AV1 image-item fixtures and exact native-plane comparisons.
- [ ] Return the explicitly selected spatial layer or the final displayed layer, keeping reference reconstruction separate from display-only film grain.
- [ ] Verify color and auxiliary-alpha output exactly against both pinned libavif progressive fixtures under normal SIMD dispatch and all required `FeatureTestRunner` fallbacks.

15
src/ImageSharp/Formats/Heif/Av1/Entropy/Av1DefaultDistributions.cs

@ -9,7 +9,7 @@ namespace SixLabors.ImageSharp.Formats.Heif.Av1.Entropy;
internal static class Av1DefaultDistributions
{
/// <summary>
/// Gets the intra-frame luma-mode distributions indexed by block-size group.
/// Gets the inter-frame intra luma-mode distributions indexed by block-size group.
/// </summary>
public static Av1Distribution[] FrameYMode =>
[
@ -19,6 +19,11 @@ internal static class Av1DefaultDistributions
new(20155, 21301, 22838, 23178, 23261, 23533, 23703, 24804, 25352, 26575, 27016, 28049)
];
/// <summary>
/// Gets the distributions that select intra or inter prediction from the available spatial neighbors.
/// </summary>
public static Av1Distribution[] IntraInter => [new(806), new(16662), new(20186), new(26538)];
/// <summary>
/// Gets the key-frame luma-mode distributions indexed by the above and left intra-mode contexts.
/// </summary>
@ -260,6 +265,14 @@ internal static class Av1DefaultDistributions
new(27527, 28487, 28723, 28890, 32397, 32647, 32679),
];
/// <summary>
/// Gets the temporal segment-map prediction distributions indexed by the predicted state of the above and left blocks.
/// </summary>
/// <remarks>
/// AV1 initializes each binary context to 16384, the equiprobable midpoint of its Q15 probability domain.
/// </remarks>
public static Av1Distribution[] SegmentIdPredicted => [new(16384), new(16384), new(16384)];
/// <summary>
/// Gets the key-frame luma intra-mode distributions indexed by the above and left mode contexts.
/// </summary>

27
src/ImageSharp/Formats/Heif/Av1/Entropy/Av1FrameEntropyContext.cs

@ -76,13 +76,16 @@ internal sealed class Av1FrameEntropyContext
this.PaletteYColorIndex = Av1DefaultDistributions.PaletteYColorIndex;
this.PaletteUvColorIndex = Av1DefaultDistributions.PaletteUvColorIndex;
this.PartitionTypes = Av1DefaultDistributions.PartitionTypes;
this.FrameYMode = Av1DefaultDistributions.FrameYMode;
this.KeyFrameYMode = Av1DefaultDistributions.KeyFrameYMode;
this.IntraInter = Av1DefaultDistributions.IntraInter;
this.UvMode = Av1DefaultDistributions.UvMode;
this.Skip = Av1DefaultDistributions.Skip;
this.SkipMode = Av1DefaultDistributions.SkipMode;
this.DeltaLoopFilterAbsolute = Av1DefaultDistributions.DeltaLoopFilterAbsolute;
this.DeltaQuantizerAbsolute = Av1DefaultDistributions.DeltaQuantizerAbsolute;
this.SegmentId = Av1DefaultDistributions.SegmentId;
this.SegmentIdPredicted = Av1DefaultDistributions.SegmentIdPredicted;
this.AngleDelta = Av1DefaultDistributions.AngleDelta;
this.FilterIntraMode = Av1DefaultDistributions.FilterIntraMode;
this.FilterIntra = Av1DefaultDistributions.FilterIntra;
@ -124,13 +127,16 @@ internal sealed class Av1FrameEntropyContext
this.PaletteYColorIndex = Av1Distribution.CreateCopy(source.PaletteYColorIndex);
this.PaletteUvColorIndex = Av1Distribution.CreateCopy(source.PaletteUvColorIndex);
this.PartitionTypes = Av1Distribution.CreateCopy(source.PartitionTypes);
this.FrameYMode = Av1Distribution.CreateCopy(source.FrameYMode);
this.KeyFrameYMode = Av1Distribution.CreateCopy(source.KeyFrameYMode);
this.IntraInter = Av1Distribution.CreateCopy(source.IntraInter);
this.UvMode = Av1Distribution.CreateCopy(source.UvMode);
this.Skip = Av1Distribution.CreateCopy(source.Skip);
this.SkipMode = Av1Distribution.CreateCopy(source.SkipMode);
this.DeltaLoopFilterAbsolute = source.DeltaLoopFilterAbsolute.CreateCopy();
this.DeltaQuantizerAbsolute = source.DeltaQuantizerAbsolute.CreateCopy();
this.SegmentId = Av1Distribution.CreateCopy(source.SegmentId);
this.SegmentIdPredicted = Av1Distribution.CreateCopy(source.SegmentIdPredicted);
this.AngleDelta = Av1Distribution.CreateCopy(source.AngleDelta);
this.FilterIntraMode = source.FilterIntraMode.CreateCopy();
this.FilterIntra = Av1Distribution.CreateCopy(source.FilterIntra);
@ -208,11 +214,21 @@ internal sealed class Av1FrameEntropyContext
/// </summary>
public Av1Distribution[] PartitionTypes { get; }
/// <summary>
/// Gets the inter-frame intra luma-mode distributions indexed by the normative block-size group.
/// </summary>
public Av1Distribution[] FrameYMode { get; }
/// <summary>
/// Gets the key-frame luma-mode distributions.
/// </summary>
public Av1Distribution[][] KeyFrameYMode { get; }
/// <summary>
/// Gets the distributions that select intra or inter prediction from the available spatial neighbors.
/// </summary>
public Av1Distribution[] IntraInter { get; }
/// <summary>
/// Gets the chroma intra-mode distributions.
/// </summary>
@ -243,6 +259,11 @@ internal sealed class Av1FrameEntropyContext
/// </summary>
public Av1Distribution[] SegmentId { get; }
/// <summary>
/// Gets the temporal segment-map prediction distributions.
/// </summary>
public Av1Distribution[] SegmentIdPredicted { get; }
/// <summary>
/// Gets the directional angle-delta distributions.
/// </summary>
@ -363,13 +384,16 @@ internal sealed class Av1FrameEntropyContext
CopyState(source.PaletteYColorIndex, this.PaletteYColorIndex);
CopyState(source.PaletteUvColorIndex, this.PaletteUvColorIndex);
CopyState(source.PartitionTypes, this.PartitionTypes);
CopyState(source.FrameYMode, this.FrameYMode);
CopyState(source.KeyFrameYMode, this.KeyFrameYMode);
CopyState(source.IntraInter, this.IntraInter);
CopyState(source.UvMode, this.UvMode);
CopyState(source.Skip, this.Skip);
CopyState(source.SkipMode, this.SkipMode);
this.DeltaLoopFilterAbsolute.CopyFrom(source.DeltaLoopFilterAbsolute);
this.DeltaQuantizerAbsolute.CopyFrom(source.DeltaQuantizerAbsolute);
CopyState(source.SegmentId, this.SegmentId);
CopyState(source.SegmentIdPredicted, this.SegmentIdPredicted);
CopyState(source.AngleDelta, this.AngleDelta);
this.FilterIntraMode.CopyFrom(source.FilterIntraMode);
CopyState(source.FilterIntra, this.FilterIntra);
@ -418,13 +442,16 @@ internal sealed class Av1FrameEntropyContext
ResetUpdateCounts(this.PaletteYColorIndex);
ResetUpdateCounts(this.PaletteUvColorIndex);
ResetUpdateCounts(this.PartitionTypes);
ResetUpdateCounts(this.FrameYMode);
ResetUpdateCounts(this.KeyFrameYMode);
ResetUpdateCounts(this.IntraInter);
ResetUpdateCounts(this.UvMode);
ResetUpdateCounts(this.Skip);
ResetUpdateCounts(this.SkipMode);
this.DeltaLoopFilterAbsolute.ResetUpdateCount();
this.DeltaQuantizerAbsolute.ResetUpdateCount();
ResetUpdateCounts(this.SegmentId);
ResetUpdateCounts(this.SegmentIdPredicted);
ResetUpdateCounts(this.AngleDelta);
this.FilterIntraMode.ResetUpdateCount();
ResetUpdateCounts(this.FilterIntra);

51
src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolContextHelper.cs

@ -586,6 +586,57 @@ internal static class Av1SymbolContextHelper
public static int GetSegmentId(int[][] segmentIds, int rowIndex, int columnIndex)
=> segmentIds[rowIndex][columnIndex];
/// <summary>
/// Gets the intra/inter prediction context from the immediately above and left blocks.
/// </summary>
/// <param name="above">The above block, or <see langword="null"/> at a tile boundary.</param>
/// <param name="left">The left block, or <see langword="null"/> at a tile boundary.</param>
/// <returns>The context in the inclusive range zero through three.</returns>
public static int GetIntraInterContext(Av1BlockModeInfo? above, Av1BlockModeInfo? left)
{
if (above is not null && left is not null)
{
bool aboveIsIntra = above.ReferenceFrames[0] <= Av1ReferenceFrameType.Intra;
bool leftIsIntra = left.ReferenceFrames[0] <= Av1ReferenceFrameType.Intra;
// AV1 reserves context three for two intra neighbors, context one for a mixed pair, and context zero for
// two inter neighbors. These values directly index intra_inter_cdf and are not probability ranks.
if (aboveIsIntra && leftIsIntra)
{
return 3;
}
return aboveIsIntra || leftIsIntra ? 1 : 0;
}
// A single intra neighbor uses context two. A single inter neighbor and a block with no neighbors both use
// context zero, matching the unavailable-neighbor behavior in libaom's av1_get_intra_inter_context.
if (above is not null)
{
return above.ReferenceFrames[0] <= Av1ReferenceFrameType.Intra ? 2 : 0;
}
if (left is not null)
{
return left.ReferenceFrames[0] <= Av1ReferenceFrameType.Intra ? 2 : 0;
}
return 0;
}
/// <summary>
/// Gets the temporal segment-prediction context from the immediately above and left blocks.
/// </summary>
/// <param name="aboveModeInfo">The above block, or <see langword="null"/> at a tile boundary.</param>
/// <param name="leftModeInfo">The left block, or <see langword="null"/> at a tile boundary.</param>
/// <returns>The context in the inclusive range zero through two.</returns>
public static int GetSegmentIdPredictedContext(Av1BlockModeInfo? aboveModeInfo, Av1BlockModeInfo? leftModeInfo)
{
int abovePredicted = aboveModeInfo is not null && aboveModeInfo.SegmentIdPredicted ? 1 : 0;
int leftPredicted = leftModeInfo is not null && leftModeInfo.SegmentIdPredicted ? 1 : 0;
return abovePredicted + leftPredicted;
}
/// <summary>
/// Gets the minimum encoded segment identifier across a block's clipped mode-info coverage.
/// </summary>

36
src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolDecoder.cs

@ -351,6 +351,31 @@ internal ref struct Av1SymbolDecoder
return (Av1PredictionMode)r.ReadSymbol(this.context.KeyFrameYMode[aboveContext][leftContext]);
}
/// <summary>
/// Reads an intra luma prediction mode for a block coded inside an inter frame.
/// </summary>
/// <param name="blockSize">The decoded block size that selects the luma-mode distribution.</param>
/// <returns>The decoded intra luma prediction mode.</returns>
public Av1PredictionMode ReadInterFrameYMode(Av1BlockSize blockSize)
{
// AV1 section 9.3 groups blocks by the smaller base-two dimension in 4x4 units, capped at group three.
// Calculating it from the existing logarithms exactly matches libaom's size_group_lookup without another table.
int sizeGroup = Math.Min(3, Math.Min(blockSize.Get4x4WidthLog2(), blockSize.Get4x4HeightLog2()));
ref Av1SymbolReader r = ref this.reader;
return (Av1PredictionMode)r.ReadSymbol(this.context.FrameYMode[sizeGroup]);
}
/// <summary>
/// Reads whether an inter-frame block uses inter prediction.
/// </summary>
/// <param name="context">The spatial intra/inter context in the inclusive range zero through three.</param>
/// <returns><see langword="true"/> when the block uses inter prediction; otherwise, <see langword="false"/>.</returns>
public bool ReadIsInter(int context)
{
ref Av1SymbolReader r = ref this.reader;
return r.ReadSymbol(this.context.IntraInter[context]) != 0;
}
/// <summary>
/// Reads a chroma intra prediction mode conditioned on the luma mode and chroma-from-luma availability.
/// </summary>
@ -445,6 +470,17 @@ internal ref struct Av1SymbolDecoder
return r.ReadSymbol(this.context.SegmentId[context]);
}
/// <summary>
/// Reads whether the current segment identifier is predicted from the retained primary-frame map.
/// </summary>
/// <param name="context">The sum of the above and left blocks' temporal-prediction flags.</param>
/// <returns><see langword="true"/> when the retained map supplies the segment identifier.</returns>
public bool ReadSegmentIdPredicted(int context)
{
ref Av1SymbolReader r = ref this.reader;
return r.ReadSymbol(this.context.SegmentIdPredicted[context]) > 0;
}
/// <summary>
/// Reads the unsigned directional angle-delta symbol for a prediction mode.
/// </summary>

23
src/ImageSharp/Formats/Heif/Av1/OpenBitstreamUnit/ObuOrderHintInfo.cs

@ -27,4 +27,27 @@ internal class ObuOrderHintInfo
/// Gets or sets the number of bits used to encode order hints.
/// </summary>
public int OrderHintBits { get; set; }
/// <summary>
/// Computes the signed distance between two order hints in the sequence's modulo order-hint domain.
/// </summary>
/// <param name="first">The first order hint.</param>
/// <param name="second">The order hint subtracted from <paramref name="first"/>.</param>
/// <returns>
/// The shortest signed modulo distance, or zero when order hints are disabled for the sequence.
/// </returns>
public int GetRelativeDistance(uint first, uint second)
{
if (!this.EnableOrderHint)
{
return 0;
}
int difference = (int)first - (int)second;
int signBit = 1 << (this.OrderHintBits - 1);
// Folding around the sign bit maps the unsigned difference to [-2^(bits - 1), 2^(bits - 1)), including
// the wraparound between the highest encoded order hint and zero.
return (difference & (signBit - 1)) - (difference & signBit);
}
}

20
src/ImageSharp/Formats/Heif/Av1/OpenBitstreamUnit/ObuReader.cs

@ -2399,23 +2399,9 @@ internal class ObuReader
/// <param name="frameHeader">The frame header that receives the skip-mode state.</param>
private static void ReadSkipModeParameters(ref Av1BitStreamReader reader, ObuSequenceHeader sequenceHeader, ObuFrameHeader frameHeader)
{
if (frameHeader.IsIntra || frameHeader.ReferenceMode == ObuReferenceMode.SingleReference || !sequenceHeader.OrderHintInfo.EnableOrderHint)
{
frameHeader.SkipModeParameters.SkipModeAllowed = false;
}
else
{
// Not applicable for INTRA frames.
}
if (frameHeader.SkipModeParameters.SkipModeAllowed)
{
frameHeader.SkipModeParameters.SkipModeFlag = reader.ReadBoolean();
}
else
{
frameHeader.SkipModeParameters.SkipModeFlag = false;
}
ObuSkipModeParameters parameters = frameHeader.SkipModeParameters;
parameters.Derive(sequenceHeader.OrderHintInfo, frameHeader);
parameters.SkipModeFlag = parameters.SkipModeAllowed && reader.ReadBoolean();
}
/// <summary>

95
src/ImageSharp/Formats/Heif/Av1/OpenBitstreamUnit/ObuSkipModeParameters.cs

@ -1,6 +1,8 @@
// Copyright (c) Six Labors.
// Licensed under the Six Labors Split License.
using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling;
namespace SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit;
/// <summary>
@ -9,12 +11,99 @@ namespace SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit;
internal class ObuSkipModeParameters
{
/// <summary>
/// Gets or sets a value indicating whether the frame is permitted to use skip mode.
/// Gets a value indicating whether the frame is permitted to use skip mode.
/// </summary>
public bool SkipModeAllowed { get; set; }
public bool SkipModeAllowed { get; private set; }
/// <summary>
/// Gets or sets a value indicating whether skip mode is enabled for the frame.
/// </summary>
public bool SkipModeFlag { get; internal set; }
public bool SkipModeFlag { get; set; }
/// <summary>
/// Gets the first canonical inter-reference type selected for skip-mode blocks.
/// </summary>
public Av1ReferenceFrameType FirstReferenceFrame { get; private set; } = Av1ReferenceFrameType.None;
/// <summary>
/// Gets the second canonical inter-reference type selected for skip-mode blocks.
/// </summary>
public Av1ReferenceFrameType SecondReferenceFrame { get; private set; } = Av1ReferenceFrameType.None;
/// <summary>
/// Derives skip-mode availability and its reference pair from the current frame's retained-reference mapping.
/// </summary>
/// <param name="orderHintInfo">The sequence-level order-hint configuration.</param>
/// <param name="frameHeader">The current frame header and its seven canonical inter-reference mappings.</param>
public void Derive(ObuOrderHintInfo orderHintInfo, ObuFrameHeader frameHeader)
{
this.SkipModeAllowed = false;
this.FirstReferenceFrame = Av1ReferenceFrameType.None;
this.SecondReferenceFrame = Av1ReferenceFrameType.None;
if (!orderHintInfo.EnableOrderHint || frameHeader.IsIntra || frameHeader.ReferenceMode == ObuReferenceMode.SingleReference)
{
return;
}
ReadOnlySpan<uint> referenceFrameIndices = frameHeader.GetReferenceFrameIndices();
ReadOnlySpan<uint> referenceOrderHints = frameHeader.GetReferenceOrderHints();
int nearestForwardOrderHint = -1;
int nearestBackwardOrderHint = int.MaxValue;
int nearestForwardReferenceIndex = -1;
int nearestBackwardReferenceIndex = -1;
// The seven entries are canonical roles, while each value selects one physical reference-map slot. Compare
// the selected slot's order hint so duplicate roles retain the same deterministic ordering as libaom.
for (int referenceIndex = 0; referenceIndex < Av1Constants.ReferencesPerFrame; referenceIndex++)
{
uint referenceOrderHint = referenceOrderHints[(int)referenceFrameIndices[referenceIndex]];
int distanceFromCurrent = orderHintInfo.GetRelativeDistance(referenceOrderHint, frameHeader.OrderHint);
if (distanceFromCurrent < 0 &&
(nearestForwardOrderHint == -1 || orderHintInfo.GetRelativeDistance(referenceOrderHint, (uint)nearestForwardOrderHint) > 0))
{
// Among past frames, the greatest relative order is the closest frame before the current one.
nearestForwardOrderHint = (int)referenceOrderHint;
nearestForwardReferenceIndex = referenceIndex;
}
else if (distanceFromCurrent > 0 &&
(nearestBackwardOrderHint == int.MaxValue || orderHintInfo.GetRelativeDistance(referenceOrderHint, (uint)nearestBackwardOrderHint) < 0))
{
// Among future frames, the smallest relative order is the closest frame after the current one.
nearestBackwardOrderHint = (int)referenceOrderHint;
nearestBackwardReferenceIndex = referenceIndex;
}
}
if (nearestForwardReferenceIndex >= 0 && nearestBackwardReferenceIndex < 0)
{
nearestBackwardOrderHint = -1;
// A forward-only sequence pairs the nearest past frame with the closest distinct frame preceding it.
for (int referenceIndex = 0; referenceIndex < Av1Constants.ReferencesPerFrame; referenceIndex++)
{
uint referenceOrderHint = referenceOrderHints[(int)referenceFrameIndices[referenceIndex]];
bool precedesNearestForward = orderHintInfo.GetRelativeDistance(referenceOrderHint, (uint)nearestForwardOrderHint) < 0;
if (precedesNearestForward &&
(nearestBackwardOrderHint == -1 || orderHintInfo.GetRelativeDistance(referenceOrderHint, (uint)nearestBackwardOrderHint) > 0))
{
nearestBackwardOrderHint = (int)referenceOrderHint;
nearestBackwardReferenceIndex = referenceIndex;
}
}
}
if (nearestForwardReferenceIndex < 0 || nearestBackwardReferenceIndex < 0)
{
return;
}
int firstReferenceIndex = Math.Min(nearestForwardReferenceIndex, nearestBackwardReferenceIndex);
int secondReferenceIndex = Math.Max(nearestForwardReferenceIndex, nearestBackwardReferenceIndex);
this.FirstReferenceFrame = (Av1ReferenceFrameType)(firstReferenceIndex + (int)Av1ReferenceFrameType.Last);
this.SecondReferenceFrame = (Av1ReferenceFrameType)(secondReferenceIndex + (int)Av1ReferenceFrameType.Last);
this.SkipModeAllowed = true;
}
}

5
src/ImageSharp/Formats/Heif/Av1/Tiling/Av1BlockModeInfo.cs

@ -222,6 +222,11 @@ internal class Av1BlockModeInfo
/// </summary>
public int SegmentId { get; set; }
/// <summary>
/// Gets or sets a value indicating whether temporal prediction supplied the segment identifier.
/// </summary>
public bool SegmentIdPredicted { get; set; }
/// <summary>
/// Gets or sets the chroma intra-prediction mode.
/// </summary>

38
src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.MotionField.cs

@ -137,8 +137,8 @@ internal partial class Av1FrameInfo
}
InlineArray8<Av1ReferenceFrame?> selectedReferences = default;
Span<uint> referenceFrameIndices = frameHeader.GetReferenceFrameIndices();
int orderHintBits = sequenceHeader.OrderHintInfo.OrderHintBits;
ReadOnlySpan<uint> referenceFrameIndices = frameHeader.GetReferenceFrameIndices();
ObuOrderHintInfo orderHintInfo = sequenceHeader.OrderHintInfo;
// Capture the seven logical-role order hints before this frame refreshes any physical map slots. Libaom keeps
// the same snapshot on RefCntBuffer so a later frame can project this frame's stored motion vectors.
@ -150,7 +150,7 @@ internal partial class Av1FrameInfo
selectedReferences[(int)referenceFrameType] = referenceFrame;
this.motionFieldReferenceOrderHints[(int)referenceFrameType] = referenceOrderHint;
int relativeDistance = GetRelativeDistance(referenceOrderHint, frameHeader.OrderHint, orderHintBits);
int relativeDistance = orderHintInfo.GetRelativeDistance(referenceOrderHint, frameHeader.OrderHint);
this.motionFieldReferenceSides[(int)referenceFrameType] = relativeDistance > 0
? (sbyte)1
: referenceOrderHint == frameHeader.OrderHint ? (sbyte)-1 : (sbyte)0;
@ -194,7 +194,7 @@ internal partial class Av1FrameInfo
remainingProjectionCount--;
Av1ReferenceFrame backwardFrame = selectedReferences[(int)Av1ReferenceFrameType.Backward]!;
if (GetRelativeDistance(backwardFrame.FrameHeader.OrderHint, frameHeader.OrderHint, orderHintBits) > 0 &&
if (orderHintInfo.GetRelativeDistance(backwardFrame.FrameHeader.OrderHint, frameHeader.OrderHint) > 0 &&
this.ProjectMotionField(sequenceHeader, frameHeader, backwardFrame, reverseDirection: false))
{
remainingProjectionCount--;
@ -202,7 +202,7 @@ internal partial class Av1FrameInfo
Av1ReferenceFrame alternate2Frame = selectedReferences[(int)Av1ReferenceFrameType.Alternate2]!;
if (GetRelativeDistance(alternate2Frame.FrameHeader.OrderHint, frameHeader.OrderHint, orderHintBits) > 0 &&
if (orderHintInfo.GetRelativeDistance(alternate2Frame.FrameHeader.OrderHint, frameHeader.OrderHint) > 0 &&
this.ProjectMotionField(sequenceHeader, frameHeader, alternate2Frame, reverseDirection: false))
{
remainingProjectionCount--;
@ -211,7 +211,7 @@ internal partial class Av1FrameInfo
Av1ReferenceFrame alternateFrame = selectedReferences[(int)Av1ReferenceFrameType.Alternate]!;
if (remainingProjectionCount > 0 &&
GetRelativeDistance(alternateFrame.FrameHeader.OrderHint, frameHeader.OrderHint, orderHintBits) > 0 &&
orderHintInfo.GetRelativeDistance(alternateFrame.FrameHeader.OrderHint, frameHeader.OrderHint) > 0 &&
this.ProjectMotionField(sequenceHeader, frameHeader, alternateFrame, reverseDirection: false))
{
remainingProjectionCount--;
@ -331,11 +331,10 @@ internal partial class Av1FrameInfo
}
Av1FrameInfo startFrameInfo = startFrame.FrameInfo;
int orderHintBits = sequenceHeader.OrderHintInfo.OrderHintBits;
int startToCurrentFrameOffset = GetRelativeDistance(
ObuOrderHintInfo orderHintInfo = sequenceHeader.OrderHintInfo;
int startToCurrentFrameOffset = orderHintInfo.GetRelativeDistance(
startFrameHeader.OrderHint,
frameHeader.OrderHint,
orderHintBits);
frameHeader.OrderHint);
if (reverseDirection)
{
@ -358,10 +357,9 @@ internal partial class Av1FrameInfo
continue;
}
int referenceFrameOffset = GetRelativeDistance(
int referenceFrameOffset = orderHintInfo.GetRelativeDistance(
startFrameHeader.OrderHint,
startFrameInfo.motionFieldReferenceOrderHints[(int)source.ReferenceFrame],
orderHintBits);
startFrameInfo.motionFieldReferenceOrderHints[(int)source.ReferenceFrame]);
bool positionIsValid = Math.Abs(referenceFrameOffset) <= MaximumFrameDistance &&
referenceFrameOffset > 0 &&
@ -466,20 +464,6 @@ internal partial class Av1FrameInfo
projectedColumn < baseBlockColumn + 8 + MaximumHorizontalFieldOffset;
}
/// <summary>
/// Computes the signed distance between two order hints in their modulo domain.
/// </summary>
/// <param name="first">The first order hint.</param>
/// <param name="second">The order hint subtracted from <paramref name="first"/>.</param>
/// <param name="orderHintBits">The number of bits in the order-hint domain.</param>
/// <returns>The shortest signed modulo distance.</returns>
private static int GetRelativeDistance(uint first, uint second, int orderHintBits)
{
int difference = (int)first - (int)second;
int signBit = 1 << (orderHintBits - 1);
return (difference & (signBit - 1)) - (difference & signBit);
}
/// <summary>
/// Stores one motion vector and logical reference retained for projection by a later frame.
/// </summary>

19
src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.cs

@ -246,16 +246,23 @@ internal partial class Av1FrameInfo
/// <summary>
/// Gets the minimum retained segment identifier across a block's clipped mode-information coverage.
/// </summary>
/// <param name="primaryReferenceFrameInfo">
/// The retained primary-frame state, or <see langword="null"/> when no compatible map is available.
/// </param>
/// <param name="blockSize">The block size whose 4x4 coverage is inspected.</param>
/// <param name="modeInfoPosition">The block origin in frame-relative 4x4 units.</param>
/// <returns>
/// The minimum retained segment identifier, or zero when the retained frame has no enabled segmentation map.
/// The minimum retained segment identifier, or zero when no same-sized retained segmentation map is available.
/// </returns>
public int GetPredictedSegmentId(Av1BlockSize blockSize, Point modeInfoPosition)
public int GetPredictedSegmentId(Av1FrameInfo? primaryReferenceFrameInfo, Av1BlockSize blockSize, Point modeInfoPosition)
{
if (this.segmentIds.Length == 0)
if (primaryReferenceFrameInfo is null ||
primaryReferenceFrameInfo.segmentIds.Length == 0 ||
primaryReferenceFrameInfo.segmentIdColumnCount != this.segmentIdColumnCount ||
primaryReferenceFrameInfo.segmentIdRowCount != this.segmentIdRowCount)
{
// libaom represents an unavailable prior map with a null pointer and predicts segment zero.
// libaom exposes the prior map only when both mode-info dimensions match the active frame. Treating a
// differently sized retained map as absent prevents coordinates from being reinterpreted with a new stride.
return 0;
}
@ -267,8 +274,8 @@ internal partial class Av1FrameInfo
// dec_get_segment_id rule used when segmentation_temporal_update selects the retained primary map.
for (int row = 0; row < rowCount; row++)
{
int offset = ((modeInfoPosition.Y + row) * this.segmentIdColumnCount) + modeInfoPosition.X;
ReadOnlySpan<byte> segmentRow = this.segmentIds.AsSpan(offset, columnCount);
int offset = ((modeInfoPosition.Y + row) * primaryReferenceFrameInfo.segmentIdColumnCount) + modeInfoPosition.X;
ReadOnlySpan<byte> segmentRow = primaryReferenceFrameInfo.segmentIds.AsSpan(offset, columnCount);
for (int column = 0; column < segmentRow.Length; column++)
{

268
src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileReader.cs

@ -1353,7 +1353,8 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable
int block4x4Width = blockSize.Get4x4WideCount();
int block4x4Height = blockSize.Get4x4HighCount();
// HEIF still-image decoding follows the independently decodable intra-frame transform-size branch.
// Both intra frames and intra-coded blocks inside inter frames use the intra transform-size branch. The true
// inter branch will replace this fixed false classification when inter reconstruction is connected.
Av1TransformSize transformSize = this.ReadTransformSize(ref reader, ref partitionInfo, superblockInfo, tileInfo, true);
this.aboveNeighborContext.UpdateTransformation(modeInfoLocation, tileInfo, transformSize, blockSize, false);
this.leftNeighborContext.UpdateTransformation(modeInfoLocation, superblockInfo, transformSize, blockSize, false);
@ -1537,16 +1538,62 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable
}
/// <summary>
/// Reads the prediction, segmentation, skip, quantizer, and filter mode information for a still-image block.
/// Reads the prediction, segmentation, skip, quantizer, and filter mode information for a coding block.
/// </summary>
/// <param name="reader">The tile symbol decoder.</param>
/// <param name="partitionInfo">The current coding block.</param>
/// <param name="tileInfo">The active tile boundaries.</param>
/// <remarks>Implements the intra-frame branch of AV1 section 5.11.6.</remarks>
/// <remarks>Implements the frame-type dispatch in AV1 section 5.11.6.</remarks>
private void ReadModeInfo(ref Av1SymbolDecoder reader, ref Av1PartitionInfo partitionInfo, Av1TileInfo tileInfo)
{
DebugGuard.IsTrue(this.FrameHeader.FrameType is ObuFrameType.KeyFrame or ObuFrameType.IntraOnlyFrame, "Only INTRA frames supported.");
this.ReadIntraFrameModeInfo(ref reader, ref partitionInfo, tileInfo);
if (this.FrameHeader.IsIntra)
{
this.ReadIntraFrameModeInfo(ref reader, ref partitionInfo, tileInfo);
}
else
{
this.ReadInterFrameModeInfo(ref reader, ref partitionInfo);
}
}
/// <summary>
/// Reads the common inter-frame block prefix and the intra-coded-block prediction branch in bitstream order.
/// </summary>
/// <param name="reader">The tile symbol decoder.</param>
/// <param name="partitionInfo">The current coding block and its neighbors.</param>
/// <remarks>Implements the prefix and intra branch of AV1 section 5.11.7.</remarks>
internal void ReadInterFrameModeInfo(ref Av1SymbolDecoder reader, ref Av1PartitionInfo partitionInfo)
{
Av1BlockModeInfo modeInfo = partitionInfo.ModeInfo;
modeInfo.MotionVectors.Clear();
this.ReadInterSegmentId(ref reader, ref partitionInfo, beforeSkip: true);
modeInfo.SkipMode = this.ReadSkipMode(ref reader, ref partitionInfo);
modeInfo.Skip = modeInfo.SkipMode || this.ReadSkip(ref reader, ref partitionInfo);
if (!this.FrameHeader.SegmentationParameters.SegmentIdPrecedesSkip)
{
this.ReadInterSegmentId(ref reader, ref partitionInfo, beforeSkip: false);
}
this.ReadCdef(ref reader, ref partitionInfo);
if (this.FrameHeader.DeltaQParameters.IsPresent)
{
this.ReadDeltaQuantizerIndex(ref reader, ref partitionInfo);
this.ReadDeltaLoopFilter(ref reader, ref partitionInfo);
}
bool isInterBlock = modeInfo.SkipMode || this.ReadIsInter(ref reader, ref partitionInfo);
if (isInterBlock)
{
throw new NotSupportedException("AV1 inter-coded block prediction is not implemented.");
}
modeInfo.ReferenceFrames[0] = Av1ReferenceFrameType.Intra;
modeInfo.ReferenceFrames[1] = Av1ReferenceFrameType.None;
modeInfo.SetPaletteSizes(0, 0);
this.ReadConventionalIntraMode(ref reader, ref partitionInfo, reader.ReadInterFrameYMode(modeInfo.BlockSize));
}
/// <summary>
@ -1610,48 +1657,61 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable
}
else
{
partitionInfo.ModeInfo.YMode = reader.ReadYMode(partitionInfo.AboveModeInfo, partitionInfo.LeftModeInfo);
partitionInfo.ModeInfo.SetAngleDelta(
Av1PlaneType.Y,
IntraAngleInfo(ref reader, partitionInfo.ModeInfo.YMode, partitionInfo.ModeInfo.BlockSize));
Av1PredictionMode yMode = reader.ReadYMode(partitionInfo.AboveModeInfo, partitionInfo.LeftModeInfo);
this.ReadConventionalIntraMode(ref reader, ref partitionInfo, yMode);
}
}
if (partitionInfo.IsChroma && !this.SequenceHeader.ColorConfig.IsMonochrome)
{
partitionInfo.ModeInfo.UvMode = reader.ReadIntraModeUv(
partitionInfo.ModeInfo.YMode,
this.IsChromaForLumaAllowed(ref partitionInfo));
/// <summary>
/// Reads conventional luma and chroma intra-prediction details after the frame branch selects the luma mode CDF.
/// </summary>
/// <param name="reader">The tile symbol decoder.</param>
/// <param name="partitionInfo">The current coding block.</param>
/// <param name="yMode">The luma prediction mode selected by the frame-appropriate distribution.</param>
private void ReadConventionalIntraMode(
ref Av1SymbolDecoder reader,
ref Av1PartitionInfo partitionInfo,
Av1PredictionMode yMode)
{
Av1BlockModeInfo modeInfo = partitionInfo.ModeInfo;
modeInfo.YMode = yMode;
modeInfo.SetAngleDelta(Av1PlaneType.Y, IntraAngleInfo(ref reader, yMode, modeInfo.BlockSize));
if (partitionInfo.ModeInfo.UvMode == Av1ChromaPredictionMode.ChromaFromLuma)
{
ReadChromaFromLumaAlphas(ref reader, partitionInfo.ModeInfo);
}
if (partitionInfo.IsChroma && !this.SequenceHeader.ColorConfig.IsMonochrome)
{
modeInfo.UvMode = reader.ReadIntraModeUv(yMode, this.IsChromaForLumaAllowed(ref partitionInfo));
partitionInfo.ModeInfo.SetAngleDelta(
Av1PlaneType.Uv,
IntraAngleInfo(ref reader, partitionInfo.ModeInfo.UvMode.ToLumaMode(), partitionInfo.ModeInfo.BlockSize));
}
else
if (modeInfo.UvMode == Av1ChromaPredictionMode.ChromaFromLuma)
{
partitionInfo.ModeInfo.UvMode = Av1ChromaPredictionMode.DC;
ReadChromaFromLumaAlphas(ref reader, modeInfo);
}
if (partitionInfo.ModeInfo.BlockSize >= Av1BlockSize.Block8x8 &&
partitionInfo.ModeInfo.BlockSize.GetWidth() <= 64 &&
partitionInfo.ModeInfo.BlockSize.GetHeight() <= 64 &&
this.FrameHeader.AllowScreenContentTools)
{
this.PaletteModeInfo(ref reader, ref partitionInfo);
}
modeInfo.SetAngleDelta(
Av1PlaneType.Uv,
IntraAngleInfo(ref reader, modeInfo.UvMode.ToLumaMode(), modeInfo.BlockSize));
}
else
{
modeInfo.UvMode = Av1ChromaPredictionMode.DC;
}
this.FilterIntraModeInfo(ref reader, ref partitionInfo);
if (modeInfo.BlockSize >= Av1BlockSize.Block8x8 &&
modeInfo.BlockSize.GetWidth() <= 64 &&
modeInfo.BlockSize.GetHeight() <= 64 &&
this.FrameHeader.AllowScreenContentTools)
{
this.PaletteModeInfo(ref reader, ref partitionInfo);
}
this.FilterIntraModeInfo(ref reader, ref partitionInfo);
}
/// <summary>
/// Determines whether the frame header permits intra block copy for an intra still image.
/// Determines whether the frame header permits intra block copy for an intra frame.
/// </summary>
/// <returns><see langword="true"/> when the frame and sequence enable intra block copy; otherwise, <see langword="false"/>.</returns>
/// <returns>
/// <see langword="true"/> when the frame and sequence enable intra block copy; otherwise, <see langword="false"/>.
/// </returns>
private bool AllowIntraBlockCopy()
=> (this.FrameHeader.FrameType is ObuFrameType.KeyFrame or ObuFrameType.IntraOnlyFrame) &&
(this.SequenceHeader.ForceScreenContentTools > 0) &&
@ -1661,7 +1721,10 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable
/// Determines whether chroma-from-luma prediction is available for a coding block.
/// </summary>
/// <param name="partitionInfo">The current coding block.</param>
/// <returns><see langword="true"/> when the lossless transform or block dimensions permit chroma-from-luma prediction; otherwise, <see langword="false"/>.</returns>
/// <returns>
/// <see langword="true"/> when the lossless transform or block dimensions permit chroma-from-luma prediction;
/// otherwise, <see langword="false"/>.
/// </returns>
private bool IsChromaForLumaAllowed(ref Av1PartitionInfo partitionInfo)
{
if (this.FrameHeader.LosslessArray[partitionInfo.ModeInfo.SegmentId])
@ -2239,7 +2302,82 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable
}
/// <summary>
/// Predicts and, when required, decodes the segment identifier for an intra block.
/// Reads or inherits the segment identifier for one inter-frame block and updates its 4x4 map coverage.
/// </summary>
/// <param name="reader">The tile symbol decoder.</param>
/// <param name="partitionInfo">The current coding block and its available neighbors.</param>
/// <param name="beforeSkip">Whether this invocation precedes the block's residual-skip decision.</param>
/// <remarks>
/// Implements <c>read_inter_segment_id</c> from AV1 section 5.11.8.
/// </remarks>
internal void ReadInterSegmentId(ref Av1SymbolDecoder reader, ref Av1PartitionInfo partitionInfo, bool beforeSkip)
{
ObuSegmentationParameters segmentationParameters = this.FrameHeader.SegmentationParameters;
Av1BlockModeInfo modeInfo = partitionInfo.ModeInfo;
if (!segmentationParameters.Enabled)
{
// Disabled segmentation has no allocated map and normatively assigns segment zero.
modeInfo.SegmentId = 0;
return;
}
Point modeInfoPosition = new(partitionInfo.ColumnIndex, partitionInfo.RowIndex);
if (segmentationParameters.SegmentationUpdateMap == 0)
{
// The frame map was inherited as one contiguous copy during reader construction. Resolve the same clipped
// minimum that libaom obtains from last_frame_seg_map so block state and the already copied map agree.
modeInfo.SegmentId = this.FrameInfo.GetPredictedSegmentId(this.primaryReferenceFrameInfo, modeInfo.BlockSize, modeInfoPosition);
return;
}
if (beforeSkip)
{
if (!segmentationParameters.SegmentIdPrecedesSkip)
{
// The caller invokes this once before skip for every inter block; post-skip segment syntax owns this case.
return;
}
}
else if (modeInfo.Skip)
{
if (segmentationParameters.SegmentationTemporalUpdate == 1)
{
// Skipped blocks use the spatial segment predictor and signal no temporal-prediction bit.
modeInfo.SegmentIdPredicted = false;
}
this.ReadSegmentId(ref reader, ref partitionInfo);
this.FrameInfo.SetSegmentId(modeInfo.BlockSize, modeInfoPosition, modeInfo.SegmentId);
return;
}
if (segmentationParameters.SegmentationTemporalUpdate == 1)
{
// The binary context counts only neighboring blocks that themselves selected the retained map. Segment
// values do not participate in this decision.
int context = Av1SymbolContextHelper.GetSegmentIdPredictedContext(partitionInfo.AboveModeInfo, partitionInfo.LeftModeInfo);
modeInfo.SegmentIdPredicted = reader.ReadSegmentIdPredicted(context);
if (modeInfo.SegmentIdPredicted)
{
modeInfo.SegmentId = this.FrameInfo.GetPredictedSegmentId(this.primaryReferenceFrameInfo, modeInfo.BlockSize, modeInfoPosition);
}
else
{
this.ReadSegmentId(ref reader, ref partitionInfo);
}
}
else
{
this.ReadSegmentId(ref reader, ref partitionInfo);
}
this.FrameInfo.SetSegmentId(modeInfo.BlockSize, modeInfoPosition, modeInfo.SegmentId);
}
/// <summary>
/// Predicts and, when required, decodes the spatially coded segment identifier for a block.
/// </summary>
/// <param name="reader">The tile symbol decoder.</param>
/// <param name="partitionInfo">The current coding block and its available neighbors.</param>
@ -2400,6 +2538,62 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable
}
}
/// <summary>
/// Reads compound skip-mode selection when the frame, segment, and block geometry permit it.
/// </summary>
/// <param name="reader">The tile symbol decoder.</param>
/// <param name="partitionInfo">The current coding block and its available neighbors.</param>
/// <returns><see langword="true"/> when the block selects the frame's derived skip-mode reference pair.</returns>
private bool ReadSkipMode(ref Av1SymbolDecoder reader, ref Av1PartitionInfo partitionInfo)
{
Av1BlockModeInfo modeInfo = partitionInfo.ModeInfo;
ObuSegmentationParameters segmentationParameters = this.FrameHeader.SegmentationParameters;
int segmentId = modeInfo.SegmentId;
if (!this.FrameHeader.SkipModeParameters.SkipModeFlag ||
segmentationParameters.IsFeatureActive(segmentId, ObuSegmentationLevelFeature.Skip) ||
Math.Min(modeInfo.BlockSize.GetWidth(), modeInfo.BlockSize.GetHeight()) < 8 ||
segmentationParameters.IsFeatureActive(segmentId, ObuSegmentationLevelFeature.ReferenceFrame) ||
segmentationParameters.IsFeatureActive(segmentId, ObuSegmentationLevelFeature.GlobalMotionVector))
{
// Segment reference and global-motion features force single-reference prediction, while skip mode always
// selects the derived compound pair. The syntax therefore omits the skip-mode symbol in either case.
return false;
}
int aboveSkipMode = partitionInfo.AboveModeInfo is not null && partitionInfo.AboveModeInfo.SkipMode ? 1 : 0;
int leftSkipMode = partitionInfo.LeftModeInfo is not null && partitionInfo.LeftModeInfo.SkipMode ? 1 : 0;
return reader.ReadSkipMode(aboveSkipMode + leftSkipMode);
}
/// <summary>
/// Reads or infers whether an inter-frame coding block uses inter prediction.
/// </summary>
/// <param name="reader">The tile symbol decoder.</param>
/// <param name="partitionInfo">The current coding block and its available neighbors.</param>
/// <returns><see langword="true"/> for an inter-coded block; otherwise, <see langword="false"/>.</returns>
private bool ReadIsInter(ref Av1SymbolDecoder reader, ref Av1PartitionInfo partitionInfo)
{
ObuSegmentationParameters segmentationParameters = this.FrameHeader.SegmentationParameters;
int segmentId = partitionInfo.ModeInfo.SegmentId;
if (segmentationParameters.IsFeatureActive(segmentId, ObuSegmentationLevelFeature.ReferenceFrame))
{
// Reference feature values use the same numeric labels as Av1ReferenceFrameType. INTRA_FRAME is zero;
// every canonical inter reference begins at LAST_FRAME and therefore has a positive value.
int referenceFrame = segmentationParameters.FeatureData[segmentId, (int)ObuSegmentationLevelFeature.ReferenceFrame];
return referenceFrame >= (int)Av1ReferenceFrameType.Last;
}
if (segmentationParameters.IsFeatureActive(segmentId, ObuSegmentationLevelFeature.GlobalMotionVector))
{
return true;
}
int context = Av1SymbolContextHelper.GetIntraInterContext(partitionInfo.AboveModeInfo, partitionInfo.LeftModeInfo);
return reader.ReadIsInter(context);
}
/// <summary>
/// Reads and accumulates a superblock quantizer-index delta when the block carries one.
/// </summary>

257
tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterFrameIntraEntropyTests.cs

@ -0,0 +1,257 @@
// Copyright (c) Six Labors.
// Licensed under the Six Labors Split License.
using System.Buffers;
using SixLabors.ImageSharp.Formats.Heif.Av1;
using SixLabors.ImageSharp.Formats.Heif.Av1.Entropy;
using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction;
using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling;
using SixLabors.ImageSharp.Memory;
namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1;
/// <summary>
/// Verifies the entropy state and spatial contexts used by intra-coded blocks inside AV1 inter frames.
/// </summary>
[Trait("Format", "Avif")]
public class Av1InterFrameIntraEntropyTests
{
/// <summary>
/// Gets libaom's four forward Q15 luma-mode CDF rows in block-size-group order.
/// </summary>
private static ReadOnlySpan<ushort> FrameYModeForwardThresholds =>
[
22801, 23489, 24293, 24756, 25601, 26123, 26606, 27418, 27945, 29228, 29685, 30349,
18673, 19845, 22631, 23318, 23950, 24649, 25527, 27364, 28152, 29701, 29984, 30852,
19770, 20979, 23396, 23939, 24241, 24654, 25136, 27073, 27830, 29360, 29730, 30659,
20155, 21301, 22838, 23178, 23261, 23533, 23703, 24804, 25352, 26575, 27016, 28049,
];
/// <summary>
/// Verifies the four normative intra/inter distributions against libaom's forward Q15 defaults.
/// </summary>
[Fact]
public void IntraInterDefaultsMatchLibaom()
{
uint[] forwardThresholds = [806, 16662, 20186, 26538];
Av1Distribution[] distributions = Av1DefaultDistributions.IntraInter;
Assert.Equal(forwardThresholds.Length, distributions.Length);
for (int context = 0; context < distributions.Length; context++)
{
// Av1Distribution stores inverse cumulative thresholds, so compare each libaom default after the same
// forward-to-inverse conversion performed by its constructor.
Assert.Equal((uint)Av1Distribution.ProbabilityTop - forwardThresholds[context], distributions[context][0]);
Assert.Equal(2, distributions[context].NumberOfSymbols);
}
}
/// <summary>
/// Verifies every inter-frame intra luma-mode threshold against libaom's forward Q15 defaults.
/// </summary>
[Fact]
public void FrameYModeDefaultsMatchLibaom()
{
const int thresholdsPerGroup = 12;
ReadOnlySpan<ushort> forwardThresholds = FrameYModeForwardThresholds;
Av1Distribution[] distributions = Av1DefaultDistributions.FrameYMode;
Assert.Equal(4, distributions.Length);
for (int group = 0; group < distributions.Length; group++)
{
Assert.Equal(thresholdsPerGroup + 1, distributions[group].NumberOfSymbols);
for (int threshold = 0; threshold < thresholdsPerGroup; threshold++)
{
uint expected = (uint)Av1Distribution.ProbabilityTop - forwardThresholds[(group * thresholdsPerGroup) + threshold];
Assert.Equal(expected, distributions[group][threshold]);
}
}
}
/// <summary>
/// Verifies that the intra/inter reader selects and adapts each of the four spatial-context distributions.
/// </summary>
/// <param name="context">The intra/inter spatial context.</param>
[Theory]
[InlineData(0)]
[InlineData(1)]
[InlineData(2)]
[InlineData(3)]
public void ReadIsInterUsesRequestedContext(int context)
{
bool[] expected = [false, true, true, false, true, false, false, true];
Av1Distribution writerDistribution = Av1DefaultDistributions.IntraInter[context];
using Av1SymbolWriter writer = new(Configuration.Default, 8, updateCdf: true);
foreach (bool value in expected)
{
writer.WriteSymbol(value, writerDistribution);
}
using IMemoryOwner<byte> encoded = writer.Exit();
Av1SymbolDecoder decoder = new(Configuration.Default, encoded.GetSpan(), 0, updateCdf: true);
foreach (bool value in expected)
{
Assert.Equal(value, decoder.ReadIsInter(context));
}
}
/// <summary>
/// Verifies that inter-frame intra luma modes use the normative size group for every AV1 block size.
/// </summary>
/// <param name="blockSizeValue">The AV1 block-size enumeration value.</param>
/// <param name="sizeGroup">The normative size group from AV1 section 9.3.</param>
[Theory]
[MemberData(nameof(GetBlockSizeGroups))]
public void ReadInterFrameYModeUsesNormativeSizeGroup(int blockSizeValue, int sizeGroup)
{
Av1BlockSize blockSize = (Av1BlockSize)blockSizeValue;
Av1PredictionMode[] expected =
[
Av1PredictionMode.DC,
Av1PredictionMode.Directional45Degrees,
Av1PredictionMode.Smooth,
Av1PredictionMode.Paeth,
Av1PredictionMode.Horizontal,
Av1PredictionMode.Directional157Degrees,
];
Av1Distribution writerDistribution = Av1DefaultDistributions.FrameYMode[sizeGroup];
using Av1SymbolWriter writer = new(Configuration.Default, 8, updateCdf: true);
foreach (Av1PredictionMode mode in expected)
{
writer.WriteSymbol((int)mode, writerDistribution);
}
using IMemoryOwner<byte> encoded = writer.Exit();
Av1SymbolDecoder decoder = new(Configuration.Default, encoded.GetSpan(), 0, updateCdf: true);
foreach (Av1PredictionMode mode in expected)
{
Assert.Equal(mode, decoder.ReadInterFrameYMode(blockSize));
}
}
/// <summary>
/// Verifies that all four intra/inter contexts follow the normative above-and-left neighbor classification.
/// </summary>
/// <param name="hasAbove">Whether the above block is available.</param>
/// <param name="aboveIsInter">Whether the available above block uses inter prediction.</param>
/// <param name="hasLeft">Whether the left block is available.</param>
/// <param name="leftIsInter">Whether the available left block uses inter prediction.</param>
/// <param name="expected">The expected intra/inter context.</param>
[Theory]
[InlineData(false, false, false, false, 0)]
[InlineData(true, true, false, false, 0)]
[InlineData(true, false, false, false, 2)]
[InlineData(false, false, true, true, 0)]
[InlineData(false, false, true, false, 2)]
[InlineData(true, true, true, true, 0)]
[InlineData(true, false, true, true, 1)]
[InlineData(true, true, true, false, 1)]
[InlineData(true, false, true, false, 3)]
public void IntraInterContextMatchesNeighborPredictionTypes(
bool hasAbove,
bool aboveIsInter,
bool hasLeft,
bool leftIsInter,
int expected)
{
Av1BlockModeInfo above = hasAbove ? CreateModeInfo(aboveIsInter) : null;
Av1BlockModeInfo left = hasLeft ? CreateModeInfo(leftIsInter) : null;
int actual = Av1SymbolContextHelper.GetIntraInterContext(above, left);
Assert.Equal(expected, actual);
}
/// <summary>
/// Verifies that frame-context copies retain adapted inter-frame intra state without sharing mutable distributions.
/// </summary>
[Fact]
public void FrameEntropyCopyRetainsIndependentInterFrameIntraState()
{
Av1FrameEntropyContext source = new(0);
Av1FrameEntropyContext destination = new(0);
source.FrameYMode[2].Update((int)Av1PredictionMode.Smooth);
source.IntraInter[3].Update(1);
destination.CopyFrom(source);
Assert.Equal(source.FrameYMode[2][0], destination.FrameYMode[2][0]);
Assert.Equal(source.IntraInter[3][0], destination.IntraInter[3][0]);
source.FrameYMode[2].Update((int)Av1PredictionMode.Paeth);
source.IntraInter[3].Update(0);
Assert.NotEqual(source.FrameYMode[2][0], destination.FrameYMode[2][0]);
Assert.NotEqual(source.IntraInter[3][0], destination.IntraInter[3][0]);
}
/// <summary>
/// Verifies that a published frame snapshot preserves adapted thresholds but resets their update-rate history.
/// </summary>
[Fact]
public void FrameEntropySnapshotResetsInterFrameIntraUpdateCounts()
{
const int updateCount = 20;
Av1FrameEntropyContext source = new(0);
Av1FrameEntropyContext snapshot = new(0);
for (int i = 0; i < updateCount; i++)
{
source.FrameYMode[1].Update((int)Av1PredictionMode.Vertical);
source.IntraInter[1].Update(1);
}
source.SnapshotTo(snapshot);
Assert.Equal(source.FrameYMode[1][0], snapshot.FrameYMode[1][0]);
Assert.Equal(source.IntraInter[1][0], snapshot.IntraInter[1][0]);
// The source retains twenty observations while the published snapshot restarts at zero. Applying the same
// symbol therefore moves identical thresholds by different update rates only when reset wiring is complete.
source.FrameYMode[1].Update((int)Av1PredictionMode.DC);
snapshot.FrameYMode[1].Update((int)Av1PredictionMode.DC);
source.IntraInter[1].Update(0);
snapshot.IntraInter[1].Update(0);
Assert.NotEqual(source.FrameYMode[1][0], snapshot.FrameYMode[1][0]);
Assert.NotEqual(source.IntraInter[1][0], snapshot.IntraInter[1][0]);
}
/// <summary>
/// Provides the normative AV1 size-group table in block-size enumeration order.
/// </summary>
/// <returns>Every decoded block size paired with its luma-mode size group.</returns>
public static TheoryData<int, int> GetBlockSizeGroups()
{
// This is size_group_lookup from AV1 section 9.3 and libaom common_data.h. Keeping expected values explicit
// ensures that the test does not reproduce the production formula it is intended to verify.
int[] sizeGroups = [0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 0, 0, 1, 1, 2, 2];
TheoryData<int, int> result = [];
for (int blockSize = 0; blockSize < sizeGroups.Length; blockSize++)
{
result.Add(blockSize, sizeGroups[blockSize]);
}
return result;
}
/// <summary>
/// Creates decoded neighbor state with either an intra or inter primary reference.
/// </summary>
/// <param name="isInter">Whether the neighbor uses inter prediction.</param>
/// <returns>The initialized block mode state.</returns>
private static Av1BlockModeInfo CreateModeInfo(bool isInter)
{
Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block4x4, Point.Empty);
modeInfo.ReferenceFrames[0] = isInter ? Av1ReferenceFrameType.Last : Av1ReferenceFrameType.Intra;
modeInfo.ReferenceFrames[1] = Av1ReferenceFrameType.None;
return modeInfo;
}
}

133
tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterFrameModeInfoTests.cs

@ -0,0 +1,133 @@
// Copyright (c) Six Labors.
// Licensed under the Six Labors Split License.
using System.Buffers;
using SixLabors.ImageSharp.Formats.Heif.Av1;
using SixLabors.ImageSharp.Formats.Heif.Av1.Entropy;
using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit;
using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction;
using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling;
using SixLabors.ImageSharp.Memory;
namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1;
/// <summary>
/// Verifies the common inter-frame mode prefix and its intra-coded-block branch.
/// </summary>
[Trait("Format", "Avif")]
public class Av1InterFrameModeInfoTests
{
/// <summary>
/// Verifies that an inter frame can select an intra-coded block using the block-size luma distribution.
/// </summary>
[Fact]
public void ReadInterFrameModeInfoReadsIntraCodedBlock()
{
ObuSequenceHeader sequenceHeader = CreateSequenceHeader();
ObuFrameHeader frameHeader = CreateFrameHeader();
using Av1TileReader tileReader = new(Configuration.Default, sequenceHeader, frameHeader);
Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block8x8, Point.Empty);
Av1SuperblockInfo superblockInfo = new(tileReader.FrameInfo, Point.Empty);
Av1PartitionInfo partitionInfo = new(modeInfo, superblockInfo, false, Av1PartitionType.None);
Av1Distribution skip = Av1DefaultDistributions.Skip[0];
Av1Distribution intraInter = Av1DefaultDistributions.IntraInter[0];
Av1Distribution yMode = Av1DefaultDistributions.FrameYMode[1];
using Av1SymbolWriter writer = new(Configuration.Default, 1, updateCdf: true);
writer.WriteSymbol(false, skip);
writer.WriteSymbol(false, intraInter);
writer.WriteSymbol((int)Av1PredictionMode.DC, yMode);
using IMemoryOwner<byte> encoded = writer.Exit();
Av1SymbolDecoder decoder = new(Configuration.Default, encoded.GetSpan(), 0, updateCdf: true);
tileReader.ReadInterFrameModeInfo(ref decoder, ref partitionInfo);
Assert.False(modeInfo.SkipMode);
Assert.False(modeInfo.Skip);
Assert.Equal(Av1ReferenceFrameType.Intra, modeInfo.ReferenceFrames[0]);
Assert.Equal(Av1ReferenceFrameType.None, modeInfo.ReferenceFrames[1]);
Assert.Equal(Av1PredictionMode.DC, modeInfo.YMode);
Assert.Equal(Av1ChromaPredictionMode.DC, modeInfo.UvMode);
}
/// <summary>
/// Verifies that skip mode omits the residual-skip and intra-inter symbols and marks the block as inter coded.
/// </summary>
[Fact]
public void ReadInterFrameModeInfoSkipModeForcesInterBlockAndResidualSkip()
{
ObuSequenceHeader sequenceHeader = CreateSequenceHeader();
ObuFrameHeader frameHeader = CreateFrameHeader();
frameHeader.SkipModeParameters.SkipModeFlag = true;
using Av1TileReader tileReader = new(Configuration.Default, sequenceHeader, frameHeader);
Av1BlockModeInfo aboveModeInfo = new(Av1BlockSize.Block8x8, Point.Empty) { SkipMode = true };
Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block8x8, Point.Empty);
Av1Distribution skipMode = Av1DefaultDistributions.SkipMode[1];
using Av1SymbolWriter writer = new(Configuration.Default, 1, updateCdf: true);
writer.WriteSymbol(true, skipMode);
using IMemoryOwner<byte> encoded = writer.Exit();
Memory<byte> encodedMemory = encoded.Memory;
Assert.Throws<NotSupportedException>(() => ReadInterFrameModeInfo(tileReader, encodedMemory, modeInfo, aboveModeInfo));
Assert.True(modeInfo.SkipMode);
Assert.True(modeInfo.Skip);
}
/// <summary>
/// Invokes the ref-struct mode parser for exception assertions that cannot capture its parameters directly.
/// </summary>
/// <param name="tileReader">The tile reader.</param>
/// <param name="encoded">The range-coded block-prefix symbols.</param>
/// <param name="modeInfo">The current coding block.</param>
/// <param name="aboveModeInfo">The available above block supplying skip-mode context.</param>
private static void ReadInterFrameModeInfo(
Av1TileReader tileReader,
Memory<byte> encoded,
Av1BlockModeInfo modeInfo,
Av1BlockModeInfo aboveModeInfo)
{
Av1SuperblockInfo superblockInfo = new(tileReader.FrameInfo, Point.Empty);
Av1PartitionInfo partitionInfo = new(modeInfo, superblockInfo, false, Av1PartitionType.None)
{
AvailableAbove = true,
AboveModeInfo = aboveModeInfo,
};
Av1SymbolDecoder decoder = new(Configuration.Default, encoded.Span, 0, updateCdf: true);
tileReader.ReadInterFrameModeInfo(ref decoder, ref partitionInfo);
}
/// <summary>
/// Creates the monochrome 64x64 sequence geometry used by direct mode-prefix tests.
/// </summary>
/// <returns>The initialized sequence header.</returns>
private static ObuSequenceHeader CreateSequenceHeader()
=> new()
{
MaxFrameWidth = 64,
MaxFrameHeight = 64,
Use128x128Superblock = false,
EnableCdef = false,
EnableFilterIntra = false,
ColorConfig = new ObuColorConfig
{
IsMonochrome = true,
BitDepth = Av1BitDepth.EightBit,
},
};
/// <summary>
/// Creates an inter-frame header whose optional block-prefix tools are disabled.
/// </summary>
/// <returns>The initialized frame header.</returns>
private static ObuFrameHeader CreateFrameHeader()
=> new()
{
FrameType = ObuFrameType.InterFrame,
ModeInfoColumnCount = 16,
ModeInfoRowCount = 16,
CodedLossless = true,
AllowScreenContentTools = false,
};
}

275
tests/ImageSharp.Tests/Formats/Heif/Av1/Av1TemporalSegmentationTests.cs

@ -0,0 +1,275 @@
// Copyright (c) Six Labors.
// Licensed under the Six Labors Split License.
using System.Buffers;
using SixLabors.ImageSharp.Formats.Heif.Av1;
using SixLabors.ImageSharp.Formats.Heif.Av1.Entropy;
using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit;
using SixLabors.ImageSharp.Formats.Heif.Av1.ReferenceFrames;
using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling;
using SixLabors.ImageSharp.Memory;
namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1;
/// <summary>
/// Verifies AV1 temporal segment-map prediction against libaom's decoder rules.
/// </summary>
[Trait("Format", "Avif")]
public class Av1TemporalSegmentationTests
{
/// <summary>
/// Verifies temporal segment-map prediction symbols through each of AV1's three neighbor contexts.
/// </summary>
/// <param name="context">The sum of predicted above and left neighbors.</param>
[Theory]
[InlineData(0)]
[InlineData(1)]
[InlineData(2)]
public void SegmentIdPredictedRoundTrips(int context)
{
bool[] expected = [false, true, true, false, true, false];
using Av1SymbolWriter writer = new(Configuration.Default, 1, updateCdf: true);
Av1Distribution writerDistribution = Av1DefaultDistributions.SegmentIdPredicted[context];
foreach (bool value in expected)
{
writer.WriteSymbol(value, writerDistribution);
}
using IMemoryOwner<byte> encoded = writer.Exit();
Av1SymbolDecoder decoder = new(Configuration.Default, encoded.GetSpan(), 0, updateCdf: true);
foreach (bool value in expected)
{
Assert.Equal(value, decoder.ReadSegmentIdPredicted(context));
}
}
/// <summary>
/// Verifies that the frame entropy graph copies adapted temporal-prediction state instead of restoring defaults.
/// </summary>
[Fact]
public void FrameEntropyCopyRetainsAdaptedSegmentPrediction()
{
Av1FrameEntropyContext source = new(0);
Av1FrameEntropyContext destination = new(0);
source.SegmentIdPredicted[2].Update(1);
destination.CopyFrom(source);
Assert.Equal(source.SegmentIdPredicted[2][0], destination.SegmentIdPredicted[2][0]);
Assert.NotEqual(16384U, destination.SegmentIdPredicted[2][0]);
}
/// <summary>
/// Verifies that only neighboring blocks which selected temporal prediction contribute to the binary CDF context.
/// </summary>
/// <param name="hasAbove">Whether an above block is available.</param>
/// <param name="abovePredicted">Whether the available above block selected temporal prediction.</param>
/// <param name="hasLeft">Whether a left block is available.</param>
/// <param name="leftPredicted">Whether the available left block selected temporal prediction.</param>
/// <param name="expected">The expected context in the inclusive range zero through two.</param>
[Theory]
[InlineData(false, false, false, false, 0)]
[InlineData(true, false, true, false, 0)]
[InlineData(true, true, false, false, 1)]
[InlineData(false, false, true, true, 1)]
[InlineData(true, true, true, true, 2)]
public void SegmentPredictionContextCountsPredictedNeighbors(
bool hasAbove,
bool abovePredicted,
bool hasLeft,
bool leftPredicted,
int expected)
{
Av1BlockModeInfo aboveModeInfo = hasAbove ? CreateModeInfo(abovePredicted) : null;
Av1BlockModeInfo leftModeInfo = hasLeft ? CreateModeInfo(leftPredicted) : null;
int actual = Av1SymbolContextHelper.GetSegmentIdPredictedContext(aboveModeInfo, leftModeInfo);
Assert.Equal(expected, actual);
}
/// <summary>
/// Verifies that a temporal-prediction symbol selects the minimum retained segment across the complete block and writes it to the current map.
/// </summary>
/// <param name="segmentIdPrecedesSkip">Whether segment syntax precedes the residual-skip flag.</param>
[Theory]
[InlineData(false)]
[InlineData(true)]
public void ReadInterSegmentIdUsesRetainedPrimaryMap(bool segmentIdPrecedesSkip)
{
const int modeInfoSize = 16;
ObuSequenceHeader sequenceHeader = CreateSequenceHeader(64, 64);
ObuFrameHeader primaryHeader = CreateFrameHeader(modeInfoSize, modeInfoSize, segmentationUpdateMap: 1, segmentationTemporalUpdate: 0);
Av1FrameInfo primaryFrameInfo = new(sequenceHeader);
primaryFrameInfo.InitializeSegmentIds(primaryHeader, null);
primaryFrameInfo.SetSegmentId(Av1BlockSize.Block64x64, Point.Empty, 6);
// The target 16x16 block covers sixteen 4x4 cells. One lower retained value proves that prediction scans the
// complete clipped coverage rather than reading only the block origin.
Point lowSegmentPosition = new(4, 4);
primaryFrameInfo.SetSegmentId(Av1BlockSize.Block4x4, lowSegmentPosition, 2);
// The production reference store owns complete reconstructed frames. A minimal monochrome frame buffer keeps
// this test on the real ownership path while the assertions remain confined to retained segmentation state.
Av1FrameBuffer<byte> primaryBuffer = new(Configuration.Default, sequenceHeader, Av1ColorFormat.Yuv400, false);
Av1ReferenceFrame primaryFrame = new(primaryBuffer, primaryHeader, primaryFrameInfo);
using Av1ReferenceFrameStore referenceFrames = new();
referenceFrames.Commit(1, primaryFrame, showFrame: false);
ObuFrameHeader currentHeader = CreateFrameHeader(modeInfoSize, modeInfoSize, segmentationUpdateMap: 1, segmentationTemporalUpdate: 1);
currentHeader.FrameType = ObuFrameType.InterFrame;
currentHeader.PrimaryReferenceFrame = 0;
currentHeader.PrimaryReferenceSlot = 0;
currentHeader.SegmentationParameters.SegmentIdPrecedesSkip = segmentIdPrecedesSkip;
Av1FrameEntropyContexts entropyContexts = new(0);
using Av1TileReader tileReader = new(Configuration.Default, sequenceHeader, currentHeader, entropyContexts, null, referenceFrames);
Point blockPosition = new(2, 2);
Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block16x16, blockPosition);
Av1SuperblockInfo superblockInfo = new(tileReader.FrameInfo, Point.Empty);
Av1PartitionInfo partitionInfo = new(modeInfo, superblockInfo, false, Av1PartitionType.None)
{
ColumnIndex = blockPosition.X,
RowIndex = blockPosition.Y,
AvailableAbove = true,
AvailableLeft = true,
AboveModeInfo = CreateModeInfo(predicted: true),
LeftModeInfo = CreateModeInfo(predicted: false)
};
const int predictionContext = 1;
using Av1SymbolWriter writer = new(Configuration.Default, 1, updateCdf: true);
writer.WriteSymbol(true, Av1DefaultDistributions.SegmentIdPredicted[predictionContext]);
using IMemoryOwner<byte> encoded = writer.Exit();
Av1SymbolDecoder decoder = new(Configuration.Default, encoded.GetSpan(), 0, updateCdf: true);
tileReader.ReadInterSegmentId(ref decoder, ref partitionInfo, beforeSkip: segmentIdPrecedesSkip);
Assert.True(modeInfo.SegmentIdPredicted);
Assert.Equal(2, modeInfo.SegmentId);
for (int row = blockPosition.Y; row < blockPosition.Y + modeInfo.BlockSize.Get4x4HighCount(); row++)
{
for (int column = blockPosition.X; column < blockPosition.X + modeInfo.BlockSize.Get4x4WideCount(); column++)
{
Assert.Equal(2, tileReader.FrameInfo.GetSegmentId(row, column));
}
}
}
/// <summary>
/// Verifies that a skipped inter block uses the spatial predictor without reading a temporal-prediction symbol.
/// </summary>
[Fact]
public void SkippedInterBlockClearsTemporalPredictionAndUsesSpatialSegment()
{
ObuSequenceHeader sequenceHeader = CreateSequenceHeader(64, 64);
ObuFrameHeader frameHeader = CreateFrameHeader(16, 16, segmentationUpdateMap: 1, segmentationTemporalUpdate: 1);
using Av1TileReader tileReader = new(Configuration.Default, sequenceHeader, frameHeader);
Point blockPosition = new(2, 2);
// Three equal spatial neighbors select segment three without consuming a spatial segment symbol. The block is
// initialized as predicted to prove that the normative skipped-block branch explicitly clears the stale flag.
tileReader.FrameInfo.SetSegmentId(Av1BlockSize.Block4x4, new Point(1, 1), 3);
tileReader.FrameInfo.SetSegmentId(Av1BlockSize.Block4x4, new Point(2, 1), 3);
tileReader.FrameInfo.SetSegmentId(Av1BlockSize.Block4x4, new Point(1, 2), 3);
Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block8x8, blockPosition)
{
Skip = true,
SegmentIdPredicted = true
};
Av1SuperblockInfo superblockInfo = new(tileReader.FrameInfo, Point.Empty);
Av1PartitionInfo partitionInfo = new(modeInfo, superblockInfo, false, Av1PartitionType.None)
{
ColumnIndex = blockPosition.X,
RowIndex = blockPosition.Y,
AvailableAbove = true,
AvailableLeft = true
};
using Av1SymbolWriter writer = new(Configuration.Default, 1, updateCdf: true);
using IMemoryOwner<byte> encoded = writer.Exit();
Av1SymbolDecoder decoder = new(Configuration.Default, encoded.GetSpan(), 0, updateCdf: true);
tileReader.ReadInterSegmentId(ref decoder, ref partitionInfo, beforeSkip: false);
Assert.False(modeInfo.SegmentIdPredicted);
Assert.Equal(3, modeInfo.SegmentId);
Assert.Equal(3, tileReader.FrameInfo.GetSegmentId(blockPosition.Y, blockPosition.X));
}
/// <summary>
/// Verifies that retained segmentation maps with different mode-info geometry are unavailable for temporal prediction.
/// </summary>
[Fact]
public void PredictedSegmentIdIsZeroForMismatchedPrimaryGeometry()
{
ObuSequenceHeader sequenceHeader = CreateSequenceHeader(64, 64);
ObuFrameHeader currentHeader = CreateFrameHeader(16, 16, segmentationUpdateMap: 1, segmentationTemporalUpdate: 1);
ObuFrameHeader primaryHeader = CreateFrameHeader(8, 16, segmentationUpdateMap: 1, segmentationTemporalUpdate: 0);
Av1FrameInfo currentFrameInfo = new(sequenceHeader);
Av1FrameInfo primaryFrameInfo = new(sequenceHeader);
currentFrameInfo.InitializeSegmentIds(currentHeader, null);
primaryFrameInfo.InitializeSegmentIds(primaryHeader, null);
primaryFrameInfo.SetSegmentId(Av1BlockSize.Block32x64, Point.Empty, 5);
int actual = currentFrameInfo.GetPredictedSegmentId(primaryFrameInfo, Av1BlockSize.Block16x16, Point.Empty);
Assert.Equal(0, actual);
}
/// <summary>
/// Creates block mode state with the requested temporal segment-prediction flag.
/// </summary>
/// <param name="predicted">Whether the block selected its segment identifier from the retained map.</param>
/// <returns>The initialized block mode state.</returns>
private static Av1BlockModeInfo CreateModeInfo(bool predicted)
=> new(Av1BlockSize.Block4x4, Point.Empty) { SegmentIdPredicted = predicted };
/// <summary>
/// Creates the fixed 64x64-superblock sequence geometry used by segmentation-map tests.
/// </summary>
/// <param name="width">The maximum coded width in pixels.</param>
/// <param name="height">The maximum coded height in pixels.</param>
/// <returns>The initialized monochrome sequence header.</returns>
private static ObuSequenceHeader CreateSequenceHeader(int width, int height)
=> new()
{
MaxFrameWidth = width,
MaxFrameHeight = height,
Use128x128Superblock = false,
ColorConfig = new ObuColorConfig
{
IsMonochrome = true,
BitDepth = Av1BitDepth.EightBit
}
};
/// <summary>
/// Creates the frame geometry and segmentation controls used by direct map tests.
/// </summary>
/// <param name="modeInfoColumnCount">The active width in 4x4 mode-info units.</param>
/// <param name="modeInfoRowCount">The active height in 4x4 mode-info units.</param>
/// <param name="segmentationUpdateMap">Whether the frame updates its segment map.</param>
/// <param name="segmentationTemporalUpdate">Whether map updates may select the retained primary map.</param>
/// <returns>The initialized frame header.</returns>
private static ObuFrameHeader CreateFrameHeader(
int modeInfoColumnCount,
int modeInfoRowCount,
int segmentationUpdateMap,
int segmentationTemporalUpdate)
=> new()
{
ModeInfoColumnCount = modeInfoColumnCount,
ModeInfoRowCount = modeInfoRowCount,
SegmentationParameters = new ObuSegmentationParameters
{
Enabled = true,
LastActiveSegmentId = Av1Constants.MaxSegmentCount - 1,
SegmentationUpdateMap = segmentationUpdateMap,
SegmentationTemporalUpdate = segmentationTemporalUpdate
}
};
}

1
tests/ImageSharp.Tests/Formats/Heif/Av1/ObuFrameLifecycleTests.cs

@ -41,6 +41,7 @@ public class ObuFrameLifecycleTests
// spatial layers, so the lifecycle test exercises real progressive item framing.
private static ReadOnlySpan<byte> ProgressiveTwoFrameObuStream =>
[
// Temporal delimiter and progressive sequence header.
0x12, 0x00,
0x0A, 0x0F, 0x20, 0x13, 0x01, 0x00, 0x80, 0x81, 0x4E, 0x0A, 0x36, 0xBE, 0x48, 0x08, 0x20, 0x34, 0x80,

246
tests/ImageSharp.Tests/Formats/Heif/Av1/ObuSkipModeParametersTests.cs

@ -0,0 +1,246 @@
// Copyright (c) Six Labors.
// Licensed under the Six Labors Split License.
using SixLabors.ImageSharp.Formats.Heif.Av1;
using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit;
using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling;
namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1;
/// <summary>
/// Verifies the frame-level derivation of AV1 skip-mode reference pairs.
/// </summary>
[Trait("Format", "Avif")]
public class ObuSkipModeParametersTests
{
/// <summary>
/// The public theory-data representation of <see cref="ObuFrameType.KeyFrame"/>.
/// </summary>
private const int KeyFrameValue = (int)ObuFrameType.KeyFrame;
/// <summary>
/// The public theory-data representation of <see cref="ObuFrameType.InterFrame"/>.
/// </summary>
private const int InterFrameValue = (int)ObuFrameType.InterFrame;
/// <summary>
/// The public theory-data representation of <see cref="ObuReferenceMode.SingleReference"/>.
/// </summary>
private const int SingleReferenceValue = (int)ObuReferenceMode.SingleReference;
/// <summary>
/// The public theory-data representation of <see cref="ObuReferenceMode.ReferenceModeSelect"/>.
/// </summary>
private const int ReferenceModeSelectValue = (int)ObuReferenceMode.ReferenceModeSelect;
/// <summary>
/// Verifies signed order-hint distances across the modulo-domain boundary.
/// </summary>
[Fact]
public void GetRelativeDistanceWrapsWithinConfiguredDomain()
{
ObuOrderHintInfo orderHintInfo = CreateOrderHintInfo();
Assert.Equal(-2, orderHintInfo.GetRelativeDistance(15, 1));
Assert.Equal(2, orderHintInfo.GetRelativeDistance(1, 15));
}
/// <summary>
/// Verifies that disabled order hints have no temporal ordering.
/// </summary>
[Fact]
public void GetRelativeDistanceReturnsZeroWhenOrderHintsAreDisabled()
{
ObuOrderHintInfo orderHintInfo = new();
Assert.Equal(0, orderHintInfo.GetRelativeDistance(15, 1));
}
/// <summary>
/// Verifies that skip mode selects the nearest past and future canonical reference roles.
/// </summary>
[Fact]
public void DeriveSelectsNearestForwardAndBackwardReferences()
{
ObuOrderHintInfo orderHintInfo = CreateOrderHintInfo();
ObuFrameHeader frameHeader = CreateInterFrame(8, [7, 3, 6, 2, 10, 12, 15]);
frameHeader.SkipModeParameters.Derive(orderHintInfo, frameHeader);
Assert.True(frameHeader.SkipModeParameters.SkipModeAllowed);
Assert.Equal(Av1ReferenceFrameType.Last, frameHeader.SkipModeParameters.FirstReferenceFrame);
Assert.Equal(Av1ReferenceFrameType.Backward, frameHeader.SkipModeParameters.SecondReferenceFrame);
}
/// <summary>
/// Verifies that the derived pair identifies canonical roles rather than their physical reference-map slots.
/// </summary>
[Fact]
public void DeriveOrdersCanonicalRolesIndependentlyOfMappedSlots()
{
ObuOrderHintInfo orderHintInfo = CreateOrderHintInfo();
ObuFrameHeader frameHeader = CreateInterFrame(8, [7, 3, 6, 2, 10, 12, 15]);
Span<uint> referenceFrameIndices = frameHeader.GetReferenceFrameIndices();
Span<uint> referenceOrderHints = frameHeader.GetReferenceOrderHints();
// Several canonical roles deliberately share physical slot seven. The first matching role remains LAST, while
// the future BWDREF role maps to slot four; neither physical slot number becomes part of the derived pair.
referenceFrameIndices.Fill(7);
referenceFrameIndices[(int)Av1ReferenceFrameType.Backward - 1] = 4;
referenceOrderHints[7] = 7;
frameHeader.SkipModeParameters.Derive(orderHintInfo, frameHeader);
Assert.True(frameHeader.SkipModeParameters.SkipModeAllowed);
Assert.Equal(Av1ReferenceFrameType.Last, frameHeader.SkipModeParameters.FirstReferenceFrame);
Assert.Equal(Av1ReferenceFrameType.Backward, frameHeader.SkipModeParameters.SecondReferenceFrame);
}
/// <summary>
/// Verifies that a frame with only future references cannot use skip mode.
/// </summary>
[Fact]
public void DeriveDisallowsSkipModeWithoutForwardReference()
{
ObuOrderHintInfo orderHintInfo = CreateOrderHintInfo();
ObuFrameHeader frameHeader = CreateInterFrame(8, [9, 10, 11, 12, 13, 14, 15]);
frameHeader.SkipModeParameters.Derive(orderHintInfo, frameHeader);
Assert.False(frameHeader.SkipModeParameters.SkipModeAllowed);
Assert.Equal(Av1ReferenceFrameType.None, frameHeader.SkipModeParameters.FirstReferenceFrame);
Assert.Equal(Av1ReferenceFrameType.None, frameHeader.SkipModeParameters.SecondReferenceFrame);
}
/// <summary>
/// Verifies that a forward-only frame selects the two closest distinct past reference orders.
/// </summary>
[Fact]
public void DeriveSelectsTwoNearestForwardReferencesWhenNoBackwardReferenceExists()
{
ObuOrderHintInfo orderHintInfo = CreateOrderHintInfo();
ObuFrameHeader frameHeader = CreateInterFrame(8, [7, 3, 6, 2, 1, 5, 4]);
frameHeader.SkipModeParameters.Derive(orderHintInfo, frameHeader);
Assert.True(frameHeader.SkipModeParameters.SkipModeAllowed);
Assert.Equal(Av1ReferenceFrameType.Last, frameHeader.SkipModeParameters.FirstReferenceFrame);
Assert.Equal(Av1ReferenceFrameType.Last3, frameHeader.SkipModeParameters.SecondReferenceFrame);
}
/// <summary>
/// Verifies that modulo wraparound participates in nearest-reference selection.
/// </summary>
[Fact]
public void DeriveSelectsReferencesAcrossOrderHintWraparound()
{
ObuOrderHintInfo orderHintInfo = CreateOrderHintInfo();
ObuFrameHeader frameHeader = CreateInterFrame(1, [12, 15, 11, 10, 2, 5, 7]);
frameHeader.SkipModeParameters.Derive(orderHintInfo, frameHeader);
Assert.True(frameHeader.SkipModeParameters.SkipModeAllowed);
Assert.Equal(Av1ReferenceFrameType.Last2, frameHeader.SkipModeParameters.FirstReferenceFrame);
Assert.Equal(Av1ReferenceFrameType.Backward, frameHeader.SkipModeParameters.SecondReferenceFrame);
}
/// <summary>
/// Verifies that skip mode remains unavailable without two temporally distinct usable reference orders.
/// </summary>
[Fact]
public void DeriveDisallowsSkipModeWithoutReferencePair()
{
ObuOrderHintInfo orderHintInfo = CreateOrderHintInfo();
ObuFrameHeader frameHeader = CreateInterFrame(8, [7, 8, 8, 8, 8, 8, 8]);
frameHeader.SkipModeParameters.Derive(orderHintInfo, frameHeader);
Assert.False(frameHeader.SkipModeParameters.SkipModeAllowed);
Assert.Equal(Av1ReferenceFrameType.None, frameHeader.SkipModeParameters.FirstReferenceFrame);
Assert.Equal(Av1ReferenceFrameType.None, frameHeader.SkipModeParameters.SecondReferenceFrame);
}
/// <summary>
/// Verifies that deriving an ineligible frame clears a reference pair retained by an earlier derivation.
/// </summary>
[Fact]
public void DeriveClearsPreviousReferencePair()
{
ObuOrderHintInfo orderHintInfo = CreateOrderHintInfo();
ObuFrameHeader frameHeader = CreateInterFrame(8, [7, 3, 6, 2, 10, 12, 15]);
frameHeader.SkipModeParameters.Derive(orderHintInfo, frameHeader);
frameHeader.ReferenceMode = ObuReferenceMode.SingleReference;
frameHeader.SkipModeParameters.Derive(orderHintInfo, frameHeader);
Assert.False(frameHeader.SkipModeParameters.SkipModeAllowed);
Assert.Equal(Av1ReferenceFrameType.None, frameHeader.SkipModeParameters.FirstReferenceFrame);
Assert.Equal(Av1ReferenceFrameType.None, frameHeader.SkipModeParameters.SecondReferenceFrame);
}
/// <summary>
/// Verifies the frame modes for which the AV1 syntax forbids skip-mode signaling.
/// </summary>
/// <param name="enableOrderHint">Whether the sequence enables order hints.</param>
/// <param name="frameTypeValue">The numeric coded-frame-type value.</param>
/// <param name="referenceModeValue">The numeric frame-level reference-mode value.</param>
[Theory]
[InlineData(false, InterFrameValue, ReferenceModeSelectValue)]
[InlineData(true, KeyFrameValue, ReferenceModeSelectValue)]
[InlineData(true, InterFrameValue, SingleReferenceValue)]
public void DeriveDisallowsSkipModeForIneligibleFrameSyntax(
bool enableOrderHint,
int frameTypeValue,
int referenceModeValue)
{
ObuOrderHintInfo orderHintInfo = CreateOrderHintInfo();
orderHintInfo.EnableOrderHint = enableOrderHint;
ObuFrameHeader frameHeader = CreateInterFrame(8, [7, 3, 6, 2, 10, 12, 15]);
frameHeader.FrameType = (ObuFrameType)frameTypeValue;
frameHeader.ReferenceMode = (ObuReferenceMode)referenceModeValue;
frameHeader.SkipModeParameters.Derive(orderHintInfo, frameHeader);
Assert.False(frameHeader.SkipModeParameters.SkipModeAllowed);
Assert.Equal(Av1ReferenceFrameType.None, frameHeader.SkipModeParameters.FirstReferenceFrame);
Assert.Equal(Av1ReferenceFrameType.None, frameHeader.SkipModeParameters.SecondReferenceFrame);
}
/// <summary>
/// Creates the four-bit modulo order-hint configuration used by the derivation scenarios.
/// </summary>
/// <returns>The enabled order-hint configuration.</returns>
private static ObuOrderHintInfo CreateOrderHintInfo()
=> new()
{
EnableOrderHint = true,
OrderHintBits = 4,
};
/// <summary>
/// Creates an inter frame whose seven canonical roles map directly to slots zero through six.
/// </summary>
/// <param name="currentOrderHint">The current frame order hint.</param>
/// <param name="referenceOrderHints">The order hint selected by each canonical role.</param>
/// <returns>The initialized inter-frame header.</returns>
private static ObuFrameHeader CreateInterFrame(uint currentOrderHint, ReadOnlySpan<uint> referenceOrderHints)
{
ObuFrameHeader frameHeader = new()
{
FrameType = ObuFrameType.InterFrame,
OrderHint = currentOrderHint,
ReferenceMode = ObuReferenceMode.ReferenceModeSelect,
};
Span<uint> referenceFrameIndices = frameHeader.GetReferenceFrameIndices();
Span<uint> referenceMapOrderHints = frameHeader.GetReferenceOrderHints();
for (int referenceIndex = 0; referenceIndex < Av1Constants.ReferencesPerFrame; referenceIndex++)
{
referenceFrameIndices[referenceIndex] = (uint)referenceIndex;
referenceMapOrderHints[referenceIndex] = referenceOrderHints[referenceIndex];
}
return frameHeader;
}
}
Loading…
Cancel
Save