diff --git a/HEIF_IMPLEMENTATION_PLAN.md b/HEIF_IMPLEMENTATION_PLAN.md index 662ff1e21..099774d55 100644 --- a/HEIF_IMPLEMENTATION_PLAN.md +++ b/HEIF_IMPLEMENTATION_PLAN.md @@ -38,7 +38,7 @@ Status meanings: - **Not started:** supporting primitives may exist, but the production format path is absent. - **Current:** the only work item that should be advanced before taking the next queued item. -Current development stage: **Stage 3 — complete AV1 still-image decoding.** The transform checkpoint is closed: forward transforms use one libaom-shaped SIMD-first operator architecture across `Vector512`, `Vector256`, and `Vector128`, with scalar fallback; inverse production traversal uses the verified `Vector256` and `Vector128` tiers with scalar fallback; and implementation-mechanic type and file suffixes have been removed. Neither AV1 nor HEVC production encoding is implemented. +Current development stage: **Stage 3 — complete AV1 still-image decoding.** The transform checkpoint is closed. Layered decoding now retains reference/header/CDF/motion-field state, derives frame-level skip-mode references, consumes temporal segment prediction, and decodes intra-coded blocks inside inter frames. True inter-coded blocks still stop before reference/MV parsing and reconstruction. Neither AV1 nor HEVC production encoding is implemented. Immediate checkpoint: **complete layered AV1 image-item decoding through the existing image-only container surface.** This includes `a1op`, `lsel`, and `a1lx` properties, operating-point selection, dependency-preserving layer consumption, and final or explicitly selected spatial-layer output for color, alpha, and grid items. The bounded decoder session now retains reference owners and the header, entropy, segmentation, loop-filter, global-motion, and temporal motion-field state required by dependent layers. Inter tile syntax and reconstruction still need to consume that state. This work must not be represented as animation or expanded into a general ISO BMFF/video model. @@ -46,7 +46,7 @@ Immediate checkpoint: **complete layered AV1 image-item decoding through the exi | --- | --- | --- | --- | --- | | 1 | Baseline, provenance, documentation, and public contract | In progress | Pinned codec references, a bounded image-only scope, encoder options, typed bit depth, decoder-option propagation, and extensive HEIF documentation exist. | Complete the all-file documentation audit, record a fresh Release baseline, finish distinct public HEIC/AVIF save boundaries, and close API review. | | 2 | Bounded HEIF item and image-sequence container | In progress | Still-item parsing, grids, auxiliary alpha, metadata properties, bounded image-sequence tracks, Identify, and all-sync AV1 sequence presentation are connected. | Complete adversarial boundary coverage, remaining item/property behavior, reference-dependent sequence reconstruction, and the bounded sequence writer. | -| 3 | Still-image AV1 and HEVC decoding | **Current** | HEVC reconstruction reaches exact HM/libheif fixtures across the recorded 8/10/12-bit and chroma cases. AV1 includes bounded OBU framing, reconstruction, filters, grain, color, transforms, intra-block copy, an exact independent 12-profile bit-depth/chroma matrix through every dispatch tier, and retained layered reference/header/CDF state. | Complete inter tile syntax and reconstruction, including temporal segmentation consumption and motion/global/warped prediction, remove every other valid AV1 still-image unsupported branch with independent compression-tool vectors, then complete the remaining HEVC profile and Range Extensions matrix. | +| 3 | Still-image AV1 and HEVC decoding | **Current** | HEVC reconstruction reaches exact HM/libheif fixtures across the recorded 8/10/12-bit and chroma cases. AV1 includes bounded OBU framing, reconstruction, filters, grain, color, transforms, intra-block copy, an exact independent 12-profile bit-depth/chroma matrix through every dispatch tier, retained layered reference/header/CDF state, temporal segment prediction, and the inter-frame intra-coded-block branch. | Complete true inter tile syntax and reconstruction, including reference/MV, compound, global, and warped prediction; remove every other valid AV1 still-image unsupported branch with independent compression-tool vectors; then complete the remaining HEVC profile and Range Extensions matrix. | | 4 | Complete decoded presentation and animation | In progress | Shared SIMD-first AV1/HEVC color conversion, ICC application, grids, transforms, direct planar alpha composition, frame metadata, repetition, and independently decodable AV1 sequence samples exist. | Close the full color/ICC cross-product, HEVC sequence decoding, AV1/HEVC reference-dependent samples, frame-local metadata/alpha behavior, and independent animated decode vectors. | | 5 | AV1/AVIF encoding | Not started | RGB-to-planar conversion, forward transforms, OBU writer foundations, options, and container-writing infrastructure exist. | `HeifEncoderCore` still rejects AV1. Implement a real independently decodable lossy/lossless AV1 payload and the complete AVIF item/metadata matrix. | | 6 | HEVC/HEIC encoding | Not started | Shared input color conversion, options, and HEIF writer infrastructure exist. | `HeifEncoderCore` still rejects HEVC. Implement a real independently decodable lossy/lossless HEVC payload and the complete HEIC item/metadata matrix. | @@ -68,10 +68,12 @@ Immediate checkpoint: **complete layered AV1 image-item decoding through the exi - [x] Parse full and short reference signaling against retained-slot occupancy separately from frame-ID validity, resolve the primary-reference slot, implement `frame_size_with_refs`, and read high-precision-motion-vector, interpolation-filter, switchable-motion-mode, and reference-frame-motion-vector flags. - [x] Initialize each frame's working CDF graph from the resolved primary snapshot, inherit segmentation feature data and an unchanged contiguous segmentation map, inherit loop-filter delta state, parse and inherit global-motion parameters, and initialize/project the retained per-8x8 temporal motion field. - [x] Match libaom's bounded range-decoder consumed-bit accounting and tile trailing-bit validation so implicit zero refill cannot conceal truncated entropy data. Validate before publishing the selected CDF, reset decoder state after failure, and unwind every successful frame/block workspace rent when a later constructor allocation fails. A real truncated palette tile, decoder reuse, parser lifecycle, and allocator-identity tests cover these boundaries. - - [ ] Consume the retained segmentation map through `seg_id_predicted` when temporal segmentation is enabled. The current intra-only tile path does not read that inter-block decision. + - [x] Consume the retained segmentation map through `seg_id_predicted` when temporal segmentation is enabled, including all three neighbor CDF contexts, pre-skip/post-skip ordering, skipped-block spatial inference, compatible retained-map geometry, and current-map updates. - [ ] Implement the complete inter-frame entropy, mode, motion-vector, compound-prediction, inter-prediction, and warped/global-motion paths permitted by the image profile. + - [x] Derive the frame-level skip-mode reference pair from mapped order hints, including modulo wraparound and the two-forward fallback, then decode the common inter-frame block prefix and intra-coded-block branch with retained CDF state and block-size luma contexts. - [x] Implement allocation-free SIMD-first translational single-reference interpolation for regular, smooth, sharp, and bilinear filters across 8/10/12-bit samples. The predictor mirrors JPEG's closed static operator architecture, descends through `Vector512`, `Vector256`, and `Vector128` before scalar fallback, and passes the exact independent convolution oracle through `FeatureTestRunner`. - - [ ] Decode inter-block reference selection, motion-vector derivation, compound and inter-intra modes, masked blending, OBMC, and warped/global-motion prediction, then write predicted and residual samples through tile reconstruction. + - [ ] **Current:** decode single-reference inter selection, the spatial/temporal reference-MV stack, NEAREST/NEAR/NEW/GLOBAL motion modes, DRL and interpolation filters, then reconstruct the complete block once through the existing SIMD-first translational predictor before residual traversal. + - [ ] Decode compound and inter-intra modes, masked blending, OBMC, scaled references, and warped/global-motion prediction without changing the single-reference predictor contract or rounding model. - [ ] Verify every connected mode and filter with independently encoded dependent-layer AV1 image-item fixtures and exact native-plane comparisons. - [ ] Return the explicitly selected spatial layer or the final displayed layer, keeping reference reconstruction separate from display-only film grain. - [ ] Verify color and auxiliary-alpha output exactly against both pinned libavif progressive fixtures under normal SIMD dispatch and all required `FeatureTestRunner` fallbacks. diff --git a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1DefaultDistributions.cs b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1DefaultDistributions.cs index f40d7de25..842ffcb28 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1DefaultDistributions.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1DefaultDistributions.cs @@ -9,7 +9,7 @@ namespace SixLabors.ImageSharp.Formats.Heif.Av1.Entropy; internal static class Av1DefaultDistributions { /// - /// Gets the intra-frame luma-mode distributions indexed by block-size group. + /// Gets the inter-frame intra luma-mode distributions indexed by block-size group. /// public static Av1Distribution[] FrameYMode => [ @@ -19,6 +19,11 @@ internal static class Av1DefaultDistributions new(20155, 21301, 22838, 23178, 23261, 23533, 23703, 24804, 25352, 26575, 27016, 28049) ]; + /// + /// Gets the distributions that select intra or inter prediction from the available spatial neighbors. + /// + public static Av1Distribution[] IntraInter => [new(806), new(16662), new(20186), new(26538)]; + /// /// Gets the key-frame luma-mode distributions indexed by the above and left intra-mode contexts. /// @@ -260,6 +265,14 @@ internal static class Av1DefaultDistributions new(27527, 28487, 28723, 28890, 32397, 32647, 32679), ]; + /// + /// Gets the temporal segment-map prediction distributions indexed by the predicted state of the above and left blocks. + /// + /// + /// AV1 initializes each binary context to 16384, the equiprobable midpoint of its Q15 probability domain. + /// + public static Av1Distribution[] SegmentIdPredicted => [new(16384), new(16384), new(16384)]; + /// /// Gets the key-frame luma intra-mode distributions indexed by the above and left mode contexts. /// diff --git a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1FrameEntropyContext.cs b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1FrameEntropyContext.cs index f0c1a3f6b..de899d117 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1FrameEntropyContext.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1FrameEntropyContext.cs @@ -76,13 +76,16 @@ internal sealed class Av1FrameEntropyContext this.PaletteYColorIndex = Av1DefaultDistributions.PaletteYColorIndex; this.PaletteUvColorIndex = Av1DefaultDistributions.PaletteUvColorIndex; this.PartitionTypes = Av1DefaultDistributions.PartitionTypes; + this.FrameYMode = Av1DefaultDistributions.FrameYMode; this.KeyFrameYMode = Av1DefaultDistributions.KeyFrameYMode; + this.IntraInter = Av1DefaultDistributions.IntraInter; this.UvMode = Av1DefaultDistributions.UvMode; this.Skip = Av1DefaultDistributions.Skip; this.SkipMode = Av1DefaultDistributions.SkipMode; this.DeltaLoopFilterAbsolute = Av1DefaultDistributions.DeltaLoopFilterAbsolute; this.DeltaQuantizerAbsolute = Av1DefaultDistributions.DeltaQuantizerAbsolute; this.SegmentId = Av1DefaultDistributions.SegmentId; + this.SegmentIdPredicted = Av1DefaultDistributions.SegmentIdPredicted; this.AngleDelta = Av1DefaultDistributions.AngleDelta; this.FilterIntraMode = Av1DefaultDistributions.FilterIntraMode; this.FilterIntra = Av1DefaultDistributions.FilterIntra; @@ -124,13 +127,16 @@ internal sealed class Av1FrameEntropyContext this.PaletteYColorIndex = Av1Distribution.CreateCopy(source.PaletteYColorIndex); this.PaletteUvColorIndex = Av1Distribution.CreateCopy(source.PaletteUvColorIndex); this.PartitionTypes = Av1Distribution.CreateCopy(source.PartitionTypes); + this.FrameYMode = Av1Distribution.CreateCopy(source.FrameYMode); this.KeyFrameYMode = Av1Distribution.CreateCopy(source.KeyFrameYMode); + this.IntraInter = Av1Distribution.CreateCopy(source.IntraInter); this.UvMode = Av1Distribution.CreateCopy(source.UvMode); this.Skip = Av1Distribution.CreateCopy(source.Skip); this.SkipMode = Av1Distribution.CreateCopy(source.SkipMode); this.DeltaLoopFilterAbsolute = source.DeltaLoopFilterAbsolute.CreateCopy(); this.DeltaQuantizerAbsolute = source.DeltaQuantizerAbsolute.CreateCopy(); this.SegmentId = Av1Distribution.CreateCopy(source.SegmentId); + this.SegmentIdPredicted = Av1Distribution.CreateCopy(source.SegmentIdPredicted); this.AngleDelta = Av1Distribution.CreateCopy(source.AngleDelta); this.FilterIntraMode = source.FilterIntraMode.CreateCopy(); this.FilterIntra = Av1Distribution.CreateCopy(source.FilterIntra); @@ -208,11 +214,21 @@ internal sealed class Av1FrameEntropyContext /// public Av1Distribution[] PartitionTypes { get; } + /// + /// Gets the inter-frame intra luma-mode distributions indexed by the normative block-size group. + /// + public Av1Distribution[] FrameYMode { get; } + /// /// Gets the key-frame luma-mode distributions. /// public Av1Distribution[][] KeyFrameYMode { get; } + /// + /// Gets the distributions that select intra or inter prediction from the available spatial neighbors. + /// + public Av1Distribution[] IntraInter { get; } + /// /// Gets the chroma intra-mode distributions. /// @@ -243,6 +259,11 @@ internal sealed class Av1FrameEntropyContext /// public Av1Distribution[] SegmentId { get; } + /// + /// Gets the temporal segment-map prediction distributions. + /// + public Av1Distribution[] SegmentIdPredicted { get; } + /// /// Gets the directional angle-delta distributions. /// @@ -363,13 +384,16 @@ internal sealed class Av1FrameEntropyContext CopyState(source.PaletteYColorIndex, this.PaletteYColorIndex); CopyState(source.PaletteUvColorIndex, this.PaletteUvColorIndex); CopyState(source.PartitionTypes, this.PartitionTypes); + CopyState(source.FrameYMode, this.FrameYMode); CopyState(source.KeyFrameYMode, this.KeyFrameYMode); + CopyState(source.IntraInter, this.IntraInter); CopyState(source.UvMode, this.UvMode); CopyState(source.Skip, this.Skip); CopyState(source.SkipMode, this.SkipMode); this.DeltaLoopFilterAbsolute.CopyFrom(source.DeltaLoopFilterAbsolute); this.DeltaQuantizerAbsolute.CopyFrom(source.DeltaQuantizerAbsolute); CopyState(source.SegmentId, this.SegmentId); + CopyState(source.SegmentIdPredicted, this.SegmentIdPredicted); CopyState(source.AngleDelta, this.AngleDelta); this.FilterIntraMode.CopyFrom(source.FilterIntraMode); CopyState(source.FilterIntra, this.FilterIntra); @@ -418,13 +442,16 @@ internal sealed class Av1FrameEntropyContext ResetUpdateCounts(this.PaletteYColorIndex); ResetUpdateCounts(this.PaletteUvColorIndex); ResetUpdateCounts(this.PartitionTypes); + ResetUpdateCounts(this.FrameYMode); ResetUpdateCounts(this.KeyFrameYMode); + ResetUpdateCounts(this.IntraInter); ResetUpdateCounts(this.UvMode); ResetUpdateCounts(this.Skip); ResetUpdateCounts(this.SkipMode); this.DeltaLoopFilterAbsolute.ResetUpdateCount(); this.DeltaQuantizerAbsolute.ResetUpdateCount(); ResetUpdateCounts(this.SegmentId); + ResetUpdateCounts(this.SegmentIdPredicted); ResetUpdateCounts(this.AngleDelta); this.FilterIntraMode.ResetUpdateCount(); ResetUpdateCounts(this.FilterIntra); diff --git a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolContextHelper.cs b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolContextHelper.cs index b61ccd038..00c9205a9 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolContextHelper.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolContextHelper.cs @@ -586,6 +586,57 @@ internal static class Av1SymbolContextHelper public static int GetSegmentId(int[][] segmentIds, int rowIndex, int columnIndex) => segmentIds[rowIndex][columnIndex]; + /// + /// Gets the intra/inter prediction context from the immediately above and left blocks. + /// + /// The above block, or at a tile boundary. + /// The left block, or at a tile boundary. + /// The context in the inclusive range zero through three. + public static int GetIntraInterContext(Av1BlockModeInfo? above, Av1BlockModeInfo? left) + { + if (above is not null && left is not null) + { + bool aboveIsIntra = above.ReferenceFrames[0] <= Av1ReferenceFrameType.Intra; + bool leftIsIntra = left.ReferenceFrames[0] <= Av1ReferenceFrameType.Intra; + + // AV1 reserves context three for two intra neighbors, context one for a mixed pair, and context zero for + // two inter neighbors. These values directly index intra_inter_cdf and are not probability ranks. + if (aboveIsIntra && leftIsIntra) + { + return 3; + } + + return aboveIsIntra || leftIsIntra ? 1 : 0; + } + + // A single intra neighbor uses context two. A single inter neighbor and a block with no neighbors both use + // context zero, matching the unavailable-neighbor behavior in libaom's av1_get_intra_inter_context. + if (above is not null) + { + return above.ReferenceFrames[0] <= Av1ReferenceFrameType.Intra ? 2 : 0; + } + + if (left is not null) + { + return left.ReferenceFrames[0] <= Av1ReferenceFrameType.Intra ? 2 : 0; + } + + return 0; + } + + /// + /// Gets the temporal segment-prediction context from the immediately above and left blocks. + /// + /// The above block, or at a tile boundary. + /// The left block, or at a tile boundary. + /// The context in the inclusive range zero through two. + public static int GetSegmentIdPredictedContext(Av1BlockModeInfo? aboveModeInfo, Av1BlockModeInfo? leftModeInfo) + { + int abovePredicted = aboveModeInfo is not null && aboveModeInfo.SegmentIdPredicted ? 1 : 0; + int leftPredicted = leftModeInfo is not null && leftModeInfo.SegmentIdPredicted ? 1 : 0; + return abovePredicted + leftPredicted; + } + /// /// Gets the minimum encoded segment identifier across a block's clipped mode-info coverage. /// diff --git a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolDecoder.cs b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolDecoder.cs index 308ca9751..c3a0de0ae 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolDecoder.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolDecoder.cs @@ -351,6 +351,31 @@ internal ref struct Av1SymbolDecoder return (Av1PredictionMode)r.ReadSymbol(this.context.KeyFrameYMode[aboveContext][leftContext]); } + /// + /// Reads an intra luma prediction mode for a block coded inside an inter frame. + /// + /// The decoded block size that selects the luma-mode distribution. + /// The decoded intra luma prediction mode. + public Av1PredictionMode ReadInterFrameYMode(Av1BlockSize blockSize) + { + // AV1 section 9.3 groups blocks by the smaller base-two dimension in 4x4 units, capped at group three. + // Calculating it from the existing logarithms exactly matches libaom's size_group_lookup without another table. + int sizeGroup = Math.Min(3, Math.Min(blockSize.Get4x4WidthLog2(), blockSize.Get4x4HeightLog2())); + ref Av1SymbolReader r = ref this.reader; + return (Av1PredictionMode)r.ReadSymbol(this.context.FrameYMode[sizeGroup]); + } + + /// + /// Reads whether an inter-frame block uses inter prediction. + /// + /// The spatial intra/inter context in the inclusive range zero through three. + /// when the block uses inter prediction; otherwise, . + public bool ReadIsInter(int context) + { + ref Av1SymbolReader r = ref this.reader; + return r.ReadSymbol(this.context.IntraInter[context]) != 0; + } + /// /// Reads a chroma intra prediction mode conditioned on the luma mode and chroma-from-luma availability. /// @@ -445,6 +470,17 @@ internal ref struct Av1SymbolDecoder return r.ReadSymbol(this.context.SegmentId[context]); } + /// + /// Reads whether the current segment identifier is predicted from the retained primary-frame map. + /// + /// The sum of the above and left blocks' temporal-prediction flags. + /// when the retained map supplies the segment identifier. + public bool ReadSegmentIdPredicted(int context) + { + ref Av1SymbolReader r = ref this.reader; + return r.ReadSymbol(this.context.SegmentIdPredicted[context]) > 0; + } + /// /// Reads the unsigned directional angle-delta symbol for a prediction mode. /// diff --git a/src/ImageSharp/Formats/Heif/Av1/OpenBitstreamUnit/ObuOrderHintInfo.cs b/src/ImageSharp/Formats/Heif/Av1/OpenBitstreamUnit/ObuOrderHintInfo.cs index 06f93f00e..d8d0aacd6 100644 --- a/src/ImageSharp/Formats/Heif/Av1/OpenBitstreamUnit/ObuOrderHintInfo.cs +++ b/src/ImageSharp/Formats/Heif/Av1/OpenBitstreamUnit/ObuOrderHintInfo.cs @@ -27,4 +27,27 @@ internal class ObuOrderHintInfo /// Gets or sets the number of bits used to encode order hints. /// public int OrderHintBits { get; set; } + + /// + /// Computes the signed distance between two order hints in the sequence's modulo order-hint domain. + /// + /// The first order hint. + /// The order hint subtracted from . + /// + /// The shortest signed modulo distance, or zero when order hints are disabled for the sequence. + /// + public int GetRelativeDistance(uint first, uint second) + { + if (!this.EnableOrderHint) + { + return 0; + } + + int difference = (int)first - (int)second; + int signBit = 1 << (this.OrderHintBits - 1); + + // Folding around the sign bit maps the unsigned difference to [-2^(bits - 1), 2^(bits - 1)), including + // the wraparound between the highest encoded order hint and zero. + return (difference & (signBit - 1)) - (difference & signBit); + } } diff --git a/src/ImageSharp/Formats/Heif/Av1/OpenBitstreamUnit/ObuReader.cs b/src/ImageSharp/Formats/Heif/Av1/OpenBitstreamUnit/ObuReader.cs index e00ccf668..7b888d779 100644 --- a/src/ImageSharp/Formats/Heif/Av1/OpenBitstreamUnit/ObuReader.cs +++ b/src/ImageSharp/Formats/Heif/Av1/OpenBitstreamUnit/ObuReader.cs @@ -2399,23 +2399,9 @@ internal class ObuReader /// The frame header that receives the skip-mode state. private static void ReadSkipModeParameters(ref Av1BitStreamReader reader, ObuSequenceHeader sequenceHeader, ObuFrameHeader frameHeader) { - if (frameHeader.IsIntra || frameHeader.ReferenceMode == ObuReferenceMode.SingleReference || !sequenceHeader.OrderHintInfo.EnableOrderHint) - { - frameHeader.SkipModeParameters.SkipModeAllowed = false; - } - else - { - // Not applicable for INTRA frames. - } - - if (frameHeader.SkipModeParameters.SkipModeAllowed) - { - frameHeader.SkipModeParameters.SkipModeFlag = reader.ReadBoolean(); - } - else - { - frameHeader.SkipModeParameters.SkipModeFlag = false; - } + ObuSkipModeParameters parameters = frameHeader.SkipModeParameters; + parameters.Derive(sequenceHeader.OrderHintInfo, frameHeader); + parameters.SkipModeFlag = parameters.SkipModeAllowed && reader.ReadBoolean(); } /// diff --git a/src/ImageSharp/Formats/Heif/Av1/OpenBitstreamUnit/ObuSkipModeParameters.cs b/src/ImageSharp/Formats/Heif/Av1/OpenBitstreamUnit/ObuSkipModeParameters.cs index eb394a3b2..ca83961df 100644 --- a/src/ImageSharp/Formats/Heif/Av1/OpenBitstreamUnit/ObuSkipModeParameters.cs +++ b/src/ImageSharp/Formats/Heif/Av1/OpenBitstreamUnit/ObuSkipModeParameters.cs @@ -1,6 +1,8 @@ // Copyright (c) Six Labors. // Licensed under the Six Labors Split License. +using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; + namespace SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit; /// @@ -9,12 +11,99 @@ namespace SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit; internal class ObuSkipModeParameters { /// - /// Gets or sets a value indicating whether the frame is permitted to use skip mode. + /// Gets a value indicating whether the frame is permitted to use skip mode. /// - public bool SkipModeAllowed { get; set; } + public bool SkipModeAllowed { get; private set; } /// /// Gets or sets a value indicating whether skip mode is enabled for the frame. /// - public bool SkipModeFlag { get; internal set; } + public bool SkipModeFlag { get; set; } + + /// + /// Gets the first canonical inter-reference type selected for skip-mode blocks. + /// + public Av1ReferenceFrameType FirstReferenceFrame { get; private set; } = Av1ReferenceFrameType.None; + + /// + /// Gets the second canonical inter-reference type selected for skip-mode blocks. + /// + public Av1ReferenceFrameType SecondReferenceFrame { get; private set; } = Av1ReferenceFrameType.None; + + /// + /// Derives skip-mode availability and its reference pair from the current frame's retained-reference mapping. + /// + /// The sequence-level order-hint configuration. + /// The current frame header and its seven canonical inter-reference mappings. + public void Derive(ObuOrderHintInfo orderHintInfo, ObuFrameHeader frameHeader) + { + this.SkipModeAllowed = false; + this.FirstReferenceFrame = Av1ReferenceFrameType.None; + this.SecondReferenceFrame = Av1ReferenceFrameType.None; + + if (!orderHintInfo.EnableOrderHint || frameHeader.IsIntra || frameHeader.ReferenceMode == ObuReferenceMode.SingleReference) + { + return; + } + + ReadOnlySpan referenceFrameIndices = frameHeader.GetReferenceFrameIndices(); + ReadOnlySpan referenceOrderHints = frameHeader.GetReferenceOrderHints(); + int nearestForwardOrderHint = -1; + int nearestBackwardOrderHint = int.MaxValue; + int nearestForwardReferenceIndex = -1; + int nearestBackwardReferenceIndex = -1; + + // The seven entries are canonical roles, while each value selects one physical reference-map slot. Compare + // the selected slot's order hint so duplicate roles retain the same deterministic ordering as libaom. + for (int referenceIndex = 0; referenceIndex < Av1Constants.ReferencesPerFrame; referenceIndex++) + { + uint referenceOrderHint = referenceOrderHints[(int)referenceFrameIndices[referenceIndex]]; + int distanceFromCurrent = orderHintInfo.GetRelativeDistance(referenceOrderHint, frameHeader.OrderHint); + + if (distanceFromCurrent < 0 && + (nearestForwardOrderHint == -1 || orderHintInfo.GetRelativeDistance(referenceOrderHint, (uint)nearestForwardOrderHint) > 0)) + { + // Among past frames, the greatest relative order is the closest frame before the current one. + nearestForwardOrderHint = (int)referenceOrderHint; + nearestForwardReferenceIndex = referenceIndex; + } + else if (distanceFromCurrent > 0 && + (nearestBackwardOrderHint == int.MaxValue || orderHintInfo.GetRelativeDistance(referenceOrderHint, (uint)nearestBackwardOrderHint) < 0)) + { + // Among future frames, the smallest relative order is the closest frame after the current one. + nearestBackwardOrderHint = (int)referenceOrderHint; + nearestBackwardReferenceIndex = referenceIndex; + } + } + + if (nearestForwardReferenceIndex >= 0 && nearestBackwardReferenceIndex < 0) + { + nearestBackwardOrderHint = -1; + + // A forward-only sequence pairs the nearest past frame with the closest distinct frame preceding it. + for (int referenceIndex = 0; referenceIndex < Av1Constants.ReferencesPerFrame; referenceIndex++) + { + uint referenceOrderHint = referenceOrderHints[(int)referenceFrameIndices[referenceIndex]]; + bool precedesNearestForward = orderHintInfo.GetRelativeDistance(referenceOrderHint, (uint)nearestForwardOrderHint) < 0; + + if (precedesNearestForward && + (nearestBackwardOrderHint == -1 || orderHintInfo.GetRelativeDistance(referenceOrderHint, (uint)nearestBackwardOrderHint) > 0)) + { + nearestBackwardOrderHint = (int)referenceOrderHint; + nearestBackwardReferenceIndex = referenceIndex; + } + } + } + + if (nearestForwardReferenceIndex < 0 || nearestBackwardReferenceIndex < 0) + { + return; + } + + int firstReferenceIndex = Math.Min(nearestForwardReferenceIndex, nearestBackwardReferenceIndex); + int secondReferenceIndex = Math.Max(nearestForwardReferenceIndex, nearestBackwardReferenceIndex); + this.FirstReferenceFrame = (Av1ReferenceFrameType)(firstReferenceIndex + (int)Av1ReferenceFrameType.Last); + this.SecondReferenceFrame = (Av1ReferenceFrameType)(secondReferenceIndex + (int)Av1ReferenceFrameType.Last); + this.SkipModeAllowed = true; + } } diff --git a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1BlockModeInfo.cs b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1BlockModeInfo.cs index edeb939d5..62345222b 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1BlockModeInfo.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1BlockModeInfo.cs @@ -222,6 +222,11 @@ internal class Av1BlockModeInfo /// public int SegmentId { get; set; } + /// + /// Gets or sets a value indicating whether temporal prediction supplied the segment identifier. + /// + public bool SegmentIdPredicted { get; set; } + /// /// Gets or sets the chroma intra-prediction mode. /// diff --git a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.MotionField.cs b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.MotionField.cs index d68a63e9c..0eff7d739 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.MotionField.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.MotionField.cs @@ -137,8 +137,8 @@ internal partial class Av1FrameInfo } InlineArray8 selectedReferences = default; - Span referenceFrameIndices = frameHeader.GetReferenceFrameIndices(); - int orderHintBits = sequenceHeader.OrderHintInfo.OrderHintBits; + ReadOnlySpan referenceFrameIndices = frameHeader.GetReferenceFrameIndices(); + ObuOrderHintInfo orderHintInfo = sequenceHeader.OrderHintInfo; // Capture the seven logical-role order hints before this frame refreshes any physical map slots. Libaom keeps // the same snapshot on RefCntBuffer so a later frame can project this frame's stored motion vectors. @@ -150,7 +150,7 @@ internal partial class Av1FrameInfo selectedReferences[(int)referenceFrameType] = referenceFrame; this.motionFieldReferenceOrderHints[(int)referenceFrameType] = referenceOrderHint; - int relativeDistance = GetRelativeDistance(referenceOrderHint, frameHeader.OrderHint, orderHintBits); + int relativeDistance = orderHintInfo.GetRelativeDistance(referenceOrderHint, frameHeader.OrderHint); this.motionFieldReferenceSides[(int)referenceFrameType] = relativeDistance > 0 ? (sbyte)1 : referenceOrderHint == frameHeader.OrderHint ? (sbyte)-1 : (sbyte)0; @@ -194,7 +194,7 @@ internal partial class Av1FrameInfo remainingProjectionCount--; Av1ReferenceFrame backwardFrame = selectedReferences[(int)Av1ReferenceFrameType.Backward]!; - if (GetRelativeDistance(backwardFrame.FrameHeader.OrderHint, frameHeader.OrderHint, orderHintBits) > 0 && + if (orderHintInfo.GetRelativeDistance(backwardFrame.FrameHeader.OrderHint, frameHeader.OrderHint) > 0 && this.ProjectMotionField(sequenceHeader, frameHeader, backwardFrame, reverseDirection: false)) { remainingProjectionCount--; @@ -202,7 +202,7 @@ internal partial class Av1FrameInfo Av1ReferenceFrame alternate2Frame = selectedReferences[(int)Av1ReferenceFrameType.Alternate2]!; - if (GetRelativeDistance(alternate2Frame.FrameHeader.OrderHint, frameHeader.OrderHint, orderHintBits) > 0 && + if (orderHintInfo.GetRelativeDistance(alternate2Frame.FrameHeader.OrderHint, frameHeader.OrderHint) > 0 && this.ProjectMotionField(sequenceHeader, frameHeader, alternate2Frame, reverseDirection: false)) { remainingProjectionCount--; @@ -211,7 +211,7 @@ internal partial class Av1FrameInfo Av1ReferenceFrame alternateFrame = selectedReferences[(int)Av1ReferenceFrameType.Alternate]!; if (remainingProjectionCount > 0 && - GetRelativeDistance(alternateFrame.FrameHeader.OrderHint, frameHeader.OrderHint, orderHintBits) > 0 && + orderHintInfo.GetRelativeDistance(alternateFrame.FrameHeader.OrderHint, frameHeader.OrderHint) > 0 && this.ProjectMotionField(sequenceHeader, frameHeader, alternateFrame, reverseDirection: false)) { remainingProjectionCount--; @@ -331,11 +331,10 @@ internal partial class Av1FrameInfo } Av1FrameInfo startFrameInfo = startFrame.FrameInfo; - int orderHintBits = sequenceHeader.OrderHintInfo.OrderHintBits; - int startToCurrentFrameOffset = GetRelativeDistance( + ObuOrderHintInfo orderHintInfo = sequenceHeader.OrderHintInfo; + int startToCurrentFrameOffset = orderHintInfo.GetRelativeDistance( startFrameHeader.OrderHint, - frameHeader.OrderHint, - orderHintBits); + frameHeader.OrderHint); if (reverseDirection) { @@ -358,10 +357,9 @@ internal partial class Av1FrameInfo continue; } - int referenceFrameOffset = GetRelativeDistance( + int referenceFrameOffset = orderHintInfo.GetRelativeDistance( startFrameHeader.OrderHint, - startFrameInfo.motionFieldReferenceOrderHints[(int)source.ReferenceFrame], - orderHintBits); + startFrameInfo.motionFieldReferenceOrderHints[(int)source.ReferenceFrame]); bool positionIsValid = Math.Abs(referenceFrameOffset) <= MaximumFrameDistance && referenceFrameOffset > 0 && @@ -466,20 +464,6 @@ internal partial class Av1FrameInfo projectedColumn < baseBlockColumn + 8 + MaximumHorizontalFieldOffset; } - /// - /// Computes the signed distance between two order hints in their modulo domain. - /// - /// The first order hint. - /// The order hint subtracted from . - /// The number of bits in the order-hint domain. - /// The shortest signed modulo distance. - private static int GetRelativeDistance(uint first, uint second, int orderHintBits) - { - int difference = (int)first - (int)second; - int signBit = 1 << (orderHintBits - 1); - return (difference & (signBit - 1)) - (difference & signBit); - } - /// /// Stores one motion vector and logical reference retained for projection by a later frame. /// diff --git a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.cs b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.cs index a20f0aa79..3992d38b6 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.cs @@ -246,16 +246,23 @@ internal partial class Av1FrameInfo /// /// Gets the minimum retained segment identifier across a block's clipped mode-information coverage. /// + /// + /// The retained primary-frame state, or when no compatible map is available. + /// /// The block size whose 4x4 coverage is inspected. /// The block origin in frame-relative 4x4 units. /// - /// The minimum retained segment identifier, or zero when the retained frame has no enabled segmentation map. + /// The minimum retained segment identifier, or zero when no same-sized retained segmentation map is available. /// - public int GetPredictedSegmentId(Av1BlockSize blockSize, Point modeInfoPosition) + public int GetPredictedSegmentId(Av1FrameInfo? primaryReferenceFrameInfo, Av1BlockSize blockSize, Point modeInfoPosition) { - if (this.segmentIds.Length == 0) + if (primaryReferenceFrameInfo is null || + primaryReferenceFrameInfo.segmentIds.Length == 0 || + primaryReferenceFrameInfo.segmentIdColumnCount != this.segmentIdColumnCount || + primaryReferenceFrameInfo.segmentIdRowCount != this.segmentIdRowCount) { - // libaom represents an unavailable prior map with a null pointer and predicts segment zero. + // libaom exposes the prior map only when both mode-info dimensions match the active frame. Treating a + // differently sized retained map as absent prevents coordinates from being reinterpreted with a new stride. return 0; } @@ -267,8 +274,8 @@ internal partial class Av1FrameInfo // dec_get_segment_id rule used when segmentation_temporal_update selects the retained primary map. for (int row = 0; row < rowCount; row++) { - int offset = ((modeInfoPosition.Y + row) * this.segmentIdColumnCount) + modeInfoPosition.X; - ReadOnlySpan segmentRow = this.segmentIds.AsSpan(offset, columnCount); + int offset = ((modeInfoPosition.Y + row) * primaryReferenceFrameInfo.segmentIdColumnCount) + modeInfoPosition.X; + ReadOnlySpan segmentRow = primaryReferenceFrameInfo.segmentIds.AsSpan(offset, columnCount); for (int column = 0; column < segmentRow.Length; column++) { diff --git a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileReader.cs b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileReader.cs index d640d2a22..c800a9ad6 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileReader.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileReader.cs @@ -1353,7 +1353,8 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable int block4x4Width = blockSize.Get4x4WideCount(); int block4x4Height = blockSize.Get4x4HighCount(); - // HEIF still-image decoding follows the independently decodable intra-frame transform-size branch. + // Both intra frames and intra-coded blocks inside inter frames use the intra transform-size branch. The true + // inter branch will replace this fixed false classification when inter reconstruction is connected. Av1TransformSize transformSize = this.ReadTransformSize(ref reader, ref partitionInfo, superblockInfo, tileInfo, true); this.aboveNeighborContext.UpdateTransformation(modeInfoLocation, tileInfo, transformSize, blockSize, false); this.leftNeighborContext.UpdateTransformation(modeInfoLocation, superblockInfo, transformSize, blockSize, false); @@ -1537,16 +1538,62 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable } /// - /// Reads the prediction, segmentation, skip, quantizer, and filter mode information for a still-image block. + /// Reads the prediction, segmentation, skip, quantizer, and filter mode information for a coding block. /// /// The tile symbol decoder. /// The current coding block. /// The active tile boundaries. - /// Implements the intra-frame branch of AV1 section 5.11.6. + /// Implements the frame-type dispatch in AV1 section 5.11.6. private void ReadModeInfo(ref Av1SymbolDecoder reader, ref Av1PartitionInfo partitionInfo, Av1TileInfo tileInfo) { - DebugGuard.IsTrue(this.FrameHeader.FrameType is ObuFrameType.KeyFrame or ObuFrameType.IntraOnlyFrame, "Only INTRA frames supported."); - this.ReadIntraFrameModeInfo(ref reader, ref partitionInfo, tileInfo); + if (this.FrameHeader.IsIntra) + { + this.ReadIntraFrameModeInfo(ref reader, ref partitionInfo, tileInfo); + } + else + { + this.ReadInterFrameModeInfo(ref reader, ref partitionInfo); + } + } + + /// + /// Reads the common inter-frame block prefix and the intra-coded-block prediction branch in bitstream order. + /// + /// The tile symbol decoder. + /// The current coding block and its neighbors. + /// Implements the prefix and intra branch of AV1 section 5.11.7. + internal void ReadInterFrameModeInfo(ref Av1SymbolDecoder reader, ref Av1PartitionInfo partitionInfo) + { + Av1BlockModeInfo modeInfo = partitionInfo.ModeInfo; + modeInfo.MotionVectors.Clear(); + this.ReadInterSegmentId(ref reader, ref partitionInfo, beforeSkip: true); + + modeInfo.SkipMode = this.ReadSkipMode(ref reader, ref partitionInfo); + modeInfo.Skip = modeInfo.SkipMode || this.ReadSkip(ref reader, ref partitionInfo); + + if (!this.FrameHeader.SegmentationParameters.SegmentIdPrecedesSkip) + { + this.ReadInterSegmentId(ref reader, ref partitionInfo, beforeSkip: false); + } + + this.ReadCdef(ref reader, ref partitionInfo); + + if (this.FrameHeader.DeltaQParameters.IsPresent) + { + this.ReadDeltaQuantizerIndex(ref reader, ref partitionInfo); + this.ReadDeltaLoopFilter(ref reader, ref partitionInfo); + } + + bool isInterBlock = modeInfo.SkipMode || this.ReadIsInter(ref reader, ref partitionInfo); + if (isInterBlock) + { + throw new NotSupportedException("AV1 inter-coded block prediction is not implemented."); + } + + modeInfo.ReferenceFrames[0] = Av1ReferenceFrameType.Intra; + modeInfo.ReferenceFrames[1] = Av1ReferenceFrameType.None; + modeInfo.SetPaletteSizes(0, 0); + this.ReadConventionalIntraMode(ref reader, ref partitionInfo, reader.ReadInterFrameYMode(modeInfo.BlockSize)); } /// @@ -1610,48 +1657,61 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable } else { - partitionInfo.ModeInfo.YMode = reader.ReadYMode(partitionInfo.AboveModeInfo, partitionInfo.LeftModeInfo); - - partitionInfo.ModeInfo.SetAngleDelta( - Av1PlaneType.Y, - IntraAngleInfo(ref reader, partitionInfo.ModeInfo.YMode, partitionInfo.ModeInfo.BlockSize)); + Av1PredictionMode yMode = reader.ReadYMode(partitionInfo.AboveModeInfo, partitionInfo.LeftModeInfo); + this.ReadConventionalIntraMode(ref reader, ref partitionInfo, yMode); + } + } - if (partitionInfo.IsChroma && !this.SequenceHeader.ColorConfig.IsMonochrome) - { - partitionInfo.ModeInfo.UvMode = reader.ReadIntraModeUv( - partitionInfo.ModeInfo.YMode, - this.IsChromaForLumaAllowed(ref partitionInfo)); + /// + /// Reads conventional luma and chroma intra-prediction details after the frame branch selects the luma mode CDF. + /// + /// The tile symbol decoder. + /// The current coding block. + /// The luma prediction mode selected by the frame-appropriate distribution. + private void ReadConventionalIntraMode( + ref Av1SymbolDecoder reader, + ref Av1PartitionInfo partitionInfo, + Av1PredictionMode yMode) + { + Av1BlockModeInfo modeInfo = partitionInfo.ModeInfo; + modeInfo.YMode = yMode; + modeInfo.SetAngleDelta(Av1PlaneType.Y, IntraAngleInfo(ref reader, yMode, modeInfo.BlockSize)); - if (partitionInfo.ModeInfo.UvMode == Av1ChromaPredictionMode.ChromaFromLuma) - { - ReadChromaFromLumaAlphas(ref reader, partitionInfo.ModeInfo); - } + if (partitionInfo.IsChroma && !this.SequenceHeader.ColorConfig.IsMonochrome) + { + modeInfo.UvMode = reader.ReadIntraModeUv(yMode, this.IsChromaForLumaAllowed(ref partitionInfo)); - partitionInfo.ModeInfo.SetAngleDelta( - Av1PlaneType.Uv, - IntraAngleInfo(ref reader, partitionInfo.ModeInfo.UvMode.ToLumaMode(), partitionInfo.ModeInfo.BlockSize)); - } - else + if (modeInfo.UvMode == Av1ChromaPredictionMode.ChromaFromLuma) { - partitionInfo.ModeInfo.UvMode = Av1ChromaPredictionMode.DC; + ReadChromaFromLumaAlphas(ref reader, modeInfo); } - if (partitionInfo.ModeInfo.BlockSize >= Av1BlockSize.Block8x8 && - partitionInfo.ModeInfo.BlockSize.GetWidth() <= 64 && - partitionInfo.ModeInfo.BlockSize.GetHeight() <= 64 && - this.FrameHeader.AllowScreenContentTools) - { - this.PaletteModeInfo(ref reader, ref partitionInfo); - } + modeInfo.SetAngleDelta( + Av1PlaneType.Uv, + IntraAngleInfo(ref reader, modeInfo.UvMode.ToLumaMode(), modeInfo.BlockSize)); + } + else + { + modeInfo.UvMode = Av1ChromaPredictionMode.DC; + } - this.FilterIntraModeInfo(ref reader, ref partitionInfo); + if (modeInfo.BlockSize >= Av1BlockSize.Block8x8 && + modeInfo.BlockSize.GetWidth() <= 64 && + modeInfo.BlockSize.GetHeight() <= 64 && + this.FrameHeader.AllowScreenContentTools) + { + this.PaletteModeInfo(ref reader, ref partitionInfo); } + + this.FilterIntraModeInfo(ref reader, ref partitionInfo); } /// - /// Determines whether the frame header permits intra block copy for an intra still image. + /// Determines whether the frame header permits intra block copy for an intra frame. /// - /// when the frame and sequence enable intra block copy; otherwise, . + /// + /// when the frame and sequence enable intra block copy; otherwise, . + /// private bool AllowIntraBlockCopy() => (this.FrameHeader.FrameType is ObuFrameType.KeyFrame or ObuFrameType.IntraOnlyFrame) && (this.SequenceHeader.ForceScreenContentTools > 0) && @@ -1661,7 +1721,10 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable /// Determines whether chroma-from-luma prediction is available for a coding block. /// /// The current coding block. - /// when the lossless transform or block dimensions permit chroma-from-luma prediction; otherwise, . + /// + /// when the lossless transform or block dimensions permit chroma-from-luma prediction; + /// otherwise, . + /// private bool IsChromaForLumaAllowed(ref Av1PartitionInfo partitionInfo) { if (this.FrameHeader.LosslessArray[partitionInfo.ModeInfo.SegmentId]) @@ -2239,7 +2302,82 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable } /// - /// Predicts and, when required, decodes the segment identifier for an intra block. + /// Reads or inherits the segment identifier for one inter-frame block and updates its 4x4 map coverage. + /// + /// The tile symbol decoder. + /// The current coding block and its available neighbors. + /// Whether this invocation precedes the block's residual-skip decision. + /// + /// Implements read_inter_segment_id from AV1 section 5.11.8. + /// + internal void ReadInterSegmentId(ref Av1SymbolDecoder reader, ref Av1PartitionInfo partitionInfo, bool beforeSkip) + { + ObuSegmentationParameters segmentationParameters = this.FrameHeader.SegmentationParameters; + Av1BlockModeInfo modeInfo = partitionInfo.ModeInfo; + + if (!segmentationParameters.Enabled) + { + // Disabled segmentation has no allocated map and normatively assigns segment zero. + modeInfo.SegmentId = 0; + return; + } + + Point modeInfoPosition = new(partitionInfo.ColumnIndex, partitionInfo.RowIndex); + + if (segmentationParameters.SegmentationUpdateMap == 0) + { + // The frame map was inherited as one contiguous copy during reader construction. Resolve the same clipped + // minimum that libaom obtains from last_frame_seg_map so block state and the already copied map agree. + modeInfo.SegmentId = this.FrameInfo.GetPredictedSegmentId(this.primaryReferenceFrameInfo, modeInfo.BlockSize, modeInfoPosition); + return; + } + + if (beforeSkip) + { + if (!segmentationParameters.SegmentIdPrecedesSkip) + { + // The caller invokes this once before skip for every inter block; post-skip segment syntax owns this case. + return; + } + } + else if (modeInfo.Skip) + { + if (segmentationParameters.SegmentationTemporalUpdate == 1) + { + // Skipped blocks use the spatial segment predictor and signal no temporal-prediction bit. + modeInfo.SegmentIdPredicted = false; + } + + this.ReadSegmentId(ref reader, ref partitionInfo); + this.FrameInfo.SetSegmentId(modeInfo.BlockSize, modeInfoPosition, modeInfo.SegmentId); + return; + } + + if (segmentationParameters.SegmentationTemporalUpdate == 1) + { + // The binary context counts only neighboring blocks that themselves selected the retained map. Segment + // values do not participate in this decision. + int context = Av1SymbolContextHelper.GetSegmentIdPredictedContext(partitionInfo.AboveModeInfo, partitionInfo.LeftModeInfo); + modeInfo.SegmentIdPredicted = reader.ReadSegmentIdPredicted(context); + if (modeInfo.SegmentIdPredicted) + { + modeInfo.SegmentId = this.FrameInfo.GetPredictedSegmentId(this.primaryReferenceFrameInfo, modeInfo.BlockSize, modeInfoPosition); + } + else + { + this.ReadSegmentId(ref reader, ref partitionInfo); + } + } + else + { + this.ReadSegmentId(ref reader, ref partitionInfo); + } + + this.FrameInfo.SetSegmentId(modeInfo.BlockSize, modeInfoPosition, modeInfo.SegmentId); + } + + /// + /// Predicts and, when required, decodes the spatially coded segment identifier for a block. /// /// The tile symbol decoder. /// The current coding block and its available neighbors. @@ -2400,6 +2538,62 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable } } + /// + /// Reads compound skip-mode selection when the frame, segment, and block geometry permit it. + /// + /// The tile symbol decoder. + /// The current coding block and its available neighbors. + /// when the block selects the frame's derived skip-mode reference pair. + private bool ReadSkipMode(ref Av1SymbolDecoder reader, ref Av1PartitionInfo partitionInfo) + { + Av1BlockModeInfo modeInfo = partitionInfo.ModeInfo; + ObuSegmentationParameters segmentationParameters = this.FrameHeader.SegmentationParameters; + int segmentId = modeInfo.SegmentId; + + if (!this.FrameHeader.SkipModeParameters.SkipModeFlag || + segmentationParameters.IsFeatureActive(segmentId, ObuSegmentationLevelFeature.Skip) || + Math.Min(modeInfo.BlockSize.GetWidth(), modeInfo.BlockSize.GetHeight()) < 8 || + segmentationParameters.IsFeatureActive(segmentId, ObuSegmentationLevelFeature.ReferenceFrame) || + segmentationParameters.IsFeatureActive(segmentId, ObuSegmentationLevelFeature.GlobalMotionVector)) + { + // Segment reference and global-motion features force single-reference prediction, while skip mode always + // selects the derived compound pair. The syntax therefore omits the skip-mode symbol in either case. + return false; + } + + int aboveSkipMode = partitionInfo.AboveModeInfo is not null && partitionInfo.AboveModeInfo.SkipMode ? 1 : 0; + int leftSkipMode = partitionInfo.LeftModeInfo is not null && partitionInfo.LeftModeInfo.SkipMode ? 1 : 0; + return reader.ReadSkipMode(aboveSkipMode + leftSkipMode); + } + + /// + /// Reads or infers whether an inter-frame coding block uses inter prediction. + /// + /// The tile symbol decoder. + /// The current coding block and its available neighbors. + /// for an inter-coded block; otherwise, . + private bool ReadIsInter(ref Av1SymbolDecoder reader, ref Av1PartitionInfo partitionInfo) + { + ObuSegmentationParameters segmentationParameters = this.FrameHeader.SegmentationParameters; + int segmentId = partitionInfo.ModeInfo.SegmentId; + + if (segmentationParameters.IsFeatureActive(segmentId, ObuSegmentationLevelFeature.ReferenceFrame)) + { + // Reference feature values use the same numeric labels as Av1ReferenceFrameType. INTRA_FRAME is zero; + // every canonical inter reference begins at LAST_FRAME and therefore has a positive value. + int referenceFrame = segmentationParameters.FeatureData[segmentId, (int)ObuSegmentationLevelFeature.ReferenceFrame]; + return referenceFrame >= (int)Av1ReferenceFrameType.Last; + } + + if (segmentationParameters.IsFeatureActive(segmentId, ObuSegmentationLevelFeature.GlobalMotionVector)) + { + return true; + } + + int context = Av1SymbolContextHelper.GetIntraInterContext(partitionInfo.AboveModeInfo, partitionInfo.LeftModeInfo); + return reader.ReadIsInter(context); + } + /// /// Reads and accumulates a superblock quantizer-index delta when the block carries one. /// diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterFrameIntraEntropyTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterFrameIntraEntropyTests.cs new file mode 100644 index 000000000..cf2baf76a --- /dev/null +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterFrameIntraEntropyTests.cs @@ -0,0 +1,257 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Buffers; +using SixLabors.ImageSharp.Formats.Heif.Av1; +using SixLabors.ImageSharp.Formats.Heif.Av1.Entropy; +using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; +using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; +using SixLabors.ImageSharp.Memory; + +namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; + +/// +/// Verifies the entropy state and spatial contexts used by intra-coded blocks inside AV1 inter frames. +/// +[Trait("Format", "Avif")] +public class Av1InterFrameIntraEntropyTests +{ + /// + /// Gets libaom's four forward Q15 luma-mode CDF rows in block-size-group order. + /// + private static ReadOnlySpan FrameYModeForwardThresholds => + [ + 22801, 23489, 24293, 24756, 25601, 26123, 26606, 27418, 27945, 29228, 29685, 30349, + 18673, 19845, 22631, 23318, 23950, 24649, 25527, 27364, 28152, 29701, 29984, 30852, + 19770, 20979, 23396, 23939, 24241, 24654, 25136, 27073, 27830, 29360, 29730, 30659, + 20155, 21301, 22838, 23178, 23261, 23533, 23703, 24804, 25352, 26575, 27016, 28049, + ]; + + /// + /// Verifies the four normative intra/inter distributions against libaom's forward Q15 defaults. + /// + [Fact] + public void IntraInterDefaultsMatchLibaom() + { + uint[] forwardThresholds = [806, 16662, 20186, 26538]; + Av1Distribution[] distributions = Av1DefaultDistributions.IntraInter; + + Assert.Equal(forwardThresholds.Length, distributions.Length); + for (int context = 0; context < distributions.Length; context++) + { + // Av1Distribution stores inverse cumulative thresholds, so compare each libaom default after the same + // forward-to-inverse conversion performed by its constructor. + Assert.Equal((uint)Av1Distribution.ProbabilityTop - forwardThresholds[context], distributions[context][0]); + Assert.Equal(2, distributions[context].NumberOfSymbols); + } + } + + /// + /// Verifies every inter-frame intra luma-mode threshold against libaom's forward Q15 defaults. + /// + [Fact] + public void FrameYModeDefaultsMatchLibaom() + { + const int thresholdsPerGroup = 12; + ReadOnlySpan forwardThresholds = FrameYModeForwardThresholds; + Av1Distribution[] distributions = Av1DefaultDistributions.FrameYMode; + + Assert.Equal(4, distributions.Length); + for (int group = 0; group < distributions.Length; group++) + { + Assert.Equal(thresholdsPerGroup + 1, distributions[group].NumberOfSymbols); + + for (int threshold = 0; threshold < thresholdsPerGroup; threshold++) + { + uint expected = (uint)Av1Distribution.ProbabilityTop - forwardThresholds[(group * thresholdsPerGroup) + threshold]; + Assert.Equal(expected, distributions[group][threshold]); + } + } + } + + /// + /// Verifies that the intra/inter reader selects and adapts each of the four spatial-context distributions. + /// + /// The intra/inter spatial context. + [Theory] + [InlineData(0)] + [InlineData(1)] + [InlineData(2)] + [InlineData(3)] + public void ReadIsInterUsesRequestedContext(int context) + { + bool[] expected = [false, true, true, false, true, false, false, true]; + Av1Distribution writerDistribution = Av1DefaultDistributions.IntraInter[context]; + using Av1SymbolWriter writer = new(Configuration.Default, 8, updateCdf: true); + + foreach (bool value in expected) + { + writer.WriteSymbol(value, writerDistribution); + } + + using IMemoryOwner encoded = writer.Exit(); + Av1SymbolDecoder decoder = new(Configuration.Default, encoded.GetSpan(), 0, updateCdf: true); + + foreach (bool value in expected) + { + Assert.Equal(value, decoder.ReadIsInter(context)); + } + } + + /// + /// Verifies that inter-frame intra luma modes use the normative size group for every AV1 block size. + /// + /// The AV1 block-size enumeration value. + /// The normative size group from AV1 section 9.3. + [Theory] + [MemberData(nameof(GetBlockSizeGroups))] + public void ReadInterFrameYModeUsesNormativeSizeGroup(int blockSizeValue, int sizeGroup) + { + Av1BlockSize blockSize = (Av1BlockSize)blockSizeValue; + Av1PredictionMode[] expected = + [ + Av1PredictionMode.DC, + Av1PredictionMode.Directional45Degrees, + Av1PredictionMode.Smooth, + Av1PredictionMode.Paeth, + Av1PredictionMode.Horizontal, + Av1PredictionMode.Directional157Degrees, + ]; + + Av1Distribution writerDistribution = Av1DefaultDistributions.FrameYMode[sizeGroup]; + using Av1SymbolWriter writer = new(Configuration.Default, 8, updateCdf: true); + + foreach (Av1PredictionMode mode in expected) + { + writer.WriteSymbol((int)mode, writerDistribution); + } + + using IMemoryOwner encoded = writer.Exit(); + Av1SymbolDecoder decoder = new(Configuration.Default, encoded.GetSpan(), 0, updateCdf: true); + + foreach (Av1PredictionMode mode in expected) + { + Assert.Equal(mode, decoder.ReadInterFrameYMode(blockSize)); + } + } + + /// + /// Verifies that all four intra/inter contexts follow the normative above-and-left neighbor classification. + /// + /// Whether the above block is available. + /// Whether the available above block uses inter prediction. + /// Whether the left block is available. + /// Whether the available left block uses inter prediction. + /// The expected intra/inter context. + [Theory] + [InlineData(false, false, false, false, 0)] + [InlineData(true, true, false, false, 0)] + [InlineData(true, false, false, false, 2)] + [InlineData(false, false, true, true, 0)] + [InlineData(false, false, true, false, 2)] + [InlineData(true, true, true, true, 0)] + [InlineData(true, false, true, true, 1)] + [InlineData(true, true, true, false, 1)] + [InlineData(true, false, true, false, 3)] + public void IntraInterContextMatchesNeighborPredictionTypes( + bool hasAbove, + bool aboveIsInter, + bool hasLeft, + bool leftIsInter, + int expected) + { + Av1BlockModeInfo above = hasAbove ? CreateModeInfo(aboveIsInter) : null; + Av1BlockModeInfo left = hasLeft ? CreateModeInfo(leftIsInter) : null; + + int actual = Av1SymbolContextHelper.GetIntraInterContext(above, left); + + Assert.Equal(expected, actual); + } + + /// + /// Verifies that frame-context copies retain adapted inter-frame intra state without sharing mutable distributions. + /// + [Fact] + public void FrameEntropyCopyRetainsIndependentInterFrameIntraState() + { + Av1FrameEntropyContext source = new(0); + Av1FrameEntropyContext destination = new(0); + source.FrameYMode[2].Update((int)Av1PredictionMode.Smooth); + source.IntraInter[3].Update(1); + + destination.CopyFrom(source); + + Assert.Equal(source.FrameYMode[2][0], destination.FrameYMode[2][0]); + Assert.Equal(source.IntraInter[3][0], destination.IntraInter[3][0]); + + source.FrameYMode[2].Update((int)Av1PredictionMode.Paeth); + source.IntraInter[3].Update(0); + + Assert.NotEqual(source.FrameYMode[2][0], destination.FrameYMode[2][0]); + Assert.NotEqual(source.IntraInter[3][0], destination.IntraInter[3][0]); + } + + /// + /// Verifies that a published frame snapshot preserves adapted thresholds but resets their update-rate history. + /// + [Fact] + public void FrameEntropySnapshotResetsInterFrameIntraUpdateCounts() + { + const int updateCount = 20; + Av1FrameEntropyContext source = new(0); + Av1FrameEntropyContext snapshot = new(0); + + for (int i = 0; i < updateCount; i++) + { + source.FrameYMode[1].Update((int)Av1PredictionMode.Vertical); + source.IntraInter[1].Update(1); + } + + source.SnapshotTo(snapshot); + + Assert.Equal(source.FrameYMode[1][0], snapshot.FrameYMode[1][0]); + Assert.Equal(source.IntraInter[1][0], snapshot.IntraInter[1][0]); + + // The source retains twenty observations while the published snapshot restarts at zero. Applying the same + // symbol therefore moves identical thresholds by different update rates only when reset wiring is complete. + source.FrameYMode[1].Update((int)Av1PredictionMode.DC); + snapshot.FrameYMode[1].Update((int)Av1PredictionMode.DC); + source.IntraInter[1].Update(0); + snapshot.IntraInter[1].Update(0); + + Assert.NotEqual(source.FrameYMode[1][0], snapshot.FrameYMode[1][0]); + Assert.NotEqual(source.IntraInter[1][0], snapshot.IntraInter[1][0]); + } + + /// + /// Provides the normative AV1 size-group table in block-size enumeration order. + /// + /// Every decoded block size paired with its luma-mode size group. + public static TheoryData GetBlockSizeGroups() + { + // This is size_group_lookup from AV1 section 9.3 and libaom common_data.h. Keeping expected values explicit + // ensures that the test does not reproduce the production formula it is intended to verify. + int[] sizeGroups = [0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 0, 0, 1, 1, 2, 2]; + TheoryData result = []; + + for (int blockSize = 0; blockSize < sizeGroups.Length; blockSize++) + { + result.Add(blockSize, sizeGroups[blockSize]); + } + + return result; + } + + /// + /// Creates decoded neighbor state with either an intra or inter primary reference. + /// + /// Whether the neighbor uses inter prediction. + /// The initialized block mode state. + private static Av1BlockModeInfo CreateModeInfo(bool isInter) + { + Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block4x4, Point.Empty); + modeInfo.ReferenceFrames[0] = isInter ? Av1ReferenceFrameType.Last : Av1ReferenceFrameType.Intra; + modeInfo.ReferenceFrames[1] = Av1ReferenceFrameType.None; + return modeInfo; + } +} diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterFrameModeInfoTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterFrameModeInfoTests.cs new file mode 100644 index 000000000..5b35a4391 --- /dev/null +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterFrameModeInfoTests.cs @@ -0,0 +1,133 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Buffers; +using SixLabors.ImageSharp.Formats.Heif.Av1; +using SixLabors.ImageSharp.Formats.Heif.Av1.Entropy; +using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit; +using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; +using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; +using SixLabors.ImageSharp.Memory; + +namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; + +/// +/// Verifies the common inter-frame mode prefix and its intra-coded-block branch. +/// +[Trait("Format", "Avif")] +public class Av1InterFrameModeInfoTests +{ + /// + /// Verifies that an inter frame can select an intra-coded block using the block-size luma distribution. + /// + [Fact] + public void ReadInterFrameModeInfoReadsIntraCodedBlock() + { + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(); + ObuFrameHeader frameHeader = CreateFrameHeader(); + using Av1TileReader tileReader = new(Configuration.Default, sequenceHeader, frameHeader); + Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block8x8, Point.Empty); + Av1SuperblockInfo superblockInfo = new(tileReader.FrameInfo, Point.Empty); + Av1PartitionInfo partitionInfo = new(modeInfo, superblockInfo, false, Av1PartitionType.None); + + Av1Distribution skip = Av1DefaultDistributions.Skip[0]; + Av1Distribution intraInter = Av1DefaultDistributions.IntraInter[0]; + Av1Distribution yMode = Av1DefaultDistributions.FrameYMode[1]; + using Av1SymbolWriter writer = new(Configuration.Default, 1, updateCdf: true); + writer.WriteSymbol(false, skip); + writer.WriteSymbol(false, intraInter); + writer.WriteSymbol((int)Av1PredictionMode.DC, yMode); + using IMemoryOwner encoded = writer.Exit(); + Av1SymbolDecoder decoder = new(Configuration.Default, encoded.GetSpan(), 0, updateCdf: true); + + tileReader.ReadInterFrameModeInfo(ref decoder, ref partitionInfo); + + Assert.False(modeInfo.SkipMode); + Assert.False(modeInfo.Skip); + Assert.Equal(Av1ReferenceFrameType.Intra, modeInfo.ReferenceFrames[0]); + Assert.Equal(Av1ReferenceFrameType.None, modeInfo.ReferenceFrames[1]); + Assert.Equal(Av1PredictionMode.DC, modeInfo.YMode); + Assert.Equal(Av1ChromaPredictionMode.DC, modeInfo.UvMode); + } + + /// + /// Verifies that skip mode omits the residual-skip and intra-inter symbols and marks the block as inter coded. + /// + [Fact] + public void ReadInterFrameModeInfoSkipModeForcesInterBlockAndResidualSkip() + { + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(); + ObuFrameHeader frameHeader = CreateFrameHeader(); + frameHeader.SkipModeParameters.SkipModeFlag = true; + using Av1TileReader tileReader = new(Configuration.Default, sequenceHeader, frameHeader); + Av1BlockModeInfo aboveModeInfo = new(Av1BlockSize.Block8x8, Point.Empty) { SkipMode = true }; + Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block8x8, Point.Empty); + + Av1Distribution skipMode = Av1DefaultDistributions.SkipMode[1]; + using Av1SymbolWriter writer = new(Configuration.Default, 1, updateCdf: true); + writer.WriteSymbol(true, skipMode); + using IMemoryOwner encoded = writer.Exit(); + Memory encodedMemory = encoded.Memory; + + Assert.Throws(() => ReadInterFrameModeInfo(tileReader, encodedMemory, modeInfo, aboveModeInfo)); + Assert.True(modeInfo.SkipMode); + Assert.True(modeInfo.Skip); + } + + /// + /// Invokes the ref-struct mode parser for exception assertions that cannot capture its parameters directly. + /// + /// The tile reader. + /// The range-coded block-prefix symbols. + /// The current coding block. + /// The available above block supplying skip-mode context. + private static void ReadInterFrameModeInfo( + Av1TileReader tileReader, + Memory encoded, + Av1BlockModeInfo modeInfo, + Av1BlockModeInfo aboveModeInfo) + { + Av1SuperblockInfo superblockInfo = new(tileReader.FrameInfo, Point.Empty); + Av1PartitionInfo partitionInfo = new(modeInfo, superblockInfo, false, Av1PartitionType.None) + { + AvailableAbove = true, + AboveModeInfo = aboveModeInfo, + }; + + Av1SymbolDecoder decoder = new(Configuration.Default, encoded.Span, 0, updateCdf: true); + tileReader.ReadInterFrameModeInfo(ref decoder, ref partitionInfo); + } + + /// + /// Creates the monochrome 64x64 sequence geometry used by direct mode-prefix tests. + /// + /// The initialized sequence header. + private static ObuSequenceHeader CreateSequenceHeader() + => new() + { + MaxFrameWidth = 64, + MaxFrameHeight = 64, + Use128x128Superblock = false, + EnableCdef = false, + EnableFilterIntra = false, + ColorConfig = new ObuColorConfig + { + IsMonochrome = true, + BitDepth = Av1BitDepth.EightBit, + }, + }; + + /// + /// Creates an inter-frame header whose optional block-prefix tools are disabled. + /// + /// The initialized frame header. + private static ObuFrameHeader CreateFrameHeader() + => new() + { + FrameType = ObuFrameType.InterFrame, + ModeInfoColumnCount = 16, + ModeInfoRowCount = 16, + CodedLossless = true, + AllowScreenContentTools = false, + }; +} diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1TemporalSegmentationTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1TemporalSegmentationTests.cs new file mode 100644 index 000000000..ad7f021d4 --- /dev/null +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1TemporalSegmentationTests.cs @@ -0,0 +1,275 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Buffers; +using SixLabors.ImageSharp.Formats.Heif.Av1; +using SixLabors.ImageSharp.Formats.Heif.Av1.Entropy; +using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit; +using SixLabors.ImageSharp.Formats.Heif.Av1.ReferenceFrames; +using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; +using SixLabors.ImageSharp.Memory; + +namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; + +/// +/// Verifies AV1 temporal segment-map prediction against libaom's decoder rules. +/// +[Trait("Format", "Avif")] +public class Av1TemporalSegmentationTests +{ + /// + /// Verifies temporal segment-map prediction symbols through each of AV1's three neighbor contexts. + /// + /// The sum of predicted above and left neighbors. + [Theory] + [InlineData(0)] + [InlineData(1)] + [InlineData(2)] + public void SegmentIdPredictedRoundTrips(int context) + { + bool[] expected = [false, true, true, false, true, false]; + using Av1SymbolWriter writer = new(Configuration.Default, 1, updateCdf: true); + Av1Distribution writerDistribution = Av1DefaultDistributions.SegmentIdPredicted[context]; + + foreach (bool value in expected) + { + writer.WriteSymbol(value, writerDistribution); + } + + using IMemoryOwner encoded = writer.Exit(); + Av1SymbolDecoder decoder = new(Configuration.Default, encoded.GetSpan(), 0, updateCdf: true); + + foreach (bool value in expected) + { + Assert.Equal(value, decoder.ReadSegmentIdPredicted(context)); + } + } + + /// + /// Verifies that the frame entropy graph copies adapted temporal-prediction state instead of restoring defaults. + /// + [Fact] + public void FrameEntropyCopyRetainsAdaptedSegmentPrediction() + { + Av1FrameEntropyContext source = new(0); + Av1FrameEntropyContext destination = new(0); + source.SegmentIdPredicted[2].Update(1); + + destination.CopyFrom(source); + + Assert.Equal(source.SegmentIdPredicted[2][0], destination.SegmentIdPredicted[2][0]); + Assert.NotEqual(16384U, destination.SegmentIdPredicted[2][0]); + } + + /// + /// Verifies that only neighboring blocks which selected temporal prediction contribute to the binary CDF context. + /// + /// Whether an above block is available. + /// Whether the available above block selected temporal prediction. + /// Whether a left block is available. + /// Whether the available left block selected temporal prediction. + /// The expected context in the inclusive range zero through two. + [Theory] + [InlineData(false, false, false, false, 0)] + [InlineData(true, false, true, false, 0)] + [InlineData(true, true, false, false, 1)] + [InlineData(false, false, true, true, 1)] + [InlineData(true, true, true, true, 2)] + public void SegmentPredictionContextCountsPredictedNeighbors( + bool hasAbove, + bool abovePredicted, + bool hasLeft, + bool leftPredicted, + int expected) + { + Av1BlockModeInfo aboveModeInfo = hasAbove ? CreateModeInfo(abovePredicted) : null; + Av1BlockModeInfo leftModeInfo = hasLeft ? CreateModeInfo(leftPredicted) : null; + + int actual = Av1SymbolContextHelper.GetSegmentIdPredictedContext(aboveModeInfo, leftModeInfo); + + Assert.Equal(expected, actual); + } + + /// + /// Verifies that a temporal-prediction symbol selects the minimum retained segment across the complete block and writes it to the current map. + /// + /// Whether segment syntax precedes the residual-skip flag. + [Theory] + [InlineData(false)] + [InlineData(true)] + public void ReadInterSegmentIdUsesRetainedPrimaryMap(bool segmentIdPrecedesSkip) + { + const int modeInfoSize = 16; + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(64, 64); + ObuFrameHeader primaryHeader = CreateFrameHeader(modeInfoSize, modeInfoSize, segmentationUpdateMap: 1, segmentationTemporalUpdate: 0); + Av1FrameInfo primaryFrameInfo = new(sequenceHeader); + primaryFrameInfo.InitializeSegmentIds(primaryHeader, null); + primaryFrameInfo.SetSegmentId(Av1BlockSize.Block64x64, Point.Empty, 6); + + // The target 16x16 block covers sixteen 4x4 cells. One lower retained value proves that prediction scans the + // complete clipped coverage rather than reading only the block origin. + Point lowSegmentPosition = new(4, 4); + primaryFrameInfo.SetSegmentId(Av1BlockSize.Block4x4, lowSegmentPosition, 2); + + // The production reference store owns complete reconstructed frames. A minimal monochrome frame buffer keeps + // this test on the real ownership path while the assertions remain confined to retained segmentation state. + Av1FrameBuffer primaryBuffer = new(Configuration.Default, sequenceHeader, Av1ColorFormat.Yuv400, false); + Av1ReferenceFrame primaryFrame = new(primaryBuffer, primaryHeader, primaryFrameInfo); + using Av1ReferenceFrameStore referenceFrames = new(); + referenceFrames.Commit(1, primaryFrame, showFrame: false); + + ObuFrameHeader currentHeader = CreateFrameHeader(modeInfoSize, modeInfoSize, segmentationUpdateMap: 1, segmentationTemporalUpdate: 1); + currentHeader.FrameType = ObuFrameType.InterFrame; + currentHeader.PrimaryReferenceFrame = 0; + currentHeader.PrimaryReferenceSlot = 0; + currentHeader.SegmentationParameters.SegmentIdPrecedesSkip = segmentIdPrecedesSkip; + Av1FrameEntropyContexts entropyContexts = new(0); + using Av1TileReader tileReader = new(Configuration.Default, sequenceHeader, currentHeader, entropyContexts, null, referenceFrames); + + Point blockPosition = new(2, 2); + Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block16x16, blockPosition); + Av1SuperblockInfo superblockInfo = new(tileReader.FrameInfo, Point.Empty); + Av1PartitionInfo partitionInfo = new(modeInfo, superblockInfo, false, Av1PartitionType.None) + { + ColumnIndex = blockPosition.X, + RowIndex = blockPosition.Y, + AvailableAbove = true, + AvailableLeft = true, + AboveModeInfo = CreateModeInfo(predicted: true), + LeftModeInfo = CreateModeInfo(predicted: false) + }; + + const int predictionContext = 1; + using Av1SymbolWriter writer = new(Configuration.Default, 1, updateCdf: true); + writer.WriteSymbol(true, Av1DefaultDistributions.SegmentIdPredicted[predictionContext]); + using IMemoryOwner encoded = writer.Exit(); + Av1SymbolDecoder decoder = new(Configuration.Default, encoded.GetSpan(), 0, updateCdf: true); + + tileReader.ReadInterSegmentId(ref decoder, ref partitionInfo, beforeSkip: segmentIdPrecedesSkip); + + Assert.True(modeInfo.SegmentIdPredicted); + Assert.Equal(2, modeInfo.SegmentId); + for (int row = blockPosition.Y; row < blockPosition.Y + modeInfo.BlockSize.Get4x4HighCount(); row++) + { + for (int column = blockPosition.X; column < blockPosition.X + modeInfo.BlockSize.Get4x4WideCount(); column++) + { + Assert.Equal(2, tileReader.FrameInfo.GetSegmentId(row, column)); + } + } + } + + /// + /// Verifies that a skipped inter block uses the spatial predictor without reading a temporal-prediction symbol. + /// + [Fact] + public void SkippedInterBlockClearsTemporalPredictionAndUsesSpatialSegment() + { + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(64, 64); + ObuFrameHeader frameHeader = CreateFrameHeader(16, 16, segmentationUpdateMap: 1, segmentationTemporalUpdate: 1); + using Av1TileReader tileReader = new(Configuration.Default, sequenceHeader, frameHeader); + Point blockPosition = new(2, 2); + + // Three equal spatial neighbors select segment three without consuming a spatial segment symbol. The block is + // initialized as predicted to prove that the normative skipped-block branch explicitly clears the stale flag. + tileReader.FrameInfo.SetSegmentId(Av1BlockSize.Block4x4, new Point(1, 1), 3); + tileReader.FrameInfo.SetSegmentId(Av1BlockSize.Block4x4, new Point(2, 1), 3); + tileReader.FrameInfo.SetSegmentId(Av1BlockSize.Block4x4, new Point(1, 2), 3); + Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block8x8, blockPosition) + { + Skip = true, + SegmentIdPredicted = true + }; + + Av1SuperblockInfo superblockInfo = new(tileReader.FrameInfo, Point.Empty); + Av1PartitionInfo partitionInfo = new(modeInfo, superblockInfo, false, Av1PartitionType.None) + { + ColumnIndex = blockPosition.X, + RowIndex = blockPosition.Y, + AvailableAbove = true, + AvailableLeft = true + }; + + using Av1SymbolWriter writer = new(Configuration.Default, 1, updateCdf: true); + using IMemoryOwner encoded = writer.Exit(); + Av1SymbolDecoder decoder = new(Configuration.Default, encoded.GetSpan(), 0, updateCdf: true); + + tileReader.ReadInterSegmentId(ref decoder, ref partitionInfo, beforeSkip: false); + + Assert.False(modeInfo.SegmentIdPredicted); + Assert.Equal(3, modeInfo.SegmentId); + Assert.Equal(3, tileReader.FrameInfo.GetSegmentId(blockPosition.Y, blockPosition.X)); + } + + /// + /// Verifies that retained segmentation maps with different mode-info geometry are unavailable for temporal prediction. + /// + [Fact] + public void PredictedSegmentIdIsZeroForMismatchedPrimaryGeometry() + { + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(64, 64); + ObuFrameHeader currentHeader = CreateFrameHeader(16, 16, segmentationUpdateMap: 1, segmentationTemporalUpdate: 1); + ObuFrameHeader primaryHeader = CreateFrameHeader(8, 16, segmentationUpdateMap: 1, segmentationTemporalUpdate: 0); + Av1FrameInfo currentFrameInfo = new(sequenceHeader); + Av1FrameInfo primaryFrameInfo = new(sequenceHeader); + currentFrameInfo.InitializeSegmentIds(currentHeader, null); + primaryFrameInfo.InitializeSegmentIds(primaryHeader, null); + primaryFrameInfo.SetSegmentId(Av1BlockSize.Block32x64, Point.Empty, 5); + + int actual = currentFrameInfo.GetPredictedSegmentId(primaryFrameInfo, Av1BlockSize.Block16x16, Point.Empty); + + Assert.Equal(0, actual); + } + + /// + /// Creates block mode state with the requested temporal segment-prediction flag. + /// + /// Whether the block selected its segment identifier from the retained map. + /// The initialized block mode state. + private static Av1BlockModeInfo CreateModeInfo(bool predicted) + => new(Av1BlockSize.Block4x4, Point.Empty) { SegmentIdPredicted = predicted }; + + /// + /// Creates the fixed 64x64-superblock sequence geometry used by segmentation-map tests. + /// + /// The maximum coded width in pixels. + /// The maximum coded height in pixels. + /// The initialized monochrome sequence header. + private static ObuSequenceHeader CreateSequenceHeader(int width, int height) + => new() + { + MaxFrameWidth = width, + MaxFrameHeight = height, + Use128x128Superblock = false, + ColorConfig = new ObuColorConfig + { + IsMonochrome = true, + BitDepth = Av1BitDepth.EightBit + } + }; + + /// + /// Creates the frame geometry and segmentation controls used by direct map tests. + /// + /// The active width in 4x4 mode-info units. + /// The active height in 4x4 mode-info units. + /// Whether the frame updates its segment map. + /// Whether map updates may select the retained primary map. + /// The initialized frame header. + private static ObuFrameHeader CreateFrameHeader( + int modeInfoColumnCount, + int modeInfoRowCount, + int segmentationUpdateMap, + int segmentationTemporalUpdate) + => new() + { + ModeInfoColumnCount = modeInfoColumnCount, + ModeInfoRowCount = modeInfoRowCount, + SegmentationParameters = new ObuSegmentationParameters + { + Enabled = true, + LastActiveSegmentId = Av1Constants.MaxSegmentCount - 1, + SegmentationUpdateMap = segmentationUpdateMap, + SegmentationTemporalUpdate = segmentationTemporalUpdate + } + }; +} diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/ObuFrameLifecycleTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/ObuFrameLifecycleTests.cs index 699066d70..e53139a4b 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/ObuFrameLifecycleTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/ObuFrameLifecycleTests.cs @@ -41,6 +41,7 @@ public class ObuFrameLifecycleTests // spatial layers, so the lifecycle test exercises real progressive item framing. private static ReadOnlySpan ProgressiveTwoFrameObuStream => [ + // Temporal delimiter and progressive sequence header. 0x12, 0x00, 0x0A, 0x0F, 0x20, 0x13, 0x01, 0x00, 0x80, 0x81, 0x4E, 0x0A, 0x36, 0xBE, 0x48, 0x08, 0x20, 0x34, 0x80, diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/ObuSkipModeParametersTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/ObuSkipModeParametersTests.cs new file mode 100644 index 000000000..0511b615a --- /dev/null +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/ObuSkipModeParametersTests.cs @@ -0,0 +1,246 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using SixLabors.ImageSharp.Formats.Heif.Av1; +using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit; +using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; + +namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; + +/// +/// Verifies the frame-level derivation of AV1 skip-mode reference pairs. +/// +[Trait("Format", "Avif")] +public class ObuSkipModeParametersTests +{ + /// + /// The public theory-data representation of . + /// + private const int KeyFrameValue = (int)ObuFrameType.KeyFrame; + + /// + /// The public theory-data representation of . + /// + private const int InterFrameValue = (int)ObuFrameType.InterFrame; + + /// + /// The public theory-data representation of . + /// + private const int SingleReferenceValue = (int)ObuReferenceMode.SingleReference; + + /// + /// The public theory-data representation of . + /// + private const int ReferenceModeSelectValue = (int)ObuReferenceMode.ReferenceModeSelect; + + /// + /// Verifies signed order-hint distances across the modulo-domain boundary. + /// + [Fact] + public void GetRelativeDistanceWrapsWithinConfiguredDomain() + { + ObuOrderHintInfo orderHintInfo = CreateOrderHintInfo(); + + Assert.Equal(-2, orderHintInfo.GetRelativeDistance(15, 1)); + Assert.Equal(2, orderHintInfo.GetRelativeDistance(1, 15)); + } + + /// + /// Verifies that disabled order hints have no temporal ordering. + /// + [Fact] + public void GetRelativeDistanceReturnsZeroWhenOrderHintsAreDisabled() + { + ObuOrderHintInfo orderHintInfo = new(); + + Assert.Equal(0, orderHintInfo.GetRelativeDistance(15, 1)); + } + + /// + /// Verifies that skip mode selects the nearest past and future canonical reference roles. + /// + [Fact] + public void DeriveSelectsNearestForwardAndBackwardReferences() + { + ObuOrderHintInfo orderHintInfo = CreateOrderHintInfo(); + ObuFrameHeader frameHeader = CreateInterFrame(8, [7, 3, 6, 2, 10, 12, 15]); + + frameHeader.SkipModeParameters.Derive(orderHintInfo, frameHeader); + + Assert.True(frameHeader.SkipModeParameters.SkipModeAllowed); + Assert.Equal(Av1ReferenceFrameType.Last, frameHeader.SkipModeParameters.FirstReferenceFrame); + Assert.Equal(Av1ReferenceFrameType.Backward, frameHeader.SkipModeParameters.SecondReferenceFrame); + } + + /// + /// Verifies that the derived pair identifies canonical roles rather than their physical reference-map slots. + /// + [Fact] + public void DeriveOrdersCanonicalRolesIndependentlyOfMappedSlots() + { + ObuOrderHintInfo orderHintInfo = CreateOrderHintInfo(); + ObuFrameHeader frameHeader = CreateInterFrame(8, [7, 3, 6, 2, 10, 12, 15]); + Span referenceFrameIndices = frameHeader.GetReferenceFrameIndices(); + Span referenceOrderHints = frameHeader.GetReferenceOrderHints(); + + // Several canonical roles deliberately share physical slot seven. The first matching role remains LAST, while + // the future BWDREF role maps to slot four; neither physical slot number becomes part of the derived pair. + referenceFrameIndices.Fill(7); + referenceFrameIndices[(int)Av1ReferenceFrameType.Backward - 1] = 4; + referenceOrderHints[7] = 7; + + frameHeader.SkipModeParameters.Derive(orderHintInfo, frameHeader); + + Assert.True(frameHeader.SkipModeParameters.SkipModeAllowed); + Assert.Equal(Av1ReferenceFrameType.Last, frameHeader.SkipModeParameters.FirstReferenceFrame); + Assert.Equal(Av1ReferenceFrameType.Backward, frameHeader.SkipModeParameters.SecondReferenceFrame); + } + + /// + /// Verifies that a frame with only future references cannot use skip mode. + /// + [Fact] + public void DeriveDisallowsSkipModeWithoutForwardReference() + { + ObuOrderHintInfo orderHintInfo = CreateOrderHintInfo(); + ObuFrameHeader frameHeader = CreateInterFrame(8, [9, 10, 11, 12, 13, 14, 15]); + + frameHeader.SkipModeParameters.Derive(orderHintInfo, frameHeader); + + Assert.False(frameHeader.SkipModeParameters.SkipModeAllowed); + Assert.Equal(Av1ReferenceFrameType.None, frameHeader.SkipModeParameters.FirstReferenceFrame); + Assert.Equal(Av1ReferenceFrameType.None, frameHeader.SkipModeParameters.SecondReferenceFrame); + } + + /// + /// Verifies that a forward-only frame selects the two closest distinct past reference orders. + /// + [Fact] + public void DeriveSelectsTwoNearestForwardReferencesWhenNoBackwardReferenceExists() + { + ObuOrderHintInfo orderHintInfo = CreateOrderHintInfo(); + ObuFrameHeader frameHeader = CreateInterFrame(8, [7, 3, 6, 2, 1, 5, 4]); + + frameHeader.SkipModeParameters.Derive(orderHintInfo, frameHeader); + + Assert.True(frameHeader.SkipModeParameters.SkipModeAllowed); + Assert.Equal(Av1ReferenceFrameType.Last, frameHeader.SkipModeParameters.FirstReferenceFrame); + Assert.Equal(Av1ReferenceFrameType.Last3, frameHeader.SkipModeParameters.SecondReferenceFrame); + } + + /// + /// Verifies that modulo wraparound participates in nearest-reference selection. + /// + [Fact] + public void DeriveSelectsReferencesAcrossOrderHintWraparound() + { + ObuOrderHintInfo orderHintInfo = CreateOrderHintInfo(); + ObuFrameHeader frameHeader = CreateInterFrame(1, [12, 15, 11, 10, 2, 5, 7]); + + frameHeader.SkipModeParameters.Derive(orderHintInfo, frameHeader); + + Assert.True(frameHeader.SkipModeParameters.SkipModeAllowed); + Assert.Equal(Av1ReferenceFrameType.Last2, frameHeader.SkipModeParameters.FirstReferenceFrame); + Assert.Equal(Av1ReferenceFrameType.Backward, frameHeader.SkipModeParameters.SecondReferenceFrame); + } + + /// + /// Verifies that skip mode remains unavailable without two temporally distinct usable reference orders. + /// + [Fact] + public void DeriveDisallowsSkipModeWithoutReferencePair() + { + ObuOrderHintInfo orderHintInfo = CreateOrderHintInfo(); + ObuFrameHeader frameHeader = CreateInterFrame(8, [7, 8, 8, 8, 8, 8, 8]); + + frameHeader.SkipModeParameters.Derive(orderHintInfo, frameHeader); + + Assert.False(frameHeader.SkipModeParameters.SkipModeAllowed); + Assert.Equal(Av1ReferenceFrameType.None, frameHeader.SkipModeParameters.FirstReferenceFrame); + Assert.Equal(Av1ReferenceFrameType.None, frameHeader.SkipModeParameters.SecondReferenceFrame); + } + + /// + /// Verifies that deriving an ineligible frame clears a reference pair retained by an earlier derivation. + /// + [Fact] + public void DeriveClearsPreviousReferencePair() + { + ObuOrderHintInfo orderHintInfo = CreateOrderHintInfo(); + ObuFrameHeader frameHeader = CreateInterFrame(8, [7, 3, 6, 2, 10, 12, 15]); + frameHeader.SkipModeParameters.Derive(orderHintInfo, frameHeader); + + frameHeader.ReferenceMode = ObuReferenceMode.SingleReference; + frameHeader.SkipModeParameters.Derive(orderHintInfo, frameHeader); + + Assert.False(frameHeader.SkipModeParameters.SkipModeAllowed); + Assert.Equal(Av1ReferenceFrameType.None, frameHeader.SkipModeParameters.FirstReferenceFrame); + Assert.Equal(Av1ReferenceFrameType.None, frameHeader.SkipModeParameters.SecondReferenceFrame); + } + + /// + /// Verifies the frame modes for which the AV1 syntax forbids skip-mode signaling. + /// + /// Whether the sequence enables order hints. + /// The numeric coded-frame-type value. + /// The numeric frame-level reference-mode value. + [Theory] + [InlineData(false, InterFrameValue, ReferenceModeSelectValue)] + [InlineData(true, KeyFrameValue, ReferenceModeSelectValue)] + [InlineData(true, InterFrameValue, SingleReferenceValue)] + public void DeriveDisallowsSkipModeForIneligibleFrameSyntax( + bool enableOrderHint, + int frameTypeValue, + int referenceModeValue) + { + ObuOrderHintInfo orderHintInfo = CreateOrderHintInfo(); + orderHintInfo.EnableOrderHint = enableOrderHint; + ObuFrameHeader frameHeader = CreateInterFrame(8, [7, 3, 6, 2, 10, 12, 15]); + frameHeader.FrameType = (ObuFrameType)frameTypeValue; + frameHeader.ReferenceMode = (ObuReferenceMode)referenceModeValue; + + frameHeader.SkipModeParameters.Derive(orderHintInfo, frameHeader); + + Assert.False(frameHeader.SkipModeParameters.SkipModeAllowed); + Assert.Equal(Av1ReferenceFrameType.None, frameHeader.SkipModeParameters.FirstReferenceFrame); + Assert.Equal(Av1ReferenceFrameType.None, frameHeader.SkipModeParameters.SecondReferenceFrame); + } + + /// + /// Creates the four-bit modulo order-hint configuration used by the derivation scenarios. + /// + /// The enabled order-hint configuration. + private static ObuOrderHintInfo CreateOrderHintInfo() + => new() + { + EnableOrderHint = true, + OrderHintBits = 4, + }; + + /// + /// Creates an inter frame whose seven canonical roles map directly to slots zero through six. + /// + /// The current frame order hint. + /// The order hint selected by each canonical role. + /// The initialized inter-frame header. + private static ObuFrameHeader CreateInterFrame(uint currentOrderHint, ReadOnlySpan referenceOrderHints) + { + ObuFrameHeader frameHeader = new() + { + FrameType = ObuFrameType.InterFrame, + OrderHint = currentOrderHint, + ReferenceMode = ObuReferenceMode.ReferenceModeSelect, + }; + + Span referenceFrameIndices = frameHeader.GetReferenceFrameIndices(); + Span referenceMapOrderHints = frameHeader.GetReferenceOrderHints(); + + for (int referenceIndex = 0; referenceIndex < Av1Constants.ReferencesPerFrame; referenceIndex++) + { + referenceFrameIndices[referenceIndex] = (uint)referenceIndex; + referenceMapOrderHints[referenceIndex] = referenceOrderHints[referenceIndex]; + } + + return frameHeader; + } +}