From 8b9aa4b2cfe59ff0f8c348d9bb06678a5ff12f2a Mon Sep 17 00:00:00 2001 From: James Jackson-South Date: Fri, 28 Aug 2026 23:44:01 +1000 Subject: [PATCH] Complete AV1 single-reference inter decoding --- HEIF_IMPLEMENTATION_PLAN.md | 59 +- .../Heif/Av1/Av1BlockSizeExtensions.cs | 12 + src/ImageSharp/Formats/Heif/Av1/Av1Decoder.cs | 94 +- .../Formats/Heif/Av1/Av1FrameBuffer.cs | 53 +- .../Av1/Entropy/Av1DefaultDistributions.cs | 121 +++ .../Entropy/Av1DisplacementVectorContext.cs | 228 ----- .../Av1/Entropy/Av1FrameEntropyContext.cs | 105 +- .../Av1/Entropy/Av1MotionVectorContext.cs | 319 ++++++ .../Av1/Entropy/Av1SymbolContextHelper.cs | 361 +++++++ .../Heif/Av1/Entropy/Av1SymbolDecoder.cs | 176 +++- .../Heif/Av1/Entropy/Av1SymbolEncoder.cs | 2 +- .../Av1/Motion/Av1GlobalMotionParameters.cs | 60 ++ .../Heif/Av1/Motion/Av1IntraBlockCopy.cs | 54 +- .../Motion/Av1MotionVariationCandidates.cs | 297 ++++++ .../Heif/Av1/Motion/Av1MotionVector.cs | 138 +++ .../Av1/Motion/Av1MotionVectorPrecision.cs | 25 + .../Av1/Motion/Av1ReferenceMotionVectors.cs | 906 ++++++++++++++++++ .../Heif/Av1/OpenBitstreamUnit/ObuReader.cs | 7 +- .../Heif/Av1/Pipeline/Av1FrameDecoder.cs | 25 +- .../Av1/ReferenceFrames/Av1ReferenceFrame.cs | 12 +- .../Heif/Av1/Tiling/Av1BlockModeInfo.cs | 2 +- .../Av1/Tiling/Av1FrameInfo.MotionField.cs | 203 ++-- .../Formats/Heif/Av1/Tiling/Av1FrameInfo.cs | 2 +- .../Heif/Av1/Tiling/Av1PartitionInfo.cs | 59 ++ .../Formats/Heif/Av1/Tiling/Av1TileReader.cs | 332 ++++++- .../Heif/Av1/Transform/Av1BlockDecoder.cs | 147 ++- .../Formats/Heif/Av1HeifItemDecoder.cs | 5 +- .../Formats/Heif/Av1/Av1FrameBufferTests.cs | 33 +- .../Av1/Av1GlobalMotionParametersTests.cs | 85 ++ .../Heif/Av1/Av1InterFrameModeInfoTests.cs | 89 +- .../Heif/Av1/Av1InterIntraEntropyTests.cs | 170 ++++ .../Heif/Av1/Av1InterModeEntropyTests.cs | 223 +++++ .../Av1/Av1InterpolationFilterEntropyTests.cs | 314 ++++++ .../Heif/Av1/Av1MotionModeEntropyTests.cs | 223 +++++ .../Heif/Av1/Av1MotionModeInfoTests.cs | 206 ++++ .../Av1/Av1MotionVariationCandidatesTests.cs | 392 ++++++++ .../Heif/Av1/Av1MotionVectorEntropyTests.cs | 345 +++++++ .../Formats/Heif/Av1/Av1MotionVectorTests.cs | 158 +++ .../Av1/Av1ReconstructionConformanceTests.cs | 209 ++++ .../Heif/Av1/Av1ReferenceFrameStoreTests.cs | 199 +++- .../Av1/Av1ReferenceMotionVectorsTests.cs | 498 ++++++++++ .../Av1/Av1SingleReferenceEntropyTests.cs | 330 +++++++ tests/ImageSharp.Tests/TestImages.cs | 4 + .../Input/Heif/Av1/Conformance/README.md | 149 +-- ...-progressive-draw-points-8b-libaom-y4m.yuv | 3 + .../libavif-progressive-draw-points-8b.avif | 3 + .../libavif-progressive-draw-points-8b.bit | 3 + .../libavif-progressive-draw-points-8b.png | 3 + 48 files changed, 6920 insertions(+), 523 deletions(-) delete mode 100644 src/ImageSharp/Formats/Heif/Av1/Entropy/Av1DisplacementVectorContext.cs create mode 100644 src/ImageSharp/Formats/Heif/Av1/Entropy/Av1MotionVectorContext.cs create mode 100644 src/ImageSharp/Formats/Heif/Av1/Motion/Av1MotionVariationCandidates.cs create mode 100644 src/ImageSharp/Formats/Heif/Av1/Motion/Av1MotionVectorPrecision.cs create mode 100644 src/ImageSharp/Formats/Heif/Av1/Motion/Av1ReferenceMotionVectors.cs create mode 100644 tests/ImageSharp.Tests/Formats/Heif/Av1/Av1GlobalMotionParametersTests.cs create mode 100644 tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterIntraEntropyTests.cs create mode 100644 tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterModeEntropyTests.cs create mode 100644 tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterpolationFilterEntropyTests.cs create mode 100644 tests/ImageSharp.Tests/Formats/Heif/Av1/Av1MotionModeEntropyTests.cs create mode 100644 tests/ImageSharp.Tests/Formats/Heif/Av1/Av1MotionModeInfoTests.cs create mode 100644 tests/ImageSharp.Tests/Formats/Heif/Av1/Av1MotionVariationCandidatesTests.cs create mode 100644 tests/ImageSharp.Tests/Formats/Heif/Av1/Av1MotionVectorEntropyTests.cs create mode 100644 tests/ImageSharp.Tests/Formats/Heif/Av1/Av1MotionVectorTests.cs create mode 100644 tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReferenceMotionVectorsTests.cs create mode 100644 tests/ImageSharp.Tests/Formats/Heif/Av1/Av1SingleReferenceEntropyTests.cs create mode 100644 tests/Images/Input/Heif/Av1/Conformance/libavif-progressive-draw-points-8b-libaom-y4m.yuv create mode 100644 tests/Images/Input/Heif/Av1/Conformance/libavif-progressive-draw-points-8b.avif create mode 100644 tests/Images/Input/Heif/Av1/Conformance/libavif-progressive-draw-points-8b.bit create mode 100644 tests/Images/Input/Heif/Av1/Conformance/libavif-progressive-draw-points-8b.png diff --git a/HEIF_IMPLEMENTATION_PLAN.md b/HEIF_IMPLEMENTATION_PLAN.md index 099774d55..ceabda57b 100644 --- a/HEIF_IMPLEMENTATION_PLAN.md +++ b/HEIF_IMPLEMENTATION_PLAN.md @@ -29,24 +29,25 @@ Checkboxes may be marked complete only when the implementation and the verificat ## Delivery dashboard -Last reconciled with the source tree on 2026-08-27 against the worktree based on commit `f6d3da2b31`, including the completed AV1 transform, OBU-framing, intra-block-copy, 12-profile reconstruction, layered-item property, and layered reference/header-state checkpoints. This dashboard is the authoritative delivery order. The detailed phase checklists below provide subsystem evidence; they do not override the current-stage marker or permit work to skip ahead. +Last reconciled with the source tree on 2026-08-28 against production checkpoint `096fb9af8` and the uncommitted single-reference inter-decoding work identified below. Committed checkpoints include the AV1 transform architecture, OBU framing, intra-block copy, 12-profile reconstruction matrix, layered-item properties, layered reference/header state, inter-frame intra blocks, SIMD-first translational prediction, and normalized self-guided-filter dispatch. The progressive fixture audit found that the initial `.bit` file appended a physically adjacent auxiliary-alpha extent instead of the color item's second `iloc` extent. The corrected 55-plus-17-byte logical color payload decodes as two frames with exact pinned libaom output, and the three production-path progressive tests now pass for native planes, final libavif presentation, constrained allocation, and decoder-result ownership. The complete focused Release matrix, documentation review, and final diff checks remain open. This dashboard is the authoritative delivery order. The detailed phase checklists below provide subsystem evidence; they do not override the current-stage marker or permit work to skip ahead. Status meanings: - **Complete:** the implementation and its phase exit evidence are recorded. - **In progress:** usable implementation exists, but one or more required behaviors or verification gates remain open. +- **Implemented locally; unverified:** source exists in the working tree, but its checkbox remains open until the required Release and independent evidence pass. - **Not started:** supporting primitives may exist, but the production format path is absent. - **Current:** the only work item that should be advanced before taking the next queued item. -Current development stage: **Stage 3 — complete AV1 still-image decoding.** The transform checkpoint is closed. Layered decoding now retains reference/header/CDF/motion-field state, derives frame-level skip-mode references, consumes temporal segment prediction, and decodes intra-coded blocks inside inter frames. True inter-coded blocks still stop before reference/MV parsing and reconstruction. Neither AV1 nor HEVC production encoding is implemented. +Current development stage: **Stage 3 — complete AV1 still-image decoding.** The committed decoder retains reference/header/CDF/motion-field state, derives frame-level skip-mode references, consumes temporal segment prediction, and decodes intra-coded blocks inside inter frames. The uncommitted working tree adds single-reference selection, spatial and temporal reference-MV derivation, NEAREST/NEAR/NEW/GLOBAL mode parsing, DRL, interpolation-filter syntax, motion-mode eligibility, and simple translational reconstruction before residual traversal. The exact dependent-frame oracle, constrained-allocation path, and decoder-result motion-field ownership test pass in Release; the complete focused Release matrix and final source/documentation review remain open. Compound prediction, inter-intra, OBMC, warped motion, scaled references, and non-translational global prediction remain explicitly unsupported. Neither AV1 nor HEVC production encoding is implemented. -Immediate checkpoint: **complete layered AV1 image-item decoding through the existing image-only container surface.** This includes `a1op`, `lsel`, and `a1lx` properties, operating-point selection, dependency-preserving layer consumption, and final or explicitly selected spatial-layer output for color, alpha, and grid items. The bounded decoder session now retains reference owners and the header, entropy, segmentation, loop-filter, global-motion, and temporal motion-field state required by dependent layers. Inter tile syntax and reconstruction still need to consume that state. This work must not be represented as animation or expanded into a general ISO BMFF/video model. +Immediate checkpoint: **finish the source audit, independently verify, document, and commit the existing single-reference AV1 inter-block slice before implementing another codec feature.** Pinned libaom source confirms that interpolation syntax is omitted for an identity `GLOBALMV` block of sufficient size and that the spatial single-reference extension loops stop at two candidates; the current predicates match those two call paths. The corrected logical color payload and two-frame YUV444-alpha reference now prove the exact final native planes and libavif presentation through the complete production decoder, including constrained frame-plane allocation and decoder-result motion-field ownership. The complete focused Release verification and final source/documentation review remain required. This work remains inside the existing image-item and bounded image-sequence surfaces and must not expand into a general ISO BMFF/video model. | Order | Delivery stage | State | Delivered state | Gate that remains open | | --- | --- | --- | --- | --- | | 1 | Baseline, provenance, documentation, and public contract | In progress | Pinned codec references, a bounded image-only scope, encoder options, typed bit depth, decoder-option propagation, and extensive HEIF documentation exist. | Complete the all-file documentation audit, record a fresh Release baseline, finish distinct public HEIC/AVIF save boundaries, and close API review. | | 2 | Bounded HEIF item and image-sequence container | In progress | Still-item parsing, grids, auxiliary alpha, metadata properties, bounded image-sequence tracks, Identify, and all-sync AV1 sequence presentation are connected. | Complete adversarial boundary coverage, remaining item/property behavior, reference-dependent sequence reconstruction, and the bounded sequence writer. | -| 3 | Still-image AV1 and HEVC decoding | **Current** | HEVC reconstruction reaches exact HM/libheif fixtures across the recorded 8/10/12-bit and chroma cases. AV1 includes bounded OBU framing, reconstruction, filters, grain, color, transforms, intra-block copy, an exact independent 12-profile bit-depth/chroma matrix through every dispatch tier, retained layered reference/header/CDF state, temporal segment prediction, and the inter-frame intra-coded-block branch. | Complete true inter tile syntax and reconstruction, including reference/MV, compound, global, and warped prediction; remove every other valid AV1 still-image unsupported branch with independent compression-tool vectors; then complete the remaining HEVC profile and Range Extensions matrix. | +| 3 | Still-image AV1 and HEVC decoding | **Current** | HEVC reconstruction reaches exact HM/libheif fixtures across the recorded 8/10/12-bit and chroma cases. Committed AV1 work includes bounded OBU framing, reconstruction, filters, grain, color, transforms, intra-block copy, an exact independent 12-profile bit-depth/chroma matrix through every dispatch tier, retained layered reference/header/CDF state, temporal segment prediction, inter-frame intra blocks, and SIMD-first translational prediction. The working tree contains an unverified simple single-reference inter path. | Close the simple single-reference checkpoint with exact dependent-frame evidence; implement compound, inter-intra, OBMC, scaled-reference, warped, and non-translational global prediction; remove every other valid AV1 still-image unsupported branch with independent vectors; then complete the remaining HEVC profile and Range Extensions matrix. | | 4 | Complete decoded presentation and animation | In progress | Shared SIMD-first AV1/HEVC color conversion, ICC application, grids, transforms, direct planar alpha composition, frame metadata, repetition, and independently decodable AV1 sequence samples exist. | Close the full color/ICC cross-product, HEVC sequence decoding, AV1/HEVC reference-dependent samples, frame-local metadata/alpha behavior, and independent animated decode vectors. | | 5 | AV1/AVIF encoding | Not started | RGB-to-planar conversion, forward transforms, OBU writer foundations, options, and container-writing infrastructure exist. | `HeifEncoderCore` still rejects AV1. Implement a real independently decodable lossy/lossless AV1 payload and the complete AVIF item/metadata matrix. | | 6 | HEVC/HEIC encoding | Not started | Shared input color conversion, options, and HEIF writer infrastructure exist. | `HeifEncoderCore` still rejects HEVC. Implement a real independently decodable lossy/lossless HEVC payload and the complete HEIC item/metadata matrix. | @@ -58,7 +59,7 @@ Immediate checkpoint: **complete layered AV1 image-item decoding through the exi - [x] Finish the libaom-shaped AV1 forward-transform architecture, measured production dispatch, inverse-tier correction, suffix cleanup, `FeatureTestRunner` matrix, and focused Release verification recorded below. - [x] Close the base AV1 profile matrix with exact native-plane and presented-image comparisons for 8/10/12-bit monochrome, 4:2:0, 4:2:2, and 4:4:4 fixtures under normal, AVX2, 128-bit, and scalar dispatch. - [x] Accept the AV1-ISOBMFF final low-overhead OBU form that omits its payload-size field and uses the bounded image-item remainder; focused Release coverage reconstructs a valid combined frame in that form. -- [ ] **Current:** implement layered AV1 image-item properties and stateful dependency reconstruction, then verify default final-layer output against the two pinned libavif progressive fixtures. +- [ ] **In progress:** complete layered AV1 image-item dependency reconstruction and verify default final-layer output against the pinned libavif progressive fixtures. - [x] Parse and associate `a1op`, `lsel`, and `a1lx` through the bounded image-item property model, including normative essential flags, duplicate handling, exact property lengths, and the four-layer limit. - [x] Validate `a1lx` layer boundaries against the logical item size and restrict concrete `lsel` decoding to the cumulative payload through the selected spatial layer without copying item bytes. - [x] Apply the selected `a1op` operating-point mask while consuming extended OBUs and validate the selected index against the parsed sequence header. @@ -72,8 +73,21 @@ Immediate checkpoint: **complete layered AV1 image-item decoding through the exi - [ ] Implement the complete inter-frame entropy, mode, motion-vector, compound-prediction, inter-prediction, and warped/global-motion paths permitted by the image profile. - [x] Derive the frame-level skip-mode reference pair from mapped order hints, including modulo wraparound and the two-forward fallback, then decode the common inter-frame block prefix and intra-coded-block branch with retained CDF state and block-size luma contexts. - [x] Implement allocation-free SIMD-first translational single-reference interpolation for regular, smooth, sharp, and bilinear filters across 8/10/12-bit samples. The predictor mirrors JPEG's closed static operator architecture, descends through `Vector512`, `Vector256`, and `Vector128` before scalar fallback, and passes the exact independent convolution oracle through `FeatureTestRunner`. - - [ ] **Current:** decode single-reference inter selection, the spatial/temporal reference-MV stack, NEAREST/NEAR/NEW/GLOBAL motion modes, DRL and interpolation filters, then reconstruct the complete block once through the existing SIMD-first translational predictor before residual traversal. - - [ ] Decode compound and inter-intra modes, masked blending, OBMC, scaled references, and warped/global-motion prediction without changing the single-reference predictor contract or rounding model. + - [x] Close the single-reference inter path before advancing to another mode. + - [x] Decode single-reference selection and own the exact adaptive reference and inter-mode distributions in the frame entropy context. + - [x] Derive the fixed-capacity spatial and temporal reference-MV stack with normative ordering, precision, clamping, global fallback, and DRL selection without per-block allocation. + - [x] Audit the disputed interpolation omission against pinned libaom `03087864cf4bea6abb0d28f95cf7843511413d8f`: `av1_is_interp_needed()` calls `is_nontrans_global_motion()`, whose global-motion-type test returns false only for `TRANSLATION`. An identity `GLOBALMV` block of sufficient size therefore omits switchable-filter symbols, matching the current `!= Translation` predicate. The separate `is_global_mv_block()` `> Translation` test governs a different global-motion classification. + - [x] Audit the disputed spatial extension bound against the same pinned libaom: both loops in `setup_ref_mv_list()` stop at `MAX_MV_REF_CANDIDATES`, which is two. `MAX_REF_MV_STACK_SIZE`, which is eight, is the total stack capacity used by other candidate paths. The current spatial-loop bound matches the pinned source. + - [x] Decode NEW motion-vector differences with the independent normal-motion-vector entropy context and validate the final component range. + - [x] Decode or infer both directional interpolation filters after finalized motion-mode syntax, including fixed, switchable, dual-filter, and no-symbol paths. + - [x] Reconstruct one complete single-reference block through the retained padded plane and existing SIMD-first predictor before transform-unit residual traversal. + - [x] Consume inter-intra and motion-mode syntax in normative order when those modes are not selected, including the exact `Block8x8` through `Block32x32` inter-intra enum interval and binary/ternary motion-mode CDF selection. + - [x] Use counted frame/reference ownership for allocator-owned retained and temporal motion fields, with allocation tracking for initialization, retained-slot aliases, failure unwinding, presentation ownership, decoder-result ownership, and exactly-once final disposal. + - [x] Request the existing contiguous ImageSharp allocation contract for every padded AV1 frame plane. Constrained-allocator coverage verifies complete-plane block reconstruction without copying or per-block allocation. + - [x] Complete source review of the new motion-mode and single-reference tests. + - [x] Pass the exact current tree's Release verification: `net10.0` and `net11.0` source builds and the `net10.0` test-project analyzer build complete with zero warnings and errors; 293 focused `net10.0` entropy, candidate, motion, interpolation, lifecycle, reconstruction, ownership, and `FeatureTestRunner` cases pass with zero failures or skips; and `git diff --check` is clean. + - [x] Correct the progressive dependent-frame extraction and compare the final frame's first three native planes with pinned libaom output and its final RGBA presentation with pinned libavif exactly. The unmodified AVIF has the recorded SHA-256 and stores the primary color item's 55-byte base extent at offset 511 and 17-byte dependent extent at offset 583. The corrected logical `.bit` payload decodes as two YUV444 frames with pinned libaom `03087864cf4bea6abb0d28f95cf7843511413d8f`; the retained two-frame YUV444-alpha reference and final PNG come from pinned libavif linked to that build. The production test selects the second native frame, requires inter-coded blocks, and passes exact native and presentation comparisons through `FeatureTestRunner`. + - [ ] Decode compound and inter-intra modes, masked blending, OBMC, scaled references, and warped and non-translational global-motion prediction without changing the single-reference predictor contract or rounding model. - [ ] Verify every connected mode and filter with independently encoded dependent-layer AV1 image-item fixtures and exact native-plane comparisons. - [ ] Return the explicitly selected spatial layer or the final displayed layer, keeping reference reconstruction separate from display-only film grain. - [ ] Verify color and auxiliary-alpha output exactly against both pinned libavif progressive fixtures under normal SIMD dispatch and all required `FeatureTestRunner` fallbacks. @@ -335,7 +349,7 @@ This table is intentionally incomplete. Add a row before each additional AV1 or ## Current implementation assessment -This assessment was reconciled with the source tree on 2026-08-26. Unless a result is stated explicitly, each item is a source-inspection finding rather than a verified interoperability claim. +This assessment was reconciled with the source tree on 2026-08-27, including production checkpoint `096fb9af8` and the explicitly identified uncommitted work. Unless a result is stated explicitly, each item is a source-inspection finding rather than a verified interoperability claim. ### Public integration @@ -373,6 +387,9 @@ This assessment was reconciled with the source tree on 2026-08-26. Unless a resu ### AV1 decoder - The bounded `Av1Decoder` session parses all tile state before allocating and reconstructing each coded image layer. After successful completion it retains the ungrained reference planes, frame header, frame information, and published entropy snapshot in the refreshed slots, while presentation-only ownership remains separate. A new accepted sequence header resets both parser and retained-owner state. This is dependency reconstruction within one bounded image item; `show_existing_frame` playback remains rejected and no animation/video reference model is exposed. +- Committed inter-frame support reaches the intra-coded-block branch and provides SIMD-first translational prediction. The uncommitted working tree additionally parses single-reference selection, builds the fixed-capacity spatial and temporal reference-MV stack, decodes NEAREST/NEAR/NEW/GLOBAL and DRL syntax, decodes or infers interpolation filters, checks inter-intra and motion-mode eligibility, and invokes simple translational prediction before residual reconstruction. The exact corrected dependent-frame fixture now passes native-plane and final-presentation comparisons through the production decoder, but the slice is not delivered until the complete focused Release matrix, final review, and checkpoint commit pass. +- Source inspection against pinned libaom `03087864cf4bea6abb0d28f95cf7843511413d8f` resolves two disputed audit claims in favor of the current predicates. The interpolation call path uses `is_nontrans_global_motion()`, which returns false only for `TRANSLATION`, so an identity `GLOBALMV` block omits filter symbols. The single-reference spatial extension loops use `MAX_MV_REF_CANDIDATES`, which is two, while the full reference-MV stack capacity is eight. These loops are spatial extension, not temporal extension. +- The remaining single-reference audit issues are concrete and open. The working tree now requests contiguous allocation for all padded AV1 frame planes and adds constrained-allocator coverage, but that contract has not passed the complete focused matrix. Motion fields use allocator-owned storage and counted leases, with new allocation tracking for aliases, success ownership, failure unwinding, and exactly-once disposal; those tests are also unverified. The supplied progressive fixture cannot reach the production inter branch: ImageSharp and exact pinned libaom both reject the enhancement frame's nonzero byte-alignment padding, while the supplied one-frame Y4M's first three planes exactly match the separately decoded base layer. - Transform coefficient entropy derivation and updates now address the above contexts relative to the tile column and the left contexts relative to the current superblock row, preserve luma coordinates independently of chroma subsampling, and test every packed context entry for the libaom any-nonzero rule. Extended vertical partition updates advance the mode-information column rather than the row. The existing multi-superblock 4:4:4 AVIF fixture now completes tile parsing; independent coefficient-context vectors across tile boundaries, chroma layouts, bit depths, and edge-clipped transforms remain required. - The reconstruction pipeline now records plane-relative transform geometry, preserves tile-local delta-Q and delta-LF predictors, derives segmentation and reference-adjusted filter levels, and runs the exact AV1 4-, 6-, 8-, and 14-tap deblocking kernels in normative vertical-then-horizontal order. Deblocking uses the same closed edge-operator architecture as the HEVC filter, with operators specialized by sample storage and orientation, `Vector128` lanes representing the four rows or columns along an edge, and an allocation-free scalar fallback for disabled intrinsics. Exact native-plane comparison with pinned scalar libaom output now verifies active deblocking and complete reconstruction for real 8-bit 4:2:0, 10-bit 4:4:4, and 12-bit 4:4:4 content; genuine AVIF containers separately verify presentation and public bit-depth metadata. The pipeline then applies CDEF through one semantic filter architecture: paired AVX2 and single-block `Vector128` direction analysis, closed primary/secondary strength operators, packed 4x4/4x8/8x4/8x8 constrained filtering, byte/16-bit output operators, and an exact allocation-free scalar fallback. Decoder orchestration now owns the immutable plane snapshots and clean direction/variance maps through ImageSharp's memory allocator, widens 8-bit source rows with the same AVX2/128-bit/scalar tiers as libaom, lists each unit's non-skipped blocks in fixed inline storage, analyzes listed blocks in pairs, and writes filtered bytes or 16-bit samples directly to the frame planes. Independently encoded active-CDEF 8-bit 4:2:0 and 10/12-bit 4:4:4 streams match every visible native sample produced by pinned scalar libaom with restoration disabled. Independently encoded AVIF containers at the same three bit depths also match pinned scalar-libavif presentation exactly under normal, 256-bit, 128-bit, and scalar color-conversion dispatch. Active super-resolution derives the Appendix A bounded coded width and applies the exact 64-phase, 8-tap horizontal filter with aligned reconstruction-edge input, 8/10/12-bit clipping, and the existing cross-platform `Vector128_.MultiplyAddAdjacent` helper. Independently encoded active-super-resolution AV1 streams at 8, 10, and 12 bits match every visible native sample produced by pinned scalar libaom under normal and forced-scalar dispatch. Independently packaged AVIF containers at the same bit depths retain matching libavif profile, dimensions, chroma, and CICP properties, require active super-resolution in their actual AV1 item, and match pinned scalar-libavif presentation exactly under normal, 256-bit, 128-bit, and scalar color-conversion dispatch. Loop restoration follows super-resolution, preserves the required pre-CDEF deblocked context at internal stripes, and applies decoded Wiener or self-guided units from immutable plane snapshots. Independently encoded active-restoration streams at 8, 10, and 12 bits now match every native sample from pinned scalar libaom across AVX2, 128-bit, and scalar dispatch, with the fixture matrix proving both Wiener and self-guided unit selection. The matching AVIF containers also match pinned scalar-libavif presentation exactly, and an independent direct-window definition verifies all sixteen self-guided parameter sets at each supported bit depth across vector and scalar dispatch. Combined 8-bit 4:2:0, 10-bit 4:2:2, and 12-bit 4:4:4 streams additionally verify restoration-unit boundaries after super-resolution, including clipped chroma transform traversal at a coded-frame edge. - The visible still-image path applies the complete self-contained film-grain parameter set after all in-loop filters. Independently encoded pinned-libaom vectors match every native sample at 8, 10, and 12 bits across monochrome, 4:2:0, 4:2:2, and 4:4:4 layouts, full and restricted ranges, identity-matrix signaling, overlap, and odd 33x11 frame extension. `FeatureTestRunner` verifies normal, AVX-disabled, and fully scalar dispatch. A full-HD-equivalent 4:2:0 benchmark reports zero allocation: 8-bit AVX2 is 2.335 ms versus 5.806 ms scalar, while 12-bit AVX2 is 3.195 ms, cross-platform 128-bit is 7.382 ms, and scalar is 8.614 ms on the measured Ryzen platform. The slower 8-bit 128-bit lookup construction is deliberately not dispatched. @@ -381,8 +398,8 @@ This assessment was reconciled with the source tree on 2026-08-26. Unless a resu - Loop-restoration unit parsing records tile-local switchable/Wiener/self-guided filter selections and coefficients in frame-owned plane grids, including super-resolution-adjusted unit corners and the corrected conditional 64x64-superblock unit-size bit. The active restoration stage implements the normative unit geometry, striped deblocked boundaries, Wiener filtering, self-guided projection, and 8/10/12-bit clipping. Self-guided filtering now follows libaom's summed-area-table, coefficient-grid, alternating-row radius-two, full radius-one, and projection stages through AVX2 and cross-platform 128-bit traversals with one exact scalar fallback and caller-owned scratch. Independently encoded fixtures cover active Wiener and self-guided reconstruction and exact AVIF presentation at every supported bit depth. An independent direct-window definition covers all sixteen self-guided parameter sets, narrow and odd processing units, both vector-tail widths, padded strides, and the scalar fallback. Combined active-restoration and super-resolution fixtures cover 4:2:0, 4:2:2, and 4:4:4 at 8, 10, and 12 bits, including restoration-unit boundaries and clipped chroma transform traversal. Other normative independently decodable still-image syntax paths still contain `NotImplementedException` or equivalent unsupported branches. Tile-local palette CDF adaptation is present; the remaining still-image frame-context behavior requires a separate source audit without introducing sequence playback state. - The frame buffer now establishes two-byte native sample storage, logical plane rows, and sample-unit block strides for 10/12-bit frames. The active intra-prediction, inverse-transform, and block-reconstruction path selects native 16-bit samples for 10/12-bit frames and has focused pipeline wiring coverage. Chroma-from-luma storage, 4:4:4/4:2:2/4:2:0 subsampling, rounded mean subtraction, U/V sharing, and 8/10/12-bit prediction now traverse AVX2 and cross-platform `Vector128` paths before an exact scalar fallback. `FeatureTestRunner` verifies every tier against independent fixed-point definitions across every supported block width; independently encoded high-bit-depth and chroma-from-luma AVIF conformance files are still required. - `Av1YuvConverter` now consumes the signaled full or limited range, every non-reserved AV1 H.273 matrix coefficient, transfer characteristics where the matrix definition requires them, subsampling, and chroma sample position for 8, 10, and 12-bit output. Its high-bit-depth decode and encode paths use allocator-backed `Rgb48` rows and the existing `PixelOperations` conversions, avoiding the former eight-bit intermediate. Encoder conversion covers monochrome, YUV 4:2:0, 4:2:2, and 4:4:4 with libavif-compatible box averaging. Identity, full/limited-range YCgCo, the fixed non-constant-luminance matrices, both fixed and chromaticity-derived constant/non-constant-luminance systems, SMPTE ST 2085, and PQ/HLG ICtCp are active in both directions. Independent vectors for every matrix, transfer, range, bit depth, sampling layout, and chroma position remain required before the complete color matrix is externally verified. -- Forward and inverse transforms use operation-owned workspace, explicit sequential fixed storage, and stateless static-generic operator structs for every valid DCT, ADST, and identity size. Named configuration factories keep the encoder's three shifts and variable cosine precision separate from the decoder's two shifts, fixed 12-bit cosine precision, and 8/10/12-bit clamp ranges. Forward traversal shares one libaom-shaped stage network across scalar, `Vector128`, `Vector256`, and `Vector512`; inverse production traversal uses the verified scalar, `Vector128`, and `Vector256` tiers. Lossless segments now bypass the DCT pipeline and apply the reversible four-by-four inverse Walsh-Hadamard transform through the same byte/high-bit-depth output operators, with a `Vector128` production path and caller-workspace scalar fallback. The focused Release matrix passes across the `FeatureTestRunner` hardware configurations, and the production transform benchmarks report zero allocation. -- Core intra prediction, chroma-from-luma, palette reconstruction, and nonlinear self-guided restoration now use SIMD-first operator or packed-kernel architectures with exact scalar fallbacks. Self-guided restoration keeps one semantic type while internal overloads select the libaom-shaped AVX2 or cross-platform 128-bit traversal; no namespace, file, or type name exposes SIMD width, ISA, storage, or bit depth. Transform traversal is already SIMD-first, while normative super-resolution and Wiener horizontal products reuse ImageSharp's cross-platform adjacent multiply/add SIMD helper with exact scalar fallbacks. +- Forward and inverse transforms use operation-owned workspace, explicit sequential fixed storage, and stateless static-generic operator structs for every valid DCT, ADST, and identity size. Named configuration factories keep the encoder's three shifts and variable cosine precision separate from the decoder's two shifts, fixed 12-bit cosine precision, and 8/10/12-bit clamp ranges. Forward traversal shares one libaom-shaped stage network across scalar, `Vector128`, `Vector256`, and `Vector512`; inverse production traversal uses the verified scalar, `Vector128`, and `Vector256` tiers. Lossless segments now bypass the DCT pipeline and apply the reversible four-by-four inverse Walsh-Hadamard transform through the same byte/high-bit-depth output operators, with a `Vector128` production path and caller-workspace scalar fallback. The focused Release matrix passes across the `FeatureTestRunner` hardware configurations, and the production transform benchmarks report zero allocation. The separately audited 12-bit inverse ADST4, Identity4, and Identity16 widening correction remains open, so the complete Phase 3 transform correctness gate is not closed. +- Core intra prediction, chroma-from-luma, palette reconstruction, and nonlinear self-guided restoration now use SIMD-first operator or packed-kernel architectures with exact scalar fallbacks. Self-guided restoration dispatches portably through `Vector256`, `Vector128`, and scalar tiers; AVX2-only gather and scan operations remain local fast paths within the 256-bit implementation. `FeatureTestRunner` verifies AVX2, portable AVX-only 256-bit, 128-bit, and scalar execution. No namespace, file, or type name exposes SIMD width, ISA, storage, or bit depth. Transform traversal is already SIMD-first, while normative super-resolution and Wiener horizontal products reuse ImageSharp's cross-platform adjacent multiply/add SIMD helper with exact scalar fallbacks. ### AV1 encoder @@ -401,7 +418,7 @@ This assessment was reconciled with the source tree on 2026-08-26. Unless a resu ### Tests - HEVC coverage includes exact native-plane comparison with HM output, exact complete-image comparison with pinned libheif/libde265 references, and the 10 official Sony GENERAL Range Extensions first-picture fixtures across 8/10/12-bit monochrome, 4:2:0, 4:2:2, and 4:4:4 reconstruction. The remaining exposed profiles and individual Range Extensions tools still need exact independent vectors. -- AV1 has focused bitstream, prediction, entropy, reconstruction, filter, film-grain, color, and transform coverage, plus real libavif inputs. A real two-layer libavif-derived OBU stream verifies the bounded frame lifecycle, retained-slot occupancy, resolved inter references, and `frame_size_with_refs` dimensions through a fake tile lifecycle. A real palette stream truncated inside its tile entropy payload verifies libaom-equivalent overflow/trailing-bit rejection and decoder-session recovery. The current Release checkpoint passes all 2,422 selected entropy, ownership, reference, predictor, intra-block-copy, and transform cases. It does not decode inter tile syntax or compare dependent-layer reconstructed pixels. Valid still-image syntax paths still contain explicit unsupported branches, so the independent AV1 decode matrix is not complete. +- AV1 has focused bitstream, prediction, entropy, reconstruction, filter, film-grain, color, and transform coverage, plus real libavif inputs. A real two-layer libavif-derived OBU stream verifies the bounded frame lifecycle, retained-slot occupancy, resolved inter references, and `frame_size_with_refs` dimensions through a fake tile lifecycle. A real palette stream truncated inside its tile entropy payload verifies libaom-equivalent overflow/trailing-bit rejection and decoder-session recovery. The last broad committed Release checkpoint passed all 2,422 selected entropy, ownership, reference, predictor, intra-block-copy, and transform cases. Commit `096fb9af8` separately passes the self-guided filter test under four `FeatureTestRunner` configurations. The 191-case `net10.0` result and zero-error `net10.0`/`net11.0` builds apply only to the pre-fixture tree. The exact current tree builds the focused `net10.0` test project, but its dependent-frame test fails before reconstruction; no current-tree claim inherits the older evidence. No test currently decodes a real dependent inter frame and compares its reconstructed pixels with libaom. Valid still-image syntax paths still contain explicit unsupported branches, so the independent AV1 decode matrix is not complete. - The AV1 transform matrix verifies scalar, `Vector128`, `Vector256`, and `Vector512` forward representations plus the production inverse tiers across every valid size/type combination and supported bit depth. All 511 focused forward and inverse cases pass in Release; `FeatureTestRunner` isolates hardware tiers, every two-dimensional configuration exercises production dispatch, and the complete-block benchmark records zero managed allocation. - Independent libavif fixtures cover primary, grid, auxiliary-alpha, ICC, metadata-skipping, and all-sync image-sequence presentation. Reference-dependent AV1 and HEVC sequence reconstruction and independent HEVC ICC sequence coverage remain open. - Focused decoder-option tests cover strict, ancillary-only, image-data, and metadata-skipping behavior for still items and sequence samples. Complete adversarial dimension, allocation, malformed-container, and resource-limit coverage remains open. @@ -535,8 +552,18 @@ Implement and verify in dependency order: - [x] Retain completed ungrained reference planes, frame headers, frame information, and published CDF snapshots in one eight-slot owner; apply refresh flags only after successful completion; resolve full and short reference signaling against occupancy and frame-ID validity; and implement primary-reference selection and `frame_size_with_refs`. - [x] Inherit primary-reference CDFs, segmentation features and unchanged maps, loop-filter deltas, and same-role global-motion parameters. Initialize the frame-owned per-8x8 temporal motion field and project eligible retained motion vectors in normative reference order. - [x] Implement allocation-free SIMD-first translational single-reference interpolation for regular, smooth, sharp, and bilinear filters; reduced four-sample kernels; horizontal, vertical, and separable two-dimensional convolution; exact AV1 rounding; 8/10/12-bit clipping; padded reference origins; and guarded destination strides. The operator contract and concrete operator files mirror JPEG color conversion, and `FeatureTestRunner` verifies normal, AVX-512-disabled, AVX-disabled, and scalar execution against an independent fixed-point oracle. - - [ ] Decode temporal segmentation prediction and consume the retained segmentation map when `segmentation_update_map == 1` and temporal update is enabled. - - [ ] Decode and connect inter-block reference indices, motion vectors, compound prediction, inter-intra prediction, masked blending, warped motion, global motion, and OBMC through reconstructed reference planes. The current tile mode reader accepts intra frames only. + - [x] Decode temporal segmentation prediction and consume the retained segmentation map when `segmentation_update_map == 1` and temporal update is enabled. + - [x] Finish the simple single-reference path. + - [x] Own and decode the adaptive single-reference, inter-mode, DRL, normal-motion-vector, interpolation-filter, inter-intra, OBMC, and motion-mode distributions through the frame CDF lifecycle. + - [x] Derive the spatial and temporal reference-MV stack, select NEAREST/NEAR/NEW/GLOBAL and DRL candidates, decode NEW motion-vector differences, validate the final vector, and preserve libaom's exact duplicate and weighting behavior. + - [x] Verify the disputed source predicates against pinned libaom: identity `GLOBALMV` blocks of sufficient size omit interpolation symbols through `is_nontrans_global_motion()`, and the single-reference spatial extension loops stop at `MAX_MV_REF_CANDIDATES` (two), not the full eight-entry stack capacity. + - [x] Consume the exact inter-intra and binary or ternary motion-mode syntax that precedes interpolation filters, then reconstruct simple translational blocks through the retained padded reference and existing SIMD-first predictor before residual traversal. + - [x] Store frame-sized retained and temporal motion fields in ImageSharp allocator-owned memory with deterministic counted disposal. Allocation tracking verifies aliases, success ownership, failure unwinding, presentation and decoder-result ownership, and exactly-once final disposal. + - [x] Request the established contiguous allocation contract for every padded frame plane and verify complete-plane reconstruction with a constrained ImageSharp allocator without copying. + - [x] Complete source review of the new motion-mode and single-reference tests. + - [x] Pass the exact current tree's Release verification. The `net10.0` and `net11.0` source builds and the `net10.0` test-project analyzer build complete with zero warnings and errors. All 293 focused `net10.0` syntax, CDF lifecycle, candidate, vector, interpolation, lifecycle, reconstruction, ownership, and `FeatureTestRunner` cases pass with zero failures or skips, including exact dependent-frame native-plane and presentation comparisons, constrained allocation, and motion-field lifetime coverage. `git diff --check` is clean. + - [x] Correct the logical progressive color payload and compare the final dependent frame's native planes and libavif presentation exactly. Pinned libaom decodes both layers from the primary item's two `iloc` extents, and pinned libavif produces the retained two-frame YUV444-alpha reference and final PNG. The exact production-path comparisons pass through `FeatureTestRunner`. + - [ ] Decode and reconstruct compound prediction, selected inter-intra prediction, masked blending, OBMC, scaled references, warped motion, and non-translational global motion through reconstructed reference planes. - [ ] Verify every connected inter mode and filter with independently encoded dependent-layer AV1 image-item fixtures and exact native-plane comparisons. - [ ] Lossless and high-bit-depth reconstruction with correct clipping and intermediate precision. - [x] Route lossless 4x4 blocks through allocation-free reversible inverse Walsh-Hadamard reconstruction for 8/10/12-bit samples, including the DC-only specialization, `Vector128` production traversal, scalar fallback, exact clipping, and `FeatureTestRunner` parity. @@ -569,7 +596,7 @@ For each SIMD-suitable item, design the data layout, operator contract, scratch Exit gate: -- [ ] Independently encoded, opaque, single-item AVIF files reconstruct correctly across all AVIF profiles, bit depths, subsampling modes, and normative still-image compression tools. Pixel comparisons are made after applying the same signaled color conversion in the reference path. +- [ ] Independently encoded opaque AVIF still images, including reference-dependent layered items, reconstruct correctly across all exposed AV1 profiles, bit depths, subsampling modes, and normative image compression tools. Native planes are compared exactly with pinned libaom output, and presented pixels are compared after applying the same signaled color conversion in the reference path. ### Phase 4: complete HEVC still-image reconstruction @@ -773,7 +800,9 @@ No valid HEVC or AV1 color, compression, or bit-depth row may remain `unsupporte The dashboard and immediate execution queue define the remaining critical path. In phase terms, work proceeds as follows: - [x] Finish the Phase 8 checkpoint for the implemented AV1 `Vector512` transforms, including Release, feature-isolation, and benchmark evidence. -- [ ] **Current:** close Phase 3 by removing every unsupported valid AV1 still-image syntax path and proving the complete AVIF decode matrix with independent inputs and scalar/SIMD parity. +- [x] Finish the complete focused Release matrix, final source/documentation review, and checkpoint implementation for the simple single-reference inter slice. The corrected real dependent-frame AVIF passes exact pinned-libaom native-plane and pinned-libavif presentation comparisons, and all 293 focused cases pass with zero failures or skips. +- [ ] Complete the remaining Phase 3 inter modes in dependency order: compound reference selection and averaging, inter-intra and masked blending, OBMC, scaled references, warped motion, and non-translational global prediction. Each mode requires an independent fixture before the next begins. +- [ ] Remove every other unsupported valid AV1 still-image syntax path, correct the audited 12-bit inverse-transform arithmetic, and prove the complete AVIF decode matrix with independent inputs and scalar/SIMD parity. - [ ] Close Phase 4 by completing the remaining HEVC profile and Range Extensions matrix with exact native-plane and presented-image evidence. - [ ] Close Phase 5 and the decode portion of the bounded sequence ledger: color, ICC, alpha, grids, presentation transforms, reference-dependent samples, and complete animated AVIF/HEIC decode. - [ ] Close the still-image portions of Phases 0, 1, and 2 that remain as release gates: documentation, provenance, public format boundaries, API review, parser hardening, and malformed-input coverage. diff --git a/src/ImageSharp/Formats/Heif/Av1/Av1BlockSizeExtensions.cs b/src/ImageSharp/Formats/Heif/Av1/Av1BlockSizeExtensions.cs index eafde6bef..3f46a3b41 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Av1BlockSizeExtensions.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Av1BlockSizeExtensions.cs @@ -140,6 +140,18 @@ internal static class Av1BlockSizeExtensions public static int Get4x4HeightLog2(this Av1BlockSize blockSize) => Av1Math.Log2(Get4x4HighCount(blockSize)); + /// + /// Gets the entropy context group associated with the block size. + /// + /// The block size. + /// The zero-based size group in the inclusive range zero through three. + public static int GetSizeGroup(this Av1BlockSize blockSize) + { + // AV1 section 9.3 groups a block by its smaller dimension in 4x4 units and caps that logarithm at three. + // Deriving the value from the existing geometry tables exactly matches libaom's size_group_lookup table. + return Math.Min(3, Math.Min(blockSize.Get4x4WidthLog2(), blockSize.Get4x4HeightLog2())); + } + /// /// Gets the residual-plane block size for Boolean chroma subsampling flags. /// diff --git a/src/ImageSharp/Formats/Heif/Av1/Av1Decoder.cs b/src/ImageSharp/Formats/Heif/Av1/Av1Decoder.cs index 89379f245..737651774 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Av1Decoder.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Av1Decoder.cs @@ -112,14 +112,22 @@ internal sealed class Av1Decoder : IAv1TileReader, IDisposable /// /// The item-associated AV1 codec configuration validated against the coded sequence header. /// + /// The optional byte boundaries of a layered AV1 image item. /// The decoded image. public Image Decode( Span buffer, CicpProfile? containerColorProfile = null, - Av1CodecConfiguration? codecConfiguration = null) + Av1CodecConfiguration? codecConfiguration = null, + Av1LayeredImageIndex? layeredImageIndex = null) where TPixel : unmanaged, IPixel { - ImageFrame frame = this.DecodeFrame(buffer, containerColorProfile, codecConfiguration, out CicpProfile effectiveColorProfile); + ImageFrame frame = this.DecodeFrame( + buffer, + containerColorProfile, + codecConfiguration, + out CicpProfile effectiveColorProfile, + layeredImageIndex); + ImageMetadata metadata = new() { CicpProfile = effectiveColorProfile @@ -149,19 +157,22 @@ internal sealed class Av1Decoder : IAv1TileReader, IDisposable /// The AV1 codec configuration validated against the coded sequence header. /// /// Receives the effective CICP description used for conversion. + /// The optional byte boundaries of a layered AV1 image item. /// The decoded frame. Ownership transfers to the caller. public ImageFrame DecodeFrame( Span buffer, CicpProfile? containerColorProfile, Av1CodecConfiguration? codecConfiguration, - out CicpProfile effectiveColorProfile) + out CicpProfile effectiveColorProfile, + Av1LayeredImageIndex? layeredImageIndex = null) where TPixel : unmanaged, IPixel { using Av1FrameBuffer frameBuffer = this.DecodeFrameBuffer( buffer, containerColorProfile, codecConfiguration, - out effectiveColorProfile); + out effectiveColorProfile, + layeredImageIndex); ImageFrame? resultFrame = null; try @@ -196,6 +207,7 @@ internal sealed class Av1Decoder : IAv1TileReader, IDisposable /// The complete presented size of the auxiliary image or grid tile. /// The destination region receiving the top-left portion of the presented alpha image. /// Whether stored color samples must be converted to unassociated alpha. + /// The optional byte boundaries of a layered AV1 image item. public void DecodeAlpha( Span buffer, CicpProfile? containerColorProfile, @@ -204,10 +216,17 @@ internal sealed class Av1Decoder : IAv1TileReader, IDisposable ImageFrame destination, Size outputSize, Rectangle destinationRectangle, - bool premultiplied) + bool premultiplied, + Av1LayeredImageIndex? layeredImageIndex = null) where TPixel : unmanaged, IPixel { - using Av1FrameBuffer frameBuffer = this.DecodeFrameBuffer(buffer, containerColorProfile, codecConfiguration, out _); + using Av1FrameBuffer frameBuffer = this.DecodeFrameBuffer( + buffer, + containerColorProfile, + codecConfiguration, + out _, + layeredImageIndex); + if (expectedCodedSize != default && (frameBuffer.Width != expectedCodedSize.Width || frameBuffer.Height != expectedCodedSize.Height)) { throw new InvalidImageContentException("The decoded alpha sample dimensions do not match its visual sample entry."); @@ -238,24 +257,60 @@ internal sealed class Av1Decoder : IAv1TileReader, IDisposable /// /// The AV1 codec configuration validated against the coded sequence header. /// Receives the effective CICP description associated with the native planes. + /// The optional byte boundaries of a layered AV1 image item. /// The reconstructed native frame buffer. Ownership transfers to the caller. public Av1FrameBuffer DecodeFrameBuffer( Span buffer, CicpProfile? containerColorProfile, Av1CodecConfiguration? codecConfiguration, - out CicpProfile effectiveColorProfile) + out CicpProfile effectiveColorProfile, + Av1LayeredImageIndex? layeredImageIndex = null) { this.codecConfiguration = codecConfiguration; this.containerColorProfile = containerColorProfile; this.validatedSequenceHeader = null; this.SequenceHeader = null; this.FrameHeader = null; + this.FrameInfo?.ReleaseOwner(); this.FrameInfo = null; - Av1BitStreamReader reader = new(buffer); try { - this.obuReader.ReadAll(ref reader, buffer.Length, () => this, false); + if (layeredImageIndex is null) + { + Av1BitStreamReader reader = new(buffer); + this.obuReader.ReadAll(ref reader, buffer.Length, () => this, false); + } + else + { + int layerOffset = 0; + for (int layer = 0; layer < Av1Constants.MaxSpatialLayerCount - 1 && layerOffset < buffer.Length; layer++) + { + uint declaredLayerSize = layer switch + { + 0 => layeredImageIndex.Value.FirstLayerSize, + 1 => layeredImageIndex.Value.SecondLayerSize, + _ => layeredImageIndex.Value.ThirdLayerSize + }; + + if (declaredLayerSize == 0) + { + break; + } + + int layerSize = (int)declaredLayerSize; + Av1BitStreamReader layerReader = new(buffer.Slice(layerOffset, layerSize)); + this.obuReader.ReadAll(ref layerReader, layerSize, () => this, false); + layerOffset += layerSize; + } + + if (layerOffset < buffer.Length) + { + Span finalLayer = buffer[layerOffset..]; + Av1BitStreamReader finalLayerReader = new(finalLayer); + this.obuReader.ReadAll(ref finalLayerReader, finalLayer.Length, () => this, false); + } + } Guard.NotNull(this.referenceFrames.OutputFrame, nameof(this.referenceFrames.OutputFrame)); Guard.NotNull(this.SequenceHeader, nameof(this.SequenceHeader)); @@ -284,6 +339,7 @@ internal sealed class Av1Decoder : IAv1TileReader, IDisposable this.entropySequenceHeader = null; this.SequenceHeader = null; this.FrameHeader = null; + this.FrameInfo?.ReleaseOwner(); this.FrameInfo = null; throw; } @@ -430,7 +486,12 @@ internal sealed class Av1Decoder : IAv1TileReader, IDisposable sequenceHeader.ColorConfig.GetColorFormat(), false); - using Av1FrameDecoder frameDecoder = new(sequenceHeader, frameHeader, frameInfo, frameBuffer); + // Plane allocations use the sequence maxima, while every reconstruction stage must see the active coded + // dimensions. Super-resolution replaces Width after decoding; Height remains the coded frame height. + frameBuffer.Width = frameHeader.FrameSize.FrameWidth; + frameBuffer.Height = frameHeader.FrameSize.FrameHeight; + + using Av1FrameDecoder frameDecoder = new(sequenceHeader, frameHeader, frameInfo, frameBuffer, this.referenceFrames); frameDecoder.DecodeFrame(); bool retainsReference = (frameHeader.RefreshFrameFlags & byte.MaxValue) != 0; @@ -493,13 +554,20 @@ internal sealed class Av1Decoder : IAv1TileReader, IDisposable { this.SequenceHeader = sequenceHeader; this.FrameHeader = frameHeader; + + // The output frame owner is released after its sample buffer transfers to the caller. Retain the + // parsed state independently so diagnostics and conformance inspection remain valid until the next + // bounded decode or decoder disposal. + frameInfo.AddOwner(); + this.FrameInfo?.ReleaseOwner(); this.FrameInfo = frameInfo; } } finally { - // A non-shown frame or failed reconstruction never escapes this callback. FrameInfo uses managed storage, - // so it remains inspectable for a retained frame after the pooled entropy-neighbor contexts are returned. + // A non-shown frame or failed reconstruction never escapes this callback. The tile reader releases only + // its initial frame-state lease; retained frames and the decoder result keep allocator-owned motion fields + // alive independently after the entropy-neighbor contexts are returned. presentationBuffer?.Dispose(); frameBuffer?.Dispose(); tileReader.Dispose(); @@ -515,5 +583,7 @@ internal sealed class Av1Decoder : IAv1TileReader, IDisposable this.tileReader?.Dispose(); this.tileReader = null; this.referenceFrames.Dispose(); + this.FrameInfo?.ReleaseOwner(); + this.FrameInfo = null; } } diff --git a/src/ImageSharp/Formats/Heif/Av1/Av1FrameBuffer.cs b/src/ImageSharp/Formats/Heif/Av1/Av1FrameBuffer.cs index f3a79566c..eccc0228d 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Av1FrameBuffer.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Av1FrameBuffer.cs @@ -16,9 +16,11 @@ internal class Av1FrameBuffer : IDisposable where T : unmanaged { /// - /// The number of border samples reserved for intra prediction and in-loop filtering. + /// The number of luma border samples reserved for prediction and in-loop filtering. /// - private const int DecoderPaddingValue = 72; + // A 128-sample UMV block plus filter support reaches 135 luma samples beyond an edge. The 144-sample value also + // leaves 72 samples on a horizontally subsampled plane, exceeding its corresponding 71-sample maximum. + private const int DecoderPaddingValue = 144; /// /// The allocation-mask bit for the luma plane. @@ -108,21 +110,24 @@ internal class Av1FrameBuffer : IDisposable this.BufferY = null; this.BufferCb = null; this.BufferCr = null; + + // Block reconstruction and the SIMD predictors address decoder padding through one span plus a constant row + // stride. Establish that invariant at the plane owner instead of copying fragmented groups in every hot path. try { if ((bufferEnableMask & PictureBufferYFlag) != 0) { - this.BufferY = configuration.MemoryAllocator.Allocate2D(strideY * this.storageElementsPerSample, heightY); + this.BufferY = configuration.MemoryAllocator.Allocate2D(strideY * this.storageElementsPerSample, heightY, preferContiguosImageBuffers: true); } if ((bufferEnableMask & PictureBufferCbFlag) != 0) { - this.BufferCb = configuration.MemoryAllocator.Allocate2D(strideChroma * this.storageElementsPerSample, heightChroma); + this.BufferCb = configuration.MemoryAllocator.Allocate2D(strideChroma * this.storageElementsPerSample, heightChroma, preferContiguosImageBuffers: true); } if ((bufferEnableMask & PictureBufferCrFlag) != 0) { - this.BufferCr = configuration.MemoryAllocator.Allocate2D(strideChroma * this.storageElementsPerSample, heightChroma); + this.BufferCr = configuration.MemoryAllocator.Allocate2D(strideChroma * this.storageElementsPerSample, heightChroma, preferContiguosImageBuffers: true); } } catch @@ -369,6 +374,44 @@ internal class Av1FrameBuffer : IDisposable return samples.Slice(originX, width); } + /// + /// Gets the complete padded storage allocation for one plane. + /// + /// The luma or chroma plane. + /// The horizontal chroma subsampling shift. + /// The vertical chroma subsampling shift. + /// Receives the number of logical samples between adjacent rows. + /// Receives the visible plane origin within the padded allocation. + /// The complete plane allocation, including decoder padding. + public Span GetPaddedPlaneSpan(Av1Plane plane, int subX, int subY, out int stride, out Point origin) + { + this.GetPlaneLayout( + plane, + subX, + subY, + out Buffer2D buffer, + out int originX, + out int originY, + out _, + out _); + + stride = buffer.Width / this.storageElementsPerSample; + origin = new(originX, originY); + return buffer.DangerousGetSingleSpan(); + } + + /// + /// Gets the complete padded storage allocation for one native 16-bit plane. + /// + /// The luma or chroma plane. + /// The horizontal chroma subsampling shift. + /// The vertical chroma subsampling shift. + /// Receives the number of logical samples between adjacent rows. + /// Receives the visible plane origin within the padded allocation. + /// The complete plane allocation, including decoder padding. + public Span GetPaddedPlaneSpan16(Av1Plane plane, int subX, int subY, out int stride, out Point origin) + => MemoryMarshal.Cast(this.GetPaddedPlaneSpan(plane, subX, subY, out stride, out origin)); + /// /// Resolves a plane allocation and its visible padded layout. /// diff --git a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1DefaultDistributions.cs b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1DefaultDistributions.cs index 842ffcb28..13c6a51c1 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1DefaultDistributions.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1DefaultDistributions.cs @@ -24,6 +24,127 @@ internal static class Av1DefaultDistributions /// public static Av1Distribution[] IntraInter => [new(806), new(16662), new(20186), new(26538)]; + /// + /// Gets the distributions that select a newly decoded motion vector before the remaining single-reference modes. + /// + public static Av1Distribution[] NewMv => [new(24035), new(16630), new(15339), new(8386), new(12222), new(4676)]; + + /// + /// Gets the distributions that select global motion before the spatial reference-motion-vector modes. + /// + public static Av1Distribution[] ZeroMv => [new(2175), new(1054)]; + + /// + /// Gets the distributions that select the nearest or near spatial reference motion vector. + /// + public static Av1Distribution[] RefMv => [new(23974), new(24188), new(17848), new(28622), new(24312), new(19923)]; + + /// + /// Gets the distributions that advance through the dynamic reference-motion-vector candidate list. + /// + public static Av1Distribution[] Drl => [new(13104), new(24560), new(18945)]; + + /// + /// Gets the single-reference selection distributions indexed by spatial context and tree decision. + /// + public static Av1Distribution[][] SingleReference => + [ + [new(4897), new(1555), new(4236), new(8650), new(904), new(1444)], + [new(16973), new(16751), new(19647), new(24773), new(11014), new(15087)], + [new(29744), new(30279), new(31194), new(31895), new(26875), new(30304)], + ]; + + /// + /// Gets the distributions that select single-reference or compound-reference prediction for a block. + /// + public static Av1Distribution[] CompInter => [new(26828), new(24035), new(12031), new(10640), new(2901)]; + + /// + /// Gets the inter-intra prediction flag distributions indexed by block-size group. + /// + public static Av1Distribution[] InterIntra => [new(16384), new(26887), new(27597), new(30237)]; + + /// + /// Gets the Simple Translation, OBMC, or Warped motion-mode distributions indexed by block size. + /// + public static Av1Distribution[] MotionMode => + [ + new(10923, 21845), + new(10923, 21845), + new(10923, 21845), + new(7651, 24760), + new(4738, 24765), + new(5391, 25528), + new(19419, 26810), + new(5123, 23606), + new(11606, 24308), + new(26260, 29116), + new(20360, 28062), + new(21679, 26830), + new(29516, 30701), + new(28898, 30397), + new(30878, 31335), + new(32507, 32558), + new(10923, 21845), + new(10923, 21845), + new(28799, 31390), + new(26431, 30774), + new(28973, 31594), + new(29742, 31203), + ]; + + /// + /// Gets the Simple Translation or OBMC motion-mode distributions indexed by block size. + /// + public static Av1Distribution[] Obmc => + [ + new(16384), + new(16384), + new(16384), + new(10437), + new(9371), + new(9301), + new(17432), + new(14423), + new(15142), + new(25817), + new(22823), + new(22083), + new(30128), + new(31014), + new(31560), + new(32638), + new(16384), + new(16384), + new(23664), + new(20901), + new(24008), + new(26879), + ]; + + /// + /// Gets the switchable interpolation-filter distributions indexed by reference type, direction, and neighbor state. + /// + public static Av1Distribution[] SwitchableInterpolation => + [ + new(31935, 32720), + new(5568, 32719), + new(422, 2938), + new(28244, 32608), + new(31206, 31953), + new(4862, 32121), + new(770, 1152), + new(20889, 25637), + new(31910, 32724), + new(4120, 32712), + new(305, 2247), + new(27403, 32636), + new(31022, 32009), + new(2963, 32093), + new(601, 943), + new(14969, 21398), + ]; + /// /// Gets the key-frame luma-mode distributions indexed by the above and left intra-mode contexts. /// diff --git a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1DisplacementVectorContext.cs b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1DisplacementVectorContext.cs deleted file mode 100644 index 23483c4b5..000000000 --- a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1DisplacementVectorContext.cs +++ /dev/null @@ -1,228 +0,0 @@ -// Copyright (c) Six Labors. -// Licensed under the Six Labors Split License. - -using SixLabors.ImageSharp.Formats.Heif.Av1.Motion; - -namespace SixLabors.ImageSharp.Formats.Heif.Av1.Entropy; - -/// -/// Decodes integer intra-block-copy displacement vectors with tile-adaptive AV1 distributions. -/// -internal sealed class Av1DisplacementVectorContext -{ - /// - /// The number of magnitude classes defined by AV1. - /// - private const int MagnitudeClassCount = 11; - - /// - /// The number of class-zero integer magnitude bits. - /// - private const int ClassZeroBitCount = 1; - - /// - /// The tile-adaptive distribution selecting which vector components are nonzero. - /// - private readonly Av1Distribution joint = new(4096, 11264, 19328); - - /// - /// The tile-adaptive vertical component distributions. - /// - private readonly Component vertical = new(); - - /// - /// The tile-adaptive horizontal component distributions. - /// - private readonly Component horizontal = new(); - - /// - /// Replaces every displacement-vector distribution with state copied from another context. - /// - /// The displacement-vector context state to copy. - public void CopyFrom(Av1DisplacementVectorContext source) - { - this.joint.CopyFrom(source.joint); - this.vertical.CopyFrom(source.vertical); - this.horizontal.CopyFrom(source.horizontal); - } - - /// - /// Resets every observation count used to adapt displacement-vector distributions. - /// - public void ResetUpdateCounts() - { - this.joint.ResetUpdateCount(); - this.vertical.ResetUpdateCounts(); - this.horizontal.ResetUpdateCounts(); - } - - /// - /// Reads an integer displacement vector relative to a spatially derived reference. - /// - /// The tile range decoder. - /// The reference displacement vector. - /// The decoded displacement vector in one-eighth-sample units. - public Av1MotionVector Read(ref Av1SymbolReader reader, Av1MotionVector reference) - { - int jointType = reader.ReadSymbol(this.joint); - - // Joint values 1 and 3 carry a horizontal delta; values 2 and 3 carry a vertical delta. Intra-block copy - // fixes precision to whole luma samples, so the component reader consumes no fractional or high-precision CDFs. - int row = jointType >= 2 ? this.vertical.Read(ref reader) : 0; - int column = (jointType & 1) != 0 ? this.horizontal.Read(ref reader) : 0; - return reference + new Av1MotionVector(row, column); - } - - /// - /// Writes an integer displacement vector relative to a spatially derived reference. - /// - /// The tile range encoder. - /// The displacement vector to encode. - /// The spatially derived reference vector. - public void Write(Av1SymbolWriter writer, Av1MotionVector value, Av1MotionVector reference) - { - int row = value.Row - reference.Row; - int column = value.Column - reference.Column; - int jointType = (row != 0 ? 2 : 0) | (column != 0 ? 1 : 0); - writer.WriteSymbol(jointType, this.joint); - - if (row != 0) - { - this.vertical.Write(writer, row); - } - - if (column != 0) - { - this.horizontal.Write(writer, column); - } - } - - /// - /// Stores the adaptive magnitude distributions for one displacement-vector component. - /// - private sealed class Component - { - /// - /// The distribution selecting the signed magnitude class. - /// - private readonly Av1Distribution magnitudeClass = new(28672, 30976, 31858, 32320, 32551, 32656, 32740, 32757, 32762, 32767); - - /// - /// The distribution selecting the sign of a nonzero component. - /// - private readonly Av1Distribution sign = new(16384); - - /// - /// The distribution selecting either of the two class-zero integer magnitudes. - /// - private readonly Av1Distribution classZero = new(27648); - - /// - /// The binary distributions that reconstruct larger magnitude offsets from least to most significant bit. - /// - private readonly Av1Distribution[] offsetBits = - [ - new(17408), new(17920), new(18944), new(20480), new(22528), - new(24576), new(28672), new(29952), new(29952), new(30720) - ]; - - /// - /// Replaces every component distribution with state copied from another component. - /// - /// The component state to copy. - public void CopyFrom(Component source) - { - this.magnitudeClass.CopyFrom(source.magnitudeClass); - this.sign.CopyFrom(source.sign); - this.classZero.CopyFrom(source.classZero); - - for (int bit = 0; bit < this.offsetBits.Length; bit++) - { - this.offsetBits[bit].CopyFrom(source.offsetBits[bit]); - } - } - - /// - /// Resets every observation count used to adapt one component's distributions. - /// - public void ResetUpdateCounts() - { - this.magnitudeClass.ResetUpdateCount(); - this.sign.ResetUpdateCount(); - this.classZero.ResetUpdateCount(); - - for (int bit = 0; bit < this.offsetBits.Length; bit++) - { - this.offsetBits[bit].ResetUpdateCount(); - } - } - - /// - /// Reads one signed integer-precision component. - /// - /// The tile range decoder. - /// The signed component in one-eighth-sample units. - public int Read(ref Av1SymbolReader reader) - { - bool isNegative = reader.ReadSymbol(this.sign) != 0; - int magnitudeClass = reader.ReadSymbol(this.magnitudeClass); - int offset; - int magnitudeBase; - - if (magnitudeClass == 0) - { - offset = reader.ReadSymbol(this.classZero); - magnitudeBase = 0; - } - else - { - int bitCount = magnitudeClass + ClassZeroBitCount - 1; - offset = 0; - for (int bit = 0; bit < bitCount; bit++) - { - // AV1 transmits the integer offset least-significant bit first, with an independently adapting - // distribution for every bit position. - offset |= reader.ReadSymbol(this.offsetBits[bit]) << bit; - } - - magnitudeBase = (1 << ClassZeroBitCount) << (magnitudeClass + 2); - } - - // Integer precision substitutes the normative fractional values fr=3 and hp=1. The low three bits are - // consequently all one, and the final increment converts the zero-based magnitude representation. - int magnitude = magnitudeBase + (offset << 3) + 8; - return isNegative ? -magnitude : magnitude; - } - - /// - /// Writes one signed integer-precision component. - /// - /// The tile range encoder. - /// The nonzero component in one-eighth-sample units. - public void Write(Av1SymbolWriter writer, int value) - { - int magnitude = Math.Abs(value); - DebugGuard.IsTrue(magnitude > 0 && (magnitude & 7) == 0, "Displacement-vector components must use whole-sample precision."); - - int magnitudeClass = magnitude <= 16 ? 0 : Av1Math.MostSignificantBit((uint)(magnitude - 1)) - 3; - DebugGuard.MustBeLessThan(magnitudeClass, MagnitudeClassCount, nameof(magnitudeClass)); - writer.WriteSymbol(value < 0, this.sign); - writer.WriteSymbol(magnitudeClass, this.magnitudeClass); - - if (magnitudeClass == 0) - { - writer.WriteSymbol((magnitude >> 3) - 1, this.classZero); - return; - } - - int magnitudeBase = 8 << magnitudeClass; - int offset = (magnitude - magnitudeBase - 8) >> 3; - for (int bit = 0; bit < magnitudeClass; bit++) - { - // The decoder reconstructs offsets least-significant bit first, so each adaptive bit model must be - // updated in the same order during encoding. - writer.WriteSymbol(((offset >> bit) & 1) != 0, this.offsetBits[bit]); - } - } - } -} diff --git a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1FrameEntropyContext.cs b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1FrameEntropyContext.cs index de899d117..1d0b5cc48 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1FrameEntropyContext.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1FrameEntropyContext.cs @@ -65,6 +65,10 @@ internal sealed class Av1FrameEntropyContext // Every default-distribution accessor constructs independently mutable state. Retaining those returned // graphs directly confines generated-table construction to the four process-wide quantizer-band prototypes. this.IntraBlockCopy = Av1DefaultDistributions.IntraBlockCopy; + + // Normal motion vectors and intra-block-copy displacement vectors start from identical defaults, but AV1 + // adapts NMVC and NDVC independently. Distinct object graphs preserve that separation for the prototype too. + this.MotionVector = new(); this.DisplacementVector = new(); this.SwitchableRestoration = Av1DefaultDistributions.SwitchableRestoration; this.WienerRestoration = Av1DefaultDistributions.WienerRestoration; @@ -79,6 +83,16 @@ internal sealed class Av1FrameEntropyContext this.FrameYMode = Av1DefaultDistributions.FrameYMode; this.KeyFrameYMode = Av1DefaultDistributions.KeyFrameYMode; this.IntraInter = Av1DefaultDistributions.IntraInter; + this.NewMv = Av1DefaultDistributions.NewMv; + this.ZeroMv = Av1DefaultDistributions.ZeroMv; + this.RefMv = Av1DefaultDistributions.RefMv; + this.Drl = Av1DefaultDistributions.Drl; + this.SingleReference = Av1DefaultDistributions.SingleReference; + this.CompInter = Av1DefaultDistributions.CompInter; + this.InterIntra = Av1DefaultDistributions.InterIntra; + this.MotionMode = Av1DefaultDistributions.MotionMode; + this.Obmc = Av1DefaultDistributions.Obmc; + this.SwitchableInterpolation = Av1DefaultDistributions.SwitchableInterpolation; this.UvMode = Av1DefaultDistributions.UvMode; this.Skip = Av1DefaultDistributions.Skip; this.SkipMode = Av1DefaultDistributions.SkipMode; @@ -115,6 +129,8 @@ internal sealed class Av1FrameEntropyContext // Session and retained-frame contexts need one mutable graph, not four generated quantizer-band graphs whose // unused bands are immediately discarded. Deep-copy the already selected prototype shape exactly once. this.IntraBlockCopy = source.IntraBlockCopy.CreateCopy(); + this.MotionVector = new(); + this.MotionVector.CopyFrom(source.MotionVector); this.DisplacementVector = new(); this.DisplacementVector.CopyFrom(source.DisplacementVector); this.SwitchableRestoration = source.SwitchableRestoration.CreateCopy(); @@ -130,6 +146,16 @@ internal sealed class Av1FrameEntropyContext this.FrameYMode = Av1Distribution.CreateCopy(source.FrameYMode); this.KeyFrameYMode = Av1Distribution.CreateCopy(source.KeyFrameYMode); this.IntraInter = Av1Distribution.CreateCopy(source.IntraInter); + this.NewMv = Av1Distribution.CreateCopy(source.NewMv); + this.ZeroMv = Av1Distribution.CreateCopy(source.ZeroMv); + this.RefMv = Av1Distribution.CreateCopy(source.RefMv); + this.Drl = Av1Distribution.CreateCopy(source.Drl); + this.SingleReference = Av1Distribution.CreateCopy(source.SingleReference); + this.CompInter = Av1Distribution.CreateCopy(source.CompInter); + this.InterIntra = Av1Distribution.CreateCopy(source.InterIntra); + this.MotionMode = Av1Distribution.CreateCopy(source.MotionMode); + this.Obmc = Av1Distribution.CreateCopy(source.Obmc); + this.SwitchableInterpolation = Av1Distribution.CreateCopy(source.SwitchableInterpolation); this.UvMode = Av1Distribution.CreateCopy(source.UvMode); this.Skip = Av1Distribution.CreateCopy(source.Skip); this.SkipMode = Av1Distribution.CreateCopy(source.SkipMode); @@ -159,10 +185,15 @@ internal sealed class Av1FrameEntropyContext /// public Av1Distribution IntraBlockCopy { get; } + /// + /// Gets the entropy context used by normal inter-prediction motion vectors. + /// + public Av1MotionVectorContext MotionVector { get; } + /// /// Gets the integer displacement-vector context used by intra-block copy. /// - public Av1DisplacementVectorContext DisplacementVector { get; } + public Av1MotionVectorContext DisplacementVector { get; } /// /// Gets the switchable loop-restoration distribution. @@ -229,6 +260,56 @@ internal sealed class Av1FrameEntropyContext /// public Av1Distribution[] IntraInter { get; } + /// + /// Gets the distributions that select a newly decoded motion vector before the remaining single-reference modes. + /// + public Av1Distribution[] NewMv { get; } + + /// + /// Gets the distributions that select global motion before the spatial reference-motion-vector modes. + /// + public Av1Distribution[] ZeroMv { get; } + + /// + /// Gets the distributions that select the nearest or near spatial reference motion vector. + /// + public Av1Distribution[] RefMv { get; } + + /// + /// Gets the distributions that advance through the dynamic reference-motion-vector candidate list. + /// + public Av1Distribution[] Drl { get; } + + /// + /// Gets the single-reference selection distributions indexed by spatial context and tree decision. + /// + public Av1Distribution[][] SingleReference { get; } + + /// + /// Gets the distributions that select single-reference or compound-reference prediction for a block. + /// + public Av1Distribution[] CompInter { get; } + + /// + /// Gets the inter-intra prediction flag distributions indexed by block-size group. + /// + public Av1Distribution[] InterIntra { get; } + + /// + /// Gets the three-way motion-mode distributions indexed by block size. + /// + public Av1Distribution[] MotionMode { get; } + + /// + /// Gets the binary Simple Translation or OBMC distributions indexed by block size. + /// + public Av1Distribution[] Obmc { get; } + + /// + /// Gets the switchable interpolation-filter distributions. + /// + public Av1Distribution[] SwitchableInterpolation { get; } + /// /// Gets the chroma intra-mode distributions. /// @@ -373,6 +454,7 @@ internal sealed class Av1FrameEntropyContext public void CopyFrom(Av1FrameEntropyContext source) { this.IntraBlockCopy.CopyFrom(source.IntraBlockCopy); + this.MotionVector.CopyFrom(source.MotionVector); this.DisplacementVector.CopyFrom(source.DisplacementVector); this.SwitchableRestoration.CopyFrom(source.SwitchableRestoration); this.WienerRestoration.CopyFrom(source.WienerRestoration); @@ -387,6 +469,16 @@ internal sealed class Av1FrameEntropyContext CopyState(source.FrameYMode, this.FrameYMode); CopyState(source.KeyFrameYMode, this.KeyFrameYMode); CopyState(source.IntraInter, this.IntraInter); + CopyState(source.NewMv, this.NewMv); + CopyState(source.ZeroMv, this.ZeroMv); + CopyState(source.RefMv, this.RefMv); + CopyState(source.Drl, this.Drl); + CopyState(source.SingleReference, this.SingleReference); + CopyState(source.CompInter, this.CompInter); + CopyState(source.InterIntra, this.InterIntra); + CopyState(source.MotionMode, this.MotionMode); + CopyState(source.Obmc, this.Obmc); + CopyState(source.SwitchableInterpolation, this.SwitchableInterpolation); CopyState(source.UvMode, this.UvMode); CopyState(source.Skip, this.Skip); CopyState(source.SkipMode, this.SkipMode); @@ -431,6 +523,7 @@ internal sealed class Av1FrameEntropyContext private void ResetUpdateCounts() { this.IntraBlockCopy.ResetUpdateCount(); + this.MotionVector.ResetUpdateCounts(); this.DisplacementVector.ResetUpdateCounts(); this.SwitchableRestoration.ResetUpdateCount(); this.WienerRestoration.ResetUpdateCount(); @@ -445,6 +538,16 @@ internal sealed class Av1FrameEntropyContext ResetUpdateCounts(this.FrameYMode); ResetUpdateCounts(this.KeyFrameYMode); ResetUpdateCounts(this.IntraInter); + ResetUpdateCounts(this.NewMv); + ResetUpdateCounts(this.ZeroMv); + ResetUpdateCounts(this.RefMv); + ResetUpdateCounts(this.Drl); + ResetUpdateCounts(this.SingleReference); + ResetUpdateCounts(this.CompInter); + ResetUpdateCounts(this.InterIntra); + ResetUpdateCounts(this.MotionMode); + ResetUpdateCounts(this.Obmc); + ResetUpdateCounts(this.SwitchableInterpolation); ResetUpdateCounts(this.UvMode); ResetUpdateCounts(this.Skip); ResetUpdateCounts(this.SkipMode); diff --git a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1MotionVectorContext.cs b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1MotionVectorContext.cs new file mode 100644 index 000000000..29f477dc6 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1MotionVectorContext.cs @@ -0,0 +1,319 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using SixLabors.ImageSharp.Formats.Heif.Av1.Motion; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Entropy; + +/// +/// Owns one independently adaptive AV1 motion-vector entropy context. +/// +/// +/// Normal inter-prediction vectors and intra-block-copy displacement vectors use identical initial distributions, but +/// each syntax domain owns a separate instance so observations from one domain cannot adapt the other. +/// +internal sealed class Av1MotionVectorContext +{ + /// + /// The number of magnitude classes defined by AV1. + /// + private const int MagnitudeClassCount = 11; + + /// + /// The number of integer magnitude bits coded directly for class zero. + /// + private const int ClassZeroBitCount = 1; + + /// + /// The number of integer magnitude offsets represented by class zero. + /// + private const int ClassZeroSize = 1 << ClassZeroBitCount; + + /// + /// Gets the distribution selecting which vector components are nonzero. + /// + public Av1Distribution Joint { get; } = new(4096, 11264, 19328); + + /// + /// Gets the adaptive distributions for the vertical vector component. + /// + public Component Vertical { get; } = new(); + + /// + /// Gets the adaptive distributions for the horizontal vector component. + /// + public Component Horizontal { get; } = new(); + + /// + /// Replaces every motion-vector distribution with state copied from another context. + /// + /// The motion-vector context state to copy. + public void CopyFrom(Av1MotionVectorContext source) + { + this.Joint.CopyFrom(source.Joint); + this.Vertical.CopyFrom(source.Vertical); + this.Horizontal.CopyFrom(source.Horizontal); + } + + /// + /// Resets every observation count used to adapt the motion-vector distributions. + /// + public void ResetUpdateCounts() + { + this.Joint.ResetUpdateCount(); + this.Vertical.ResetUpdateCounts(); + this.Horizontal.ResetUpdateCounts(); + } + + /// + /// Reads a motion-vector delta relative to a spatially derived reference. + /// + /// The tile range decoder. + /// The reference motion vector. + /// The fractional precision allowed by the current frame. + /// The decoded motion vector in one-eighth-sample units. + public Av1MotionVector Read(ref Av1SymbolReader reader, Av1MotionVector reference, Av1MotionVectorPrecision precision) + { + int jointType = reader.ReadSymbol(this.Joint); + + // Joint values 1 and 3 carry a horizontal delta; values 2 and 3 carry a vertical delta. Reading only the + // signaled components preserves the normative entropy-symbol order and leaves zero components unadapted. + int row = jointType >= 2 ? this.Vertical.Read(ref reader, precision) : 0; + int column = (jointType & 1) != 0 ? this.Horizontal.Read(ref reader, precision) : 0; + + return reference + new Av1MotionVector(row, column); + } + + /// + /// Writes an integer displacement vector relative to a spatially derived reference. + /// + /// The tile range encoder. + /// The displacement vector to encode. + /// The spatially derived reference vector. + public void Write(Av1SymbolWriter writer, Av1MotionVector value, Av1MotionVector reference) + { + int row = value.Row - reference.Row; + int column = value.Column - reference.Column; + + // Bit zero signals a horizontal delta and bit one signals a vertical delta, producing the four normative + // zero/horizontal/vertical/both joint symbols without a lookup. + int jointType = (row != 0 ? 2 : 0) | (column != 0 ? 1 : 0); + + writer.WriteSymbol(jointType, this.Joint); + if (row != 0) + { + this.Vertical.Write(writer, row); + } + + if (column != 0) + { + this.Horizontal.Write(writer, column); + } + } + + /// + /// Owns the adaptive magnitude distributions for one motion-vector component. + /// + public sealed class Component + { + /// + /// Gets the distribution selecting the magnitude class of a nonzero component. + /// + public Av1Distribution MagnitudeClass { get; } = new(28672, 30976, 31858, 32320, 32551, 32656, 32740, 32757, 32762, 32767); + + /// + /// Gets the fractional distributions selected by the two class-zero integer offsets. + /// + public Av1Distribution[] ClassZeroFractional { get; } = + [ + new(16384, 24576, 26624), + new(12288, 21248, 24128) + ]; + + /// + /// Gets the fractional distribution used by nonzero magnitude classes. + /// + public Av1Distribution Fractional { get; } = new(8192, 17408, 21248); + + /// + /// Gets the distribution selecting the sign of a nonzero component. + /// + public Av1Distribution Sign { get; } = new(16384); + + /// + /// Gets the eighth-sample distribution used by class-zero magnitudes. + /// + public Av1Distribution ClassZeroHighPrecision { get; } = new(20480); + + /// + /// Gets the eighth-sample distribution used by nonzero magnitude classes. + /// + public Av1Distribution HighPrecision { get; } = new(16384); + + /// + /// Gets the distribution selecting either of the two class-zero integer magnitude offsets. + /// + public Av1Distribution ClassZero { get; } = new(27648); + + /// + /// Gets the binary distributions that reconstruct larger integer magnitude offsets from least to most significant bit. + /// + public Av1Distribution[] OffsetBits { get; } = + [ + new(17408), new(17920), new(18944), new(20480), new(22528), + new(24576), new(28672), new(29952), new(29952), new(30720) + ]; + + /// + /// Replaces every component distribution with state copied from another component. + /// + /// The component state to copy. + public void CopyFrom(Component source) + { + this.MagnitudeClass.CopyFrom(source.MagnitudeClass); + + for (int offset = 0; offset < this.ClassZeroFractional.Length; offset++) + { + this.ClassZeroFractional[offset].CopyFrom(source.ClassZeroFractional[offset]); + } + + this.Fractional.CopyFrom(source.Fractional); + this.Sign.CopyFrom(source.Sign); + this.ClassZeroHighPrecision.CopyFrom(source.ClassZeroHighPrecision); + this.HighPrecision.CopyFrom(source.HighPrecision); + this.ClassZero.CopyFrom(source.ClassZero); + + for (int bit = 0; bit < this.OffsetBits.Length; bit++) + { + this.OffsetBits[bit].CopyFrom(source.OffsetBits[bit]); + } + } + + /// + /// Resets every observation count used to adapt one component's distributions. + /// + public void ResetUpdateCounts() + { + this.MagnitudeClass.ResetUpdateCount(); + + for (int offset = 0; offset < this.ClassZeroFractional.Length; offset++) + { + this.ClassZeroFractional[offset].ResetUpdateCount(); + } + + this.Fractional.ResetUpdateCount(); + this.Sign.ResetUpdateCount(); + this.ClassZeroHighPrecision.ResetUpdateCount(); + this.HighPrecision.ResetUpdateCount(); + this.ClassZero.ResetUpdateCount(); + + for (int bit = 0; bit < this.OffsetBits.Length; bit++) + { + this.OffsetBits[bit].ResetUpdateCount(); + } + } + + /// + /// Reads one signed motion-vector component at the requested precision. + /// + /// The tile range decoder. + /// The fractional precision allowed by the current frame. + /// The signed component in one-eighth-sample units. + public int Read(ref Av1SymbolReader reader, Av1MotionVectorPrecision precision) + { + bool isNegative = reader.ReadSymbol(this.Sign) != 0; + int magnitudeClass = reader.ReadSymbol(this.MagnitudeClass); + bool isClassZero = magnitudeClass == 0; + int integerOffset; + int magnitudeBase; + + if (isClassZero) + { + integerOffset = reader.ReadSymbol(this.ClassZero); + magnitudeBase = 0; + } + else + { + int bitCount = magnitudeClass + ClassZeroBitCount - 1; + integerOffset = 0; + + for (int bit = 0; bit < bitCount; bit++) + { + // AV1 transmits the integer offset least-significant bit first, with an independently adapting + // distribution for every bit position. + integerOffset |= reader.ReadSymbol(this.OffsetBits[bit]) << bit; + } + + // Class one uses a base of two whole samples, or sixteen eighth-sample units, and every later class doubles + // that base. CLASS0_SIZE shifted by class + 2 expresses the same scale directly in eighth-sample units. + magnitudeBase = ClassZeroSize << (magnitudeClass + 2); + } + + int fractional; + int highPrecision; + + if (precision != Av1MotionVectorPrecision.Integer) + { + // Class-zero magnitudes select one of two fractional CDFs using the already decoded integer offset; + // larger classes share one fractional CDF because their expanded integer range supplies the context. + Av1Distribution fractionalDistribution = isClassZero ? this.ClassZeroFractional[integerOffset] : this.Fractional; + fractional = reader.ReadSymbol(fractionalDistribution); + + // Quarter-sample motion omits the eighth-sample symbol. The normative implicit one, combined with the + // final increment below, constrains the result to even one-eighth-sample units. + highPrecision = precision == Av1MotionVectorPrecision.EighthSample + ? reader.ReadSymbol(isClassZero ? this.ClassZeroHighPrecision : this.HighPrecision) + : 1; + } + else + { + // Integer motion omits both fractional symbols. The implicit maximum values make the low three bits + // all one before the final increment, constraining the result to whole-sample multiples of eight. + fractional = 3; + highPrecision = 1; + } + + // The entropy syntax represents magnitude minus one. Integer offset occupies bits three and above, + // fractional occupies bits one and two, and high precision occupies bit zero, all in one-eighth-sample units. + int magnitude = magnitudeBase + ((integerOffset << 3) | (fractional << 1) | highPrecision) + 1; + + return isNegative ? -magnitude : magnitude; + } + + /// + /// Writes one signed integer-precision component. + /// + /// The tile range encoder. + /// The nonzero component in one-eighth-sample units. + public void Write(Av1SymbolWriter writer, int value) + { + int magnitude = Math.Abs(value); + DebugGuard.IsTrue(magnitude > 0 && (magnitude & 7) == 0, "Displacement-vector components must use whole-sample precision."); + + // Class zero contains the two whole-sample magnitudes 8 and 16. Above it, the highest set bit of magnitude + // minus one selects the doubling range; subtracting three converts the eighth-sample bit index to the class. + int magnitudeClass = magnitude <= (ClassZeroSize << 3) ? 0 : Av1Math.MostSignificantBit((uint)(magnitude - 1)) - 3; + DebugGuard.MustBeLessThan(magnitudeClass, MagnitudeClassCount, nameof(magnitudeClass)); + writer.WriteSymbol(value < 0, this.Sign); + writer.WriteSymbol(magnitudeClass, this.MagnitudeClass); + + if (magnitudeClass == 0) + { + writer.WriteSymbol((magnitude >> 3) - 1, this.ClassZero); + return; + } + + // Remove the class base and the implicit low-bit value 7 plus the final one before coding the remaining + // whole-sample offset least-significant bit first. + int magnitudeBase = ClassZeroSize << (magnitudeClass + 2); + int integerOffset = (magnitude - magnitudeBase - 8) >> 3; + + for (int bit = 0; bit < magnitudeClass; bit++) + { + // The decoder reconstructs offsets least-significant bit first, so each adaptive bit model must be + // updated in the same order during encoding. + writer.WriteSymbol(((integerOffset >> bit) & 1) != 0, this.OffsetBits[bit]); + } + } + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolContextHelper.cs b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolContextHelper.cs index 00c9205a9..22aeb0c54 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolContextHelper.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolContextHelper.cs @@ -12,6 +12,41 @@ namespace SixLabors.ImageSharp.Formats.Heif.Av1.Entropy; /// internal static class Av1SymbolContextHelper { + /// + /// The bit offset of the global-motion decision context in a packed inter-mode context. + /// + private const int GlobalMvContextOffset = 3; + + /// + /// The bit offset of the spatial reference-motion-vector context in a packed inter-mode context. + /// + private const int RefMvContextOffset = 4; + + /// + /// The low-three-bit mask containing the new-motion-vector context. + /// + private const int NewMvContextMask = (1 << GlobalMvContextOffset) - 1; + + /// + /// The mask selecting the two-value global-motion context from its single packed bit. + /// + private const int ZeroMvContextMask = (1 << (RefMvContextOffset - GlobalMvContextOffset)) - 1; + + /// + /// The high-nibble mask containing the spatial reference-motion-vector context. + /// + private const int RefMvContextMask = (1 << (8 - RefMvContextOffset)) - 1; + + /// + /// The weight at which AV1 classifies a reference-motion-vector candidate as a strong spatial match. + /// + private const int ReferenceCategoryLevel = 640; + + /// + /// The number of interpolation filters selectable by per-block switchable syntax. + /// + private const int SwitchableInterpolationFilterCount = 3; + /// /// The number of transform types represented by each flattened transform-set row. /// @@ -624,6 +659,271 @@ internal static class Av1SymbolContextHelper return 0; } + /// + /// Gets the block reference-mode context from the immediately above and left blocks. + /// + /// The above block, or at a tile boundary. + /// The left block, or at a tile boundary. + /// The context in the inclusive range zero through four. + public static int GetReferenceModeContext(Av1BlockModeInfo? above, Av1BlockModeInfo? left) + { + // Libaom first classifies whether each neighbor uses a second inter reference. Single neighbors then contribute + // their forward/backward direction, while intra neighbors take the same branch as a non-forward reference. + if (above is not null && left is not null) + { + bool aboveIsCompound = above.ReferenceFrames[1] > Av1ReferenceFrameType.Intra; + bool leftIsCompound = left.ReferenceFrames[1] > Av1ReferenceFrameType.Intra; + + if (!aboveIsCompound && !leftIsCompound) + { + bool aboveIsBackward = above.ReferenceFrames[0] >= Av1ReferenceFrameType.Backward; + bool leftIsBackward = left.ReferenceFrames[0] >= Av1ReferenceFrameType.Backward; + + return aboveIsBackward == leftIsBackward ? 0 : 1; + } + + if (!aboveIsCompound) + { + bool aboveIsBackward = above.ReferenceFrames[0] >= Av1ReferenceFrameType.Backward; + bool aboveIsIntra = above.ReferenceFrames[0] <= Av1ReferenceFrameType.Intra; + + return 2 + (aboveIsBackward || aboveIsIntra ? 1 : 0); + } + + if (!leftIsCompound) + { + bool leftIsBackward = left.ReferenceFrames[0] >= Av1ReferenceFrameType.Backward; + bool leftIsIntra = left.ReferenceFrames[0] <= Av1ReferenceFrameType.Intra; + + return 2 + (leftIsBackward || leftIsIntra ? 1 : 0); + } + + return 4; + } + + Av1BlockModeInfo? neighbor = above ?? left; + + if (neighbor is not null) + { + bool isCompound = neighbor.ReferenceFrames[1] > Av1ReferenceFrameType.Intra; + + if (isCompound) + { + return 3; + } + + return neighbor.ReferenceFrames[0] >= Av1ReferenceFrameType.Backward ? 1 : 0; + } + + // With no spatial votes, AV1 uses the neutral single-versus-compound context rather than context zero. + return 1; + } + + /// + /// Gets the switchable interpolation-filter context for one prediction direction. + /// + /// The current inter block. + /// The above block, or at a tile boundary. + /// The left block, or at a tile boundary. + /// Zero for the vertical filter or one for the horizontal filter. + /// The context in the inclusive range zero through fifteen. + public static int GetSwitchableInterpolationContext( + Av1BlockModeInfo modeInfo, + Av1BlockModeInfo? above, + Av1BlockModeInfo? left, + int direction) + { + const int filterContextCount = SwitchableInterpolationFilterCount + 1; + const int horizontalContextOffset = filterContextCount * 2; + ReadOnlySpan referenceFrames = modeInfo.ReferenceFrames; + Av1ReferenceFrameType primaryReference = referenceFrames[0]; + bool isCompound = referenceFrames[1] > Av1ReferenceFrameType.Intra; + + // The sixteen rows are laid out as single vertical, compound vertical, single horizontal, then compound + // horizontal, with four neighbor states in each group. + int context = (isCompound ? filterContextCount : 0) + (direction * horizontalContextOffset); + int leftFilter = GetReferenceInterpolationFilterContext(left, primaryReference, direction); + int aboveFilter = GetReferenceInterpolationFilterContext(above, primaryReference, direction); + + if (leftFilter == aboveFilter) + { + return context + leftFilter; + } + + // The fourth neighbor state is not a selectable Bilinear filter. It is the value libaom uses when a neighbor + // does not share the current primary reference, and when two contributing neighbors selected different filters. + if (leftFilter == SwitchableInterpolationFilterCount) + { + return context + aboveFilter; + } + + if (aboveFilter == SwitchableInterpolationFilterCount) + { + return context + leftFilter; + } + + return context + SwitchableInterpolationFilterCount; + } + + /// + /// Gets the new-motion-vector decision context from a packed single-reference inter-mode context. + /// + /// The packed mode context produced by reference-motion-vector candidate analysis. + /// For a valid packed mode context, the context in the inclusive range zero through five. + public static int GetNewMvContext(int modeContext) => modeContext & NewMvContextMask; + + /// + /// Gets the global-motion decision context from a packed single-reference inter-mode context. + /// + /// The packed mode context produced by reference-motion-vector candidate analysis. + /// The context in the inclusive range zero through one. + public static int GetZeroMvContext(int modeContext) => (modeContext >> GlobalMvContextOffset) & ZeroMvContextMask; + + /// + /// Gets the spatial reference-motion-vector decision context from a packed single-reference inter-mode context. + /// + /// The packed mode context produced by reference-motion-vector candidate analysis. + /// For a valid packed mode context, the context in the inclusive range zero through five. + public static int GetRefMvContext(int modeContext) => (modeContext >> RefMvContextOffset) & RefMvContextMask; + + /// + /// Gets the dynamic reference-list context for two adjacent motion-vector candidates. + /// + /// The candidate weights in dynamic reference-list order. + /// The zero-based index of the first candidate in the pair. + /// The context in the inclusive range zero through two. + public static int GetDrlContext(ReadOnlySpan referenceWeights, int referenceIndex) + { + int currentWeight = referenceWeights[referenceIndex]; + int nextWeight = referenceWeights[referenceIndex + 1]; + + // Candidate weights at or above the reference-category threshold carry a strong spatial match. The four + // normative pairings use context zero for strong/strong and weak/strong, one for strong/weak, and two for weak/weak. + if (currentWeight >= ReferenceCategoryLevel && nextWeight >= ReferenceCategoryLevel) + { + return 0; + } + + if (currentWeight >= ReferenceCategoryLevel && nextWeight < ReferenceCategoryLevel) + { + return 1; + } + + return currentWeight < ReferenceCategoryLevel && nextWeight < ReferenceCategoryLevel ? 2 : 0; + } + + /// + /// Counts the reference-frame labels used by the immediately above and left inter blocks. + /// + /// The above block, or at a tile boundary. + /// The left block, or at a tile boundary. + /// The eight-entry reference-count destination indexed by . + public static void CollectNeighborReferenceCounts(Av1BlockModeInfo? above, Av1BlockModeInfo? left, Span referenceCounts) + { + // The caller reuses fixed inline storage across blocks. Clearing all eight entries matches libaom's + // av1_collect_neighbors_ref_counts and prevents an unavailable neighbor from retaining an earlier block's vote. + referenceCounts.Clear(); + + if (above is not null) + { + AddNeighborReferenceCounts(above, referenceCounts); + } + + if (left is not null) + { + AddNeighborReferenceCounts(left, referenceCounts); + } + } + + /// + /// Gets the context that selects a backward instead of forward single reference. + /// + /// The neighboring reference counts indexed by . + /// The context in the inclusive range zero through two. + public static int GetSingleReferenceBackwardContext(ReadOnlySpan referenceCounts) + { + int forwardCount = referenceCounts[(int)Av1ReferenceFrameType.Last] + + referenceCounts[(int)Av1ReferenceFrameType.Last2] + + referenceCounts[(int)Av1ReferenceFrameType.Last3] + + referenceCounts[(int)Av1ReferenceFrameType.Golden]; + + int backwardCount = referenceCounts[(int)Av1ReferenceFrameType.Backward] + + referenceCounts[(int)Av1ReferenceFrameType.Alternate2] + + referenceCounts[(int)Av1ReferenceFrameType.Alternate]; + + return GetBinaryReferenceContext(forwardCount, backwardCount); + } + + /// + /// Gets the context that selects Alternate instead of Backward or Alternate2. + /// + /// The neighboring reference counts indexed by . + /// The context in the inclusive range zero through two. + public static int GetSingleReferenceAlternateContext(ReadOnlySpan referenceCounts) + { + int backwardOrAlternate2Count = referenceCounts[(int)Av1ReferenceFrameType.Backward] + + referenceCounts[(int)Av1ReferenceFrameType.Alternate2]; + + int alternateCount = referenceCounts[(int)Av1ReferenceFrameType.Alternate]; + + return GetBinaryReferenceContext(backwardOrAlternate2Count, alternateCount); + } + + /// + /// Gets the context that selects Last3 or Golden instead of Last or Last2. + /// + /// The neighboring reference counts indexed by . + /// The context in the inclusive range zero through two. + public static int GetSingleReferenceLast3OrGoldenContext(ReadOnlySpan referenceCounts) + { + int lastOrLast2Count = referenceCounts[(int)Av1ReferenceFrameType.Last] + + referenceCounts[(int)Av1ReferenceFrameType.Last2]; + + int last3OrGoldenCount = referenceCounts[(int)Av1ReferenceFrameType.Last3] + + referenceCounts[(int)Av1ReferenceFrameType.Golden]; + + return GetBinaryReferenceContext(lastOrLast2Count, last3OrGoldenCount); + } + + /// + /// Gets the context that selects Last2 instead of Last. + /// + /// The neighboring reference counts indexed by . + /// The context in the inclusive range zero through two. + public static int GetSingleReferenceLast2Context(ReadOnlySpan referenceCounts) + { + int lastCount = referenceCounts[(int)Av1ReferenceFrameType.Last]; + int last2Count = referenceCounts[(int)Av1ReferenceFrameType.Last2]; + + return GetBinaryReferenceContext(lastCount, last2Count); + } + + /// + /// Gets the context that selects Golden instead of Last3. + /// + /// The neighboring reference counts indexed by . + /// The context in the inclusive range zero through two. + public static int GetSingleReferenceGoldenContext(ReadOnlySpan referenceCounts) + { + int last3Count = referenceCounts[(int)Av1ReferenceFrameType.Last3]; + int goldenCount = referenceCounts[(int)Av1ReferenceFrameType.Golden]; + + return GetBinaryReferenceContext(last3Count, goldenCount); + } + + /// + /// Gets the context that selects Alternate2 instead of Backward. + /// + /// The neighboring reference counts indexed by . + /// The context in the inclusive range zero through two. + public static int GetSingleReferenceAlternate2Context(ReadOnlySpan referenceCounts) + { + int backwardCount = referenceCounts[(int)Av1ReferenceFrameType.Backward]; + int alternate2Count = referenceCounts[(int)Av1ReferenceFrameType.Alternate2]; + + return GetBinaryReferenceContext(backwardCount, alternate2Count); + } + /// /// Gets the temporal segment-prediction context from the immediately above and left blocks. /// @@ -719,4 +1019,65 @@ internal static class Av1SymbolContextHelper return max - (diff + 1); } } + + /// + /// Adds one decoded inter neighbor's primary and optional secondary reference votes. + /// + /// The decoded neighboring block. + /// The reference counts updated in place. + private static void AddNeighborReferenceCounts(Av1BlockModeInfo modeInfo, Span referenceCounts) + { + ReadOnlySpan referenceFrames = modeInfo.ReferenceFrames; + + if (referenceFrames[0] <= Av1ReferenceFrameType.Intra) + { + return; + } + + referenceCounts[(int)referenceFrames[0]]++; + + // A current block may use one reference, but the conditioning neighbors may be compound blocks. Libaom counts + // both labels so later single-reference decisions remain bit-exact when compound support is enabled. + if (referenceFrames[1] > Av1ReferenceFrameType.Intra) + { + referenceCounts[(int)referenceFrames[1]]++; + } + } + + /// + /// Converts neighboring votes for a binary reference-tree decision to its three-state AV1 context. + /// + /// The votes for the branch represented by symbol zero. + /// The votes for the branch represented by symbol one. + /// One for tied votes, zero when symbol one has more votes, or two when symbol zero has more votes. + private static int GetBinaryReferenceContext(int zeroSymbolCount, int oneSymbolCount) + => zeroSymbolCount == oneSymbolCount ? 1 : zeroSymbolCount < oneSymbolCount ? 0 : 2; + + /// + /// Gets one neighbor's interpolation-filter contribution for the requested reference and direction. + /// + /// The decoded neighboring block, or when unavailable. + /// The current block's primary reference. + /// Zero for the vertical filter or one for the horizontal filter. + /// The selected filter index, or three when the neighbor does not contribute. + private static int GetReferenceInterpolationFilterContext( + Av1BlockModeInfo? modeInfo, + Av1ReferenceFrameType referenceFrame, + int direction) + { + if (modeInfo is null) + { + return SwitchableInterpolationFilterCount; + } + + ReadOnlySpan referenceFrames = modeInfo.ReferenceFrames; + + // A compound neighbor contributes when either of its references matches the current primary reference. + if (referenceFrames[0] != referenceFrame && referenceFrames[1] != referenceFrame) + { + return SwitchableInterpolationFilterCount; + } + + return (int)modeInfo.InterpolationFilters[direction]; + } } diff --git a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolDecoder.cs b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolDecoder.cs index c3a0de0ae..cf02b6f4f 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolDecoder.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolDecoder.cs @@ -4,6 +4,7 @@ using SixLabors.ImageSharp.Formats.Heif.Av1.Motion; using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.ChromaFromLuma; +using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; using SixLabors.ImageSharp.Formats.Heif.Av1.Transform; @@ -286,7 +287,16 @@ internal ref struct Av1SymbolDecoder /// The spatially derived reference vector. /// The decoded displacement vector in one-eighth-sample units. public Av1MotionVector ReadDisplacementVector(Av1MotionVector reference) - => this.context.DisplacementVector.Read(ref this.reader, reference); + => this.context.DisplacementVector.Read(ref this.reader, reference, Av1MotionVectorPrecision.Integer); + + /// + /// Reads a normal inter-prediction motion vector relative to a selected reference candidate. + /// + /// The selected reference motion vector. + /// The fractional precision allowed by the current frame. + /// The decoded motion vector in one-eighth-sample units. + public Av1MotionVector ReadMotionVector(Av1MotionVector reference, Av1MotionVectorPrecision precision) + => this.context.MotionVector.Read(ref this.reader, reference, precision); /// /// Reads a complete block partition type from the selected partition context. @@ -358,13 +368,42 @@ internal ref struct Av1SymbolDecoder /// The decoded intra luma prediction mode. public Av1PredictionMode ReadInterFrameYMode(Av1BlockSize blockSize) { - // AV1 section 9.3 groups blocks by the smaller base-two dimension in 4x4 units, capped at group three. - // Calculating it from the existing logarithms exactly matches libaom's size_group_lookup without another table. - int sizeGroup = Math.Min(3, Math.Min(blockSize.Get4x4WidthLog2(), blockSize.Get4x4HeightLog2())); + int sizeGroup = blockSize.GetSizeGroup(); ref Av1SymbolReader r = ref this.reader; return (Av1PredictionMode)r.ReadSymbol(this.context.FrameYMode[sizeGroup]); } + /// + /// Reads whether a single-reference inter block uses inter-intra prediction. + /// + /// The decoded block size that selects the inter-intra flag distribution. + /// when an intra predictor is blended with the inter predictor. + public bool ReadIsInterIntra(Av1BlockSize blockSize) + { + int sizeGroup = blockSize.GetSizeGroup(); + ref Av1SymbolReader r = ref this.reader; + return r.ReadSymbol(this.context.InterIntra[sizeGroup]) != 0; + } + + /// + /// Reads the motion model selected for an eligible single-reference inter block. + /// + /// The decoded block size that selects the motion-mode distribution. + /// + /// A value indicating whether the block may select Warped in addition to Simple Translation and OBMC. + /// + /// The decoded motion mode. + public Av1MotionMode ReadMotionMode(Av1BlockSize blockSize, bool allowWarpedMotion) + { + ref Av1SymbolReader r = ref this.reader; + + // AV1 uses a separate binary CDF when Warped is ineligible; reading the first two leaves from the three-way + // CDF would use different probabilities and desynchronize the range decoder even when Simple is selected. + return allowWarpedMotion + ? (Av1MotionMode)r.ReadSymbol(this.context.MotionMode[(int)blockSize]) + : (Av1MotionMode)r.ReadSymbol(this.context.Obmc[(int)blockSize]); + } + /// /// Reads whether an inter-frame block uses inter prediction. /// @@ -376,6 +415,135 @@ internal ref struct Av1SymbolDecoder return r.ReadSymbol(this.context.IntraInter[context]) != 0; } + /// + /// Reads whether an inter block uses compound-reference instead of single-reference prediction. + /// + /// The spatial block reference-mode context in the inclusive range zero through four. + /// for compound-reference prediction; otherwise, . + public bool ReadIsCompoundReference(int context) + { + ref Av1SymbolReader r = ref this.reader; + + return r.ReadSymbol(this.context.CompInter[context]) != 0; + } + + /// + /// Reads one per-block interpolation filter selected by a switchable frame. + /// + /// The reference, direction, and neighbor filter context. + /// The selected Regular, Smooth, or Sharp interpolation filter. + public Av1InterpolationFilter ReadSwitchableInterpolationFilter(int context) + { + ref Av1SymbolReader r = ref this.reader; + + return (Av1InterpolationFilter)r.ReadSymbol(this.context.SwitchableInterpolation[context]); + } + + /// + /// Reads the prediction mode for a single-reference inter block. + /// + /// The packed mode context produced by reference-motion-vector candidate analysis. + /// The selected new, global, nearest, or near motion-vector mode. + public Av1PredictionMode ReadInterMode(int modeContext) + { + ref Av1SymbolReader r = ref this.reader; + int newMvContext = Av1SymbolContextHelper.GetNewMvContext(modeContext); + + // AV1 assigns symbol zero to the NEWMV leaf and symbol one to the rest of the tree. Returning at the leaf is + // required both for the selected mode and to avoid consuming the unrelated lower decisions. + if (r.ReadSymbol(this.context.NewMv[newMvContext]) == 0) + { + return Av1PredictionMode.NewMotionVector; + } + + int zeroMvContext = Av1SymbolContextHelper.GetZeroMvContext(modeContext); + if (r.ReadSymbol(this.context.ZeroMv[zeroMvContext]) == 0) + { + return Av1PredictionMode.GlobalMotionVector; + } + + // The final zero symbol selects the nearest spatial candidate; one selects the near candidate and may be + // followed by dynamic-reference-list syntax when more than one near candidate is available. + int refMvContext = Av1SymbolContextHelper.GetRefMvContext(modeContext); + return r.ReadSymbol(this.context.RefMv[refMvContext]) == 0 + ? Av1PredictionMode.NearestMotionVector + : Av1PredictionMode.NearMotionVector; + } + + /// + /// Reads one dynamic reference-list decision for adjacent motion-vector candidates. + /// + /// The candidate-weight context in the inclusive range zero through two. + /// + /// when selection advances past the current candidate; otherwise, . + /// + public bool ReadDrl(int context) + { + ref Av1SymbolReader r = ref this.reader; + return r.ReadSymbol(this.context.Drl[context]) != 0; + } + + /// + /// Reads whether a single-reference block selects the backward-reference group. + /// + /// The neighboring forward-versus-backward vote context. + /// for a backward reference; otherwise, . + public bool ReadSingleReferenceIsBackward(int context) + => this.ReadSingleReferenceDecision(context, decision: 0); + + /// + /// Reads whether a backward single-reference block selects Alternate. + /// + /// The neighboring Backward-or-Alternate2-versus-Alternate vote context. + /// for Alternate; otherwise, . + public bool ReadSingleReferenceIsAlternate(int context) + => this.ReadSingleReferenceDecision(context, decision: 1); + + /// + /// Reads whether a forward single-reference block selects the Last3-or-Golden group. + /// + /// The neighboring near-forward-versus-far-forward vote context. + /// for Last3 or Golden; otherwise, . + public bool ReadSingleReferenceIsLast3OrGolden(int context) + => this.ReadSingleReferenceDecision(context, decision: 2); + + /// + /// Reads whether a near-forward single-reference block selects Last2. + /// + /// The neighboring Last-versus-Last2 vote context. + /// for Last2; otherwise, . + public bool ReadSingleReferenceIsLast2(int context) + => this.ReadSingleReferenceDecision(context, decision: 3); + + /// + /// Reads whether a far-forward single-reference block selects Golden. + /// + /// The neighboring Last3-versus-Golden vote context. + /// for Golden; otherwise, . + public bool ReadSingleReferenceIsGolden(int context) + => this.ReadSingleReferenceDecision(context, decision: 4); + + /// + /// Reads whether a non-Alternate backward single-reference block selects Alternate2. + /// + /// The neighboring Backward-versus-Alternate2 vote context. + /// for Alternate2; otherwise, . + public bool ReadSingleReferenceIsAlternate2(int context) + => this.ReadSingleReferenceDecision(context, decision: 5); + + /// + /// Reads one binary decision from the single-reference selection tree. + /// + /// The neighboring reference-vote context. + /// The zero-based tree decision matching one single_ref_cdf column. + /// when the decision selects symbol one; otherwise, . + private bool ReadSingleReferenceDecision(int context, int decision) + { + ref Av1SymbolReader r = ref this.reader; + + return r.ReadSymbol(this.context.SingleReference[context][decision]) != 0; + } + /// /// Reads a chroma intra prediction mode conditioned on the luma mode and chroma-from-luma availability. /// diff --git a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolEncoder.cs b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolEncoder.cs index aa1fe9cc7..b00cf8da9 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolEncoder.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolEncoder.cs @@ -24,7 +24,7 @@ internal class Av1SymbolEncoder : IDisposable /// /// The tile-adaptive integer displacement-vector context. /// - private readonly Av1DisplacementVectorContext displacementVector = new(); + private readonly Av1MotionVectorContext displacementVector = new(); /// /// The tile-adaptive partition-type distributions. diff --git a/src/ImageSharp/Formats/Heif/Av1/Motion/Av1GlobalMotionParameters.cs b/src/ImageSharp/Formats/Heif/Av1/Motion/Av1GlobalMotionParameters.cs index 4f86ad80c..6e068dee1 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Motion/Av1GlobalMotionParameters.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Motion/Av1GlobalMotionParameters.cs @@ -3,6 +3,7 @@ using System.Numerics; using System.Runtime.CompilerServices; +using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; namespace SixLabors.ImageSharp.Formats.Heif.Av1.Motion; @@ -126,6 +127,65 @@ internal struct Av1GlobalMotionParameters set => this.matrix[index] = value; } + /// + /// Gets the translational motion vector represented by this model at the center of a coding block. + /// + /// + /// A value indicating whether motion vectors may retain one-eighth-sample precision. + /// + /// The coding block size. + /// The block origin in 4x4 mode-information units. + /// + /// A value indicating whether the result is rounded to an integer-sample displacement. + /// + /// The global motion vector in one-eighth-sample units. + public readonly Av1MotionVector GetMotionVector( + bool allowHighPrecisionMotionVector, + Av1BlockSize blockSize, + Point modeInfoPosition, + bool forceIntegerMotionVector) + { + if (this.Type == Av1GlobalMotionType.Identity) + { + return default; + } + + int row; + int column; + if (this.Type == Av1GlobalMotionType.Translation) + { + // AV1 accidentally assigns the horizontal translation parameter to the row component and the vertical + // parameter to the column component. Decoders preserve that published bitstream behavior for conformance. + row = this.matrix[0] >> (ModelPrecisionBits - 3); + column = this.matrix[1] >> (ModelPrecisionBits - 3); + } + else + { + int blockCenterX = (modeInfoPosition.X << Av1Constants.ModeInfoSizeLog2) + (blockSize.GetWidth() >> 1) - 1; + int blockCenterY = (modeInfoPosition.Y << Av1Constants.ModeInfoSizeLog2) + (blockSize.GetHeight() >> 1) - 1; + int horizontal = ((this.matrix[2] - ModelScale) * blockCenterX) + + (this.matrix[3] * blockCenterY) + + this.matrix[0]; + + int vertical = (this.matrix[4] * blockCenterX) + + ((this.matrix[5] - ModelScale) * blockCenterY) + + this.matrix[1]; + + int precisionBits = allowHighPrecisionMotionVector ? ModelPrecisionBits - 3 : ModelPrecisionBits - 2; + column = Av1Math.RoundPowerOf2Signed(horizontal, precisionBits); + row = Av1Math.RoundPowerOf2Signed(vertical, precisionBits); + if (!allowHighPrecisionMotionVector) + { + column *= 2; + row *= 2; + } + } + + return new Av1MotionVector(row, column).LowerPrecision( + allowHighPrecision: allowHighPrecisionMotionVector, + forceInteger: forceIntegerMotionVector); + } + /// /// Derives the reduced shear parameters and records whether the complete affine model is valid. /// diff --git a/src/ImageSharp/Formats/Heif/Av1/Motion/Av1IntraBlockCopy.cs b/src/ImageSharp/Formats/Heif/Av1/Motion/Av1IntraBlockCopy.cs index 06aea4e91..ff6653de8 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Motion/Av1IntraBlockCopy.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Motion/Av1IntraBlockCopy.cs @@ -77,7 +77,7 @@ internal static class Av1IntraBlockCopy ScanColumn(ref partitionInfo, -1, maximumColumnOffset, candidates, weights, ref candidateCount, ref processedColumns); } - if (HasTopRight(ref partitionInfo, superblockModeInfoSize)) + if (partitionInfo.HasTopRight(superblockModeInfoSize)) { AddBlock(ref partitionInfo, -1, width, tileInfo, candidates, weights, ref candidateCount); } @@ -396,56 +396,4 @@ internal static class Av1IntraBlockCopy length = lastSwap; } } - - /// - /// Determines whether the current partition is parsed after the block at its top-right search position. - /// - private static bool HasTopRight(ref Av1PartitionInfo partitionInfo, int superblockModeInfoSize) - { - int width = partitionInfo.ModeInfo.BlockSize.Get4x4WideCount(); - int height = partitionInfo.ModeInfo.BlockSize.Get4x4HighCount(); - int blockSize = Math.Max(width, height); - if (blockSize > 16) - { - return false; - } - - int row = partitionInfo.RowIndex & (superblockModeInfoSize - 1); - int column = partitionInfo.ColumnIndex & (superblockModeInfoSize - 1); - bool hasTopRight = !((row & blockSize) != 0 && (column & blockSize) != 0); - int traversalSize = blockSize; - while (traversalSize < superblockModeInfoSize) - { - if ((column & traversalSize) == 0) - { - break; - } - - if ((column & (traversalSize << 1)) != 0 && (row & (traversalSize << 1)) != 0) - { - hasTopRight = false; - break; - } - - traversalSize <<= 1; - } - - if (width < height && ((partitionInfo.ColumnIndex + width) & (height - 1)) != 0) - { - hasTopRight = true; - } - - if (width > height && (partitionInfo.RowIndex & (width - 1)) != 0) - { - hasTopRight = false; - } - - // The lower-left square of a vertical-A partition is decoded before its right-hand rectangle. - if (partitionInfo.Type == Av1PartitionType.VerticalA && width == height && (row & traversalSize) != 0) - { - hasTopRight = false; - } - - return hasTopRight; - } } diff --git a/src/ImageSharp/Formats/Heif/Av1/Motion/Av1MotionVariationCandidates.cs b/src/ImageSharp/Formats/Heif/Av1/Motion/Av1MotionVariationCandidates.cs new file mode 100644 index 000000000..3987ecd90 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Motion/Av1MotionVariationCandidates.cs @@ -0,0 +1,297 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit; +using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Motion; + +/// +/// Derives the neighboring-block state and fixed-capacity projection samples used to select an AV1 motion mode. +/// +internal sealed class Av1MotionVariationCandidates +{ + /// + /// The maximum number of neighboring motion samples retained for a local warped-motion projection. + /// + private const int ProjectionSampleCapacity = 8; + + /// + /// The largest neighbor step used by overlapping motion compensation, measured in 4x4 mode-information units. + /// + private const int MaximumNeighborStep = 16; + + /// + /// The number of fractional bits in an AV1 motion vector and warped-motion sample position. + /// + private const int MotionVectorSubpixelBits = 3; + + /// + /// Stores sample positions relative to the current block origin in one-eighth-sample units. + /// + private InlineArray8 sourcePoints; + + /// + /// Stores the corresponding reference-frame positions in one-eighth-sample units. + /// + private InlineArray8 referencePoints; + + /// + /// Gets the number of valid entries in and . + /// + public int Count { get; private set; } + + /// + /// Gets a value indicating whether an inter-coded block overlaps the current block's above or left edge. + /// + public bool HasOverlappableNeighbor { get; private set; } + + /// + /// Gets the retained current-frame sample positions in one-eighth-sample units relative to the current block. + /// + public ReadOnlySpan SourcePoints => this.sourcePoints[..this.Count]; + + /// + /// Gets the retained reference-frame sample positions in one-eighth-sample units relative to the current block. + /// + public ReadOnlySpan ReferencePoints => this.referencePoints[..this.Count]; + + /// + /// Derives the spatial state used to select Simple Translation, OBMC, or Warped motion for one inter block. + /// + /// The current block geometry and frame-wide decoded mode map. + /// The active tile boundaries. + /// The sequence-level superblock geometry. + /// The current frame dimensions. + /// The current block's primary canonical reference. + public void Build( + ref Av1PartitionInfo partitionInfo, + Av1TileInfo tileInfo, + ObuSequenceHeader sequenceHeader, + ObuFrameHeader frameHeader, + Av1ReferenceFrameType referenceFrame) + { + this.Count = 0; + this.CollectProjectionSamples(ref partitionInfo, tileInfo, sequenceHeader, frameHeader, referenceFrame); + this.HasOverlappableNeighbor = FindOverlappableNeighbor(ref partitionInfo, frameHeader); + } + + /// + /// Collects the at most eight spatial samples permitted by AV1's local warped-motion model. + /// + /// The current block geometry and frame-wide decoded mode map. + /// The active tile boundaries. + /// The sequence-level superblock geometry. + /// The current frame dimensions. + /// The current block's primary canonical reference. + private void CollectProjectionSamples( + ref Av1PartitionInfo partitionInfo, + Av1TileInfo tileInfo, + ObuSequenceHeader sequenceHeader, + ObuFrameHeader frameHeader, + Av1ReferenceFrameType referenceFrame) + { + Av1BlockSize blockSize = partitionInfo.ModeInfo.BlockSize; + int width = blockSize.Get4x4WideCount(); + int height = blockSize.Get4x4HighCount(); + int row = partitionInfo.RowIndex; + int column = partitionInfo.ColumnIndex; + bool includeTopLeft = true; + bool includeTopRight = true; + + if (partitionInfo.AvailableAbove) + { + Av1BlockModeInfo candidate = partitionInfo.SuperblockInfo.GetModeInfoAt(new Point(column, row - 1)); + int candidateWidth = candidate.BlockSize.Get4x4WideCount(); + if (width <= candidateWidth) + { + // A wider above block can also cover the diagonal search positions. The signed alignment offset + // prevents those positions from contributing the same block a second time. + int columnOffset = -column % candidateWidth; + includeTopLeft = columnOffset >= 0; + includeTopRight = columnOffset + candidateWidth <= width; + this.AddProjectionSample(candidate, referenceFrame, 0, -1, columnOffset, 1); + } + else + { + int end = Math.Min(width, frameHeader.ModeInfoColumnCount - column); + for (int index = 0; index < end && this.Count < ProjectionSampleCapacity; index += candidateWidth) + { + candidate = partitionInfo.SuperblockInfo.GetModeInfoAt(new Point(column + index, row - 1)); + candidateWidth = candidate.BlockSize.Get4x4WideCount(); + this.AddProjectionSample(candidate, referenceFrame, 0, -1, index, 1); + } + } + } + + if (partitionInfo.AvailableLeft && this.Count < ProjectionSampleCapacity) + { + Av1BlockModeInfo candidate = partitionInfo.SuperblockInfo.GetModeInfoAt(new Point(column - 1, row)); + int candidateHeight = candidate.BlockSize.Get4x4HighCount(); + if (height <= candidateHeight) + { + // The same alignment rule suppresses a duplicate top-left sample when one tall left block covers it. + int rowOffset = -row % candidateHeight; + includeTopLeft &= rowOffset >= 0; + this.AddProjectionSample(candidate, referenceFrame, rowOffset, 1, 0, -1); + } + else + { + int end = Math.Min(height, frameHeader.ModeInfoRowCount - row); + for (int index = 0; index < end && this.Count < ProjectionSampleCapacity; index += candidateHeight) + { + candidate = partitionInfo.SuperblockInfo.GetModeInfoAt(new Point(column - 1, row + index)); + candidateHeight = candidate.BlockSize.Get4x4HighCount(); + this.AddProjectionSample(candidate, referenceFrame, index, 1, 0, -1); + } + } + } + + if (includeTopLeft && partitionInfo.AvailableAbove && partitionInfo.AvailableLeft && this.Count < ProjectionSampleCapacity) + { + Av1BlockModeInfo candidate = partitionInfo.SuperblockInfo.GetModeInfoAt(new Point(column - 1, row - 1)); + this.AddProjectionSample(candidate, referenceFrame, 0, -1, 0, -1); + } + + int topRightRow = row - 1; + int topRightColumn = column + width; + bool topRightInsideTile = + topRightRow >= tileInfo.ModeInfoRowStart && + topRightRow < tileInfo.ModeInfoRowEnd && + topRightColumn >= tileInfo.ModeInfoColumnStart && + topRightColumn < tileInfo.ModeInfoColumnEnd; + + if (includeTopRight && + this.Count < ProjectionSampleCapacity && + partitionInfo.HasTopRight(sequenceHeader.SuperblockModeInfoSize) && + topRightInsideTile) + { + Av1BlockModeInfo candidate = partitionInfo.SuperblockInfo.GetModeInfoAt(new Point(topRightColumn, topRightRow)); + this.AddProjectionSample(candidate, referenceFrame, 0, -1, width, 1); + } + } + + /// + /// Determines whether an inter-coded neighbor covers either complete prediction edge of the current block. + /// + /// The current block geometry and frame-wide decoded mode map. + /// The current frame dimensions. + /// when an above or left inter block can contribute overlapping prediction. + private static bool FindOverlappableNeighbor(ref Av1PartitionInfo partitionInfo, ObuFrameHeader frameHeader) + { + Av1BlockSize blockSize = partitionInfo.ModeInfo.BlockSize; + int width = blockSize.Get4x4WideCount(); + int height = blockSize.Get4x4HighCount(); + int row = partitionInfo.RowIndex; + int column = partitionInfo.ColumnIndex; + + if (partitionInfo.AvailableAbove) + { + int endColumn = Math.Min(column + width, frameHeader.ModeInfoColumnCount); + for (int aboveColumn = column; aboveColumn < endColumn;) + { + Av1BlockModeInfo candidate = partitionInfo.SuperblockInfo.GetModeInfoAt(new Point(aboveColumn, row - 1)); + int step = Math.Min(candidate.BlockSize.Get4x4WideCount(), MaximumNeighborStep); + if (step == 1) + { + // AV1 treats a 4-sample-wide neighbor as one half of an 8-sample pair and reads the mode record + // attached to the pair's second cell before advancing across both cells. + aboveColumn &= ~1; + candidate = partitionInfo.SuperblockInfo.GetModeInfoAt(new Point(aboveColumn + 1, row - 1)); + step = 2; + } + + if (IsOverlappable(candidate)) + { + return true; + } + + aboveColumn += step; + } + } + + if (partitionInfo.AvailableLeft) + { + int endRow = Math.Min(row + height, frameHeader.ModeInfoRowCount); + for (int leftRow = row; leftRow < endRow;) + { + Av1BlockModeInfo candidate = partitionInfo.SuperblockInfo.GetModeInfoAt(new Point(column - 1, leftRow)); + int step = Math.Min(candidate.BlockSize.Get4x4HighCount(), MaximumNeighborStep); + if (step == 1) + { + // The vertical scan applies the corresponding 4-sample-high pairing rule. + leftRow &= ~1; + candidate = partitionInfo.SuperblockInfo.GetModeInfoAt(new Point(column - 1, leftRow + 1)); + step = 2; + } + + if (IsOverlappable(candidate)) + { + return true; + } + + leftRow += step; + } + } + + return false; + } + + /// + /// Appends one neighboring single-reference sample when it uses the current block's primary reference. + /// + /// The neighboring decoded block. + /// The current block's primary canonical reference. + /// The neighbor center row offset in 4x4 mode-information units. + /// The direction from the current block toward the neighbor on the vertical axis. + /// The neighbor center column offset in 4x4 mode-information units. + /// The direction from the current block toward the neighbor on the horizontal axis. + private void AddProjectionSample( + Av1BlockModeInfo candidate, + Av1ReferenceFrameType referenceFrame, + int rowOffset, + int rowSign, + int columnOffset, + int columnSign) + { + Span candidateReferences = candidate.ReferenceFrames; + if (candidateReferences[0] != referenceFrame || candidateReferences[1] != Av1ReferenceFrameType.None) + { + return; + } + + const int modeInfoSampleSize = 1 << Av1Constants.ModeInfoSizeLog2; + int sourceX = (columnOffset * modeInfoSampleSize) + (columnSign * (candidate.BlockSize.GetWidth() >> 1)) - 1; + int sourceY = (rowOffset * modeInfoSampleSize) + (rowSign * (candidate.BlockSize.GetHeight() >> 1)) - 1; + Point sourcePoint = new(sourceX << MotionVectorSubpixelBits, sourceY << MotionVectorSubpixelBits); + Av1MotionVector motionVector = candidate.MotionVectors[0]; + + // Neighbor centers and motion vectors share Q3 precision. Adding them directly produces the corresponding + // reference position without rounding away the fractional displacement needed by the projection solver. + this.sourcePoints[this.Count] = sourcePoint; + this.referencePoints[this.Count] = new Point(sourcePoint.X + motionVector.Column, sourcePoint.Y + motionVector.Row); + this.Count++; + } + + /// + /// Determines whether a decoded neighbor can participate in overlapping motion compensation. + /// + /// The neighboring decoded block. + /// for inter prediction or intra-block copy; otherwise, . + private static bool IsOverlappable(Av1BlockModeInfo candidate) + => candidate.UseIntraBlockCopy || candidate.ReferenceFrames[0] > Av1ReferenceFrameType.Intra; + + /// + /// Provides fixed storage for AV1's eight local warped-motion projection samples. + /// + /// The source or reference point type stored in the inline buffer. + [InlineArray(ProjectionSampleCapacity)] + private struct InlineArray8 + { + /// + /// The first element in the compiler-expanded inline buffer. + /// + private T element; + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Motion/Av1MotionVector.cs b/src/ImageSharp/Formats/Heif/Av1/Motion/Av1MotionVector.cs index de953309d..92d6009fa 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Motion/Av1MotionVector.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Motion/Av1MotionVector.cs @@ -8,6 +8,26 @@ namespace SixLabors.ImageSharp.Formats.Heif.Av1.Motion; /// internal readonly struct Av1MotionVector : IEquatable { + /// + /// The greatest absolute temporal distance used by AV1 motion-vector projection. + /// + public const int MaximumTemporalDistance = 31; + + /// + /// The reserved lower endpoint of the signed AV1 motion-vector domain. + /// + private const int LowerBound = -16384; + + /// + /// The exclusive upper endpoint of the signed AV1 motion-vector domain. + /// + private const int UpperBound = 16384; + + /// + /// The additional sixteen-sample border admitted while deriving spatial reference candidates, in one-eighth-sample units. + /// + private const int ReferenceBorder = 16 << 3; + /// /// Initializes a new instance of the struct. /// @@ -34,6 +54,22 @@ internal readonly struct Av1MotionVector : IEquatable /// public bool IsZero => this.Row == 0 && this.Column == 0; + /// + /// Gets a value indicating whether both components lie strictly between the two reserved AV1 endpoints. + /// + public bool IsValid => + this.Row > LowerBound && + this.Row < UpperBound && + this.Column > LowerBound && + this.Column < UpperBound; + + /// + /// Gets the reciprocal table used by AV1 temporal projection in fourteen-bit fixed-point precision. + /// + private static ReadOnlySpan ProjectionDivisors => + [0, 16384, 8192, 5461, 4096, 3276, 2730, 2340, 2048, 1820, 1638, 1489, 1365, 1260, 1170, 1092, + 1024, 963, 910, 862, 819, 780, 744, 712, 682, 655, 630, 606, 585, 564, 546, 528]; + /// /// Adds a component delta to this vector. /// @@ -59,6 +95,88 @@ internal readonly struct Av1MotionVector : IEquatable /// when either component differs; otherwise, . public static bool operator !=(Av1MotionVector left, Av1MotionVector right) => !left.Equals(right); + /// + /// Reduces this vector to the motion-vector precision selected by the current frame. + /// + /// + /// A value indicating whether one-eighth-sample precision may be retained. + /// + /// + /// A value indicating whether both components must be rounded to integer-sample precision. + /// + /// The precision-reduced vector. + public Av1MotionVector LowerPrecision(bool allowHighPrecision, bool forceInteger) + { + if (forceInteger) + { + return new(RoundToIntegerPrecision(this.Row), RoundToIntegerPrecision(this.Column)); + } + + if (allowHighPrecision) + { + return this; + } + + // Low precision removes the one-eighth-sample bit. Odd components move toward zero rather than rounding to + // the nearest even value, which is the normative lower_mv_precision behavior used by spatial and temporal MVs. + int row = (this.Row & 1) != 0 ? this.Row + (this.Row > 0 ? -1 : 1) : this.Row; + int column = (this.Column & 1) != 0 ? this.Column + (this.Column > 0 ? -1 : 1) : this.Column; + return new(row, column); + } + + /// + /// Clamps this vector to the spatial reference-candidate limits for a coding block. + /// + /// The coding-block width in luma samples. + /// The coding-block height in luma samples. + /// The signed distance to the left frame edge in one-eighth-sample units. + /// The signed distance to the right frame edge in one-eighth-sample units. + /// The signed distance to the top frame edge in one-eighth-sample units. + /// The signed distance to the bottom frame edge in one-eighth-sample units. + /// The vector clamped to the permitted spatial reference-candidate range. + public Av1MotionVector ClampReference( + int blockWidth, + int blockHeight, + int blockToLeftEdge, + int blockToRightEdge, + int blockToTopEdge, + int blockToBottomEdge) + { + int blockWidthSubpixel = blockWidth << 3; + int blockHeightSubpixel = blockHeight << 3; + + // Candidate derivation permits the complete block extent plus sixteen further luma samples beyond each + // visible frame edge. These are stack limits, not the tighter UMV limits applied later while sampling pixels. + int minimumColumn = blockToLeftEdge - blockWidthSubpixel - ReferenceBorder; + int maximumColumn = blockToRightEdge + blockWidthSubpixel + ReferenceBorder; + int minimumRow = blockToTopEdge - blockHeightSubpixel - ReferenceBorder; + int maximumRow = blockToBottomEdge + blockHeightSubpixel + ReferenceBorder; + return new( + Av1Math.Clip3(minimumRow, maximumRow, this.Row), + Av1Math.Clip3(minimumColumn, maximumColumn, this.Column)); + } + + /// + /// Projects this vector across a ratio of temporal frame distances. + /// + /// The signed source-to-target frame distance. + /// The positive source-to-reference frame distance. + /// The projected vector clamped inside the AV1 motion-vector domain. + public Av1MotionVector ProjectTemporal(int numerator, int denominator) + { + denominator = Math.Min(denominator, MaximumTemporalDistance); + numerator = Av1Math.Clip3(-MaximumTemporalDistance, MaximumTemporalDistance, numerator); + + // The reciprocal table represents 1 / denominator in Q14. Signed power-of-two rounding preserves symmetry + // for negative components, and AV1 excludes the two reserved endpoints from projected motion vectors. + // Motion-field retention limits each source component to 4095, keeping the complete Q14 product inside Int32. + int row = Av1Math.RoundPowerOf2Signed(this.Row * numerator * ProjectionDivisors[denominator], 14); + int column = Av1Math.RoundPowerOf2Signed(this.Column * numerator * ProjectionDivisors[denominator], 14); + row = Av1Math.Clip3(LowerBound + 1, UpperBound - 1, row); + column = Av1Math.Clip3(LowerBound + 1, UpperBound - 1, column); + return new(row, column); + } + /// /// Determines whether this vector has the same components as another vector. /// @@ -71,4 +189,24 @@ internal readonly struct Av1MotionVector : IEquatable /// public override int GetHashCode() => HashCode.Combine(this.Row, this.Column); + + /// + /// Rounds one component to the nearest integer-sample displacement. + /// + /// The component in one-eighth-sample units. + /// The integer-precision component in one-eighth-sample units. + private static int RoundToIntegerPrecision(int value) + { + int remainder = value % 8; + value -= remainder; + + // Exactly half an integer sample has magnitude four. AV1 leaves that truncated base unchanged, so both + // positive and negative half ties move toward zero; only larger remainders advance to the adjacent sample. + if (Math.Abs(remainder) > 4) + { + value += remainder > 0 ? 8 : -8; + } + + return value; + } } diff --git a/src/ImageSharp/Formats/Heif/Av1/Motion/Av1MotionVectorPrecision.cs b/src/ImageSharp/Formats/Heif/Av1/Motion/Av1MotionVectorPrecision.cs new file mode 100644 index 000000000..79b89bb6a --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Motion/Av1MotionVectorPrecision.cs @@ -0,0 +1,25 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Motion; + +/// +/// Identifies the fractional precision used to decode an AV1 motion-vector delta. +/// +internal enum Av1MotionVectorPrecision : sbyte +{ + /// + /// Restricts components to whole-sample increments. + /// + Integer = -1, + + /// + /// Allows components in quarter-sample increments. + /// + QuarterSample, + + /// + /// Allows components in eighth-sample increments. + /// + EighthSample +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Motion/Av1ReferenceMotionVectors.cs b/src/ImageSharp/Formats/Heif/Av1/Motion/Av1ReferenceMotionVectors.cs new file mode 100644 index 000000000..b5e614771 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Motion/Av1ReferenceMotionVectors.cs @@ -0,0 +1,906 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit; +using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; +using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Motion; + +/// +/// Derives the weighted AV1 reference-motion-vector candidates for one single-reference inter block. +/// +internal sealed class Av1ReferenceMotionVectors +{ + /// + /// The number of surrounding mode-information rows and columns examined by the spatial search. + /// + private const int ReferenceSearchDistance = 3; + + /// + /// The weight separating immediately adjacent candidates from temporal and outer spatial candidates. + /// + private const int NearestCandidateWeight = 640; + + /// + /// The maximum number of distinct candidates retained by AV1. + /// + private const int CandidateCapacity = 8; + + /// + /// The width and height of the outer spatial and temporal search boundary in 4x4 mode-information units. + /// + private const int MaximumSearchBlockSize = 16; + + /// + /// The packed mode-context bit containing temporal availability relative to global motion. + /// + private const int GlobalMotionContextBit = 1 << 3; + + /// + /// The bit offset of the reference-motion-vector context in the packed mode context. + /// + private const int ReferenceMotionVectorContextOffset = 4; + + /// + /// Stores the unique candidates in their normative weighted order. + /// + private InlineArray8 candidates; + + /// + /// Stores the accumulated spatial or temporal weight corresponding to each candidate. + /// + private InlineArray8 weights; + + /// + /// Stores the nearest and near references after applying AV1 fallback and precision rules. + /// + private InlineArray2 references; + + /// + /// Gets the number of valid entries in and . + /// + public int Count { get; private set; } + + /// + /// Gets the packed entropy context derived from adjacent, outer, and temporal candidates. + /// + public int ModeContext { get; private set; } + + /// + /// Gets the derived candidates in normative nearest-region then outer-region order. + /// + public ReadOnlySpan Candidates => this.candidates[..this.Count]; + + /// + /// Gets the accumulated weight corresponding to each entry in . + /// + public ReadOnlySpan Weights => this.weights[..this.Count]; + + /// + /// Gets the nearest reference, or the current block's global-motion vector when no candidate exists. + /// + public Av1MotionVector Nearest => this.references[0]; + + /// + /// Derives all single-reference motion-vector candidates for the current block. + /// + /// The current block geometry and decoded spatial neighbors. + /// The active tile boundaries. + /// The frame-wide spatial map and projected temporal motion field. + /// The sequence-level superblock and order-hint configuration. + /// The frame-level global-motion and motion-vector precision configuration. + /// The canonical inter reference selected for the current block. + public void Build( + ref Av1PartitionInfo partitionInfo, + Av1TileInfo tileInfo, + Av1FrameInfo frameInfo, + ObuSequenceHeader sequenceHeader, + ObuFrameHeader frameHeader, + Av1ReferenceFrameType referenceFrame) + { + Av1BlockSize blockSize = partitionInfo.ModeInfo.BlockSize; + int width = blockSize.Get4x4WideCount(); + int height = blockSize.Get4x4HighCount(); + int row = partitionInfo.RowIndex; + int column = partitionInfo.ColumnIndex; + int rowAdjustment = height < 2 && (row & 1) != 0 ? 1 : 0; + int columnAdjustment = width < 2 && (column & 1) != 0 ? 1 : 0; + int maximumRowOffset = 0; + int maximumColumnOffset = 0; + + this.Count = 0; + this.ModeContext = 0; + + if (partitionInfo.AvailableAbove) + { + maximumRowOffset = height < 2 ? -4 + rowAdjustment : -(ReferenceSearchDistance << 1) + rowAdjustment; + maximumRowOffset = Math.Clamp(maximumRowOffset, tileInfo.ModeInfoRowStart - row, tileInfo.ModeInfoRowEnd - row - 1); + } + + if (partitionInfo.AvailableLeft) + { + maximumColumnOffset = width < 2 ? -4 + columnAdjustment : -(ReferenceSearchDistance << 1) + columnAdjustment; + maximumColumnOffset = Math.Clamp(maximumColumnOffset, tileInfo.ModeInfoColumnStart - column, tileInfo.ModeInfoColumnEnd - column - 1); + } + + Av1GlobalMotionParameters globalMotion = frameHeader.GetGlobalMotionParameters()[(int)referenceFrame - 1]; + Av1MotionVector globalMotionVector = globalMotion.GetMotionVector( + frameHeader.AllowHighPrecisionMotionVector, + blockSize, + new Point(column, row), + frameHeader.ForceIntegerMotionVector); + + int processedRows = 0; + int processedColumns = 0; + int rowMatchCount = 0; + int columnMatchCount = 0; + int newMotionVectorCount = 0; + + // Immediate above and left scans form a distinct high-priority region. Their direction-level match counts, + // rather than their number of unique vectors, drive the packed inter-mode entropy context. + if (Math.Abs(maximumRowOffset) >= 1) + { + this.ScanRow( + ref partitionInfo, + referenceFrame, + in globalMotion, + globalMotionVector, + -1, + maximumRowOffset, + ref rowMatchCount, + ref newMotionVectorCount, + ref processedRows); + } + + if (Math.Abs(maximumColumnOffset) >= 1) + { + this.ScanColumn( + ref partitionInfo, + referenceFrame, + in globalMotion, + globalMotionVector, + -1, + maximumColumnOffset, + ref columnMatchCount, + ref newMotionVectorCount, + ref processedColumns); + } + + if (partitionInfo.HasTopRight(sequenceHeader.SuperblockModeInfoSize)) + { + this.AddSpatialBlock( + ref partitionInfo, + tileInfo, + referenceFrame, + in globalMotion, + globalMotionVector, + -1, + width, + ref rowMatchCount, + ref newMotionVectorCount); + } + + int nearestMatch = (rowMatchCount > 0 ? 1 : 0) + (columnMatchCount > 0 ? 1 : 0); + int nearestCandidateCount = this.Count; + for (int index = 0; index < nearestCandidateCount; index++) + { + this.weights[index] += NearestCandidateWeight; + } + + if (frameHeader.UseReferenceFrameMotionVectors) + { + this.AddTemporalCandidates( + ref partitionInfo, + tileInfo, + frameInfo, + sequenceHeader.OrderHintInfo, + frameHeader, + referenceFrame, + globalMotionVector); + } + + int ignoredNewMotionVectorCount = 0; + + // The top-left block begins the lower-priority outer region. Candidate deduplication still spans both + // regions, while the two independent stable sorts below preserve the normative nearest-before-outer order. + this.AddSpatialBlock( + ref partitionInfo, + tileInfo, + referenceFrame, + in globalMotion, + globalMotionVector, + -1, + -1, + ref rowMatchCount, + ref ignoredNewMotionVectorCount); + + for (int index = 2; index <= ReferenceSearchDistance; index++) + { + int rowOffset = -(index << 1) + 1 + rowAdjustment; + int columnOffset = -(index << 1) + 1 + columnAdjustment; + if (Math.Abs(rowOffset) <= Math.Abs(maximumRowOffset) && Math.Abs(rowOffset) > processedRows) + { + this.ScanRow( + ref partitionInfo, + referenceFrame, + in globalMotion, + globalMotionVector, + rowOffset, + maximumRowOffset, + ref rowMatchCount, + ref ignoredNewMotionVectorCount, + ref processedRows); + } + + if (Math.Abs(columnOffset) <= Math.Abs(maximumColumnOffset) && Math.Abs(columnOffset) > processedColumns) + { + this.ScanColumn( + ref partitionInfo, + referenceFrame, + in globalMotion, + globalMotionVector, + columnOffset, + maximumColumnOffset, + ref columnMatchCount, + ref ignoredNewMotionVectorCount, + ref processedColumns); + } + } + + int referenceMatchCount = (rowMatchCount > 0 ? 1 : 0) + (columnMatchCount > 0 ? 1 : 0); + this.ModeContext |= nearestMatch switch + { + 0 => (referenceMatchCount >= 1 ? 1 : 0) | + (referenceMatchCount == 1 ? 1 << ReferenceMotionVectorContextOffset : + referenceMatchCount >= 2 ? 2 << ReferenceMotionVectorContextOffset : 0), + 1 => (newMotionVectorCount > 0 ? 2 : 3) | + (referenceMatchCount == 1 ? 3 << ReferenceMotionVectorContextOffset : + referenceMatchCount >= 2 ? 4 << ReferenceMotionVectorContextOffset : 0), + _ => (newMotionVectorCount >= 1 ? 4 : 5) | (5 << ReferenceMotionVectorContextOffset), + }; + + this.SortByWeight(0, nearestCandidateCount); + this.SortByWeight(nearestCandidateCount, this.Count); + + int frameWidth = frameHeader.ModeInfoColumnCount; + int frameHeight = frameHeader.ModeInfoRowCount; + int modeInfoWidth = Math.Min(Math.Min(MaximumSearchBlockSize, width), frameWidth - column); + int modeInfoHeight = Math.Min(Math.Min(MaximumSearchBlockSize, height), frameHeight - row); + int extensionLength = Math.Min(modeInfoWidth, modeInfoHeight); + + // When the direct stack has fewer than two entries, AV1 extends it from every inter reference on the + // immediate above and left blocks. Opposite temporal directions are sign-reversed into the target role. + for (int index = 0; Math.Abs(maximumRowOffset) >= 1 && index < extensionLength && this.Count < 2;) + { + Av1BlockModeInfo candidate = partitionInfo.SuperblockInfo.GetModeInfoAt(new Point(column + index, row - 1)); + this.AddExtensionCandidate(candidate, frameInfo, referenceFrame); + index += candidate.BlockSize.Get4x4WideCount(); + } + + for (int index = 0; Math.Abs(maximumColumnOffset) >= 1 && index < extensionLength && this.Count < 2;) + { + Av1BlockModeInfo candidate = partitionInfo.SuperblockInfo.GetModeInfoAt(new Point(column - 1, row + index)); + this.AddExtensionCandidate(candidate, frameInfo, referenceFrame); + index += candidate.BlockSize.Get4x4HighCount(); + } + + for (int index = 0; index < this.Count; index++) + { + this.candidates[index] = this.candidates[index].ClampReference( + blockSize.GetWidth(), + blockSize.GetHeight(), + partitionInfo.ModeBlockToLeftEdge, + partitionInfo.ModeBlockToRightEdge, + partitionInfo.ModeBlockToTopEdge, + partitionInfo.ModeBlockToBottomEdge); + } + + // The two-element reference list is separate from the full DRL stack. Missing entries use global motion, + // and both entries undergo the same precision reduction as libaom's av1_find_best_ref_mvs output. + this.references[0] = (this.Count > 0 ? this.candidates[0] : globalMotionVector).LowerPrecision( + frameHeader.AllowHighPrecisionMotionVector, + frameHeader.ForceIntegerMotionVector); + + this.references[1] = (this.Count > 1 ? this.candidates[1] : globalMotionVector).LowerPrecision( + frameHeader.AllowHighPrecisionMotionVector, + frameHeader.ForceIntegerMotionVector); + } + + /// + /// Gets the near reference selected by a decoded dynamic-reference-list index. + /// + /// The decoded zero-based dynamic-reference-list index. + /// The selected near motion vector. + public Av1MotionVector GetNearReference(int referenceMotionVectorIndex) + => referenceMotionVectorIndex == 0 ? this.references[1] : this.candidates[referenceMotionVectorIndex + 1]; + + /// + /// Gets the differential reference used to decode a new motion vector. + /// + /// The decoded zero-based dynamic-reference-list index. + /// The selected stack candidate, or the nearest fallback when the stack contains one or no entries. + public Av1MotionVector GetNewReference(int referenceMotionVectorIndex) + => this.Count > 1 ? this.candidates[referenceMotionVectorIndex] : this.references[0]; + + /// + /// Scans one spatial row using AV1's block-size-dependent steps and weights. + /// + /// The current block geometry and frame-wide mode map. + /// The canonical inter reference selected for the current block. + /// The selected reference's global-motion model. + /// The selected reference's global-motion vector at the current block. + /// The signed row offset from the current block in 4x4 units. + /// The farthest permitted row offset inside the tile. + /// Accumulates matching reference labels found in this scan direction. + /// Accumulates matching neighbors whose inter mode contains a new vector. + /// Receives the spatial depth covered by block-height weighting. + private void ScanRow( + ref Av1PartitionInfo partitionInfo, + Av1ReferenceFrameType referenceFrame, + in Av1GlobalMotionParameters globalMotion, + Av1MotionVector globalMotionVector, + int rowOffset, + int maximumRowOffset, + ref int referenceMatchCount, + ref int newMotionVectorCount, + ref int processedRows) + { + int width = partitionInfo.ModeInfo.BlockSize.Get4x4WideCount(); + int end = Math.Min(partitionInfo.GetMaxBlockWide(partitionInfo.ModeInfo.BlockSize, false), MaximumSearchBlockSize); + int columnOffset = 0; + if (Math.Abs(rowOffset) > 1) + { + columnOffset = 1; + if ((partitionInfo.ColumnIndex & 1) != 0 && width < 2) + { + columnOffset--; + } + } + + bool useFourUnitStep = width >= 4; + for (int index = 0; index < end;) + { + Av1BlockModeInfo candidate = partitionInfo.SuperblockInfo.GetModeInfoAt( + new Point(partitionInfo.ColumnIndex + columnOffset + index, partitionInfo.RowIndex + rowOffset)); + + int candidateWidth = candidate.BlockSize.Get4x4WideCount(); + int length = Math.Min(width, candidateWidth); + if (useFourUnitStep) + { + length = Math.Max(4, length); + } + else if (Math.Abs(rowOffset) > 1) + { + length = Math.Max(2, length); + } + + int weight = 2; + if (width >= 2 && width <= candidateWidth) + { + int increment = Math.Min(-maximumRowOffset + rowOffset + 1, candidate.BlockSize.Get4x4HighCount()); + weight = Math.Max(weight, increment); + processedRows = increment - rowOffset - 1; + } + + this.AddCandidate( + candidate, + referenceFrame, + in globalMotion, + globalMotionVector, + length * weight, + ref referenceMatchCount, + ref newMotionVectorCount); + + index += length; + } + } + + /// + /// Scans one spatial column using AV1's block-size-dependent steps and weights. + /// + /// The current block geometry and frame-wide mode map. + /// The canonical inter reference selected for the current block. + /// The selected reference's global-motion model. + /// The selected reference's global-motion vector at the current block. + /// The signed column offset from the current block in 4x4 units. + /// The farthest permitted column offset inside the tile. + /// Accumulates matching reference labels found in this scan direction. + /// Accumulates matching neighbors whose inter mode contains a new vector. + /// Receives the spatial depth covered by block-width weighting. + private void ScanColumn( + ref Av1PartitionInfo partitionInfo, + Av1ReferenceFrameType referenceFrame, + in Av1GlobalMotionParameters globalMotion, + Av1MotionVector globalMotionVector, + int columnOffset, + int maximumColumnOffset, + ref int referenceMatchCount, + ref int newMotionVectorCount, + ref int processedColumns) + { + int height = partitionInfo.ModeInfo.BlockSize.Get4x4HighCount(); + int end = Math.Min(partitionInfo.GetMaxBlockHigh(partitionInfo.ModeInfo.BlockSize, false), MaximumSearchBlockSize); + int rowOffset = 0; + if (Math.Abs(columnOffset) > 1) + { + rowOffset = 1; + if ((partitionInfo.RowIndex & 1) != 0 && height < 2) + { + rowOffset--; + } + } + + bool useFourUnitStep = height >= 4; + for (int index = 0; index < end;) + { + Av1BlockModeInfo candidate = partitionInfo.SuperblockInfo.GetModeInfoAt( + new Point(partitionInfo.ColumnIndex + columnOffset, partitionInfo.RowIndex + rowOffset + index)); + + int candidateHeight = candidate.BlockSize.Get4x4HighCount(); + int length = Math.Min(height, candidateHeight); + if (useFourUnitStep) + { + length = Math.Max(4, length); + } + else if (Math.Abs(columnOffset) > 1) + { + length = Math.Max(2, length); + } + + int weight = 2; + if (height >= 2 && height <= candidateHeight) + { + int increment = Math.Min(-maximumColumnOffset + columnOffset + 1, candidate.BlockSize.Get4x4WideCount()); + weight = Math.Max(weight, increment); + processedColumns = increment - columnOffset - 1; + } + + this.AddCandidate( + candidate, + referenceFrame, + in globalMotion, + globalMotionVector, + length * weight, + ref referenceMatchCount, + ref newMotionVectorCount); + + index += length; + } + } + + /// + /// Adds the candidate at one tile-relative spatial search position. + /// + /// The current block geometry and frame-wide mode map. + /// The active tile boundaries. + /// The canonical inter reference selected for the current block. + /// The selected reference's global-motion model. + /// The selected reference's global-motion vector at the current block. + /// The signed row offset from the current block in 4x4 units. + /// The signed column offset from the current block in 4x4 units. + /// Accumulates matching reference labels at the search position. + /// Accumulates matching neighbors whose inter mode contains a new vector. + private void AddSpatialBlock( + ref Av1PartitionInfo partitionInfo, + Av1TileInfo tileInfo, + Av1ReferenceFrameType referenceFrame, + in Av1GlobalMotionParameters globalMotion, + Av1MotionVector globalMotionVector, + int rowOffset, + int columnOffset, + ref int referenceMatchCount, + ref int newMotionVectorCount) + { + int row = partitionInfo.RowIndex + rowOffset; + int column = partitionInfo.ColumnIndex + columnOffset; + if (row < tileInfo.ModeInfoRowStart || row >= tileInfo.ModeInfoRowEnd || + column < tileInfo.ModeInfoColumnStart || column >= tileInfo.ModeInfoColumnEnd) + { + return; + } + + Av1BlockModeInfo candidate = partitionInfo.SuperblockInfo.GetModeInfoAt(new Point(column, row)); + this.AddCandidate( + candidate, + referenceFrame, + in globalMotion, + globalMotionVector, + 4, + ref referenceMatchCount, + ref newMotionVectorCount); + } + + /// + /// Accumulates matching references from one decoded inter block. + /// + /// The decoded neighboring block. + /// The canonical inter reference selected for the current block. + /// The selected reference's global-motion model. + /// The selected reference's global-motion vector at the current block. + /// The spatial weight contributed by each matching reference. + /// Accumulates matching reference labels in the active scan direction. + /// Accumulates matching neighbors whose inter mode contains a new vector. + private void AddCandidate( + Av1BlockModeInfo candidate, + Av1ReferenceFrameType referenceFrame, + in Av1GlobalMotionParameters globalMotion, + Av1MotionVector globalMotionVector, + int weight, + ref int referenceMatchCount, + ref int newMotionVectorCount) + { + Span candidateReferences = candidate.ReferenceFrames; + if (candidateReferences[0] <= Av1ReferenceFrameType.Intra) + { + return; + } + + Span candidateMotionVectors = candidate.MotionVectors; + for (int referenceIndex = 0; referenceIndex < 2; referenceIndex++) + { + if (candidateReferences[referenceIndex] != referenceFrame) + { + continue; + } + + // A non-translational global block has no independent translational candidate at the neighbor. AV1 + // therefore evaluates the selected reference's global model at the current block and contributes that + // vector, but only for blocks large enough to use affine global prediction. + bool useGlobalMotion = + (candidate.YMode is Av1PredictionMode.GlobalMotionVector or Av1PredictionMode.GlobalGlobalMotionVector) && + globalMotion.Type > Av1GlobalMotionType.Translation && + Math.Min(candidate.BlockSize.GetWidth(), candidate.BlockSize.GetHeight()) >= 8; + + Av1MotionVector motionVector = useGlobalMotion + ? globalMotionVector + : candidateMotionVectors[referenceIndex]; + + this.AddUnique(motionVector, weight); + + // Every matching reference in a neighbor carrying a NEW component contributes to the adjacent NEWMV + // context even when its vector deduplicates against an earlier stack entry. + if (candidate.YMode is Av1PredictionMode.NewMotionVector or + Av1PredictionMode.NewNewMotionVector or + Av1PredictionMode.NearestNewMotionVector or + Av1PredictionMode.NewNearestMotionVector or + Av1PredictionMode.NearNewMotionVector or + Av1PredictionMode.NewNearMotionVector) + { + newMotionVectorCount++; + } + + referenceMatchCount++; + } + } + + /// + /// Adds projected temporal candidates over the current block and its permitted extension positions. + /// + /// The current block geometry. + /// The active tile boundaries. + /// The projected temporal motion field. + /// The sequence modulo order-hint configuration. + /// The frame-level motion-vector precision configuration. + /// The canonical inter reference selected for the current block. + /// The selected reference's global-motion vector at the current block. + private void AddTemporalCandidates( + ref Av1PartitionInfo partitionInfo, + Av1TileInfo tileInfo, + Av1FrameInfo frameInfo, + ObuOrderHintInfo orderHintInfo, + ObuFrameHeader frameHeader, + Av1ReferenceFrameType referenceFrame, + Av1MotionVector globalMotionVector) + { + int width = partitionInfo.ModeInfo.BlockSize.Get4x4WideCount(); + int height = partitionInfo.ModeInfo.BlockSize.Get4x4HighCount(); + int verticalOffset = Math.Max(2, height); + int horizontalOffset = Math.Max(2, width); + int blockRowEnd = Math.Min(height, MaximumSearchBlockSize); + int blockColumnEnd = Math.Min(width, MaximumSearchBlockSize); + int rowStep = height >= MaximumSearchBlockSize ? 4 : 2; + int columnStep = width >= MaximumSearchBlockSize ? 4 : 2; + bool firstSampleAvailable = false; + + for (int blockRow = 0; blockRow < blockRowEnd; blockRow += rowStep) + { + for (int blockColumn = 0; blockColumn < blockColumnEnd; blockColumn += columnStep) + { + bool available = this.AddTemporalCandidate( + ref partitionInfo, + tileInfo, + frameInfo, + orderHintInfo, + frameHeader, + referenceFrame, + globalMotionVector, + blockRow, + blockColumn); + + if (blockRow == 0 && blockColumn == 0) + { + firstSampleAvailable = available; + } + } + } + + if (!firstSampleAvailable) + { + this.ModeContext |= GlobalMotionContextBit; + } + + bool allowExtension = height >= 2 && height < MaximumSearchBlockSize && width >= 2 && width < MaximumSearchBlockSize; + if (!allowExtension) + { + return; + } + + // These three positions extend the temporal search below-left, below-right, and above-right. The 64x64 + // boundary test is normative even when the sequence uses 128x128 superblocks. + this.AddTemporalExtension( + ref partitionInfo, + tileInfo, + frameInfo, + orderHintInfo, + frameHeader, + referenceFrame, + globalMotionVector, + verticalOffset, + -2); + + this.AddTemporalExtension( + ref partitionInfo, + tileInfo, + frameInfo, + orderHintInfo, + frameHeader, + referenceFrame, + globalMotionVector, + verticalOffset, + horizontalOffset); + + this.AddTemporalExtension( + ref partitionInfo, + tileInfo, + frameInfo, + orderHintInfo, + frameHeader, + referenceFrame, + globalMotionVector, + verticalOffset - 2, + horizontalOffset); + } + + /// + /// Adds one optional temporal extension candidate after applying the normative 64x64 boundary rule. + /// + /// The current block geometry. + /// The active tile boundaries. + /// The projected temporal motion field. + /// The sequence modulo order-hint configuration. + /// The frame-level motion-vector precision configuration. + /// The canonical inter reference selected for the current block. + /// The selected reference's global-motion vector at the current block. + /// The temporal sample row relative to the current block in 4x4 units. + /// The temporal sample column relative to the current block in 4x4 units. + private void AddTemporalExtension( + ref Av1PartitionInfo partitionInfo, + Av1TileInfo tileInfo, + Av1FrameInfo frameInfo, + ObuOrderHintInfo orderHintInfo, + ObuFrameHeader frameHeader, + Av1ReferenceFrameType referenceFrame, + Av1MotionVector globalMotionVector, + int blockRow, + int blockColumn) + { + int rowWithinBlock64 = partitionInfo.RowIndex & (MaximumSearchBlockSize - 1); + int columnWithinBlock64 = partitionInfo.ColumnIndex & (MaximumSearchBlockSize - 1); + if (rowWithinBlock64 + blockRow < 0 || rowWithinBlock64 + blockRow >= MaximumSearchBlockSize || + columnWithinBlock64 + blockColumn < 0 || columnWithinBlock64 + blockColumn >= MaximumSearchBlockSize) + { + return; + } + + _ = this.AddTemporalCandidate( + ref partitionInfo, + tileInfo, + frameInfo, + orderHintInfo, + frameHeader, + referenceFrame, + globalMotionVector, + blockRow, + blockColumn); + } + + /// + /// Projects and accumulates one temporal motion-field sample. + /// + /// The current block geometry. + /// The active tile boundaries. + /// The projected temporal motion field. + /// The sequence modulo order-hint configuration. + /// The frame-level motion-vector precision configuration. + /// The canonical inter reference selected for the current block. + /// The selected reference's global-motion vector at the current block. + /// The temporal sample row relative to the current block in 4x4 units. + /// The temporal sample column relative to the current block in 4x4 units. + /// when the projected motion field covers the requested position. + private bool AddTemporalCandidate( + ref Av1PartitionInfo partitionInfo, + Av1TileInfo tileInfo, + Av1FrameInfo frameInfo, + ObuOrderHintInfo orderHintInfo, + ObuFrameHeader frameHeader, + Av1ReferenceFrameType referenceFrame, + Av1MotionVector globalMotionVector, + int blockRow, + int blockColumn) + { + int rowOffset = (partitionInfo.RowIndex & 1) != 0 ? blockRow : blockRow + 1; + int columnOffset = (partitionInfo.ColumnIndex & 1) != 0 ? blockColumn : blockColumn + 1; + int row = partitionInfo.RowIndex + rowOffset; + int column = partitionInfo.ColumnIndex + columnOffset; + if (row < tileInfo.ModeInfoRowStart || row >= tileInfo.ModeInfoRowEnd || + column < tileInfo.ModeInfoColumnStart || column >= tileInfo.ModeInfoColumnEnd) + { + return false; + } + + if (!frameInfo.TryGetProjectedTemporalMotionVector( + row, + column, + referenceFrame, + orderHintInfo, + frameHeader.AllowHighPrecisionMotionVector, + frameHeader.ForceIntegerMotionVector, + out Av1MotionVector motionVector)) + { + return false; + } + + if (blockRow == 0 && blockColumn == 0 && + (Math.Abs(motionVector.Row - globalMotionVector.Row) >= 16 || + Math.Abs(motionVector.Column - globalMotionVector.Column) >= 16)) + { + // The packed global-motion context records whether the first temporal sample is absent or differs from + // global motion by at least two full samples in either one-eighth-sample component. + this.ModeContext |= GlobalMotionContextBit; + } + + this.AddUnique(motionVector, 2); + return true; + } + + /// + /// Extends a short stack with inter vectors from a neighboring block, correcting their temporal direction. + /// + /// The decoded neighboring block. + /// The reference-side classification for the current frame. + /// The canonical inter reference selected for the current block. + private void AddExtensionCandidate( + Av1BlockModeInfo candidate, + Av1FrameInfo frameInfo, + Av1ReferenceFrameType referenceFrame) + { + Span candidateReferences = candidate.ReferenceFrames; + Span candidateMotionVectors = candidate.MotionVectors; + bool targetSignBias = frameInfo.IsReferenceSignBiased(referenceFrame); + + for (int referenceIndex = 0; referenceIndex < 2; referenceIndex++) + { + Av1ReferenceFrameType candidateReference = candidateReferences[referenceIndex]; + if (candidateReference <= Av1ReferenceFrameType.Intra) + { + continue; + } + + Av1MotionVector motionVector = candidateMotionVectors[referenceIndex]; + if (frameInfo.IsReferenceSignBiased(candidateReference) != targetSignBias) + { + motionVector = new Av1MotionVector(-motionVector.Row, -motionVector.Column); + } + + int candidateIndex; + for (candidateIndex = 0; candidateIndex < this.Count; candidateIndex++) + { + if (this.candidates[candidateIndex] == motionVector) + { + break; + } + } + + if (candidateIndex == this.Count && this.Count < CandidateCapacity) + { + // AV1's outer spatial extension only initializes a new stack entry. Unlike the weighted nearest and + // temporal scans, finding an existing vector here must not change its previously accumulated rank. + this.candidates[this.Count] = motionVector; + this.weights[this.Count] = 2; + this.Count++; + } + } + } + + /// + /// Adds a unique candidate or accumulates the weight of an existing candidate. + /// + /// The candidate vector in one-eighth-sample units. + /// The spatial or temporal weight contributed by this occurrence. + private void AddUnique(Av1MotionVector motionVector, int weight) + { + for (int index = 0; index < this.Count; index++) + { + if (this.candidates[index] == motionVector) + { + this.weights[index] += (ushort)weight; + return; + } + } + + if (this.Count < CandidateCapacity) + { + this.candidates[this.Count] = motionVector; + this.weights[this.Count] = (ushort)weight; + this.Count++; + } + } + + /// + /// Sorts one candidate region by descending accumulated weight while retaining scan order for equal weights. + /// + /// The inclusive first candidate index in the region. + /// The exclusive end candidate index in the region. + private void SortByWeight(int start, int end) + { + int length = end; + while (length > start) + { + int lastSwap = start; + for (int index = start + 1; index < length; index++) + { + if (this.weights[index - 1] < this.weights[index]) + { + Av1MotionVector candidate = this.candidates[index - 1]; + this.candidates[index - 1] = this.candidates[index]; + this.candidates[index] = candidate; + + ushort weight = this.weights[index - 1]; + this.weights[index - 1] = this.weights[index]; + this.weights[index] = weight; + lastSwap = index; + } + } + + length = lastSwap; + } + } + + /// + /// Provides fixed storage for AV1's eight reference-motion-vector candidates. + /// + /// The motion-vector or weight type stored in the inline buffer. + [InlineArray(CandidateCapacity)] + private struct InlineArray8 + { + /// + /// The first element in the compiler-expanded inline buffer. + /// + private T element; + } + + /// + /// Provides fixed storage for the nearest and near motion-vector references. + /// + /// The motion-vector type stored in the inline buffer. + [InlineArray(2)] + private struct InlineArray2 + { + /// + /// The first element in the compiler-expanded inline buffer. + /// + private T element; + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/OpenBitstreamUnit/ObuReader.cs b/src/ImageSharp/Formats/Heif/Av1/OpenBitstreamUnit/ObuReader.cs index 7b888d779..776fbb4c1 100644 --- a/src/ImageSharp/Formats/Heif/Av1/OpenBitstreamUnit/ObuReader.cs +++ b/src/ImageSharp/Formats/Heif/Av1/OpenBitstreamUnit/ObuReader.cs @@ -539,11 +539,16 @@ internal class ObuReader /// The reader to align. private static void AlignToByteBoundary(ref Av1BitStreamReader reader) { + int alignmentStartPosition = reader.BitPosition; while ((reader.BitPosition & 0x7) > 0) { + int paddingBitPosition = reader.BitPosition; if (reader.ReadBoolean()) { - throw new ImageFormatException("Incorrect byte alignment padding bits."); + string message = + $"Incorrect byte alignment padding bit at offset {paddingBitPosition}; alignment started at offset {alignmentStartPosition}."; + + throw new ImageFormatException(message); } } } diff --git a/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1FrameDecoder.cs b/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1FrameDecoder.cs index 9a95d94d9..6a09e20f0 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1FrameDecoder.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1FrameDecoder.cs @@ -7,6 +7,7 @@ using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.LoopFilter; using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.LoopRestoration; using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.Quantizers; using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.SuperResolution; +using SixLabors.ImageSharp.Formats.Heif.Av1.ReferenceFrames; using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; using SixLabors.ImageSharp.Formats.Heif.Av1.Transform; @@ -37,6 +38,11 @@ internal sealed class Av1FrameDecoder : IAv1FrameDecoder, IDisposable /// private readonly Av1FrameBuffer frameBuffer; + /// + /// The retained reconstructed frames addressable by inter prediction. + /// + private readonly Av1ReferenceFrameStore? referenceFrames; + /// /// The coefficient inverse-quantization stage shared across superblocks. /// @@ -64,16 +70,31 @@ internal sealed class Av1FrameDecoder : IAv1FrameDecoder, IDisposable /// The parsed AV1 frame header. /// The parsed superblock and block-mode information. /// The destination planar sample buffers. - public Av1FrameDecoder(ObuSequenceHeader sequenceHeader, ObuFrameHeader frameHeader, Av1FrameInfo frameInfo, Av1FrameBuffer frameBuffer) + /// + /// The retained reconstructed frames selected by inter blocks, or for intra-only reconstruction. + /// + public Av1FrameDecoder( + ObuSequenceHeader sequenceHeader, + ObuFrameHeader frameHeader, + Av1FrameInfo frameInfo, + Av1FrameBuffer frameBuffer, + Av1ReferenceFrameStore? referenceFrames = null) { this.sequenceHeader = sequenceHeader; this.frameHeader = frameHeader; this.frameInfo = frameInfo; this.frameBuffer = frameBuffer; + this.referenceFrames = referenceFrames; this.inverseQuantizer = new(sequenceHeader, frameHeader); this.deQuants = new(sequenceHeader, frameHeader); this.loopFilterContext = new(sequenceHeader); - this.blockDecoder = new(this.sequenceHeader, this.frameHeader, this.frameBuffer, this.loopFilterContext, this.inverseQuantizer); + this.blockDecoder = new( + this.sequenceHeader, + this.frameHeader, + this.frameBuffer, + this.loopFilterContext, + this.inverseQuantizer, + this.referenceFrames); } /// diff --git a/src/ImageSharp/Formats/Heif/Av1/ReferenceFrames/Av1ReferenceFrame.cs b/src/ImageSharp/Formats/Heif/Av1/ReferenceFrames/Av1ReferenceFrame.cs index a1953c483..d67c050c5 100644 --- a/src/ImageSharp/Formats/Heif/Av1/ReferenceFrames/Av1ReferenceFrame.cs +++ b/src/ImageSharp/Formats/Heif/Av1/ReferenceFrames/Av1ReferenceFrame.cs @@ -31,6 +31,11 @@ internal sealed class Av1ReferenceFrame : IDisposable /// private Av1FrameEntropyContexts? entropyContextOwner; + /// + /// The shared decoded per-block state while this frame owns one lifetime lease. + /// + private Av1FrameInfo? frameInfo; + /// /// Initializes a new instance of the class and takes ownership of the decoded /// sample buffer. @@ -50,7 +55,8 @@ internal sealed class Av1ReferenceFrame : IDisposable { this.frameBuffer = frameBuffer; this.FrameHeader = frameHeader; - this.FrameInfo = frameInfo; + this.frameInfo = frameInfo; + frameInfo.AddOwner(); } /// @@ -95,7 +101,7 @@ internal sealed class Av1ReferenceFrame : IDisposable /// /// Gets the decoded per-block mode, motion, transform, and filter state associated with the retained frame. /// - public Av1FrameInfo FrameInfo { get; } + public Av1FrameInfo FrameInfo => this.frameInfo!; /// /// Gets the entropy context retained for primary-reference use, or for a presentation-only @@ -131,5 +137,7 @@ internal sealed class Av1ReferenceFrame : IDisposable this.frameBuffer?.Dispose(); this.frameBuffer = null; + this.frameInfo?.ReleaseOwner(); + this.frameInfo = null; } } diff --git a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1BlockModeInfo.cs b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1BlockModeInfo.cs index 62345222b..0255ae0eb 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1BlockModeInfo.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1BlockModeInfo.cs @@ -147,7 +147,7 @@ internal class Av1BlockModeInfo /// Gets or sets the selected index in the derived reference-motion-vector stack. /// /// - /// The AV1 syntax constrains this value to the inclusive range 0 through 3. + /// The AV1 syntax constrains this value to the inclusive range zero through two. /// public byte ReferenceMotionVectorIndex { get; set; } diff --git a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.MotionField.cs b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.MotionField.cs index 0eff7d739..ae7757dd7 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.MotionField.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.MotionField.cs @@ -1,9 +1,11 @@ // Copyright (c) Six Labors. // Licensed under the Six Labors Split License. +using System.Buffers; using SixLabors.ImageSharp.Formats.Heif.Av1.Motion; using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit; using SixLabors.ImageSharp.Formats.Heif.Av1.ReferenceFrames; +using SixLabors.ImageSharp.Memory; namespace SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; @@ -12,11 +14,6 @@ namespace SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; /// internal partial class Av1FrameInfo { - /// - /// The maximum absolute temporal distance accepted by AV1 motion-field projection. - /// - private const int MaximumFrameDistance = 31; - /// /// The maximum number of reference frames projected into one temporal motion field. /// @@ -27,16 +24,6 @@ internal partial class Av1FrameInfo /// private const int ReferenceMotionVectorLimit = 4095; - /// - /// The exclusive upper bound of an AV1 motion-vector component in one-eighth-sample units. - /// - private const int MotionVectorUpperBound = 16384; - - /// - /// The reserved lower endpoint of an AV1 motion-vector component in one-eighth-sample units. - /// - private const int MotionVectorLowerBound = -16384; - /// /// The width or height of the largest AV1 superblock in 4x4 mode-information units. /// @@ -63,14 +50,24 @@ internal partial class Av1FrameInfo private const int MaximumHorizontalFieldOffset = 8; /// - /// Stores the selected motion vector and logical reference for every retained 8x8 frame position. + /// Owns the selected motion vector and logical reference for every retained 8x8 frame position. + /// + private IMemoryOwner? retainedMotionField; + + /// + /// Owns motion vectors projected from retained frames into the current frame's 8x8 grid. /// - private RetainedMotionFieldEntry[] retainedMotionField = []; + private IMemoryOwner? temporalMotionField; /// - /// Stores motion vectors projected from retained frames into the current frame's 8x8 grid. + /// The number of tile-reader, reference-frame, and decoder-result owners retaining this frame state. /// - private TemporalMotionFieldEntry[] temporalMotionField = []; + private int ownerCount = 1; + + /// + /// Indicates whether this frame state still owns the initial lease created with the instance. + /// + private bool ownsInitialLease = true; /// /// Stores the order hint selected by each logical inter-reference type for later projections from this frame. @@ -82,6 +79,11 @@ internal partial class Av1FrameInfo /// private InlineArray8 motionFieldReferenceSides; + /// + /// The order hint of the current frame represented by this mode-information owner. + /// + private uint motionFieldOrderHint; + /// /// The number of retained motion-field entries in one active 8x8 row. /// @@ -102,29 +104,25 @@ internal partial class Av1FrameInfo /// private int activeModeInfoRowCount; - /// - /// Gets the reciprocal table used by AV1 motion-vector projection in 14-bit fixed-point precision. - /// - private static ReadOnlySpan ProjectionDivisors => - [0, 16384, 8192, 5461, 4096, 3276, 2730, 2340, 2048, 1820, 1638, 1489, 1365, 1260, 1170, 1092, - 1024, 963, 910, 862, 819, 780, 744, 712, 682, 655, 630, 606, 585, 564, 546, 528]; - /// /// Allocates and derives the motion fields required by one decoded frame. /// + /// The decoder configuration providing motion-field storage. /// The sequence header defining motion-field enablement and order-hint precision. /// The current frame header and its seven resolved inter-reference roles. /// The retained reconstructed frames selected by the current reference map. public void InitializeMotionField( + Configuration configuration, ObuSequenceHeader sequenceHeader, ObuFrameHeader frameHeader, Av1ReferenceFrameStore referenceFrames) { - if (!sequenceHeader.OrderHintInfo.EnableReferenceFrameMotionVectors) + if (!sequenceHeader.OrderHintInfo.EnableOrderHint) { return; } + this.motionFieldOrderHint = frameHeader.OrderHint; this.activeModeInfoColumnCount = frameHeader.ModeInfoColumnCount; this.activeModeInfoRowCount = frameHeader.ModeInfoRowCount; this.retainedMotionFieldStride = (this.activeModeInfoColumnCount + 1) >> MotionFieldModeInfoShift; @@ -156,10 +154,20 @@ internal partial class Av1FrameInfo : referenceOrderHint == frameHeader.OrderHint ? (sbyte)-1 : (sbyte)0; } - // FrameInfo is transferred directly into each retained frame owner, so allocate only the active 8x8 source - // grid whose completed block vectors can be projected by a later frame. + if (!sequenceHeader.OrderHintInfo.EnableReferenceFrameMotionVectors) + { + // Spatial reference extension still needs the sign classification above when temporal motion fields are + // disabled. Retained and projected 8x8 storage belongs exclusively to the temporal-motion-vector tool. + return; + } + + // FrameInfo is shared directly by the tile reader, reference frames, and the final decoder result. The + // allocator-owned motion fields therefore follow that shared lifetime without placing frame-sized arrays on + // the managed heap. Clean storage is required because an all-zero entry denotes the normative empty field. int retainedRowCount = (this.activeModeInfoRowCount + 1) >> MotionFieldModeInfoShift; - this.retainedMotionField = new RetainedMotionFieldEntry[this.retainedMotionFieldStride * retainedRowCount]; + this.retainedMotionField = configuration.MemoryAllocator.Allocate( + this.retainedMotionFieldStride * retainedRowCount, + AllocationOptions.Clean); if (!frameHeader.UseReferenceFrameMotionVectors) { @@ -174,7 +182,9 @@ internal partial class Av1FrameInfo this.temporalMotionFieldStride = alignedModeInfoColumnCount >> MotionFieldModeInfoShift; int temporalRowCount = (this.activeModeInfoRowCount + MaximumSuperblockModeInfoSize) >> MotionFieldModeInfoShift; - this.temporalMotionField = new TemporalMotionFieldEntry[this.temporalMotionFieldStride * temporalRowCount]; + this.temporalMotionField = configuration.MemoryAllocator.Allocate( + this.temporalMotionFieldStride * temporalRowCount, + AllocationOptions.Clean); // AV1 examines LAST, BWDREF, ALTREF2, ALTREF, and LAST2 in this normative order and admits at most three // projection sources. LAST always consumes the first budget position, forward references consume one only @@ -241,12 +251,62 @@ internal partial class Av1FrameInfo int index = ((modeInfoRow >> MotionFieldModeInfoShift) * this.temporalMotionFieldStride) + (modeInfoColumn >> MotionFieldModeInfoShift); - TemporalMotionFieldEntry entry = this.temporalMotionField[index]; + TemporalMotionFieldEntry entry = this.temporalMotionField!.Memory.Span[index]; motionVector = entry.MotionVector; referenceFrameOffset = entry.ReferenceFrameOffset; return referenceFrameOffset > 0; } + /// + /// Gets a value indicating whether a canonical inter reference has positive AV1 sign bias. + /// + /// The canonical inter-reference role. + /// for a future reference; otherwise, . + public bool IsReferenceSignBiased(Av1ReferenceFrameType referenceFrame) => this.motionFieldReferenceSides[(int)referenceFrame] > 0; + + /// + /// Gets a temporal motion-field entry projected to a selected canonical inter reference. + /// + /// The zero-based 4x4 row. + /// The zero-based 4x4 column. + /// The canonical inter-reference role targeted by the candidate. + /// The sequence modulo order-hint configuration. + /// A value indicating whether one-eighth-sample precision may be retained. + /// A value indicating whether integer-sample precision is required. + /// Receives the projected and precision-reduced candidate. + /// when a temporal entry covers the requested position. + public bool TryGetProjectedTemporalMotionVector( + int modeInfoRow, + int modeInfoColumn, + Av1ReferenceFrameType referenceFrame, + ObuOrderHintInfo orderHintInfo, + bool allowHighPrecision, + bool forceInteger, + out Av1MotionVector motionVector) + { + if (!this.TryGetTemporalMotionVector( + modeInfoRow, + modeInfoColumn, + out Av1MotionVector sourceMotionVector, + out int sourceReferenceOffset)) + { + motionVector = default; + return false; + } + + int targetReferenceOffset = orderHintInfo.GetRelativeDistance( + this.motionFieldOrderHint, + this.motionFieldReferenceOrderHints[(int)referenceFrame]); + + // The projected field retains the source frame's original vector and its source-to-reference distance. + // Candidate construction therefore applies the second normative ratio for the current frame's selected role. + motionVector = sourceMotionVector + .ProjectTemporal(targetReferenceOffset, sourceReferenceOffset) + .LowerPrecision(allowHighPrecision, forceInteger); + + return true; + } + /// /// Writes the retained per-8x8 motion-field entries covered by one completed mode-information block. /// @@ -254,11 +314,14 @@ internal partial class Av1FrameInfo /// The block origin in frame-relative 4x4 units. private void UpdateRetainedMotionField(Av1BlockModeInfo modeInfo, Point modeInfoPosition) { - if (this.retainedMotionField.Length == 0) + IMemoryOwner? retainedMotionField = this.retainedMotionField; + if (retainedMotionField is null) { return; } + Span retainedEntries = retainedMotionField.Memory.Span; + Av1ReferenceFrameType selectedReference = Av1ReferenceFrameType.None; Av1MotionVector selectedMotionVector = default; Span referenceFrames = modeInfo.ReferenceFrames; @@ -295,12 +358,13 @@ internal partial class Av1FrameInfo int firstFieldColumn = modeInfoPosition.X >> MotionFieldModeInfoShift; RetainedMotionFieldEntry entry = new(selectedMotionVector, selectedReference); - // One decoded block supplies the same retained candidate to every covered 8x8 cell. Array.Fill preserves the - // native contiguous-row write and lets later sub-8x8 blocks overwrite the shared cell in traversal order. + // One decoded block supplies the same retained candidate to every covered 8x8 cell. Filling each contiguous + // row lets the runtime select its optimized span implementation while later sub-8x8 blocks retain the + // normative ability to overwrite the shared cell in traversal order. for (int row = 0; row < fieldHeight; row++) { int rowOffset = ((firstFieldRow + row) * this.retainedMotionFieldStride) + firstFieldColumn; - Array.Fill(this.retainedMotionField, entry, rowOffset, fieldWidth); + retainedEntries.Slice(rowOffset, fieldWidth).Fill(entry); } } @@ -345,13 +409,15 @@ internal partial class Av1FrameInfo int sourceColumnCount = (this.activeModeInfoColumnCount + 1) >> MotionFieldModeInfoShift; int destinationRowCount = this.activeModeInfoRowCount >> MotionFieldModeInfoShift; int destinationColumnCount = this.activeModeInfoColumnCount >> MotionFieldModeInfoShift; + ReadOnlySpan sourceEntries = startFrameInfo.retainedMotionField!.Memory.Span; + Span destinationEntries = this.temporalMotionField!.Memory.Span; for (int blockRow = 0; blockRow < sourceRowCount; blockRow++) { int sourceRowOffset = blockRow * startFrameInfo.retainedMotionFieldStride; for (int blockColumn = 0; blockColumn < sourceColumnCount; blockColumn++) { - RetainedMotionFieldEntry source = startFrameInfo.retainedMotionField[sourceRowOffset + blockColumn]; + RetainedMotionFieldEntry source = sourceEntries[sourceRowOffset + blockColumn]; if (source.ReferenceFrame <= Av1ReferenceFrameType.Intra) { continue; @@ -361,19 +427,16 @@ internal partial class Av1FrameInfo startFrameHeader.OrderHint, startFrameInfo.motionFieldReferenceOrderHints[(int)source.ReferenceFrame]); - bool positionIsValid = Math.Abs(referenceFrameOffset) <= MaximumFrameDistance && + bool positionIsValid = Math.Abs(referenceFrameOffset) <= Av1MotionVector.MaximumTemporalDistance && referenceFrameOffset > 0 && - Math.Abs(startToCurrentFrameOffset) <= MaximumFrameDistance; + Math.Abs(startToCurrentFrameOffset) <= Av1MotionVector.MaximumTemporalDistance; if (!positionIsValid) { continue; } - Av1MotionVector projected = ProjectMotionVector( - source.MotionVector, - startToCurrentFrameOffset, - referenceFrameOffset); + Av1MotionVector projected = source.MotionVector.ProjectTemporal(startToCurrentFrameOffset, referenceFrameOffset); if (!TryGetProjectedBlockPosition( blockRow, @@ -391,7 +454,7 @@ internal partial class Av1FrameInfo // The projected vector selects the destination cell, but AV1 stores the original forward vector and // its source-to-reference distance there. Candidate scaling later uses both values for its own target. int destinationOffset = (projectedRow * this.temporalMotionFieldStride) + projectedColumn; - this.temporalMotionField[destinationOffset] = new(source.MotionVector, referenceFrameOffset); + destinationEntries[destinationOffset] = new(source.MotionVector, referenceFrameOffset); } } @@ -399,24 +462,44 @@ internal partial class Av1FrameInfo } /// - /// Scales a retained motion vector by a signed ratio of temporal distances. + /// Adds one owner for this frame state. + /// + public void AddOwner() + { + // One decoder session serializes tile parsing, reference-map updates, and output transfer. A direct count is + // therefore sufficient and avoids both atomic operations and a separately allocated shared-owner object. + this.ownerCount++; + } + + /// + /// Releases the initial owner created with this frame state. /// - /// The retained vector in one-eighth-sample units. - /// The signed start-to-current temporal distance. - /// The positive start-to-reference temporal distance. - /// The projected and AV1-range-clamped vector. - private static Av1MotionVector ProjectMotionVector(Av1MotionVector motionVector, int numerator, int denominator) + public void Dispose() { - denominator = Math.Min(denominator, MaximumFrameDistance); - numerator = Av1Math.Clip3(-MaximumFrameDistance, MaximumFrameDistance, numerator); - - // ProjectionDivisors represents 1 / denominator in Q14. Symmetric power-of-two rounding matches libaom for - // negative vectors, and the final clamp excludes the two reserved extreme motion-vector values. - int row = Av1Math.RoundPowerOf2Signed(motionVector.Row * numerator * ProjectionDivisors[denominator], 14); - int column = Av1Math.RoundPowerOf2Signed(motionVector.Column * numerator * ProjectionDivisors[denominator], 14); - row = Av1Math.Clip3(MotionVectorLowerBound + 1, MotionVectorUpperBound - 1, row); - column = Av1Math.Clip3(MotionVectorLowerBound + 1, MotionVectorUpperBound - 1, column); - return new(row, column); + if (this.ownsInitialLease) + { + // Av1TileReader can complete through both the OBU lifecycle and decoder cleanup. Keeping the initial lease + // idempotent lets either path dispose safely without affecting reference-frame or result-state owners. + this.ownsInitialLease = false; + this.ReleaseOwner(); + } + } + + /// + /// Releases one owner and returns motion-field storage after the final owner is released. + /// + public void ReleaseOwner() + { + this.ownerCount--; + if (this.ownerCount == 0) + { + // Retained and temporal fields can each be frame-sized. Return both together only after the tile reader, + // every reference or presentation frame, and the decoder's inspectable result have released ownership. + this.retainedMotionField?.Dispose(); + this.retainedMotionField = null; + this.temporalMotionField?.Dispose(); + this.temporalMotionField = null; + } } /// diff --git a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.cs b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.cs index 3992d38b6..6f944cf56 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1FrameInfo.cs @@ -8,7 +8,7 @@ namespace SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; /// /// Owns the mode, motion, segmentation, transform, coefficient, quantizer, and filter state decoded for one AV1 frame. /// -internal partial class Av1FrameInfo +internal partial class Av1FrameInfo : IDisposable { /// /// The coefficient slots reserved for one 4x4 mode-information unit: one end index followed by 16 coefficients. diff --git a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1PartitionInfo.cs b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1PartitionInfo.cs index 0f51e07aa..5d45cb100 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1PartitionInfo.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1PartitionInfo.cs @@ -295,4 +295,63 @@ internal ref struct Av1PartitionInfo return maxBlockHigh >> 2; } + + /// + /// Determines whether the current block can use the block at its top-right search position. + /// + /// The superblock width in 4x4 mode-information units. + /// when the top-right block has already been decoded; otherwise, . + public bool HasTopRight(int superblockModeInfoSize) + { + int width = this.ModeInfo.BlockSize.Get4x4WideCount(); + int height = this.ModeInfo.BlockSize.Get4x4HighCount(); + int blockSize = Math.Max(width, height); + if (blockSize > 16) + { + return false; + } + + int row = this.RowIndex & (superblockModeInfoSize - 1); + int column = this.ColumnIndex & (superblockModeInfoSize - 1); + bool hasTopRight = !((row & blockSize) != 0 && (column & blockSize) != 0); + int traversalSize = blockSize; + + // Split partitions decode three quadrants before the bottom-right quadrant. Walking the enclosing split levels + // excludes a right-hand block whenever traversal has not reached that block yet. + while (traversalSize < superblockModeInfoSize) + { + if ((column & traversalSize) == 0) + { + break; + } + + if ((column & (traversalSize << 1)) != 0 && (row & (traversalSize << 1)) != 0) + { + hasTopRight = false; + break; + } + + traversalSize <<= 1; + } + + // Rectangular partitions override the square traversal rule because their sub-blocks are decoded along the + // long axis. Earlier vertical rectangles have a completed row above; later horizontal rectangles do not. + if (width < height && ((this.ColumnIndex + width) & (height - 1)) != 0) + { + hasTopRight = true; + } + + if (width > height && (this.RowIndex & (width - 1)) != 0) + { + hasTopRight = false; + } + + // The lower-left square of a vertical-A partition precedes its right-hand rectangle in bitstream order. + if (this.Type == Av1PartitionType.VerticalA && width == height && (row & traversalSize) != 0) + { + hasTopRight = false; + } + + return hasTopRight; + } } diff --git a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileReader.cs b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileReader.cs index c800a9ad6..dd9b30dc1 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileReader.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileReader.cs @@ -8,6 +8,7 @@ using SixLabors.ImageSharp.Formats.Heif.Av1.Motion; using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit; using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline; using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; +using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; using SixLabors.ImageSharp.Formats.Heif.Av1.ReferenceFrames; using SixLabors.ImageSharp.Formats.Heif.Av1.Transform; @@ -104,6 +105,21 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable /// private InlineArray8 displacementVectorWeights; + /// + /// Reusable counts of the canonical references selected by the immediately above and left blocks. + /// + private InlineArray8 neighborReferenceCounts; + + /// + /// Reusable fixed-capacity storage for one block's weighted reference-motion-vector candidates. + /// + private readonly Av1ReferenceMotionVectors referenceMotionVectors = new(); + + /// + /// Reusable fixed-capacity state for motion-mode eligibility and local warped-motion projection. + /// + private readonly Av1MotionVariationCandidates motionVariationCandidates = new(); + /// /// Provides allocator and decoder configuration to tile entropy decoding. /// @@ -124,6 +140,11 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable /// private readonly Av1FrameInfo? primaryReferenceFrameInfo; + /// + /// The retained reconstructed frames used to determine reference scaling during inter mode parsing. + /// + private readonly Av1ReferenceFrameStore? referenceFrames; + /// /// Initializes a new instance of the class for syntax parsing without reconstruction. /// @@ -160,16 +181,13 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable this.configuration = configuration; this.SequenceHeader = sequenceHeader; this.entropyContexts = entropyContexts; + this.referenceFrames = referenceFrames; this.entropyContexts.BeginFrame(frameHeader.QuantizationParameters.BaseQIndex, primaryReferenceContext); // FrameInfo owns all traversal-order records and coefficient storage produced by the tile readers. this.FrameInfo = new(this.SequenceHeader); if (referenceFrames is not null) { - // Only the production decoder owns reconstructed references. Header-only intra readers retain their - // existing allocation profile and cannot reach inter mode parsing. - this.FrameInfo.InitializeMotionField(this.SequenceHeader, this.FrameHeader, referenceFrames); - byte? primaryReferenceSlot = this.FrameHeader.PrimaryReferenceSlot; if (primaryReferenceSlot is not null) { @@ -201,11 +219,28 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable } catch { - // The reader is not returned when its second context allocation fails, so release - // the first rent here rather than relying on an owner that the caller cannot reach. + // The reader is not returned when its second context allocation fails, so release the first rent here + // rather than relying on an owner that the caller cannot reach. this.aboveNeighborContext.Dispose(); throw; } + + if (referenceFrames is not null) + { + try + { + // Motion storage is acquired after every other constructor allocation. If this final acquisition + // fails, the catch can return every successfully created allocator-owned resource in one place. + this.FrameInfo.InitializeMotionField(configuration, this.SequenceHeader, this.FrameHeader, referenceFrames); + } + catch + { + this.aboveNeighborContext.Dispose(); + this.leftNeighborContext.Dispose(); + this.FrameInfo.Dispose(); + throw; + } + } } /// @@ -287,12 +322,13 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable public Av1FrameEntropyContext FrameEntropyContext => this.entropyContexts.Published; /// - /// Returns the tile-neighbor context storage to the configured memory allocator. + /// Returns tile-neighbor storage and the reader's frame-state lease to the configured memory allocator. /// public void Dispose() { this.aboveNeighborContext.Dispose(); this.leftNeighborContext.Dispose(); + this.FrameInfo.Dispose(); } /// @@ -1552,17 +1588,18 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable } else { - this.ReadInterFrameModeInfo(ref reader, ref partitionInfo); + this.ReadInterFrameModeInfo(ref reader, ref partitionInfo, tileInfo); } } /// - /// Reads the common inter-frame block prefix and the intra-coded-block prediction branch in bitstream order. + /// Reads the common inter-frame block prefix and the supported intra or single-reference inter prediction branch in bitstream order. /// /// The tile symbol decoder. /// The current coding block and its neighbors. - /// Implements the prefix and intra branch of AV1 section 5.11.7. - internal void ReadInterFrameModeInfo(ref Av1SymbolDecoder reader, ref Av1PartitionInfo partitionInfo) + /// The active tile boundaries used by reference-motion-vector searches. + /// Implements the prefix, intra, and single-reference translational branches of AV1 section 5.11.7. + internal void ReadInterFrameModeInfo(ref Av1SymbolDecoder reader, ref Av1PartitionInfo partitionInfo, Av1TileInfo tileInfo) { Av1BlockModeInfo modeInfo = partitionInfo.ModeInfo; modeInfo.MotionVectors.Clear(); @@ -1587,7 +1624,189 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable bool isInterBlock = modeInfo.SkipMode || this.ReadIsInter(ref reader, ref partitionInfo); if (isInterBlock) { - throw new NotSupportedException("AV1 inter-coded block prediction is not implemented."); + modeInfo.SetPaletteSizes(0, 0); + modeInfo.UvMode = Av1ChromaPredictionMode.DC; + this.ReadReferenceFrames(ref reader, ref partitionInfo); + + Av1ReferenceFrameType referenceFrame = modeInfo.ReferenceFrames[0]; + if (modeInfo.SkipMode || modeInfo.ReferenceFrames[1] > Av1ReferenceFrameType.Intra) + { + throw new NotSupportedException("AV1 compound-reference block prediction is not implemented."); + } + + Av1ReferenceMotionVectors referenceMotionVectors = this.referenceMotionVectors; + referenceMotionVectors.Build( + ref partitionInfo, + tileInfo, + this.FrameInfo, + this.SequenceHeader, + this.FrameHeader, + referenceFrame); + + ObuSegmentationParameters segmentationParameters = this.FrameHeader.SegmentationParameters; + int segmentId = modeInfo.SegmentId; + bool usesForcedGlobalMotion = + segmentationParameters.IsFeatureActive(segmentId, ObuSegmentationLevelFeature.Skip) || + segmentationParameters.IsFeatureActive(segmentId, ObuSegmentationLevelFeature.GlobalMotionVector); + + modeInfo.ReferenceMotionVectorIndex = 0; + modeInfo.YMode = usesForcedGlobalMotion + ? Av1PredictionMode.GlobalMotionVector + : reader.ReadInterMode(referenceMotionVectors.ModeContext); + + if (modeInfo.YMode == Av1PredictionMode.NewMotionVector) + { + // NEWMV can advance across candidates zero through two. Each transmitted one selects the next + // candidate and exposes one further DRL decision when the stack contains it. + for (int index = 0; index < 2 && referenceMotionVectors.Count > index + 1; index++) + { + int context = Av1SymbolContextHelper.GetDrlContext(referenceMotionVectors.Weights, index); + bool advance = reader.ReadDrl(context); + modeInfo.ReferenceMotionVectorIndex = (byte)(index + (advance ? 1 : 0)); + if (!advance) + { + break; + } + } + } + else if (modeInfo.YMode == Av1PredictionMode.NearMotionVector) + { + // NEARMV reserves candidate zero for NEARESTMV, so its two DRL decisions examine pairs one/two and + // two/three while storing a zero-based offset from the first near candidate. + for (int index = 1; index < 3 && referenceMotionVectors.Count > index + 1; index++) + { + int context = Av1SymbolContextHelper.GetDrlContext(referenceMotionVectors.Weights, index); + bool advance = reader.ReadDrl(context); + modeInfo.ReferenceMotionVectorIndex = (byte)(index + (advance ? 1 : 0) - 1); + if (!advance) + { + break; + } + } + } + + Av1MotionVectorPrecision precision = this.FrameHeader.ForceIntegerMotionVector + ? Av1MotionVectorPrecision.Integer + : this.FrameHeader.AllowHighPrecisionMotionVector ? Av1MotionVectorPrecision.EighthSample : Av1MotionVectorPrecision.QuarterSample; + + Av1MotionVector motionVector = modeInfo.YMode switch + { + Av1PredictionMode.NewMotionVector => reader.ReadMotionVector( + referenceMotionVectors.GetNewReference(modeInfo.ReferenceMotionVectorIndex), + precision), + Av1PredictionMode.NearestMotionVector => referenceMotionVectors.Nearest, + Av1PredictionMode.NearMotionVector => referenceMotionVectors.GetNearReference(modeInfo.ReferenceMotionVectorIndex), + Av1PredictionMode.GlobalMotionVector => this.FrameHeader.GetGlobalMotionParameters()[(int)referenceFrame - 1].GetMotionVector( + this.FrameHeader.AllowHighPrecisionMotionVector, + modeInfo.BlockSize, + new Point(partitionInfo.ColumnIndex, partitionInfo.RowIndex), + this.FrameHeader.ForceIntegerMotionVector), + _ => throw new InvalidImageContentException("Invalid single-reference AV1 inter mode.") + }; + + if (!motionVector.IsValid) + { + throw new InvalidImageContentException("AV1 motion-vector component is outside the permitted range."); + } + + modeInfo.MotionVectors[0] = motionVector; + modeInfo.MotionMode = Av1MotionMode.SimpleTranslation; + + int minimumBlockDimension = Math.Min(modeInfo.BlockSize.GetWidth(), modeInfo.BlockSize.GetHeight()); + if (this.SequenceHeader.EnableInterIntraCompound && + modeInfo.BlockSize is >= Av1BlockSize.Block8x8 and <= Av1BlockSize.Block32x32 && + reader.ReadIsInterIntra(modeInfo.BlockSize)) + { + // AV1 assigns this syntax only to the contiguous Block8x8 through Block32x32 enum range; similarly + // dimensioned extended rectangles occur later in the enum and must not consume a flag. A false flag + // continues into motion-mode syntax even while the selected inter-intra predictor remains unsupported. + throw new NotSupportedException("AV1 inter-intra block prediction is not implemented."); + } + + if (this.FrameHeader.IsMotionModeSwitchable && minimumBlockDimension >= 8 && !modeInfo.SkipMode) + { + Av1MotionVariationCandidates candidates = this.motionVariationCandidates; + candidates.Build(ref partitionInfo, tileInfo, this.SequenceHeader, this.FrameHeader, referenceFrame); + + Av1GlobalMotionParameters selectedGlobalMotion = this.FrameHeader.GetGlobalMotionParameters()[(int)referenceFrame - 1]; + bool hasFixedGlobalMotionMode = + !this.FrameHeader.ForceIntegerMotionVector && + modeInfo.YMode == Av1PredictionMode.GlobalMotionVector && + selectedGlobalMotion.Type > Av1GlobalMotionType.Translation; + + if (candidates.HasOverlappableNeighbor && !hasFixedGlobalMotionMode) + { + bool allowWarpedMotion = false; + if (candidates.Count > 0 && this.FrameHeader.AllowWarpedMotion && !this.FrameHeader.ForceIntegerMotionVector) + { + int canonicalReferenceIndex = (int)referenceFrame - (int)Av1ReferenceFrameType.Last; + uint referenceSlot = this.FrameHeader.GetReferenceFrameIndices()[canonicalReferenceIndex]; + Av1FrameBuffer referenceFrameBuffer = this.referenceFrames!.Resolve((int)referenceSlot)!.FrameBuffer; + + // Local warped motion is excluded for a scaled reference. Width and height equality are the + // identity-scale test because both dimensions form the decoder's reference scale factors. + allowWarpedMotion = + referenceFrameBuffer.Width == this.FrameHeader.FrameSize.FrameWidth && + referenceFrameBuffer.Height == this.FrameHeader.FrameSize.FrameHeight; + } + + modeInfo.MotionMode = reader.ReadMotionMode(modeInfo.BlockSize, allowWarpedMotion); + if (modeInfo.MotionMode != Av1MotionMode.SimpleTranslation) + { + throw new NotSupportedException($"AV1 {modeInfo.MotionMode} block prediction is not implemented."); + } + } + } + + Span interpolationFilters = modeInfo.InterpolationFilters; + Av1InterpolationFilter frameInterpolationFilter = this.FrameHeader.InterpolationFilter; + Av1GlobalMotionParameters globalMotion = this.FrameHeader.GetGlobalMotionParameters()[(int)referenceFrame - 1]; + bool usesNonTranslationalGlobalMotion = + modeInfo.YMode == Av1PredictionMode.GlobalMotionVector && + minimumBlockDimension >= 8 && + globalMotion.Type != Av1GlobalMotionType.Translation; + + if (modeInfo.SkipMode || modeInfo.MotionMode == Av1MotionMode.Warped || usesNonTranslationalGlobalMotion) + { + // Blocks that do not use separable interpolation carry no filter symbols. A switchable frame falls + // back to the regular family so every stored mode record contains an actual predictor selection. + interpolationFilters.Fill( + frameInterpolationFilter == Av1InterpolationFilter.Switchable + ? Av1InterpolationFilter.Regular + : frameInterpolationFilter); + } + else if (frameInterpolationFilter != Av1InterpolationFilter.Switchable) + { + interpolationFilters.Fill(frameInterpolationFilter); + } + else + { + // Filter storage is vertical then horizontal. AV1 transmits in the same order and reuses the vertical + // choice for both axes when the sequence disables independent dual-filter selection. + int verticalContext = Av1SymbolContextHelper.GetSwitchableInterpolationContext( + modeInfo, + partitionInfo.AboveModeInfo, + partitionInfo.LeftModeInfo, + direction: 0); + + interpolationFilters[0] = reader.ReadSwitchableInterpolationFilter(verticalContext); + if (this.SequenceHeader.EnableDualFilter) + { + int horizontalContext = Av1SymbolContextHelper.GetSwitchableInterpolationContext( + modeInfo, + partitionInfo.AboveModeInfo, + partitionInfo.LeftModeInfo, + direction: 1); + + interpolationFilters[1] = reader.ReadSwitchableInterpolationFilter(horizontalContext); + } + else + { + interpolationFilters[1] = interpolationFilters[0]; + } + } + + return; } modeInfo.ReferenceFrames[0] = Av1ReferenceFrameType.Intra; @@ -2594,6 +2813,95 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable return reader.ReadIsInter(context); } + /// + /// Reads or infers the retained reference-frame labels selected by an inter-coded block. + /// + /// The tile symbol decoder. + /// The current coding block and its available neighbors. + private void ReadReferenceFrames(ref Av1SymbolDecoder reader, ref Av1PartitionInfo partitionInfo) + { + Av1BlockModeInfo modeInfo = partitionInfo.ModeInfo; + Span references = modeInfo.ReferenceFrames; + if (modeInfo.SkipMode) + { + ObuSkipModeParameters skipModeParameters = this.FrameHeader.SkipModeParameters; + references[0] = skipModeParameters.FirstReferenceFrame; + references[1] = skipModeParameters.SecondReferenceFrame; + return; + } + + ObuSegmentationParameters segmentationParameters = this.FrameHeader.SegmentationParameters; + int segmentId = modeInfo.SegmentId; + if (segmentationParameters.IsFeatureActive(segmentId, ObuSegmentationLevelFeature.ReferenceFrame)) + { + references[0] = (Av1ReferenceFrameType)segmentationParameters.FeatureData[ + segmentId, + (int)ObuSegmentationLevelFeature.ReferenceFrame]; + + references[1] = Av1ReferenceFrameType.None; + return; + } + + if (segmentationParameters.IsFeatureActive(segmentId, ObuSegmentationLevelFeature.Skip) || + segmentationParameters.IsFeatureActive(segmentId, ObuSegmentationLevelFeature.GlobalMotionVector)) + { + references[0] = Av1ReferenceFrameType.Last; + references[1] = Av1ReferenceFrameType.None; + return; + } + + bool compoundReferenceAllowed = Math.Min(modeInfo.BlockSize.GetWidth(), modeInfo.BlockSize.GetHeight()) >= 8; + if (compoundReferenceAllowed && this.FrameHeader.ReferenceMode == ObuReferenceMode.ReferenceModeSelect) + { + int context = Av1SymbolContextHelper.GetReferenceModeContext( + partitionInfo.AboveModeInfo, + partitionInfo.LeftModeInfo); + + if (reader.ReadIsCompoundReference(context)) + { + throw new NotSupportedException("AV1 compound-reference block prediction is not implemented."); + } + } + + Span referenceCounts = this.neighborReferenceCounts; + Av1SymbolContextHelper.CollectNeighborReferenceCounts( + partitionInfo.AboveModeInfo, + partitionInfo.LeftModeInfo, + referenceCounts); + + Av1ReferenceFrameType reference; + if (reader.ReadSingleReferenceIsBackward(Av1SymbolContextHelper.GetSingleReferenceBackwardContext(referenceCounts))) + { + if (reader.ReadSingleReferenceIsAlternate(Av1SymbolContextHelper.GetSingleReferenceAlternateContext(referenceCounts))) + { + reference = Av1ReferenceFrameType.Alternate; + } + else + { + reference = reader.ReadSingleReferenceIsAlternate2( + Av1SymbolContextHelper.GetSingleReferenceAlternate2Context(referenceCounts)) + ? Av1ReferenceFrameType.Alternate2 + : Av1ReferenceFrameType.Backward; + } + } + else if (reader.ReadSingleReferenceIsLast3OrGolden( + Av1SymbolContextHelper.GetSingleReferenceLast3OrGoldenContext(referenceCounts))) + { + reference = reader.ReadSingleReferenceIsGolden(Av1SymbolContextHelper.GetSingleReferenceGoldenContext(referenceCounts)) + ? Av1ReferenceFrameType.Golden + : Av1ReferenceFrameType.Last3; + } + else + { + reference = reader.ReadSingleReferenceIsLast2(Av1SymbolContextHelper.GetSingleReferenceLast2Context(referenceCounts)) + ? Av1ReferenceFrameType.Last2 + : Av1ReferenceFrameType.Last; + } + + references[0] = reference; + references[1] = Av1ReferenceFrameType.None; + } + /// /// Reads and accumulates a superblock quantizer-index delta when the block carries one. /// diff --git a/src/ImageSharp/Formats/Heif/Av1/Transform/Av1BlockDecoder.cs b/src/ImageSharp/Formats/Heif/Av1/Transform/Av1BlockDecoder.cs index 1c59d01f7..6dd636d01 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Transform/Av1BlockDecoder.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Transform/Av1BlockDecoder.cs @@ -3,12 +3,16 @@ using System.Buffers; using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using SixLabors.ImageSharp.Formats.Heif.Av1.Motion; using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit; using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.LoopFilter; using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.Quantizers; using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.ChromaFromLuma; +using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.IntraBlockCopy; +using SixLabors.ImageSharp.Formats.Heif.Av1.ReferenceFrames; using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; namespace SixLabors.ImageSharp.Formats.Heif.Av1.Transform; @@ -43,6 +47,11 @@ internal sealed class Av1BlockDecoder : IDisposable /// private readonly Av1InverseQuantizer inverseQuantizer; + /// + /// The retained reconstructed frames addressable by inter prediction, when reconstruction belongs to a decoder session. + /// + private readonly Av1ReferenceFrameStore? referenceFrames; + /// /// Owns the reusable raster-order inverse-quantization buffer. /// @@ -86,18 +95,21 @@ internal sealed class Av1BlockDecoder : IDisposable /// The frame buffer receiving reconstructed samples. /// The transform-size map populated while reconstructing blocks. /// The inverse quantizer carrying the active superblock delta-Q state. + /// The retained reconstructed frames selected by inter blocks, or for intra-only use. public Av1BlockDecoder( ObuSequenceHeader sequenceHeader, ObuFrameHeader frameHeader, Av1FrameBuffer frameBuffer, Av1LoopFilterContext loopFilterContext, - Av1InverseQuantizer inverseQuantizer) + Av1InverseQuantizer inverseQuantizer, + Av1ReferenceFrameStore? referenceFrames = null) { this.sequenceHeader = sequenceHeader; this.frameHeader = frameHeader; this.frameBuffer = frameBuffer; this.loopFilterContext = loopFilterContext; this.inverseQuantizer = inverseQuantizer; + this.referenceFrames = referenceFrames; int ySize = (1 << this.sequenceHeader.SuperblockSizeLog2) * (1 << this.sequenceHeader.SuperblockSizeLog2); // One scratch plane is reused for every transform unit. Its maximum size must cover a complete superblock @@ -113,7 +125,12 @@ internal sealed class Av1BlockDecoder : IDisposable { inverseQuantizationOwner = this.frameBuffer.MemoryAllocator.Allocate(inverseQuantizationSize); transformWorkspaceOwner = this.frameBuffer.MemoryAllocator.Allocate(Av1TransformWorkspace.MaximumLength); - predictionScratchOwner = this.frameBuffer.MemoryAllocator.Allocate(Av1PredictionDecoder.ScratchLength); + int maximumBlockLength = 1 << sequenceHeader.SuperblockSizeLog2; + int predictionScratchLength = Math.Max( + Av1PredictionDecoder.ScratchLength, + Av1InterPredictor.GetScratchLength(maximumBlockLength, maximumBlockLength)); + + predictionScratchOwner = this.frameBuffer.MemoryAllocator.Allocate(predictionScratchLength); this.inverseQuantizationOwner = inverseQuantizationOwner; this.transformWorkspaceOwner = transformWorkspaceOwner; @@ -216,6 +233,34 @@ internal sealed class Av1BlockDecoder : IDisposable ? (maxBlocksWide * maxBlocksHigh) >> ((colorConfig.SubSamplingX ? 1 : 0) + (colorConfig.SubSamplingY ? 1 : 0)) : modeInfo.GetTransformUnitCount(Av1Plane.U); + bool isInterBlock = modeInfo.ReferenceFrames[0] >= Av1ReferenceFrameType.Last; + Av1FrameBuffer? referenceFrameBuffer = null; + if (isInterBlock) + { + int canonicalReferenceIndex = (int)modeInfo.ReferenceFrames[0] - (int)Av1ReferenceFrameType.Last; + Av1GlobalMotionParameters globalMotion = this.frameHeader.GetGlobalMotionParameters()[canonicalReferenceIndex]; + if (modeInfo.YMode == Av1PredictionMode.GlobalMotionVector && + Math.Min(modeInfo.BlockSize.GetWidth(), modeInfo.BlockSize.GetHeight()) >= 8 && + globalMotion.Type > Av1GlobalMotionType.Translation) + { + // A qualifying rotation/zoom or affine GLOBALMV block samples the complete warped model. Its center + // vector is a stack fallback only and cannot be substituted into the translational predictor. + throw new NotSupportedException("AV1 non-translational global prediction is not implemented."); + } + + uint referenceSlot = this.frameHeader.GetReferenceFrameIndices()[canonicalReferenceIndex]; + + // The uncompressed-header parser validates each selected slot and the reference store remains unchanged + // until frame reconstruction completes, so every parsed inter block resolves the same retained owner. + referenceFrameBuffer = this.referenceFrames!.Resolve((int)referenceSlot)!.FrameBuffer; + if (referenceFrameBuffer.Width != this.frameHeader.FrameSize.FrameWidth || referenceFrameBuffer.Height != this.frameHeader.FrameSize.FrameHeight) + { + // Scaled prediction changes both the source coordinate and the per-output-sample step. Running the + // unit-step predictor here would silently reconstruct valid scaled-reference streams incorrectly. + throw new NotSupportedException("AV1 scaled-reference inter prediction is not implemented."); + } + } + bool highBitDepth = this.frameBuffer.BytesPerSample == 2; for (int plane = 0; plane < colorConfig.PlaneCount; plane++) { @@ -270,6 +315,102 @@ internal sealed class Av1BlockDecoder : IDisposable blockReconstructionBuffer = this.frameBuffer.DeriveBlockPointer((Av1Plane)plane, pixelPosition, subX, subY, out reconstructionStride); } + if (isInterBlock) + { + Av1MotionVector motionVector = modeInfo.MotionVectors[0]; + int predictionWidth = Math.Max(4, blockSize.GetWidth() >> subX); + int predictionHeight = Math.Max(4, blockSize.GetHeight() >> subY); + + // AV1 predicts the complete declared plane block even when its luma extent crosses the frame boundary. + // Subsampled dimensions retain the mandatory four-sample minimum used by set_plane_n4 in libaom. + int horizontalMotionQ4 = motionVector.Column << (1 - subX); + int verticalMotionQ4 = motionVector.Row << (1 - subY); + int horizontalExtensionQ4 = (4 + predictionWidth) << 4; + int verticalExtensionQ4 = (4 + predictionHeight) << 4; + int horizontalEdgeScale = 1 << (1 - subX); + int verticalEdgeScale = 1 << (1 - subY); + + // The UMV clamp is expressed in one-sixteenth plane-sample units. A 128-sample block can legally + // address 135 samples beyond an edge once its prediction extent and eight-tap filter support are + // included; the frame-owned 144-sample luma border keeps that source directly addressable. + horizontalMotionQ4 = Av1Math.Clip3( + (partitionInfo.ModeBlockToLeftEdge * horizontalEdgeScale) - horizontalExtensionQ4, + (partitionInfo.ModeBlockToRightEdge * horizontalEdgeScale) + horizontalExtensionQ4 - 16, + horizontalMotionQ4); + + verticalMotionQ4 = Av1Math.Clip3( + (partitionInfo.ModeBlockToTopEdge * verticalEdgeScale) - verticalExtensionQ4, + (partitionInfo.ModeBlockToBottomEdge * verticalEdgeScale) + verticalExtensionQ4 - 16, + verticalMotionQ4); + + int sourceColumnQ4 = (pixelPosition.X << 4) + horizontalMotionQ4; + int sourceRowQ4 = (pixelPosition.Y << 4) + verticalMotionQ4; + + // Motion vectors use one-eighth luma-sample units. Shifting by one minus the plane subsampling converts + // them directly to the predictor's one-sixteenth-plane-sample phase; masking then preserves the signed + // floor used to select the integer source sample. + int horizontalPhase = sourceColumnQ4 & 15; + int verticalPhase = sourceRowQ4 & 15; + Span predictionScratch = this.predictionScratchOwner.Memory.Span; + + if (highBitDepth) + { + Span source = referenceFrameBuffer!.GetPaddedPlaneSpan16( + (Av1Plane)plane, + subX, + subY, + out int sourceStride, + out Point sourceOrigin); + + int sourceIndex = + ((sourceOrigin.Y + (sourceRowQ4 >> 4)) * sourceStride) + sourceOrigin.X + (sourceColumnQ4 >> 4); + + Span destination = + MemoryMarshal.Cast(highBitDepthBlockReconstructionBuffer[reconstructionStride..]); + + Av1InterPredictor.Predict( + source, + sourceStride, + sourceIndex, + destination, + reconstructionStride, + predictionWidth, + predictionHeight, + modeInfo.InterpolationFilters[1], + modeInfo.InterpolationFilters[0], + horizontalPhase, + verticalPhase, + this.frameBuffer.BitDepth.GetBitCount(), + predictionScratch); + } + else + { + Span source = referenceFrameBuffer!.GetPaddedPlaneSpan( + (Av1Plane)plane, + subX, + subY, + out int sourceStride, + out Point sourceOrigin); + + int sourceIndex = + ((sourceOrigin.Y + (sourceRowQ4 >> 4)) * sourceStride) + sourceOrigin.X + (sourceColumnQ4 >> 4); + + Av1InterPredictor.Predict( + source, + sourceStride, + sourceIndex, + blockReconstructionBuffer[reconstructionStride..], + reconstructionStride, + predictionWidth, + predictionHeight, + modeInfo.InterpolationFilters[1], + modeInfo.InterpolationFilters[0], + horizontalPhase, + verticalPhase, + predictionScratch); + } + } + for (int tu = 0; tu < transformUnitCount; tu++) { Span transformBlockReconstructionBuffer = default; @@ -372,7 +513,7 @@ internal sealed class Av1BlockDecoder : IDisposable sourcePhaseY != 0); } } - else + else if (!isInterBlock) { // Conventional intra prediction consumes the reference-prefixed destination span before the // transform residual is reconstructed over its first output row. diff --git a/src/ImageSharp/Formats/Heif/Av1HeifItemDecoder.cs b/src/ImageSharp/Formats/Heif/Av1HeifItemDecoder.cs index 479ad8931..bb74c3260 100644 --- a/src/ImageSharp/Formats/Heif/Av1HeifItemDecoder.cs +++ b/src/ImageSharp/Formats/Heif/Av1HeifItemDecoder.cs @@ -55,7 +55,7 @@ internal class Av1HeifItemDecoder : IHeifItemDecoder, IHeifAlpha byte operatingPointIndex = item.Av1OperatingPointSelector?.Index ?? 0; using Av1Decoder decoder = new(options.Configuration, operatingPointIndex); - Image image = decoder.Decode(itemData, colorProfile, codecConfiguration); + Image image = decoder.Decode(itemData, colorProfile, codecConfiguration, item.Av1LayeredImageIndex); HeifMetadata metadata = image.Metadata.GetHeifMetadata(); metadata.CompressionMethod = this.CompressionMethod; metadata.BitDepth = codecConfiguration.BitDepth; @@ -95,7 +95,8 @@ internal class Av1HeifItemDecoder : IHeifItemDecoder, IHeifAlpha destination, outputSize, destinationRectangle, - premultiplied); + premultiplied, + item.Av1LayeredImageIndex); } /// diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1FrameBufferTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1FrameBufferTests.cs index 37f356af0..d012f3709 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1FrameBufferTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1FrameBufferTests.cs @@ -12,11 +12,42 @@ using SixLabors.ImageSharp.Tests.Memory; namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; /// -/// Verifies AV1 frame and block-decoder allocation ownership during constructor rollback. +/// Verifies AV1 frame-plane allocation contracts and constructor rollback ownership. /// [Trait("Format", "Avif")] public class Av1FrameBufferTests { + /// + /// Verifies that padded frame planes remain contiguous when the allocator would otherwise split the buffer. + /// + [Fact] + public void ConstructorRequestsContiguousPaddedPlanes() + { + TestMemoryAllocator allocator = new() { BufferCapacityInBytes = 10_000 }; + Configuration configuration = Configuration.Default.Clone(); + configuration.MemoryAllocator = allocator; + ObuSequenceHeader sequenceHeader = new() + { + MaxFrameWidth = 64, + MaxFrameHeight = 64, + ColorConfig = new ObuColorConfig + { + IsMonochrome = true, + BitDepth = Av1BitDepth.EightBit + } + }; + + using Av1FrameBuffer frameBuffer = new( + configuration, + sequenceHeader, + Av1ColorFormat.Yuv400, + false); + + MemoryGroup memoryGroup = frameBuffer.BufferY!.FastMemoryGroup; + Assert.Equal(1, memoryGroup.Count); + Assert.True(memoryGroup.TotalLength > allocator.BufferCapacityInBytes); + } + /// /// Verifies that a failure while renting the final chroma plane releases every previously rented plane. /// diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1GlobalMotionParametersTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1GlobalMotionParametersTests.cs new file mode 100644 index 000000000..37e56b0fc --- /dev/null +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1GlobalMotionParametersTests.cs @@ -0,0 +1,85 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using SixLabors.ImageSharp.Formats.Heif.Av1; +using SixLabors.ImageSharp.Formats.Heif.Av1.Motion; +using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; + +namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; + +/// +/// Verifies AV1 global-motion-vector derivation against the fixed-point rules used by the reference decoder. +/// +[Trait("Format", "Avif")] +public class Av1GlobalMotionParametersTests +{ + /// + /// Verifies that an identity model produces no displacement at every block position. + /// + [Fact] + public void IdentityModelProducesZeroMotionVector() + { + Av1GlobalMotionParameters parameters = Av1GlobalMotionParameters.Identity; + + Av1MotionVector actual = parameters.GetMotionVector( + allowHighPrecisionMotionVector: true, + Av1BlockSize.Block128x128, + new Point(31, 17), + forceIntegerMotionVector: false); + + Assert.Equal(default, actual); + } + + /// + /// Verifies the published AV1 translation-component ordering and optional integer precision reduction. + /// + [Theory] + [InlineData(false, 19, -21)] + [InlineData(true, 16, -24)] + public void TranslationModelMatchesNormativeComponentOrdering(bool forceIntegerMotionVector, int expectedRow, int expectedColumn) + { + Av1GlobalMotionParameters parameters = Av1GlobalMotionParameters.Identity; + parameters.Type = Av1GlobalMotionType.Translation; + + // Translation parameters retain sixteen fractional bits; the derived vector retains three. + parameters[0] = 19 << 13; + parameters[1] = -21 << 13; + + Av1MotionVector actual = parameters.GetMotionVector( + allowHighPrecisionMotionVector: true, + Av1BlockSize.Block16x16, + new Point(4, 7), + forceIntegerMotionVector); + + Assert.Equal(new Av1MotionVector(expectedRow, expectedColumn), actual); + } + + /// + /// Verifies affine evaluation at the AV1 block center for high- and low-precision vector output. + /// + [Theory] + [InlineData(true, 1, 3)] + [InlineData(false, 0, 2)] + public void AffineModelEvaluatesBlockCenter(bool allowHighPrecisionMotionVector, int expectedRow, int expectedColumn) + { + Av1GlobalMotionParameters parameters = Av1GlobalMotionParameters.Identity; + parameters.Type = Av1GlobalMotionType.Affine; + + // The 8x8 block at mode-info position (2, 3) has center (11, 15). These deltas produce horizontal and + // vertical fixed-point offsets that exercise signed rounding at the selected output precision. + parameters[0] = 2048; + parameters[1] = -1024; + parameters[2] = Av1GlobalMotionParameters.ModelScale + 1024; + parameters[3] = 512; + parameters[4] = -256; + parameters[5] = Av1GlobalMotionParameters.ModelScale + 768; + + Av1MotionVector actual = parameters.GetMotionVector( + allowHighPrecisionMotionVector, + Av1BlockSize.Block8x8, + new Point(2, 3), + forceIntegerMotionVector: false); + + Assert.Equal(new Av1MotionVector(expectedRow, expectedColumn), actual); + } +} diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterFrameModeInfoTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterFrameModeInfoTests.cs index 5b35a4391..a0f3a40f8 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterFrameModeInfoTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterFrameModeInfoTests.cs @@ -4,15 +4,17 @@ using System.Buffers; using SixLabors.ImageSharp.Formats.Heif.Av1; using SixLabors.ImageSharp.Formats.Heif.Av1.Entropy; +using SixLabors.ImageSharp.Formats.Heif.Av1.Motion; using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit; using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; +using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; using SixLabors.ImageSharp.Memory; namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; /// -/// Verifies the common inter-frame mode prefix and its intra-coded-block branch. +/// Verifies inter-frame block-prefix, intra-block selection, skip-mode, and interpolation-filter syntax. /// [Trait("Format", "Avif")] public class Av1InterFrameModeInfoTests @@ -38,9 +40,9 @@ public class Av1InterFrameModeInfoTests writer.WriteSymbol(false, intraInter); writer.WriteSymbol((int)Av1PredictionMode.DC, yMode); using IMemoryOwner encoded = writer.Exit(); - Av1SymbolDecoder decoder = new(Configuration.Default, encoded.GetSpan(), 0, updateCdf: true); + Av1SymbolDecoder decoder = new(Configuration.Default, encoded.Memory.Span, 0, updateCdf: true); - tileReader.ReadInterFrameModeInfo(ref decoder, ref partitionInfo); + tileReader.ReadInterFrameModeInfo(ref decoder, ref partitionInfo, new Av1TileInfo(0, 0, frameHeader)); Assert.False(modeInfo.SkipMode); Assert.False(modeInfo.Skip); @@ -74,6 +76,85 @@ public class Av1InterFrameModeInfoTests Assert.True(modeInfo.Skip); } + /// + /// Verifies switchable interpolation-filter decoding with shared and independent axis selections. + /// + /// Whether the horizontal axis carries an independent filter symbol. + /// The expected horizontal interpolation filter. + [Theory] + [InlineData(false, (int)Av1InterpolationFilter.Smooth)] + [InlineData(true, (int)Av1InterpolationFilter.Sharp)] + public void ReadInterFrameModeInfoReadsInterpolationFilters( + bool enableDualFilter, + int expectedHorizontalFilter) + { + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(); + sequenceHeader.EnableDualFilter = enableDualFilter; + ObuFrameHeader frameHeader = CreateFrameHeader(); + frameHeader.InterpolationFilter = Av1InterpolationFilter.Switchable; + + // Forcing segment zero to GLOBALMV removes reference and inter-mode symbols from this focused fixture. A + // translational global model still requires interpolation, leaving only the filter branch under test. + ObuSegmentationParameters segmentationParameters = frameHeader.SegmentationParameters; + segmentationParameters.Enabled = true; + segmentationParameters.FeatureEnabled[0, (int)ObuSegmentationLevelFeature.GlobalMotionVector] = true; + frameHeader.GetGlobalMotionParameters()[0].Type = Av1GlobalMotionType.Translation; + + using Av1TileReader tileReader = new(Configuration.Default, sequenceHeader, frameHeader); + Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block8x8, Point.Empty); + Av1SuperblockInfo superblockInfo = new(tileReader.FrameInfo, Point.Empty); + Av1PartitionInfo partitionInfo = new(modeInfo, superblockInfo, false, Av1PartitionType.None); + using Av1SymbolWriter writer = new(Configuration.Default, 2, updateCdf: true); + writer.WriteSymbol(false, Av1DefaultDistributions.Skip[0]); + writer.WriteSymbol((int)Av1InterpolationFilter.Smooth, Av1DefaultDistributions.SwitchableInterpolation[3]); + if (enableDualFilter) + { + writer.WriteSymbol((int)Av1InterpolationFilter.Sharp, Av1DefaultDistributions.SwitchableInterpolation[11]); + } + + using IMemoryOwner encoded = writer.Exit(); + Av1SymbolDecoder decoder = new(Configuration.Default, encoded.GetSpan(), 0, updateCdf: true); + + tileReader.ReadInterFrameModeInfo(ref decoder, ref partitionInfo, new Av1TileInfo(0, 0, frameHeader)); + + Assert.Equal(Av1InterpolationFilter.Smooth, modeInfo.InterpolationFilters[0]); + Assert.Equal((Av1InterpolationFilter)expectedHorizontalFilter, modeInfo.InterpolationFilters[1]); + } + + /// + /// Verifies that an identity global-motion block omits switchable interpolation-filter symbols. + /// + [Fact] + public void ReadInterFrameModeInfoOmitsInterpolationFiltersForIdentityGlobalMotion() + { + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(); + sequenceHeader.EnableDualFilter = true; + ObuFrameHeader frameHeader = CreateFrameHeader(); + frameHeader.InterpolationFilter = Av1InterpolationFilter.Switchable; + ObuSegmentationParameters segmentationParameters = frameHeader.SegmentationParameters; + segmentationParameters.Enabled = true; + segmentationParameters.FeatureEnabled[0, (int)ObuSegmentationLevelFeature.GlobalMotionVector] = true; + + using Av1TileReader tileReader = new(Configuration.Default, sequenceHeader, frameHeader); + Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block8x8, Point.Empty); + Av1SuperblockInfo superblockInfo = new(tileReader.FrameInfo, Point.Empty); + Av1PartitionInfo partitionInfo = new(modeInfo, superblockInfo, false, Av1PartitionType.None); + using Av1SymbolWriter writer = new(Configuration.Default, 3, updateCdf: true); + writer.WriteSymbol(false, Av1DefaultDistributions.Skip[0]); + + // These sentinel symbols remain unread because pinned libaom classifies every GLOBALMV model other than + // TRANSLATION as non-translational for interpolation syntax, including the default identity model. + writer.WriteSymbol((int)Av1InterpolationFilter.Smooth, Av1DefaultDistributions.SwitchableInterpolation[3]); + writer.WriteSymbol((int)Av1InterpolationFilter.Sharp, Av1DefaultDistributions.SwitchableInterpolation[11]); + using IMemoryOwner encoded = writer.Exit(); + Av1SymbolDecoder decoder = new(Configuration.Default, encoded.GetSpan(), 0, updateCdf: true); + + tileReader.ReadInterFrameModeInfo(ref decoder, ref partitionInfo, new Av1TileInfo(0, 0, frameHeader)); + + Assert.Equal(Av1InterpolationFilter.Regular, modeInfo.InterpolationFilters[0]); + Assert.Equal(Av1InterpolationFilter.Regular, modeInfo.InterpolationFilters[1]); + } + /// /// Invokes the ref-struct mode parser for exception assertions that cannot capture its parameters directly. /// @@ -95,7 +176,7 @@ public class Av1InterFrameModeInfoTests }; Av1SymbolDecoder decoder = new(Configuration.Default, encoded.Span, 0, updateCdf: true); - tileReader.ReadInterFrameModeInfo(ref decoder, ref partitionInfo); + tileReader.ReadInterFrameModeInfo(ref decoder, ref partitionInfo, new Av1TileInfo(0, 0, tileReader.FrameHeader)); } /// diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterIntraEntropyTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterIntraEntropyTests.cs new file mode 100644 index 000000000..e31f38492 --- /dev/null +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterIntraEntropyTests.cs @@ -0,0 +1,170 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Buffers; +using SixLabors.ImageSharp.Formats.Heif.Av1; +using SixLabors.ImageSharp.Formats.Heif.Av1.Entropy; +using SixLabors.ImageSharp.Memory; + +namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; + +/// +/// Verifies the entropy state and block-size groups used by the AV1 inter-intra prediction flag. +/// +[Trait("Format", "Avif")] +public class Av1InterIntraEntropyTests +{ + /// + /// Verifies the four block-size-group distributions against libaom's forward Q15 defaults. + /// + [Fact] + public void DefaultsMatchLibaom() + { + ReadOnlySpan forwardThresholds = [16384, 26887, 27597, 30237]; + Av1Distribution[] distributions = Av1DefaultDistributions.InterIntra; + + Assert.Equal(forwardThresholds.Length, distributions.Length); + for (int group = 0; group < distributions.Length; group++) + { + // Av1Distribution stores inverse cumulative thresholds, so convert libaom's forward threshold before + // comparing the exact Q15 state consumed by the range decoder. + uint expected = (uint)Av1Distribution.ProbabilityTop - forwardThresholds[group]; + + Assert.Equal(expected, distributions[group][0]); + Assert.Equal(2, distributions[group].NumberOfSymbols); + } + } + + /// + /// Verifies every AV1 block size against the normative size-group conversion table. + /// + /// The AV1 block-size enumeration value. + /// The normative zero-based size group. + [Theory] + [MemberData(nameof(GetBlockSizeGroups))] + public void GetSizeGroupMatchesNormativeTable(int blockSizeValue, int expectedGroup) + { + Av1BlockSize blockSize = (Av1BlockSize)blockSizeValue; + + Assert.Equal(expectedGroup, blockSize.GetSizeGroup()); + } + + /// + /// Verifies that the inter-intra flag reader selects and adapts the distribution for each size group. + /// + /// A block-size enumeration value representing one size group. + /// The expected zero-based size group. + [Theory] + [InlineData((int)Av1BlockSize.Block4x4, 0)] + [InlineData((int)Av1BlockSize.Block8x8, 1)] + [InlineData((int)Av1BlockSize.Block16x16, 2)] + [InlineData((int)Av1BlockSize.Block32x32, 3)] + public void ReaderUsesBlockSizeGroup(int blockSizeValue, int sizeGroup) + { + bool[] expected = [false, true, true, false, true, false, false, true]; + Av1Distribution writerDistribution = Av1DefaultDistributions.InterIntra[sizeGroup]; + using Av1SymbolWriter writer = new(Configuration.Default, expected.Length, updateCdf: true); + + foreach (bool value in expected) + { + writer.WriteSymbol(value, writerDistribution); + } + + using IMemoryOwner encoded = writer.Exit(); + Av1SymbolDecoder decoder = new(Configuration.Default, encoded.GetSpan(), 0, updateCdf: true); + Av1BlockSize blockSize = (Av1BlockSize)blockSizeValue; + + foreach (bool value in expected) + { + Assert.Equal(value, decoder.ReadIsInterIntra(blockSize)); + } + } + + /// + /// Verifies that frame-context copies retain adapted inter-intra state without sharing mutable distributions. + /// + [Fact] + public void FrameEntropyCopyRetainsIndependentState() + { + Av1FrameEntropyContext source = new(0); + Av1FrameEntropyContext destination = new(0); + source.InterIntra[2].Update(1); + + destination.CopyFrom(source); + + Assert.NotSame(source.InterIntra[2], destination.InterIntra[2]); + Assert.Equal(source.InterIntra[2][0], destination.InterIntra[2][0]); + + source.InterIntra[2].Update(0); + + Assert.NotEqual(source.InterIntra[2][0], destination.InterIntra[2][0]); + } + + /// + /// Verifies that resetting a frame context restores the default threshold and adaptation state. + /// + [Fact] + public void FrameEntropyResetRestoresDefaultState() + { + Av1FrameEntropyContext context = new(0); + Av1FrameEntropyContext expected = new(0); + context.InterIntra[3].Update(1); + + context.ResetToDefaults(0); + + Assert.Equal(expected.InterIntra[3][0], context.InterIntra[3][0]); + + // Applying the same next observation proves that reset restored the update-rate history as well as the visible + // threshold; otherwise two equal thresholds would diverge because their adaptation rates differ. + context.InterIntra[3].Update(0); + expected.InterIntra[3].Update(0); + + Assert.Equal(expected.InterIntra[3][0], context.InterIntra[3][0]); + } + + /// + /// Verifies that a published frame snapshot preserves adapted thresholds and resets their update-rate history. + /// + [Fact] + public void FrameEntropySnapshotResetsUpdateCount() + { + const int updateCount = 20; + Av1FrameEntropyContext source = new(0); + Av1FrameEntropyContext snapshot = new(0); + + for (int i = 0; i < updateCount; i++) + { + source.InterIntra[1].Update(1); + } + + source.SnapshotTo(snapshot); + + Assert.Equal(source.InterIntra[1][0], snapshot.InterIntra[1][0]); + + // The source retains twenty observations while the published snapshot restarts at zero. Their next identical + // observation must therefore move the shared starting threshold by different update rates. + source.InterIntra[1].Update(0); + snapshot.InterIntra[1].Update(0); + + Assert.NotEqual(source.InterIntra[1][0], snapshot.InterIntra[1][0]); + } + + /// + /// Provides the normative AV1 size-group table in block-size enumeration order. + /// + /// Every decoded block size paired with its size group. + public static TheoryData GetBlockSizeGroups() + { + // These are the explicit Size_Group values from AV1 section 9.3 and libaom common_data.h. The test keeps the + // expected table independent from the production geometry formula so a shared calculation cannot mask errors. + int[] sizeGroups = [0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 0, 0, 1, 1, 2, 2]; + TheoryData result = []; + + for (int blockSize = 0; blockSize < sizeGroups.Length; blockSize++) + { + result.Add(blockSize, sizeGroups[blockSize]); + } + + return result; + } +} diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterModeEntropyTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterModeEntropyTests.cs new file mode 100644 index 000000000..316c73785 --- /dev/null +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterModeEntropyTests.cs @@ -0,0 +1,223 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Buffers; +using SixLabors.ImageSharp.Formats.Heif.Av1.Entropy; +using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; + +namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; + +/// +/// Verifies the adaptive distributions and packed contexts used to select an AV1 single-reference inter mode. +/// +[Trait("Format", "Avif")] +public class Av1InterModeEntropyTests +{ + /// + /// Verifies the normative single-reference inter-mode distributions against libaom's forward Q15 defaults. + /// + [Fact] + public void InterModeDefaultsMatchLibaom() + { + AssertBinaryDefaults(Av1DefaultDistributions.NewMv, [24035, 16630, 15339, 8386, 12222, 4676]); + AssertBinaryDefaults(Av1DefaultDistributions.ZeroMv, [2175, 1054]); + AssertBinaryDefaults(Av1DefaultDistributions.RefMv, [23974, 24188, 17848, 28622, 24312, 19923]); + AssertBinaryDefaults(Av1DefaultDistributions.Drl, [13104, 24560, 18945]); + } + + /// + /// Verifies that each inter-mode context occupies the normative field in the packed mode context. + /// + /// The packed mode context. + /// The expected newly decoded motion-vector context. + /// The expected global-motion context. + /// The expected spatial reference-motion-vector context. + [Theory] + [InlineData(0, 0, 0, 0)] + [InlineData(77, 5, 1, 4)] + [InlineData(93, 5, 1, 5)] + public void PackedInterModeContextMatchesLibaom(int modeContext, int expectedNewMv, int expectedZeroMv, int expectedRefMv) + { + Assert.Equal(expectedNewMv, Av1SymbolContextHelper.GetNewMvContext(modeContext)); + Assert.Equal(expectedZeroMv, Av1SymbolContextHelper.GetZeroMvContext(modeContext)); + Assert.Equal(expectedRefMv, Av1SymbolContextHelper.GetRefMvContext(modeContext)); + } + + /// + /// Verifies the exact short-circuit order and symbol polarity of the single-reference inter-mode tree. + /// + /// The expected prediction mode. + /// The new-motion-vector decision. + /// The global-motion decision, or negative when the leaf precedes it. + /// The spatial reference-motion-vector decision, or negative when the leaf precedes it. + [Theory] + [InlineData((int)Av1PredictionMode.NewMotionVector, 0, -1, -1)] + [InlineData((int)Av1PredictionMode.GlobalMotionVector, 1, 0, -1)] + [InlineData((int)Av1PredictionMode.NearestMotionVector, 1, 1, 0)] + [InlineData((int)Av1PredictionMode.NearMotionVector, 1, 1, 1)] + public void ReadInterModeMatchesLibaomTree(int expectedMode, int newMvSymbol, int zeroMvSymbol, int refMvSymbol) + { + const int modeContext = 77; + Av1Distribution newMv = Av1DefaultDistributions.NewMv[5]; + Av1Distribution zeroMv = Av1DefaultDistributions.ZeroMv[1]; + Av1Distribution refMv = Av1DefaultDistributions.RefMv[4]; + Av1Distribution drl = Av1DefaultDistributions.Drl[2]; + using Av1SymbolWriter writer = new(Configuration.Default, 8, updateCdf: true); + + writer.WriteSymbol(newMvSymbol, newMv); + if (zeroMvSymbol >= 0) + { + writer.WriteSymbol(zeroMvSymbol, zeroMv); + } + + if (refMvSymbol >= 0) + { + writer.WriteSymbol(refMvSymbol, refMv); + } + + // A symbol after the selected leaf proves that the decoder consumed exactly the decisions on that branch. + writer.WriteSymbol(true, drl); + + using IMemoryOwner encoded = writer.Exit(); + Av1SymbolDecoder decoder = new(Configuration.Default, encoded.Memory.Span, 0, updateCdf: true); + + Assert.Equal((Av1PredictionMode)expectedMode, decoder.ReadInterMode(modeContext)); + Assert.True(decoder.ReadDrl(2)); + } + + /// + /// Verifies that the dynamic reference-list reader selects each requested context distribution. + /// + /// The dynamic reference-list context. + [Theory] + [InlineData(0)] + [InlineData(1)] + [InlineData(2)] + public void ReadDrlUsesRequestedContext(int context) + { + bool[] expected = [false, true, true, false, true, false, false, true]; + Av1Distribution writerDistribution = Av1DefaultDistributions.Drl[context]; + using Av1SymbolWriter writer = new(Configuration.Default, 8, updateCdf: true); + + foreach (bool value in expected) + { + writer.WriteSymbol(value, writerDistribution); + } + + using IMemoryOwner encoded = writer.Exit(); + Av1SymbolDecoder decoder = new(Configuration.Default, encoded.Memory.Span, 0, updateCdf: true); + + foreach (bool value in expected) + { + Assert.Equal(value, decoder.ReadDrl(context)); + } + } + + /// + /// Verifies the four candidate-weight pairings used to select a dynamic reference-list context. + /// + /// The current candidate's weight. + /// The next candidate's weight. + /// The expected dynamic reference-list context. + [Theory] + [InlineData(640, 640, 0)] + [InlineData(640, 639, 1)] + [InlineData(639, 639, 2)] + [InlineData(639, 640, 0)] + public void DrlContextMatchesCandidateWeightCategories(ushort currentWeight, ushort nextWeight, int expected) + { + ushort[] referenceWeights = [currentWeight, nextWeight]; + + Assert.Equal(expected, Av1SymbolContextHelper.GetDrlContext(referenceWeights, 0)); + } + + /// + /// Verifies that frame-context copies retain inter-mode adaptation without sharing mutable distributions. + /// + [Fact] + public void FrameEntropyCopyRetainsIndependentInterModeState() + { + Av1FrameEntropyContext source = new(0); + Av1FrameEntropyContext destination = new(0); + source.NewMv[5].Update(1); + source.ZeroMv[1].Update(1); + source.RefMv[4].Update(1); + source.Drl[2].Update(1); + + destination.CopyFrom(source); + + Assert.Equal(source.NewMv[5][0], destination.NewMv[5][0]); + Assert.Equal(source.ZeroMv[1][0], destination.ZeroMv[1][0]); + Assert.Equal(source.RefMv[4][0], destination.RefMv[4][0]); + Assert.Equal(source.Drl[2][0], destination.Drl[2][0]); + + source.NewMv[5].Update(0); + source.ZeroMv[1].Update(0); + source.RefMv[4].Update(0); + source.Drl[2].Update(0); + + Assert.NotEqual(source.NewMv[5][0], destination.NewMv[5][0]); + Assert.NotEqual(source.ZeroMv[1][0], destination.ZeroMv[1][0]); + Assert.NotEqual(source.RefMv[4][0], destination.RefMv[4][0]); + Assert.NotEqual(source.Drl[2][0], destination.Drl[2][0]); + } + + /// + /// Verifies that publishing frame state resets the inter-mode distributions' update-rate history. + /// + [Fact] + public void FrameEntropySnapshotResetsInterModeUpdateCounts() + { + const int updateCount = 20; + Av1FrameEntropyContext source = new(0); + Av1FrameEntropyContext snapshot = new(0); + + for (int i = 0; i < updateCount; i++) + { + source.NewMv[5].Update(1); + source.ZeroMv[1].Update(1); + source.RefMv[4].Update(1); + source.Drl[2].Update(1); + } + + source.SnapshotTo(snapshot); + + Assert.Equal(source.NewMv[5][0], snapshot.NewMv[5][0]); + Assert.Equal(source.ZeroMv[1][0], snapshot.ZeroMv[1][0]); + Assert.Equal(source.RefMv[4][0], snapshot.RefMv[4][0]); + Assert.Equal(source.Drl[2][0], snapshot.Drl[2][0]); + + // The source retains twenty observations while the snapshot restarts at zero. Applying the same next symbol + // therefore moves identical thresholds by different amounts only when the new distributions participate in reset. + source.NewMv[5].Update(0); + snapshot.NewMv[5].Update(0); + source.ZeroMv[1].Update(0); + snapshot.ZeroMv[1].Update(0); + source.RefMv[4].Update(0); + snapshot.RefMv[4].Update(0); + source.Drl[2].Update(0); + snapshot.Drl[2].Update(0); + + Assert.NotEqual(source.NewMv[5][0], snapshot.NewMv[5][0]); + Assert.NotEqual(source.ZeroMv[1][0], snapshot.ZeroMv[1][0]); + Assert.NotEqual(source.RefMv[4][0], snapshot.RefMv[4][0]); + Assert.NotEqual(source.Drl[2][0], snapshot.Drl[2][0]); + } + + /// + /// Verifies binary distribution defaults after their conversion to the inverse cumulative representation. + /// + /// The distributions under test. + /// The normative forward Q15 thresholds. + private static void AssertBinaryDefaults(Av1Distribution[] distributions, ReadOnlySpan forwardThresholds) + { + Assert.Equal(forwardThresholds.Length, distributions.Length); + for (int context = 0; context < distributions.Length; context++) + { + uint expected = (uint)Av1Distribution.ProbabilityTop - forwardThresholds[context]; + + Assert.Equal(expected, distributions[context][0]); + Assert.Equal(2, distributions[context].NumberOfSymbols); + } + } +} diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterpolationFilterEntropyTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterpolationFilterEntropyTests.cs new file mode 100644 index 000000000..129f4fe44 --- /dev/null +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterpolationFilterEntropyTests.cs @@ -0,0 +1,314 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Buffers; +using SixLabors.ImageSharp.Formats.Heif.Av1; +using SixLabors.ImageSharp.Formats.Heif.Av1.Entropy; +using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; +using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; + +namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; + +/// +/// Verifies the adaptive distributions and spatial contexts used by AV1 switchable interpolation filters. +/// +[Trait("Format", "Avif")] +public class Av1InterpolationFilterEntropyTests +{ + /// + /// The number of reference, direction, and neighbor-state combinations represented by the distribution table. + /// + private const int SwitchableInterpolationContextCount = 16; + + /// + /// The interpolation-filter direction index for vertical prediction. + /// + private const int VerticalDirection = 0; + + /// + /// The interpolation-filter direction index for horizontal prediction. + /// + private const int HorizontalDirection = 1; + + /// + /// Gets libaom's forward Q15 switchable interpolation-filter thresholds in context order. + /// + private static ReadOnlySpan ForwardThresholds => + [ + 31935, 32720, + 5568, 32719, + 422, 2938, + 28244, 32608, + 31206, 31953, + 4862, 32121, + 770, 1152, + 20889, 25637, + 31910, 32724, + 4120, 32712, + 305, 2247, + 27403, 32636, + 31022, 32009, + 2963, 32093, + 601, 943, + 14969, 21398, + ]; + + /// + /// Verifies every normative switchable interpolation-filter distribution against libaom's forward Q15 defaults. + /// + [Fact] + public void DefaultsMatchLibaom() + { + const int thresholdCount = 2; + ReadOnlySpan forwardThresholds = ForwardThresholds; + Av1Distribution[] distributions = Av1DefaultDistributions.SwitchableInterpolation; + + Assert.Equal(SwitchableInterpolationContextCount, distributions.Length); + for (int context = 0; context < distributions.Length; context++) + { + Assert.Equal(thresholdCount + 1, distributions[context].NumberOfSymbols); + + for (int threshold = 0; threshold < thresholdCount; threshold++) + { + // Av1Distribution stores inverse cumulative thresholds. Complement each published forward value by + // the same Q15 probability top used during production construction before comparing exact state. + uint expected = (uint)Av1Distribution.ProbabilityTop - forwardThresholds[(context * thresholdCount) + threshold]; + + Assert.Equal(expected, distributions[context][threshold]); + } + } + } + + /// + /// Verifies that the symbol reader selects and adapts each of the sixteen switchable interpolation contexts. + /// + /// The reference, direction, and neighbor filter context. + [Theory] + [MemberData(nameof(GetContexts))] + public void ReaderUsesRequestedContext(int context) + { + Av1InterpolationFilter[] expected = + [ + Av1InterpolationFilter.Regular, + Av1InterpolationFilter.Sharp, + Av1InterpolationFilter.Smooth, + Av1InterpolationFilter.Regular, + Av1InterpolationFilter.Smooth, + Av1InterpolationFilter.Sharp, + ]; + + Av1Distribution writerDistribution = Av1DefaultDistributions.SwitchableInterpolation[context]; + using Av1SymbolWriter writer = new(Configuration.Default, expected.Length, updateCdf: true); + + foreach (Av1InterpolationFilter filter in expected) + { + writer.WriteSymbol((int)filter, writerDistribution); + } + + using IMemoryOwner encoded = writer.Exit(); + Av1SymbolDecoder decoder = new(Configuration.Default, encoded.Memory.Span, 0, updateCdf: true); + + foreach (Av1InterpolationFilter filter in expected) + { + Assert.Equal(filter, decoder.ReadSwitchableInterpolationFilter(context)); + } + } + + /// + /// Verifies all sixteen combinations of reference type, direction, and contributing neighbor filter state. + /// + [Fact] + public void ContextLayoutMatchesLibaom() + { + Av1BlockModeInfo single = CreateModeInfo( + Av1ReferenceFrameType.Last, + Av1ReferenceFrameType.None, + Av1InterpolationFilter.Regular, + Av1InterpolationFilter.Regular); + + Av1BlockModeInfo compound = CreateModeInfo( + Av1ReferenceFrameType.Last, + Av1ReferenceFrameType.Backward, + Av1InterpolationFilter.Regular, + Av1InterpolationFilter.Regular); + + Av1BlockModeInfo regular = CreateModeInfo( + Av1ReferenceFrameType.Last, + Av1ReferenceFrameType.None, + Av1InterpolationFilter.Regular, + Av1InterpolationFilter.Regular); + + Av1BlockModeInfo smooth = CreateModeInfo( + Av1ReferenceFrameType.Last, + Av1ReferenceFrameType.None, + Av1InterpolationFilter.Smooth, + Av1InterpolationFilter.Smooth); + + Av1BlockModeInfo sharp = CreateModeInfo( + Av1ReferenceFrameType.Last, + Av1ReferenceFrameType.None, + Av1InterpolationFilter.Sharp, + Av1InterpolationFilter.Sharp); + + // Contexts zero through three are single-reference vertical contexts. Compound prediction adds four, while + // horizontal prediction adds eight. The mixed Regular/Smooth pair selects the fourth neighbor state. + Assert.Equal(0, Av1SymbolContextHelper.GetSwitchableInterpolationContext(single, regular, null, VerticalDirection)); + Assert.Equal(1, Av1SymbolContextHelper.GetSwitchableInterpolationContext(single, smooth, null, VerticalDirection)); + Assert.Equal(2, Av1SymbolContextHelper.GetSwitchableInterpolationContext(single, sharp, null, VerticalDirection)); + Assert.Equal(3, Av1SymbolContextHelper.GetSwitchableInterpolationContext(single, regular, smooth, VerticalDirection)); + Assert.Equal(4, Av1SymbolContextHelper.GetSwitchableInterpolationContext(compound, regular, null, VerticalDirection)); + Assert.Equal(5, Av1SymbolContextHelper.GetSwitchableInterpolationContext(compound, smooth, null, VerticalDirection)); + Assert.Equal(6, Av1SymbolContextHelper.GetSwitchableInterpolationContext(compound, sharp, null, VerticalDirection)); + Assert.Equal(7, Av1SymbolContextHelper.GetSwitchableInterpolationContext(compound, regular, smooth, VerticalDirection)); + Assert.Equal(8, Av1SymbolContextHelper.GetSwitchableInterpolationContext(single, regular, null, HorizontalDirection)); + Assert.Equal(9, Av1SymbolContextHelper.GetSwitchableInterpolationContext(single, smooth, null, HorizontalDirection)); + Assert.Equal(10, Av1SymbolContextHelper.GetSwitchableInterpolationContext(single, sharp, null, HorizontalDirection)); + Assert.Equal(11, Av1SymbolContextHelper.GetSwitchableInterpolationContext(single, regular, smooth, HorizontalDirection)); + Assert.Equal(12, Av1SymbolContextHelper.GetSwitchableInterpolationContext(compound, regular, null, HorizontalDirection)); + Assert.Equal(13, Av1SymbolContextHelper.GetSwitchableInterpolationContext(compound, smooth, null, HorizontalDirection)); + Assert.Equal(14, Av1SymbolContextHelper.GetSwitchableInterpolationContext(compound, sharp, null, HorizontalDirection)); + Assert.Equal(15, Av1SymbolContextHelper.GetSwitchableInterpolationContext(compound, regular, smooth, HorizontalDirection)); + } + + /// + /// Verifies that only neighbors sharing the current primary reference contribute their directional filter. + /// + [Fact] + public void ContextUsesMatchingPrimaryOrSecondaryNeighborReference() + { + Av1BlockModeInfo current = CreateModeInfo( + Av1ReferenceFrameType.Last, + Av1ReferenceFrameType.None, + Av1InterpolationFilter.Regular, + Av1InterpolationFilter.Regular); + + Av1BlockModeInfo secondaryMatch = CreateModeInfo( + Av1ReferenceFrameType.Golden, + Av1ReferenceFrameType.Last, + Av1InterpolationFilter.Smooth, + Av1InterpolationFilter.Sharp); + + Av1BlockModeInfo mismatch = CreateModeInfo( + Av1ReferenceFrameType.Golden, + Av1ReferenceFrameType.None, + Av1InterpolationFilter.Regular, + Av1InterpolationFilter.Regular); + + Assert.Equal(1, Av1SymbolContextHelper.GetSwitchableInterpolationContext(current, secondaryMatch, mismatch, VerticalDirection)); + Assert.Equal(10, Av1SymbolContextHelper.GetSwitchableInterpolationContext(current, secondaryMatch, mismatch, HorizontalDirection)); + Assert.Equal(3, Av1SymbolContextHelper.GetSwitchableInterpolationContext(current, mismatch, null, VerticalDirection)); + Assert.Equal(11, Av1SymbolContextHelper.GetSwitchableInterpolationContext(current, null, null, HorizontalDirection)); + } + + /// + /// Verifies that frame-context copies retain interpolation adaptation without sharing mutable distributions. + /// + [Fact] + public void FrameEntropyCopyRetainsIndependentInterpolationState() + { + Av1FrameEntropyContext source = new(0); + Av1FrameEntropyContext destination = new(0); + source.SwitchableInterpolation[15].Update((int)Av1InterpolationFilter.Sharp); + + destination.CopyFrom(source); + + Assert.Equal(source.SwitchableInterpolation[15][0], destination.SwitchableInterpolation[15][0]); + + source.SwitchableInterpolation[15].Update((int)Av1InterpolationFilter.Regular); + + Assert.NotEqual(source.SwitchableInterpolation[15][0], destination.SwitchableInterpolation[15][0]); + } + + /// + /// Verifies that restoring frame defaults replaces adapted interpolation thresholds and update history. + /// + [Fact] + public void FrameEntropyResetRestoresInterpolationDefaults() + { + const int updateCount = 20; + Av1FrameEntropyContext context = new(0); + + for (int i = 0; i < updateCount; i++) + { + context.SwitchableInterpolation[5].Update((int)Av1InterpolationFilter.Sharp); + } + + context.ResetToDefaults(0); + + Av1Distribution expected = Av1DefaultDistributions.SwitchableInterpolation[5]; + + Assert.Equal(expected[0], context.SwitchableInterpolation[5][0]); + Assert.Equal(expected[1], context.SwitchableInterpolation[5][1]); + + expected.Update((int)Av1InterpolationFilter.Smooth); + context.SwitchableInterpolation[5].Update((int)Av1InterpolationFilter.Smooth); + + Assert.Equal(expected[0], context.SwitchableInterpolation[5][0]); + Assert.Equal(expected[1], context.SwitchableInterpolation[5][1]); + } + + /// + /// Verifies that publishing frame state resets interpolation update history while retaining adapted thresholds. + /// + [Fact] + public void FrameEntropySnapshotResetsInterpolationUpdateCounts() + { + const int updateCount = 20; + Av1FrameEntropyContext source = new(0); + Av1FrameEntropyContext snapshot = new(0); + + for (int i = 0; i < updateCount; i++) + { + source.SwitchableInterpolation[7].Update((int)Av1InterpolationFilter.Smooth); + } + + source.SnapshotTo(snapshot); + + Assert.Equal(source.SwitchableInterpolation[7][0], snapshot.SwitchableInterpolation[7][0]); + + // The source retains its observations while the snapshot restarts at zero. Applying the same next symbol moves + // identical thresholds by different amounts only when the new distribution participates in snapshot reset. + source.SwitchableInterpolation[7].Update((int)Av1InterpolationFilter.Regular); + snapshot.SwitchableInterpolation[7].Update((int)Av1InterpolationFilter.Regular); + + Assert.NotEqual(source.SwitchableInterpolation[7][0], snapshot.SwitchableInterpolation[7][0]); + } + + /// + /// Provides every switchable interpolation-filter context. + /// + /// The sixteen zero-based context indices. + public static TheoryData GetContexts() + { + TheoryData result = []; + + for (int context = 0; context < SwitchableInterpolationContextCount; context++) + { + result.Add(context); + } + + return result; + } + + /// + /// Creates decoded block state with the requested references and directional interpolation filters. + /// + /// The primary reference label. + /// The optional secondary reference label. + /// The vertical interpolation filter. + /// The horizontal interpolation filter. + /// The initialized block mode state. + private static Av1BlockModeInfo CreateModeInfo( + Av1ReferenceFrameType primaryReference, + Av1ReferenceFrameType secondaryReference, + Av1InterpolationFilter verticalFilter, + Av1InterpolationFilter horizontalFilter) + { + Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block8x8, Point.Empty); + modeInfo.ReferenceFrames[0] = primaryReference; + modeInfo.ReferenceFrames[1] = secondaryReference; + modeInfo.InterpolationFilters[0] = verticalFilter; + modeInfo.InterpolationFilters[1] = horizontalFilter; + return modeInfo; + } +} diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1MotionModeEntropyTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1MotionModeEntropyTests.cs new file mode 100644 index 000000000..58a447d2f --- /dev/null +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1MotionModeEntropyTests.cs @@ -0,0 +1,223 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Buffers; +using SixLabors.ImageSharp.Formats.Heif.Av1; +using SixLabors.ImageSharp.Formats.Heif.Av1.Entropy; +using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; +using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; + +namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; + +/// +/// Verifies the entropy defaults, lifecycle, and range-decoder alignment used by AV1 motion-mode syntax. +/// +[Trait("Format", "Avif")] +public class Av1MotionModeEntropyTests +{ + /// + /// Gets libaom's forward Q15 Simple Translation, OBMC, and Warped thresholds in block-size order. + /// + private static ReadOnlySpan MotionModeForwardThresholds => + [ + 10923, 21845, + 10923, 21845, + 10923, 21845, + 7651, 24760, + 4738, 24765, + 5391, 25528, + 19419, 26810, + 5123, 23606, + 11606, 24308, + 26260, 29116, + 20360, 28062, + 21679, 26830, + 29516, 30701, + 28898, 30397, + 30878, 31335, + 32507, 32558, + 10923, 21845, + 10923, 21845, + 28799, 31390, + 26431, 30774, + 28973, 31594, + 29742, 31203, + ]; + + /// + /// Gets libaom's forward Q15 Simple Translation and OBMC thresholds in block-size order. + /// + private static ReadOnlySpan ObmcForwardThresholds => + [ + 16384, 16384, 16384, 10437, 9371, 9301, 17432, 14423, 15142, 25817, 22823, + 22083, 30128, 31014, 31560, 32638, 16384, 16384, 23664, 20901, 24008, 26879, + ]; + + /// + /// Verifies all twenty-two ternary and binary motion-mode distributions against libaom's forward Q15 defaults. + /// + [Fact] + public void DefaultsMatchLibaomForEveryBlockSize() + { + const int blockSizeCount = (int)Av1BlockSize.AllSizes; + const int ternaryThresholdCount = 2; + ReadOnlySpan motionModeForwardThresholds = MotionModeForwardThresholds; + ReadOnlySpan obmcForwardThresholds = ObmcForwardThresholds; + Av1Distribution[] motionMode = Av1DefaultDistributions.MotionMode; + Av1Distribution[] obmc = Av1DefaultDistributions.Obmc; + + Assert.Equal(blockSizeCount * ternaryThresholdCount, motionModeForwardThresholds.Length); + Assert.Equal(blockSizeCount, obmcForwardThresholds.Length); + Assert.Equal(blockSizeCount, motionMode.Length); + Assert.Equal(blockSizeCount, obmc.Length); + + for (int blockSize = 0; blockSize < blockSizeCount; blockSize++) + { + Assert.Equal(3, motionMode[blockSize].NumberOfSymbols); + Assert.Equal(2, obmc[blockSize].NumberOfSymbols); + + for (int threshold = 0; threshold < ternaryThresholdCount; threshold++) + { + // Av1Distribution stores inverse cumulative thresholds, so complement libaom's published forward + // Q15 values before comparing the exact state consumed by the range decoder. + uint expected = (uint)Av1Distribution.ProbabilityTop - + motionModeForwardThresholds[(blockSize * ternaryThresholdCount) + threshold]; + + Assert.Equal(expected, motionMode[blockSize][threshold]); + } + + uint expectedObmc = (uint)Av1Distribution.ProbabilityTop - obmcForwardThresholds[blockSize]; + + Assert.Equal(expectedObmc, obmc[blockSize][0]); + } + } + + /// + /// Verifies that newly created frame contexts and explicit copies own independent motion-mode distributions. + /// + [Fact] + public void FrameEntropyContextsDeepCopyAndCopyFromMotionModeState() + { + int blockSize = (int)Av1BlockSize.Block16x16; + Av1FrameEntropyContext source = new(0); + Av1FrameEntropyContext destination = new(0); + + Assert.NotSame(source.MotionMode[blockSize], destination.MotionMode[blockSize]); + Assert.NotSame(source.Obmc[blockSize], destination.Obmc[blockSize]); + + source.MotionMode[blockSize].Update((int)Av1MotionMode.Warped); + source.Obmc[blockSize].Update((int)Av1MotionMode.Obmc); + + Assert.NotEqual(source.MotionMode[blockSize][0], destination.MotionMode[blockSize][0]); + Assert.NotEqual(source.Obmc[blockSize][0], destination.Obmc[blockSize][0]); + + destination.CopyFrom(source); + + Assert.Equal(source.MotionMode[blockSize][0], destination.MotionMode[blockSize][0]); + Assert.Equal(source.MotionMode[blockSize][1], destination.MotionMode[blockSize][1]); + Assert.Equal(source.Obmc[blockSize][0], destination.Obmc[blockSize][0]); + + source.MotionMode[blockSize].Update((int)Av1MotionMode.SimpleTranslation); + source.Obmc[blockSize].Update((int)Av1MotionMode.SimpleTranslation); + + Assert.NotEqual(source.MotionMode[blockSize][0], destination.MotionMode[blockSize][0]); + Assert.NotEqual(source.Obmc[blockSize][0], destination.Obmc[blockSize][0]); + } + + /// + /// Verifies that resetting a frame context restores both motion-mode thresholds and adaptation history. + /// + [Fact] + public void FrameEntropyResetRestoresMotionModeState() + { + const int updateCount = 20; + int blockSize = (int)Av1BlockSize.Block32x16; + Av1FrameEntropyContext context = new(0); + Av1FrameEntropyContext expected = new(0); + + for (int update = 0; update < updateCount; update++) + { + context.MotionMode[blockSize].Update((int)Av1MotionMode.Warped); + context.Obmc[blockSize].Update((int)Av1MotionMode.Obmc); + } + + context.ResetToDefaults(0); + + Assert.Equal(expected.MotionMode[blockSize][0], context.MotionMode[blockSize][0]); + Assert.Equal(expected.MotionMode[blockSize][1], context.MotionMode[blockSize][1]); + Assert.Equal(expected.Obmc[blockSize][0], context.Obmc[blockSize][0]); + + // Equal thresholds can still carry different observation counts. Applying the same next symbols proves that + // ResetToDefaults restored the update-rate history as well as the visible probability thresholds. + context.MotionMode[blockSize].Update((int)Av1MotionMode.Obmc); + expected.MotionMode[blockSize].Update((int)Av1MotionMode.Obmc); + context.Obmc[blockSize].Update((int)Av1MotionMode.SimpleTranslation); + expected.Obmc[blockSize].Update((int)Av1MotionMode.SimpleTranslation); + + Assert.Equal(expected.MotionMode[blockSize][0], context.MotionMode[blockSize][0]); + Assert.Equal(expected.MotionMode[blockSize][1], context.MotionMode[blockSize][1]); + Assert.Equal(expected.Obmc[blockSize][0], context.Obmc[blockSize][0]); + } + + /// + /// Verifies that a published frame snapshot retains adapted motion-mode thresholds but resets update counts. + /// + [Fact] + public void FrameEntropySnapshotResetsMotionModeUpdateCounts() + { + const int updateCount = 20; + int blockSize = (int)Av1BlockSize.Block16x32; + Av1FrameEntropyContext source = new(0); + Av1FrameEntropyContext snapshot = new(0); + + for (int update = 0; update < updateCount; update++) + { + source.MotionMode[blockSize].Update((int)Av1MotionMode.Warped); + source.Obmc[blockSize].Update((int)Av1MotionMode.Obmc); + } + + source.SnapshotTo(snapshot); + + Assert.Equal(source.MotionMode[blockSize][0], snapshot.MotionMode[blockSize][0]); + Assert.Equal(source.MotionMode[blockSize][1], snapshot.MotionMode[blockSize][1]); + Assert.Equal(source.Obmc[blockSize][0], snapshot.Obmc[blockSize][0]); + + // The source retains twenty observations while the snapshot restarts at zero. Identical next observations + // therefore move their equal starting thresholds by different update rates. + source.MotionMode[blockSize].Update((int)Av1MotionMode.SimpleTranslation); + snapshot.MotionMode[blockSize].Update((int)Av1MotionMode.SimpleTranslation); + source.Obmc[blockSize].Update((int)Av1MotionMode.SimpleTranslation); + snapshot.Obmc[blockSize].Update((int)Av1MotionMode.SimpleTranslation); + + Assert.NotEqual(source.MotionMode[blockSize][0], snapshot.MotionMode[blockSize][0]); + Assert.NotEqual(source.Obmc[blockSize][0], snapshot.Obmc[blockSize][0]); + } + + /// + /// Verifies that both motion-mode alphabets leave the range decoder aligned for the immediately following filter symbol. + /// + /// Whether the motion-mode symbol uses the ternary rather than binary distribution. + [Theory] + [InlineData(false)] + [InlineData(true)] + public void ReadMotionModePreservesFollowingInterpolationSymbolAlignment(bool allowWarpedMotion) + { + Av1BlockSize blockSize = Av1BlockSize.Block16x16; + const int interpolationContext = 3; + Av1Distribution motionModeDistribution = allowWarpedMotion + ? Av1DefaultDistributions.MotionMode[(int)blockSize] + : Av1DefaultDistributions.Obmc[(int)blockSize]; + + using Av1SymbolWriter writer = new(Configuration.Default, 2, updateCdf: true); + writer.WriteSymbol((int)Av1MotionMode.SimpleTranslation, motionModeDistribution); + writer.WriteSymbol( + (int)Av1InterpolationFilter.Sharp, + Av1DefaultDistributions.SwitchableInterpolation[interpolationContext]); + + using IMemoryOwner encoded = writer.Exit(); + Av1SymbolDecoder decoder = new(Configuration.Default, encoded.Memory.Span, 0, updateCdf: true); + + Assert.Equal(Av1MotionMode.SimpleTranslation, decoder.ReadMotionMode(blockSize, allowWarpedMotion)); + Assert.Equal(Av1InterpolationFilter.Sharp, decoder.ReadSwitchableInterpolationFilter(interpolationContext)); + } +} diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1MotionModeInfoTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1MotionModeInfoTests.cs new file mode 100644 index 000000000..1f183d31c --- /dev/null +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1MotionModeInfoTests.cs @@ -0,0 +1,206 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Buffers; +using SixLabors.ImageSharp.Formats.Heif.Av1; +using SixLabors.ImageSharp.Formats.Heif.Av1.Entropy; +using SixLabors.ImageSharp.Formats.Heif.Av1.Motion; +using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit; +using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline; +using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; +using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; +using SixLabors.ImageSharp.Formats.Heif.Av1.ReferenceFrames; +using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; + +namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; + +/// +/// Verifies inter-intra, motion-mode, and interpolation-filter syntax ordering in inter-frame mode parsing. +/// +[Trait("Format", "Avif")] +public class Av1MotionModeInfoTests +{ + /// + /// Verifies that an extended 8x32 rectangle omits inter-intra syntax and reads the following interpolation filter. + /// + [Fact] + public void ReadInterFrameModeInfoOmitsInterIntraFlagForExtendedRectangle() + { + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(); + sequenceHeader.EnableInterIntraCompound = true; + ObuFrameHeader frameHeader = CreateFrameHeader(); + ConfigureForcedTranslationalGlobalMotion(frameHeader); + + using Av1TileReader tileReader = new(Configuration.Default, sequenceHeader, frameHeader); + Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block8x32, Point.Empty); + Av1SuperblockInfo superblockInfo = new(tileReader.FrameInfo, Point.Empty); + Av1PartitionInfo partitionInfo = new(modeInfo, superblockInfo, false, Av1PartitionType.None); + + using Av1SymbolWriter writer = new(Configuration.Default, 2, updateCdf: true); + writer.WriteSymbol(false, Av1DefaultDistributions.Skip[0]); + + // With no matching above or left filter, a single-reference vertical filter uses context three. Writing the + // filter immediately after Skip makes any accidental extended-rectangle inter-intra read desynchronize it. + writer.WriteSymbol((int)Av1InterpolationFilter.Sharp, Av1DefaultDistributions.SwitchableInterpolation[3]); + + using IMemoryOwner encoded = writer.Exit(); + Av1SymbolDecoder decoder = new(Configuration.Default, encoded.Memory.Span, 0, updateCdf: true); + + tileReader.ReadInterFrameModeInfo(ref decoder, ref partitionInfo, new Av1TileInfo(0, 0, frameHeader)); + + Assert.Equal(Av1MotionMode.SimpleTranslation, modeInfo.MotionMode); + Assert.Equal(Av1InterpolationFilter.Sharp, modeInfo.InterpolationFilters[0]); + Assert.Equal(Av1InterpolationFilter.Sharp, modeInfo.InterpolationFilters[1]); + } + + /// + /// Verifies that a false inter-intra flag continues through omitted, binary, and ternary Simple Translation syntax into interpolation. + /// + /// Whether the frame enables per-block motion-mode syntax. + /// Whether the eligible block uses the ternary rather than binary motion-mode distribution. + [Theory] + [InlineData(false, false)] + [InlineData(true, false)] + [InlineData(true, true)] + public void ReadInterFrameModeInfoContinuesFromFalseInterIntraThroughMotionModeIntoInterpolation( + bool isMotionModeSwitchable, + bool allowWarpedMotion) + { + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(); + sequenceHeader.EnableInterIntraCompound = true; + ObuFrameHeader frameHeader = CreateFrameHeader(); + frameHeader.IsMotionModeSwitchable = isMotionModeSwitchable; + frameHeader.AllowWarpedMotion = allowWarpedMotion; + ConfigureForcedTranslationalGlobalMotion(frameHeader); + + using Av1ReferenceFrameStore referenceFrames = new(); + using Av1FrameInfo retainedFrameInfo = new(sequenceHeader); + Av1ReferenceFrame retainedFrame = new( + new Av1FrameBuffer(Configuration.Default, sequenceHeader, Av1ColorFormat.Yuv400, false), + CreateFrameHeader(), + retainedFrameInfo); + + Assert.True(referenceFrames.Commit(1, retainedFrame, showFrame: false)); + + Av1FrameEntropyContexts entropyContexts = new(0); + using Av1TileReader tileReader = new( + Configuration.Default, + sequenceHeader, + frameHeader, + entropyContexts, + null, + referenceFrames); + + Av1SuperblockInfo superblockInfo = tileReader.FrameInfo.GetSuperblock(Point.Empty); + Av1BlockModeInfo aboveModeInfo = new(Av1BlockSize.Block8x8, Point.Empty) + { + YMode = Av1PredictionMode.NearestMotionVector, + }; + + aboveModeInfo.ReferenceFrames[0] = Av1ReferenceFrameType.Last; + aboveModeInfo.ReferenceFrames[1] = Av1ReferenceFrameType.None; + aboveModeInfo.InterpolationFilters.Fill(Av1InterpolationFilter.Regular); + tileReader.FrameInfo.UpdateModeInfo(aboveModeInfo, superblockInfo); + superblockInfo.BlockCount++; + + Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block8x8, new Point(0, 2)); + Av1PartitionInfo partitionInfo = new(modeInfo, superblockInfo, false, Av1PartitionType.None) + { + ColumnIndex = 0, + RowIndex = 2, + AvailableAbove = true, + AboveModeInfo = aboveModeInfo, + }; + + using Av1SymbolWriter writer = new(Configuration.Default, 4, updateCdf: true); + writer.WriteSymbol(false, Av1DefaultDistributions.Skip[0]); + writer.WriteSymbol(false, Av1DefaultDistributions.InterIntra[Av1BlockSize.Block8x8.GetSizeGroup()]); + + if (isMotionModeSwitchable) + { + Av1Distribution motionModeDistribution = allowWarpedMotion + ? Av1DefaultDistributions.MotionMode[(int)Av1BlockSize.Block8x8] + : Av1DefaultDistributions.Obmc[(int)Av1BlockSize.Block8x8]; + + writer.WriteSymbol((int)Av1MotionMode.SimpleTranslation, motionModeDistribution); + } + + // The matching regular above neighbor selects vertical context zero. Sharp is deliberately non-default so the + // assertion proves that every preceding conditional symbol consumed exactly its own range-coded interval. + writer.WriteSymbol((int)Av1InterpolationFilter.Sharp, Av1DefaultDistributions.SwitchableInterpolation[0]); + + using IMemoryOwner encoded = writer.Exit(); + Av1SymbolDecoder decoder = new(Configuration.Default, encoded.Memory.Span, 0, updateCdf: true); + + tileReader.ReadInterFrameModeInfo(ref decoder, ref partitionInfo, new Av1TileInfo(0, 0, frameHeader)); + + Assert.Equal(Av1ReferenceFrameType.Last, modeInfo.ReferenceFrames[0]); + Assert.Equal(Av1ReferenceFrameType.None, modeInfo.ReferenceFrames[1]); + Assert.Equal(Av1MotionMode.SimpleTranslation, modeInfo.MotionMode); + Assert.Equal(Av1InterpolationFilter.Sharp, modeInfo.InterpolationFilters[0]); + Assert.Equal(Av1InterpolationFilter.Sharp, modeInfo.InterpolationFilters[1]); + } + + /// + /// Creates the monochrome 64x64 sequence geometry used by direct inter-mode syntax tests. + /// + /// The initialized sequence header. + private static ObuSequenceHeader CreateSequenceHeader() + => new() + { + MaxFrameWidth = 64, + MaxFrameHeight = 64, + Use128x128Superblock = false, + EnableDualFilter = false, + EnableCdef = false, + EnableFilterIntra = false, + ColorConfig = new ObuColorConfig + { + IsMonochrome = true, + BitDepth = Av1BitDepth.EightBit, + }, + }; + + /// + /// Creates an inter-frame header whose one tile and coded dimensions cover the complete test frame. + /// + /// The initialized frame header. + private static ObuFrameHeader CreateFrameHeader() + => new() + { + FrameType = ObuFrameType.InterFrame, + ModeInfoColumnCount = 16, + ModeInfoRowCount = 16, + CodedLossless = true, + AllowScreenContentTools = false, + InterpolationFilter = Av1InterpolationFilter.Switchable, + FrameSize = new ObuFrameSize + { + FrameWidth = 64, + FrameHeight = 64, + SuperResolutionUpscaledWidth = 64, + RenderWidth = 64, + RenderHeight = 64, + }, + TilesInfo = new ObuTileGroupHeader + { + TileColumnCount = 1, + TileRowCount = 1, + TileColumnStartModeInfo = [0, 16], + TileRowStartModeInfo = [0, 16], + }, + }; + + /// + /// Forces segment zero to a translational global-motion mode that omits reference and inter-mode symbols but still carries interpolation. + /// + /// The frame header to configure. + private static void ConfigureForcedTranslationalGlobalMotion(ObuFrameHeader frameHeader) + { + ObuSegmentationParameters segmentationParameters = frameHeader.SegmentationParameters; + segmentationParameters.Enabled = true; + segmentationParameters.FeatureEnabled[0, (int)ObuSegmentationLevelFeature.GlobalMotionVector] = true; + frameHeader.GetGlobalMotionParameters()[0].Type = Av1GlobalMotionType.Translation; + frameHeader.GetReferenceFrameIndices()[0] = 0; + } +} diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1MotionVariationCandidatesTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1MotionVariationCandidatesTests.cs new file mode 100644 index 000000000..6cbe1ce40 --- /dev/null +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1MotionVariationCandidatesTests.cs @@ -0,0 +1,392 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using SixLabors.ImageSharp.Formats.Heif.Av1; +using SixLabors.ImageSharp.Formats.Heif.Av1.Motion; +using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit; +using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; +using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; + +namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; + +/// +/// Verifies the spatial neighbor and projection-sample rules used to select an AV1 motion mode. +/// +[Trait("Format", "Avif")] +public class Av1MotionVariationCandidatesTests +{ + /// + /// Verifies that overlap detection exhausts the complete above edge before falling back to the complete left edge. + /// + [Fact] + public void BuildScansCompleteAboveEdgeThenFallsBackToLeftEdge() + { + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(); + ObuFrameHeader frameHeader = CreateFrameHeader(); + Av1FrameInfo frameInfo = new(sequenceHeader); + + for (int offset = 0; offset < 8; offset += 2) + { + AddModeInfo( + frameInfo, + sequenceHeader, + new Point(4 + offset, 2), + Av1BlockSize.Block8x8, + Av1ReferenceFrameType.Intra, + Av1ReferenceFrameType.None, + default); + + Av1ReferenceFrameType leftReference = offset == 6 + ? Av1ReferenceFrameType.Last + : Av1ReferenceFrameType.Intra; + + AddModeInfo( + frameInfo, + sequenceHeader, + new Point(2, 4 + offset), + Av1BlockSize.Block8x8, + leftReference, + Av1ReferenceFrameType.None, + default); + } + + Av1PartitionInfo partitionInfo = CreatePartitionInfo( + frameInfo, + sequenceHeader, + new Point(4, 4), + Av1BlockSize.Block32x32, + availableAbove: true, + availableLeft: true); + + Av1MotionVariationCandidates candidates = new(); + + candidates.Build( + ref partitionInfo, + new Av1TileInfo(0, 0, frameHeader), + sequenceHeader, + frameHeader, + Av1ReferenceFrameType.Last); + + Assert.True(candidates.HasOverlappableNeighbor); + Assert.Equal(1, candidates.Count); + } + + /// + /// Verifies that 4x4 neighbors use the second mode record of each horizontal or vertical 8-sample pair. + /// + [Fact] + public void BuildUsesSecondCellForFourSampleNeighborPairs() + { + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(); + ObuFrameHeader frameHeader = CreateFrameHeader(); + Av1FrameInfo horizontalFrameInfo = new(sequenceHeader); + AddModeInfo( + horizontalFrameInfo, + sequenceHeader, + new Point(4, 3), + Av1BlockSize.Block4x4, + Av1ReferenceFrameType.Intra, + Av1ReferenceFrameType.None, + default); + + AddModeInfo( + horizontalFrameInfo, + sequenceHeader, + new Point(5, 3), + Av1BlockSize.Block4x4, + Av1ReferenceFrameType.Last, + Av1ReferenceFrameType.None, + default); + + Av1PartitionInfo horizontalPartition = CreatePartitionInfo( + horizontalFrameInfo, + sequenceHeader, + new Point(4, 4), + Av1BlockSize.Block8x8, + availableAbove: true, + availableLeft: false); + + Av1MotionVariationCandidates horizontalCandidates = new(); + horizontalCandidates.Build( + ref horizontalPartition, + new Av1TileInfo(0, 0, frameHeader), + sequenceHeader, + frameHeader, + Av1ReferenceFrameType.Last); + + Av1FrameInfo verticalFrameInfo = new(sequenceHeader); + AddModeInfo( + verticalFrameInfo, + sequenceHeader, + new Point(3, 4), + Av1BlockSize.Block4x4, + Av1ReferenceFrameType.Intra, + Av1ReferenceFrameType.None, + default); + + AddModeInfo( + verticalFrameInfo, + sequenceHeader, + new Point(3, 5), + Av1BlockSize.Block4x4, + Av1ReferenceFrameType.Last, + Av1ReferenceFrameType.None, + default); + + Av1PartitionInfo verticalPartition = CreatePartitionInfo( + verticalFrameInfo, + sequenceHeader, + new Point(4, 4), + Av1BlockSize.Block8x8, + availableAbove: false, + availableLeft: true); + + Av1MotionVariationCandidates verticalCandidates = new(); + verticalCandidates.Build( + ref verticalPartition, + new Av1TileInfo(0, 0, frameHeader), + sequenceHeader, + frameHeader, + Av1ReferenceFrameType.Last); + + Assert.True(horizontalCandidates.HasOverlappableNeighbor); + Assert.True(verticalCandidates.HasOverlappableNeighbor); + } + + /// + /// Verifies that projection samples require a matching single reference and stop at the normative capacity of eight. + /// + [Fact] + public void BuildCollectsMatchingSingleReferenceSamplesUpToCapacity() + { + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(); + ObuFrameHeader frameHeader = CreateFrameHeader(); + Av1FrameInfo frameInfo = new(sequenceHeader); + + // The first candidate has the wrong primary reference and the second is compound. Ten following candidates + // are eligible, so the retained range must begin at offset two and stop after eight samples at offset nine. + for (int offset = 0; offset < 12; offset++) + { + Av1ReferenceFrameType primaryReference = offset == 0 + ? Av1ReferenceFrameType.Golden + : Av1ReferenceFrameType.Last; + + Av1ReferenceFrameType secondaryReference = offset == 1 + ? Av1ReferenceFrameType.Golden + : Av1ReferenceFrameType.None; + + AddModeInfo( + frameInfo, + sequenceHeader, + new Point(8 + offset, 15), + Av1BlockSize.Block4x4, + primaryReference, + secondaryReference, + new Av1MotionVector(offset, offset + 1)); + } + + Av1PartitionInfo partitionInfo = CreatePartitionInfo( + frameInfo, + sequenceHeader, + new Point(8, 16), + Av1BlockSize.Block64x64, + availableAbove: true, + availableLeft: false); + + Av1MotionVariationCandidates candidates = new(); + + candidates.Build( + ref partitionInfo, + new Av1TileInfo(0, 0, frameHeader), + sequenceHeader, + frameHeader, + Av1ReferenceFrameType.Last); + + Assert.Equal(8, candidates.Count); + + // Positions are Q3 neighbor centers relative to the current block. Reference points add the corresponding + // Q3 motion vector without rounding, which also proves that the rejected first two candidates were skipped. + Assert.Equal(new Point(72, -24), candidates.SourcePoints[0]); + Assert.Equal(new Point(75, -22), candidates.ReferencePoints[0]); + Assert.Equal(new Point(296, -24), candidates.SourcePoints[7]); + Assert.Equal(new Point(306, -15), candidates.ReferencePoints[7]); + } + + /// + /// Verifies that eligible top-left and top-right diagonal blocks contribute after the direct edge neighbors. + /// + [Fact] + public void BuildIncludesEligibleTopLeftAndTopRightSamples() + { + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(); + ObuFrameHeader frameHeader = CreateFrameHeader(); + Av1FrameInfo frameInfo = new(sequenceHeader); + AddModeInfo(frameInfo, sequenceHeader, new Point(4, 2), Av1BlockSize.Block8x8, Av1ReferenceFrameType.Last, Av1ReferenceFrameType.None, default); + AddModeInfo(frameInfo, sequenceHeader, new Point(2, 4), Av1BlockSize.Block8x8, Av1ReferenceFrameType.Last, Av1ReferenceFrameType.None, default); + AddModeInfo(frameInfo, sequenceHeader, new Point(2, 2), Av1BlockSize.Block8x8, Av1ReferenceFrameType.Last, Av1ReferenceFrameType.None, default); + AddModeInfo(frameInfo, sequenceHeader, new Point(6, 2), Av1BlockSize.Block8x8, Av1ReferenceFrameType.Last, Av1ReferenceFrameType.None, default); + + Av1PartitionInfo partitionInfo = CreatePartitionInfo( + frameInfo, + sequenceHeader, + new Point(4, 4), + Av1BlockSize.Block8x8, + availableAbove: true, + availableLeft: true); + + Av1MotionVariationCandidates candidates = new(); + + candidates.Build( + ref partitionInfo, + new Av1TileInfo(0, 0, frameHeader), + sequenceHeader, + frameHeader, + Av1ReferenceFrameType.Last); + + Assert.Equal(4, candidates.Count); + Assert.Equal(new Point(24, -40), candidates.SourcePoints[0]); + Assert.Equal(new Point(-40, 24), candidates.SourcePoints[1]); + Assert.Equal(new Point(-40, -40), candidates.SourcePoints[2]); + Assert.Equal(new Point(88, -40), candidates.SourcePoints[3]); + } + + /// + /// Verifies that edge blocks already covering the diagonal positions suppress duplicate top-left and top-right samples. + /// + [Fact] + public void BuildSuppressesDiagonalSamplesCoveredByEdgeNeighbors() + { + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(); + ObuFrameHeader frameHeader = CreateFrameHeader(); + Av1FrameInfo frameInfo = new(sequenceHeader); + + // The aligned 16x8 above block covers the top-right position. The 8x16 left block begins two mode-info rows + // above the current block and therefore covers its top-left position. + AddModeInfo(frameInfo, sequenceHeader, new Point(4, 4), Av1BlockSize.Block16x8, Av1ReferenceFrameType.Last, Av1ReferenceFrameType.None, default); + AddModeInfo(frameInfo, sequenceHeader, new Point(2, 4), Av1BlockSize.Block8x16, Av1ReferenceFrameType.Last, Av1ReferenceFrameType.None, default); + + Av1PartitionInfo partitionInfo = CreatePartitionInfo( + frameInfo, + sequenceHeader, + new Point(4, 6), + Av1BlockSize.Block8x8, + availableAbove: true, + availableLeft: true); + + Av1MotionVariationCandidates candidates = new(); + + candidates.Build( + ref partitionInfo, + new Av1TileInfo(0, 0, frameHeader), + sequenceHeader, + frameHeader, + Av1ReferenceFrameType.Last); + + Assert.Equal(2, candidates.Count); + } + + /// + /// Creates the monochrome 128x128 sequence geometry used by spatial motion-mode tests. + /// + /// The initialized sequence header. + private static ObuSequenceHeader CreateSequenceHeader() + => new() + { + MaxFrameWidth = 128, + MaxFrameHeight = 128, + Use128x128Superblock = false, + ColorConfig = new ObuColorConfig + { + IsMonochrome = true, + BitDepth = Av1BitDepth.EightBit, + }, + }; + + /// + /// Creates a single-tile frame covering the complete sequence geometry. + /// + /// The initialized frame header. + private static ObuFrameHeader CreateFrameHeader() + => new() + { + FrameType = ObuFrameType.InterFrame, + ModeInfoColumnCount = 32, + ModeInfoRowCount = 32, + TilesInfo = new ObuTileGroupHeader + { + TileColumnCount = 1, + TileRowCount = 1, + TileColumnStartModeInfo = [0, 32], + TileRowStartModeInfo = [0, 32], + }, + }; + + /// + /// Creates one current partition at a frame-relative mode-information position. + /// + /// The frame map containing the neighboring mode records. + /// The sequence geometry defining superblock-relative addressing. + /// The frame-relative block origin in 4x4 units. + /// The current block geometry. + /// Whether the above edge is available. + /// Whether the left edge is available. + /// The initialized partition state. + private static Av1PartitionInfo CreatePartitionInfo( + Av1FrameInfo frameInfo, + ObuSequenceHeader sequenceHeader, + Point position, + Av1BlockSize blockSize, + bool availableAbove, + bool availableLeft) + { + int superblockSize = sequenceHeader.SuperblockModeInfoSize; + Point superblockPosition = new(position.X / superblockSize, position.Y / superblockSize); + Point relativePosition = new(position.X % superblockSize, position.Y % superblockSize); + Av1BlockModeInfo modeInfo = new(blockSize, relativePosition); + + return new Av1PartitionInfo(modeInfo, frameInfo.GetSuperblock(superblockPosition), false, Av1PartitionType.None) + { + ColumnIndex = position.X, + RowIndex = position.Y, + AvailableAbove = availableAbove, + AvailableLeft = availableLeft, + }; + } + + /// + /// Creates and maps one decoded neighbor at a frame-relative mode-information position. + /// + /// The frame map that owns the neighbor. + /// The sequence geometry defining superblock-relative addressing. + /// The frame-relative block origin in 4x4 units. + /// The neighboring block geometry. + /// The primary prediction reference. + /// The optional secondary prediction reference. + /// The primary motion vector in one-eighth-sample units. + private static void AddModeInfo( + Av1FrameInfo frameInfo, + ObuSequenceHeader sequenceHeader, + Point position, + Av1BlockSize blockSize, + Av1ReferenceFrameType primaryReference, + Av1ReferenceFrameType secondaryReference, + Av1MotionVector motionVector) + { + int superblockSize = sequenceHeader.SuperblockModeInfoSize; + Point superblockPosition = new(position.X / superblockSize, position.Y / superblockSize); + Point relativePosition = new(position.X % superblockSize, position.Y % superblockSize); + Av1SuperblockInfo superblockInfo = frameInfo.GetSuperblock(superblockPosition); + Av1BlockModeInfo modeInfo = new(blockSize, relativePosition) + { + YMode = primaryReference == Av1ReferenceFrameType.Intra + ? Av1PredictionMode.DC + : Av1PredictionMode.NearestMotionVector, + }; + + modeInfo.ReferenceFrames[0] = primaryReference; + modeInfo.ReferenceFrames[1] = secondaryReference; + modeInfo.MotionVectors[0] = motionVector; + frameInfo.UpdateModeInfo(modeInfo, superblockInfo); + superblockInfo.BlockCount++; + } +} diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1MotionVectorEntropyTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1MotionVectorEntropyTests.cs new file mode 100644 index 000000000..9c2ab1c92 --- /dev/null +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1MotionVectorEntropyTests.cs @@ -0,0 +1,345 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Buffers; +using SixLabors.ImageSharp.Formats.Heif.Av1.Entropy; +using SixLabors.ImageSharp.Formats.Heif.Av1.Motion; + +namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; + +/// +/// Verifies the adaptive AV1 normal and displacement motion-vector entropy contexts. +/// +[Trait("Format", "Avif")] +public class Av1MotionVectorEntropyTests +{ + /// + /// Verifies both motion-vector contexts against every normative forward Q15 default from libaom. + /// + [Fact] + public void MotionVectorDefaultsMatchLibaom() + { + Av1FrameEntropyContext context = new(0); + + Assert.NotSame(context.MotionVector, context.DisplacementVector); + AssertContextDefaults(context.MotionVector); + AssertContextDefaults(context.DisplacementVector); + } + + /// + /// Verifies integer, quarter-sample, and eighth-sample syntax consumption and reconstruction. + /// + /// The numeric motion-vector precision. + /// Indicates whether the coded delta occupies the horizontal component. + /// The coded magnitude class. + /// The coded integer magnitude offset. + /// The coded fractional symbol, or negative when omitted. + /// The coded eighth-sample symbol, or negative when omitted. + /// The expected positive component in one-eighth-sample units. + [Theory] + [InlineData((int)Av1MotionVectorPrecision.Integer, false, 0, 1, -1, -1, 16)] + [InlineData((int)Av1MotionVectorPrecision.QuarterSample, true, 0, 0, 2, -1, 6)] + [InlineData((int)Av1MotionVectorPrecision.EighthSample, false, 0, 1, 1, 0, 11)] + [InlineData((int)Av1MotionVectorPrecision.EighthSample, true, 1, 1, 3, 1, 32)] + public void ReadMotionVectorUsesRequestedPrecision( + int precisionValue, + bool horizontal, + int magnitudeClass, + int integerOffset, + int fractional, + int highPrecision, + int expectedMagnitude) + { + Av1MotionVectorPrecision precision = (Av1MotionVectorPrecision)precisionValue; + Av1MotionVectorContext writerContext = new(); + Av1MotionVectorContext.Component component = horizontal ? writerContext.Horizontal : writerContext.Vertical; + Av1Distribution trailingDistribution = Av1DefaultDistributions.Drl[1]; + using Av1SymbolWriter writer = new(Configuration.Default, 8, updateCdf: true); + + writer.WriteSymbol(horizontal ? 1 : 2, writerContext.Joint); + writer.WriteSymbol(false, component.Sign); + writer.WriteSymbol(magnitudeClass, component.MagnitudeClass); + + if (magnitudeClass == 0) + { + writer.WriteSymbol(integerOffset, component.ClassZero); + } + else + { + // CLASS0_BITS is one, so a nonzero class transmits exactly magnitudeClass integer-offset bits. + for (int bit = 0; bit < magnitudeClass; bit++) + { + writer.WriteSymbol((integerOffset >> bit) & 1, component.OffsetBits[bit]); + } + } + + if (precision != Av1MotionVectorPrecision.Integer) + { + Av1Distribution fractionalDistribution = magnitudeClass == 0 + ? component.ClassZeroFractional[integerOffset] + : component.Fractional; + + writer.WriteSymbol(fractional, fractionalDistribution); + } + + if (precision == Av1MotionVectorPrecision.EighthSample) + { + Av1Distribution highPrecisionDistribution = magnitudeClass == 0 + ? component.ClassZeroHighPrecision + : component.HighPrecision; + + writer.WriteSymbol(highPrecision, highPrecisionDistribution); + } + + // The trailing decision detects either an omitted precision symbol being consumed or a required one being skipped. + writer.WriteSymbol(true, trailingDistribution); + + using IMemoryOwner encoded = writer.Exit(); + Av1FrameEntropyContext decoderContext = new(0); + uint normalJoint = decoderContext.MotionVector.Joint[0]; + uint displacementJoint = decoderContext.DisplacementVector.Joint[0]; + Av1SymbolDecoder decoder = new(Configuration.Default, encoded.Memory.Span, decoderContext, updateCdf: true); + Av1MotionVector reference = new(27, -11); + + Av1MotionVector actual = decoder.ReadMotionVector(reference, precision); + Av1MotionVector expected = horizontal + ? new Av1MotionVector(reference.Row, reference.Column + expectedMagnitude) + : new Av1MotionVector(reference.Row + expectedMagnitude, reference.Column); + + Assert.Equal(expected, actual); + Assert.NotEqual(normalJoint, decoderContext.MotionVector.Joint[0]); + Assert.Equal(displacementJoint, decoderContext.DisplacementVector.Joint[0]); + Assert.True(decoder.ReadDrl(1)); + } + + /// + /// Verifies that normal and displacement motion vectors never share adaptive distribution state. + /// + [Fact] + public void NormalAndDisplacementContextsAdaptIndependently() + { + Av1FrameEntropyContext context = new(0); + uint displacementJoint = context.DisplacementVector.Joint[0]; + uint normalFractional = context.MotionVector.Vertical.Fractional[0]; + + context.MotionVector.Joint.Update(3); + context.DisplacementVector.Vertical.Fractional.Update(2); + + Assert.NotEqual(displacementJoint, context.MotionVector.Joint[0]); + Assert.Equal(displacementJoint, context.DisplacementVector.Joint[0]); + Assert.Equal(normalFractional, context.MotionVector.Vertical.Fractional[0]); + Assert.NotEqual(normalFractional, context.DisplacementVector.Vertical.Fractional[0]); + } + + /// + /// Verifies that frame-context copies retain complete motion-vector state without sharing it. + /// + [Fact] + public void FrameEntropyCopyRetainsIndependentMotionVectorState() + { + Av1FrameEntropyContext source = new(0); + Av1FrameEntropyContext destination = new(0); + UpdateState(source.MotionVector, 1, 5); + UpdateState(source.DisplacementVector, 1, 7); + + destination.CopyFrom(source); + + AssertStateEqual(source.MotionVector, destination.MotionVector); + AssertStateEqual(source.DisplacementVector, destination.DisplacementVector); + + UpdateState(source.MotionVector, 0, 1); + UpdateState(source.DisplacementVector, 0, 1); + + AssertStateNotEqual(source.MotionVector, destination.MotionVector); + AssertStateNotEqual(source.DisplacementVector, destination.DisplacementVector); + } + + /// + /// Verifies that publishing frame state resets every motion-vector update count. + /// + [Fact] + public void FrameEntropySnapshotResetsMotionVectorUpdateCounts() + { + Av1FrameEntropyContext source = new(0); + Av1FrameEntropyContext snapshot = new(0); + UpdateState(source.MotionVector, 1, 20); + UpdateState(source.DisplacementVector, 1, 20); + + source.SnapshotTo(snapshot); + + AssertStateEqual(source.MotionVector, snapshot.MotionVector); + AssertStateEqual(source.DisplacementVector, snapshot.DisplacementVector); + + // The source retains twenty observations while the snapshot restarts at zero. The same next symbol therefore + // moves identical thresholds by different amounts only when every new distribution participates in reset. + UpdateState(source.MotionVector, 0, 1); + UpdateState(snapshot.MotionVector, 0, 1); + UpdateState(source.DisplacementVector, 0, 1); + UpdateState(snapshot.DisplacementVector, 0, 1); + + AssertStateNotEqual(source.MotionVector, snapshot.MotionVector); + AssertStateNotEqual(source.DisplacementVector, snapshot.DisplacementVector); + } + + /// + /// Verifies one complete motion-vector context against the normative defaults. + /// + /// The context under test. + private static void AssertContextDefaults(Av1MotionVectorContext context) + { + Assert.NotSame(context.Vertical, context.Horizontal); + AssertDistribution(context.Joint, [4096, 11264, 19328]); + AssertComponentDefaults(context.Vertical); + AssertComponentDefaults(context.Horizontal); + } + + /// + /// Verifies one component's complete set of normative defaults. + /// + /// The component under test. + private static void AssertComponentDefaults(Av1MotionVectorContext.Component component) + { + AssertDistribution(component.MagnitudeClass, [28672, 30976, 31858, 32320, 32551, 32656, 32740, 32757, 32762, 32767]); + Assert.Equal(2, component.ClassZeroFractional.Length); + AssertDistribution(component.ClassZeroFractional[0], [16384, 24576, 26624]); + AssertDistribution(component.ClassZeroFractional[1], [12288, 21248, 24128]); + AssertDistribution(component.Fractional, [8192, 17408, 21248]); + AssertDistribution(component.Sign, [16384]); + AssertDistribution(component.ClassZeroHighPrecision, [20480]); + AssertDistribution(component.HighPrecision, [16384]); + AssertDistribution(component.ClassZero, [27648]); + + ReadOnlySpan offsetThresholds = [17408, 17920, 18944, 20480, 22528, 24576, 28672, 29952, 29952, 30720]; + + Assert.Equal(offsetThresholds.Length, component.OffsetBits.Length); + for (int bit = 0; bit < offsetThresholds.Length; bit++) + { + uint expected = (uint)Av1Distribution.ProbabilityTop - offsetThresholds[bit]; + + Assert.Equal(2, component.OffsetBits[bit].NumberOfSymbols); + Assert.Equal(expected, component.OffsetBits[bit][0]); + } + } + + /// + /// Verifies one distribution after conversion from forward to inverse cumulative thresholds. + /// + /// The distribution under test. + /// The normative forward Q15 thresholds. + private static void AssertDistribution(Av1Distribution distribution, ReadOnlySpan forwardThresholds) + { + Assert.Equal(forwardThresholds.Length + 1, distribution.NumberOfSymbols); + for (int threshold = 0; threshold < forwardThresholds.Length; threshold++) + { + uint expected = (uint)Av1Distribution.ProbabilityTop - forwardThresholds[threshold]; + + Assert.Equal(expected, distribution[threshold]); + } + } + + /// + /// Applies the same observations to every distribution in a motion-vector context. + /// + /// The context to adapt. + /// The coded symbol used for each observation. + /// The number of observations. + private static void UpdateState(Av1MotionVectorContext context, int symbol, int count) + { + for (int update = 0; update < count; update++) + { + context.Joint.Update(symbol); + UpdateState(context.Vertical, symbol); + UpdateState(context.Horizontal, symbol); + } + } + + /// + /// Applies one observation to every distribution in one motion-vector component. + /// + /// The component to adapt. + /// The coded symbol used for the observation. + private static void UpdateState(Av1MotionVectorContext.Component component, int symbol) + { + component.MagnitudeClass.Update(symbol); + component.ClassZeroFractional[0].Update(symbol); + component.ClassZeroFractional[1].Update(symbol); + component.Fractional.Update(symbol); + component.Sign.Update(symbol); + component.ClassZeroHighPrecision.Update(symbol); + component.HighPrecision.Update(symbol); + component.ClassZero.Update(symbol); + + for (int bit = 0; bit < component.OffsetBits.Length; bit++) + { + component.OffsetBits[bit].Update(symbol); + } + } + + /// + /// Verifies equal adaptive thresholds across two motion-vector contexts. + /// + /// The expected context. + /// The actual context. + private static void AssertStateEqual(Av1MotionVectorContext expected, Av1MotionVectorContext actual) + { + Assert.Equal(expected.Joint[0], actual.Joint[0]); + AssertStateEqual(expected.Vertical, actual.Vertical); + AssertStateEqual(expected.Horizontal, actual.Horizontal); + } + + /// + /// Verifies equal adaptive thresholds across two motion-vector components. + /// + /// The expected component. + /// The actual component. + private static void AssertStateEqual(Av1MotionVectorContext.Component expected, Av1MotionVectorContext.Component actual) + { + Assert.Equal(expected.MagnitudeClass[0], actual.MagnitudeClass[0]); + Assert.Equal(expected.ClassZeroFractional[0][0], actual.ClassZeroFractional[0][0]); + Assert.Equal(expected.ClassZeroFractional[1][0], actual.ClassZeroFractional[1][0]); + Assert.Equal(expected.Fractional[0], actual.Fractional[0]); + Assert.Equal(expected.Sign[0], actual.Sign[0]); + Assert.Equal(expected.ClassZeroHighPrecision[0], actual.ClassZeroHighPrecision[0]); + Assert.Equal(expected.HighPrecision[0], actual.HighPrecision[0]); + Assert.Equal(expected.ClassZero[0], actual.ClassZero[0]); + + for (int bit = 0; bit < expected.OffsetBits.Length; bit++) + { + Assert.Equal(expected.OffsetBits[bit][0], actual.OffsetBits[bit][0]); + } + } + + /// + /// Verifies independently adaptive thresholds across two motion-vector contexts. + /// + /// The independently adapted context. + /// The copied or reset context. + private static void AssertStateNotEqual(Av1MotionVectorContext expected, Av1MotionVectorContext actual) + { + Assert.NotEqual(expected.Joint[0], actual.Joint[0]); + AssertStateNotEqual(expected.Vertical, actual.Vertical); + AssertStateNotEqual(expected.Horizontal, actual.Horizontal); + } + + /// + /// Verifies independently adaptive thresholds across two motion-vector components. + /// + /// The independently adapted component. + /// The copied or reset component. + private static void AssertStateNotEqual(Av1MotionVectorContext.Component expected, Av1MotionVectorContext.Component actual) + { + Assert.NotEqual(expected.MagnitudeClass[0], actual.MagnitudeClass[0]); + Assert.NotEqual(expected.ClassZeroFractional[0][0], actual.ClassZeroFractional[0][0]); + Assert.NotEqual(expected.ClassZeroFractional[1][0], actual.ClassZeroFractional[1][0]); + Assert.NotEqual(expected.Fractional[0], actual.Fractional[0]); + Assert.NotEqual(expected.Sign[0], actual.Sign[0]); + Assert.NotEqual(expected.ClassZeroHighPrecision[0], actual.ClassZeroHighPrecision[0]); + Assert.NotEqual(expected.HighPrecision[0], actual.HighPrecision[0]); + Assert.NotEqual(expected.ClassZero[0], actual.ClassZero[0]); + + for (int bit = 0; bit < expected.OffsetBits.Length; bit++) + { + Assert.NotEqual(expected.OffsetBits[bit][0], actual.OffsetBits[bit][0]); + } + } +} diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1MotionVectorTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1MotionVectorTests.cs new file mode 100644 index 000000000..706e12f29 --- /dev/null +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1MotionVectorTests.cs @@ -0,0 +1,158 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using SixLabors.ImageSharp.Formats.Heif.Av1.Motion; + +namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; + +/// +/// Verifies AV1 motion-vector precision, range, spatial-clamp, and temporal-projection semantics. +/// +[Trait("Format", "Avif")] +public class Av1MotionVectorTests +{ + /// + /// Verifies that high-precision vectors retain their one-eighth-sample components unchanged. + /// + [Fact] + public void LowerPrecisionRetainsHighPrecisionComponents() + { + Av1MotionVector vector = new(15, -15); + + Assert.Equal(vector, vector.LowerPrecision(allowHighPrecision: true, forceInteger: false)); + } + + /// + /// Verifies that low precision removes odd one-eighth-sample components toward zero. + /// + /// The original vertical component. + /// The original horizontal component. + /// The expected low-precision vertical component. + /// The expected low-precision horizontal component. + [Theory] + [InlineData(15, -15, 14, -14)] + [InlineData(14, -14, 14, -14)] + [InlineData(1, -1, 0, 0)] + public void LowerPrecisionReducesOddComponentsTowardZero(int row, int column, int expectedRow, int expectedColumn) + { + Av1MotionVector actual = new Av1MotionVector(row, column).LowerPrecision(allowHighPrecision: false, forceInteger: false); + + Assert.Equal(new Av1MotionVector(expectedRow, expectedColumn), actual); + } + + /// + /// Verifies AV1 integer-sample rounding, including half-sample ties toward zero on both signs. + /// + /// The original component in one-eighth-sample units. + /// The expected integer-precision component. + [Theory] + [InlineData(3, 0)] + [InlineData(4, 0)] + [InlineData(5, 8)] + [InlineData(11, 8)] + [InlineData(12, 8)] + [InlineData(13, 16)] + [InlineData(16, 16)] + [InlineData(-3, 0)] + [InlineData(-4, 0)] + [InlineData(-5, -8)] + [InlineData(-11, -8)] + [InlineData(-12, -8)] + [InlineData(-13, -16)] + [InlineData(-16, -16)] + public void LowerPrecisionRoundsIntegerHalfTiesTowardZero(int component, int expected) + { + Av1MotionVector actual = new Av1MotionVector(component, -component).LowerPrecision( + allowHighPrecision: true, + forceInteger: true); + + Assert.Equal(new Av1MotionVector(expected, -expected), actual); + } + + /// + /// Verifies that AV1 reserves both signed motion-vector endpoints. + /// + /// The vertical component. + /// The horizontal component. + /// The expected validity. + [Theory] + [InlineData(-16383, 16383, true)] + [InlineData(-16384, 0, false)] + [InlineData(-16385, 0, false)] + [InlineData(16384, 0, false)] + [InlineData(16385, 0, false)] + [InlineData(0, -16384, false)] + [InlineData(0, 16384, false)] + public void IsValidUsesExclusiveMotionVectorEndpoints(int row, int column, bool expected) + => Assert.Equal(expected, new Av1MotionVector(row, column).IsValid); + + /// + /// Verifies the complete-block and sixteen-sample borders used to clamp spatial reference candidates. + /// + [Fact] + public void ClampReferenceMatchesLibaomSpatialLimits() + { + const int blockWidth = 16; + const int blockHeight = 8; + const int blockToLeftEdge = -256; + const int blockToRightEdge = 512; + const int blockToTopEdge = -128; + const int blockToBottomEdge = 384; + + Av1MotionVector upper = new Av1MotionVector(1000, 1000).ClampReference( + blockWidth, + blockHeight, + blockToLeftEdge, + blockToRightEdge, + blockToTopEdge, + blockToBottomEdge); + + Av1MotionVector lower = new Av1MotionVector(-1000, -1000).ClampReference( + blockWidth, + blockHeight, + blockToLeftEdge, + blockToRightEdge, + blockToTopEdge, + blockToBottomEdge); + + Av1MotionVector inside = new Av1MotionVector(48, -64).ClampReference( + blockWidth, + blockHeight, + blockToLeftEdge, + blockToRightEdge, + blockToTopEdge, + blockToBottomEdge); + + Assert.Equal(new Av1MotionVector(576, 768), upper); + Assert.Equal(new Av1MotionVector(-320, -512), lower); + Assert.Equal(new Av1MotionVector(48, -64), inside); + } + + /// + /// Verifies AV1 fixed-point temporal projection, distance limiting, symmetric rounding, and endpoint clamping. + /// + /// The source vertical component. + /// The source horizontal component. + /// The signed source-to-target frame distance. + /// The positive source-to-reference frame distance. + /// The expected projected vertical component. + /// The expected projected horizontal component. + [Theory] + [InlineData(64, -96, 2, 4, 32, -48)] + [InlineData(64, -96, -2, 4, -32, 48)] + [InlineData(2, -2, 1, 3, 1, -1)] + [InlineData(31, -31, 40, 40, 31, -31)] + [InlineData(4095, -4095, 31, 1, 16383, -16383)] + public void ProjectTemporalMatchesLibaom( + int row, + int column, + int numerator, + int denominator, + int expectedRow, + int expectedColumn) + { + Av1MotionVector actual = new Av1MotionVector(row, column).ProjectTemporal(numerator, denominator); + + Assert.Equal(new Av1MotionVector(expectedRow, expectedColumn), actual); + } +} diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReconstructionConformanceTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReconstructionConformanceTests.cs index 39ca33fd0..7e75fce67 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReconstructionConformanceTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReconstructionConformanceTests.cs @@ -12,6 +12,7 @@ using SixLabors.ImageSharp.Formats.Heif.Av1.Transform; using SixLabors.ImageSharp.Memory; using SixLabors.ImageSharp.Metadata.Profiles.Cicp; using SixLabors.ImageSharp.PixelFormats; +using SixLabors.ImageSharp.Tests.Memory; using SixLabors.ImageSharp.Tests.TestUtilities; using SixLabors.ImageSharp.Tests.TestUtilities.ImageComparison; @@ -112,6 +113,21 @@ public class Av1ReconstructionConformanceTests /// private const int ProfileFixtureHeight = 256; + /// + /// The displayed width of the independent two-layer progressive fixture. + /// + private const int ProgressiveFixtureWidth = 33; + + /// + /// The displayed height of the independent two-layer progressive fixture. + /// + private const int ProgressiveFixtureHeight = 11; + + /// + /// The byte length of the fixture's base layer as declared by its a1lx property. + /// + private const int ProgressiveFirstLayerSize = 55; + /// /// The hardware configurations covering the available vector widths and the scalar color-conversion fallback. /// @@ -326,6 +342,87 @@ public class Av1ReconstructionConformanceTests public void DecodeWithIntraBlockCopyMatchesPinnedLibavifPresentation() => FeatureTestRunner.RunWithHwIntrinsicsFeature(ValidateIntraBlockCopyPresentedFixtures, PresentationConfigurations); + /// + /// Verifies the production single-reference inter-reconstruction path against exact native and presentation + /// references across the available vector widths and scalar fallback. + /// + [Fact] + public void DecodeProgressiveSingleReferenceMatchesPinnedReferences() + => FeatureTestRunner.RunWithHwIntrinsicsFeature( + ValidateProgressiveSingleReferenceFixtureWithDefaultConfiguration, + ReconstructionConfigurations); + + /// + /// Verifies production single-reference inter reconstruction with a constrained allocator. + /// + [Fact] + public void DecodeProgressiveSingleReferenceWithConstrainedAllocator() + { + TestMemoryAllocator allocator = new() { BufferCapacityInBytes = 1_024 }; + Configuration configuration = Configuration.Default.Clone(); + configuration.MemoryAllocator = allocator; + + ValidateProgressiveSingleReferenceFixture(configuration, verifyPresentation: false); + } + + /// + /// Verifies that the production dependent-frame result owns its motion-field storage until decoder disposal. + /// + [Fact] + public void DecodeProgressiveSingleReferenceTracksMotionFieldResultOwnership() + { + TestMemoryAllocator allocator = new(); + allocator.EnableNonThreadSafeLogging(); + Configuration configuration = Configuration.Default.Clone(); + configuration.MemoryAllocator = allocator; + byte[] payload = TestFile.Create(TestImages.Heif.Av1Progressive8BitPayload).Bytes; + + using Av1Decoder decoder = new(configuration); + using Av1FrameBuffer frameBuffer = decoder.DecodeFrameBuffer( + payload, + null, + null, + out _, + new Av1LayeredImageIndex(ProgressiveFirstLayerSize, 0, 0)); + + TestMemoryAllocator.AllocationRequest retainedMotionField = Assert.Single( + allocator.AllocationLog, + request => request.ElementType.Name == "RetainedMotionFieldEntry"); + + TestMemoryAllocator.AllocationRequest temporalMotionField = Assert.Single( + allocator.AllocationLog, + request => request.ElementType.Name == "TemporalMotionFieldEntry"); + + // Reference-slot and presentation owners are released while DecodeFrameBuffer transfers the native planes. + // The decoder's inspectable FrameInfo result remains the final motion-field owner until decoder disposal. + Assert.DoesNotContain( + allocator.ReturnLog, + returned => returned.HashCodeOfBuffer == retainedMotionField.HashCodeOfBuffer); + + Assert.DoesNotContain( + allocator.ReturnLog, + returned => returned.HashCodeOfBuffer == temporalMotionField.HashCodeOfBuffer); + + frameBuffer.Dispose(); + + Assert.DoesNotContain( + allocator.ReturnLog, + returned => returned.HashCodeOfBuffer == retainedMotionField.HashCodeOfBuffer); + + Assert.DoesNotContain( + allocator.ReturnLog, + returned => returned.HashCodeOfBuffer == temporalMotionField.HashCodeOfBuffer); + + decoder.Dispose(); + decoder.Dispose(); + + Assert.Single( + allocator.ReturnLog, + returned => returned.HashCodeOfBuffer == retainedMotionField.HashCodeOfBuffer); + + Assert.Single(allocator.ReturnLog, returned => returned.HashCodeOfBuffer == temporalMotionField.HashCodeOfBuffer); + } + /// /// Verifies lossless syntax, residual reconstruction, and exact native samples against scalar libaom for /// independently encoded eight-, ten-, and twelve-bit AVIF images. @@ -886,6 +983,118 @@ public class Av1ReconstructionConformanceTests requireIntraBlockCopy: true); } + /// + /// Runs the exact final-layer native and presentation comparisons with the default configuration. + /// + private static void ValidateProgressiveSingleReferenceFixtureWithDefaultConfiguration() + => ValidateProgressiveSingleReferenceFixture(Configuration.Default, verifyPresentation: true); + + /// + /// Verifies the final dependent layer with the requested allocator. + /// + /// The decoder configuration. + /// Whether to verify the final public RGBA presentation. + private static void ValidateProgressiveSingleReferenceFixture( + Configuration configuration, + bool verifyPresentation) + { + byte[] payload = TestFile.Create(TestImages.Heif.Av1Progressive8BitPayload).Bytes; + byte[] referenceBytes = TestFile.Create(TestImages.Heif.Av1Progressive8BitReference).Bytes; + ReadOnlySpan fileHeader = + "YUV4MPEG2 W33 H11 F25:1 Ip A0:0 C444alpha XYSCSS=444 XCOLORRANGE=FULL\n"u8; + + ReadOnlySpan frameHeader = "FRAME\n"u8; + + int planeSampleCount = ProgressiveFixtureWidth * ProgressiveFixtureHeight; + int frameSampleCount = planeSampleCount * 4; + + // The pinned reference contains both progressive YUV444-alpha outputs in decode order. Select the second frame + // so this assertion cannot pass by comparing only the independently decodable base layer. + ReadOnlySpan nativeReference = referenceBytes; + Assert.True(nativeReference.StartsWith(fileHeader)); + nativeReference = nativeReference[fileHeader.Length..]; + Assert.True(nativeReference.StartsWith(frameHeader)); + int storedFrameSize = frameHeader.Length + frameSampleCount; + Assert.Equal(storedFrameSize * 2, nativeReference.Length); + + ReadOnlySpan finalFrameReference = nativeReference[storedFrameSize..]; + Assert.True(finalFrameReference.StartsWith(frameHeader)); + finalFrameReference = finalFrameReference[frameHeader.Length..]; + Assert.Equal(frameSampleCount, finalFrameReference.Length); + + // The Y4M stores the color item's Y, U, and V planes before the auxiliary alpha plane. Native AV1 reconstruction + // is compared with exactly those first three planes of the final dependent frame. + ReadOnlySpan colorReference = finalFrameReference[..(planeSampleCount * 3)]; + + using Av1Decoder decoder = new(configuration); + using Av1FrameBuffer frameBuffer = decoder.DecodeFrameBuffer( + payload, + null, + null, + out _, + new Av1LayeredImageIndex(ProgressiveFirstLayerSize, 0, 0)); + + Assert.Equal(ProgressiveFixtureWidth, frameBuffer.Width); + Assert.Equal(ProgressiveFixtureHeight, frameBuffer.Height); + Assert.Equal(Av1BitDepth.EightBit, frameBuffer.BitDepth); + Assert.Equal(Av1ColorFormat.Yuv444, frameBuffer.ColorFormat); + Assert.Equal(1, frameBuffer.BufferY!.FastMemoryGroup.Count); + Assert.Equal(1, frameBuffer.BufferCb!.FastMemoryGroup.Count); + Assert.Equal(1, frameBuffer.BufferCr!.FastMemoryGroup.Count); + + ObuSequenceHeader sequenceHeader = Assert.IsType(decoder.SequenceHeader); + ObuFrameHeader finalFrameHeader = Assert.IsType(decoder.FrameHeader); + Av1FrameInfo frameInfo = Assert.IsType(decoder.FrameInfo); + + Assert.Equal(ObuFrameType.InterFrame, finalFrameHeader.FrameType); + + int superblockSizeLog2 = sequenceHeader.SuperblockSizeLog2; + int superblockColumnCount = Av1Math.AlignPowerOf2(sequenceHeader.MaxFrameWidth, superblockSizeLog2) >> superblockSizeLog2; + int superblockRowCount = Av1Math.AlignPowerOf2(sequenceHeader.MaxFrameHeight, superblockSizeLog2) >> superblockSizeLog2; + int interBlockCount = 0; + + // Traverse the final coding-block records once rather than revisiting every 4x4 map cell covered by each + // block. The syntax assertions ensure that this fixture reaches only the completed single-reference path. + for (int superblockRow = 0; superblockRow < superblockRowCount; superblockRow++) + { + for (int superblockColumn = 0; superblockColumn < superblockColumnCount; superblockColumn++) + { + Av1SuperblockInfo superblock = frameInfo.GetSuperblock(new Point(superblockColumn, superblockRow)); + foreach (Av1BlockModeInfo modeInfo in superblock.GetModeInfos()) + { + if (modeInfo.ReferenceFrames[0] < Av1ReferenceFrameType.Last) + { + continue; + } + + Assert.Equal(Av1ReferenceFrameType.None, modeInfo.ReferenceFrames[1]); + Assert.Equal(Av1MotionMode.SimpleTranslation, modeInfo.MotionMode); + interBlockCount++; + } + } + } + + Assert.NotEqual(0, interBlockCount); + AssertNativePlanesEqual(decoder, frameBuffer, colorReference); + + if (!verifyPresentation) + { + return; + } + + DecoderOptions options = new() { Configuration = configuration, MaxFrames = 1 }; + byte[] imageBytes = TestFile.Create(TestImages.Heif.Av1Progressive8BitAvif).Bytes; + byte[] presentationBytes = TestFile.Create(TestImages.Heif.Av1Progressive8BitPresentationReference).Bytes; + using Image image = Image.Load(options, imageBytes); + using Image presentationReference = Image.Load(presentationBytes); + + Assert.Equal(ProgressiveFixtureWidth, image.Width); + Assert.Equal(ProgressiveFixtureHeight, image.Height); + Assert.Single(image.Frames); + Assert.Equal(HeifBitDepth.Bit8, image.Metadata.GetHeifMetadata().BitDepth); + ImageComparer.Exact.VerifySimilarity(presentationReference, image); + } + /// /// Validates every lossless native fixture under the hardware configuration selected by /// . diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReferenceFrameStoreTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReferenceFrameStoreTests.cs index dddb14aaf..75d122098 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReferenceFrameStoreTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReferenceFrameStoreTests.cs @@ -1,13 +1,16 @@ // Copyright (c) Six Labors. // Licensed under the Six Labors Split License. +using System.Buffers; using System.Runtime.InteropServices; using SixLabors.ImageSharp.Formats.Heif.Av1; +using SixLabors.ImageSharp.Formats.Heif.Av1.Entropy; using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit; using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.FilmGrain; using SixLabors.ImageSharp.Formats.Heif.Av1.ReferenceFrames; using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; using SixLabors.ImageSharp.Memory; +using SixLabors.ImageSharp.Tests.Memory; namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; @@ -200,6 +203,143 @@ public class Av1ReferenceFrameStoreTests Assert.Null(store.Resolve(1)); } + /// + /// Verifies motion-field ownership across frame initialization, reference aliases, shown output, and final disposal. + /// + [Fact] + public void MotionFieldsFollowReferenceAliasesAndPresentationOwnership() + { + TestMemoryAllocator allocator = new(); + allocator.EnableNonThreadSafeLogging(); + Configuration configuration = Configuration.Default.Clone(); + configuration.MemoryAllocator = allocator; + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(64, 64, Av1BitDepth.EightBit, true, false, false); + sequenceHeader.OrderHintInfo.EnableOrderHint = true; + sequenceHeader.OrderHintInfo.EnableReferenceFrameMotionVectors = true; + + using Av1ReferenceFrameStore sourceReferences = new(); + using Av1FrameInfo sourceFrameInfo = new(sequenceHeader); + ObuFrameHeader sourceHeader = new() + { + FrameType = ObuFrameType.KeyFrame, + ShowFrame = true, + OrderHint = 0, + ModeInfoColumnCount = 16, + ModeInfoRowCount = 16 + }; + + Av1ReferenceFrame sourceFrame = new( + new Av1FrameBuffer(Configuration.Default, sequenceHeader, Av1ColorFormat.Yuv400, false), + sourceHeader, + sourceFrameInfo); + + Assert.True(sourceReferences.Commit(byte.MaxValue, sourceFrame, showFrame: false)); + + ObuFrameHeader frameHeader = new() + { + FrameType = ObuFrameType.InterFrame, + OrderHint = 1, + ModeInfoColumnCount = 16, + ModeInfoRowCount = 16, + UseReferenceFrameMotionVectors = true + }; + + using Av1FrameInfo frameInfo = new(sequenceHeader); + frameInfo.InitializeMotionField(configuration, sequenceHeader, frameHeader, sourceReferences); + + Assert.Equal(2, allocator.AllocationLog.Count); + Assert.Contains(allocator.AllocationLog, request => request.ElementType.Name == "RetainedMotionFieldEntry"); + Assert.Contains(allocator.AllocationLog, request => request.ElementType.Name == "TemporalMotionFieldEntry"); + + using Av1ReferenceFrameStore store = new(); + Av1ReferenceFrame frame = new( + new Av1FrameBuffer(Configuration.Default, sequenceHeader, Av1ColorFormat.Yuv400, false), + frameHeader, + frameInfo); + + Assert.True(store.Commit(byte.MaxValue, frame, showFrame: true)); + + // The reference frame owns the shared FrameInfo after the tile-reader lease ends. Physical reference slots + // and the shown-output pointer are aliases of that owner and must not release either motion field early. + frameInfo.Dispose(); + Assert.Empty(allocator.ReturnLog); + + Av1ReferenceFrame output = store.TakeOutput(); + Assert.Empty(allocator.ReturnLog); + + output.Dispose(); + output.Dispose(); + store.Dispose(); + + Assert.All( + allocator.AllocationLog, + allocation => Assert.Single( + allocator.ReturnLog, + returned => returned.HashCodeOfBuffer == allocation.HashCodeOfBuffer)); + + Assert.Equal(2, allocator.ReturnLog.Count); + } + + /// + /// Verifies that tile-reader construction unwinds every successful allocation when temporal-field allocation fails. + /// + [Fact] + public void MotionFieldAllocationFailureUnwindsTileReaderOwnership() + { + FailingTemporalMotionFieldAllocator allocator = new(); + Configuration configuration = Configuration.Default.Clone(); + configuration.MemoryAllocator = allocator; + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(64, 64, Av1BitDepth.EightBit, true, false, false); + sequenceHeader.OrderHintInfo.EnableOrderHint = true; + sequenceHeader.OrderHintInfo.EnableReferenceFrameMotionVectors = true; + + using Av1ReferenceFrameStore referenceFrames = new(); + using Av1FrameInfo retainedFrameInfo = new(sequenceHeader); + ObuFrameHeader retainedHeader = new() + { + FrameType = ObuFrameType.KeyFrame, + ShowFrame = true, + ModeInfoColumnCount = 16, + ModeInfoRowCount = 16 + }; + + Av1ReferenceFrame retainedFrame = new( + new Av1FrameBuffer(Configuration.Default, sequenceHeader, Av1ColorFormat.Yuv400, false), + retainedHeader, + retainedFrameInfo); + + Assert.True(referenceFrames.Commit(byte.MaxValue, retainedFrame, showFrame: false)); + + ObuFrameHeader frameHeader = new() + { + FrameType = ObuFrameType.InterFrame, + OrderHint = 1, + ModeInfoColumnCount = 16, + ModeInfoRowCount = 16, + UseReferenceFrameMotionVectors = true + }; + + Av1FrameEntropyContexts entropyContexts = new(0); + + Assert.Throws( + () => new Av1TileReader( + configuration, + sequenceHeader, + frameHeader, + entropyContexts, + null, + referenceFrames)); + + Assert.NotEmpty(allocator.AllocationLog); + Assert.All( + allocator.AllocationLog, + allocation => Assert.Single( + allocator.ReturnLog, + returned => returned.HashCodeOfBuffer == allocation.HashCodeOfBuffer)); + + Assert.Equal(allocator.AllocationLog.Count, allocator.ReturnLog.Count); + } + /// /// Verifies that an eight-bit presentation copy contains every byte of each padded plane and the complete active geometry. /// @@ -214,6 +354,36 @@ public class Av1ReferenceFrameStoreTests public void CopyToCopiesCompletePaddedHighBitDepthFrame() => ValidateCompleteFrameCopy(Av1BitDepth.TwelveBit); + /// + /// Verifies that luma and subsampled chroma allocations cover the greatest legal unscaled UMV prediction extent. + /// + [Fact] + public void PaddedPlanesCoverMaximumUnscaledMotionVectorExtent() + { + const int maximumLumaExtent = 135; + const int maximumSubsampledExtent = 71; + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(128, 128, Av1BitDepth.EightBit, false, true, true); + using Av1FrameBuffer frameBuffer = new(Configuration.Default, sequenceHeader, Av1ColorFormat.Yuv420, false); + + Span luma = frameBuffer.GetPaddedPlaneSpan(Av1Plane.Y, 0, 0, out int lumaStride, out Point lumaOrigin); + int lumaHeight = luma.Length / lumaStride; + + Assert.True(lumaOrigin.X >= maximumLumaExtent); + Assert.True(lumaOrigin.Y >= maximumLumaExtent); + Assert.True(lumaStride - lumaOrigin.X - frameBuffer.Width >= maximumLumaExtent); + Assert.True(lumaHeight - lumaOrigin.Y - frameBuffer.Height >= maximumLumaExtent); + + Span chroma = frameBuffer.GetPaddedPlaneSpan(Av1Plane.U, 1, 1, out int chromaStride, out Point chromaOrigin); + int chromaWidth = Av1Math.DivideLog2Ceiling(frameBuffer.Width, 1); + int chromaHeight = Av1Math.DivideLog2Ceiling(frameBuffer.Height, 1); + int chromaAllocationHeight = chroma.Length / chromaStride; + + Assert.True(chromaOrigin.X >= maximumSubsampledExtent); + Assert.True(chromaOrigin.Y >= maximumSubsampledExtent); + Assert.True(chromaStride - chromaOrigin.X - chromaWidth >= maximumSubsampledExtent); + Assert.True(chromaAllocationHeight - chromaOrigin.Y - chromaHeight >= maximumSubsampledExtent); + } + /// /// Verifies that AV1 reference-border extension repeats the nearest visible edge across every allocated plane sample. /// @@ -571,8 +741,35 @@ public class Av1ReferenceFrameStoreTests { ObuSequenceHeader sequenceHeader = CreateSequenceHeader(1, 1, Av1BitDepth.EightBit, true, false, false); Av1FrameBuffer frameBuffer = new(Configuration.Default, sequenceHeader, Av1ColorFormat.Yuv400, false); + using Av1FrameInfo frameInfo = new(sequenceHeader); + + return new Av1ReferenceFrame(frameBuffer, new ObuFrameHeader(), frameInfo); + } + + /// + /// Fails the temporal motion-field rent after allowing every earlier tile-reader allocation to succeed. + /// + private sealed class FailingTemporalMotionFieldAllocator : TestMemoryAllocator + { + /// + /// Initializes a new instance of the class. + /// + public FailingTemporalMotionFieldAllocator() => this.EnableNonThreadSafeLogging(); + + /// + protected override AllocationTrackedMemoryManager AllocateCore( + int length, + AllocationOptions options = AllocationOptions.None) + { + if (typeof(T).Name == "TemporalMotionFieldEntry") + { + // Fail before the owner is published so the allocation log contains only resources that the + // Av1TileReader constructor must unwind. + throw new InvalidMemoryOperationException("The configured temporal motion-field allocation failed."); + } - return new Av1ReferenceFrame(frameBuffer, new ObuFrameHeader(), new Av1FrameInfo(sequenceHeader)); + return base.AllocateCore(length, options); + } } /// diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReferenceMotionVectorsTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReferenceMotionVectorsTests.cs new file mode 100644 index 000000000..941beebfc --- /dev/null +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReferenceMotionVectorsTests.cs @@ -0,0 +1,498 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using SixLabors.ImageSharp.Formats.Heif.Av1; +using SixLabors.ImageSharp.Formats.Heif.Av1.Motion; +using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit; +using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; +using SixLabors.ImageSharp.Formats.Heif.Av1.ReferenceFrames; +using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; + +namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; + +/// +/// Verifies the spatial, temporal, global, and extension rules used to derive single-reference AV1 motion vectors. +/// +[Trait("Format", "Avif")] +public class Av1ReferenceMotionVectorsTests +{ + /// + /// Verifies adjacent-direction counting, duplicate weighting, stable ordering, and the nearest, near, and new-reference accessors. + /// + [Fact] + public void BuildOrdersAdjacentCandidatesAndPacksModeContext() + { + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(enableTemporalMotionVectors: false); + ObuFrameHeader frameHeader = CreateFrameHeader(orderHint: 0, useReferenceFrameMotionVectors: false); + using Av1FrameInfo frameInfo = new(sequenceHeader); + FillFrameWithIntraBlocks(frameInfo, sequenceHeader); + + Av1MotionVector above = new(24, -10); + Av1MotionVector left = new(-14, 30); + AddModeInfo(frameInfo, sequenceHeader, new Point(8, 4), Av1BlockSize.Block16x16, Av1ReferenceFrameType.Last, above, Av1PredictionMode.NewMotionVector); + AddModeInfo( + frameInfo, + sequenceHeader, + new Point(4, 8), + Av1BlockSize.Block16x16, + Av1ReferenceFrameType.Last, + left, + Av1PredictionMode.NearestMotionVector); + + AddModeInfo( + frameInfo, + sequenceHeader, + new Point(12, 7), + Av1BlockSize.Block4x4, + Av1ReferenceFrameType.Last, + above, + Av1PredictionMode.NearestMotionVector); + + Av1SuperblockInfo superblockInfo = frameInfo.GetSuperblock(Point.Empty); + Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block16x16, new Point(8, 8)); + Av1PartitionInfo partitionInfo = new(modeInfo, superblockInfo, true, Av1PartitionType.None) + { + ColumnIndex = 8, + RowIndex = 8, + }; + + Av1TileInfo tileInfo = new(0, 0, frameHeader); + partitionInfo.ComputeBoundaryOffsets(sequenceHeader, frameHeader, tileInfo); + Av1ReferenceMotionVectors referenceMotionVectors = new(); + + referenceMotionVectors.Build( + ref partitionInfo, + tileInfo, + frameInfo, + sequenceHeader, + frameHeader, + Av1ReferenceFrameType.Last); + + Assert.Equal(2, referenceMotionVectors.Count); + Assert.Equal(84, referenceMotionVectors.ModeContext); + Assert.Equal(above, referenceMotionVectors.Candidates[0]); + Assert.Equal(left, referenceMotionVectors.Candidates[1]); + Assert.Equal((ushort)660, referenceMotionVectors.Weights[0]); + Assert.Equal((ushort)656, referenceMotionVectors.Weights[1]); + Assert.Equal(above, referenceMotionVectors.Nearest); + Assert.Equal(left, referenceMotionVectors.GetNearReference(0)); + Assert.Equal(above, referenceMotionVectors.GetNewReference(0)); + } + + /// + /// Verifies that outer candidates are weight-sorted independently without crossing the nearest-region boundary. + /// + [Fact] + public void BuildSortsOuterCandidatesInsideTheirOwnRegion() + { + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(enableTemporalMotionVectors: false); + ObuFrameHeader frameHeader = CreateFrameHeader(orderHint: 0, useReferenceFrameMotionVectors: false); + Av1FrameInfo frameInfo = new(sequenceHeader); + FillFrameWithIntraBlocks(frameInfo, sequenceHeader); + + Av1MotionVector nearest = new(8, 16); + Av1MotionVector topLeft = new(24, 32); + Av1MotionVector outerRow = new(40, 48); + AddModeInfo( + frameInfo, + sequenceHeader, + new Point(8, 7), + Av1BlockSize.Block4x4, + Av1ReferenceFrameType.Last, + nearest, + Av1PredictionMode.NearestMotionVector); + + // A 4x4 intra neighbor keeps the adjacent scan from marking the deeper row as covered by a large background block. + AddModeInfo( + frameInfo, + sequenceHeader, + new Point(9, 7), + Av1BlockSize.Block4x4, + Av1ReferenceFrameType.Intra, + default, + Av1PredictionMode.DC); + + AddModeInfo( + frameInfo, + sequenceHeader, + new Point(7, 7), + Av1BlockSize.Block4x4, + Av1ReferenceFrameType.Last, + topLeft, + Av1PredictionMode.NearestMotionVector); + + AddModeInfo( + frameInfo, + sequenceHeader, + new Point(9, 3), + Av1BlockSize.Block8x16, + Av1ReferenceFrameType.Last, + outerRow, + Av1PredictionMode.NearestMotionVector); + + Av1SuperblockInfo superblockInfo = frameInfo.GetSuperblock(Point.Empty); + Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block8x8, new Point(8, 8)); + Av1PartitionInfo partitionInfo = new(modeInfo, superblockInfo, true, Av1PartitionType.None) + { + ColumnIndex = 8, + RowIndex = 8, + }; + + Av1TileInfo tileInfo = new(0, 0, frameHeader); + partitionInfo.ComputeBoundaryOffsets(sequenceHeader, frameHeader, tileInfo); + Av1ReferenceMotionVectors referenceMotionVectors = new(); + + referenceMotionVectors.Build( + ref partitionInfo, + tileInfo, + frameInfo, + sequenceHeader, + frameHeader, + Av1ReferenceFrameType.Last); + + Assert.Equal(3, referenceMotionVectors.Count); + Assert.Equal(nearest, referenceMotionVectors.Candidates[0]); + Assert.Equal(outerRow, referenceMotionVectors.Candidates[1]); + Assert.Equal(topLeft, referenceMotionVectors.Candidates[2]); + Assert.Equal((ushort)642, referenceMotionVectors.Weights[0]); + Assert.Equal((ushort)8, referenceMotionVectors.Weights[1]); + Assert.Equal((ushort)4, referenceMotionVectors.Weights[2]); + Assert.Equal(51, referenceMotionVectors.ModeContext); + } + + /// + /// Verifies that an affine global-motion neighbor contributes the current block's global vector while extension retains its decoded vector. + /// + [Fact] + public void BuildSubstitutesAffineGlobalMotionForDirectCandidate() + { + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(enableTemporalMotionVectors: false); + ObuFrameHeader frameHeader = CreateFrameHeader(orderHint: 0, useReferenceFrameMotionVectors: false); + Av1GlobalMotionParameters globalMotion = Av1GlobalMotionParameters.Identity; + globalMotion.Type = Av1GlobalMotionType.Affine; + globalMotion[0] = 4096; + globalMotion[1] = -2048; + globalMotion[2] = Av1GlobalMotionParameters.ModelScale + 512; + globalMotion[5] = Av1GlobalMotionParameters.ModelScale; + frameHeader.GetGlobalMotionParameters()[0] = globalMotion; + + Av1FrameInfo frameInfo = new(sequenceHeader); + FillFrameWithIntraBlocks(frameInfo, sequenceHeader); + Av1MotionVector decoded = new(40, -24); + AddModeInfo( + frameInfo, + sequenceHeader, + new Point(8, 4), + Av1BlockSize.Block16x16, + Av1ReferenceFrameType.Last, + decoded, + Av1PredictionMode.GlobalMotionVector); + + Av1SuperblockInfo superblockInfo = frameInfo.GetSuperblock(Point.Empty); + Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block16x16, new Point(8, 8)); + Av1PartitionInfo partitionInfo = new(modeInfo, superblockInfo, true, Av1PartitionType.None) + { + ColumnIndex = 8, + RowIndex = 8, + }; + + Av1TileInfo tileInfo = new(0, 0, frameHeader); + partitionInfo.ComputeBoundaryOffsets(sequenceHeader, frameHeader, tileInfo); + Av1ReferenceMotionVectors referenceMotionVectors = new(); + + referenceMotionVectors.Build( + ref partitionInfo, + tileInfo, + frameInfo, + sequenceHeader, + frameHeader, + Av1ReferenceFrameType.Last); + + Av1MotionVector expectedGlobal = globalMotion.GetMotionVector( + frameHeader.AllowHighPrecisionMotionVector, + modeInfo.BlockSize, + new Point(partitionInfo.ColumnIndex, partitionInfo.RowIndex), + frameHeader.ForceIntegerMotionVector); + + Assert.Equal(2, referenceMotionVectors.Count); + Assert.Equal(expectedGlobal, referenceMotionVectors.Candidates[0]); + Assert.Equal(decoded, referenceMotionVectors.Candidates[1]); + Assert.Equal((ushort)656, referenceMotionVectors.Weights[0]); + Assert.Equal((ushort)2, referenceMotionVectors.Weights[1]); + } + + /// + /// Verifies that stack extension reverses an opposite-side vector without reweighting a candidate already in the + /// direct stack. + /// + [Fact] + public void BuildReversesOppositeDirectionExtensionCandidate() + { + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(enableTemporalMotionVectors: false); + sequenceHeader.OrderHintInfo.EnableOrderHint = true; + sequenceHeader.OrderHintInfo.OrderHintBits = 5; + ObuFrameHeader frameHeader = CreateFrameHeader(orderHint: 10, useReferenceFrameMotionVectors: false); + frameHeader.GetReferenceFrameIndices()[0] = 0; + frameHeader.GetReferenceFrameIndices()[4] = 1; + + using Av1ReferenceFrameStore referenceFrames = new(); + Av1ReferenceFrame past = CreateReferenceFrame(sequenceHeader, orderHint: 8); + Av1ReferenceFrame future = CreateReferenceFrame(sequenceHeader, orderHint: 12); + Assert.True(referenceFrames.Commit(1, past, showFrame: false)); + Assert.True(referenceFrames.Commit(2, future, showFrame: false)); + + using Av1FrameInfo frameInfo = new(sequenceHeader); + frameInfo.InitializeMotionField(Configuration.Default, sequenceHeader, frameHeader, referenceFrames); + FillFrameWithIntraBlocks(frameInfo, sequenceHeader); + Av1MotionVector direct = new(16, 24); + Av1BlockModeInfo candidate = AddModeInfo( + frameInfo, + sequenceHeader, + new Point(8, 4), + Av1BlockSize.Block16x16, + Av1ReferenceFrameType.Last, + direct, + Av1PredictionMode.NearestMotionVector); + + // The direct scan adds the first reference with its normative adjacent weight. Extension visits both entries: + // it must ignore that duplicate and append only the sign-corrected backward-reference vector. + candidate.ReferenceFrames[1] = Av1ReferenceFrameType.Backward; + candidate.MotionVectors[1] = new Av1MotionVector(40, -24); + + Av1SuperblockInfo superblockInfo = frameInfo.GetSuperblock(Point.Empty); + Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block16x16, new Point(8, 8)); + Av1PartitionInfo partitionInfo = new(modeInfo, superblockInfo, true, Av1PartitionType.None) + { + ColumnIndex = 8, + RowIndex = 8, + }; + + Av1TileInfo tileInfo = new(0, 0, frameHeader); + partitionInfo.ComputeBoundaryOffsets(sequenceHeader, frameHeader, tileInfo); + Av1ReferenceMotionVectors referenceMotionVectors = new(); + + referenceMotionVectors.Build( + ref partitionInfo, + tileInfo, + frameInfo, + sequenceHeader, + frameHeader, + Av1ReferenceFrameType.Last); + + Av1MotionVector expected = new(-40, 24); + Assert.Equal(2, referenceMotionVectors.Count); + Assert.Equal(direct, referenceMotionVectors.Candidates[0]); + Assert.Equal(expected, referenceMotionVectors.Candidates[1]); + Assert.Equal((ushort)656, referenceMotionVectors.Weights[0]); + Assert.Equal((ushort)2, referenceMotionVectors.Weights[1]); + Assert.Equal(direct, referenceMotionVectors.Nearest); + Assert.Equal(direct, referenceMotionVectors.GetNewReference(0)); + Assert.Equal(expected, referenceMotionVectors.GetNearReference(0)); + } + + /// + /// Verifies temporal field sampling, candidate deduplication, accumulated weight, and the global-motion context bit. + /// + [Fact] + public void BuildAccumulatesProjectedTemporalCandidates() + { + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(enableTemporalMotionVectors: true); + sequenceHeader.OrderHintInfo.EnableOrderHint = true; + sequenceHeader.OrderHintInfo.OrderHintBits = 5; + + using Av1ReferenceFrameStore priorReferences = new(); + Av1ReferenceFrame prior = CreateReferenceFrame(sequenceHeader, orderHint: 6); + Assert.True(priorReferences.Commit(1, prior, showFrame: false)); + + ObuFrameHeader sourceHeader = CreateFrameHeader(orderHint: 8, useReferenceFrameMotionVectors: false); + using Av1FrameInfo sourceFrameInfo = new(sequenceHeader); + sourceFrameInfo.InitializeMotionField(Configuration.Default, sequenceHeader, sourceHeader, priorReferences); + FillFrameWithInterBlocks(sourceFrameInfo, sequenceHeader, Av1ReferenceFrameType.Last, default); + + using Av1ReferenceFrameStore sourceReferences = new(); + Av1ReferenceFrame source = new( + new Av1FrameBuffer(Configuration.Default, sequenceHeader, Av1ColorFormat.Yuv400, false), + sourceHeader, + sourceFrameInfo); + + Assert.True(sourceReferences.Commit(1, source, showFrame: false)); + + ObuFrameHeader frameHeader = CreateFrameHeader(orderHint: 10, useReferenceFrameMotionVectors: true); + using Av1FrameInfo frameInfo = new(sequenceHeader); + frameInfo.InitializeMotionField(Configuration.Default, sequenceHeader, frameHeader, sourceReferences); + FillFrameWithIntraBlocks(frameInfo, sequenceHeader); + + Av1SuperblockInfo superblockInfo = frameInfo.GetSuperblock(Point.Empty); + Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block16x16, new Point(8, 8)); + Av1PartitionInfo partitionInfo = new(modeInfo, superblockInfo, true, Av1PartitionType.None) + { + ColumnIndex = 8, + RowIndex = 8, + }; + + Av1TileInfo tileInfo = new(0, 0, frameHeader); + partitionInfo.ComputeBoundaryOffsets(sequenceHeader, frameHeader, tileInfo); + Av1ReferenceMotionVectors referenceMotionVectors = new(); + + referenceMotionVectors.Build( + ref partitionInfo, + tileInfo, + frameInfo, + sequenceHeader, + frameHeader, + Av1ReferenceFrameType.Last); + + Assert.Equal(1, referenceMotionVectors.Count); + Assert.Equal(default, referenceMotionVectors.Candidates[0]); + Assert.Equal((ushort)14, referenceMotionVectors.Weights[0]); + Assert.Equal(0, referenceMotionVectors.ModeContext); + } + + /// + /// Creates the monochrome 128-by-128 sequence geometry shared by reference-motion-vector tests. + /// + /// Whether projected reference-frame motion vectors are enabled. + /// The configured sequence header. + private static ObuSequenceHeader CreateSequenceHeader(bool enableTemporalMotionVectors) + => new() + { + MaxFrameWidth = 128, + MaxFrameHeight = 128, + Use128x128Superblock = false, + ColorConfig = new ObuColorConfig + { + IsMonochrome = true, + BitDepth = Av1BitDepth.EightBit, + }, + OrderHintInfo = new ObuOrderHintInfo + { + EnableReferenceFrameMotionVectors = enableTemporalMotionVectors, + }, + }; + + /// + /// Creates one inter-frame header whose single tile covers the complete test frame. + /// + /// The frame's modulo display-order hint. + /// Whether this frame consumes its projected temporal motion field. + /// The configured frame header. + private static ObuFrameHeader CreateFrameHeader(uint orderHint, bool useReferenceFrameMotionVectors) + => new() + { + FrameType = ObuFrameType.InterFrame, + OrderHint = orderHint, + ModeInfoColumnCount = 32, + ModeInfoRowCount = 32, + AllowHighPrecisionMotionVector = true, + UseReferenceFrameMotionVectors = useReferenceFrameMotionVectors, + TilesInfo = new ObuTileGroupHeader + { + TileColumnCount = 1, + TileRowCount = 1, + TileColumnStartModeInfo = [0, 32], + TileRowStartModeInfo = [0, 32], + }, + }; + + /// + /// Maps one intra block over each 64-by-64 superblock so every spatial search position has initialized mode information. + /// + /// The frame map to initialize. + /// The sequence geometry defining the superblock grid. + private static void FillFrameWithIntraBlocks(Av1FrameInfo frameInfo, ObuSequenceHeader sequenceHeader) + { + for (int row = 0; row < 32; row += sequenceHeader.SuperblockModeInfoSize) + { + for (int column = 0; column < 32; column += sequenceHeader.SuperblockModeInfoSize) + { + AddModeInfo( + frameInfo, + sequenceHeader, + new Point(column, row), + Av1BlockSize.Block64x64, + Av1ReferenceFrameType.Intra, + default, + Av1PredictionMode.DC); + } + } + } + + /// + /// Maps one inter block over each 64-by-64 superblock and publishes its vector to the retained motion field. + /// + /// The frame map and retained field to initialize. + /// The sequence geometry defining the superblock grid. + /// The canonical reference selected by each block. + /// The retained motion vector. + private static void FillFrameWithInterBlocks( + Av1FrameInfo frameInfo, + ObuSequenceHeader sequenceHeader, + Av1ReferenceFrameType referenceFrame, + Av1MotionVector motionVector) + { + for (int row = 0; row < 32; row += sequenceHeader.SuperblockModeInfoSize) + { + for (int column = 0; column < 32; column += sequenceHeader.SuperblockModeInfoSize) + { + AddModeInfo( + frameInfo, + sequenceHeader, + new Point(column, row), + Av1BlockSize.Block64x64, + referenceFrame, + motionVector, + Av1PredictionMode.NearestMotionVector); + } + } + } + + /// + /// Creates and maps one mode-information block at a frame-relative position. + /// + /// The frame map that owns the block. + /// The sequence geometry defining superblock-relative addressing. + /// The block origin in frame-relative 4x4 units. + /// The block geometry. + /// The primary prediction reference. + /// The primary motion vector. + /// The decoded luma or inter prediction mode. + /// The mapped mode-information block. + private static Av1BlockModeInfo AddModeInfo( + Av1FrameInfo frameInfo, + ObuSequenceHeader sequenceHeader, + Point position, + Av1BlockSize blockSize, + Av1ReferenceFrameType referenceFrame, + Av1MotionVector motionVector, + Av1PredictionMode predictionMode) + { + int superblockSize = sequenceHeader.SuperblockModeInfoSize; + Point superblockPosition = new(position.X / superblockSize, position.Y / superblockSize); + Point relativePosition = new(position.X % superblockSize, position.Y % superblockSize); + Av1SuperblockInfo superblockInfo = frameInfo.GetSuperblock(superblockPosition); + Av1BlockModeInfo modeInfo = new(blockSize, relativePosition) + { + YMode = predictionMode, + }; + + modeInfo.ReferenceFrames[0] = referenceFrame; + modeInfo.MotionVectors[0] = motionVector; + frameInfo.UpdateModeInfo(modeInfo, superblockInfo); + superblockInfo.BlockCount++; + return modeInfo; + } + + /// + /// Creates a retained monochrome frame at one display-order hint. + /// + /// The sequence geometry used by the retained frame. + /// The retained frame's modulo display-order hint. + /// A frame owner whose sample buffer and mode state are ready for reference-map ownership. + private static Av1ReferenceFrame CreateReferenceFrame(ObuSequenceHeader sequenceHeader, uint orderHint) + { + ObuFrameHeader frameHeader = CreateFrameHeader(orderHint, useReferenceFrameMotionVectors: false); + Av1FrameBuffer frameBuffer = new(Configuration.Default, sequenceHeader, Av1ColorFormat.Yuv400, false); + using Av1FrameInfo frameInfo = new(sequenceHeader); + return new Av1ReferenceFrame(frameBuffer, frameHeader, frameInfo); + } +} diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1SingleReferenceEntropyTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1SingleReferenceEntropyTests.cs new file mode 100644 index 000000000..9e8908761 --- /dev/null +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1SingleReferenceEntropyTests.cs @@ -0,0 +1,330 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Buffers; +using SixLabors.ImageSharp.Formats.Heif.Av1; +using SixLabors.ImageSharp.Formats.Heif.Av1.Entropy; +using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; + +namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; + +/// +/// Verifies the adaptive distributions and spatial contexts used to select an AV1 inter block's reference mode and frame. +/// +[Trait("Format", "Avif")] +public class Av1SingleReferenceEntropyTests +{ + /// + /// Verifies all eighteen normative single-reference distributions against libaom's forward Q15 defaults. + /// + [Fact] + public void SingleReferenceDefaultsMatchLibaom() + { + uint[][] forwardThresholds = + [ + [4897, 1555, 4236, 8650, 904, 1444], + [16973, 16751, 19647, 24773, 11014, 15087], + [29744, 30279, 31194, 31895, 26875, 30304], + ]; + + Av1Distribution[][] distributions = Av1DefaultDistributions.SingleReference; + + Assert.Equal(forwardThresholds.Length, distributions.Length); + for (int context = 0; context < distributions.Length; context++) + { + Assert.Equal(forwardThresholds[context].Length, distributions[context].Length); + for (int decision = 0; decision < distributions[context].Length; decision++) + { + // Av1Distribution stores inverse cumulative thresholds. Convert each published forward default by the + // same Q15 complement used by production construction before comparing the exact value. + uint expected = (uint)Av1Distribution.ProbabilityTop - forwardThresholds[context][decision]; + + Assert.Equal(expected, distributions[context][decision][0]); + Assert.Equal(2, distributions[context][decision].NumberOfSymbols); + } + } + } + + /// + /// Verifies the five normative block reference-mode distributions against libaom's forward Q15 defaults. + /// + [Fact] + public void CompInterDefaultsMatchLibaom() + { + uint[] forwardThresholds = [26828, 24035, 12031, 10640, 2901]; + Av1Distribution[] distributions = Av1DefaultDistributions.CompInter; + + Assert.Equal(forwardThresholds.Length, distributions.Length); + for (int context = 0; context < distributions.Length; context++) + { + uint expected = (uint)Av1Distribution.ProbabilityTop - forwardThresholds[context]; + + Assert.Equal(expected, distributions[context][0]); + Assert.Equal(2, distributions[context].NumberOfSymbols); + } + } + + /// + /// Verifies that every semantic reader selects its exact context row and single-reference tree column. + /// + /// The zero-based single-reference tree decision. + /// The neighboring reference-vote context. + [Theory] + [MemberData(nameof(GetReaderCases))] + public void SingleReferenceReadersUseRequestedDistribution(int decision, int context) + { + bool[] expected = [false, true, true, false, true, false, false, true]; + Av1Distribution writerDistribution = Av1DefaultDistributions.SingleReference[context][decision]; + using Av1SymbolWriter writer = new(Configuration.Default, 8, updateCdf: true); + + foreach (bool value in expected) + { + writer.WriteSymbol(value, writerDistribution); + } + + using IMemoryOwner encoded = writer.Exit(); + Av1SymbolDecoder decoder = new(Configuration.Default, encoded.Memory.Span, 0, updateCdf: true); + + foreach (bool value in expected) + { + Assert.Equal(value, ReadDecision(ref decoder, decision, context)); + } + } + + /// + /// Verifies that the reference-mode reader selects each of the five spatial-context distributions. + /// + /// The block reference-mode context. + [Theory] + [InlineData(0)] + [InlineData(1)] + [InlineData(2)] + [InlineData(3)] + [InlineData(4)] + public void ReferenceModeReaderUsesRequestedContext(int context) + { + bool[] expected = [false, true, true, false, true, false, false, true]; + Av1Distribution writerDistribution = Av1DefaultDistributions.CompInter[context]; + using Av1SymbolWriter writer = new(Configuration.Default, 8, updateCdf: true); + + foreach (bool value in expected) + { + writer.WriteSymbol(value, writerDistribution); + } + + using IMemoryOwner encoded = writer.Exit(); + Av1SymbolDecoder decoder = new(Configuration.Default, encoded.Memory.Span, 0, updateCdf: true); + + foreach (bool value in expected) + { + Assert.Equal(value, decoder.ReadIsCompoundReference(context)); + } + } + + /// + /// Verifies one-pass neighbor collection, compound-neighbor votes, clearing, and intra-neighbor exclusion. + /// + [Fact] + public void CollectNeighborReferenceCountsMatchesLibaom() + { + Av1BlockModeInfo above = CreateModeInfo(Av1ReferenceFrameType.Last, Av1ReferenceFrameType.None); + Av1BlockModeInfo left = CreateModeInfo(Av1ReferenceFrameType.Backward, Av1ReferenceFrameType.Alternate); + InlineArray8 referenceCountStorage = default; + Span referenceCounts = referenceCountStorage; + referenceCounts.Fill(7); + + Av1SymbolContextHelper.CollectNeighborReferenceCounts(above, left, referenceCounts); + + ReadOnlySpan expected = [0, 1, 0, 0, 0, 1, 0, 1]; + + for (int reference = 0; reference < referenceCounts.Length; reference++) + { + Assert.Equal(expected[reference], referenceCounts[reference]); + } + + Av1BlockModeInfo intra = CreateModeInfo(Av1ReferenceFrameType.Intra, Av1ReferenceFrameType.None); + Av1SymbolContextHelper.CollectNeighborReferenceCounts(intra, null, referenceCounts); + + for (int reference = 0; reference < referenceCounts.Length; reference++) + { + Assert.Equal((byte)0, referenceCounts[reference]); + } + } + + /// + /// Verifies that the six context functions aggregate the exact reference groups used by libaom. + /// + [Fact] + public void SingleReferenceContextsAggregateNormativeReferenceGroups() + { + InlineArray8 referenceCountStorage = default; + Span referenceCounts = referenceCountStorage; + referenceCounts[(int)Av1ReferenceFrameType.Last] = 5; + referenceCounts[(int)Av1ReferenceFrameType.Last2] = 1; + referenceCounts[(int)Av1ReferenceFrameType.Last3] = 2; + referenceCounts[(int)Av1ReferenceFrameType.Golden] = 2; + referenceCounts[(int)Av1ReferenceFrameType.Backward] = 3; + referenceCounts[(int)Av1ReferenceFrameType.Alternate2] = 3; + referenceCounts[(int)Av1ReferenceFrameType.Alternate] = 6; + + Assert.Equal(0, Av1SymbolContextHelper.GetSingleReferenceBackwardContext(referenceCounts)); + Assert.Equal(1, Av1SymbolContextHelper.GetSingleReferenceAlternateContext(referenceCounts)); + Assert.Equal(2, Av1SymbolContextHelper.GetSingleReferenceLast3OrGoldenContext(referenceCounts)); + Assert.Equal(2, Av1SymbolContextHelper.GetSingleReferenceLast2Context(referenceCounts)); + Assert.Equal(1, Av1SymbolContextHelper.GetSingleReferenceGoldenContext(referenceCounts)); + Assert.Equal(1, Av1SymbolContextHelper.GetSingleReferenceAlternate2Context(referenceCounts)); + } + + /// + /// Verifies every branch of libaom's five-state single-versus-compound reference-mode context. + /// + [Fact] + public void ReferenceModeContextMatchesLibaom() + { + Av1BlockModeInfo singleForward = CreateModeInfo(Av1ReferenceFrameType.Last, Av1ReferenceFrameType.None); + Av1BlockModeInfo singleBackward = CreateModeInfo(Av1ReferenceFrameType.Backward, Av1ReferenceFrameType.None); + Av1BlockModeInfo intra = CreateModeInfo(Av1ReferenceFrameType.Intra, Av1ReferenceFrameType.None); + Av1BlockModeInfo compound = CreateModeInfo(Av1ReferenceFrameType.Last, Av1ReferenceFrameType.Backward); + Av1BlockModeInfo secondCompound = CreateModeInfo(Av1ReferenceFrameType.Last2, Av1ReferenceFrameType.Alternate); + + Assert.Equal(1, Av1SymbolContextHelper.GetReferenceModeContext(null, null)); + Assert.Equal(0, Av1SymbolContextHelper.GetReferenceModeContext(singleForward, null)); + Assert.Equal(1, Av1SymbolContextHelper.GetReferenceModeContext(singleBackward, null)); + Assert.Equal(3, Av1SymbolContextHelper.GetReferenceModeContext(compound, null)); + Assert.Equal(0, Av1SymbolContextHelper.GetReferenceModeContext(singleForward, singleForward)); + Assert.Equal(1, Av1SymbolContextHelper.GetReferenceModeContext(singleForward, singleBackward)); + Assert.Equal(2, Av1SymbolContextHelper.GetReferenceModeContext(singleForward, compound)); + Assert.Equal(3, Av1SymbolContextHelper.GetReferenceModeContext(intra, compound)); + Assert.Equal(2, Av1SymbolContextHelper.GetReferenceModeContext(compound, singleForward)); + Assert.Equal(3, Av1SymbolContextHelper.GetReferenceModeContext(compound, singleBackward)); + Assert.Equal(4, Av1SymbolContextHelper.GetReferenceModeContext(compound, secondCompound)); + } + + /// + /// Verifies the tied, symbol-one-majority, and symbol-zero-majority context states. + /// + /// The votes for the forward branch represented by symbol zero. + /// The votes for the backward branch represented by symbol one. + /// The expected context. + [Theory] + [InlineData(1, 1, 1)] + [InlineData(1, 2, 0)] + [InlineData(2, 1, 2)] + public void SingleReferenceContextReflectsNeighborVoteBalance(byte forwardCount, byte backwardCount, int expected) + { + InlineArray8 referenceCountStorage = default; + Span referenceCounts = referenceCountStorage; + referenceCounts[(int)Av1ReferenceFrameType.Last] = forwardCount; + referenceCounts[(int)Av1ReferenceFrameType.Backward] = backwardCount; + + int actual = Av1SymbolContextHelper.GetSingleReferenceBackwardContext(referenceCounts); + + Assert.Equal(expected, actual); + } + + /// + /// Verifies that frame-context copies retain reference-selection adaptation without sharing mutable distributions. + /// + [Fact] + public void FrameEntropyCopyRetainsIndependentReferenceSelectionState() + { + Av1FrameEntropyContext source = new(0); + Av1FrameEntropyContext destination = new(0); + source.SingleReference[2][5].Update(1); + source.CompInter[4].Update(1); + + destination.CopyFrom(source); + + Assert.Equal(source.SingleReference[2][5][0], destination.SingleReference[2][5][0]); + Assert.Equal(source.CompInter[4][0], destination.CompInter[4][0]); + + source.SingleReference[2][5].Update(0); + source.CompInter[4].Update(0); + + Assert.NotEqual(source.SingleReference[2][5][0], destination.SingleReference[2][5][0]); + Assert.NotEqual(source.CompInter[4][0], destination.CompInter[4][0]); + } + + /// + /// Verifies that publishing frame state resets the reference-selection distributions' update-rate history. + /// + [Fact] + public void FrameEntropySnapshotResetsReferenceSelectionUpdateCounts() + { + const int updateCount = 20; + Av1FrameEntropyContext source = new(0); + Av1FrameEntropyContext snapshot = new(0); + + for (int i = 0; i < updateCount; i++) + { + source.SingleReference[1][3].Update(1); + source.CompInter[2].Update(1); + } + + source.SnapshotTo(snapshot); + + Assert.Equal(source.SingleReference[1][3][0], snapshot.SingleReference[1][3][0]); + Assert.Equal(source.CompInter[2][0], snapshot.CompInter[2][0]); + + // The source retains twenty observations while the snapshot restarts at zero. The same next symbol therefore + // moves identical thresholds by different amounts only when the new distribution participates in reset. + source.SingleReference[1][3].Update(0); + snapshot.SingleReference[1][3].Update(0); + source.CompInter[2].Update(0); + snapshot.CompInter[2].Update(0); + + Assert.NotEqual(source.SingleReference[1][3][0], snapshot.SingleReference[1][3][0]); + Assert.NotEqual(source.CompInter[2][0], snapshot.CompInter[2][0]); + } + + /// + /// Provides every context and decision pairing in the single-reference distribution matrix. + /// + /// The eighteen context and decision combinations. + public static TheoryData GetReaderCases() + { + TheoryData result = []; + + for (int decision = 0; decision < 6; decision++) + { + for (int context = 0; context < 3; context++) + { + result.Add(decision, context); + } + } + + return result; + } + + /// + /// Reads one semantic single-reference decision through its production entry point. + /// + /// The tile symbol decoder. + /// The zero-based single-reference tree decision. + /// The neighboring reference-vote context. + /// The decoded binary decision. + private static bool ReadDecision(ref Av1SymbolDecoder decoder, int decision, int context) + => decision switch + { + 0 => decoder.ReadSingleReferenceIsBackward(context), + 1 => decoder.ReadSingleReferenceIsAlternate(context), + 2 => decoder.ReadSingleReferenceIsLast3OrGolden(context), + 3 => decoder.ReadSingleReferenceIsLast2(context), + 4 => decoder.ReadSingleReferenceIsGolden(context), + _ => decoder.ReadSingleReferenceIsAlternate2(context), + }; + + /// + /// Creates decoded block-mode state with the requested primary and secondary reference labels. + /// + /// The primary reference label. + /// The optional secondary reference label. + /// The initialized block mode state. + private static Av1BlockModeInfo CreateModeInfo(Av1ReferenceFrameType primary, Av1ReferenceFrameType secondary) + { + Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block4x4, Point.Empty); + modeInfo.ReferenceFrames[0] = primary; + modeInfo.ReferenceFrames[1] = secondary; + return modeInfo; + } +} diff --git a/tests/ImageSharp.Tests/TestImages.cs b/tests/ImageSharp.Tests/TestImages.cs index 45cdb3bc4..ada332e4a 100644 --- a/tests/ImageSharp.Tests/TestImages.cs +++ b/tests/ImageSharp.Tests/TestImages.cs @@ -1353,6 +1353,10 @@ public static class TestImages public const string Av1Deblocking12BitAvif = "Heif/Av1/Conformance/libavif-colors-12b.avif"; public const string Av1Deblocking12BitPayload = "Heif/Av1/Conformance/libaom-cosmos1650-12b.bit"; public const string Av1Deblocking12BitReference = "Heif/Av1/Conformance/libaom-cosmos1650-12b-libaom.yuv"; + public const string Av1Progressive8BitAvif = "Heif/Av1/Conformance/libavif-progressive-draw-points-8b.avif"; + public const string Av1Progressive8BitPayload = "Heif/Av1/Conformance/libavif-progressive-draw-points-8b.bit"; + public const string Av1Progressive8BitReference = "Heif/Av1/Conformance/libavif-progressive-draw-points-8b-libaom-y4m.yuv"; + public const string Av1Progressive8BitPresentationReference = "Heif/Av1/Conformance/libavif-progressive-draw-points-8b.png"; public const string Av1Cdef8BitPayload = "Heif/Av1/Conformance/libaom-cdef-kodim23-8b.bit"; public const string Av1Cdef8BitReference = "Heif/Av1/Conformance/libaom-cdef-kodim23-8b-libaom.yuv"; public const string Av1Cdef8BitAvif = "Heif/Av1/Conformance/libavif-cdef-kodim23-8b.avif"; diff --git a/tests/Images/Input/Heif/Av1/Conformance/README.md b/tests/Images/Input/Heif/Av1/Conformance/README.md index b2212809c..e51df3c32 100644 --- a/tests/Images/Input/Heif/Av1/Conformance/README.md +++ b/tests/Images/Input/Heif/Av1/Conformance/README.md @@ -1,135 +1,56 @@ # AV1 reconstruction conformance fixtures -The original AVIF and Y4M source files come from `libavif/tests/data` at commit `062e582e8afda88e6baf988fdcf046a801efa0f5`. Derived fixtures retain the licenses recorded in libavif's `tests/data/README.md`: the Kodak image is released for unrestricted use, the Cosmos Laundromat frame uses CC BY 3.0, and the libavif color animation is distributed with the libavif test corpus under its BSD-2-Clause license. +These fixtures provide independent reference output for AV1 reconstruction and AVIF presentation tests. ImageSharp output is compared exactly with the retained native YUV planes and presented PNG files; the tests do not use a tolerance. -The 8- and 10-bit `.bit` files contain the exact AV1 item payloads from the corresponding AVIF files. Each still file has one item occupying the complete `mdat` payload. The genuine 12-bit libavif sequence is retained for container, presentation, alpha, and metadata coverage, but its first color frame disables deblocking and therefore cannot prove the 12-bit filter path. +## Provenance -`libaom-cosmos1650-12b.bit` was encoded from libavif's real 10-bit 4:4:4 Cosmos Laundromat Y4M source with the pinned libaom encoder. libaom promotes the input samples to a 12-bit AV1 profile-2 still-picture stream. The constant-quality level is deliberately lossy so the frame signals nonzero loop-filter levels. The material command options were `--usage=2 --passes=1 --limit=1 --obu --bit-depth=12 --input-bit-depth=10 --profile=2 --end-usage=q --cq-level=30 --cpu-used=6 --threads=1 --lag-in-frames=0 --full-still-picture-hdr`. +The source images and original AVIF files come from `libavif/tests/data` at commit `062e582e8afda88e6baf988fdcf046a801efa0f5`. Their licenses are recorded in libavif's `tests/data/README.md` and continue to apply to the derived fixtures. This includes the unrestricted Kodak image, the CC BY 3.0 Cosmos Laundromat frame, and files distributed under libavif's BSD-2-Clause license. -The `_libaom.yuv` files were decoded from those exact payloads with `aomdec` built from libaom commit `03087864cf4bea6abb0d28f95cf7843511413d8f`. The reference build used `AOM_TARGET_CPU=generic`, so these files come from libaom's scalar decoder rather than ImageSharp or an architecture-specific implementation. +Reference files were generated with scalar builds of: -The `libaom-cdef-*` elementary streams were encoded separately with the same pinned generic libaom build so CDEF could be verified independently of the original corpus. The 8-bit stream uses `kodim23_yuv420_8bpc.y4m`; the 10- and 12-bit streams use `cosmos1650_yuv444_10bpc_p3pq.y4m`. Both source files are retained in libavif's test data at commit `062e582e8afda88e6baf988fdcf046a801efa0f5`. +- libaom commit `03087864cf4bea6abb0d28f95cf7843511413d8f`; +- libavif 1.4.2 from commit `062e582e8afda88e6baf988fdcf046a801efa0f5`, linked to that libaom build. -The material encoder options were `--usage=2 --passes=1 --limit=1 --obu --end-usage=q --cq-level=30 --cpu-used=4 --threads=1 --lag-in-frames=0 --full-still-picture-hdr --enable-cdef=1 --enable-restoration=0`. Each command also supplied the matching `--bit-depth`, `--input-bit-depth`, and `--profile` values. The 12-bit stream promotes the 10-bit 4:4:4 input through libaom's native 12-bit pipeline. Loop restoration is explicitly disabled so exact output equality exercises deblocking followed by active CDEF without a later restoration stage changing those samples. +The reference builds use `AOM_TARGET_CPU=generic` and disable libyuv. Native reconstruction therefore comes from libaom, and AVIF presentation comes from libavif's own conversion path, without architecture-specific SIMD or ImageSharp code. -The `libavif-cdef-*` AVIF files were independently encoded with `avifenc` 1.4.2 from libavif commit `062e582e8afda88e6baf988fdcf046a801efa0f5` and its pinned libaom 3.14.1 dependency. The material options were `-j 1 -s 4 -q 60`, `enable-cdef=1`, and `enable-restoration=0`. The 8-bit 4:2:0 file uses CICP 1/13/6 and the Kodak Y4M source. The 10-bit 4:4:4 file uses CICP 12/16/12 and the Cosmos Laundromat Y4M source. The 12-bit 4:4:4 input wraps the pinned 12-bit scalar-libaom reference planes as `C444p12` Y4M and also uses CICP 12/16/12. +## File conventions -The matching `.png` files were produced by `avifdec` from the same scalar build with `-j 1 -d 8`; the 8-bit 4:2:0 reference additionally selected bilinear chroma upsampling. The build uses `AOM_TARGET_CPU=generic` and `AVIF_LIBYUV=OFF`, so both AV1 reconstruction and YUV-to-RGB presentation come from the pinned scalar libaom/libavif paths. ImageSharp compares every presented RGBA byte exactly, without a tolerance. +- `.avif` files exercise the complete container and presentation path. +- `.bit` files contain the exact AV1 elementary-stream payload used by reconstruction tests. +- `-libaom.yuv` files contain headerless planar Y, U, and V reference samples. Samples above eight bits are stored as little-endian 16-bit values. +- `-libaom-y4m.yuv` files retain the Y4M header together with the native planar frame. +- `.png` files contain the eight-bit RGBA presentation reference produced by the pinned scalar libavif build. -The native reference layouts are: +## Coverage -- `libavif-kodim23-8b-libaom.yuv`: 768x512, 8-bit YUV 4:2:0, planar Y/U/V. -- `libavif-cosmos1650-10b-libaom.yuv`: 1024x428, 10-bit YUV 4:4:4, planar Y/U/V with little-endian 16-bit samples. -- `libaom-cosmos1650-12b-libaom.yuv`: 1024x428, 12-bit YUV 4:4:4, planar Y/U/V with little-endian 16-bit samples. -- `libaom-cdef-kodim23-8b-libaom.yuv`: 768x512, 8-bit YUV 4:2:0, planar Y/U/V. -- `libaom-cdef-cosmos-10b-libaom.yuv`: 1024x428, 10-bit YUV 4:4:4, planar Y/U/V with little-endian 16-bit samples. -- `libaom-cdef-cosmos-12b-libaom.yuv`: 1024x428, 12-bit YUV 4:4:4, planar Y/U/V with little-endian 16-bit samples. +| Fixture family | Coverage | +| --- | --- | +| `libavif-kodim23`, `libavif-cosmos1650`, `libaom-cosmos1650` | Baseline 8-, 10-, and 12-bit reconstruction, chroma subsampling, and active deblocking | +| `*-cdef-*` | Active CDEF with loop restoration disabled | +| `*-superres-*` | Active horizontal super-resolution with CDEF and restoration disabled | +| `*-restoration-*` | Wiener and self-guided loop restoration | +| `*-restoration-superres-*` | Restoration after super-resolution, including 10-bit 4:2:2 clipped-edge transform coverage | +| `libavif-profile-*` | The 8-, 10-, and 12-bit matrix across monochrome, 4:2:0, 4:2:2, and 4:4:4 | +| `*-palette-*` | Luma and chroma palette prediction | +| `*-intrabc-*` | Intra-block copy at every supported bit depth | +| `*-lossless-*` | Lossless quantization, reversible transforms, and exact presentation | +| `*-film-grain-*` | Full and restricted range, monochrome, identity matrix, 8/10/12-bit synthesis, overlap, and odd frame dimensions | +| `libavif-progressive-draw-points-8b` | A real two-layer color item whose final frame uses single-reference inter reconstruction, plus its progressive auxiliary alpha item | -The conformance tests compare every visible reconstructed sample with these files. The deblocking corpus verifies nonzero loop-filter levels. The CDEF corpus additionally verifies sequence-level CDEF enablement, a selected nonzero frame strength, and disabled loop restoration, so a disabled or bypassed CDEF stage cannot satisfy the exact native-plane comparison accidentally. +The corresponding tests also assert the syntax required by each family before comparing output. This prevents an inactive tool or an incorrectly substituted stream from passing solely because its final pixels happen to match. -Across the three active-CDEF elementary streams, the decoded mode records select every terminal AV1 partition shape. Their nested blocks also require recursive square splits, which produce no terminal mode record of their own. The tests verify that complete ten-type coverage before relying on the exact native-plane comparisons. +## Progressive dependent-frame fixture -The `libaom-superres-*` streams were encoded from the same Kodak and Cosmos sources with the pinned generic libaom build. Their material options were `--usage=2 --passes=1 --limit=1 --obu --end-usage=q --cq-level=30 --cpu-used=4 --threads=1 --lag-in-frames=0 --full-still-picture-hdr --enable-cdef=0 --enable-restoration=0 --superres-mode=1 --superres-denominator=12 --superres-kf-denominator=12`, together with the matching input depth, output depth, and profile. Disabling CDEF and restoration isolates the normative horizontal upscaling result, while the tests separately require a coded width smaller than the displayed width so an unscaled stream cannot satisfy the reference comparison. - -The matching `libaom-superres-*-libaom.yuv` files were decoded by `aomdec --rawvideo` from that exact generic build. They retain the displayed 768x512 8-bit YUV 4:2:0 and 1024x428 10/12-bit YUV 4:4:4 layouts described above. - -The `libavif-superres-*` containers retain the matching libavif-generated 8-, 10-, and 12-bit restoration container layouts described below. Each container's sole AV1 item was replaced mechanically with the corresponding active-super-resolution payload. Only the single `iloc` extent length and terminal `mdat` box size changed; the libavif-generated codec configuration, dimensions, CICP properties, item relationships, and remaining container layout were retained. - -The matching `libavif-superres-*.png` files were decoded from those exact containers with the pinned generic `avifdec -j 1 -d 8`; the 8-bit 4:2:0 reference additionally selected bilinear chroma upsampling. Tests decode the complete `mdat` payload to require a coded width smaller than the displayed width, then compare every presented RGBA byte with the scalar-libavif PNG exactly and without a tolerance. - -The `libaom-restoration-*` streams were encoded from the same Kodak and Cosmos sources with the pinned generic libaom build. Their material options were `--usage=2 --passes=1 --limit=1 --obu --end-usage=q --cq-level=30 --cpu-used=4 --threads=1 --lag-in-frames=0 --full-still-picture-hdr --enable-cdef=0 --enable-restoration=1 --superres-mode=0`, together with the matching input depth, output depth, and profile. The matching `*-libaom.yuv` files were decoded by that build's `aomdec --rawvideo` and retain the 768x512 8-bit YUV 4:2:0 and 1024x428 10/12-bit YUV 4:4:4 layouts. The tests require at least one signaled restoration unit and compare every resulting native sample exactly. - -The `libavif-restoration-*` container templates were encoded from the same sources with the pinned generic libavif build. Pinned libavif forcibly disables restoration for 12-bit libaom encoding, and its default all-intra settings did not select active restoration for the other templates. Each template's sole AV1 item was therefore replaced mechanically with the matching active-restoration payload above. Only the single `iloc` extent length and terminal `mdat` box size changed; the libavif-generated codec configuration, dimensions, CICP properties, item relationships, and remaining container layout were retained. - -The matching `libavif-restoration-*.png` files were decoded from those exact AVIF containers with the pinned generic `avifdec -j 1 -d 8`; the 8-bit 4:2:0 reference additionally selected bilinear chroma upsampling. The tests first decode each container's actual `mdat` payload to require both Wiener and self-guided unit selection, then compare every presented RGBA byte with the scalar-libavif PNG exactly and without a tolerance. - -The `libaom-restoration-superres-*` streams combine active restoration with a coded width reduced by super-resolution denominator 12. They use the same pinned generic libaom build and material encoder options as the restoration streams, with `--superres-mode=1 --superres-denominator=12 --superres-kf-denominator=12`. The 8-bit fixture is 768x512 YUV 4:2:0, the 10-bit fixture is 512x256 YUV 4:2:2, and the 12-bit fixture is 1024x428 YUV 4:4:4. Their matching `*-libaom.yuv` files were decoded from the exact payloads by the pinned generic `aomdec --rawvideo` build. - -The 10-bit 4:2:2 source was produced from libavif's `abc.png` with pinned generic `avifenc` using `-j 1 -s 8 -q 100 -d 10 -y 422`, then decoded to Y4M before the combined libaom encode. Its clipped rightmost 128x128 coding block crosses a second 64x64 residual region. This independently exercises the required conversion of the luma-region cursor to the subsampled chroma transform grid instead of relying only on full-width 4:4:4 blocks. - -## AV1 profile matrix - -The `libavif-profile-*` fixtures were generated from `tests/data/abc.png` at the pinned libavif revision. The source SHA-256 is `5561862FBD409A3F86B02DB73EBB8572D0E2A307EB45ECF9017A1B2137B9F729`; libavif's test-data manifest licenses it under the libavif license. Alpha was deliberately ignored so the matrix isolates the color planes. - -The tools were the retained generic `avifenc` and `avifdec` 1.4.2 builds linked to libaom 3.14.1 at commit `03087864cf4bea6abb0d28f95cf7843511413d8f`. The libaom build used `AOM_TARGET_CPU=generic` with its encoder and decoder enabled; its generated configuration disables AVX, AVX2, AVX-512, MMX, Neon, SSE, SSE2, SSE3, SSSE3, SSE4.1, and SSE4.2. The static libavif build used that `aom.lib`, disabled libyuv, and received `WITH_SIMD=OFF`, so the native and presentation references do not depend on ImageSharp or an architecture-specific decode path. - -The complete generation loop was: - -```powershell -foreach ($depth in 8, 10, 12) { - foreach ($format in 400, 420, 422, 444) { - $stem = "libavif-profile-${depth}b-${format}" - & $encoder -j 1 -s 6 -q 60 --ignore-alpha -d $depth -y $format --cicp 1/13/6 -a enable-palette=0 -a enable-intrabc=0 $source "$matrixDirectory\$stem.avif" - & $decoder -j 1 "$matrixDirectory\$stem.avif" "$matrixDirectory\$stem-libaom.y4m" - & $decoder -j 1 -d 8 "$matrixDirectory\$stem.avif" "$matrixDirectory\$stem.png" - } -} -``` - -The six subsampled PNG references were then regenerated with explicit bilinear chroma reconstruction: - -```powershell -foreach ($depth in 8, 10, 12) { - foreach ($format in 420, 422) { - $stem = "libavif-profile-${depth}b-${format}" - & $decoder -j 1 -d 8 -u bilinear "$matrixDirectory\$stem.avif" "$matrixDirectory\$stem.png" - } -} -``` - -Each AVIF is a lossy 512x256 opaque still image with full-range CICP 1/13/6 signaling and no ICC, XMP, or Exif payload. The retained Y4M output is a complete container decode with its native monochrome, 4:2:0, 4:2:2, or 4:4:4 header and 8-, 10-, or 12-bit planes. It is stored in the test corpus with `-libaom-y4m.yuv` replacing the generated `-libaom.y4m` suffix. SHA-256 comparison confirms that every committed AVIF, Y4M, and PNG is byte-identical to its retained generation artifact. - -## Palette coverage - -The palette fixture was encoded independently from ImageSharp using `tests/data/draw_points.png` from the pinned libavif revision. The source is a 33x11 flat-color image whose AV1 item selects both luma and chroma palette prediction. The pinned generic `avifenc` command used `-j 1 -s 0 -q 100 --ignore-alpha -y 444 --cicp 12/16/12 -a enable-palette=1 -a enable-intrabc=0 -a tune-content=screen`. - -`libaom-palette-draw-points-8b-444.bit` is the exact sole AV1 item extracted from `libavif-palette-draw-points-8b.avif`. The matching native YUV reference was decoded from that payload by the pinned scalar `aomdec --rawvideo` build. The presented PNG was decoded from the complete AVIF container by the pinned scalar `avifdec -j 1 -d 8` build. The tests require both palette planes to be selected, compare every native YUV sample exactly, and compare every presented RGBA byte exactly across the available vector widths and scalar fallback. No tolerance is used. - -## Intra-block-copy coverage - -The intra-block-copy fixtures were encoded independently from ImageSharp using `tests/data/abc.png` from the pinned libavif revision. This real 512x256 screen-content image provides repeated glyph and background regions beyond AV1's required 256-pixel reconstruction delay. Each AVIF is opaque YUV 4:4:4 with palette prediction disabled, so selected screen-content reuse must traverse intra-block-copy syntax and prediction rather than palette reconstruction. - -The common pinned generic `avifenc` options were `-j 1 -s 0 -l --ignore-alpha -y 444 -a enable-palette=0 -a enable-intrabc=1 -a tune-content=screen`. The 8-bit fixture uses `--cicp 1/13/0`; the 10- and 12-bit fixtures add the matching `-d` value and use `--cicp 12/16/0`. The high-depth encodes promote the 8-bit source, so `-l` configures lossless codec quantization but does not claim reversible conversion back to the original 8-bit PNG. - -The matching Y4M files were decoded from the complete AVIF containers with the pinned generic `avifdec -j 1` build. Their retained headers record the 512x256 full-range YUV 4:4:4 layouts at 8, 10, and 12 bits, followed by one planar frame. The matching PNG files were decoded with `avifdec -j 1 -d 8`. The build uses `AOM_TARGET_CPU=generic` and `AVIF_LIBYUV=OFF`, so the native planes and presented pixels come from the pinned scalar libaom/libavif paths. - -Tests require the frame header to allow intra-block copy and at least one final coding block to select it. They then compare every native Y, U, and V sample and every presented RGBA byte exactly under normal hardware dispatch, with AVX-512 disabled, with AVX disabled, and with all hardware intrinsics disabled. Displacement-vector entropy, spatial reference derivation, legal reconstruction order, inter transform selection, and prediction must therefore agree with the independent decoder for all three supported bit depths. No tolerance is used. - -## Lossless coverage - -The lossless fixtures were encoded independently from ImageSharp using `tests/data/circle-trns-after-plte.png` from the pinned libavif revision. Alpha was intentionally ignored so the native references isolate color-plane reconstruction. The 8-bit input uses CICP 1/13/0; the 10- and 12-bit YUV 4:4:4 inputs use CICP 12/16/0. The material `avifenc` options were `-j 1 -s 0 -l --ignore-alpha -y 444 -a enable-palette=0 -a enable-intrabc=0`, together with the matching depth and CICP values. Disabling palette and intra-block copy ensures the exact result traverses ordinary prediction, coefficient decoding, inverse quantization, and the reversible lossless transform. - -The `libavif-lossless-circle-*-444-libaom.yuv` files contain the headerless native planes decoded from the complete AVIF containers by the pinned generic `avifdec -j 1` build. Each file stores one 100x60 full-range YUV 4:4:4 frame at 8, 10, or 12 bits. The matching PNG files were decoded by the same build with `-d 8`. Tests require coded and complete losslessness, base quantizer zero, identity matrix coefficients, disabled palette and intra-block copy, and at least one coded residual. Every native Y, U, and V sample and every presented RGBA byte is compared exactly across normal hardware dispatch and the scalar fallback. No tolerance is used. - -## Film-grain coverage - -The film-grain pairs were generated independently from ImageSharp. Each `.bit` file is an AV1 still-picture OBU stream, and the matching `-libaom.yuv` file is the exact visible planar output from the pinned scalar libaom decoder. - -The source images are `tests/data/circle-trns-after-plte.png` and `tests/data/draw_points.png` from the pinned libavif revision above. The streams and native references use the same pinned libaom revision. Intermediate Y4M inputs were produced with libavif 1.4.2 linked to that libaom revision. - -| Stream | libaom vector | Native layout | Range | Covered behavior | -| --- | ---: | --- | --- | --- | -| `libaom-film-grain-circle-8b-420.bit` | 2 | 8-bit 4:2:0 | Full | Lag-three templates, boundary overlap, and independent luma and chroma scaling | -| `libaom-film-grain-circle-10b-422.bit` | 15 | 10-bit 4:2:2 | Full | Lag-two templates, boundary overlap, and chroma scaling derived from luma | -| `libaom-film-grain-circle-12b-444.bit` | 16 | 12-bit 4:4:4 | Full | Lag-three templates, boundary overlap, high-depth interpolation, and grain scale shift two | -| `libaom-film-grain-circle-8b-420-limited.bit` | 1 | 8-bit 4:2:0 | Restricted | Independent restricted luma and chroma endpoints | -| `libaom-film-grain-circle-8b-400-limited.bit` | 3 | 8-bit monochrome | Restricted | Monochrome synthesis, overlap, and restricted luma clipping | -| `libaom-film-grain-circle-12b-444-identity-limited.bit` | 14 | 12-bit 4:4:4 identity | Restricted | High-depth identity-matrix clipping, including luma endpoints for all three planes | -| `libaom-film-grain-draw-points-8b-420-odd.bit` | 2 | 8-bit 4:2:0, 33×11 | Full | Odd-width and odd-height extension, a partial final block, and overlap at the visible frame edge | - -The common libaom encoder options were: +The `libavif-progressive-draw-points-8b.avif` fixture is the unmodified `tests/data/draw_points_idat_progressive.avif` file from the pinned libavif tree. Its SHA-256 is `077AB2AD1E46DD912A973E4F024CB1EB242A08298BE2DBF1A52A058E88C48A4A`. It was generated with: ```text ---usage=2 --passes=1 --limit=1 --obu --end-usage=q --cq-level=30 --cpu-used=4 ---threads=1 --lag-in-frames=0 --full-still-picture-hdr --enable-cdef=0 --enable-restoration=0 +./avifenc -q 100 --progressive ../tests/data/draw_points.png ../tests/data/draw_points_idat_progressive.avif ``` -Each stream adds the bit depth, input bit depth, profile, monochrome or identity-matrix flag where applicable, and the `--film-grain-test` value shown above. The twelve-bit streams use a ten-bit Y4M input and `--bit-depth=12 --input-bit-depth=10`; this is the supported high-depth promotion path in the pinned generic aomenc build. +The primary color item's `a1lx` property divides its logical 72-byte AV1 payload into a 55-byte base layer and a 17-byte dependent layer. The container stores those layers in separate `iloc` extents at AVIF offsets 511 and 583. The `.bit` fixture concatenates those two logical color extents; it does not copy the physically adjacent auxiliary-alpha extent between them. -References were decoded with: +Exact pinned libaom decodes the corrected logical payload into two 33x11 YUV444 frames. Both frames' 1,089 color samples match the corresponding first three planes of the pinned libavif YUV444-alpha outputs exactly. The retained Y4M contains both progressive YUV444-alpha frames, and the PNG contains pinned libavif's final RGBA presentation. The production-path test selects the second native frame, requires inter-coded blocks in the final ImageSharp frame, and compares both native color and final presentation without a tolerance. -```text -aomdec --rawvideo --output=.yuv .bit -``` +## Updating fixtures -Tests compare every visible native Y, U, and V sample exactly. No tolerant image comparison is used. +Do not create conformance references with ImageSharp. Generate both the native-plane and presentation references with an independent decoder, record the exact upstream revisions and source license, and preserve exact comparisons. A new tool-specific fixture should demonstrate that the relevant syntax is active and should be no larger than required to cover that behavior. diff --git a/tests/Images/Input/Heif/Av1/Conformance/libavif-progressive-draw-points-8b-libaom-y4m.yuv b/tests/Images/Input/Heif/Av1/Conformance/libavif-progressive-draw-points-8b-libaom-y4m.yuv new file mode 100644 index 000000000..24494db84 --- /dev/null +++ b/tests/Images/Input/Heif/Av1/Conformance/libavif-progressive-draw-points-8b-libaom-y4m.yuv @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:cc70e15d31a25289492469bb637ca7e00f11de24889de5a9c6b9bd3f20be4b7a +size 2986 diff --git a/tests/Images/Input/Heif/Av1/Conformance/libavif-progressive-draw-points-8b.avif b/tests/Images/Input/Heif/Av1/Conformance/libavif-progressive-draw-points-8b.avif new file mode 100644 index 000000000..bc5d0a0d8 --- /dev/null +++ b/tests/Images/Input/Heif/Av1/Conformance/libavif-progressive-draw-points-8b.avif @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:077ab2ad1e46dd912a973e4f024cb1eb242a08298be2dbf1a52a058e88c48a4a +size 600 diff --git a/tests/Images/Input/Heif/Av1/Conformance/libavif-progressive-draw-points-8b.bit b/tests/Images/Input/Heif/Av1/Conformance/libavif-progressive-draw-points-8b.bit new file mode 100644 index 000000000..a6409ce40 --- /dev/null +++ b/tests/Images/Input/Heif/Av1/Conformance/libavif-progressive-draw-points-8b.bit @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:93ff341e1a7a4c849c3f713c4d20589fce93ddd7154106bce574106395e1498e +size 72 diff --git a/tests/Images/Input/Heif/Av1/Conformance/libavif-progressive-draw-points-8b.png b/tests/Images/Input/Heif/Av1/Conformance/libavif-progressive-draw-points-8b.png new file mode 100644 index 000000000..4e753f21b --- /dev/null +++ b/tests/Images/Input/Heif/Av1/Conformance/libavif-progressive-draw-points-8b.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:0758c17dc36e38aee9f4389a335c2bf332ab91e4c79d7b0b22994fddd0fd1605 +size 186