diff --git a/HEIF_IMPLEMENTATION_PLAN.md b/HEIF_IMPLEMENTATION_PLAN.md index ceabda57b..a92be5f56 100644 --- a/HEIF_IMPLEMENTATION_PLAN.md +++ b/HEIF_IMPLEMENTATION_PLAN.md @@ -29,7 +29,7 @@ Checkboxes may be marked complete only when the implementation and the verificat ## Delivery dashboard -Last reconciled with the source tree on 2026-08-28 against production checkpoint `096fb9af8` and the uncommitted single-reference inter-decoding work identified below. Committed checkpoints include the AV1 transform architecture, OBU framing, intra-block copy, 12-profile reconstruction matrix, layered-item properties, layered reference/header state, inter-frame intra blocks, SIMD-first translational prediction, and normalized self-guided-filter dispatch. The progressive fixture audit found that the initial `.bit` file appended a physically adjacent auxiliary-alpha extent instead of the color item's second `iloc` extent. The corrected 55-plus-17-byte logical color payload decodes as two frames with exact pinned libaom output, and the three production-path progressive tests now pass for native planes, final libavif presentation, constrained allocation, and decoder-result ownership. The complete focused Release matrix, documentation review, and final diff checks remain open. This dashboard is the authoritative delivery order. The detailed phase checklists below provide subsystem evidence; they do not override the current-stage marker or permit work to skip ahead. +Last reconciled with the source tree on 2026-08-29 against production checkpoint `8b9aa4b2cfe59ff0f8c348d9bb06678a5ff12f2a` and the verified equal-average compound work described below. Committed checkpoints include the AV1 transform architecture, OBU framing, intra-block copy, 12-profile reconstruction matrix, layered-item properties, layered reference/header/CDF/motion-field state, inter-frame intra blocks, SIMD-first translational prediction, and complete single-reference inter reconstruction. The current checkpoint adds compound reference trees and modes, paired reference-MV derivation, reference-dependent bounded sequence decoding, and allocation-free SIMD-first equal averaging. A pinned libavif/libaom 19-frame YUV444 sequence passes exact final native-plane and RGBA presentation comparison, production `DecodeBlock()` coverage, constrained allocation, and exactly-once allocator-return tracking. Both source target builds pass with zero warnings and errors; the test-project analyzer build passes with zero errors and only pre-existing repository warnings; 119 direct focused cases and three `FeatureTestRunner` cases pass with zero failures or skips; and `git diff --check` is clean. This dashboard is the authoritative delivery order. The detailed phase checklists below provide subsystem evidence; they do not override the current-stage marker or permit work to skip ahead. Status meanings: @@ -39,15 +39,15 @@ Status meanings: - **Not started:** supporting primitives may exist, but the production format path is absent. - **Current:** the only work item that should be advanced before taking the next queued item. -Current development stage: **Stage 3 — complete AV1 still-image decoding.** The committed decoder retains reference/header/CDF/motion-field state, derives frame-level skip-mode references, consumes temporal segment prediction, and decodes intra-coded blocks inside inter frames. The uncommitted working tree adds single-reference selection, spatial and temporal reference-MV derivation, NEAREST/NEAR/NEW/GLOBAL mode parsing, DRL, interpolation-filter syntax, motion-mode eligibility, and simple translational reconstruction before residual traversal. The exact dependent-frame oracle, constrained-allocation path, and decoder-result motion-field ownership test pass in Release; the complete focused Release matrix and final source/documentation review remain open. Compound prediction, inter-intra, OBMC, warped motion, scaled references, and non-translational global prediction remain explicitly unsupported. Neither AV1 nor HEVC production encoding is implemented. +Current development stage: **Stage 3 — complete AV1 still-image decoding.** The decoder retains reference/header/CDF/motion-field state, derives frame-level skip-mode references, consumes temporal segment prediction, decodes intra-coded blocks inside inter frames, reconstructs single-reference NEAREST/NEAR/NEW/GLOBAL translational blocks, and now reconstructs two retained predictors with equal-weight compound averaging before residual traversal. Exact dependent-frame and 19-frame compound-sequence native/presentation comparisons, constrained allocation, and ownership evidence pass. Selectable distance-weighted and masked compound blending, inter-intra prediction, OBMC, warped motion, scaled references, and non-translational global prediction remain explicitly unsupported. Neither AV1 nor HEVC production encoding is implemented. -Immediate checkpoint: **finish the source audit, independently verify, document, and commit the existing single-reference AV1 inter-block slice before implementing another codec feature.** Pinned libaom source confirms that interpolation syntax is omitted for an identity `GLOBALMV` block of sufficient size and that the spatial single-reference extension loops stop at two candidates; the current predicates match those two call paths. The corrected logical color payload and two-frame YUV444-alpha reference now prove the exact final native planes and libavif presentation through the complete production decoder, including constrained frame-plane allocation and decoder-result motion-field ownership. The complete focused Release verification and final source/documentation review remain required. This work remains inside the existing image-item and bounded image-sequence surfaces and must not expand into a general ISO BMFF/video model. +Immediate checkpoint: **implement and independently verify selected inter-intra prediction and selectable compound blending before advancing to OBMC.** Preserve the committed translational predictor and equal-average rounding contracts, consume each selection tree in normative order, keep mask and predictor workspaces allocator-owned and allocation-free per block, and prove each production branch with independently encoded bounded AVIF content and exact native-plane comparison. This work remains inside the existing image-item and bounded image-sequence surfaces and must not expand into a general ISO BMFF/video model. | Order | Delivery stage | State | Delivered state | Gate that remains open | | --- | --- | --- | --- | --- | | 1 | Baseline, provenance, documentation, and public contract | In progress | Pinned codec references, a bounded image-only scope, encoder options, typed bit depth, decoder-option propagation, and extensive HEIF documentation exist. | Complete the all-file documentation audit, record a fresh Release baseline, finish distinct public HEIC/AVIF save boundaries, and close API review. | | 2 | Bounded HEIF item and image-sequence container | In progress | Still-item parsing, grids, auxiliary alpha, metadata properties, bounded image-sequence tracks, Identify, and all-sync AV1 sequence presentation are connected. | Complete adversarial boundary coverage, remaining item/property behavior, reference-dependent sequence reconstruction, and the bounded sequence writer. | -| 3 | Still-image AV1 and HEVC decoding | **Current** | HEVC reconstruction reaches exact HM/libheif fixtures across the recorded 8/10/12-bit and chroma cases. Committed AV1 work includes bounded OBU framing, reconstruction, filters, grain, color, transforms, intra-block copy, an exact independent 12-profile bit-depth/chroma matrix through every dispatch tier, retained layered reference/header/CDF state, temporal segment prediction, inter-frame intra blocks, and SIMD-first translational prediction. The working tree contains an unverified simple single-reference inter path. | Close the simple single-reference checkpoint with exact dependent-frame evidence; implement compound, inter-intra, OBMC, scaled-reference, warped, and non-translational global prediction; remove every other valid AV1 still-image unsupported branch with independent vectors; then complete the remaining HEVC profile and Range Extensions matrix. | +| 3 | Still-image AV1 and HEVC decoding | **Current** | HEVC reconstruction reaches exact HM/libheif fixtures across the recorded 8/10/12-bit and chroma cases. AV1 work includes bounded OBU framing, reconstruction, filters, grain, color, transforms, intra-block copy, an exact independent 12-profile bit-depth/chroma matrix through every dispatch tier, retained layered reference/header/CDF/motion-field state, temporal segment prediction, inter-frame intra blocks, SIMD-first translational prediction, exact single-reference inter reconstruction, and exact equal-average compound reconstruction through a reference-dependent bounded sequence. | Implement inter-intra and selectable compound blending, then OBMC, scaled-reference, warped, and non-translational global prediction; remove every other valid AV1 still-image unsupported branch with independent vectors; then complete the remaining HEVC profile and Range Extensions matrix. | | 4 | Complete decoded presentation and animation | In progress | Shared SIMD-first AV1/HEVC color conversion, ICC application, grids, transforms, direct planar alpha composition, frame metadata, repetition, and independently decodable AV1 sequence samples exist. | Close the full color/ICC cross-product, HEVC sequence decoding, AV1/HEVC reference-dependent samples, frame-local metadata/alpha behavior, and independent animated decode vectors. | | 5 | AV1/AVIF encoding | Not started | RGB-to-planar conversion, forward transforms, OBU writer foundations, options, and container-writing infrastructure exist. | `HeifEncoderCore` still rejects AV1. Implement a real independently decodable lossy/lossless AV1 payload and the complete AVIF item/metadata matrix. | | 6 | HEVC/HEIC encoding | Not started | Shared input color conversion, options, and HEIF writer infrastructure exist. | `HeifEncoderCore` still rejects HEVC. Implement a real independently decodable lossy/lossless HEVC payload and the complete HEIC item/metadata matrix. | @@ -85,10 +85,13 @@ Immediate checkpoint: **finish the source audit, independently verify, document, - [x] Use counted frame/reference ownership for allocator-owned retained and temporal motion fields, with allocation tracking for initialization, retained-slot aliases, failure unwinding, presentation ownership, decoder-result ownership, and exactly-once final disposal. - [x] Request the existing contiguous ImageSharp allocation contract for every padded AV1 frame plane. Constrained-allocator coverage verifies complete-plane block reconstruction without copying or per-block allocation. - [x] Complete source review of the new motion-mode and single-reference tests. - - [x] Pass the exact current tree's Release verification: `net10.0` and `net11.0` source builds and the `net10.0` test-project analyzer build complete with zero warnings and errors; 293 focused `net10.0` entropy, candidate, motion, interpolation, lifecycle, reconstruction, ownership, and `FeatureTestRunner` cases pass with zero failures or skips; and `git diff --check` is clean. + - [x] Pass the exact single-reference checkpoint's Release verification: `net10.0` and `net11.0` source builds complete with zero warnings and errors; the `net10.0` test-project analyzer build completes with zero errors and pre-existing repository warnings; 293 focused `net10.0` entropy, candidate, motion, interpolation, lifecycle, reconstruction, ownership, and `FeatureTestRunner` cases pass with zero failures or skips; and `git diff --check` is clean. - [x] Correct the progressive dependent-frame extraction and compare the final frame's first three native planes with pinned libaom output and its final RGBA presentation with pinned libavif exactly. The unmodified AVIF has the recorded SHA-256 and stores the primary color item's 55-byte base extent at offset 511 and 17-byte dependent extent at offset 583. The corrected logical `.bit` payload decodes as two YUV444 frames with pinned libaom `03087864cf4bea6abb0d28f95cf7843511413d8f`; the retained two-frame YUV444-alpha reference and final PNG come from pinned libavif linked to that build. The production test selects the second native frame, requires inter-coded blocks, and passes exact native and presentation comparisons through `FeatureTestRunner`. - - [ ] Decode compound and inter-intra modes, masked blending, OBMC, scaled references, and warped and non-translational global-motion prediction without changing the single-reference predictor contract or rounding model. - - [ ] Verify every connected mode and filter with independently encoded dependent-layer AV1 image-item fixtures and exact native-plane comparisons. + - [x] Decode all compound reference-pair trees and inter modes, retain paired spatial/temporal reference-MV candidates through DRL selection, and reconstruct two translational predictors with allocation-free equal-weight averaging across 8/10/12-bit samples without changing the single-reference predictor contract or rounding model. + - [x] Verify the equal-average production branch with the pinned 19-frame libavif YUV444 sequence. The final native Y, U, and V planes match pinned libaom exactly, final RGBA presentation matches pinned libavif exactly, production `DecodeBlock()` is covered at 8/10/12 bit, constrained plane allocation passes, and allocator tracking returns every retained and temporal motion-field buffer exactly once. + - [ ] Decode selected inter-intra prediction and selectable distance-weighted, wedge, and difference-weighted compound blending. + - [ ] Decode OBMC, scaled references, warped motion, and non-translational global-motion prediction. + - [ ] Verify every remaining connected mode and filter with independently encoded bounded AV1 content and exact native-plane comparisons. - [ ] Return the explicitly selected spatial layer or the final displayed layer, keeping reference reconstruction separate from display-only film grain. - [ ] Verify color and auxiliary-alpha output exactly against both pinned libavif progressive fixtures under normal SIMD dispatch and all required `FeatureTestRunner` fallbacks. - [ ] Correct the audited 12-bit inverse ADST4, Identity4, and Identity16 SIMD arithmetic by widening only the libaom-widened multiply/accumulate operations, with exact conformant-range vectors and `FeatureTestRunner` coverage. @@ -349,7 +352,7 @@ This table is intentionally incomplete. Add a row before each additional AV1 or ## Current implementation assessment -This assessment was reconciled with the source tree on 2026-08-27, including production checkpoint `096fb9af8` and the explicitly identified uncommitted work. Unless a result is stated explicitly, each item is a source-inspection finding rather than a verified interoperability claim. +This assessment was reconciled with the source tree on 2026-08-29, including production checkpoint `8b9aa4b2cfe59ff0f8c348d9bb06678a5ff12f2a` and the verified equal-average compound checkpoint. Unless a result is stated explicitly, each item is a source-inspection finding rather than a verified interoperability claim. ### Public integration @@ -386,10 +389,11 @@ This assessment was reconciled with the source tree on 2026-08-27, including pro ### AV1 decoder -- The bounded `Av1Decoder` session parses all tile state before allocating and reconstructing each coded image layer. After successful completion it retains the ungrained reference planes, frame header, frame information, and published entropy snapshot in the refreshed slots, while presentation-only ownership remains separate. A new accepted sequence header resets both parser and retained-owner state. This is dependency reconstruction within one bounded image item; `show_existing_frame` playback remains rejected and no animation/video reference model is exposed. -- Committed inter-frame support reaches the intra-coded-block branch and provides SIMD-first translational prediction. The uncommitted working tree additionally parses single-reference selection, builds the fixed-capacity spatial and temporal reference-MV stack, decodes NEAREST/NEAR/NEW/GLOBAL and DRL syntax, decodes or infers interpolation filters, checks inter-intra and motion-mode eligibility, and invokes simple translational prediction before residual reconstruction. The exact corrected dependent-frame fixture now passes native-plane and final-presentation comparisons through the production decoder, but the slice is not delivered until the complete focused Release matrix, final review, and checkpoint commit pass. +- The bounded `Av1Decoder` session parses all tile state before allocating and reconstructing each coded image layer or sequence sample. After successful completion it retains the ungrained reference planes, frame header, frame information, and published entropy snapshot in the refreshed slots, while presentation-only ownership remains separate. A new accepted sequence header resets both parser and retained-owner state. Bounded image-sequence decoding accepts `show_existing_frame`, including its key-frame reference reset and separate film-grain presentation ownership, without exposing a general animation/video reference model. +- Committed inter-frame support parses single-reference selection, builds the fixed-capacity spatial and temporal reference-MV stack, decodes NEAREST/NEAR/NEW/GLOBAL and DRL syntax, decodes or infers interpolation filters, checks inter-intra and motion-mode eligibility, and invokes simple translational prediction before residual reconstruction. Commit `8b9aa4b2cfe59ff0f8c348d9bb06678a5ff12f2a` records exact corrected dependent-frame native-plane and final-presentation comparisons plus the complete focused Release matrix and ownership/allocation evidence. +- The verified equal-average compound checkpoint owns every compound reference-selection and inter-mode distribution through the retained CDF lifecycle, derives primary and secondary candidate vectors as inseparable pairs, reconstructs both retained references through the existing translational predictor, and combines them through one SIMD-first byte/ushort averaging operator with scalar parity. The pinned 19-frame libavif sequence exercises the complete production block branch and matches pinned libaom native planes and pinned libavif presentation exactly after decoding all preceding reference samples. - Source inspection against pinned libaom `03087864cf4bea6abb0d28f95cf7843511413d8f` resolves two disputed audit claims in favor of the current predicates. The interpolation call path uses `is_nontrans_global_motion()`, which returns false only for `TRANSLATION`, so an identity `GLOBALMV` block omits filter symbols. The single-reference spatial extension loops use `MAX_MV_REF_CANDIDATES`, which is two, while the full reference-MV stack capacity is eight. These loops are spatial extension, not temporal extension. -- The remaining single-reference audit issues are concrete and open. The working tree now requests contiguous allocation for all padded AV1 frame planes and adds constrained-allocator coverage, but that contract has not passed the complete focused matrix. Motion fields use allocator-owned storage and counted leases, with new allocation tracking for aliases, success ownership, failure unwinding, and exactly-once disposal; those tests are also unverified. The supplied progressive fixture cannot reach the production inter branch: ImageSharp and exact pinned libaom both reject the enhancement frame's nonzero byte-alignment padding, while the supplied one-frame Y4M's first three planes exactly match the separately decoded base layer. +- Every padded AV1 frame plane requests ImageSharp's contiguous allocation contract, and constrained-allocator reconstruction passes without copying. Motion fields use allocator-owned storage and counted leases; allocation tracking passes for initialization, retained aliases, failure unwinding, presentation ownership, decoder-result ownership, and exactly-once disposal. The corrected logical progressive color payload reaches the production inter branch and its final native planes match pinned libaom exactly. - Transform coefficient entropy derivation and updates now address the above contexts relative to the tile column and the left contexts relative to the current superblock row, preserve luma coordinates independently of chroma subsampling, and test every packed context entry for the libaom any-nonzero rule. Extended vertical partition updates advance the mode-information column rather than the row. The existing multi-superblock 4:4:4 AVIF fixture now completes tile parsing; independent coefficient-context vectors across tile boundaries, chroma layouts, bit depths, and edge-clipped transforms remain required. - The reconstruction pipeline now records plane-relative transform geometry, preserves tile-local delta-Q and delta-LF predictors, derives segmentation and reference-adjusted filter levels, and runs the exact AV1 4-, 6-, 8-, and 14-tap deblocking kernels in normative vertical-then-horizontal order. Deblocking uses the same closed edge-operator architecture as the HEVC filter, with operators specialized by sample storage and orientation, `Vector128` lanes representing the four rows or columns along an edge, and an allocation-free scalar fallback for disabled intrinsics. Exact native-plane comparison with pinned scalar libaom output now verifies active deblocking and complete reconstruction for real 8-bit 4:2:0, 10-bit 4:4:4, and 12-bit 4:4:4 content; genuine AVIF containers separately verify presentation and public bit-depth metadata. The pipeline then applies CDEF through one semantic filter architecture: paired AVX2 and single-block `Vector128` direction analysis, closed primary/secondary strength operators, packed 4x4/4x8/8x4/8x8 constrained filtering, byte/16-bit output operators, and an exact allocation-free scalar fallback. Decoder orchestration now owns the immutable plane snapshots and clean direction/variance maps through ImageSharp's memory allocator, widens 8-bit source rows with the same AVX2/128-bit/scalar tiers as libaom, lists each unit's non-skipped blocks in fixed inline storage, analyzes listed blocks in pairs, and writes filtered bytes or 16-bit samples directly to the frame planes. Independently encoded active-CDEF 8-bit 4:2:0 and 10/12-bit 4:4:4 streams match every visible native sample produced by pinned scalar libaom with restoration disabled. Independently encoded AVIF containers at the same three bit depths also match pinned scalar-libavif presentation exactly under normal, 256-bit, 128-bit, and scalar color-conversion dispatch. Active super-resolution derives the Appendix A bounded coded width and applies the exact 64-phase, 8-tap horizontal filter with aligned reconstruction-edge input, 8/10/12-bit clipping, and the existing cross-platform `Vector128_.MultiplyAddAdjacent` helper. Independently encoded active-super-resolution AV1 streams at 8, 10, and 12 bits match every visible native sample produced by pinned scalar libaom under normal and forced-scalar dispatch. Independently packaged AVIF containers at the same bit depths retain matching libavif profile, dimensions, chroma, and CICP properties, require active super-resolution in their actual AV1 item, and match pinned scalar-libavif presentation exactly under normal, 256-bit, 128-bit, and scalar color-conversion dispatch. Loop restoration follows super-resolution, preserves the required pre-CDEF deblocked context at internal stripes, and applies decoded Wiener or self-guided units from immutable plane snapshots. Independently encoded active-restoration streams at 8, 10, and 12 bits now match every native sample from pinned scalar libaom across AVX2, 128-bit, and scalar dispatch, with the fixture matrix proving both Wiener and self-guided unit selection. The matching AVIF containers also match pinned scalar-libavif presentation exactly, and an independent direct-window definition verifies all sixteen self-guided parameter sets at each supported bit depth across vector and scalar dispatch. Combined 8-bit 4:2:0, 10-bit 4:2:2, and 12-bit 4:4:4 streams additionally verify restoration-unit boundaries after super-resolution, including clipped chroma transform traversal at a coded-frame edge. - The visible still-image path applies the complete self-contained film-grain parameter set after all in-loop filters. Independently encoded pinned-libaom vectors match every native sample at 8, 10, and 12 bits across monochrome, 4:2:0, 4:2:2, and 4:4:4 layouts, full and restricted ranges, identity-matrix signaling, overlap, and odd 33x11 frame extension. `FeatureTestRunner` verifies normal, AVX-disabled, and fully scalar dispatch. A full-HD-equivalent 4:2:0 benchmark reports zero allocation: 8-bit AVX2 is 2.335 ms versus 5.806 ms scalar, while 12-bit AVX2 is 3.195 ms, cross-platform 128-bit is 7.382 ms, and scalar is 8.614 ms on the measured Ryzen platform. The slower 8-bit 128-bit lookup construction is deliberately not dispatched. @@ -418,7 +422,7 @@ This assessment was reconciled with the source tree on 2026-08-27, including pro ### Tests - HEVC coverage includes exact native-plane comparison with HM output, exact complete-image comparison with pinned libheif/libde265 references, and the 10 official Sony GENERAL Range Extensions first-picture fixtures across 8/10/12-bit monochrome, 4:2:0, 4:2:2, and 4:4:4 reconstruction. The remaining exposed profiles and individual Range Extensions tools still need exact independent vectors. -- AV1 has focused bitstream, prediction, entropy, reconstruction, filter, film-grain, color, and transform coverage, plus real libavif inputs. A real two-layer libavif-derived OBU stream verifies the bounded frame lifecycle, retained-slot occupancy, resolved inter references, and `frame_size_with_refs` dimensions through a fake tile lifecycle. A real palette stream truncated inside its tile entropy payload verifies libaom-equivalent overflow/trailing-bit rejection and decoder-session recovery. The last broad committed Release checkpoint passed all 2,422 selected entropy, ownership, reference, predictor, intra-block-copy, and transform cases. Commit `096fb9af8` separately passes the self-guided filter test under four `FeatureTestRunner` configurations. The 191-case `net10.0` result and zero-error `net10.0`/`net11.0` builds apply only to the pre-fixture tree. The exact current tree builds the focused `net10.0` test project, but its dependent-frame test fails before reconstruction; no current-tree claim inherits the older evidence. No test currently decodes a real dependent inter frame and compares its reconstructed pixels with libaom. Valid still-image syntax paths still contain explicit unsupported branches, so the independent AV1 decode matrix is not complete. +- AV1 has focused bitstream, prediction, entropy, reconstruction, filter, film-grain, color, and transform coverage, plus real libavif inputs. A real two-layer libavif-derived OBU stream verifies the bounded frame lifecycle, retained-slot occupancy, resolved inter references, and `frame_size_with_refs` dimensions through a fake tile lifecycle. The corrected progressive color item then verifies its production single-reference inter branch against exact pinned-libaom native planes and pinned-libavif presentation. The pinned 19-frame YUV444 sequence separately reaches production equal-average compound reconstruction after every preceding reference sample and matches its final native and presented references exactly. A real palette stream truncated inside its tile entropy payload verifies libaom-equivalent overflow/trailing-bit rejection and decoder-session recovery. The current compound checkpoint passes 119 direct focused cases plus three `FeatureTestRunner` cases, zero-warning `net10.0` and `net11.0` source builds, a zero-error test-project analyzer build with pre-existing repository warnings, allocator lifetime validation, and `git diff --check`. Valid still-image syntax paths still contain explicit unsupported branches, so the independent AV1 decode matrix is not complete. - The AV1 transform matrix verifies scalar, `Vector128`, `Vector256`, and `Vector512` forward representations plus the production inverse tiers across every valid size/type combination and supported bit depth. All 511 focused forward and inverse cases pass in Release; `FeatureTestRunner` isolates hardware tiers, every two-dimensional configuration exercises production dispatch, and the complete-block benchmark records zero managed allocation. - Independent libavif fixtures cover primary, grid, auxiliary-alpha, ICC, metadata-skipping, and all-sync image-sequence presentation. Reference-dependent AV1 and HEVC sequence reconstruction and independent HEVC ICC sequence coverage remain open. - Focused decoder-option tests cover strict, ancillary-only, image-data, and metadata-skipping behavior for still items and sequence samples. Complete adversarial dimension, allocation, malformed-container, and resource-limit coverage remains open. @@ -561,10 +565,13 @@ Implement and verify in dependency order: - [x] Store frame-sized retained and temporal motion fields in ImageSharp allocator-owned memory with deterministic counted disposal. Allocation tracking verifies aliases, success ownership, failure unwinding, presentation and decoder-result ownership, and exactly-once final disposal. - [x] Request the established contiguous allocation contract for every padded frame plane and verify complete-plane reconstruction with a constrained ImageSharp allocator without copying. - [x] Complete source review of the new motion-mode and single-reference tests. - - [x] Pass the exact current tree's Release verification. The `net10.0` and `net11.0` source builds and the `net10.0` test-project analyzer build complete with zero warnings and errors. All 293 focused `net10.0` syntax, CDF lifecycle, candidate, vector, interpolation, lifecycle, reconstruction, ownership, and `FeatureTestRunner` cases pass with zero failures or skips, including exact dependent-frame native-plane and presentation comparisons, constrained allocation, and motion-field lifetime coverage. `git diff --check` is clean. + - [x] Pass the exact single-reference checkpoint's Release verification. The `net10.0` and `net11.0` source builds complete with zero warnings and errors, while the `net10.0` test-project analyzer build completes with zero errors and pre-existing repository warnings. All 293 focused `net10.0` syntax, CDF lifecycle, candidate, vector, interpolation, lifecycle, reconstruction, ownership, and `FeatureTestRunner` cases pass with zero failures or skips, including exact dependent-frame native-plane and presentation comparisons, constrained allocation, and motion-field lifetime coverage. `git diff --check` is clean. - [x] Correct the logical progressive color payload and compare the final dependent frame's native planes and libavif presentation exactly. Pinned libaom decodes both layers from the primary item's two `iloc` extents, and pinned libavif produces the retained two-frame YUV444-alpha reference and final PNG. The exact production-path comparisons pass through `FeatureTestRunner`. - - [ ] Decode and reconstruct compound prediction, selected inter-intra prediction, masked blending, OBMC, scaled references, warped motion, and non-translational global motion through reconstructed reference planes. - - [ ] Verify every connected inter mode and filter with independently encoded dependent-layer AV1 image-item fixtures and exact native-plane comparisons. + - [x] Decode all compound reference trees and inter modes, derive and retain paired reference-MV candidates, reconstruct both unscaled translational references, and combine them with equal-weight averaging through `Vector512`, `Vector256`, `Vector128`, and scalar paths without per-block allocation. + - [x] Verify equal-average compound reconstruction with the pinned 19-frame libavif sequence, exact pinned-libaom final native planes, exact pinned-libavif final presentation, direct 8/10/12-bit production-block tests, constrained allocation, and exactly-once motion-field disposal. Both Release source targets pass with zero warnings and errors; the test-project analyzer build passes with zero errors and pre-existing repository warnings; 119 direct focused cases and three `FeatureTestRunner` cases pass without failures or skips; and `git diff --check` is clean. + - [ ] Decode and reconstruct selected inter-intra prediction and selectable distance-weighted, wedge, and difference-weighted compound blending through reconstructed reference planes. + - [ ] Decode and reconstruct OBMC, scaled references, warped motion, and non-translational global motion through reconstructed reference planes. + - [ ] Verify every remaining connected inter mode and filter with independently encoded bounded AV1 content and exact native-plane comparisons. - [ ] Lossless and high-bit-depth reconstruction with correct clipping and intermediate precision. - [x] Route lossless 4x4 blocks through allocation-free reversible inverse Walsh-Hadamard reconstruction for 8/10/12-bit samples, including the DC-only specialization, `Vector128` production traversal, scalar fallback, exact clipping, and `FeatureTestRunner` parity. - [x] Verify lossless syntax, inverse quantization, prediction, and presented reconstruction with independently encoded 8/10/12-bit AVIF fixtures. The tests require coded residuals with palette and intra-block copy disabled, compare every native YUV sample with the pinned generic libaom-backed decoder, and compare every presented RGBA byte with pinned generic libavif exactly under normal hardware dispatch and the scalar fallback. @@ -800,8 +807,9 @@ No valid HEVC or AV1 color, compression, or bit-depth row may remain `unsupporte The dashboard and immediate execution queue define the remaining critical path. In phase terms, work proceeds as follows: - [x] Finish the Phase 8 checkpoint for the implemented AV1 `Vector512` transforms, including Release, feature-isolation, and benchmark evidence. -- [x] Finish the complete focused Release matrix, final source/documentation review, and checkpoint implementation for the simple single-reference inter slice. The corrected real dependent-frame AVIF passes exact pinned-libaom native-plane and pinned-libavif presentation comparisons, and all 293 focused cases pass with zero failures or skips. -- [ ] Complete the remaining Phase 3 inter modes in dependency order: compound reference selection and averaging, inter-intra and masked blending, OBMC, scaled references, warped motion, and non-translational global prediction. Each mode requires an independent fixture before the next begins. +- [x] Finish the complete focused Release matrix, final source/documentation review, and checkpoint implementation for the simple single-reference inter slice. Commit `8b9aa4b2cfe59ff0f8c348d9bb06678a5ff12f2a` records the corrected real dependent-frame AVIF, exact pinned-libaom native-plane and pinned-libavif presentation comparisons, and 293 focused cases passing with zero failures or skips. +- [x] Implement compound reference selection and equal averaging with paired reference-MV derivation, allocation-free SIMD/scalar parity, and exact independent native/presentation evidence through a reference-dependent bounded AVIF sequence. +- [ ] **Current:** implement selected inter-intra prediction and selectable distance-weighted, wedge, and difference-weighted compound blending, then continue the remaining Phase 3 inter modes in dependency order: OBMC, scaled references, warped motion, and non-translational global prediction. Each mode requires independent exact evidence before the next begins. - [ ] Remove every other unsupported valid AV1 still-image syntax path, correct the audited 12-bit inverse-transform arithmetic, and prove the complete AVIF decode matrix with independent inputs and scalar/SIMD parity. - [ ] Close Phase 4 by completing the remaining HEVC profile and Range Extensions matrix with exact native-plane and presented-image evidence. - [ ] Close Phase 5 and the decode portion of the bounded sequence ledger: color, ICC, alpha, grids, presentation transforms, reference-dependent samples, and complete animated AVIF/HEIC decode. diff --git a/src/ImageSharp/Formats/Heif/Av1/Av1Decoder.cs b/src/ImageSharp/Formats/Heif/Av1/Av1Decoder.cs index 737651774..c964bd2fb 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Av1Decoder.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Av1Decoder.cs @@ -16,7 +16,7 @@ using SixLabors.ImageSharp.PixelFormats.Utils; namespace SixLabors.ImageSharp.Formats.Heif.Av1; /// -/// Decodes one bounded AV1 image payload into an ImageSharp image. +/// Decodes bounded AV1 image payloads and image-sequence samples into ImageSharp frames. /// internal sealed class Av1Decoder : IAv1TileReader, IDisposable { @@ -101,6 +101,11 @@ internal sealed class Av1Decoder : IAv1TileReader, IDisposable /// public Av1FrameInfo? FrameInfo { get; private set; } + /// + /// Gets the native planes of the current retained shown frame, or before one completes. + /// + public Av1FrameBuffer? FrameBuffer => this.referenceFrames.OutputFrame?.FrameBuffer; + /// /// Decodes a bounded AV1 image payload and presents its final shown frame. /// @@ -174,6 +179,101 @@ internal sealed class Av1Decoder : IAv1TileReader, IDisposable out effectiveColorProfile, layeredImageIndex); + return this.ConvertToFrame(frameBuffer, effectiveColorProfile); + } + + /// + /// Decodes the next visible sample in a bounded AV1 image sequence while retaining its reference state. + /// + /// The destination pixel type. + /// The complete AV1 sample payload. + /// The container color description. + /// The AV1 sample-entry configuration. + /// The independently owned decoded frame. + public ImageFrame DecodeSequenceFrame( + Span buffer, + CicpProfile? containerColorProfile, + Av1CodecConfiguration? codecConfiguration) + where TPixel : unmanaged, IPixel + { + CicpProfile effectiveColorProfile = this.DecodePayload( + buffer, + containerColorProfile, + codecConfiguration, + null, + requireShownFrame: true); + + return this.ConvertToFrame(this.referenceFrames.OutputFrame!.FrameBuffer, effectiveColorProfile); + } + + /// + /// Decodes one non-presented AV1 image-sequence sample while retaining its reference state. + /// + /// The complete AV1 sample payload. + /// The container color description. + /// The AV1 sample-entry configuration. + public void DecodeSequenceReference( + Span buffer, + CicpProfile? containerColorProfile, + Av1CodecConfiguration? codecConfiguration) + => _ = this.DecodePayload( + buffer, + containerColorProfile, + codecConfiguration, + null, + requireShownFrame: false); + + /// + /// Decodes the next visible monochrome AV1 sequence sample and composes it into a color frame. + /// + /// The destination color pixel type. + /// The complete AV1 sample payload. + /// The container color description. + /// The AV1 sample-entry configuration. + /// The required coded dimensions. + /// The packed color frame receiving alpha values. + /// The complete presented size of the auxiliary image. + /// The destination region receiving the alpha image. + /// Whether stored color samples must be converted to unassociated alpha. + public void DecodeSequenceAlpha( + Span buffer, + CicpProfile? containerColorProfile, + Av1CodecConfiguration? codecConfiguration, + Size expectedCodedSize, + ImageFrame destination, + Size outputSize, + Rectangle destinationRectangle, + bool premultiplied) + where TPixel : unmanaged, IPixel + { + _ = this.DecodePayload( + buffer, + containerColorProfile, + codecConfiguration, + null, + requireShownFrame: true); + + this.ComposeAlpha( + this.referenceFrames.OutputFrame!.FrameBuffer, + expectedCodedSize, + destination, + outputSize, + destinationRectangle, + premultiplied); + } + + /// + /// Converts native AV1 planes into one independently owned packed-pixel frame. + /// + /// The destination pixel type. + /// The decoded native planes. + /// The effective CICP description. + /// The independently owned packed-pixel frame. + private ImageFrame ConvertToFrame( + Av1FrameBuffer frameBuffer, + CicpProfile effectiveColorProfile) + where TPixel : unmanaged, IPixel + { ImageFrame? resultFrame = null; try { @@ -227,6 +327,34 @@ internal sealed class Av1Decoder : IAv1TileReader, IDisposable out _, layeredImageIndex); + this.ComposeAlpha( + frameBuffer, + expectedCodedSize, + destination, + outputSize, + destinationRectangle, + premultiplied); + } + + /// + /// Composes one decoded monochrome plane into a packed color frame. + /// + /// The destination color pixel type. + /// The decoded monochrome planes. + /// The required coded dimensions. + /// The packed color frame receiving alpha values. + /// The complete presented size of the auxiliary image. + /// The destination region receiving the alpha image. + /// Whether stored color samples must be converted to unassociated alpha. + private void ComposeAlpha( + Av1FrameBuffer frameBuffer, + Size expectedCodedSize, + ImageFrame destination, + Size outputSize, + Rectangle destinationRectangle, + bool premultiplied) + where TPixel : unmanaged, IPixel + { if (expectedCodedSize != default && (frameBuffer.Width != expectedCodedSize.Width || frameBuffer.Height != expectedCodedSize.Height)) { throw new InvalidImageContentException("The decoded alpha sample dimensions do not match its visual sample entry."); @@ -265,6 +393,33 @@ internal sealed class Av1Decoder : IAv1TileReader, IDisposable Av1CodecConfiguration? codecConfiguration, out CicpProfile effectiveColorProfile, Av1LayeredImageIndex? layeredImageIndex = null) + { + effectiveColorProfile = this.DecodePayload( + buffer, + containerColorProfile, + codecConfiguration, + layeredImageIndex, + requireShownFrame: true); + + using Av1ReferenceFrame outputFrame = this.referenceFrames.TakeOutput(); + return outputFrame.TakeFrameBuffer(); + } + + /// + /// Parses one bounded payload into the retained decoder session. + /// + /// The complete AV1 payload. + /// The container color description. + /// The AV1 codec configuration. + /// The optional layer byte boundaries. + /// Whether the payload must produce a shown frame. + /// The effective CICP description. + private CicpProfile DecodePayload( + Span buffer, + CicpProfile? containerColorProfile, + Av1CodecConfiguration? codecConfiguration, + Av1LayeredImageIndex? layeredImageIndex, + bool requireShownFrame) { this.codecConfiguration = codecConfiguration; this.containerColorProfile = containerColorProfile; @@ -312,21 +467,23 @@ internal sealed class Av1Decoder : IAv1TileReader, IDisposable } } - Guard.NotNull(this.referenceFrames.OutputFrame, nameof(this.referenceFrames.OutputFrame)); - Guard.NotNull(this.SequenceHeader, nameof(this.SequenceHeader)); - Guard.NotNull(this.FrameHeader, nameof(this.FrameHeader)); + ObuSequenceHeader sequenceHeader = this.obuReader.SequenceHeader!; + Guard.NotNull(sequenceHeader, nameof(sequenceHeader)); + if (requireShownFrame) + { + Guard.NotNull(this.referenceFrames.OutputFrame, nameof(this.referenceFrames.OutputFrame)); + Guard.NotNull(this.SequenceHeader, nameof(this.SequenceHeader)); + Guard.NotNull(this.FrameHeader, nameof(this.FrameHeader)); + } // Preserve the effective CICP description used for conversion, including container values that legally // supplied unspecified bitstream fields. This also exposes bitstream-only color metadata to callers. - ObuColorConfig effectiveColorConfig = this.SequenceHeader.ColorConfig; - effectiveColorProfile = new CicpProfile( + ObuColorConfig effectiveColorConfig = sequenceHeader.ColorConfig; + return new CicpProfile( (byte)effectiveColorConfig.ColorPrimaries, (byte)effectiveColorConfig.TransferCharacteristics, (byte)effectiveColorConfig.MatrixCoefficients, effectiveColorConfig.ColorRange); - - using Av1ReferenceFrame outputFrame = this.referenceFrames.TakeOutput(); - return outputFrame.TakeFrameBuffer(); } catch { @@ -416,7 +573,6 @@ internal sealed class Av1Decoder : IAv1TileReader, IDisposable /// public void CompleteFrame() { - Av1TileReader tileReader = this.tileReader!; ObuSequenceHeader sequenceHeader = this.obuReader.SequenceHeader!; ObuFrameHeader frameHeader = this.obuReader.FrameHeader!; Av1FrameBuffer? frameBuffer = null; @@ -479,6 +635,50 @@ internal sealed class Av1Decoder : IAv1TileReader, IDisposable this.validatedSequenceHeader = sequenceHeader; } + if (frameHeader.ShowExistingFrame) + { + Av1ReferenceFrame existingFrame = this.referenceFrames.ShowExisting((int)frameHeader.FrameToShowMapIdx); + ObuFrameHeader existingFrameHeader = existingFrame.FrameHeader; + Av1FrameInfo existingFrameInfo = existingFrame.FrameInfo; + + if (existingFrameHeader.FrameType == ObuFrameType.KeyFrame) + { + // libaom resets both the decoder working context and the context retained by the newly aliased key + // frame. Later primary-reference selection must therefore observe normative defaults. + existingFrame.ResetEntropyContext(); + this.entropyContexts?.Reset(); + } + + if (existingFrameHeader.FilmGrainParameters.ApplyGrain) + { + presentationBuffer = new Av1FrameBuffer( + this.configuration, + sequenceHeader, + sequenceHeader.ColorConfig.GetColorFormat(), + false); + + // Retained reference samples remain ungrained. Existing-frame presentation receives its own + // allocator-owned copy only when the inherited film-grain parameters actually modify the output. + existingFrame.FrameBuffer.CopyTo(presentationBuffer); + Av1FilmGrainDecoder filmGrainDecoder = new(sequenceHeader, existingFrameHeader, presentationBuffer); + filmGrainDecoder.DecodeFrame(); + + Av1ReferenceFrame presentationFrame = new(presentationBuffer, existingFrameHeader, existingFrameInfo); + presentationBuffer = null; + this.referenceFrames.CommitOutput(presentationFrame); + } + + this.SequenceHeader = sequenceHeader; + this.FrameHeader = existingFrameHeader; + + existingFrameInfo.AddOwner(); + this.FrameInfo?.ReleaseOwner(); + this.FrameInfo = existingFrameInfo; + return; + } + + Av1TileReader tileReader = this.tileReader!; + Av1FrameInfo frameInfo = tileReader.FrameInfo; frameBuffer = new Av1FrameBuffer( this.configuration, @@ -570,7 +770,7 @@ internal sealed class Av1Decoder : IAv1TileReader, IDisposable // alive independently after the entropy-neighbor contexts are returned. presentationBuffer?.Dispose(); frameBuffer?.Dispose(); - tileReader.Dispose(); + this.tileReader?.Dispose(); this.tileReader = null; } } diff --git a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1DefaultDistributions.cs b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1DefaultDistributions.cs index 13c6a51c1..4e594a779 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1DefaultDistributions.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1DefaultDistributions.cs @@ -59,6 +59,56 @@ internal static class Av1DefaultDistributions /// public static Av1Distribution[] CompInter => [new(26828), new(24035), new(12031), new(10640), new(2901)]; + /// + /// Gets the distributions that select unidirectional or bidirectional compound reference pairs. + /// + public static Av1Distribution[] CompoundReferenceType => [new(1198), new(2070), new(9166), new(7499), new(22475)]; + + /// + /// Gets the unidirectional compound-reference selection distributions indexed by spatial context and tree decision. + /// + public static Av1Distribution[][] UnidirectionalCompoundReference => + [ + [new(5284), new(3865), new(3128)], + [new(23152), new(14173), new(15270)], + [new(31774), new(25120), new(26710)], + ]; + + /// + /// Gets the bidirectional compound forward-reference distributions indexed by spatial context and tree decision. + /// + public static Av1Distribution[][] CompoundReference => + [ + [new(4946), new(9468), new(1503)], + [new(19891), new(22441), new(15160)], + [new(30731), new(31059), new(27544)], + ]; + + /// + /// Gets the bidirectional compound backward-reference distributions indexed by spatial context and tree decision. + /// + public static Av1Distribution[][] CompoundBackwardReference => + [ + [new(2235), new(1423)], + [new(17182), new(15175)], + [new(30606), new(30489)], + ]; + + /// + /// Gets the eight-symbol compound inter-mode distributions indexed by derived mode context. + /// + public static Av1Distribution[] InterCompoundMode => + [ + new(7760, 13823, 15808, 17641, 19156, 20666, 26891), + new(10730, 19452, 21145, 22749, 24039, 25131, 28724), + new(10664, 20221, 21588, 22906, 24295, 25387, 28436), + new(13298, 16984, 20471, 24182, 25067, 25736, 26422), + new(18904, 23325, 25242, 27432, 27898, 28258, 30758), + new(10725, 17454, 20124, 22820, 24195, 25168, 26046), + new(17125, 24273, 25814, 27492, 28214, 28704, 30592), + new(13046, 23214, 24505, 25942, 27435, 28442, 29330), + ]; + /// /// Gets the inter-intra prediction flag distributions indexed by block-size group. /// @@ -459,6 +509,16 @@ internal static class Av1DefaultDistributions [new(5782, 11475), new(5782, 11475), new(16803, 22759)], ]; + /// + /// Gets the binary variable-transform partition distributions indexed by transform geometry and neighbors. + /// + public static Av1Distribution[] TransformPartition => + [ + new(28581), new(23846), new(20847), new(24315), new(18196), new(12133), new(18791), + new(10887), new(11005), new(27179), new(20004), new(11281), new(26549), new(19308), + new(14224), new(28015), new(21546), new(14400), new(28165), new(22401), new(16088) + ]; + /// /// Gets the end-of-block token distributions for transforms containing at most 16 coefficients, indexed by /// quantizer, plane, and transform-class contexts. diff --git a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1FrameEntropyContext.cs b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1FrameEntropyContext.cs index 1d0b5cc48..ae3e910b2 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1FrameEntropyContext.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1FrameEntropyContext.cs @@ -89,6 +89,11 @@ internal sealed class Av1FrameEntropyContext this.Drl = Av1DefaultDistributions.Drl; this.SingleReference = Av1DefaultDistributions.SingleReference; this.CompInter = Av1DefaultDistributions.CompInter; + this.CompoundReferenceType = Av1DefaultDistributions.CompoundReferenceType; + this.UnidirectionalCompoundReference = Av1DefaultDistributions.UnidirectionalCompoundReference; + this.CompoundReference = Av1DefaultDistributions.CompoundReference; + this.CompoundBackwardReference = Av1DefaultDistributions.CompoundBackwardReference; + this.InterCompoundMode = Av1DefaultDistributions.InterCompoundMode; this.InterIntra = Av1DefaultDistributions.InterIntra; this.MotionMode = Av1DefaultDistributions.MotionMode; this.Obmc = Av1DefaultDistributions.Obmc; @@ -104,6 +109,7 @@ internal sealed class Av1FrameEntropyContext this.FilterIntraMode = Av1DefaultDistributions.FilterIntraMode; this.FilterIntra = Av1DefaultDistributions.FilterIntra; this.TransformSize = Av1DefaultDistributions.TransformSize; + this.TransformPartition = Av1DefaultDistributions.TransformPartition; this.ChromaFromLumaSign = Av1DefaultDistributions.ChromaFromLumaSign; this.ChromaFromLumaAlpha = Av1DefaultDistributions.ChromaFromLumaAlpha; this.IntraExtendedTransform = Av1DefaultDistributions.IntraExtendedTransform; @@ -152,6 +158,11 @@ internal sealed class Av1FrameEntropyContext this.Drl = Av1Distribution.CreateCopy(source.Drl); this.SingleReference = Av1Distribution.CreateCopy(source.SingleReference); this.CompInter = Av1Distribution.CreateCopy(source.CompInter); + this.CompoundReferenceType = Av1Distribution.CreateCopy(source.CompoundReferenceType); + this.UnidirectionalCompoundReference = Av1Distribution.CreateCopy(source.UnidirectionalCompoundReference); + this.CompoundReference = Av1Distribution.CreateCopy(source.CompoundReference); + this.CompoundBackwardReference = Av1Distribution.CreateCopy(source.CompoundBackwardReference); + this.InterCompoundMode = Av1Distribution.CreateCopy(source.InterCompoundMode); this.InterIntra = Av1Distribution.CreateCopy(source.InterIntra); this.MotionMode = Av1Distribution.CreateCopy(source.MotionMode); this.Obmc = Av1Distribution.CreateCopy(source.Obmc); @@ -167,6 +178,7 @@ internal sealed class Av1FrameEntropyContext this.FilterIntraMode = source.FilterIntraMode.CreateCopy(); this.FilterIntra = Av1Distribution.CreateCopy(source.FilterIntra); this.TransformSize = Av1Distribution.CreateCopy(source.TransformSize); + this.TransformPartition = Av1Distribution.CreateCopy(source.TransformPartition); this.EndOfBlockFlag = Av1Distribution.CreateCopy(source.EndOfBlockFlag); this.CoefficientsBase = Av1Distribution.CreateCopy(source.CoefficientsBase); this.BaseEndOfBlock = Av1Distribution.CreateCopy(source.BaseEndOfBlock); @@ -290,6 +302,31 @@ internal sealed class Av1FrameEntropyContext /// public Av1Distribution[] CompInter { get; } + /// + /// Gets the distributions that select unidirectional or bidirectional compound reference pairs. + /// + public Av1Distribution[] CompoundReferenceType { get; } + + /// + /// Gets the unidirectional compound-reference selection distributions. + /// + public Av1Distribution[][] UnidirectionalCompoundReference { get; } + + /// + /// Gets the bidirectional compound forward-reference selection distributions. + /// + public Av1Distribution[][] CompoundReference { get; } + + /// + /// Gets the bidirectional compound backward-reference selection distributions. + /// + public Av1Distribution[][] CompoundBackwardReference { get; } + + /// + /// Gets the compound inter-mode distributions indexed by derived mode context. + /// + public Av1Distribution[] InterCompoundMode { get; } + /// /// Gets the inter-intra prediction flag distributions indexed by block-size group. /// @@ -365,6 +402,11 @@ internal sealed class Av1FrameEntropyContext /// public Av1Distribution[][] TransformSize { get; } + /// + /// Gets the variable-transform partition distributions. + /// + public Av1Distribution[] TransformPartition { get; } + /// /// Gets the end-of-block token distributions selected for the frame base quantizer. /// @@ -475,6 +517,11 @@ internal sealed class Av1FrameEntropyContext CopyState(source.Drl, this.Drl); CopyState(source.SingleReference, this.SingleReference); CopyState(source.CompInter, this.CompInter); + CopyState(source.CompoundReferenceType, this.CompoundReferenceType); + CopyState(source.UnidirectionalCompoundReference, this.UnidirectionalCompoundReference); + CopyState(source.CompoundReference, this.CompoundReference); + CopyState(source.CompoundBackwardReference, this.CompoundBackwardReference); + CopyState(source.InterCompoundMode, this.InterCompoundMode); CopyState(source.InterIntra, this.InterIntra); CopyState(source.MotionMode, this.MotionMode); CopyState(source.Obmc, this.Obmc); @@ -490,6 +537,7 @@ internal sealed class Av1FrameEntropyContext this.FilterIntraMode.CopyFrom(source.FilterIntraMode); CopyState(source.FilterIntra, this.FilterIntra); CopyState(source.TransformSize, this.TransformSize); + CopyState(source.TransformPartition, this.TransformPartition); CopyState(source.EndOfBlockFlag, this.EndOfBlockFlag); CopyState(source.CoefficientsBase, this.CoefficientsBase); CopyState(source.BaseEndOfBlock, this.BaseEndOfBlock); @@ -544,6 +592,11 @@ internal sealed class Av1FrameEntropyContext ResetUpdateCounts(this.Drl); ResetUpdateCounts(this.SingleReference); ResetUpdateCounts(this.CompInter); + ResetUpdateCounts(this.CompoundReferenceType); + ResetUpdateCounts(this.UnidirectionalCompoundReference); + ResetUpdateCounts(this.CompoundReference); + ResetUpdateCounts(this.CompoundBackwardReference); + ResetUpdateCounts(this.InterCompoundMode); ResetUpdateCounts(this.InterIntra); ResetUpdateCounts(this.MotionMode); ResetUpdateCounts(this.Obmc); @@ -559,6 +612,7 @@ internal sealed class Av1FrameEntropyContext this.FilterIntraMode.ResetUpdateCount(); ResetUpdateCounts(this.FilterIntra); ResetUpdateCounts(this.TransformSize); + ResetUpdateCounts(this.TransformPartition); ResetUpdateCounts(this.EndOfBlockFlag); ResetUpdateCounts(this.CoefficientsBase); ResetUpdateCounts(this.BaseEndOfBlock); diff --git a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolContextHelper.cs b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolContextHelper.cs index 22aeb0c54..53aea3191 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolContextHelper.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolContextHelper.cs @@ -719,6 +719,73 @@ internal static class Av1SymbolContextHelper return 1; } + /// + /// Gets the context that selects a unidirectional or bidirectional compound reference pair. + /// + /// The above block, or at a tile boundary. + /// The left block, or at a tile boundary. + /// The context in the inclusive range zero through four. + public static int GetCompoundReferenceTypeContext(Av1BlockModeInfo? above, Av1BlockModeInfo? left) + { + if (above is not null && left is not null) + { + bool aboveIntra = !IsInterBlock(above); + bool leftIntra = !IsInterBlock(left); + if (aboveIntra && leftIntra) + { + return 2; + } + + if (aboveIntra || leftIntra) + { + Av1BlockModeInfo inter = aboveIntra ? left : above; + return HasCompoundReference(inter) ? 1 + (2 * (HasUnidirectionalCompoundReferences(inter) ? 1 : 0)) : 2; + } + + bool aboveSingle = !HasCompoundReference(above); + bool leftSingle = !HasCompoundReference(left); + Av1ReferenceFrameType abovePrimary = above.ReferenceFrames[0]; + Av1ReferenceFrameType leftPrimary = left.ReferenceFrames[0]; + if (aboveSingle && leftSingle) + { + return 1 + (2 * (IsBackwardReference(abovePrimary) == IsBackwardReference(leftPrimary) ? 1 : 0)); + } + + if (aboveSingle || leftSingle) + { + Av1BlockModeInfo compound = aboveSingle ? left : above; + if (!HasUnidirectionalCompoundReferences(compound)) + { + return 1; + } + + return 3 + (IsBackwardReference(abovePrimary) == IsBackwardReference(leftPrimary) ? 1 : 0); + } + + bool aboveUnidirectional = HasUnidirectionalCompoundReferences(above); + bool leftUnidirectional = HasUnidirectionalCompoundReferences(left); + if (!aboveUnidirectional && !leftUnidirectional) + { + return 0; + } + + if (!aboveUnidirectional || !leftUnidirectional) + { + return 2; + } + + return 3 + ((abovePrimary == Av1ReferenceFrameType.Backward) == (leftPrimary == Av1ReferenceFrameType.Backward) ? 1 : 0); + } + + Av1BlockModeInfo? edge = above ?? left; + if (edge is null || !IsInterBlock(edge) || !HasCompoundReference(edge)) + { + return 2; + } + + return HasUnidirectionalCompoundReferences(edge) ? 4 : 0; + } + /// /// Gets the switchable interpolation-filter context for one prediction direction. /// @@ -786,6 +853,25 @@ internal static class Av1SymbolContextHelper /// For a valid packed mode context, the context in the inclusive range zero through five. public static int GetRefMvContext(int modeContext) => (modeContext >> RefMvContextOffset) & RefMvContextMask; + /// + /// Maps the packed paired-reference candidate context to one of the eight compound inter-mode distributions. + /// + /// The packed mode context produced by paired reference-motion-vector analysis. + /// The compound inter-mode context in the inclusive range zero through seven. + public static int GetCompoundModeContext(int modeContext) + { + ReadOnlySpan contextMap = + [ + 0, 1, 1, 1, 1, + 1, 2, 3, 4, 4, + 4, 4, 5, 6, 7, + ]; + + int newMvContext = Math.Min(GetNewMvContext(modeContext), 4); + int referenceContextGroup = GetRefMvContext(modeContext) >> 1; + return contextMap[(referenceContextGroup * 5) + newMvContext]; + } + /// /// Gets the dynamic reference-list context for two adjacent motion-vector candidates. /// @@ -924,6 +1010,60 @@ internal static class Av1SymbolContextHelper return GetBinaryReferenceContext(backwardCount, alternate2Count); } + /// + /// Gets the first unidirectional compound-reference decision context. + /// + public static int GetUnidirectionalCompoundBackwardContext(ReadOnlySpan referenceCounts) + => GetSingleReferenceBackwardContext(referenceCounts); + + /// + /// Gets the context that selects Last3 or Golden instead of Last2 for a forward unidirectional pair. + /// + public static int GetUnidirectionalCompoundLast3OrGoldenContext(ReadOnlySpan referenceCounts) + { + int last2Count = referenceCounts[(int)Av1ReferenceFrameType.Last2]; + int last3OrGoldenCount = referenceCounts[(int)Av1ReferenceFrameType.Last3] + + referenceCounts[(int)Av1ReferenceFrameType.Golden]; + + return GetBinaryReferenceContext(last2Count, last3OrGoldenCount); + } + + /// + /// Gets the context that selects Golden instead of Last3 for a forward unidirectional pair. + /// + public static int GetUnidirectionalCompoundGoldenContext(ReadOnlySpan referenceCounts) + => GetSingleReferenceGoldenContext(referenceCounts); + + /// + /// Gets the context that selects Last3 or Golden instead of Last or Last2 for a bidirectional pair. + /// + public static int GetCompoundForwardLast3OrGoldenContext(ReadOnlySpan referenceCounts) + => GetSingleReferenceLast3OrGoldenContext(referenceCounts); + + /// + /// Gets the context that selects Last2 instead of Last for a bidirectional pair. + /// + public static int GetCompoundForwardLast2Context(ReadOnlySpan referenceCounts) + => GetSingleReferenceLast2Context(referenceCounts); + + /// + /// Gets the context that selects Golden instead of Last3 for a bidirectional pair. + /// + public static int GetCompoundForwardGoldenContext(ReadOnlySpan referenceCounts) + => GetSingleReferenceGoldenContext(referenceCounts); + + /// + /// Gets the context that selects Alternate instead of Backward or Alternate2 for a bidirectional pair. + /// + public static int GetCompoundBackwardAlternateContext(ReadOnlySpan referenceCounts) + => GetSingleReferenceAlternateContext(referenceCounts); + + /// + /// Gets the context that selects Alternate2 instead of Backward for a bidirectional pair. + /// + public static int GetCompoundBackwardAlternate2Context(ReadOnlySpan referenceCounts) + => GetSingleReferenceAlternate2Context(referenceCounts); + /// /// Gets the temporal segment-prediction context from the immediately above and left blocks. /// @@ -1053,6 +1193,30 @@ internal static class Av1SymbolContextHelper private static int GetBinaryReferenceContext(int zeroSymbolCount, int oneSymbolCount) => zeroSymbolCount == oneSymbolCount ? 1 : zeroSymbolCount < oneSymbolCount ? 0 : 2; + /// + /// Determines whether a decoded block uses an inter reference. + /// + private static bool IsInterBlock(Av1BlockModeInfo modeInfo) + => modeInfo.ReferenceFrames[0] >= Av1ReferenceFrameType.Last; + + /// + /// Determines whether a decoded block has a second inter reference. + /// + private static bool HasCompoundReference(Av1BlockModeInfo modeInfo) + => modeInfo.ReferenceFrames[1] > Av1ReferenceFrameType.Intra; + + /// + /// Determines whether both compound references point in the same display-order direction. + /// + private static bool HasUnidirectionalCompoundReferences(Av1BlockModeInfo modeInfo) + => IsBackwardReference(modeInfo.ReferenceFrames[0]) == IsBackwardReference(modeInfo.ReferenceFrames[1]); + + /// + /// Determines whether a retained reference belongs to the backward group. + /// + private static bool IsBackwardReference(Av1ReferenceFrameType referenceFrame) + => referenceFrame >= Av1ReferenceFrameType.Backward; + /// /// Gets one neighbor's interpolation-filter contribution for the requested reference and direction. /// diff --git a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolDecoder.cs b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolDecoder.cs index cf02b6f4f..7f9f94f96 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolDecoder.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolDecoder.cs @@ -427,6 +427,57 @@ internal ref struct Av1SymbolDecoder return r.ReadSymbol(this.context.CompInter[context]) != 0; } + /// + /// Reads whether a compound block uses one forward and one backward reference. + /// + /// The compound reference-type context in the inclusive range zero through four. + /// for a bidirectional pair; otherwise, . + public bool ReadCompoundReferenceIsBidirectional(int context) + { + ref Av1SymbolReader r = ref this.reader; + + return r.ReadSymbol(this.context.CompoundReferenceType[context]) != 0; + } + + /// + /// Reads one decision from the unidirectional compound-reference tree. + /// + /// The neighboring reference-vote context. + /// The zero-based tree decision in the inclusive range zero through two. + /// The decoded binary decision. + public bool ReadUnidirectionalCompoundReference(int context, int decision) + { + ref Av1SymbolReader r = ref this.reader; + + return r.ReadSymbol(this.context.UnidirectionalCompoundReference[context][decision]) != 0; + } + + /// + /// Reads one decision from the bidirectional compound forward-reference tree. + /// + /// The neighboring reference-vote context. + /// The zero-based tree decision in the inclusive range zero through two. + /// The decoded binary decision. + public bool ReadCompoundForwardReference(int context, int decision) + { + ref Av1SymbolReader r = ref this.reader; + + return r.ReadSymbol(this.context.CompoundReference[context][decision]) != 0; + } + + /// + /// Reads one decision from the bidirectional compound backward-reference tree. + /// + /// The neighboring reference-vote context. + /// The zero-based tree decision in the inclusive range zero through one. + /// The decoded binary decision. + public bool ReadCompoundBackwardReference(int context, int decision) + { + ref Av1SymbolReader r = ref this.reader; + + return r.ReadSymbol(this.context.CompoundBackwardReference[context][decision]) != 0; + } + /// /// Reads one per-block interpolation filter selected by a switchable frame. /// @@ -470,6 +521,20 @@ internal ref struct Av1SymbolDecoder : Av1PredictionMode.NearMotionVector; } + /// + /// Reads the prediction mode for a compound-reference inter block. + /// + /// The packed mode context produced by paired reference-motion-vector analysis. + /// The selected compound motion-vector mode. + public Av1PredictionMode ReadInterCompoundMode(int modeContext) + { + ref Av1SymbolReader r = ref this.reader; + int context = Av1SymbolContextHelper.GetCompoundModeContext(modeContext); + int mode = r.ReadSymbol(this.context.InterCompoundMode[context]); + + return (Av1PredictionMode)((int)Av1PredictionMode.NearestNearestMotionVector + mode); + } + /// /// Reads one dynamic reference-list decision for adjacent motion-vector candidates. /// @@ -708,6 +773,17 @@ internal ref struct Av1SymbolDecoder return transformSize; } + /// + /// Reads whether an inter transform node is subdivided. + /// + /// The variable-transform partition context. + /// when the transform node is split. + public bool ReadTransformPartition(int context) + { + ref Av1SymbolReader r = ref this.reader; + return r.ReadSymbol(this.context.TransformPartition[context]) > 0; + } + /// /// Reads a transform type from the transform set permitted for the block. /// @@ -845,7 +921,7 @@ internal ref struct Av1SymbolDecoder bool isLossless, bool useReducedTransformSet, Av1TransformType lumaTransformType, - Av1TransformInfo transformInfo, + ref Av1TransformInfo transformInfo, int modeBlocksToRightEdge, int modeBlocksToBottomEdge, Span coefficientBuffer) @@ -874,12 +950,15 @@ internal ref struct Av1SymbolDecoder return 0; } + bool usesInterTransformSet = modeInfo.ReferenceFrames[0] >= Av1ReferenceFrameType.Last || modeInfo.UseIntraBlockCopy; if (plane == (int)Av1Plane.Y) { + // Transform-set selection follows the prediction class. Intra-block copy uses inter residual syntax even + // though its reference is the current frame; ordinary inter blocks are identified by their retained ref. transformInfo.Type = this.ReadTransformType( transformSize, useReducedTransformSet, - modeInfo.UseIntraBlockCopy, + usesInterTransformSet, modeInfo.UseFilterIntra, isLossless, modeInfo.FilterIntraMode, @@ -892,8 +971,9 @@ internal ref struct Av1SymbolDecoder isLossless, transformSize, lumaTransformType, - transformInfo, - useReducedTransformSet); + in transformInfo, + useReducedTransformSet, + usesInterTransformSet); Av1TransformClass transformClass = transformInfo.Type.ToClass(); Av1ScanOrder scanOrder = Av1ScanOrderConstants.GetScanOrder(transformSize, transformInfo.Type); ReadOnlySpan scan = scanOrder.Scan; @@ -1295,6 +1375,7 @@ internal ref struct Av1SymbolDecoder /// The luma transform type shared by inter-predicted chroma. /// The transform descriptor containing the signaled luma type. /// Indicates whether the frame restricts transform choices. + /// Indicates whether prediction selects the inter transform set. /// The transform type valid for the current plane. private static Av1TransformType ComputeTransformType( Av1PlaneType planeType, @@ -1302,8 +1383,9 @@ internal ref struct Av1SymbolDecoder bool isLossless, Av1TransformSize transformSize, Av1TransformType lumaTransformType, - Av1TransformInfo transformInfo, - bool useReducedTransformSet) + in Av1TransformInfo transformInfo, + bool useReducedTransformSet, + bool usesInterTransformSet) { Av1TransformType transformType = Av1TransformType.DctDct; if (isLossless || transformSize.GetSquareUpSize() > Av1TransformSize.Size32x32) @@ -1316,10 +1398,10 @@ internal ref struct Av1SymbolDecoder { transformType = transformInfo.Type; } - else if (modeInfo.UseIntraBlockCopy) + else if (usesInterTransformSet) { - // Intra-block copy follows inter transform rules, so chroma reuses the luma transform type at the - // corresponding luma-grid position rather than deriving a type from the DC chroma mode. + // Inter prediction, including intra-block copy, shares the luma transform type with chroma at the + // corresponding luma-grid position rather than deriving an implicit type from the chroma mode. transformType = lumaTransformType; } else @@ -1331,7 +1413,7 @@ internal ref struct Av1SymbolDecoder Av1TransformSetType transformSetType = Av1SymbolContextHelper.GetExtendedTransformSetType( transformSize, - modeInfo.UseIntraBlockCopy, + usesInterTransformSet, useReducedTransformSet); if (!transformType.IsExtendedSetUsed(transformSetType)) diff --git a/src/ImageSharp/Formats/Heif/Av1/Motion/Av1ReferenceMotionVectors.cs b/src/ImageSharp/Formats/Heif/Av1/Motion/Av1ReferenceMotionVectors.cs index b5e614771..32161f73a 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Motion/Av1ReferenceMotionVectors.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Motion/Av1ReferenceMotionVectors.cs @@ -9,7 +9,7 @@ using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; namespace SixLabors.ImageSharp.Formats.Heif.Av1.Motion; /// -/// Derives the weighted AV1 reference-motion-vector candidates for one single-reference inter block. +/// Derives the weighted AV1 reference-motion-vector candidates for one inter block. /// internal sealed class Av1ReferenceMotionVectors { @@ -48,6 +48,11 @@ internal sealed class Av1ReferenceMotionVectors /// private InlineArray8 candidates; + /// + /// Stores the secondary vector of each compound candidate. + /// + private InlineArray8 compoundCandidates; + /// /// Stores the accumulated spatial or temporal weight corresponding to each candidate. /// @@ -58,6 +63,11 @@ internal sealed class Av1ReferenceMotionVectors /// private InlineArray2 references; + /// + /// Stores the nearest and near secondary references for a compound block. + /// + private InlineArray2 compoundReferences; + /// /// Gets the number of valid entries in and . /// @@ -73,6 +83,11 @@ internal sealed class Av1ReferenceMotionVectors /// public ReadOnlySpan Candidates => this.candidates[..this.Count]; + /// + /// Gets the secondary vectors corresponding to for a compound block. + /// + public ReadOnlySpan CompoundCandidates => this.compoundCandidates[..this.Count]; + /// /// Gets the accumulated weight corresponding to each entry in . /// @@ -84,21 +99,23 @@ internal sealed class Av1ReferenceMotionVectors public Av1MotionVector Nearest => this.references[0]; /// - /// Derives all single-reference motion-vector candidates for the current block. + /// Derives all single- or compound-reference motion-vector candidates for the current block. /// /// The current block geometry and decoded spatial neighbors. /// The active tile boundaries. /// The frame-wide spatial map and projected temporal motion field. /// The sequence-level superblock and order-hint configuration. /// The frame-level global-motion and motion-vector precision configuration. - /// The canonical inter reference selected for the current block. + /// The primary canonical inter reference selected for the current block. + /// The secondary compound reference, or . public void Build( ref Av1PartitionInfo partitionInfo, Av1TileInfo tileInfo, Av1FrameInfo frameInfo, ObuSequenceHeader sequenceHeader, ObuFrameHeader frameHeader, - Av1ReferenceFrameType referenceFrame) + Av1ReferenceFrameType referenceFrame, + Av1ReferenceFrameType secondaryReferenceFrame = Av1ReferenceFrameType.None) { Av1BlockSize blockSize = partitionInfo.ModeInfo.BlockSize; int width = blockSize.Get4x4WideCount(); @@ -132,6 +149,18 @@ internal sealed class Av1ReferenceMotionVectors new Point(column, row), frameHeader.ForceIntegerMotionVector); + Av1GlobalMotionParameters secondaryGlobalMotion = default; + Av1MotionVector secondaryGlobalMotionVector = default; + if (secondaryReferenceFrame > Av1ReferenceFrameType.Intra) + { + secondaryGlobalMotion = frameHeader.GetGlobalMotionParameters()[(int)secondaryReferenceFrame - 1]; + secondaryGlobalMotionVector = secondaryGlobalMotion.GetMotionVector( + frameHeader.AllowHighPrecisionMotionVector, + blockSize, + new Point(column, row), + frameHeader.ForceIntegerMotionVector); + } + int processedRows = 0; int processedColumns = 0; int rowMatchCount = 0; @@ -145,8 +174,11 @@ internal sealed class Av1ReferenceMotionVectors this.ScanRow( ref partitionInfo, referenceFrame, + secondaryReferenceFrame, in globalMotion, + in secondaryGlobalMotion, globalMotionVector, + secondaryGlobalMotionVector, -1, maximumRowOffset, ref rowMatchCount, @@ -159,8 +191,11 @@ internal sealed class Av1ReferenceMotionVectors this.ScanColumn( ref partitionInfo, referenceFrame, + secondaryReferenceFrame, in globalMotion, + in secondaryGlobalMotion, globalMotionVector, + secondaryGlobalMotionVector, -1, maximumColumnOffset, ref columnMatchCount, @@ -174,8 +209,11 @@ internal sealed class Av1ReferenceMotionVectors ref partitionInfo, tileInfo, referenceFrame, + secondaryReferenceFrame, in globalMotion, + in secondaryGlobalMotion, globalMotionVector, + secondaryGlobalMotionVector, -1, width, ref rowMatchCount, @@ -198,7 +236,9 @@ internal sealed class Av1ReferenceMotionVectors sequenceHeader.OrderHintInfo, frameHeader, referenceFrame, - globalMotionVector); + secondaryReferenceFrame, + globalMotionVector, + secondaryGlobalMotionVector); } int ignoredNewMotionVectorCount = 0; @@ -209,8 +249,11 @@ internal sealed class Av1ReferenceMotionVectors ref partitionInfo, tileInfo, referenceFrame, + secondaryReferenceFrame, in globalMotion, + in secondaryGlobalMotion, globalMotionVector, + secondaryGlobalMotionVector, -1, -1, ref rowMatchCount, @@ -225,8 +268,11 @@ internal sealed class Av1ReferenceMotionVectors this.ScanRow( ref partitionInfo, referenceFrame, + secondaryReferenceFrame, in globalMotion, + in secondaryGlobalMotion, globalMotionVector, + secondaryGlobalMotionVector, rowOffset, maximumRowOffset, ref rowMatchCount, @@ -239,8 +285,11 @@ internal sealed class Av1ReferenceMotionVectors this.ScanColumn( ref partitionInfo, referenceFrame, + secondaryReferenceFrame, in globalMotion, + in secondaryGlobalMotion, globalMotionVector, + secondaryGlobalMotionVector, columnOffset, maximumColumnOffset, ref columnMatchCount, @@ -270,20 +319,39 @@ internal sealed class Av1ReferenceMotionVectors int modeInfoHeight = Math.Min(Math.Min(MaximumSearchBlockSize, height), frameHeight - row); int extensionLength = Math.Min(modeInfoWidth, modeInfoHeight); - // When the direct stack has fewer than two entries, AV1 extends it from every inter reference on the - // immediate above and left blocks. Opposite temporal directions are sign-reversed into the target role. - for (int index = 0; Math.Abs(maximumRowOffset) >= 1 && index < extensionLength && this.Count < 2;) + if (secondaryReferenceFrame > Av1ReferenceFrameType.Intra) { - Av1BlockModeInfo candidate = partitionInfo.SuperblockInfo.GetModeInfoAt(new Point(column + index, row - 1)); - this.AddExtensionCandidate(candidate, frameInfo, referenceFrame); - index += candidate.BlockSize.Get4x4WideCount(); + if (this.Count < 2) + { + this.ExtendCompoundStack( + ref partitionInfo, + frameInfo, + referenceFrame, + secondaryReferenceFrame, + globalMotionVector, + secondaryGlobalMotionVector, + Math.Abs(maximumRowOffset) >= 1, + Math.Abs(maximumColumnOffset) >= 1, + extensionLength); + } } - - for (int index = 0; Math.Abs(maximumColumnOffset) >= 1 && index < extensionLength && this.Count < 2;) + else { - Av1BlockModeInfo candidate = partitionInfo.SuperblockInfo.GetModeInfoAt(new Point(column - 1, row + index)); - this.AddExtensionCandidate(candidate, frameInfo, referenceFrame); - index += candidate.BlockSize.Get4x4HighCount(); + // When the direct single-reference stack has fewer than two entries, AV1 extends it from every inter + // reference on the immediate above and left blocks. Opposite temporal directions are sign-reversed. + for (int index = 0; Math.Abs(maximumRowOffset) >= 1 && index < extensionLength && this.Count < 2;) + { + Av1BlockModeInfo candidate = partitionInfo.SuperblockInfo.GetModeInfoAt(new Point(column + index, row - 1)); + this.AddExtensionCandidate(candidate, frameInfo, referenceFrame); + index += candidate.BlockSize.Get4x4WideCount(); + } + + for (int index = 0; Math.Abs(maximumColumnOffset) >= 1 && index < extensionLength && this.Count < 2;) + { + Av1BlockModeInfo candidate = partitionInfo.SuperblockInfo.GetModeInfoAt(new Point(column - 1, row + index)); + this.AddExtensionCandidate(candidate, frameInfo, referenceFrame); + index += candidate.BlockSize.Get4x4HighCount(); + } } for (int index = 0; index < this.Count; index++) @@ -295,6 +363,17 @@ internal sealed class Av1ReferenceMotionVectors partitionInfo.ModeBlockToRightEdge, partitionInfo.ModeBlockToTopEdge, partitionInfo.ModeBlockToBottomEdge); + + if (secondaryReferenceFrame > Av1ReferenceFrameType.Intra) + { + this.compoundCandidates[index] = this.compoundCandidates[index].ClampReference( + blockSize.GetWidth(), + blockSize.GetHeight(), + partitionInfo.ModeBlockToLeftEdge, + partitionInfo.ModeBlockToRightEdge, + partitionInfo.ModeBlockToTopEdge, + partitionInfo.ModeBlockToBottomEdge); + } } // The two-element reference list is separate from the full DRL stack. Missing entries use global motion, @@ -306,6 +385,17 @@ internal sealed class Av1ReferenceMotionVectors this.references[1] = (this.Count > 1 ? this.candidates[1] : globalMotionVector).LowerPrecision( frameHeader.AllowHighPrecisionMotionVector, frameHeader.ForceIntegerMotionVector); + + if (secondaryReferenceFrame > Av1ReferenceFrameType.Intra) + { + this.compoundReferences[0] = this.compoundCandidates[0].LowerPrecision( + frameHeader.AllowHighPrecisionMotionVector, + frameHeader.ForceIntegerMotionVector); + + this.compoundReferences[1] = this.compoundCandidates[1].LowerPrecision( + frameHeader.AllowHighPrecisionMotionVector, + frameHeader.ForceIntegerMotionVector); + } } /// @@ -324,13 +414,52 @@ internal sealed class Av1ReferenceMotionVectors public Av1MotionVector GetNewReference(int referenceMotionVectorIndex) => this.Count > 1 ? this.candidates[referenceMotionVectorIndex] : this.references[0]; + /// + /// Gets the nearest reference for one member of a compound pair. + /// + /// Zero for the primary reference or one for the secondary reference. + /// The precision-reduced nearest vector. + public Av1MotionVector GetCompoundNearestReference(int referenceIndex) + => referenceIndex == 0 ? this.references[0] : this.compoundReferences[0]; + + /// + /// Gets the near reference for one member of a compound pair. + /// + /// The decoded zero-based dynamic-reference-list index. + /// Zero for the primary reference or one for the secondary reference. + /// The precision-reduced near vector. + public Av1MotionVector GetCompoundNearReference(int referenceMotionVectorIndex, int referenceIndex) + { + if (referenceMotionVectorIndex == 0) + { + return referenceIndex == 0 ? this.references[1] : this.compoundReferences[1]; + } + + int candidateIndex = referenceMotionVectorIndex + 1; + return referenceIndex == 0 ? this.candidates[candidateIndex] : this.compoundCandidates[candidateIndex]; + } + + /// + /// Gets the differential reference for one member of a compound pair. + /// + /// The selected stack index. + /// Zero for the primary reference or one for the secondary reference. + /// The selected raw stack vector. + public Av1MotionVector GetCompoundNewReference(int referenceMotionVectorIndex, int referenceIndex) + => referenceIndex == 0 + ? this.candidates[referenceMotionVectorIndex] + : this.compoundCandidates[referenceMotionVectorIndex]; + /// /// Scans one spatial row using AV1's block-size-dependent steps and weights. /// /// The current block geometry and frame-wide mode map. /// The canonical inter reference selected for the current block. + /// The secondary compound reference, or . /// The selected reference's global-motion model. + /// The secondary reference's global-motion model. /// The selected reference's global-motion vector at the current block. + /// The secondary reference's global-motion vector at the current block. /// The signed row offset from the current block in 4x4 units. /// The farthest permitted row offset inside the tile. /// Accumulates matching reference labels found in this scan direction. @@ -339,8 +468,11 @@ internal sealed class Av1ReferenceMotionVectors private void ScanRow( ref Av1PartitionInfo partitionInfo, Av1ReferenceFrameType referenceFrame, + Av1ReferenceFrameType secondaryReferenceFrame, in Av1GlobalMotionParameters globalMotion, + in Av1GlobalMotionParameters secondaryGlobalMotion, Av1MotionVector globalMotionVector, + Av1MotionVector secondaryGlobalMotionVector, int rowOffset, int maximumRowOffset, ref int referenceMatchCount, @@ -359,7 +491,9 @@ internal sealed class Av1ReferenceMotionVectors } } - bool useFourUnitStep = width >= 4; + // The scan advances by at least one 16x16 mode-info region only when the current block reaches 64 pixels + // on this axis. Smaller blocks must visit narrow neighbors individually so none of their candidates vanish. + bool useFourUnitStep = width >= MaximumSearchBlockSize; for (int index = 0; index < end;) { Av1BlockModeInfo candidate = partitionInfo.SuperblockInfo.GetModeInfoAt( @@ -387,8 +521,11 @@ internal sealed class Av1ReferenceMotionVectors this.AddCandidate( candidate, referenceFrame, + secondaryReferenceFrame, in globalMotion, + in secondaryGlobalMotion, globalMotionVector, + secondaryGlobalMotionVector, length * weight, ref referenceMatchCount, ref newMotionVectorCount); @@ -402,8 +539,11 @@ internal sealed class Av1ReferenceMotionVectors /// /// The current block geometry and frame-wide mode map. /// The canonical inter reference selected for the current block. + /// The secondary compound reference, or . /// The selected reference's global-motion model. + /// The secondary reference's global-motion model. /// The selected reference's global-motion vector at the current block. + /// The secondary reference's global-motion vector at the current block. /// The signed column offset from the current block in 4x4 units. /// The farthest permitted column offset inside the tile. /// Accumulates matching reference labels found in this scan direction. @@ -412,8 +552,11 @@ internal sealed class Av1ReferenceMotionVectors private void ScanColumn( ref Av1PartitionInfo partitionInfo, Av1ReferenceFrameType referenceFrame, + Av1ReferenceFrameType secondaryReferenceFrame, in Av1GlobalMotionParameters globalMotion, + in Av1GlobalMotionParameters secondaryGlobalMotion, Av1MotionVector globalMotionVector, + Av1MotionVector secondaryGlobalMotionVector, int columnOffset, int maximumColumnOffset, ref int referenceMatchCount, @@ -432,7 +575,9 @@ internal sealed class Av1ReferenceMotionVectors } } - bool useFourUnitStep = height >= 4; + // The scan advances by at least one 16x16 mode-info region only when the current block reaches 64 pixels + // on this axis. Smaller blocks must visit narrow neighbors individually so none of their candidates vanish. + bool useFourUnitStep = height >= MaximumSearchBlockSize; for (int index = 0; index < end;) { Av1BlockModeInfo candidate = partitionInfo.SuperblockInfo.GetModeInfoAt( @@ -460,8 +605,11 @@ internal sealed class Av1ReferenceMotionVectors this.AddCandidate( candidate, referenceFrame, + secondaryReferenceFrame, in globalMotion, + in secondaryGlobalMotion, globalMotionVector, + secondaryGlobalMotionVector, length * weight, ref referenceMatchCount, ref newMotionVectorCount); @@ -476,8 +624,11 @@ internal sealed class Av1ReferenceMotionVectors /// The current block geometry and frame-wide mode map. /// The active tile boundaries. /// The canonical inter reference selected for the current block. + /// The secondary compound reference, or . /// The selected reference's global-motion model. + /// The secondary reference's global-motion model. /// The selected reference's global-motion vector at the current block. + /// The secondary reference's global-motion vector at the current block. /// The signed row offset from the current block in 4x4 units. /// The signed column offset from the current block in 4x4 units. /// Accumulates matching reference labels at the search position. @@ -486,8 +637,11 @@ internal sealed class Av1ReferenceMotionVectors ref Av1PartitionInfo partitionInfo, Av1TileInfo tileInfo, Av1ReferenceFrameType referenceFrame, + Av1ReferenceFrameType secondaryReferenceFrame, in Av1GlobalMotionParameters globalMotion, + in Av1GlobalMotionParameters secondaryGlobalMotion, Av1MotionVector globalMotionVector, + Av1MotionVector secondaryGlobalMotionVector, int rowOffset, int columnOffset, ref int referenceMatchCount, @@ -505,8 +659,11 @@ internal sealed class Av1ReferenceMotionVectors this.AddCandidate( candidate, referenceFrame, + secondaryReferenceFrame, in globalMotion, + in secondaryGlobalMotion, globalMotionVector, + secondaryGlobalMotionVector, 4, ref referenceMatchCount, ref newMotionVectorCount); @@ -517,16 +674,22 @@ internal sealed class Av1ReferenceMotionVectors /// /// The decoded neighboring block. /// The canonical inter reference selected for the current block. + /// The secondary compound reference, or . /// The selected reference's global-motion model. + /// The secondary reference's global-motion model. /// The selected reference's global-motion vector at the current block. + /// The secondary reference's global-motion vector at the current block. /// The spatial weight contributed by each matching reference. /// Accumulates matching reference labels in the active scan direction. /// Accumulates matching neighbors whose inter mode contains a new vector. private void AddCandidate( Av1BlockModeInfo candidate, Av1ReferenceFrameType referenceFrame, + Av1ReferenceFrameType secondaryReferenceFrame, in Av1GlobalMotionParameters globalMotion, + in Av1GlobalMotionParameters secondaryGlobalMotion, Av1MotionVector globalMotionVector, + Av1MotionVector secondaryGlobalMotionVector, int weight, ref int referenceMatchCount, ref int newMotionVectorCount) @@ -538,6 +701,41 @@ internal sealed class Av1ReferenceMotionVectors } Span candidateMotionVectors = candidate.MotionVectors; + if (secondaryReferenceFrame > Av1ReferenceFrameType.Intra) + { + if (candidateReferences[0] != referenceFrame || candidateReferences[1] != secondaryReferenceFrame) + { + return; + } + + bool usePrimaryGlobalMotion = + candidate.YMode == Av1PredictionMode.GlobalGlobalMotionVector && + globalMotion.Type > Av1GlobalMotionType.Translation && + Math.Min(candidate.BlockSize.GetWidth(), candidate.BlockSize.GetHeight()) >= 8; + + bool useSecondaryGlobalMotion = + candidate.YMode == Av1PredictionMode.GlobalGlobalMotionVector && + secondaryGlobalMotion.Type > Av1GlobalMotionType.Translation && + Math.Min(candidate.BlockSize.GetWidth(), candidate.BlockSize.GetHeight()) >= 8; + + Av1MotionVector primaryMotionVector = usePrimaryGlobalMotion + ? globalMotionVector + : candidateMotionVectors[0]; + + Av1MotionVector secondaryMotionVector = useSecondaryGlobalMotion + ? secondaryGlobalMotionVector + : candidateMotionVectors[1]; + + this.AddUnique(primaryMotionVector, secondaryMotionVector, weight); + if (UsesNewMotionVector(candidate.YMode)) + { + newMotionVectorCount++; + } + + referenceMatchCount++; + return; + } + for (int referenceIndex = 0; referenceIndex < 2; referenceIndex++) { if (candidateReferences[referenceIndex] != referenceFrame) @@ -561,12 +759,7 @@ internal sealed class Av1ReferenceMotionVectors // Every matching reference in a neighbor carrying a NEW component contributes to the adjacent NEWMV // context even when its vector deduplicates against an earlier stack entry. - if (candidate.YMode is Av1PredictionMode.NewMotionVector or - Av1PredictionMode.NewNewMotionVector or - Av1PredictionMode.NearestNewMotionVector or - Av1PredictionMode.NewNearestMotionVector or - Av1PredictionMode.NearNewMotionVector or - Av1PredictionMode.NewNearMotionVector) + if (UsesNewMotionVector(candidate.YMode)) { newMotionVectorCount++; } @@ -584,7 +777,9 @@ internal sealed class Av1ReferenceMotionVectors /// The sequence modulo order-hint configuration. /// The frame-level motion-vector precision configuration. /// The canonical inter reference selected for the current block. + /// The secondary compound reference, or . /// The selected reference's global-motion vector at the current block. + /// The secondary reference's global-motion vector at the current block. private void AddTemporalCandidates( ref Av1PartitionInfo partitionInfo, Av1TileInfo tileInfo, @@ -592,7 +787,9 @@ internal sealed class Av1ReferenceMotionVectors ObuOrderHintInfo orderHintInfo, ObuFrameHeader frameHeader, Av1ReferenceFrameType referenceFrame, - Av1MotionVector globalMotionVector) + Av1ReferenceFrameType secondaryReferenceFrame, + Av1MotionVector globalMotionVector, + Av1MotionVector secondaryGlobalMotionVector) { int width = partitionInfo.ModeInfo.BlockSize.Get4x4WideCount(); int height = partitionInfo.ModeInfo.BlockSize.Get4x4HighCount(); @@ -615,7 +812,9 @@ internal sealed class Av1ReferenceMotionVectors orderHintInfo, frameHeader, referenceFrame, + secondaryReferenceFrame, globalMotionVector, + secondaryGlobalMotionVector, blockRow, blockColumn); @@ -646,7 +845,9 @@ internal sealed class Av1ReferenceMotionVectors orderHintInfo, frameHeader, referenceFrame, + secondaryReferenceFrame, globalMotionVector, + secondaryGlobalMotionVector, verticalOffset, -2); @@ -657,7 +858,9 @@ internal sealed class Av1ReferenceMotionVectors orderHintInfo, frameHeader, referenceFrame, + secondaryReferenceFrame, globalMotionVector, + secondaryGlobalMotionVector, verticalOffset, horizontalOffset); @@ -668,7 +871,9 @@ internal sealed class Av1ReferenceMotionVectors orderHintInfo, frameHeader, referenceFrame, + secondaryReferenceFrame, globalMotionVector, + secondaryGlobalMotionVector, verticalOffset - 2, horizontalOffset); } @@ -682,7 +887,9 @@ internal sealed class Av1ReferenceMotionVectors /// The sequence modulo order-hint configuration. /// The frame-level motion-vector precision configuration. /// The canonical inter reference selected for the current block. + /// The secondary compound reference, or . /// The selected reference's global-motion vector at the current block. + /// The secondary reference's global-motion vector at the current block. /// The temporal sample row relative to the current block in 4x4 units. /// The temporal sample column relative to the current block in 4x4 units. private void AddTemporalExtension( @@ -692,7 +899,9 @@ internal sealed class Av1ReferenceMotionVectors ObuOrderHintInfo orderHintInfo, ObuFrameHeader frameHeader, Av1ReferenceFrameType referenceFrame, + Av1ReferenceFrameType secondaryReferenceFrame, Av1MotionVector globalMotionVector, + Av1MotionVector secondaryGlobalMotionVector, int blockRow, int blockColumn) { @@ -711,7 +920,9 @@ internal sealed class Av1ReferenceMotionVectors orderHintInfo, frameHeader, referenceFrame, + secondaryReferenceFrame, globalMotionVector, + secondaryGlobalMotionVector, blockRow, blockColumn); } @@ -725,7 +936,9 @@ internal sealed class Av1ReferenceMotionVectors /// The sequence modulo order-hint configuration. /// The frame-level motion-vector precision configuration. /// The canonical inter reference selected for the current block. + /// The secondary compound reference, or . /// The selected reference's global-motion vector at the current block. + /// The secondary reference's global-motion vector at the current block. /// The temporal sample row relative to the current block in 4x4 units. /// The temporal sample column relative to the current block in 4x4 units. /// when the projected motion field covers the requested position. @@ -736,7 +949,9 @@ internal sealed class Av1ReferenceMotionVectors ObuOrderHintInfo orderHintInfo, ObuFrameHeader frameHeader, Av1ReferenceFrameType referenceFrame, + Av1ReferenceFrameType secondaryReferenceFrame, Av1MotionVector globalMotionVector, + Av1MotionVector secondaryGlobalMotionVector, int blockRow, int blockColumn) { @@ -762,16 +977,41 @@ internal sealed class Av1ReferenceMotionVectors return false; } + Av1MotionVector secondaryMotionVector = default; + if (secondaryReferenceFrame > Av1ReferenceFrameType.Intra && + !frameInfo.TryGetProjectedTemporalMotionVector( + row, + column, + secondaryReferenceFrame, + orderHintInfo, + frameHeader.AllowHighPrecisionMotionVector, + frameHeader.ForceIntegerMotionVector, + out secondaryMotionVector)) + { + return false; + } + if (blockRow == 0 && blockColumn == 0 && (Math.Abs(motionVector.Row - globalMotionVector.Row) >= 16 || - Math.Abs(motionVector.Column - globalMotionVector.Column) >= 16)) + Math.Abs(motionVector.Column - globalMotionVector.Column) >= 16 || + (secondaryReferenceFrame > Av1ReferenceFrameType.Intra && + (Math.Abs(secondaryMotionVector.Row - secondaryGlobalMotionVector.Row) >= 16 || + Math.Abs(secondaryMotionVector.Column - secondaryGlobalMotionVector.Column) >= 16)))) { // The packed global-motion context records whether the first temporal sample is absent or differs from // global motion by at least two full samples in either one-eighth-sample component. this.ModeContext |= GlobalMotionContextBit; } - this.AddUnique(motionVector, 2); + if (secondaryReferenceFrame > Av1ReferenceFrameType.Intra) + { + this.AddUnique(motionVector, secondaryMotionVector, 2); + } + else + { + this.AddUnique(motionVector, 2); + } + return true; } @@ -824,6 +1064,216 @@ internal sealed class Av1ReferenceMotionVectors } } + /// + /// Extends a short compound stack from the immediate above and left blocks. + /// + private void ExtendCompoundStack( + ref Av1PartitionInfo partitionInfo, + Av1FrameInfo frameInfo, + Av1ReferenceFrameType referenceFrame, + Av1ReferenceFrameType secondaryReferenceFrame, + Av1MotionVector globalMotionVector, + Av1MotionVector secondaryGlobalMotionVector, + bool hasAbove, + bool hasLeft, + int extensionLength) + { + InlineArray2 primaryExact = default; + InlineArray2 secondaryExact = default; + InlineArray2 primaryDifferent = default; + InlineArray2 secondaryDifferent = default; + int primaryExactCount = 0; + int secondaryExactCount = 0; + int primaryDifferentCount = 0; + int secondaryDifferentCount = 0; + int row = partitionInfo.RowIndex; + int column = partitionInfo.ColumnIndex; + + for (int index = 0; hasAbove && index < extensionLength;) + { + Av1BlockModeInfo candidate = partitionInfo.SuperblockInfo.GetModeInfoAt(new Point(column + index, row - 1)); + CollectCompoundExtensionCandidate( + candidate, + frameInfo, + referenceFrame, + secondaryReferenceFrame, + ref primaryExact, + ref primaryExactCount, + ref secondaryExact, + ref secondaryExactCount, + ref primaryDifferent, + ref primaryDifferentCount, + ref secondaryDifferent, + ref secondaryDifferentCount); + + index += candidate.BlockSize.Get4x4WideCount(); + } + + for (int index = 0; hasLeft && index < extensionLength;) + { + Av1BlockModeInfo candidate = partitionInfo.SuperblockInfo.GetModeInfoAt(new Point(column - 1, row + index)); + CollectCompoundExtensionCandidate( + candidate, + frameInfo, + referenceFrame, + secondaryReferenceFrame, + ref primaryExact, + ref primaryExactCount, + ref secondaryExact, + ref secondaryExactCount, + ref primaryDifferent, + ref primaryDifferentCount, + ref secondaryDifferent, + ref secondaryDifferentCount); + + index += candidate.BlockSize.Get4x4HighCount(); + } + + InlineArray2 primaryList = BuildCompoundExtensionList( + in primaryExact, + primaryExactCount, + in primaryDifferent, + primaryDifferentCount, + globalMotionVector); + + InlineArray2 secondaryList = BuildCompoundExtensionList( + in secondaryExact, + secondaryExactCount, + in secondaryDifferent, + secondaryDifferentCount, + secondaryGlobalMotionVector); + + if (this.Count == 1) + { + int listIndex = primaryList[0] == this.candidates[0] && secondaryList[0] == this.compoundCandidates[0] ? 1 : 0; + this.candidates[1] = primaryList[listIndex]; + this.compoundCandidates[1] = secondaryList[listIndex]; + this.weights[1] = 2; + this.Count = 2; + return; + } + + // The fallback list is positional rather than a weighted candidate scan. Preserve both entries even when + // they are equal so DRL indices have the same meaning as the pinned libaom implementation. + for (int index = 0; index < 2; index++) + { + this.candidates[index] = primaryList[index]; + this.compoundCandidates[index] = secondaryList[index]; + this.weights[index] = 2; + } + + this.Count = 2; + } + + /// + /// Collects exact-reference and temporal-direction-corrected fallback vectors from one neighboring block. + /// + private static void CollectCompoundExtensionCandidate( + Av1BlockModeInfo candidate, + Av1FrameInfo frameInfo, + Av1ReferenceFrameType referenceFrame, + Av1ReferenceFrameType secondaryReferenceFrame, + ref InlineArray2 primaryExact, + ref int primaryExactCount, + ref InlineArray2 secondaryExact, + ref int secondaryExactCount, + ref InlineArray2 primaryDifferent, + ref int primaryDifferentCount, + ref InlineArray2 secondaryDifferent, + ref int secondaryDifferentCount) + { + Span candidateReferences = candidate.ReferenceFrames; + Span candidateMotionVectors = candidate.MotionVectors; + + for (int candidateIndex = 0; candidateIndex < 2; candidateIndex++) + { + Av1ReferenceFrameType candidateReference = candidateReferences[candidateIndex]; + Av1MotionVector candidateMotionVector = candidateMotionVectors[candidateIndex]; + + for (int targetIndex = 0; targetIndex < 2; targetIndex++) + { + Av1ReferenceFrameType targetReference = targetIndex == 0 ? referenceFrame : secondaryReferenceFrame; + if (candidateReference == targetReference) + { + ref int exactCount = ref (targetIndex == 0 ? ref primaryExactCount : ref secondaryExactCount); + if (exactCount < 2) + { + if (targetIndex == 0) + { + primaryExact[exactCount] = candidateMotionVector; + } + else + { + secondaryExact[exactCount] = candidateMotionVector; + } + + exactCount++; + continue; + } + } + + if (candidateReference <= Av1ReferenceFrameType.Intra) + { + continue; + } + + ref int differentCount = ref (targetIndex == 0 ? ref primaryDifferentCount : ref secondaryDifferentCount); + if (differentCount >= 2) + { + continue; + } + + Av1MotionVector differentMotionVector = candidateMotionVector; + if (frameInfo.IsReferenceSignBiased(candidateReference) != frameInfo.IsReferenceSignBiased(targetReference)) + { + differentMotionVector = new Av1MotionVector(-differentMotionVector.Row, -differentMotionVector.Column); + } + + if (targetIndex == 0) + { + primaryDifferent[differentCount] = differentMotionVector; + } + else + { + secondaryDifferent[differentCount] = differentMotionVector; + } + + differentCount++; + } + } + } + + /// + /// Builds the two positional fallback entries for one member of a compound reference pair. + /// + private static InlineArray2 BuildCompoundExtensionList( + in InlineArray2 exact, + int exactCount, + in InlineArray2 different, + int differentCount, + Av1MotionVector globalMotionVector) + { + InlineArray2 result = default; + int resultCount = 0; + + for (int index = 0; index < exactCount && resultCount < 2; index++) + { + result[resultCount++] = exact[index]; + } + + for (int index = 0; index < differentCount && resultCount < 2; index++) + { + result[resultCount++] = different[index]; + } + + while (resultCount < 2) + { + result[resultCount++] = globalMotionVector; + } + + return result; + } + /// /// Adds a unique candidate or accumulates the weight of an existing candidate. /// @@ -848,6 +1298,29 @@ internal sealed class Av1ReferenceMotionVectors } } + /// + /// Adds a unique compound candidate or accumulates the weight of an existing vector pair. + /// + private void AddUnique(Av1MotionVector motionVector, Av1MotionVector compoundMotionVector, int weight) + { + for (int index = 0; index < this.Count; index++) + { + if (this.candidates[index] == motionVector && this.compoundCandidates[index] == compoundMotionVector) + { + this.weights[index] += (ushort)weight; + return; + } + } + + if (this.Count < CandidateCapacity) + { + this.candidates[this.Count] = motionVector; + this.compoundCandidates[this.Count] = compoundMotionVector; + this.weights[this.Count] = (ushort)weight; + this.Count++; + } + } + /// /// Sorts one candidate region by descending accumulated weight while retaining scan order for equal weights. /// @@ -867,6 +1340,10 @@ internal sealed class Av1ReferenceMotionVectors this.candidates[index - 1] = this.candidates[index]; this.candidates[index] = candidate; + Av1MotionVector compoundCandidate = this.compoundCandidates[index - 1]; + this.compoundCandidates[index - 1] = this.compoundCandidates[index]; + this.compoundCandidates[index] = compoundCandidate; + ushort weight = this.weights[index - 1]; this.weights[index - 1] = this.weights[index]; this.weights[index] = weight; @@ -878,6 +1355,17 @@ internal sealed class Av1ReferenceMotionVectors } } + /// + /// Determines whether an inter mode decodes at least one new motion-vector component. + /// + private static bool UsesNewMotionVector(Av1PredictionMode mode) + => mode is Av1PredictionMode.NewMotionVector or + Av1PredictionMode.NewNewMotionVector or + Av1PredictionMode.NearestNewMotionVector or + Av1PredictionMode.NewNearestMotionVector or + Av1PredictionMode.NearNewMotionVector or + Av1PredictionMode.NewNearMotionVector; + /// /// Provides fixed storage for AV1's eight reference-motion-vector candidates. /// diff --git a/src/ImageSharp/Formats/Heif/Av1/OpenBitstreamUnit/ObuReader.cs b/src/ImageSharp/Formats/Heif/Av1/OpenBitstreamUnit/ObuReader.cs index 776fbb4c1..a7c21e052 100644 --- a/src/ImageSharp/Formats/Heif/Av1/OpenBitstreamUnit/ObuReader.cs +++ b/src/ImageSharp/Formats/Heif/Av1/OpenBitstreamUnit/ObuReader.cs @@ -272,6 +272,14 @@ internal class ObuReader this.ReadFrameHeader(ref payloadReader, header, trailingBit: true); decodedPayloadSize = Av1Math.DivideBy8Floor(payloadReader.BitPosition); primaryFrameHeaderPayload = obuPayload[..decodedPayloadSize]; + + if (primaryFrameHeader.ShowExistingFrame) + { + // This header completes by selecting retained samples; no tile group belongs to it. + this.decoder ??= creator(); + frameDecodingFinished = true; + } + break; case ObuType.RedundantFrameHeader: if (!seenFrameHeader) @@ -311,6 +319,16 @@ internal class ObuReader this.FrameHeader = combinedFrameHeader; this.ReadFrameHeader(ref payloadReader, header, trailingBit: false); primaryFrameHeaderPayload = obuPayload[..Av1Math.DivideBy8Floor(payloadReader.BitPosition)]; + + if (combinedFrameHeader.ShowExistingFrame) + { + // A combined OBU carries no tile-group syntax when it only presents a retained frame. + this.decoder ??= creator(); + frameDecodingFinished = true; + decodedPayloadSize = Av1Math.DivideBy8Floor(payloadReader.BitPosition); + break; + } + goto TILE_GROUP; case ObuType.TileGroup: TILE_GROUP: @@ -1291,7 +1309,7 @@ internal class ObuReader } /// - /// Reads the uncompressed syntax for one coded frame in a bounded AV1 image item or layered image sequence. + /// Reads the uncompressed syntax for one coded frame in a bounded AV1 image item or image sequence. /// /// The reader positioned at the uncompressed frame header. /// The OBU header identifying the frame's temporal and spatial layers. @@ -1338,9 +1356,55 @@ internal class ObuReader frameHeader.DisplayFrameId = reader.ReadLiteral(idLength); } - // The bounded image-item decoder retains reference state only to reconstruct coded dependent layers. - // show_existing_frame is a presentation-timeline operation and remains outside that image-only scope. - throw new InvalidImageContentException("An AV1 image item cannot display a previously decoded frame."); + Av1ReferenceFrameStore? retainedReferenceFrames = this.referenceFrames; + if (retainedReferenceFrames is null) + { + throw new InvalidOperationException("AV1 existing-frame presentation requires a reconstructed reference map."); + } + + int existingFrameSlot = (int)frameHeader.FrameToShowMapIdx; + Av1ReferenceFrame? existingFrame = retainedReferenceFrames.Resolve(existingFrameSlot); + if (existingFrame is null) + { + throw new InvalidImageContentException("The AV1 existing-frame header selects an unoccupied reference-map slot."); + } + + if (!existingFrame.FrameHeader.ShowableFrame) + { + throw new InvalidImageContentException("The AV1 existing-frame header selects a frame that is not showable."); + } + + if (sequenceHeader.IsFrameIdNumbersPresent && + (!frameHeader.GetReferenceValidity()[existingFrameSlot] || + frameHeader.DisplayFrameId != frameHeader.GetReferenceFrameIds()[existingFrameSlot])) + { + throw new InvalidImageContentException("The AV1 existing-frame header has a mismatched display frame identifier."); + } + + ObuFrameHeader existingFrameHeader = existingFrame.FrameHeader; + frameHeader.FrameType = existingFrameHeader.FrameType; + frameHeader.ShowFrame = true; + frameHeader.ShowableFrame = existingFrameHeader.ShowableFrame; + frameHeader.OrderHint = existingFrameHeader.OrderHint; + frameHeader.FrameSize = existingFrameHeader.FrameSize; + frameHeader.FilmGrainParameters = existingFrameHeader.FilmGrainParameters; + + if (existingFrameHeader.FrameType == ObuFrameType.KeyFrame) + { + frameHeader.RefreshFrameFlags = byte.MaxValue; + frameHeader.CurrentFrameId = frameHeader.GetReferenceFrameIds()[existingFrameSlot]; + } + else + { + frameHeader.RefreshFrameFlags = 0; + if (this.frameReferenceState.HasCurrentFrameId) + { + // Non-key existing-frame presentation does not consume or replace decoder current_frame_id. + frameHeader.CurrentFrameId = this.frameReferenceState.CurrentFrameId; + } + } + + return; } frameHeader.FrameType = (ObuFrameType)reader.ReadLiteral(2); diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.cs new file mode 100644 index 000000000..e3365bd03 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.cs @@ -0,0 +1,245 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using System.Runtime.Intrinsics; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; + +/// +/// Combines two AV1 inter predictors with equal-weight rounded averaging. +/// +internal static class Av1CompoundInterPredictor +{ + /// + /// Averages an 8-bit predictor into an existing prediction block. + /// + /// The first predictor and combined output. + /// The distance between destination rows in samples. + /// The second predictor. + /// The distance between second-predictor rows in samples. + /// The active block width. + /// The active block height. + public static void Average( + Span destination, + int destinationStride, + ReadOnlySpan second, + int secondStride, + int width, + int height) + { + for (int row = 0; row < height; row++) + { + Span destinationRow = destination.Slice(row * destinationStride, width); + ReadOnlySpan secondRow = second.Slice(row * secondStride, width); + ref byte destinationReference = ref MemoryMarshal.GetReference(destinationRow); + ref byte secondReference = ref MemoryMarshal.GetReference(secondRow); + int column = 0; + + if (Vector512.IsHardwareAccelerated) + { + int vectorEnd = width - Vector512.Count; + for (; column <= vectorEnd; column += Vector512.Count) + { + Vector512 firstVector = Vector512.LoadUnsafe(ref destinationReference, (nuint)column); + Vector512 secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column); + Average(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + if (Vector256.IsHardwareAccelerated) + { + int vectorEnd = width - Vector256.Count; + for (; column <= vectorEnd; column += Vector256.Count) + { + Vector256 firstVector = Vector256.LoadUnsafe(ref destinationReference, (nuint)column); + Vector256 secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column); + Average(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + if (Vector128.IsHardwareAccelerated) + { + int vectorEnd = width - Vector128.Count; + for (; column <= vectorEnd; column += Vector128.Count) + { + Vector128 firstVector = Vector128.LoadUnsafe(ref destinationReference, (nuint)column); + Vector128 secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column); + Average(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + for (; column < width; column++) + { + destinationRow[column] = (byte)((destinationRow[column] + secondRow[column] + 1) >> 1); + } + } + } + + /// + /// Averages a high-bit-depth predictor into an existing prediction block. + /// + /// The first predictor and combined output. + /// The distance between destination rows in samples. + /// The second predictor. + /// The distance between second-predictor rows in samples. + /// The active block width. + /// The active block height. + public static void Average( + Span destination, + int destinationStride, + ReadOnlySpan second, + int secondStride, + int width, + int height) + { + for (int row = 0; row < height; row++) + { + Span destinationRow = destination.Slice(row * destinationStride, width); + ReadOnlySpan secondRow = second.Slice(row * secondStride, width); + ref ushort destinationReference = ref MemoryMarshal.GetReference(destinationRow); + ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow); + int column = 0; + + if (Vector512.IsHardwareAccelerated) + { + int vectorEnd = width - Vector512.Count; + for (; column <= vectorEnd; column += Vector512.Count) + { + Vector512 firstVector = Vector512.LoadUnsafe(ref destinationReference, (nuint)column); + Vector512 secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column); + Average(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + if (Vector256.IsHardwareAccelerated) + { + int vectorEnd = width - Vector256.Count; + for (; column <= vectorEnd; column += Vector256.Count) + { + Vector256 firstVector = Vector256.LoadUnsafe(ref destinationReference, (nuint)column); + Vector256 secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column); + Average(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + if (Vector128.IsHardwareAccelerated) + { + int vectorEnd = width - Vector128.Count; + for (; column <= vectorEnd; column += Vector128.Count) + { + Vector128 firstVector = Vector128.LoadUnsafe(ref destinationReference, (nuint)column); + Vector128 secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column); + Average(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + for (; column < width; column++) + { + destinationRow[column] = (ushort)((destinationRow[column] + secondRow[column] + 1) >> 1); + } + } + } + + /// + /// Averages an 8-bit predictor without explicit hardware intrinsics. + /// + /// The first predictor and combined output. + /// The distance between destination rows in samples. + /// The second predictor. + /// The distance between second-predictor rows in samples. + /// The active block width. + /// The active block height. + public static void AverageScalar( + Span destination, + int destinationStride, + ReadOnlySpan second, + int secondStride, + int width, + int height) + { + for (int row = 0; row < height; row++) + { + Span destinationRow = destination.Slice(row * destinationStride, width); + ReadOnlySpan secondRow = second.Slice(row * secondStride, width); + for (int column = 0; column < width; column++) + { + destinationRow[column] = (byte)((destinationRow[column] + secondRow[column] + 1) >> 1); + } + } + } + + /// + /// Averages a high-bit-depth predictor without explicit hardware intrinsics. + /// + /// The first predictor and combined output. + /// The distance between destination rows in samples. + /// The second predictor. + /// The distance between second-predictor rows in samples. + /// The active block width. + /// The active block height. + public static void AverageScalar( + Span destination, + int destinationStride, + ReadOnlySpan second, + int secondStride, + int width, + int height) + { + for (int row = 0; row < height; row++) + { + Span destinationRow = destination.Slice(row * destinationStride, width); + ReadOnlySpan secondRow = second.Slice(row * secondStride, width); + for (int column = 0; column < width; column++) + { + destinationRow[column] = (ushort)((destinationRow[column] + secondRow[column] + 1) >> 1); + } + } + } + + /// + /// Computes rounded unsigned averages without widening either input vector. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector512 Average(Vector512 first, Vector512 second) + { + // (a | b) - ((a ^ b) >> 1) is exactly (a + b + 1) >> 1 and cannot overflow an unsigned lane. + return (first | second) - ((first ^ second) >> 1); + } + + /// + /// Computes rounded unsigned averages without widening either input vector. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector256 Average(Vector256 first, Vector256 second) + => (first | second) - ((first ^ second) >> 1); + + /// + /// Computes rounded unsigned averages without widening either input vector. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector128 Average(Vector128 first, Vector128 second) + => (first | second) - ((first ^ second) >> 1); + + /// + /// Computes rounded unsigned averages without widening either input vector. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector512 Average(Vector512 first, Vector512 second) + => (first | second) - ((first ^ second) >> 1); + + /// + /// Computes rounded unsigned averages without widening either input vector. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector256 Average(Vector256 first, Vector256 second) + => (first | second) - ((first ^ second) >> 1); + + /// + /// Computes rounded unsigned averages without widening either input vector. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector128 Average(Vector128 first, Vector128 second) + => (first | second) - ((first ^ second) >> 1); +} diff --git a/src/ImageSharp/Formats/Heif/Av1/ReferenceFrames/Av1ReferenceFrame.cs b/src/ImageSharp/Formats/Heif/Av1/ReferenceFrames/Av1ReferenceFrame.cs index d67c050c5..ed43d164e 100644 --- a/src/ImageSharp/Formats/Heif/Av1/ReferenceFrames/Av1ReferenceFrame.cs +++ b/src/ImageSharp/Formats/Heif/Av1/ReferenceFrames/Av1ReferenceFrame.cs @@ -109,6 +109,12 @@ internal sealed class Av1ReferenceFrame : IDisposable /// public Av1FrameEntropyContext? EntropyContext => this.entropyContext; + /// + /// Restores the retained frame context to the normative defaults selected by this frame's quantizer band. + /// + public void ResetEntropyContext() + => this.entropyContext!.ResetToDefaults(this.FrameHeader.QuantizationParameters.BaseQIndex); + /// /// Transfers the completed sample planes out of this frame owner. /// diff --git a/src/ImageSharp/Formats/Heif/Av1/ReferenceFrames/Av1ReferenceFrameStore.cs b/src/ImageSharp/Formats/Heif/Av1/ReferenceFrames/Av1ReferenceFrameStore.cs index 307dde453..a900f54e0 100644 --- a/src/ImageSharp/Formats/Heif/Av1/ReferenceFrames/Av1ReferenceFrameStore.cs +++ b/src/ImageSharp/Formats/Heif/Av1/ReferenceFrames/Av1ReferenceFrameStore.cs @@ -1,6 +1,8 @@ // Copyright (c) Six Labors. // Licensed under the Six Labors Split License. +using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit; + namespace SixLabors.ImageSharp.Formats.Heif.Av1.ReferenceFrames; /// @@ -155,6 +157,60 @@ internal sealed class Av1ReferenceFrameStore : IDisposable } } + /// + /// Selects one retained reference for presentation and applies the key-frame reference-map reset when required. + /// + /// The zero-based reference-map slot selected by the frame header. + /// The retained frame selected for presentation. + public Av1ReferenceFrame ShowExisting(int slot) + { + Av1ReferenceFrame selectedFrame = this.frames[slot]!; + Av1ReferenceFrame? replacedOutputFrame = this.outputFrame; + this.outputFrame = selectedFrame; + + if (selectedFrame.FrameHeader.FrameType == ObuFrameType.KeyFrame) + { + InlineArray8 replacedFrames = this.frames; + + // Showing a hidden key frame starts a new coded-video-sequence state. All eight reference-map slots now + // identify that same reconstructed owner, so publish every alias before releasing displaced frames. + for (int mapSlot = 0; mapSlot < SlotCount; mapSlot++) + { + this.frames[mapSlot] = selectedFrame; + if (ReferenceEquals(replacedFrames[mapSlot], selectedFrame)) + { + replacedFrames[mapSlot] = null; + } + } + + // A key frame may be presented through show_existing_frame only once. The retained owner carries this + // conformance state because every slot alias must observe the transition. + selectedFrame.FrameHeader.ShowableFrame = false; + + if (replacedOutputFrame is not null) + { + // Let the displaced-output path release a detached shared owner after all of its old slot aliases have + // been removed from the replacement set. + for (int mapSlot = 0; mapSlot < SlotCount; mapSlot++) + { + if (ReferenceEquals(replacedFrames[mapSlot], replacedOutputFrame)) + { + replacedFrames[mapSlot] = null; + } + } + } + + DisposeUnique(ref replacedFrames); + } + + if (replacedOutputFrame is not null && !this.IsRetained(replacedOutputFrame)) + { + replacedOutputFrame.Dispose(); + } + + return selectedFrame; + } + /// /// Transfers the selected presentation frame out of this store and releases every other retained frame. /// diff --git a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileReader.cs b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileReader.cs index dd9b30dc1..7b1b91baa 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileReader.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileReader.cs @@ -803,8 +803,10 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable partitionInfo.PopulateModeInfoNeighbors(this.SequenceHeader.ColorConfig); this.ReadModeInfo(ref reader, ref partitionInfo, tileInfo); + this.ReadPaletteTokens(ref reader, ref partitionInfo); this.ReadBlockTransformSize(ref reader, modeInfoLocation, ref partitionInfo, superblockInfo, tileInfo); + if (partitionInfo.ModeInfo.Skip) { this.ResetSkipContext(ref partitionInfo, tileInfo); @@ -927,7 +929,7 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable DebugGuard.IsFalse(transformUnitCount == 0, nameof(transformUnitCount), string.Empty); for (int tu = 0; tu < transformUnitCount; tu++) { - Av1TransformInfo transformInfo = transformInfoSpan[transformInfoIndex]; + ref Av1TransformInfo transformInfo = ref transformInfoSpan[transformInfoIndex]; DebugGuard.MustBeLessThanOrEqualTo(transformInfo.OffsetX, maxBlocksWide, nameof(transformInfo)); DebugGuard.MustBeLessThanOrEqualTo(transformInfo.OffsetY, maxBlocksHigh, nameof(transformInfo)); @@ -952,7 +954,7 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable ref partitionInfo, tileInfo, coefficientBuffer, - transformInfo, + ref transformInfo, plane, blockColumn, blockRow, @@ -1042,7 +1044,7 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable ref Av1PartitionInfo partitionInfo, Av1TileInfo tileInfo, Span coefficientBuffer, - Av1TransformInfo transformInfo, + ref Av1TransformInfo transformInfo, int plane, int blockColumn, int blockRow, @@ -1094,7 +1096,7 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable plane, transformBlockContext, transformSize, - transformInfo, + ref transformInfo, coefficientBuffer); return endOfBlock; @@ -1128,7 +1130,7 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable int plane, Av1TransformBlockContext transformBlockContext, Av1TransformSize transformSize, - Av1TransformInfo transformInfo, + ref Av1TransformInfo transformInfo, Span coefficientBuffer) { int width = transformSize.GetWidth(); @@ -1142,7 +1144,8 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable Av1BlockSize planeBlockSize = partitionInfo.ModeInfo.BlockSize.GetSubsampled(subX, subY); int blocksWide = partitionInfo.GetMaxBlockWide(planeBlockSize, subX); int blocksHigh = partitionInfo.GetMaxBlockHigh(planeBlockSize, subY); - Av1TransformType lumaTransformType = partitionInfo.ModeInfo.UseIntraBlockCopy && plane > 0 && !isLossless + bool usesInterTransformSet = partitionInfo.ModeInfo.ReferenceFrames[0] >= Av1ReferenceFrameType.Last || partitionInfo.ModeInfo.UseIntraBlockCopy; + Av1TransformType lumaTransformType = usesInterTransformSet && plane > 0 && !isLossless ? partitionInfo.GetLumaTransformType(blockPosition, subX, subY) : Av1TransformType.DctDct; @@ -1161,7 +1164,7 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable isLossless, this.FrameHeader.UseReducedTransformSet, lumaTransformType, - transformInfo, + ref transformInfo, partitionInfo.ModeBlockToRightEdge, partitionInfo.ModeBlockToBottomEdge, coefficientBuffer); @@ -1349,6 +1352,30 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable bool hasAbove = partitionInfo.AvailableAbove; bool hasLeft = partitionInfo.AvailableLeft; + // Inter neighbors expose their coding-block extent here rather than their residual transform extent. This + // keeps intra transform-size selection independent of whether the neighboring inter block split its tree. + if (hasAbove) + { + Av1BlockModeInfo aboveModeInfo = superblockInfo.GetModeInfoAt( + new Point(partitionInfo.ColumnIndex, partitionInfo.RowIndex - 1)); + + if (aboveModeInfo.ReferenceFrames[0] > Av1ReferenceFrameType.Intra) + { + above = aboveModeInfo.BlockSize.GetWidth() >= maxTransformSize.GetWidth() ? 1 : 0; + } + } + + if (hasLeft) + { + Av1BlockModeInfo leftModeInfo = superblockInfo.GetModeInfoAt( + new Point(partitionInfo.ColumnIndex - 1, partitionInfo.RowIndex)); + + if (leftModeInfo.ReferenceFrames[0] > Av1ReferenceFrameType.Intra) + { + left = leftModeInfo.BlockSize.GetHeight() >= maxTransformSize.GetHeight() ? 1 : 0; + } + } + if (hasAbove && hasLeft) { context = above + left; @@ -1389,14 +1416,192 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable int block4x4Width = blockSize.Get4x4WideCount(); int block4x4Height = blockSize.Get4x4HighCount(); - // Both intra frames and intra-coded blocks inside inter frames use the intra transform-size branch. The true - // inter branch will replace this fixed false classification when inter reconstruction is connected. - Av1TransformSize transformSize = this.ReadTransformSize(ref reader, ref partitionInfo, superblockInfo, tileInfo, true); - this.aboveNeighborContext.UpdateTransformation(modeInfoLocation, tileInfo, transformSize, blockSize, false); - this.leftNeighborContext.UpdateTransformation(modeInfoLocation, superblockInfo, transformSize, blockSize, false); + Av1BlockModeInfo modeInfo = partitionInfo.ModeInfo; + bool usesInterTransformSyntax = modeInfo.ReferenceFrames[0] >= Av1ReferenceFrameType.Last || modeInfo.UseIntraBlockCopy; + + this.transformUnitCount[(int)Av1Plane.Y].AsSpan(0, 4).Clear(); + this.transformUnitCount[(int)Av1Plane.U].AsSpan(0, 4).Clear(); + this.transformUnitCount[(int)Av1Plane.V].AsSpan(0, 4).Clear(); + + if (usesInterTransformSyntax && + !modeInfo.Skip && + this.FrameHeader.TransformMode == Av1TransformMode.Select && + blockSize > Av1BlockSize.Block4x4) + { + this.ReadVariableTransformInfo( + ref reader, + modeInfoLocation, + ref partitionInfo, + superblockInfo, + tileInfo); + + return; + } + + // A skipped inter block derives its maximum transform size without a symbol. Intra blocks still select a + // transform size when the frame enables selection because skip_txfm does not suppress their size syntax. + bool allowSelect = !usesInterTransformSyntax || !modeInfo.Skip; + Av1TransformSize transformSize = this.ReadTransformSize( + ref reader, + ref partitionInfo, + superblockInfo, + tileInfo, + allowSelect); + + bool skippedInterBlock = usesInterTransformSyntax && modeInfo.Skip; + this.aboveNeighborContext.UpdateTransformation(modeInfoLocation, tileInfo, transformSize, blockSize, skippedInterBlock); + this.leftNeighborContext.UpdateTransformation(modeInfoLocation, superblockInfo, transformSize, blockSize, skippedInterBlock); this.UpdateTransformInfo(ref partitionInfo, superblockInfo, blockSize, transformSize); } + /// + /// Reads the recursive luma transform partition used by a non-skipped inter block. + /// + /// The tile symbol decoder. + /// The coding-block origin in frame mode-information units. + /// The current coding block. + /// The containing superblock. + /// The active tile boundaries. + private void ReadVariableTransformInfo( + ref Av1SymbolDecoder reader, + Point modeInfoLocation, + ref Av1PartitionInfo partitionInfo, + Av1SuperblockInfo superblockInfo, + Av1TileInfo tileInfo) + { + Av1BlockSize blockSize = partitionInfo.ModeInfo.BlockSize; + Av1TransformSize maximumTransformSize = blockSize.GetMaximumTransformSize(); + int maximumBlocksWide = partitionInfo.GetMaxBlockWide(blockSize, false); + int maximumBlocksHigh = partitionInfo.GetMaxBlockHigh(blockSize, false); + int regionWidth = maximumTransformSize.Get4x4WideCount(); + int regionHeight = maximumTransformSize.Get4x4HighCount(); + int transformInfoIndex = partitionInfo.ModeInfo.GetFirstTransformLocation(Av1PlaneType.Y); + int totalTransformUnitCount = 0; + int regionIndex = 0; + + // Large blocks are visited as independent maximum-transform regions. Keeping the same region order as residual + // parsing lets each region retain an exact transform count without a second map or temporary allocation. + for (int blockRow = 0; blockRow < maximumBlocksHigh; blockRow += regionHeight) + { + for (int blockColumn = 0; blockColumn < maximumBlocksWide; blockColumn += regionWidth) + { + int firstRegionTransform = totalTransformUnitCount; + this.ReadVariableTransformNode( + ref reader, + modeInfoLocation, + ref partitionInfo, + superblockInfo, + tileInfo, + maximumTransformSize, + depth: 0, + blockRow, + blockColumn, + ref transformInfoIndex, + ref totalTransformUnitCount); + + this.transformUnitCount[(int)Av1Plane.Y][regionIndex] = totalTransformUnitCount - firstRegionTransform; + regionIndex++; + } + } + + this.UpdateTransformInfo( + ref partitionInfo, + superblockInfo, + blockSize, + maximumTransformSize, + preserveLuma: true, + existingLumaTransformUnitCount: totalTransformUnitCount); + } + + /// + /// Reads one node of the inter variable-transform tree and appends its leaf transform descriptors. + /// + private void ReadVariableTransformNode( + ref Av1SymbolDecoder reader, + Point modeInfoLocation, + ref Av1PartitionInfo partitionInfo, + Av1SuperblockInfo superblockInfo, + Av1TileInfo tileInfo, + Av1TransformSize transformSize, + int depth, + int blockRow, + int blockColumn, + ref int transformInfoIndex, + ref int transformUnitCount) + { + Av1BlockSize blockSize = partitionInfo.ModeInfo.BlockSize; + int maximumBlocksWide = partitionInfo.GetMaxBlockWide(blockSize, false); + int maximumBlocksHigh = partitionInfo.GetMaxBlockHigh(blockSize, false); + if (blockRow >= maximumBlocksHigh || blockColumn >= maximumBlocksWide) + { + return; + } + + bool split = false; + if (transformSize > Av1TransformSize.Size4x4 && depth < Av1Constants.MaxVarTransform) + { + int aboveOffset = modeInfoLocation.X - tileInfo.ModeInfoColumnStart + blockColumn; + int leftOffset = modeInfoLocation.Y - superblockInfo.ModeInfoPosition.Y + blockRow; + int transformWidth = transformSize.GetWidth(); + int transformHeight = transformSize.GetHeight(); + int above = this.aboveNeighborContext.AboveTransformWidth[aboveOffset] < transformWidth ? 1 : 0; + int left = this.leftNeighborContext.LeftTransformHeight[leftOffset] < transformHeight ? 1 : 0; + int maximumDimension = Math.Max(blockSize.GetWidth(), blockSize.GetHeight()); + Av1TransformSize maximumSquareTransform = maximumDimension switch + { + >= 64 => Av1TransformSize.Size64x64, + >= 32 => Av1TransformSize.Size32x32, + >= 16 => Av1TransformSize.Size16x16, + _ => Av1TransformSize.Size8x8 + }; + + int category = ((transformSize.GetSquareUpSize() != maximumSquareTransform && maximumSquareTransform > Av1TransformSize.Size8x8) ? 1 : 0) + + ((((int)Av1TransformSize.SquareSizes - 1) - (int)maximumSquareTransform) * 2); + + int context = (category * 3) + above + left; + split = reader.ReadTransformPartition(context); + } + + if (split) + { + Av1TransformSize subTransformSize = transformSize.GetSubSize(); + int subWidth = subTransformSize.Get4x4WideCount(); + int subHeight = subTransformSize.Get4x4HighCount(); + int width = transformSize.Get4x4WideCount(); + int height = transformSize.Get4x4HighCount(); + for (int row = 0; row < height; row += subHeight) + { + for (int column = 0; column < width; column += subWidth) + { + this.ReadVariableTransformNode( + ref reader, + modeInfoLocation, + ref partitionInfo, + superblockInfo, + tileInfo, + subTransformSize, + depth + 1, + blockRow + row, + blockColumn + column, + ref transformInfoIndex, + ref transformUnitCount); + } + } + + return; + } + + Span transformInfo = superblockInfo.GetTransformInfoY(); + transformInfo[transformInfoIndex] = new Av1TransformInfo(transformSize, blockColumn, blockRow); + transformInfoIndex++; + transformUnitCount++; + + Point transformLocation = new(modeInfoLocation.X + blockColumn, modeInfoLocation.Y + blockRow); + Av1BlockSize transformBlockSize = transformSize.ToBlockSize(); + this.aboveNeighborContext.UpdateTransformation(transformLocation, tileInfo, transformSize, transformBlockSize, false); + this.leftNeighborContext.UpdateTransformation(transformLocation, superblockInfo, transformSize, transformBlockSize, false); + } + /// /// Populates luma and chroma transform-information records in residual traversal order. /// @@ -1404,17 +1609,21 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable /// The containing superblock and transform storage. /// The coding block size. /// The selected luma transform size. + /// Indicates whether variable-transform traversal already populated luma descriptors. + /// The number of luma descriptors already populated. private unsafe void UpdateTransformInfo( ref Av1PartitionInfo partitionInfo, Av1SuperblockInfo superblockInfo, Av1BlockSize blockSize, - Av1TransformSize transformSize) + Av1TransformSize transformSize, + bool preserveLuma = false, + int existingLumaTransformUnitCount = 0) { int transformInfoYIndex = partitionInfo.ModeInfo.GetFirstTransformLocation(Av1PlaneType.Y); int transformInfoUvIndex = partitionInfo.ModeInfo.GetFirstTransformLocation(Av1PlaneType.Uv); Span lumaTransformInfo = superblockInfo.GetTransformInfoY(); Span chromaTransformInfo = superblockInfo.GetTransformInfoUv(); - int totalLumaTransformUnitCount = 0; + int totalLumaTransformUnitCount = existingLumaTransformUnitCount; int totalChromaTransformUnitCount = 0; int forceSplitCount = 0; bool subX = this.SequenceHeader.ColorConfig.SubSamplingX; @@ -1438,25 +1647,27 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable int lumaTransformUnitCount = 0; int chromaTransformUnitCount = 0; - // Luma transform offsets remain relative to the coding block in 4x4 luma units. - int stepColumn = transformSize.Get4x4WideCount(); - int stepRow = transformSize.Get4x4HighCount(); - int unitHeight = Av1Math.RoundPowerOf2(Math.Min(height + idy, maxBlockHigh), 0); int unitWidth = Av1Math.RoundPowerOf2(Math.Min(width + idx, maxBlockWide), 0); - for (int blockRow = idy; blockRow < unitHeight; blockRow += stepRow) + if (!preserveLuma) { - for (int blockColumn = idx; blockColumn < unitWidth; blockColumn += stepColumn) + // Luma transform offsets remain relative to the coding block in 4x4 luma units. + int lumaStepColumn = transformSize.Get4x4WideCount(); + int lumaStepRow = transformSize.Get4x4HighCount(); + for (int blockRow = idy; blockRow < unitHeight; blockRow += lumaStepRow) { - lumaTransformInfo[transformInfoYIndex] = new Av1TransformInfo( - transformSize, blockColumn, blockRow); - transformInfoYIndex++; - lumaTransformUnitCount++; - totalLumaTransformUnitCount++; + for (int blockColumn = idx; blockColumn < unitWidth; blockColumn += lumaStepColumn) + { + lumaTransformInfo[transformInfoYIndex] = new Av1TransformInfo( + transformSize, blockColumn, blockRow); + transformInfoYIndex++; + lumaTransformUnitCount++; + totalLumaTransformUnitCount++; + } } - } - this.transformUnitCount[(int)Av1Plane.Y][forceSplitCount] = lumaTransformUnitCount; + this.transformUnitCount[(int)Av1Plane.Y][forceSplitCount] = lumaTransformUnitCount; + } if (this.SequenceHeader.ColorConfig.IsMonochrome || !partitionInfo.IsChroma) { @@ -1464,8 +1675,8 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable } // Chroma geometry is rounded to the subsampling grid before stepping its transform size. - stepColumn = transformSizeUv.Get4x4WideCount(); - stepRow = transformSizeUv.Get4x4HighCount(); + int stepColumn = transformSizeUv.Get4x4WideCount(); + int stepRow = transformSizeUv.Get4x4HighCount(); unitHeight = Av1Math.RoundPowerOf2(Math.Min(height + idy, maxBlockHigh), subY ? 1 : 0); unitWidth = Av1Math.RoundPowerOf2(Math.Min(width + idx, maxBlockWide), subX ? 1 : 0); @@ -1593,12 +1804,12 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable } /// - /// Reads the common inter-frame block prefix and the supported intra or single-reference inter prediction branch in bitstream order. + /// Reads the common inter-frame block prefix and the supported intra or inter prediction branch in bitstream order. /// /// The tile symbol decoder. /// The current coding block and its neighbors. /// The active tile boundaries used by reference-motion-vector searches. - /// Implements the prefix, intra, and single-reference translational branches of AV1 section 5.11.7. + /// Implements the prefix, intra, and translational inter branches of AV1 section 5.11.7. internal void ReadInterFrameModeInfo(ref Av1SymbolDecoder reader, ref Av1PartitionInfo partitionInfo, Av1TileInfo tileInfo) { Av1BlockModeInfo modeInfo = partitionInfo.ModeInfo; @@ -1629,10 +1840,8 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable this.ReadReferenceFrames(ref reader, ref partitionInfo); Av1ReferenceFrameType referenceFrame = modeInfo.ReferenceFrames[0]; - if (modeInfo.SkipMode || modeInfo.ReferenceFrames[1] > Av1ReferenceFrameType.Intra) - { - throw new NotSupportedException("AV1 compound-reference block prediction is not implemented."); - } + Av1ReferenceFrameType secondaryReferenceFrame = modeInfo.ReferenceFrames[1]; + bool isCompound = secondaryReferenceFrame > Av1ReferenceFrameType.Intra; Av1ReferenceMotionVectors referenceMotionVectors = this.referenceMotionVectors; referenceMotionVectors.Build( @@ -1641,7 +1850,8 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable this.FrameInfo, this.SequenceHeader, this.FrameHeader, - referenceFrame); + referenceFrame, + secondaryReferenceFrame); ObuSegmentationParameters segmentationParameters = this.FrameHeader.SegmentationParameters; int segmentId = modeInfo.SegmentId; @@ -1650,14 +1860,24 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable segmentationParameters.IsFeatureActive(segmentId, ObuSegmentationLevelFeature.GlobalMotionVector); modeInfo.ReferenceMotionVectorIndex = 0; - modeInfo.YMode = usesForcedGlobalMotion - ? Av1PredictionMode.GlobalMotionVector - : reader.ReadInterMode(referenceMotionVectors.ModeContext); + modeInfo.YMode = modeInfo.SkipMode + ? Av1PredictionMode.NearestNearestMotionVector + : usesForcedGlobalMotion + ? Av1PredictionMode.GlobalMotionVector + : isCompound + ? reader.ReadInterCompoundMode(referenceMotionVectors.ModeContext) + : reader.ReadInterMode(referenceMotionVectors.ModeContext); + + bool modeIsCompound = modeInfo.YMode is >= Av1PredictionMode.CompoundInterModeStart and < Av1PredictionMode.CompoundInterModeEnd; + if (isCompound != modeIsCompound) + { + throw new InvalidImageContentException("AV1 inter prediction mode does not match its reference-frame count."); + } - if (modeInfo.YMode == Av1PredictionMode.NewMotionVector) + if (modeInfo.YMode is Av1PredictionMode.NewMotionVector or Av1PredictionMode.NewNewMotionVector) { - // NEWMV can advance across candidates zero through two. Each transmitted one selects the next - // candidate and exposes one further DRL decision when the stack contains it. + // NEWMV and NEW_NEWMV can advance across candidates zero through two. Each transmitted one selects + // the next candidate and exposes one further DRL decision when the stack contains it. for (int index = 0; index < 2 && referenceMotionVectors.Count > index + 1; index++) { int context = Av1SymbolContextHelper.GetDrlContext(referenceMotionVectors.Weights, index); @@ -1669,10 +1889,14 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable } } } - else if (modeInfo.YMode == Av1PredictionMode.NearMotionVector) + else if (modeInfo.YMode is + Av1PredictionMode.NearMotionVector or + Av1PredictionMode.NearNearMotionVector or + Av1PredictionMode.NearNewMotionVector or + Av1PredictionMode.NewNearMotionVector) { - // NEARMV reserves candidate zero for NEARESTMV, so its two DRL decisions examine pairs one/two and - // two/three while storing a zero-based offset from the first near candidate. + // Modes containing NEARMV reserve candidate zero for NEARESTMV, so their two DRL decisions examine + // pairs one/two and two/three while storing a zero-based offset from the first near candidate. for (int index = 1; index < 3 && referenceMotionVectors.Count > index + 1; index++) { int context = Av1SymbolContextHelper.GetDrlContext(referenceMotionVectors.Weights, index); @@ -1689,31 +1913,116 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable ? Av1MotionVectorPrecision.Integer : this.FrameHeader.AllowHighPrecisionMotionVector ? Av1MotionVectorPrecision.EighthSample : Av1MotionVectorPrecision.QuarterSample; - Av1MotionVector motionVector = modeInfo.YMode switch - { - Av1PredictionMode.NewMotionVector => reader.ReadMotionVector( - referenceMotionVectors.GetNewReference(modeInfo.ReferenceMotionVectorIndex), - precision), - Av1PredictionMode.NearestMotionVector => referenceMotionVectors.Nearest, - Av1PredictionMode.NearMotionVector => referenceMotionVectors.GetNearReference(modeInfo.ReferenceMotionVectorIndex), - Av1PredictionMode.GlobalMotionVector => this.FrameHeader.GetGlobalMotionParameters()[(int)referenceFrame - 1].GetMotionVector( - this.FrameHeader.AllowHighPrecisionMotionVector, - modeInfo.BlockSize, - new Point(partitionInfo.ColumnIndex, partitionInfo.RowIndex), - this.FrameHeader.ForceIntegerMotionVector), - _ => throw new InvalidImageContentException("Invalid single-reference AV1 inter mode.") - }; + Span motionVectors = modeInfo.MotionVectors; + if (!isCompound) + { + motionVectors[0] = modeInfo.YMode switch + { + Av1PredictionMode.NewMotionVector => reader.ReadMotionVector( + referenceMotionVectors.GetNewReference(modeInfo.ReferenceMotionVectorIndex), + precision), + Av1PredictionMode.NearestMotionVector => referenceMotionVectors.Nearest, + Av1PredictionMode.NearMotionVector => referenceMotionVectors.GetNearReference(modeInfo.ReferenceMotionVectorIndex), + Av1PredictionMode.GlobalMotionVector => this.FrameHeader.GetGlobalMotionParameters()[(int)referenceFrame - 1].GetMotionVector( + this.FrameHeader.AllowHighPrecisionMotionVector, + modeInfo.BlockSize, + new Point(partitionInfo.ColumnIndex, partitionInfo.RowIndex), + this.FrameHeader.ForceIntegerMotionVector), + _ => throw new InvalidImageContentException("Invalid single-reference AV1 inter mode.") + }; + } + else + { + int referenceMotionVectorIndex = modeInfo.ReferenceMotionVectorIndex; + int newReferenceIndex = modeInfo.YMode is Av1PredictionMode.NearNewMotionVector or Av1PredictionMode.NewNearMotionVector + ? referenceMotionVectorIndex + 1 + : referenceMotionVectorIndex; + + Av1MotionVector primaryNearest = referenceMotionVectors.GetCompoundNearestReference(0); + Av1MotionVector secondaryNearest = referenceMotionVectors.GetCompoundNearestReference(1); + Av1MotionVector primaryNear = referenceMotionVectors.GetCompoundNearReference(referenceMotionVectorIndex, 0); + Av1MotionVector secondaryNear = referenceMotionVectors.GetCompoundNearReference(referenceMotionVectorIndex, 1); + + switch (modeInfo.YMode) + { + case Av1PredictionMode.NearestNearestMotionVector: + motionVectors[0] = primaryNearest; + motionVectors[1] = secondaryNearest; + break; + case Av1PredictionMode.NearNearMotionVector: + motionVectors[0] = primaryNear; + motionVectors[1] = secondaryNear; + break; + case Av1PredictionMode.NearestNewMotionVector: + motionVectors[0] = primaryNearest; + motionVectors[1] = reader.ReadMotionVector( + referenceMotionVectors.GetCompoundNewReference(newReferenceIndex, 1), + precision); - if (!motionVector.IsValid) + break; + case Av1PredictionMode.NewNearestMotionVector: + motionVectors[0] = reader.ReadMotionVector( + referenceMotionVectors.GetCompoundNewReference(newReferenceIndex, 0), + precision); + + motionVectors[1] = secondaryNearest; + break; + case Av1PredictionMode.NearNewMotionVector: + motionVectors[0] = primaryNear; + motionVectors[1] = reader.ReadMotionVector( + referenceMotionVectors.GetCompoundNewReference(newReferenceIndex, 1), + precision); + + break; + case Av1PredictionMode.NewNearMotionVector: + motionVectors[0] = reader.ReadMotionVector( + referenceMotionVectors.GetCompoundNewReference(newReferenceIndex, 0), + precision); + + motionVectors[1] = secondaryNear; + break; + case Av1PredictionMode.GlobalGlobalMotionVector: + motionVectors[0] = this.FrameHeader.GetGlobalMotionParameters()[(int)referenceFrame - 1].GetMotionVector( + this.FrameHeader.AllowHighPrecisionMotionVector, + modeInfo.BlockSize, + new Point(partitionInfo.ColumnIndex, partitionInfo.RowIndex), + this.FrameHeader.ForceIntegerMotionVector); + + motionVectors[1] = this.FrameHeader.GetGlobalMotionParameters()[(int)secondaryReferenceFrame - 1].GetMotionVector( + this.FrameHeader.AllowHighPrecisionMotionVector, + modeInfo.BlockSize, + new Point(partitionInfo.ColumnIndex, partitionInfo.RowIndex), + this.FrameHeader.ForceIntegerMotionVector); + + break; + case Av1PredictionMode.NewNewMotionVector: + motionVectors[0] = reader.ReadMotionVector( + referenceMotionVectors.GetCompoundNewReference(newReferenceIndex, 0), + precision); + + motionVectors[1] = reader.ReadMotionVector( + referenceMotionVectors.GetCompoundNewReference(newReferenceIndex, 1), + precision); + + break; + default: + throw new InvalidImageContentException("Invalid compound-reference AV1 inter mode."); + } + } + + for (int index = 0; index < (isCompound ? 2 : 1); index++) { - throw new InvalidImageContentException("AV1 motion-vector component is outside the permitted range."); + if (!motionVectors[index].IsValid) + { + throw new InvalidImageContentException("AV1 motion-vector component is outside the permitted range."); + } } - modeInfo.MotionVectors[0] = motionVector; modeInfo.MotionMode = Av1MotionMode.SimpleTranslation; int minimumBlockDimension = Math.Min(modeInfo.BlockSize.GetWidth(), modeInfo.BlockSize.GetHeight()); - if (this.SequenceHeader.EnableInterIntraCompound && + if (!isCompound && !modeInfo.SkipMode && + this.SequenceHeader.EnableInterIntraCompound && modeInfo.BlockSize is >= Av1BlockSize.Block8x8 and <= Av1BlockSize.Block32x32 && reader.ReadIsInterIntra(modeInfo.BlockSize)) { @@ -1723,7 +2032,7 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable throw new NotSupportedException("AV1 inter-intra block prediction is not implemented."); } - if (this.FrameHeader.IsMotionModeSwitchable && minimumBlockDimension >= 8 && !modeInfo.SkipMode) + if (!isCompound && this.FrameHeader.IsMotionModeSwitchable && minimumBlockDimension >= 8 && !modeInfo.SkipMode) { Av1MotionVariationCandidates candidates = this.motionVariationCandidates; candidates.Build(ref partitionInfo, tileInfo, this.SequenceHeader, this.FrameHeader, referenceFrame); @@ -1758,13 +2067,26 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable } } + modeInfo.CompoundGroupIndex = false; + modeInfo.CompoundIndex = true; + modeInfo.CompoundType = Av1CompoundType.Average; + if (isCompound && !modeInfo.SkipMode && + (this.SequenceHeader.EnableMaskedCompound || this.SequenceHeader.OrderHintInfo.EnableJointCompound)) + { + // Either enabled sequence tool adds a compound-selection symbol before interpolation syntax. Refuse + // that later checkpoint at its owning boundary so this equal-average path cannot desynchronize tiles. + throw new NotSupportedException("AV1 selectable compound blending is not implemented."); + } + Span interpolationFilters = modeInfo.InterpolationFilters; Av1InterpolationFilter frameInterpolationFilter = this.FrameHeader.InterpolationFilter; Av1GlobalMotionParameters globalMotion = this.FrameHeader.GetGlobalMotionParameters()[(int)referenceFrame - 1]; bool usesNonTranslationalGlobalMotion = - modeInfo.YMode == Av1PredictionMode.GlobalMotionVector && minimumBlockDimension >= 8 && - globalMotion.Type != Av1GlobalMotionType.Translation; + ((modeInfo.YMode == Av1PredictionMode.GlobalMotionVector && globalMotion.Type != Av1GlobalMotionType.Translation) || + (modeInfo.YMode == Av1PredictionMode.GlobalGlobalMotionVector && + globalMotion.Type != Av1GlobalMotionType.Translation && + this.FrameHeader.GetGlobalMotionParameters()[(int)secondaryReferenceFrame - 1].Type != Av1GlobalMotionType.Translation)); if (modeInfo.SkipMode || modeInfo.MotionMode == Av1MotionMode.Warped || usesNonTranslationalGlobalMotion) { @@ -2850,6 +3172,12 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable return; } + Span referenceCounts = this.neighborReferenceCounts; + Av1SymbolContextHelper.CollectNeighborReferenceCounts( + partitionInfo.AboveModeInfo, + partitionInfo.LeftModeInfo, + referenceCounts); + bool compoundReferenceAllowed = Math.Min(modeInfo.BlockSize.GetWidth(), modeInfo.BlockSize.GetHeight()) >= 8; if (compoundReferenceAllowed && this.FrameHeader.ReferenceMode == ObuReferenceMode.ReferenceModeSelect) { @@ -2859,16 +3187,72 @@ internal sealed class Av1TileReader : IAv1TileReader, IDisposable if (reader.ReadIsCompoundReference(context)) { - throw new NotSupportedException("AV1 compound-reference block prediction is not implemented."); + int typeContext = Av1SymbolContextHelper.GetCompoundReferenceTypeContext( + partitionInfo.AboveModeInfo, + partitionInfo.LeftModeInfo); + + if (!reader.ReadCompoundReferenceIsBidirectional(typeContext)) + { + int directionContext = Av1SymbolContextHelper.GetUnidirectionalCompoundBackwardContext(referenceCounts); + if (reader.ReadUnidirectionalCompoundReference(directionContext, decision: 0)) + { + references[0] = Av1ReferenceFrameType.Backward; + references[1] = Av1ReferenceFrameType.Alternate; + } + else + { + int unidirectionalForwardGroupContext = Av1SymbolContextHelper.GetUnidirectionalCompoundLast3OrGoldenContext(referenceCounts); + if (!reader.ReadUnidirectionalCompoundReference(unidirectionalForwardGroupContext, decision: 1)) + { + references[0] = Av1ReferenceFrameType.Last; + references[1] = Av1ReferenceFrameType.Last2; + } + else + { + int forwardChoiceContext = Av1SymbolContextHelper.GetUnidirectionalCompoundGoldenContext(referenceCounts); + references[0] = Av1ReferenceFrameType.Last; + references[1] = reader.ReadUnidirectionalCompoundReference(forwardChoiceContext, decision: 2) + ? Av1ReferenceFrameType.Golden + : Av1ReferenceFrameType.Last3; + } + } + + return; + } + + int forwardGroupContext = Av1SymbolContextHelper.GetCompoundForwardLast3OrGoldenContext(referenceCounts); + if (!reader.ReadCompoundForwardReference(forwardGroupContext, decision: 0)) + { + int forwardChoiceContext = Av1SymbolContextHelper.GetCompoundForwardLast2Context(referenceCounts); + references[0] = reader.ReadCompoundForwardReference(forwardChoiceContext, decision: 1) + ? Av1ReferenceFrameType.Last2 + : Av1ReferenceFrameType.Last; + } + else + { + int forwardChoiceContext = Av1SymbolContextHelper.GetCompoundForwardGoldenContext(referenceCounts); + references[0] = reader.ReadCompoundForwardReference(forwardChoiceContext, decision: 2) + ? Av1ReferenceFrameType.Golden + : Av1ReferenceFrameType.Last3; + } + + int backwardGroupContext = Av1SymbolContextHelper.GetCompoundBackwardAlternateContext(referenceCounts); + if (reader.ReadCompoundBackwardReference(backwardGroupContext, decision: 0)) + { + references[1] = Av1ReferenceFrameType.Alternate; + } + else + { + int backwardChoiceContext = Av1SymbolContextHelper.GetCompoundBackwardAlternate2Context(referenceCounts); + references[1] = reader.ReadCompoundBackwardReference(backwardChoiceContext, decision: 1) + ? Av1ReferenceFrameType.Alternate2 + : Av1ReferenceFrameType.Backward; + } + + return; } } - Span referenceCounts = this.neighborReferenceCounts; - Av1SymbolContextHelper.CollectNeighborReferenceCounts( - partitionInfo.AboveModeInfo, - partitionInfo.LeftModeInfo, - referenceCounts); - Av1ReferenceFrameType reference; if (reader.ReadSingleReferenceIsBackward(Av1SymbolContextHelper.GetSingleReferenceBackwardContext(referenceCounts))) { diff --git a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TransformInfo.cs b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TransformInfo.cs index 8cff3a790..d24611d6b 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TransformInfo.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TransformInfo.cs @@ -8,10 +8,10 @@ namespace SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; /// /// Describes the size, position, type, and residual state of one AV1 transform block. /// -internal class Av1TransformInfo +internal struct Av1TransformInfo { /// - /// Initializes a new instance of the class with a 4x4 transform at the origin. + /// Initializes a new instance of the struct with a 4x4 transform at the origin. /// public Av1TransformInfo() : this(Av1TransformSize.Size4x4, 0, 0) @@ -19,7 +19,7 @@ internal class Av1TransformInfo } /// - /// Initializes a new instance of the class. + /// Initializes a new instance of the struct. /// /// The transform size. /// The horizontal offset in mode-information units. @@ -32,7 +32,7 @@ internal class Av1TransformInfo } /// - /// Initializes a new instance of the class. + /// Initializes a new instance of the struct. /// /// The to copy the information from. public Av1TransformInfo(Av1TransformInfo originalInfo) diff --git a/src/ImageSharp/Formats/Heif/Av1/Transform/Av1BlockDecoder.cs b/src/ImageSharp/Formats/Heif/Av1/Transform/Av1BlockDecoder.cs index 6dd636d01..126a7da8a 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Transform/Av1BlockDecoder.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Transform/Av1BlockDecoder.cs @@ -126,10 +126,14 @@ internal sealed class Av1BlockDecoder : IDisposable inverseQuantizationOwner = this.frameBuffer.MemoryAllocator.Allocate(inverseQuantizationSize); transformWorkspaceOwner = this.frameBuffer.MemoryAllocator.Allocate(Av1TransformWorkspace.MaximumLength); int maximumBlockLength = 1 << sequenceHeader.SuperblockSizeLog2; - int predictionScratchLength = Math.Max( + int maximumBlockArea = maximumBlockLength * maximumBlockLength; + int predictorWorkingLength = Math.Max( Av1PredictionDecoder.ScratchLength, Av1InterPredictor.GetScratchLength(maximumBlockLength, maximumBlockLength)); + // Compound prediction retains the complete second predictor until both references have been sampled. + // Reserve it once beside the convolution workspace so block traversal never rents temporary memory. + int predictionScratchLength = maximumBlockArea + predictorWorkingLength; predictionScratchOwner = this.frameBuffer.MemoryAllocator.Allocate(predictionScratchLength); this.inverseQuantizationOwner = inverseQuantizationOwner; @@ -235,11 +239,17 @@ internal sealed class Av1BlockDecoder : IDisposable bool isInterBlock = modeInfo.ReferenceFrames[0] >= Av1ReferenceFrameType.Last; Av1FrameBuffer? referenceFrameBuffer = null; + Av1FrameBuffer? secondaryReferenceFrameBuffer = null; + bool isCompound = modeInfo.ReferenceFrames[1] > Av1ReferenceFrameType.Intra; if (isInterBlock) { int canonicalReferenceIndex = (int)modeInfo.ReferenceFrames[0] - (int)Av1ReferenceFrameType.Last; Av1GlobalMotionParameters globalMotion = this.frameHeader.GetGlobalMotionParameters()[canonicalReferenceIndex]; - if (modeInfo.YMode == Av1PredictionMode.GlobalMotionVector && + bool usesPrimaryGlobalMotion = modeInfo.YMode is + Av1PredictionMode.GlobalMotionVector or + Av1PredictionMode.GlobalGlobalMotionVector; + + if (usesPrimaryGlobalMotion && Math.Min(modeInfo.BlockSize.GetWidth(), modeInfo.BlockSize.GetHeight()) >= 8 && globalMotion.Type > Av1GlobalMotionType.Translation) { @@ -248,16 +258,24 @@ internal sealed class Av1BlockDecoder : IDisposable throw new NotSupportedException("AV1 non-translational global prediction is not implemented."); } - uint referenceSlot = this.frameHeader.GetReferenceFrameIndices()[canonicalReferenceIndex]; - - // The uncompressed-header parser validates each selected slot and the reference store remains unchanged - // until frame reconstruction completes, so every parsed inter block resolves the same retained owner. - referenceFrameBuffer = this.referenceFrames!.Resolve((int)referenceSlot)!.FrameBuffer; - if (referenceFrameBuffer.Width != this.frameHeader.FrameSize.FrameWidth || referenceFrameBuffer.Height != this.frameHeader.FrameSize.FrameHeight) + referenceFrameBuffer = this.ResolveReferenceFrame(modeInfo.ReferenceFrames[0]); + if (isCompound) { - // Scaled prediction changes both the source coordinate and the per-output-sample step. Running the - // unit-step predictor here would silently reconstruct valid scaled-reference streams incorrectly. - throw new NotSupportedException("AV1 scaled-reference inter prediction is not implemented."); + int secondaryCanonicalReferenceIndex = (int)modeInfo.ReferenceFrames[1] - (int)Av1ReferenceFrameType.Last; + Av1GlobalMotionParameters secondaryGlobalMotion = this.frameHeader.GetGlobalMotionParameters()[secondaryCanonicalReferenceIndex]; + if (modeInfo.YMode == Av1PredictionMode.GlobalGlobalMotionVector && + Math.Min(modeInfo.BlockSize.GetWidth(), modeInfo.BlockSize.GetHeight()) >= 8 && + secondaryGlobalMotion.Type > Av1GlobalMotionType.Translation) + { + throw new NotSupportedException("AV1 non-translational global prediction is not implemented."); + } + + if (modeInfo.CompoundType != Av1CompoundType.Average) + { + throw new NotSupportedException($"AV1 {modeInfo.CompoundType} compound prediction is not implemented."); + } + + secondaryReferenceFrameBuffer = this.ResolveReferenceFrame(modeInfo.ReferenceFrames[1]); } } @@ -282,7 +300,6 @@ internal sealed class Av1BlockDecoder : IDisposable _ => throw new InvalidImageContentException("Maximum of 3 color planes") }; Span transformInfo = superblockInfo.GetTransformInfo(plane)[transformInfoIndex..]; - Guard.NotNull(transformInfo[0]); if (isLosslessBlock) { @@ -317,97 +334,145 @@ internal sealed class Av1BlockDecoder : IDisposable if (isInterBlock) { - Av1MotionVector motionVector = modeInfo.MotionVectors[0]; int predictionWidth = Math.Max(4, blockSize.GetWidth() >> subX); int predictionHeight = Math.Max(4, blockSize.GetHeight() >> subY); + int maximumBlockLength = 1 << this.sequenceHeader.SuperblockSizeLog2; + int maximumBlockArea = maximumBlockLength * maximumBlockLength; + Span predictionStorage = this.predictionScratchOwner.Memory.Span; + Span secondPredictionStorage = predictionStorage[..maximumBlockArea]; + Span predictionScratch = predictionStorage[maximumBlockArea..]; + Span secondPrediction = MemoryMarshal.AsBytes(secondPredictionStorage)[..(predictionWidth * predictionHeight)]; + Span highBitDepthSecondPrediction = MemoryMarshal.Cast(secondPredictionStorage)[..(predictionWidth * predictionHeight)]; + int referenceCount = isCompound ? 2 : 1; + + for (int referenceIndex = 0; referenceIndex < referenceCount; referenceIndex++) + { + Av1FrameBuffer activeReferenceFrameBuffer = referenceIndex == 0 + ? referenceFrameBuffer! + : secondaryReferenceFrameBuffer!; + + Av1MotionVector motionVector = modeInfo.MotionVectors[referenceIndex]; + + // AV1 predicts the complete declared plane block even when its luma extent crosses the frame boundary. + // Subsampled dimensions retain the mandatory four-sample minimum used by set_plane_n4 in libaom. + int horizontalMotionQ4 = motionVector.Column << (1 - subX); + int verticalMotionQ4 = motionVector.Row << (1 - subY); + int horizontalExtensionQ4 = (4 + predictionWidth) << 4; + int verticalExtensionQ4 = (4 + predictionHeight) << 4; + int horizontalEdgeScale = 1 << (1 - subX); + int verticalEdgeScale = 1 << (1 - subY); + + // The UMV clamp is expressed in one-sixteenth plane-sample units. A 128-sample block can legally + // address 135 samples beyond an edge once its prediction extent and eight-tap filter support are + // included; the frame-owned 144-sample luma border keeps that source directly addressable. + horizontalMotionQ4 = Av1Math.Clip3( + (partitionInfo.ModeBlockToLeftEdge * horizontalEdgeScale) - horizontalExtensionQ4, + (partitionInfo.ModeBlockToRightEdge * horizontalEdgeScale) + horizontalExtensionQ4 - 16, + horizontalMotionQ4); + + verticalMotionQ4 = Av1Math.Clip3( + (partitionInfo.ModeBlockToTopEdge * verticalEdgeScale) - verticalExtensionQ4, + (partitionInfo.ModeBlockToBottomEdge * verticalEdgeScale) + verticalExtensionQ4 - 16, + verticalMotionQ4); + + int sourceColumnQ4 = (pixelPosition.X << 4) + horizontalMotionQ4; + int sourceRowQ4 = (pixelPosition.Y << 4) + verticalMotionQ4; + + // Motion vectors use one-eighth luma-sample units. Shifting by one minus the plane subsampling converts + // them directly to the predictor's one-sixteenth-plane-sample phase; masking then preserves the signed + // floor used to select the integer source sample. + int horizontalPhase = sourceColumnQ4 & 15; + int verticalPhase = sourceRowQ4 & 15; + int destinationStride = referenceIndex == 0 ? reconstructionStride : predictionWidth; - // AV1 predicts the complete declared plane block even when its luma extent crosses the frame boundary. - // Subsampled dimensions retain the mandatory four-sample minimum used by set_plane_n4 in libaom. - int horizontalMotionQ4 = motionVector.Column << (1 - subX); - int verticalMotionQ4 = motionVector.Row << (1 - subY); - int horizontalExtensionQ4 = (4 + predictionWidth) << 4; - int verticalExtensionQ4 = (4 + predictionHeight) << 4; - int horizontalEdgeScale = 1 << (1 - subX); - int verticalEdgeScale = 1 << (1 - subY); - - // The UMV clamp is expressed in one-sixteenth plane-sample units. A 128-sample block can legally - // address 135 samples beyond an edge once its prediction extent and eight-tap filter support are - // included; the frame-owned 144-sample luma border keeps that source directly addressable. - horizontalMotionQ4 = Av1Math.Clip3( - (partitionInfo.ModeBlockToLeftEdge * horizontalEdgeScale) - horizontalExtensionQ4, - (partitionInfo.ModeBlockToRightEdge * horizontalEdgeScale) + horizontalExtensionQ4 - 16, - horizontalMotionQ4); - - verticalMotionQ4 = Av1Math.Clip3( - (partitionInfo.ModeBlockToTopEdge * verticalEdgeScale) - verticalExtensionQ4, - (partitionInfo.ModeBlockToBottomEdge * verticalEdgeScale) + verticalExtensionQ4 - 16, - verticalMotionQ4); - - int sourceColumnQ4 = (pixelPosition.X << 4) + horizontalMotionQ4; - int sourceRowQ4 = (pixelPosition.Y << 4) + verticalMotionQ4; - - // Motion vectors use one-eighth luma-sample units. Shifting by one minus the plane subsampling converts - // them directly to the predictor's one-sixteenth-plane-sample phase; masking then preserves the signed - // floor used to select the integer source sample. - int horizontalPhase = sourceColumnQ4 & 15; - int verticalPhase = sourceRowQ4 & 15; - Span predictionScratch = this.predictionScratchOwner.Memory.Span; + if (highBitDepth) + { + Span source = activeReferenceFrameBuffer.GetPaddedPlaneSpan16( + (Av1Plane)plane, + subX, + subY, + out int sourceStride, + out Point sourceOrigin); - if (highBitDepth) - { - Span source = referenceFrameBuffer!.GetPaddedPlaneSpan16( - (Av1Plane)plane, - subX, - subY, - out int sourceStride, - out Point sourceOrigin); - - int sourceIndex = - ((sourceOrigin.Y + (sourceRowQ4 >> 4)) * sourceStride) + sourceOrigin.X + (sourceColumnQ4 >> 4); - - Span destination = - MemoryMarshal.Cast(highBitDepthBlockReconstructionBuffer[reconstructionStride..]); - - Av1InterPredictor.Predict( - source, - sourceStride, - sourceIndex, - destination, - reconstructionStride, - predictionWidth, - predictionHeight, - modeInfo.InterpolationFilters[1], - modeInfo.InterpolationFilters[0], - horizontalPhase, - verticalPhase, - this.frameBuffer.BitDepth.GetBitCount(), - predictionScratch); + int sourceIndex = + ((sourceOrigin.Y + (sourceRowQ4 >> 4)) * sourceStride) + sourceOrigin.X + (sourceColumnQ4 >> 4); + + Span destination = referenceIndex == 0 + ? MemoryMarshal.Cast(highBitDepthBlockReconstructionBuffer[reconstructionStride..]) + : highBitDepthSecondPrediction; + + Av1InterPredictor.Predict( + source, + sourceStride, + sourceIndex, + destination, + destinationStride, + predictionWidth, + predictionHeight, + modeInfo.InterpolationFilters[1], + modeInfo.InterpolationFilters[0], + horizontalPhase, + verticalPhase, + this.frameBuffer.BitDepth.GetBitCount(), + predictionScratch); + } + else + { + Span source = activeReferenceFrameBuffer.GetPaddedPlaneSpan( + (Av1Plane)plane, + subX, + subY, + out int sourceStride, + out Point sourceOrigin); + + int sourceIndex = + ((sourceOrigin.Y + (sourceRowQ4 >> 4)) * sourceStride) + sourceOrigin.X + (sourceColumnQ4 >> 4); + + Span destination = referenceIndex == 0 + ? blockReconstructionBuffer[reconstructionStride..] + : secondPrediction; + + Av1InterPredictor.Predict( + source, + sourceStride, + sourceIndex, + destination, + destinationStride, + predictionWidth, + predictionHeight, + modeInfo.InterpolationFilters[1], + modeInfo.InterpolationFilters[0], + horizontalPhase, + verticalPhase, + predictionScratch); + } } - else + + if (isCompound) { - Span source = referenceFrameBuffer!.GetPaddedPlaneSpan( - (Av1Plane)plane, - subX, - subY, - out int sourceStride, - out Point sourceOrigin); - - int sourceIndex = - ((sourceOrigin.Y + (sourceRowQ4 >> 4)) * sourceStride) + sourceOrigin.X + (sourceColumnQ4 >> 4); - - Av1InterPredictor.Predict( - source, - sourceStride, - sourceIndex, - blockReconstructionBuffer[reconstructionStride..], - reconstructionStride, - predictionWidth, - predictionHeight, - modeInfo.InterpolationFilters[1], - modeInfo.InterpolationFilters[0], - horizontalPhase, - verticalPhase, - predictionScratch); + if (highBitDepth) + { + Span destination = MemoryMarshal.Cast( + highBitDepthBlockReconstructionBuffer[reconstructionStride..]); + + Av1CompoundInterPredictor.Average( + destination, + reconstructionStride, + highBitDepthSecondPrediction, + predictionWidth, + predictionWidth, + predictionHeight); + } + else + { + Av1CompoundInterPredictor.Average( + blockReconstructionBuffer[reconstructionStride..], + reconstructionStride, + secondPrediction, + predictionWidth, + predictionWidth, + predictionHeight); + } } } @@ -634,6 +699,28 @@ internal sealed class Av1BlockDecoder : IDisposable } } + /// + /// Resolves one canonical reference and verifies that unit-step inter prediction can sample it. + /// + private Av1FrameBuffer ResolveReferenceFrame(Av1ReferenceFrameType referenceFrame) + { + int canonicalReferenceIndex = (int)referenceFrame - (int)Av1ReferenceFrameType.Last; + uint referenceSlot = this.frameHeader.GetReferenceFrameIndices()[canonicalReferenceIndex]; + + // The uncompressed-header parser validates each selected slot and the reference store remains unchanged + // until frame reconstruction completes, so every parsed inter block resolves the same retained owner. + Av1FrameBuffer referenceFrameBuffer = this.referenceFrames!.Resolve((int)referenceSlot)!.FrameBuffer; + if (referenceFrameBuffer.Width != this.frameHeader.FrameSize.FrameWidth || + referenceFrameBuffer.Height != this.frameHeader.FrameSize.FrameHeight) + { + // Scaled prediction changes both the source coordinate and the per-output-sample step. Running the + // unit-step predictor here would silently reconstruct valid scaled-reference streams incorrectly. + throw new NotSupportedException("AV1 scaled-reference inter prediction is not implemented."); + } + + return referenceFrameBuffer; + } + /// /// Derives a byte-addressed reconstruction span beginning one row before a block. /// diff --git a/src/ImageSharp/Formats/Heif/HeifDecoderCore.cs b/src/ImageSharp/Formats/Heif/HeifDecoderCore.cs index c588e341d..18daea6ce 100644 --- a/src/ImageSharp/Formats/Heif/HeifDecoderCore.cs +++ b/src/ImageSharp/Formats/Heif/HeifDecoderCore.cs @@ -328,17 +328,32 @@ internal sealed class HeifDecoderCore : ImageDecoderCore HeifSequenceTrack? alphaTrack = sequence.AlphaTrack; if (alphaTrack is not null) { - for (int frameIndex = 0; frameIndex < colorFrames.Length; frameIndex++) + int frameIndex = 0; + using Av1Decoder alphaDecoder = new(this.configuration); + for (int sampleIndex = 0; sampleIndex < alphaTrack.Samples.Length; sampleIndex++) { cancellationToken.ThrowIfCancellationRequested(); - HeifSequenceSample alphaSample = alphaTrack.Samples[sampleIndices[frameIndex]]; + HeifSequenceSample alphaSample = alphaTrack.Samples[sampleIndex]; + if (alphaSample.IsHidden || + frameIndex >= colorFrames.Length || + sampleIndices[frameIndex] != sampleIndex) + { + this.ExecuteImageDataSegmentAction( + () => this.DecodeSequenceReference(stream, alphaTrack, alphaSample, alphaDecoder)); + + continue; + } + this.ExecuteImageDataSegmentAction( () => this.DecodeSequenceAlphaFrame( stream, alphaTrack, alphaSample, + alphaDecoder, colorFrames[frameIndex], colorTrack.IsPremultiplied)); + + frameIndex++; } } @@ -408,6 +423,7 @@ internal sealed class HeifDecoderCore : ImageDecoderCore ImageFrame[] frames = new ImageFrame[visibleFrameCount]; sampleIndices = new int[visibleFrameCount]; int decodedFrameCount = 0; + using Av1Decoder decoder = new(this.configuration); try { for (int sampleIndex = 0; sampleIndex < track.Samples.Length; sampleIndex++) @@ -415,12 +431,17 @@ internal sealed class HeifDecoderCore : ImageDecoderCore HeifSequenceSample sample = track.Samples[sampleIndex]; if (sample.IsHidden) { + this.ExecuteImageDataSegmentAction( + () => this.DecodeSequenceReference(stream, track, sample, decoder)); + continue; } cancellationToken.ThrowIfCancellationRequested(); ImageFrame? frame = null; - this.ExecuteImageDataSegmentAction(() => frame = this.DecodeSequenceFrame(stream, track, sample)); + this.ExecuteImageDataSegmentAction( + () => frame = this.DecodeSequenceFrame(stream, track, sample, decoder)); + if (frame is null) { continue; @@ -466,11 +487,13 @@ internal sealed class HeifDecoderCore : ImageDecoderCore /// The complete seekable HEIF stream. /// The track supplying the codec configuration and color description. /// The validated sample range. + /// The decoder retaining earlier sequence references. /// The independently owned decoded frame. private ImageFrame DecodeSequenceFrame( BufferedReadStream stream, HeifSequenceTrack track, - HeifSequenceSample sample) + HeifSequenceSample sample, + Av1Decoder decoder) where TPixel : unmanaged, IPixel { if (track.CodecType != Heif4CharCode.Av01) @@ -484,12 +507,10 @@ internal sealed class HeifDecoderCore : ImageDecoderCore using IMemoryOwner sampleOwner = this.ReadSequenceSample(stream, track, sample); Span sampleData = sampleOwner.GetSpan()[..sample.Length]; - using Av1Decoder decoder = new(this.configuration); - ImageFrame frame = decoder.DecodeFrame( + ImageFrame frame = decoder.DecodeSequenceFrame( sampleData, track.CicpProfile, - codecConfiguration, - out _); + codecConfiguration); if (frame.Width != track.CodedWidth || frame.Height != track.CodedHeight) { @@ -507,12 +528,14 @@ internal sealed class HeifDecoderCore : ImageDecoderCore /// The complete seekable HEIF stream. /// The alpha track supplying the codec configuration and color description. /// The validated alpha sample range. + /// The decoder retaining earlier alpha-sequence references. /// The decoded color frame receiving alpha values. /// Whether stored color samples must be converted to unassociated alpha. private void DecodeSequenceAlphaFrame( BufferedReadStream stream, HeifSequenceTrack track, HeifSequenceSample sample, + Av1Decoder decoder, ImageFrame destination, bool premultiplied) where TPixel : unmanaged, IPixel @@ -532,8 +555,7 @@ internal sealed class HeifDecoderCore : ImageDecoderCore using IMemoryOwner sampleOwner = this.ReadSequenceSample(stream, track, sample); Span sampleData = sampleOwner.GetSpan()[..sample.Length]; - using Av1Decoder decoder = new(this.configuration); - decoder.DecodeAlpha( + decoder.DecodeSequenceAlpha( sampleData, track.CicpProfile, codecConfiguration, @@ -544,6 +566,32 @@ internal sealed class HeifDecoderCore : ImageDecoderCore premultiplied); } + /// + /// Decodes one non-presented sequence sample so later dependent samples can resolve its retained references. + /// + /// The complete seekable HEIF stream. + /// The track supplying the codec configuration and color description. + /// The validated non-presented sample. + /// The decoder retaining sequence reference state. + private void DecodeSequenceReference( + BufferedReadStream stream, + HeifSequenceTrack track, + HeifSequenceSample sample, + Av1Decoder decoder) + { + if (track.CodecType != Heif4CharCode.Av01) + { + throw new ImageFormatException($"No decoder is available for image-sequence sample type '{track.CodecType}'."); + } + + Av1CodecConfiguration codecConfiguration = track.Av1CodecConfiguration + ?? throw new InvalidImageContentException("The AV1 image-sequence track has no codec configuration."); + + using IMemoryOwner sampleOwner = this.ReadSequenceSample(stream, track, sample); + Span sampleData = sampleOwner.GetSpan()[..sample.Length]; + decoder.DecodeSequenceReference(sampleData, track.CicpProfile, codecConfiguration); + } + /// /// Reads and validates one bounded AV1 sequence sample into allocator-owned codec input storage. /// diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CoefficientsEntropyTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CoefficientsEntropyTests.cs index 120035112..24eec17c8 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CoefficientsEntropyTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CoefficientsEntropyTests.cs @@ -47,7 +47,7 @@ public class Av1CoefficientsEntropyTests decoder.ReadCoefficients( modeInfo, new Point(0, 0), aboveContexts, leftContexts, 0, 0, 0, 1, 1, transformBlockContext, transformSize, - false, true, transformType, transformInfo, 0, 0, actuals); + false, true, transformType, ref transformInfo, 0, 0, actuals); // Assert Assert.Equal(endOfBlock, actuals[0]); @@ -100,7 +100,7 @@ public class Av1CoefficientsEntropyTests decoder.ReadCoefficients( modeInfo, new Point(0, 0), aboveContexts, leftContexts, 0, 0, plane, 1, 1, transformBlockContext, transformSize, - false, true, transformType, transformInfo, 0, 0, actuals); + false, true, transformType, ref transformInfo, 0, 0, actuals); // Assert Assert.Equal(endOfBlock, actuals[0]); @@ -157,7 +157,7 @@ public class Av1CoefficientsEntropyTests decoder.ReadCoefficients( modeInfo, new Point(0, 0), aboveContexts, leftContexts, 0, 0, plane, 1, 1, transformBlockContext, transformSize, - false, true, transformType, transformInfo, 0, 0, actuals); + false, true, transformType, ref transformInfo, 0, 0, actuals); // Assert Assert.Equal(endOfBlock, actuals[0]); diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundBlockDecoderTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundBlockDecoderTests.cs new file mode 100644 index 000000000..073bf45c1 --- /dev/null +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundBlockDecoderTests.cs @@ -0,0 +1,180 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using SixLabors.ImageSharp.Formats.Heif.Av1; +using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit; +using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.LoopFilter; +using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.Quantizers; +using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; +using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; +using SixLabors.ImageSharp.Formats.Heif.Av1.ReferenceFrames; +using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; +using SixLabors.ImageSharp.Formats.Heif.Av1.Transform; + +namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; + +/// +/// Verifies compound prediction through the production block-reconstruction branch. +/// +[Trait("Format", "Avif")] +public class Av1CompoundBlockDecoderTests +{ + /// + /// Verifies that two retained reference planes are predicted and averaged before residual reconstruction. + /// + /// The native sample depth. + [Theory] + [InlineData((int)Av1BitDepth.EightBit)] + [InlineData((int)Av1BitDepth.TenBit)] + [InlineData((int)Av1BitDepth.TwelveBit)] + public void DecodeBlockReconstructsEqualAverageCompoundPrediction(int bitDepthValue) + { + Av1BitDepth bitDepth = (Av1BitDepth)bitDepthValue; + ushort firstValue = bitDepth == Av1BitDepth.EightBit ? (ushort)20 : (ushort)100; + ushort secondValue = bitDepth switch + { + Av1BitDepth.EightBit => 41, + Av1BitDepth.TenBit => 701, + _ => 3001, + }; + + ushort expected = (ushort)((firstValue + secondValue + 1) >> 1); + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(bitDepth); + ObuFrameHeader frameHeader = CreateFrameHeader(); + frameHeader.GetReferenceFrameIndices()[0] = 0; + frameHeader.GetReferenceFrameIndices()[1] = 1; + + using Av1ReferenceFrameStore referenceFrames = new(); + Assert.True(referenceFrames.Commit(1, CreateReferenceFrame(sequenceHeader, firstValue), showFrame: false)); + Assert.True(referenceFrames.Commit(2, CreateReferenceFrame(sequenceHeader, secondValue), showFrame: false)); + + using Av1FrameBuffer frameBuffer = new( + Configuration.Default, + sequenceHeader, + Av1ColorFormat.Yuv400, + false); + + using Av1FrameInfo frameInfo = new(sequenceHeader); + Av1SuperblockInfo superblockInfo = frameInfo.GetSuperblock(Point.Empty); + superblockInfo.GetTransformInfoY()[0] = new Av1TransformInfo(Av1TransformSize.Size8x8, 0, 0); + + Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block8x8, Point.Empty) + { + Skip = true, + YMode = Av1PredictionMode.NearestNearestMotionVector, + CompoundIndex = true, + CompoundType = Av1CompoundType.Average, + }; + + modeInfo.ReferenceFrames[0] = Av1ReferenceFrameType.Last; + modeInfo.ReferenceFrames[1] = Av1ReferenceFrameType.Last2; + modeInfo.InterpolationFilters.Fill(Av1InterpolationFilter.Regular); + modeInfo.SetTransformUnitCount(Av1PlaneType.Y, 1); + + Av1LoopFilterContext loopFilterContext = new(sequenceHeader); + Av1InverseQuantizer inverseQuantizer = new(sequenceHeader, frameHeader); + using Av1BlockDecoder decoder = new( + sequenceHeader, + frameHeader, + frameBuffer, + loopFilterContext, + inverseQuantizer, + referenceFrames); + + decoder.UpdateSuperblock(superblockInfo); + decoder.DecodeBlock( + modeInfo, + Point.Empty, + Av1BlockSize.Block8x8, + superblockInfo, + new Av1TileInfo(0, 0, frameHeader)); + + for (int row = 0; row < 8; row++) + { + if (bitDepth == Av1BitDepth.EightBit) + { + Span samples = frameBuffer.DeriveBlockPointer(Av1Plane.Y, 0, 0).DangerousGetRowSpan(row); + for (int column = 0; column < 8; column++) + { + Assert.Equal((byte)expected, samples[column]); + } + } + else + { + Span samples = frameBuffer.GetHighBitDepthRowSpan(Av1Plane.Y, row, 0, 0); + for (int column = 0; column < 8; column++) + { + Assert.Equal(expected, samples[column]); + } + } + } + } + + /// + /// Creates one independently owned retained frame filled with a constant visible luma value. + /// + private static Av1ReferenceFrame CreateReferenceFrame(ObuSequenceHeader sequenceHeader, ushort value) + { + Av1FrameBuffer frameBuffer = new( + Configuration.Default, + sequenceHeader, + Av1ColorFormat.Yuv400, + false); + + if (sequenceHeader.ColorConfig.BitDepth == Av1BitDepth.EightBit) + { + for (int row = 0; row < 8; row++) + { + frameBuffer.DeriveBlockPointer(Av1Plane.Y, 0, 0).DangerousGetRowSpan(row).Fill((byte)value); + } + } + else + { + for (int row = 0; row < 8; row++) + { + frameBuffer.GetHighBitDepthRowSpan(Av1Plane.Y, row, 0, 0).Fill(value); + } + } + + using Av1FrameInfo frameInfo = new(sequenceHeader); + return new Av1ReferenceFrame(frameBuffer, CreateFrameHeader(), frameInfo); + } + + /// + /// Creates the monochrome 8x8 sequence used by direct reconstruction tests. + /// + private static ObuSequenceHeader CreateSequenceHeader(Av1BitDepth bitDepth) + => new() + { + MaxFrameWidth = 8, + MaxFrameHeight = 8, + Use128x128Superblock = false, + ColorConfig = new ObuColorConfig + { + IsMonochrome = true, + BitDepth = bitDepth, + }, + }; + + /// + /// Creates an unscaled 8x8 inter-frame header with one complete tile. + /// + private static ObuFrameHeader CreateFrameHeader() + { + ObuFrameHeader frameHeader = new() + { + FrameType = ObuFrameType.InterFrame, + ModeInfoColumnCount = 2, + ModeInfoRowCount = 2, + FrameSize = new ObuFrameSize + { + FrameWidth = 8, + FrameHeight = 8, + }, + }; + + frameHeader.TilesInfo.TileColumnStartModeInfo[1] = frameHeader.ModeInfoColumnCount; + frameHeader.TilesInfo.TileRowStartModeInfo[1] = frameHeader.ModeInfoRowCount; + return frameHeader; + } +} diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundInterPredictorTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundInterPredictorTests.cs new file mode 100644 index 000000000..312d8d6c9 --- /dev/null +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundInterPredictorTests.cs @@ -0,0 +1,165 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; +using SixLabors.ImageSharp.Tests.TestUtilities; + +namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; + +/// +/// Verifies equal-weight AV1 compound prediction across every hardware-intrinsic tier. +/// +[Trait("Format", "Avif")] +public class Av1CompoundInterPredictorTests +{ + /// + /// Exercises the native vector width, 256-bit and 128-bit paths, and the complete scalar fallback. + /// + private const HwIntrinsics PredictorConfigurations = + HwIntrinsics.AllowAll | HwIntrinsics.DisableAVX512F | HwIntrinsics.DisableAVX | HwIntrinsics.DisableHWIntrinsic; + + /// + /// Verifies rounded 8-bit averaging, scalar tails, and untouched row padding under every SIMD configuration. + /// + [Fact] + public void ByteAverageMatchesIndependentOracleAcrossIntrinsicWidths() + => FeatureTestRunner.RunWithHwIntrinsicsFeature(ValidateByteAverage, PredictorConfigurations); + + /// + /// Verifies rounded 10/12-bit averaging, scalar tails, and untouched row padding under every SIMD configuration. + /// + [Fact] + public void HighBitDepthAverageMatchesIndependentOracleAcrossIntrinsicWidths() + => FeatureTestRunner.RunWithHwIntrinsicsFeature(ValidateHighBitDepthAverage, PredictorConfigurations); + + /// + /// Applies independent byte arithmetic to block widths that cross every vector and scalar boundary. + /// + private static void ValidateByteAverage() + { + ReadOnlySpan widths = [4, 7, 8, 15, 16, 23, 31, 32, 47, 64, 127, 128]; + + foreach (int width in widths) + { + const int height = 5; + int destinationStride = width + 11; + int secondStride = width + 7; + byte[] expected = new byte[destinationStride * height]; + byte[] actual = new byte[destinationStride * height]; + byte[] scalar = new byte[destinationStride * height]; + byte[] second = new byte[secondStride * height]; + + FillByteInputs(expected, second, destinationStride, secondStride, width, height); + expected.CopyTo(actual, 0); + expected.CopyTo(scalar, 0); + + for (int row = 0; row < height; row++) + { + for (int column = 0; column < width; column++) + { + int destinationIndex = (row * destinationStride) + column; + int secondIndex = (row * secondStride) + column; + expected[destinationIndex] = (byte)((expected[destinationIndex] + second[secondIndex] + 1) >> 1); + } + } + + Av1CompoundInterPredictor.Average(actual, destinationStride, second, secondStride, width, height); + Av1CompoundInterPredictor.AverageScalar(scalar, destinationStride, second, secondStride, width, height); + + Assert.Equal(expected, actual); + Assert.Equal(expected, scalar); + } + } + + /// + /// Applies independent ushort arithmetic at both supported high-bit-depth limits. + /// + private static void ValidateHighBitDepthAverage() + { + ReadOnlySpan widths = [4, 7, 8, 15, 16, 23, 31, 32, 47, 64, 127, 128]; + + foreach (int bitDepth in new[] { 10, 12 }) + { + foreach (int width in widths) + { + const int height = 5; + int destinationStride = width + 9; + int secondStride = width + 5; + ushort[] expected = new ushort[destinationStride * height]; + ushort[] actual = new ushort[destinationStride * height]; + ushort[] scalar = new ushort[destinationStride * height]; + ushort[] second = new ushort[secondStride * height]; + + FillHighBitDepthInputs(expected, second, destinationStride, secondStride, width, height, bitDepth); + expected.CopyTo(actual, 0); + expected.CopyTo(scalar, 0); + + for (int row = 0; row < height; row++) + { + for (int column = 0; column < width; column++) + { + int destinationIndex = (row * destinationStride) + column; + int secondIndex = (row * secondStride) + column; + expected[destinationIndex] = (ushort)((expected[destinationIndex] + second[secondIndex] + 1) >> 1); + } + } + + Av1CompoundInterPredictor.Average(actual, destinationStride, second, secondStride, width, height); + Av1CompoundInterPredictor.AverageScalar(scalar, destinationStride, second, secondStride, width, height); + + Assert.Equal(expected, actual); + Assert.Equal(expected, scalar); + } + } + } + + /// + /// Fills active byte samples while assigning different sentinels to the unused row tails. + /// + private static void FillByteInputs( + Span destination, + Span second, + int destinationStride, + int secondStride, + int width, + int height) + { + destination.Fill(0xD3); + second.Fill(0xA7); + + for (int row = 0; row < height; row++) + { + for (int column = 0; column < width; column++) + { + destination[(row * destinationStride) + column] = (byte)((row * 47) + (column * 29) + 3); + second[(row * secondStride) + column] = (byte)((row * 31) + (column * 53) + 11); + } + } + } + + /// + /// Fills active ushort samples across the requested precision while preserving guarded row tails. + /// + private static void FillHighBitDepthInputs( + Span destination, + Span second, + int destinationStride, + int secondStride, + int width, + int height, + int bitDepth) + { + destination.Fill(0xDEAD); + second.Fill(0xBEEF); + int mask = (1 << bitDepth) - 1; + + for (int row = 0; row < height; row++) + { + for (int column = 0; column < width; column++) + { + destination[(row * destinationStride) + column] = (ushort)(((row * 947) + (column * 613) + 17) & mask); + second[(row * secondStride) + column] = (ushort)(((row * 541) + (column * 887) + 23) & mask); + } + } + } +} diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundReferenceEntropyTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundReferenceEntropyTests.cs new file mode 100644 index 000000000..fc889aeea --- /dev/null +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundReferenceEntropyTests.cs @@ -0,0 +1,302 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Buffers; +using SixLabors.ImageSharp.Formats.Heif.Av1; +using SixLabors.ImageSharp.Formats.Heif.Av1.Entropy; +using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; +using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; + +namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; + +/// +/// Verifies AV1 compound-reference selection and compound inter-mode entropy against pinned libaom. +/// +[Trait("Format", "Avif")] +public class Av1CompoundReferenceEntropyTests +{ + /// + /// Verifies every binary compound-reference default against libaom's forward Q15 tables. + /// + [Fact] + public void CompoundReferenceDefaultsMatchLibaom() + { + AssertBinaryDefaults( + [1198, 2070, 9166, 7499, 22475], + Av1DefaultDistributions.CompoundReferenceType); + + AssertBinaryDefaults( + [ + [5284, 3865, 3128], + [23152, 14173, 15270], + [31774, 25120, 26710], + ], + Av1DefaultDistributions.UnidirectionalCompoundReference); + + AssertBinaryDefaults( + [ + [4946, 9468, 1503], + [19891, 22441, 15160], + [30731, 31059, 27544], + ], + Av1DefaultDistributions.CompoundReference); + + AssertBinaryDefaults( + [ + [2235, 1423], + [17182, 15175], + [30606, 30489], + ], + Av1DefaultDistributions.CompoundBackwardReference); + } + + /// + /// Verifies all eight compound inter-mode defaults against libaom's forward Q15 tables. + /// + [Fact] + public void InterCompoundModeDefaultsMatchLibaom() + { + uint[][] expected = + [ + [7760, 13823, 15808, 17641, 19156, 20666, 26891], + [10730, 19452, 21145, 22749, 24039, 25131, 28724], + [10664, 20221, 21588, 22906, 24295, 25387, 28436], + [13298, 16984, 20471, 24182, 25067, 25736, 26422], + [18904, 23325, 25242, 27432, 27898, 28258, 30758], + [10725, 17454, 20124, 22820, 24195, 25168, 26046], + [17125, 24273, 25814, 27492, 28214, 28704, 30592], + [13046, 23214, 24505, 25942, 27435, 28442, 29330], + ]; + + Av1Distribution[] actual = Av1DefaultDistributions.InterCompoundMode; + + Assert.Equal(expected.Length, actual.Length); + for (int context = 0; context < expected.Length; context++) + { + Assert.Equal(8, actual[context].NumberOfSymbols); + for (int threshold = 0; threshold < expected[context].Length; threshold++) + { + Assert.Equal((uint)Av1Distribution.ProbabilityTop - expected[context][threshold], actual[context][threshold]); + } + } + } + + /// + /// Verifies that each semantic reference reader selects its requested context row and tree decision. + /// + [Fact] + public void CompoundReferenceReadersUseRequestedDistributions() + { + bool[] values = [false, true, true, false, true, false]; + + for (int context = 0; context < 5; context++) + { + AssertBinaryReader( + Av1DefaultDistributions.CompoundReferenceType[context], + values, + (ref Av1SymbolDecoder decoder) => decoder.ReadCompoundReferenceIsBidirectional(context)); + } + + for (int context = 0; context < 3; context++) + { + for (int decision = 0; decision < 3; decision++) + { + AssertBinaryReader( + Av1DefaultDistributions.UnidirectionalCompoundReference[context][decision], + values, + (ref Av1SymbolDecoder decoder) => decoder.ReadUnidirectionalCompoundReference(context, decision)); + + AssertBinaryReader( + Av1DefaultDistributions.CompoundReference[context][decision], + values, + (ref Av1SymbolDecoder decoder) => decoder.ReadCompoundForwardReference(context, decision)); + } + + for (int decision = 0; decision < 2; decision++) + { + AssertBinaryReader( + Av1DefaultDistributions.CompoundBackwardReference[context][decision], + values, + (ref Av1SymbolDecoder decoder) => decoder.ReadCompoundBackwardReference(context, decision)); + } + } + } + + /// + /// Verifies the packed-mode-context mapping and all eight compound mode symbols. + /// + [Fact] + public void CompoundModeReaderUsesMappedDistribution() + { + ReadOnlySpan packedContexts = [0, 1, 33, 34, 35, 66, 67, 68]; + + for (int context = 0; context < packedContexts.Length; context++) + { + using Av1SymbolWriter writer = new(Configuration.Default, 3, updateCdf: true); + Av1Distribution writerDistribution = Av1DefaultDistributions.InterCompoundMode[context]; + writer.WriteSymbol(0, writerDistribution); + writer.WriteSymbol(7, writerDistribution); + writer.WriteSymbol(3, writerDistribution); + + using IMemoryOwner encoded = writer.Exit(); + Av1SymbolDecoder decoder = new(Configuration.Default, encoded.Memory.Span, 0, updateCdf: true); + + Assert.Equal(Av1PredictionMode.NearestNearestMotionVector, decoder.ReadInterCompoundMode(packedContexts[context])); + Assert.Equal(Av1PredictionMode.NewNewMotionVector, decoder.ReadInterCompoundMode(packedContexts[context])); + Assert.Equal(Av1PredictionMode.NewNearestMotionVector, decoder.ReadInterCompoundMode(packedContexts[context])); + } + } + + /// + /// Verifies the compound-reference type context across intra, single, bidirectional, and unidirectional neighbors. + /// + [Fact] + public void CompoundReferenceTypeContextMatchesLibaom() + { + Av1BlockModeInfo intra = CreateModeInfo(Av1ReferenceFrameType.Intra, Av1ReferenceFrameType.None); + Av1BlockModeInfo singleForward = CreateModeInfo(Av1ReferenceFrameType.Last, Av1ReferenceFrameType.None); + Av1BlockModeInfo singleBackward = CreateModeInfo(Av1ReferenceFrameType.Backward, Av1ReferenceFrameType.None); + Av1BlockModeInfo bidirectional = CreateModeInfo(Av1ReferenceFrameType.Last, Av1ReferenceFrameType.Backward); + Av1BlockModeInfo forwardUnidirectional = CreateModeInfo(Av1ReferenceFrameType.Last, Av1ReferenceFrameType.Last2); + Av1BlockModeInfo backwardUnidirectional = CreateModeInfo(Av1ReferenceFrameType.Backward, Av1ReferenceFrameType.Alternate); + + Assert.Equal(2, Av1SymbolContextHelper.GetCompoundReferenceTypeContext(null, null)); + Assert.Equal(2, Av1SymbolContextHelper.GetCompoundReferenceTypeContext(intra, null)); + Assert.Equal(2, Av1SymbolContextHelper.GetCompoundReferenceTypeContext(singleForward, null)); + Assert.Equal(0, Av1SymbolContextHelper.GetCompoundReferenceTypeContext(bidirectional, null)); + Assert.Equal(4, Av1SymbolContextHelper.GetCompoundReferenceTypeContext(forwardUnidirectional, null)); + Assert.Equal(2, Av1SymbolContextHelper.GetCompoundReferenceTypeContext(intra, singleForward)); + Assert.Equal(1, Av1SymbolContextHelper.GetCompoundReferenceTypeContext(intra, bidirectional)); + Assert.Equal(3, Av1SymbolContextHelper.GetCompoundReferenceTypeContext(intra, forwardUnidirectional)); + Assert.Equal(3, Av1SymbolContextHelper.GetCompoundReferenceTypeContext(singleForward, singleForward)); + Assert.Equal(1, Av1SymbolContextHelper.GetCompoundReferenceTypeContext(singleForward, singleBackward)); + Assert.Equal(0, Av1SymbolContextHelper.GetCompoundReferenceTypeContext(bidirectional, bidirectional)); + Assert.Equal(2, Av1SymbolContextHelper.GetCompoundReferenceTypeContext(bidirectional, forwardUnidirectional)); + Assert.Equal(4, Av1SymbolContextHelper.GetCompoundReferenceTypeContext(forwardUnidirectional, forwardUnidirectional)); + Assert.Equal(3, Av1SymbolContextHelper.GetCompoundReferenceTypeContext(forwardUnidirectional, backwardUnidirectional)); + } + + /// + /// Verifies the exact neighboring-vote groups used by every compound reference-tree decision. + /// + [Fact] + public void CompoundReferenceContextsAggregateNormativeGroups() + { + InlineArray8 referenceCountStorage = default; + Span referenceCounts = referenceCountStorage; + referenceCounts[(int)Av1ReferenceFrameType.Last] = 5; + referenceCounts[(int)Av1ReferenceFrameType.Last2] = 1; + referenceCounts[(int)Av1ReferenceFrameType.Last3] = 2; + referenceCounts[(int)Av1ReferenceFrameType.Golden] = 2; + referenceCounts[(int)Av1ReferenceFrameType.Backward] = 3; + referenceCounts[(int)Av1ReferenceFrameType.Alternate2] = 3; + referenceCounts[(int)Av1ReferenceFrameType.Alternate] = 6; + + Assert.Equal(0, Av1SymbolContextHelper.GetUnidirectionalCompoundBackwardContext(referenceCounts)); + Assert.Equal(0, Av1SymbolContextHelper.GetUnidirectionalCompoundLast3OrGoldenContext(referenceCounts)); + Assert.Equal(1, Av1SymbolContextHelper.GetUnidirectionalCompoundGoldenContext(referenceCounts)); + Assert.Equal(2, Av1SymbolContextHelper.GetCompoundForwardLast3OrGoldenContext(referenceCounts)); + Assert.Equal(2, Av1SymbolContextHelper.GetCompoundForwardLast2Context(referenceCounts)); + Assert.Equal(1, Av1SymbolContextHelper.GetCompoundForwardGoldenContext(referenceCounts)); + Assert.Equal(1, Av1SymbolContextHelper.GetCompoundBackwardAlternateContext(referenceCounts)); + Assert.Equal(1, Av1SymbolContextHelper.GetCompoundBackwardAlternate2Context(referenceCounts)); + } + + /// + /// Verifies compound CDF copying and snapshot update-count reset without sharing mutable state. + /// + [Fact] + public void FrameEntropyLifecycleIncludesCompoundDistributions() + { + const int updateCount = 20; + Av1FrameEntropyContext source = new(0); + Av1FrameEntropyContext copy = new(0); + Av1FrameEntropyContext snapshot = new(0); + + for (int i = 0; i < updateCount; i++) + { + source.CompoundReferenceType[4].Update(1); + source.UnidirectionalCompoundReference[2][2].Update(1); + source.CompoundReference[1][1].Update(1); + source.CompoundBackwardReference[0][1].Update(1); + source.InterCompoundMode[7].Update(6); + } + + copy.CopyFrom(source); + source.SnapshotTo(snapshot); + + Assert.Equal(source.CompoundReferenceType[4][0], copy.CompoundReferenceType[4][0]); + Assert.Equal(source.UnidirectionalCompoundReference[2][2][0], copy.UnidirectionalCompoundReference[2][2][0]); + Assert.Equal(source.CompoundReference[1][1][0], copy.CompoundReference[1][1][0]); + Assert.Equal(source.CompoundBackwardReference[0][1][0], copy.CompoundBackwardReference[0][1][0]); + Assert.Equal(source.InterCompoundMode[7][6], copy.InterCompoundMode[7][6]); + + source.CompoundReferenceType[4].Update(0); + snapshot.CompoundReferenceType[4].Update(0); + source.InterCompoundMode[7].Update(0); + snapshot.InterCompoundMode[7].Update(0); + + Assert.NotEqual(source.CompoundReferenceType[4][0], snapshot.CompoundReferenceType[4][0]); + Assert.NotEqual(source.InterCompoundMode[7][0], snapshot.InterCompoundMode[7][0]); + } + + /// + /// Verifies one binary reader against a separately adapted writer distribution. + /// + private static void AssertBinaryReader(Av1Distribution distribution, ReadOnlySpan values, SymbolReader reader) + { + using Av1SymbolWriter writer = new(Configuration.Default, values.Length, updateCdf: true); + foreach (bool value in values) + { + writer.WriteSymbol(value, distribution); + } + + using IMemoryOwner encoded = writer.Exit(); + Av1SymbolDecoder decoder = new(Configuration.Default, encoded.Memory.Span, 0, updateCdf: true); + foreach (bool value in values) + { + Assert.Equal(value, reader(ref decoder)); + } + } + + /// + /// Verifies one array of binary defaults stored in inverse-cumulative form. + /// + private static void AssertBinaryDefaults(ReadOnlySpan expected, Av1Distribution[] actual) + { + Assert.Equal(expected.Length, actual.Length); + for (int i = 0; i < expected.Length; i++) + { + Assert.Equal((uint)Av1Distribution.ProbabilityTop - expected[i], actual[i][0]); + Assert.Equal(2, actual[i].NumberOfSymbols); + } + } + + /// + /// Verifies a matrix of binary defaults stored in inverse-cumulative form. + /// + private static void AssertBinaryDefaults(uint[][] expected, Av1Distribution[][] actual) + { + Assert.Equal(expected.Length, actual.Length); + for (int row = 0; row < expected.Length; row++) + { + AssertBinaryDefaults(expected[row], actual[row]); + } + } + + /// + /// Creates decoded block-mode state with the requested primary and secondary references. + /// + private static Av1BlockModeInfo CreateModeInfo(Av1ReferenceFrameType primary, Av1ReferenceFrameType secondary) + { + Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block8x8, Point.Empty); + modeInfo.ReferenceFrames[0] = primary; + modeInfo.ReferenceFrames[1] = secondary; + return modeInfo; + } + + /// + /// Invokes one semantic binary symbol reader. + /// + private delegate bool SymbolReader(ref Av1SymbolDecoder decoder); +} diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterFrameModeInfoTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterFrameModeInfoTests.cs index a0f3a40f8..c447a2ab3 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterFrameModeInfoTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterFrameModeInfoTests.cs @@ -14,7 +14,7 @@ using SixLabors.ImageSharp.Memory; namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; /// -/// Verifies inter-frame block-prefix, intra-block selection, skip-mode, and interpolation-filter syntax. +/// Verifies inter-frame block-prefix, reference selection, motion-mode, and interpolation-filter syntax. /// [Trait("Format", "Avif")] public class Av1InterFrameModeInfoTests @@ -59,7 +59,25 @@ public class Av1InterFrameModeInfoTests public void ReadInterFrameModeInfoSkipModeForcesInterBlockAndResidualSkip() { ObuSequenceHeader sequenceHeader = CreateSequenceHeader(); + sequenceHeader.OrderHintInfo.EnableOrderHint = true; + sequenceHeader.OrderHintInfo.OrderHintBits = 3; ObuFrameHeader frameHeader = CreateFrameHeader(); + frameHeader.ReferenceMode = ObuReferenceMode.ReferenceModeSelect; + frameHeader.OrderHint = 4; + for (int index = 0; index < Av1Constants.ReferencesPerFrame; index++) + { + frameHeader.GetReferenceFrameIndices()[index] = (uint)index; + } + + Span referenceOrderHints = frameHeader.GetReferenceOrderHints(); + referenceOrderHints[0] = 3; + referenceOrderHints[1] = 2; + referenceOrderHints[2] = 1; + referenceOrderHints[3] = 0; + referenceOrderHints[4] = 5; + referenceOrderHints[5] = 6; + referenceOrderHints[6] = 7; + frameHeader.SkipModeParameters.Derive(sequenceHeader.OrderHintInfo, frameHeader); frameHeader.SkipModeParameters.SkipModeFlag = true; using Av1TileReader tileReader = new(Configuration.Default, sequenceHeader, frameHeader); Av1BlockModeInfo aboveModeInfo = new(Av1BlockSize.Block8x8, Point.Empty) { SkipMode = true }; @@ -71,9 +89,14 @@ public class Av1InterFrameModeInfoTests using IMemoryOwner encoded = writer.Exit(); Memory encodedMemory = encoded.Memory; - Assert.Throws(() => ReadInterFrameModeInfo(tileReader, encodedMemory, modeInfo, aboveModeInfo)); + ReadInterFrameModeInfo(tileReader, encodedMemory, modeInfo, aboveModeInfo); + Assert.True(modeInfo.SkipMode); Assert.True(modeInfo.Skip); + Assert.Equal(Av1PredictionMode.NearestNearestMotionVector, modeInfo.YMode); + Assert.Equal(Av1ReferenceFrameType.Last, modeInfo.ReferenceFrames[0]); + Assert.Equal(Av1ReferenceFrameType.Backward, modeInfo.ReferenceFrames[1]); + Assert.Equal(Av1CompoundType.Average, modeInfo.CompoundType); } /// @@ -142,8 +165,8 @@ public class Av1InterFrameModeInfoTests using Av1SymbolWriter writer = new(Configuration.Default, 3, updateCdf: true); writer.WriteSymbol(false, Av1DefaultDistributions.Skip[0]); - // These sentinel symbols remain unread because pinned libaom classifies every GLOBALMV model other than - // TRANSLATION as non-translational for interpolation syntax, including the default identity model. + // Pinned libaom's is_nontrans_global_motion rejects only TRANSLATION, so the default identity model omits these + // sentinel symbols even though is_global_mv_block uses the separate greater-than-translation classification. writer.WriteSymbol((int)Av1InterpolationFilter.Smooth, Av1DefaultDistributions.SwitchableInterpolation[3]); writer.WriteSymbol((int)Av1InterpolationFilter.Sharp, Av1DefaultDistributions.SwitchableInterpolation[11]); using IMemoryOwner encoded = writer.Exit(); @@ -156,7 +179,52 @@ public class Av1InterFrameModeInfoTests } /// - /// Invokes the ref-struct mode parser for exception assertions that cannot capture its parameters directly. + /// Verifies every unidirectional and bidirectional compound reference-tree leaf through paired motion parsing. + /// + /// The zero-based normative compound reference pair. + /// The expected primary retained-reference label. + /// The expected secondary retained-reference label. + [Theory] + [InlineData(0, (int)Av1ReferenceFrameType.Backward, (int)Av1ReferenceFrameType.Alternate)] + [InlineData(1, (int)Av1ReferenceFrameType.Last, (int)Av1ReferenceFrameType.Last2)] + [InlineData(2, (int)Av1ReferenceFrameType.Last, (int)Av1ReferenceFrameType.Last3)] + [InlineData(3, (int)Av1ReferenceFrameType.Last, (int)Av1ReferenceFrameType.Golden)] + [InlineData(4, (int)Av1ReferenceFrameType.Last, (int)Av1ReferenceFrameType.Backward)] + [InlineData(5, (int)Av1ReferenceFrameType.Last2, (int)Av1ReferenceFrameType.Alternate2)] + [InlineData(6, (int)Av1ReferenceFrameType.Last3, (int)Av1ReferenceFrameType.Alternate)] + [InlineData(7, (int)Av1ReferenceFrameType.Golden, (int)Av1ReferenceFrameType.Alternate)] + public void ReadInterFrameModeInfoReadsCompoundReferencePair( + int pairIndex, + int expectedPrimary, + int expectedSecondary) + { + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(); + ObuFrameHeader frameHeader = CreateFrameHeader(); + frameHeader.ReferenceMode = ObuReferenceMode.ReferenceModeSelect; + using Av1TileReader tileReader = new(Configuration.Default, sequenceHeader, frameHeader); + Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block8x8, Point.Empty); + using Av1SymbolWriter writer = new(Configuration.Default, 8, updateCdf: true); + writer.WriteSymbol(false, Av1DefaultDistributions.Skip[0]); + writer.WriteSymbol(true, Av1DefaultDistributions.IntraInter[0]); + writer.WriteSymbol(true, Av1DefaultDistributions.CompInter[1]); + WriteCompoundReferencePair(writer, pairIndex); + writer.WriteSymbol(0, Av1DefaultDistributions.InterCompoundMode[0]); + + using IMemoryOwner encoded = writer.Exit(); + Memory encodedMemory = encoded.Memory; + + ReadInterFrameModeInfo(tileReader, encodedMemory, modeInfo); + + Assert.Equal((Av1ReferenceFrameType)expectedPrimary, modeInfo.ReferenceFrames[0]); + Assert.Equal((Av1ReferenceFrameType)expectedSecondary, modeInfo.ReferenceFrames[1]); + Assert.Equal(Av1PredictionMode.NearestNearestMotionVector, modeInfo.YMode); + Assert.Equal(default(Av1MotionVector), modeInfo.MotionVectors[0]); + Assert.Equal(default(Av1MotionVector), modeInfo.MotionVectors[1]); + Assert.Equal(Av1CompoundType.Average, modeInfo.CompoundType); + } + + /// + /// Invokes the ref-struct mode parser with one available above neighbor. /// /// The tile reader. /// The range-coded block-prefix symbols. @@ -179,6 +247,63 @@ public class Av1InterFrameModeInfoTests tileReader.ReadInterFrameModeInfo(ref decoder, ref partitionInfo, new Av1TileInfo(0, 0, tileReader.FrameHeader)); } + /// + /// Invokes the ref-struct mode parser without spatial neighbors. + /// + private static void ReadInterFrameModeInfo( + Av1TileReader tileReader, + Memory encoded, + Av1BlockModeInfo modeInfo) + { + Av1SuperblockInfo superblockInfo = new(tileReader.FrameInfo, Point.Empty); + Av1PartitionInfo partitionInfo = new(modeInfo, superblockInfo, false, Av1PartitionType.None); + Av1SymbolDecoder decoder = new(Configuration.Default, encoded.Span, 0, updateCdf: true); + tileReader.ReadInterFrameModeInfo(ref decoder, ref partitionInfo, new Av1TileInfo(0, 0, tileReader.FrameHeader)); + } + + /// + /// Writes one complete compound-reference tree leaf using the neutral no-neighbor contexts. + /// + private static void WriteCompoundReferencePair(Av1SymbolWriter writer, int pairIndex) + { + bool bidirectional = pairIndex >= 4; + writer.WriteSymbol(bidirectional, Av1DefaultDistributions.CompoundReferenceType[2]); + if (!bidirectional) + { + bool backwardPair = pairIndex == 0; + writer.WriteSymbol(backwardPair, Av1DefaultDistributions.UnidirectionalCompoundReference[1][0]); + if (!backwardPair) + { + bool last3OrGolden = pairIndex >= 2; + writer.WriteSymbol(last3OrGolden, Av1DefaultDistributions.UnidirectionalCompoundReference[1][1]); + if (last3OrGolden) + { + writer.WriteSymbol(pairIndex == 3, Av1DefaultDistributions.UnidirectionalCompoundReference[1][2]); + } + } + + return; + } + + bool last3OrGoldenForward = pairIndex >= 6; + writer.WriteSymbol(last3OrGoldenForward, Av1DefaultDistributions.CompoundReference[1][0]); + if (last3OrGoldenForward) + { + writer.WriteSymbol(pairIndex == 7, Av1DefaultDistributions.CompoundReference[1][2]); + } + else + { + writer.WriteSymbol(pairIndex == 5, Av1DefaultDistributions.CompoundReference[1][1]); + } + + bool alternateBackward = pairIndex >= 6; + writer.WriteSymbol(alternateBackward, Av1DefaultDistributions.CompoundBackwardReference[1][0]); + if (!alternateBackward) + { + writer.WriteSymbol(pairIndex == 5, Av1DefaultDistributions.CompoundBackwardReference[1][1]); + } + } + /// /// Creates the monochrome 64x64 sequence geometry used by direct mode-prefix tests. /// @@ -203,7 +328,8 @@ public class Av1InterFrameModeInfoTests /// /// The initialized frame header. private static ObuFrameHeader CreateFrameHeader() - => new() + { + ObuFrameHeader frameHeader = new() { FrameType = ObuFrameType.InterFrame, ModeInfoColumnCount = 16, @@ -211,4 +337,9 @@ public class Av1InterFrameModeInfoTests CodedLossless = true, AllowScreenContentTools = false, }; + + frameHeader.TilesInfo.TileColumnStartModeInfo[1] = frameHeader.ModeInfoColumnCount; + frameHeader.TilesInfo.TileRowStartModeInfo[1] = frameHeader.ModeInfoRowCount; + return frameHeader; + } } diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReconstructionConformanceTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReconstructionConformanceTests.cs index 7e75fce67..896e1ebf6 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReconstructionConformanceTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReconstructionConformanceTests.cs @@ -128,6 +128,16 @@ public class Av1ReconstructionConformanceTests /// private const int ProgressiveFirstLayerSize = 55; + /// + /// The displayed width and height of the independent compound image sequence. + /// + private const int AverageCompoundFixtureSize = 80; + + /// + /// The number of presented frames in the independent compound image sequence. + /// + private const int AverageCompoundFixtureFrameCount = 19; + /// /// The hardware configurations covering the available vector widths and the scalar color-conversion fallback. /// @@ -423,6 +433,172 @@ public class Av1ReconstructionConformanceTests Assert.Single(allocator.ReturnLog, returned => returned.HashCodeOfBuffer == temporalMotionField.HashCodeOfBuffer); } + /// + /// Verifies exact native reconstruction and presentation for a genuine pinned-libavif image sequence that uses + /// equal-weight compound prediction. + /// + [Fact] + public void DecodeRealLibavifSequenceWithEqualAverageCompoundMatchesPinnedReferences() + => FeatureTestRunner.RunWithHwIntrinsicsFeature( + ValidateAverageCompoundSequenceWithDefaultConfiguration, + ReconstructionConfigurations); + + /// + /// Verifies the complete compound sequence through a constrained allocator. + /// + [Fact] + [ValidateDisposedMemoryAllocations] + public void DecodeRealLibavifSequenceWithEqualAverageCompoundUsesContiguousPlanes() + { + TestMemoryAllocator allocator = new() { BufferCapacityInBytes = 1_024 }; + allocator.EnableNonThreadSafeLogging(); + Configuration configuration = Configuration.Default.Clone(); + configuration.MemoryAllocator = allocator; + + ValidateAverageCompoundSequence(configuration, null); + + Assert.Contains(allocator.AllocationLog, request => request.ElementType.Name == "RetainedMotionFieldEntry"); + Assert.Contains(allocator.AllocationLog, request => request.ElementType.Name == "TemporalMotionFieldEntry"); + Assert.Equal(allocator.AllocationLog.Count, allocator.ReturnLog.Count); + Assert.All( + allocator.AllocationLog, + allocation => Assert.Single( + allocator.ReturnLog, + returned => returned.HashCodeOfBuffer == allocation.HashCodeOfBuffer)); + } + + /// + /// Runs the exact compound-sequence comparisons with the default configuration. + /// + private static void ValidateAverageCompoundSequenceWithDefaultConfiguration() + { + byte[] presentationBytes = TestFile.Create(TestImages.Heif.Av1AverageCompoundSequencePresentationReference).Bytes; + using Image presentationReference = Image.Load(presentationBytes); + + ValidateAverageCompoundSequence(Configuration.Default, presentationReference.Frames.RootFrame); + } + + /// + /// Validates the complete compound sequence with the requested allocator and optional presentation reference. + /// + /// The decoder configuration. + /// The exact final presented frame, or . + private static void ValidateAverageCompoundSequence( + Configuration configuration, + ImageFrame presentationReference) + { + byte[] fileBytes = TestFile.Create(TestImages.Heif.Av1AverageCompoundSequenceAvif).Bytes; + byte[] referenceBytes = TestFile.Create(TestImages.Heif.Av1AverageCompoundSequenceNativeReference).Bytes; + ReadOnlySpan fileHeader = + "YUV4MPEG2 W80 H80 F25:1 Ip A0:0 C444 XYSCSS=444 XCOLORRANGE=LIMITED\n"u8; + + ReadOnlySpan frameHeader = "FRAME\n"u8; + + ReadOnlySpan nativeReference = referenceBytes; + Assert.True(nativeReference.StartsWith(fileHeader)); + nativeReference = nativeReference[fileHeader.Length..]; + Assert.True(nativeReference.StartsWith(frameHeader)); + nativeReference = nativeReference[frameHeader.Length..]; + Assert.Equal(AverageCompoundFixtureSize * AverageCompoundFixtureSize * 3, nativeReference.Length); + + HeifSequence sequence = ParseImageSequence(fileBytes); + HeifSequenceTrack track = sequence.ColorTrack; + int compoundBlockCount = 0; + int visibleFrameCount = 0; + bool nativeCompared = false; + bool presentationCompared = false; + + using Av1Decoder decoder = new(configuration); + for (int sampleIndex = 0; sampleIndex < track.Samples.Length; sampleIndex++) + { + HeifSequenceSample sample = track.Samples[sampleIndex]; + Span sampleData = fileBytes.AsSpan((int)sample.Offset, sample.Length); + if (sample.IsHidden) + { + decoder.DecodeSequenceReference( + sampleData, + track.CicpProfile, + track.Av1CodecConfiguration); + + continue; + } + + ImageFrame decodedFrame; + try + { + decodedFrame = decoder.DecodeSequenceFrame( + sampleData, + track.CicpProfile, + track.Av1CodecConfiguration); + } + catch (InvalidImageContentException exception) + { + throw new InvalidImageContentException($"The pinned compound fixture failed at sample {sampleIndex}.", exception); + } + + using ImageFrame frame = decodedFrame; + + ObuSequenceHeader sequenceHeader = Assert.IsType(decoder.SequenceHeader); + _ = Assert.IsType(decoder.FrameHeader); + Av1FrameBuffer frameBuffer = Assert.IsType>(decoder.FrameBuffer); + Av1FrameInfo frameInfo = Assert.IsType(decoder.FrameInfo); + + // Inter prediction addresses padding with one base span and a logical row stride. The frame owner must + // preserve that contract even when the configured allocator would ordinarily split a large buffer. + Assert.Equal(1, frameBuffer.BufferY!.FastMemoryGroup.Count); + Assert.Equal(1, frameBuffer.BufferCb!.FastMemoryGroup.Count); + Assert.Equal(1, frameBuffer.BufferCr!.FastMemoryGroup.Count); + + int superblockSizeLog2 = sequenceHeader.SuperblockSizeLog2; + int superblockColumnCount = Av1Math.AlignPowerOf2(sequenceHeader.MaxFrameWidth, superblockSizeLog2) >> superblockSizeLog2; + int superblockRowCount = Av1Math.AlignPowerOf2(sequenceHeader.MaxFrameHeight, superblockSizeLog2) >> superblockSizeLog2; + + for (int superblockRow = 0; superblockRow < superblockRowCount; superblockRow++) + { + for (int superblockColumn = 0; superblockColumn < superblockColumnCount; superblockColumn++) + { + Av1SuperblockInfo superblockInfo = frameInfo.GetSuperblock(new Point(superblockColumn, superblockRow)); + foreach (Av1BlockModeInfo modeInfo in superblockInfo.GetModeInfos()) + { + if (modeInfo.ReferenceFrames[1] <= Av1ReferenceFrameType.Intra) + { + continue; + } + + Assert.Equal(Av1CompoundType.Average, modeInfo.CompoundType); + compoundBlockCount++; + } + } + } + + if (visibleFrameCount == AverageCompoundFixtureFrameCount - 1) + { + Assert.Equal(AverageCompoundFixtureSize, frameBuffer.Width); + Assert.Equal(AverageCompoundFixtureSize, frameBuffer.Height); + Assert.Equal(Av1BitDepth.EightBit, frameBuffer.BitDepth); + Assert.Equal(Av1ColorFormat.Yuv444, frameBuffer.ColorFormat); + AssertNativePlanesEqual(decoder, frameBuffer, nativeReference); + nativeCompared = true; + + if (presentationReference is not null) + { + ImageSimilarityReport report = + ImageComparer.Exact.CompareImagesOrFrames(visibleFrameCount, presentationReference, frame); + + Assert.True(report.IsEmpty, report.ToString()); + presentationCompared = true; + } + } + + visibleFrameCount++; + } + + Assert.Equal(AverageCompoundFixtureFrameCount, visibleFrameCount); + Assert.NotEqual(0, compoundBlockCount); + Assert.True(nativeCompared); + Assert.Equal(presentationReference is not null, presentationCompared); + } + /// /// Verifies lossless syntax, residual reconstruction, and exact native samples against scalar libaom for /// independently encoded eight-, ten-, and twelve-bit AVIF images. @@ -1095,6 +1271,37 @@ public class Av1ReconstructionConformanceTests ImageComparer.Exact.VerifySimilarity(presentationReference, image); } + /// + /// Parses the selected image-sequence tracks from a complete HEIF fixture. + /// + /// The complete HEIF file. + /// The bounded image-sequence model. + private static HeifSequence ParseImageSequence(byte[] fileBytes) + { + using MemoryStream stream = new(fileBytes, false); + Span scratch = stackalloc byte[32]; + while (stream.Position < stream.Length) + { + long boxLength = HeifBoxReader.ReadHeader( + stream, + stream.Length, + scratch, + out Heif4CharCode boxType, + topLevel: true); + + long boxStart = stream.Position; + if (boxType == Heif4CharCode.Moov) + { + HeifSequenceParser parser = new(new DecoderOptions { MaxFrames = 32 }); + return parser.Parse(stream, boxLength); + } + + stream.Position = checked(boxStart + boxLength); + } + + throw new InvalidImageContentException("The HEIF fixture contains no image sequence."); + } + /// /// Validates every lossless native fixture under the hardware configuration selected by /// . diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReferenceMotionVectorsTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReferenceMotionVectorsTests.cs index 941beebfc..fb7d134d0 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReferenceMotionVectorsTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReferenceMotionVectorsTests.cs @@ -11,7 +11,7 @@ using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; /// -/// Verifies the spatial, temporal, global, and extension rules used to derive single-reference AV1 motion vectors. +/// Verifies the spatial, temporal, global, and extension rules used to derive single- and compound-reference AV1 motion vectors. /// [Trait("Format", "Avif")] public class Av1ReferenceMotionVectorsTests @@ -348,6 +348,142 @@ public class Av1ReferenceMotionVectorsTests Assert.Equal(0, referenceMotionVectors.ModeContext); } + /// + /// Verifies that compound candidates retain their primary and secondary vectors through weighting, sorting, and DRL access. + /// + [Fact] + public void BuildRetainsPairedCompoundCandidates() + { + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(enableTemporalMotionVectors: false); + ObuFrameHeader frameHeader = CreateFrameHeader(orderHint: 0, useReferenceFrameMotionVectors: false); + using Av1FrameInfo frameInfo = new(sequenceHeader); + FillFrameWithIntraBlocks(frameInfo, sequenceHeader); + + Av1MotionVector abovePrimary = new(8, 16); + Av1MotionVector aboveSecondary = new(24, 32); + Av1BlockModeInfo above = AddModeInfo( + frameInfo, + sequenceHeader, + new Point(8, 4), + Av1BlockSize.Block16x16, + Av1ReferenceFrameType.Last, + abovePrimary, + Av1PredictionMode.NewNewMotionVector); + + above.ReferenceFrames[1] = Av1ReferenceFrameType.Backward; + above.MotionVectors[1] = aboveSecondary; + + Av1MotionVector leftPrimary = new(40, 48); + Av1MotionVector leftSecondary = new(56, 64); + Av1BlockModeInfo left = AddModeInfo( + frameInfo, + sequenceHeader, + new Point(4, 8), + Av1BlockSize.Block16x16, + Av1ReferenceFrameType.Last, + leftPrimary, + Av1PredictionMode.NearestNearestMotionVector); + + left.ReferenceFrames[1] = Av1ReferenceFrameType.Backward; + left.MotionVectors[1] = leftSecondary; + + Av1SuperblockInfo superblockInfo = frameInfo.GetSuperblock(Point.Empty); + Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block16x16, new Point(8, 8)); + Av1PartitionInfo partitionInfo = new(modeInfo, superblockInfo, true, Av1PartitionType.None) + { + ColumnIndex = 8, + RowIndex = 8, + }; + + Av1TileInfo tileInfo = new(0, 0, frameHeader); + partitionInfo.ComputeBoundaryOffsets(sequenceHeader, frameHeader, tileInfo); + Av1ReferenceMotionVectors referenceMotionVectors = new(); + + referenceMotionVectors.Build( + ref partitionInfo, + tileInfo, + frameInfo, + sequenceHeader, + frameHeader, + Av1ReferenceFrameType.Last, + Av1ReferenceFrameType.Backward); + + Assert.Equal(2, referenceMotionVectors.Count); + Assert.Equal(abovePrimary, referenceMotionVectors.Candidates[0]); + Assert.Equal(aboveSecondary, referenceMotionVectors.CompoundCandidates[0]); + Assert.Equal(leftPrimary, referenceMotionVectors.Candidates[1]); + Assert.Equal(leftSecondary, referenceMotionVectors.CompoundCandidates[1]); + Assert.Equal((ushort)656, referenceMotionVectors.Weights[0]); + Assert.Equal((ushort)656, referenceMotionVectors.Weights[1]); + Assert.Equal(abovePrimary, referenceMotionVectors.GetCompoundNearestReference(0)); + Assert.Equal(aboveSecondary, referenceMotionVectors.GetCompoundNearestReference(1)); + Assert.Equal(leftPrimary, referenceMotionVectors.GetCompoundNearReference(0, 0)); + Assert.Equal(leftSecondary, referenceMotionVectors.GetCompoundNearReference(0, 1)); + Assert.Equal(abovePrimary, referenceMotionVectors.GetCompoundNewReference(0, 0)); + Assert.Equal(aboveSecondary, referenceMotionVectors.GetCompoundNewReference(0, 1)); + } + + /// + /// Verifies the positional compound fallback assembled from independent exact-reference neighbor lists. + /// + [Fact] + public void BuildExtendsCompoundStackWithPairedFallbacks() + { + ObuSequenceHeader sequenceHeader = CreateSequenceHeader(enableTemporalMotionVectors: false); + ObuFrameHeader frameHeader = CreateFrameHeader(orderHint: 0, useReferenceFrameMotionVectors: false); + using Av1FrameInfo frameInfo = new(sequenceHeader); + FillFrameWithIntraBlocks(frameInfo, sequenceHeader); + + Av1MotionVector above = new(8, 16); + Av1MotionVector left = new(24, 32); + AddModeInfo( + frameInfo, + sequenceHeader, + new Point(8, 4), + Av1BlockSize.Block16x16, + Av1ReferenceFrameType.Last, + above, + Av1PredictionMode.NearestMotionVector); + + AddModeInfo( + frameInfo, + sequenceHeader, + new Point(4, 8), + Av1BlockSize.Block16x16, + Av1ReferenceFrameType.Backward, + left, + Av1PredictionMode.NearestMotionVector); + + Av1SuperblockInfo superblockInfo = frameInfo.GetSuperblock(Point.Empty); + Av1BlockModeInfo modeInfo = new(Av1BlockSize.Block16x16, new Point(8, 8)); + Av1PartitionInfo partitionInfo = new(modeInfo, superblockInfo, true, Av1PartitionType.None) + { + ColumnIndex = 8, + RowIndex = 8, + }; + + Av1TileInfo tileInfo = new(0, 0, frameHeader); + partitionInfo.ComputeBoundaryOffsets(sequenceHeader, frameHeader, tileInfo); + Av1ReferenceMotionVectors referenceMotionVectors = new(); + + referenceMotionVectors.Build( + ref partitionInfo, + tileInfo, + frameInfo, + sequenceHeader, + frameHeader, + Av1ReferenceFrameType.Last, + Av1ReferenceFrameType.Backward); + + Assert.Equal(2, referenceMotionVectors.Count); + Assert.Equal(above, referenceMotionVectors.Candidates[0]); + Assert.Equal(left, referenceMotionVectors.CompoundCandidates[0]); + Assert.Equal(left, referenceMotionVectors.Candidates[1]); + Assert.Equal(above, referenceMotionVectors.CompoundCandidates[1]); + Assert.Equal((ushort)2, referenceMotionVectors.Weights[0]); + Assert.Equal((ushort)2, referenceMotionVectors.Weights[1]); + } + /// /// Creates the monochrome 128-by-128 sequence geometry shared by reference-motion-vector tests. /// diff --git a/tests/ImageSharp.Tests/TestImages.cs b/tests/ImageSharp.Tests/TestImages.cs index ada332e4a..1995ead7e 100644 --- a/tests/ImageSharp.Tests/TestImages.cs +++ b/tests/ImageSharp.Tests/TestImages.cs @@ -1357,6 +1357,9 @@ public static class TestImages public const string Av1Progressive8BitPayload = "Heif/Av1/Conformance/libavif-progressive-draw-points-8b.bit"; public const string Av1Progressive8BitReference = "Heif/Av1/Conformance/libavif-progressive-draw-points-8b-libaom-y4m.yuv"; public const string Av1Progressive8BitPresentationReference = "Heif/Av1/Conformance/libavif-progressive-draw-points-8b.png"; + public const string Av1AverageCompoundSequenceAvif = "Heif/Av1/Conformance/libavif-webp-logo-average-compound.avif"; + public const string Av1AverageCompoundSequenceNativeReference = "Heif/Av1/Conformance/libavif-webp-logo-average-compound-libaom.y4m"; + public const string Av1AverageCompoundSequencePresentationReference = "Heif/Av1/Conformance/libavif-webp-logo-average-compound-libavif.png"; public const string Av1Cdef8BitPayload = "Heif/Av1/Conformance/libaom-cdef-kodim23-8b.bit"; public const string Av1Cdef8BitReference = "Heif/Av1/Conformance/libaom-cdef-kodim23-8b-libaom.yuv"; public const string Av1Cdef8BitAvif = "Heif/Av1/Conformance/libavif-cdef-kodim23-8b.avif"; diff --git a/tests/Images/Input/Heif/Av1/Conformance/README.md b/tests/Images/Input/Heif/Av1/Conformance/README.md index e51df3c32..d0d579db6 100644 --- a/tests/Images/Input/Heif/Av1/Conformance/README.md +++ b/tests/Images/Input/Heif/Av1/Conformance/README.md @@ -36,6 +36,7 @@ The reference builds use `AOM_TARGET_CPU=generic` and disable libyuv. Native rec | `*-lossless-*` | Lossless quantization, reversible transforms, and exact presentation | | `*-film-grain-*` | Full and restricted range, monochrome, identity matrix, 8/10/12-bit synthesis, overlap, and odd frame dimensions | | `libavif-progressive-draw-points-8b` | A real two-layer color item whose final frame uses single-reference inter reconstruction, plus its progressive auxiliary alpha item | +| `libavif-webp-logo-average-compound` | A 19-frame YUV444 image sequence whose retained references reach equal-weight compound inter reconstruction | The corresponding tests also assert the syntax required by each family before comparing output. This prevents an inactive tool or an incorrectly substituted stream from passing solely because its final pixels happen to match. @@ -51,6 +52,23 @@ The primary color item's `a1lx` property divides its logical 72-byte AV1 payload Exact pinned libaom decodes the corrected logical payload into two 33x11 YUV444 frames. Both frames' 1,089 color samples match the corresponding first three planes of the pinned libavif YUV444-alpha outputs exactly. The retained Y4M contains both progressive YUV444-alpha frames, and the PNG contains pinned libavif's final RGBA presentation. The production-path test selects the second native frame, requires inter-coded blocks in the final ImageSharp frame, and compares both native color and final presentation without a tolerance. +## Equal-average compound fixture + +The `libavif-webp-logo-average-compound.avif` fixture was encoded from the pinned libavif tree's `tests/data/webp_logo_animated.y4m` source. The source SHA-256 is `0872208D9C19B68B10A1647FA6849CFC4E2B21A19561ACD672E0629C70EFACA2`. It was generated with: + +```text +./avifenc -j 1 -c aom -s 4 -q 80 -a enable-dist-wtd-comp=0 -a enable-masked-comp=0 -a enable-interintra-comp=0 -a enable-obmc=0 -a enable-warped-motion=0 -a enable-global-motion=0 tests/data/webp_logo_animated.y4m libavif-webp-logo-average-compound.avif +``` + +Pinned scalar libavif generated the retained references with: + +```text +./avifdec -j 1 -c aom --index 18 libavif-webp-logo-average-compound.avif libavif-webp-logo-average-compound-libaom.y4m +./avifdec -j 1 -c aom --index 18 libavif-webp-logo-average-compound.avif libavif-webp-logo-average-compound-libavif.png +``` + +The AVIF SHA-256 is `7919049D367EEDB7C965E170309D6759660DDBFD4BB1AEF9496F9D66E314846A`. The retained frame-18 Y4M SHA-256 is `41FF2408DEB473D5483F3398882DF7F7AB6C7D376561C19798881595EB0C5C0C`, and the frame-18 PNG SHA-256 is `BCFABC1E1C7E17D8ECB40569849A04FFAC6CA1FCDF613F217B33816CA47337AC`. The test decodes every preceding hidden and shown sample to establish the same retained-reference state before comparing all native Y, U, and V samples and the final RGBA presentation. + ## Updating fixtures Do not create conformance references with ImageSharp. Generate both the native-plane and presentation references with an independent decoder, record the exact upstream revisions and source license, and preserve exact comparisons. A new tool-specific fixture should demonstrate that the relevant syntax is active and should be no larger than required to cover that behavior. diff --git a/tests/Images/Input/Heif/Av1/Conformance/libavif-webp-logo-average-compound-libaom.y4m b/tests/Images/Input/Heif/Av1/Conformance/libavif-webp-logo-average-compound-libaom.y4m new file mode 100644 index 000000000..37be945f1 --- /dev/null +++ b/tests/Images/Input/Heif/Av1/Conformance/libavif-webp-logo-average-compound-libaom.y4m @@ -0,0 +1,3 @@ +YUV4MPEG2 W80 H80 F25:1 Ip A0:0 C444 XYSCSS=444 XCOLORRANGE=LIMITED +FRAME +?@????????UVTTTTTTTTUvyTTtyT룰;=AB=Tdtttttv_T>BBBBBBBBBBBBBBBBBBBBB\tsXT\}}}|zyyyttttqqqqqqqqqqqqqqqqqqqqqqqqqqqrqoWU>KTWnvvvvvvvvvvvvvvvvnouonojnmnnkjgggggggggggggggggggggggggggggggcWTJBTTTUTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTB?BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB>뀀!)*)#"")37673(""55?FFF?73"ffffffffff")5ALRWRLB7*"\[[[[[[[[[""5ALWbcaWK?3"\MBBBBBBO[")7FRbotj^QB7)"\NBBBBBBNZ+'")91),(!''((''('"().38612..))())))),3769BGGNTU[oo[^aJAAAAAAAAAAA@A@@@@@@@@@@D@DDDDDDDDDDDDDDD:>621*---.---33678AFGGMTU[oo[^aJAAAAAAAAAAA@@@@@@@@@@@DDDDDDDDDDDDDDDDDDDB>561113233187=8CGGGGNTU[oo[^aJAAAAAAAAAA@@@@@@@@@@D@DDDDDDDDDDDDDDDDDBBBB9==6667675=8BFGGGGGNTU[oo[^aJAAAAAAAA@@@@@@@@@@D@DDDDDDDDDDDDDDDDDDBBBBBBAB98=>=8BBFGGGGGFFMTU[oo[^aJAAAAA@A@@@@@@@@@@DDDDDDDDDDDDDDDDDCDBBBBBBBBBB@G@FAGHGGGGGGGFFMTU[oo[^_JAAA@A@@@@@@@@@@DDDDDDDDDDDDDDKNOKECBBBBBBBBB@@AGGGGGGGGGGGGGFFMTU[oo[^_JAAA@@@@@@@@@@D@DDDDDDDEEDFQXZ\[[\ZWQKBBBBA@@F@GGGGGGGGGFFFFFFFLTU[oo[^^J@@@@@@@@@@@D@DDDDDDDDDEFS[[\fgjigf[[[RCBC@GAGGGGGGGGGGFFFFFFCCLSU[oo[^^J@@@@@@@@@@DDDDDDDDDDDCH\[]q|~|q]Y\NF@FFGGGGGGGGFFFFFFCCAALRT[oo[^^J@@@@@@@@DDDDDDDDDDDDCM[Zm~mZZPGGGGGGGGGFFFFFFFCAAAALRT[oo[]]J@@@@@@D@DDDDDDDDDDDCI[]ss][MGGGGGGFFFFFFFCAAAAAALQT[oo[]]J@?@@D@DDDDDDDDDDDDDF][ut[]JGGGGFFFFFFCCAAAAAAALQS[oo[]]J@??DDDDDDDDDDDDDDDDT[m~m[TGGGGFFFFCCAAAAAAAAALPS[oo[]]J@D@DDDDDDDDDDDDDDDI[]~][HFFFFFFCAAAAAAAAAAGLPS[oo[]]J@DDDDDDDDDDDDDDDDDS[qq[SFFGHBBAAAAAAAAKJJJKOS[oo[]\JDDDDDDDDDDDDDDDCDCV[~~[\BFCBBAAAAAAAAAJJJJKNS[oo[]ZJDDDDDDDDDDDDDDDCCJ\ed[FCBBAAAAABBAJJJJJJKMS[oo[]ZJDDDDDDDDDDDDDBBBBH\kjZJBBAAAAABCIJJJIJJIJKS[oo[[ZJDDDDDDDDDDDDBBBBBN[nm[QBAAAAABIJJJJJIHHHJJS[oo[[YJDDDDDDDDDDBBBBBAAL[onZPBAAABBIJJJJIHHHHHIJS[oo[[YJDDDDDDDDBBBBBBBABM\jj[JAABBIJJJJIHHHHHHHHIS[oo[[YJDDDDDDDBBBBBBBBAAJ\edZMAKIIJJJJIHHHHHHFFHIS[oo[[ZIDDDCDBBBBBBBBBB@GG\\\WJIJJJJJJHHHHHHFFFFHHS[oo[[WIDDDCCBBBBBBBB@G@GGRZtq[SJIJJJJHHHHHHHFFFFFFHS[oo[[VIDCCBBBBBBBB@@@GGGGN\b~~b\JJIIIHHHHHHHFFFFFFFFHS[no[ZUIBBBBBBBBB@@G@GGGGGFT[q~qZYJJIIHHHHHHFFFFFFFFGFLSZoo[ZUIBBBBBBBB@G@GGGGGGGGIZ[xv][LIIHHHHHHFFFFFFFFFHHHGR[oo[ZUIBBBAAB@@@GGGGGGGGGGGKZ\wv]\QKHHHHHHHFFFFFFFFHHHHHHS[no[YUHBBBAA??GGGGGGGGFFFFFFS[\qq[\QKHHHHHHFFFFGGGGHHHHHHHIS[oo[YVGBBAAF@GGGGGGGGGFFFFFGFR[[atvb[[PHIHHHHFFFFFGGGGHHHHHHHHIS[oo[YVGB@G@GGGGGGGGGGFFFFFFCCAMY[Z]gpqqog]Z[YJHHHHHHFFFFFFFGHHHHHHHHHHIS[oo[ZVHA@GGGGGGGGGGFFFFFFCCAAAAAJS[Z[[[\\ZSQIHHHHHFFFFFFFFHHHHHHHHHHHHIS[oo[ZVHHHGGGGGGGGGFFFFFFCAAAAAAA?AIJPRRPJEHGHHHHFFFFFFFFHHHHHHHHHHHHHHIS[oo[ZUHHHGGGGGGFFFFFFFCCAAAAAAABBIJJJJJJIHHHHHFFFFFFFFHHHHHHHHHHHHHHHHIS[oo[ZTHHHGGGGGFFFFFFCCAAAAAAAAAAJJJJJJHHHHHHGGFFFFFFFHHHHHHHHHHHHHHHHHIS[oo[ZSGGGGGGGGFFFFFCAAAAAAAAAAIJJJJIIIHHHHGFFFFFFFFHHHHHHHHHHHHHHHHHHHIS[oo[YRGGGGGFFFFEECAAAAAAAAAJJJJJJJIHHHHGGFFFFFFFFHHHHHHHHHHHHHHHHHHHHHIS[oo[YQGGGFFFFFFCCAAAAAAAAAJJJJJJJHHHHHGFFFFFFFFFHHHHHHHHHHHHHHHHHHHHHHIS[op[VPGFFFFFFDAAAAAAAAAAAJJJJJIHHHHHHHFFFFFFFFHHHHHHHHHHHHHHHHHHHHHHHHHS[op[VPLFFFFFDAAAAAAAAAAJJJJJJHHHHHHHGFFFFFFFHHHHHHHHHHHHHHHHHHHHHHHHHHHY\r[YPLFFFCAAAAAAAAAKJJJJJJJHHHHHHFFFFFFFFGHHHHHHHHHHHHHHHHHHHHHHHHHHHHY\[[PQLLGFFGJAAKKKKKKKJJJJJHHHHHHFFFFFFFGHHHHHHHHHHHHHHHHHHHHHHHHHHHHK[[e[YQQQQQPKLLLLLLKKKKKKKKKJJIIHHGGFFFFHHHHHHHHHHHHHHHHHHHHHHHHHHHHHJW[f`[WRNRNMNSSLMMMMLSSSQQKQQQQQQQQQQQQPOOOOOOOOOOOOOOOOOOOOOOOOOOPOSV[`e[[[[[[Z[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[\[Z[cfcddcccccccccccccccccbbbccccccccbccccccccccccccccccccccccccccgvvvvvvvvvvpppppppppppnddddddlppkdddddelpuuuuupojjjjjnoottttttttttttttssssssuuy}}|vqqqqppppppppppppppppppppppppppppppppppu}}vputppqqrrrrrrrrrrrrrrssssssrrqqppppppppppppy~|vqppuvqrrrrrrrrrttttttttssssrrrrrrrrrrrrrrrrpprw~yvpnnpvtqsqojjggggggggggggggggggggggggggggggggghiqs|{tokplpspqsodddddddddddddddeehdhghhhhhhggggggggghhmrz|{vojimlnpzpqsidddddddddddddddefghhhhhhhhhgggggggggggglrz}{wsphemoopzzprsidddddddddddddddfgghhhhhhhhggggggggggggghmty|{wtplfeklopzzprtiddddddddddhdhdhghhhhhhhhggggggggggggggggmntyy~{ytspkfffklopzzprtiddddddddddhdhhhhhhhhhhgggggggggggggggggggilttyzz{{{|yutoplfffgklopzzprtiddddddddddhhhhhhhhhhhggggggggggggggggggggggkpptttusttnokeffffgjlopzzprtiddddddddhhhhhhhhhhhgggggggggggggggggggggggggglmpoooplkfffffghhklopzzprsidddddhdhhhhhhhhhhggggggggggggggggggggggggggggggfgigefffffgfiiiklopzzprridddhdhhhhhhhhhhgggggggggggggihlljihgggggggggfggfeeeefffgghgiiiklopzzprridddhhhhhhhhhhhgggggggggghjloopppoppligggggggfffffffffffjiiiiijlmopzzprrhddhhhhhhhhhhgggggggggggiloposvwwvtppoljgggffffffffffffiiiiiiiilmopzzprrhhhhhhhhhhhggggggggggggjpqsz~~zspplegefffffffffiiiiiiiijjklopzzprrhhhhhhhhhgggggggggggggkopxzqomeefffffffiiiiiiiijjjjkjopzzppphhhhhhhhgggggggggggggloqzzppjefffffiiiiiiiijjjjjjjiop{zppphhhhggggggggggggggggioq{zpphffffiiiiiiiijjjjjjijiop{zppphhhhggggggggggggggggopyypofghhiiiiiijjjjjjjjijjop{zppphgggggggggggggggggggortoliiiiiiijjjjjjjjjjhjlopzzppphgggggggggggggggggglpyyoliiiijjjjjjjjjjjiihilopzzppphggggggggggggggggggppopiijjjjjjjjjjjiihihklopzzppphggggggggggggggghgiotsqjjjjjjjjjjjihhhhhhllopzzppphggggggggggggggghgkowvqjjjjjjjjjjhhhhhhhillopzzppphggggggggggggggghhjqyypljjjjjjjhhhhhhhhhiklopzzppphgggggggggggggggggjpxypmjjjjjjhhhhhhhhhhhhlopzzppphgggggggggggggggggkpvwpkjjjjhhhhhhhhhhhhhhlopzzppphgggggggggggggggggipstpjiihhhhhhhhhhhhhiiilopzzppphggggggggggggggggeepqpqhhhhhhhhhhhhhhiiiiilopzzppphggggggggggggggffeelp{~|qohhhhhhhhhhhhhiiiiiikopzzppphggggggggggggggffeelpsspkhhhhhhhhhhhiiiiiiiijopzzppphgggggggggggffffffffoqzvpohhhhhhhhhhhhiiiiiiiijopzzppphgggggggggffffffffffhpq}}ppihhhhhhhhhiiiiiiiihhhiopzzppphgggggggggefffffffffikqr||spihhhhhhhhiiiiiiiihhhhhiopzzpppigggggggeeeeffffefhiiilppzzpqlhhhhhhhiiiiiiiihhhhhhhiopzzpppigggggggeeefffffiiihiiilpoqz|rppihhhhhhiiiiiiiihhhhhhhhhiopzzppohffffffffffffghhiiiiiiijjnoppuuz{uvqopnjhhhhhhiiiiiiiihhhhhhhhhhiopzzppmiggfffffffffgghhiiiiijjjjjlnpqppppponigihhhhiiiiiiiihhhhhhhhhhhhiopzzqpliffffffffffgiiiiiiijjjjjjjjjikjllljihhhhhhiiiiiiiihhhhhhhhhhhhhhiopzzqplifffffffghhiiiiiiijjjjjjjjjihhhhhhhhhhhiiiiiiiiihhhhhhhhhhhhhhhhiopzzqplifffffffhiiiiiiijjjjjjjjjiiihhhhhhhhhhiiiiiiiiihhhhhhhhhhhhhhhhhiopzzpplifffffhgiiiiiijjjjjjjjjjihhhhhhhhhhhiiiiiiiiihhhhhhhhhhhhhhhhhhhiopzzppmifffiiiiiiiijjjjjjjjjhhhhhhhhhhhhhhiiiiiiiihhhhhhhhhhhhhhhhhhhhhiopz{ppmiffiiiiiiijjjjjjjjjjhhhhhhhhhhhhhiiiiiiiiihhhhhhhhhhhhhhhhhhhhhhiopzzpoliiiiiiiijjjjjjjjjjjhhhhhhhhhhhhhiiiiiiiihhhhhhhhhhhhhhhhhhhhhhhhhopzzpolkiiiiiiijjjjjjjjjhhhhhhhhhhhhhiiiiiiiiiihhhhhhhhhhhhhhhhhhhhhhhhhnozpnlkjiiiiiijjjjjjhhhhhhhhhhhhhhiiiiiiiiihhhhhhhhhhhhhhhhhhhhhhhhhhhhnpppljkjiiiiijjhhhhhhhhhhhhhhhhhhiiiiiiihhhhhhhhhhhhhhhhhhhhhhhhhhhhhkopupniijjiijklllllllllllllllllglhhhijjiihhhhhhhhhhhhhhhhhhhhhhhhhhhhkppwspomnlnnmmlnnnnnnlllmmnmmmimmmmiimljjiiiiiiiiiiiiiiiiiiiiiiiiiihoppsupopqppppppppppppppppppppppppqppppppppppppppppppppppppppppppppppotwttttttttttttttttttttttsttttttttsttttttttttttttttttttttttttttv \ No newline at end of file diff --git a/tests/Images/Input/Heif/Av1/Conformance/libavif-webp-logo-average-compound-libavif.png b/tests/Images/Input/Heif/Av1/Conformance/libavif-webp-logo-average-compound-libavif.png new file mode 100644 index 000000000..046e57854 --- /dev/null +++ b/tests/Images/Input/Heif/Av1/Conformance/libavif-webp-logo-average-compound-libavif.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:bcfabc1e1c7e17d8ecb40569849a04ffac6ca1fcdf613f217b33816ca47337ac +size 4728 diff --git a/tests/Images/Input/Heif/Av1/Conformance/libavif-webp-logo-average-compound.avif b/tests/Images/Input/Heif/Av1/Conformance/libavif-webp-logo-average-compound.avif new file mode 100644 index 000000000..2d6139d18 --- /dev/null +++ b/tests/Images/Input/Heif/Av1/Conformance/libavif-webp-logo-average-compound.avif @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7919049d367eedb7c965e170309d6759660ddbfd4bb1aef9496f9d66e314846a +size 6530