diff --git a/HEIF_IMPLEMENTATION_PLAN.md b/HEIF_IMPLEMENTATION_PLAN.md index 07c926331..dfae2aea6 100644 --- a/HEIF_IMPLEMENTATION_PLAN.md +++ b/HEIF_IMPLEMENTATION_PLAN.md @@ -29,9 +29,11 @@ Checkboxes may be marked complete only when the implementation and the verificat ## Delivery dashboard -Last reconciled with the source tree on 2026-08-29 against production checkpoint `c5637ea0187df35b385bf43e2fe85cd955f01099`. Committed checkpoints include the AV1 transform architecture, OBU framing, intra-block copy, 12-profile reconstruction matrix, layered-item properties, layered reference/header/CDF/motion-field state, inter-frame intra blocks, SIMD-first translational prediction, complete single-reference inter reconstruction, compound reference trees and modes, paired reference-MV derivation, reference-dependent bounded sequence decoding, allocation-free SIMD-first equal averaging, selected inter-intra prediction, selectable compound blending, OBMC, scaled-reference reconstruction, local warped prediction, and non-translational global prediction. The global-warp checkpoint reuses the committed affine filter for valid rotation/zoom and affine `GLOBALMV` models while preserving translational fallback for identity, translation, invalid shear, scaled references, integer-only frames, OBMC, and sub-8x8 plane predictions. Its genuine two-frame 256x256 bounded AVIF contains seven pinned-libaom-traced rotation/zoom `GLOBALMV` blocks and passes exact final pinned-libaom YUV444 planes, exact pinned-libavif RGBA presentation, normal and scalar `FeatureTestRunner` execution, constrained allocation, and balanced ownership tracking. A direct production-branch test drives both matrices of `GLOBAL_GLOBALMV` through compound averaging at 8, 10, and 12 bits. Both source targets build with zero warnings and errors; the `net10.0` test project builds with zero errors and pre-existing warnings; 84 focused `net10.0` cases pass without failures or skips; and the documentation and diff checks are complete. This dashboard is the authoritative delivery order. The detailed phase checklists below provide subsystem evidence; they do not override the current-stage marker or permit work to skip ahead. +Last reconciled with the source tree on 2026-08-29 against production checkpoint `1c58d855f70b024170ced9eb0a7005f0f9c955ad`. Committed checkpoints include the AV1 transform architecture, OBU framing, intra-block copy, 12-profile reconstruction matrix, layered-item properties, layered reference/header/CDF/motion-field state, inter-frame intra blocks, SIMD-first translational prediction, complete single-reference inter reconstruction, compound reference trees and modes, paired reference-MV derivation, reference-dependent bounded sequence decoding, allocation-free SIMD-first equal averaging, selected inter-intra prediction, selectable compound blending, OBMC, scaled-reference reconstruction, local warped prediction, non-translational global prediction, and official motion-vector conformance. This dashboard is the authoritative delivery order. The detailed phase checklists below provide subsystem evidence; they do not override the current-stage marker or permit work to skip ahead. -The working tree contains a complete, verified official motion-vector conformance checkpoint awaiting commit. The official IVF has SHA-1 `F064290D7FCD3B3DE19020E8AEC6C43C88D3A505`, matching the pinned libaom test-data manifest, and SHA-256 `222A9050059B254DAB17CFB802FF829C778E3F93AF18961A622C8268576C1395`; its pinned-libaom Y4M has SHA-256 `D97AC78C81782CF1507549368047769DC677DBE205706458D1EE9C807DE6EC78`. Both source targets build with zero warnings and errors, the `net10.0` test-project analyzer build completes with zero errors and 1,014 pre-existing repository warnings, Roslynk reports zero compiler errors, 3,985 focused `net10.0` cases pass without failures or skips, and `git diff --check` is clean. +Commit `1c58d855f70b024170ced9eb0a7005f0f9c955ad` records the complete official motion-vector conformance checkpoint. The official IVF has SHA-1 `F064290D7FCD3B3DE19020E8AEC6C43C88D3A505`, matching the pinned libaom test-data manifest, and SHA-256 `222A9050059B254DAB17CFB802FF829C778E3F93AF18961A622C8268576C1395`; its pinned-libaom Y4M has SHA-256 `D97AC78C81782CF1507549368047769DC677DBE205706458D1EE9C807DE6EC78`. Both source targets build with zero warnings and errors, the `net10.0` test-project analyzer build completes with zero errors and 1,014 pre-existing repository warnings, Roslynk reports zero compiler errors, 3,985 focused `net10.0` cases pass without failures or skips, and `git diff --check` is clean. + +The working tree contains a complete, verified selected-spatial-layer presentation checkpoint awaiting commit. An essential `lsel` selecting the genuine 40x40 base layer of the committed 40x40-to-80x80 progressive AVIF returns that exact pinned-libaom native reconstruction, then scales its YUV444 planes to the 80x80 `ispe` extent before color conversion and matches pinned libavif's RGBA output exactly. Retained reconstructed planes remain at coded dimensions; presentation planes use group-safe allocator-owned rows and are returned exactly once under a 1,024-byte constrained allocator. Portable `Vector128` and scalar two-times filters match pinned libyuv edge, quarter-sample, and rounding results at 8 and 12 bits through `FeatureTestRunner`. Both source targets build with zero warnings and errors, the `net10.0` test-project analyzer build completes with zero errors and 1,014 pre-existing repository warnings, Roslynk reports zero compiler errors, 156 focused reconstruction and color cases pass without failures or skips, and `git diff --check` is clean. Status meanings: @@ -41,9 +43,9 @@ Status meanings: - **Not started:** supporting primitives may exist, but the production format path is absent. - **Current:** the only work item that should be advanced before taking the next queued item. -Current development stage: **Stage 3 — complete AV1 still-image decoding.** The decoder retains reference/header/CDF/motion-field state, derives frame-level skip-mode references, consumes temporal segment prediction, decodes intra-coded blocks inside inter frames, and reconstructs translational single-reference, compound, inter-intra, OBMC, scaled-reference, local warped, and non-translational global prediction before residual traversal. Commit `c5637ea0187df35b385bf43e2fe85cd955f01099` records the global-warp checkpoint. The working tree additionally decodes the official four-frame motion-vector sequence exactly through all ordinary inter modes, all three motion modes, every regular/smooth/sharp dual-filter pair, sub-8x8 chroma prediction, and no-round compound intermediates. Neither AV1 nor HEVC production encoding is implemented. +Current development stage: **Stage 3 — complete AV1 still-image decoding.** The decoder retains reference/header/CDF/motion-field state, derives frame-level skip-mode references, consumes temporal segment prediction, decodes intra-coded blocks inside inter frames, and reconstructs translational single-reference, compound, inter-intra, OBMC, scaled-reference, local warped, and non-translational global prediction before residual traversal. Commit `1c58d855f70b024170ced9eb0a7005f0f9c955ad` adds exact official four-frame coverage through all ordinary inter modes, all three motion modes, every regular/smooth/sharp dual-filter pair, sub-8x8 chroma prediction, and no-round compound intermediates. Neither AV1 nor HEVC production encoding is implemented. -Immediate checkpoint: **commit the verified official motion-vector conformance slice before advancing.** Exact official native-plane comparison, normal/scalar dispatch, constrained allocation, sub-8x8 chroma, no-round compound arithmetic, Release builds, Roslyn/compiler checks, the focused suite, documentation review, and the diff gate all pass. +Immediate checkpoint: **commit the verified selected-spatial-layer presentation slice before advancing.** The production container path returns the exact selected native layer, scales only its short-lived presentation planes, matches pinned libavif output, preserves final-layer and display-only film-grain behavior, passes constrained allocation, and has complete focused Release evidence. | Order | Delivery stage | State | Delivered state | Gate that remains open | | --- | --- | --- | --- | --- | @@ -96,10 +98,10 @@ Immediate checkpoint: **commit the verified official motion-vector conformance s - [x] Decode and independently verify scaled-reference prediction through exact Q14/Q10 arithmetic, allocation-free SIMD/scalar convolution, 8/10/12-bit production reconstruction, scaled OBMC, active inter deblocking, constrained allocation, and a genuine size-changing bounded AVIF. - [x] Decode local warped motion through pinned integer projection and allocation-free SIMD-first affine filtering. A genuine two-frame bounded AVIF selects traced multi-sample warped blocks and passes exact final native-plane and presentation comparisons, constrained allocation, and normal/scalar dispatch. Commit `0b2db86809e26757460e719a6b59c0fe58bb0586` records the checkpoint. - [x] Decode non-translational global motion through the committed affine filter. A genuine two-frame bounded AVIF selects seven traced rotation/zoom `GLOBALMV` blocks and passes exact final native-plane and presentation comparisons, constrained allocation, and normal/scalar dispatch. Direct 8/10/12-bit production reconstruction covers both matrices of compound `GLOBAL_GLOBALMV`. Commit `c5637ea0187df35b385bf43e2fe85cd955f01099` records the checkpoint. - - [x] Decode the official `av1-1-b8-05-mv.ivf` sequence through every ordinary inter mode, all three motion modes, and every regular/smooth/sharp dual-filter pair; reconstruct sub-8x8 chroma and allocation-free no-round compound intermediates; and compare all four shown 352x288 YUV420 frames exactly with pinned-libaom output under normal/scalar dispatch and constrained tracked allocation. The official IVF SHA-1 matches the pinned libaom manifest. Both Release source targets build with zero warnings and errors; the `net10.0` test-project analyzer build completes with zero errors and 1,014 pre-existing repository warnings; Roslynk reports zero compiler errors; 3,983 focused decoder/predictor cases plus two complete official-sequence cases pass without failures or skips; and `git diff --check` is clean. The checkpoint is complete locally and awaiting commit. - - [ ] **Queued until the official motion-vector checkpoint commit:** verify every remaining connected mode and filter with independently encoded bounded AV1 content and exact native-plane comparisons. - - [ ] Return the explicitly selected spatial layer or the final displayed layer, keeping reference reconstruction separate from display-only film grain. - - [ ] Verify color and auxiliary-alpha output exactly against both pinned libavif progressive fixtures under normal SIMD dispatch and all required `FeatureTestRunner` fallbacks. + - [x] Decode the official `av1-1-b8-05-mv.ivf` sequence through every ordinary inter mode, all three motion modes, and every regular/smooth/sharp dual-filter pair; reconstruct sub-8x8 chroma and allocation-free no-round compound intermediates; and compare all four shown 352x288 YUV420 frames exactly with pinned-libaom output under normal/scalar dispatch and constrained tracked allocation. The official IVF SHA-1 matches the pinned libaom manifest. Both Release source targets build with zero warnings and errors; the `net10.0` test-project analyzer build completes with zero errors and 1,014 pre-existing repository warnings; Roslynk reports zero compiler errors; 3,983 focused decoder/predictor cases plus two complete official-sequence cases pass without failures or skips; and `git diff --check` is clean. Commit `1c58d855f70b024170ced9eb0a7005f0f9c955ad` records the checkpoint. + - [x] Verify every connected inter mode and filter with exact independent content. The official pinned-libaom sequence supplies complete bit-mask coverage for all twelve ordinary inter modes, all three motion modes, and all nine regular/smooth/sharp dual-filter pairs. The committed average, selectable compound, inter-intra, OBMC, scaled-reference, local-warp, and global-warp AVIF fixtures separately require every connected reconstruction variant and pass exact native-plane comparison; the presentation-bearing fixtures also pass exact pinned-libavif output. + - [x] Return the explicitly selected spatial layer or the final displayed layer, keeping reference reconstruction separate from display-only film grain. The essential-`lsel` production fixture reconstructs the selected 40x40 YUV444 base exactly, scales native component planes to the 80x80 item extent with pinned-libyuv integer rounding, and matches pinned-libavif RGBA presentation under normal/scalar dispatch. The committed final-layer fixture and film-grain matrix remain exact. Constrained tracked allocation returns every short-lived presentation plane exactly once. + - [ ] **Queued until the selected-spatial-layer checkpoint commit:** verify color and auxiliary-alpha output exactly against both pinned libavif progressive fixtures under normal SIMD dispatch and all required `FeatureTestRunner` fallbacks. - [ ] Correct the audited 12-bit inverse ADST4, Identity4, and Identity16 SIMD arithmetic by widening only the libaom-widened multiply/accumulate operations, with exact conformant-range vectors and `FeatureTestRunner` coverage. - [ ] Continue inventorying and removing every remaining valid AV1 still-image unsupported branch, adding exact independent compression-tool fixtures to the profile-matrix regression gate. - [ ] Complete the remaining HEVC still-image profile and Range Extensions matrix with exact independent native-plane and presentation evidence. @@ -436,6 +438,7 @@ This assessment was reconciled with the source tree on 2026-08-29, including pro - AV1 has focused bitstream, prediction, entropy, reconstruction, filter, film-grain, color, and transform coverage, plus real libavif inputs. A real two-layer libavif-derived OBU stream verifies the bounded frame lifecycle, retained-slot occupancy, resolved inter references, and `frame_size_with_refs` dimensions through a fake tile lifecycle. The corrected progressive color item then verifies its production single-reference inter branch against exact pinned-libaom native planes and pinned-libavif presentation. The pinned 19-frame YUV444 sequence separately reaches production equal-average compound reconstruction after every preceding reference sample and matches its final native and presented references exactly. A real palette stream truncated inside its tile entropy payload verifies libaom-equivalent overflow/trailing-bit rejection and decoder-session recovery. The committed equal-average checkpoint passes 119 direct focused cases plus three `FeatureTestRunner` cases, zero-warning `net10.0` and `net11.0` source builds, a zero-error test-project analyzer build with pre-existing repository warnings, allocator lifetime validation, and `git diff --check`. Four additional pinned sequences reach distance-weighted, both wedge signs, both difference-mask orientations, and smooth and wedge inter-intra reconstruction with exact final native and presented output. Commit `864b11ad9791d3254a38691246270c569001b5a1` records the selectable-compound checkpoint after both source targets, the test-project analyzer build, 25 direct focused cases, six `FeatureTestRunner` cases, constrained allocation, exactly-once tracked returns, and `git diff --check` passed. Commit `59a4fbad5ed3bc7c34331c4bdc65acc7acde3f04` adds exact traced OBMC sequence evidence, subsampled direct reconstruction, constrained allocation, both source builds, the analyzer build, 14 direct cases, the normal/scalar `FeatureTestRunner` comparison, and clean staged `git diff --check`. Commit `8f57c29b250b4291f034b31c1ea3de3fa3345467` adds a genuine size-changing layered AVIF, exact retained-base and final native planes, exact final presentation, active inter deblocking, constrained allocation, normal/scalar dispatch, direct 8/10/12-bit block reconstruction, and 87 passing focused cases. The local-warp checkpoint adds a genuine two-frame 256x256 bounded AVIF with many traced `WARPED_CAUSAL` blocks, exact pinned multi-sample projection, exact final native and presentation comparisons, constrained allocation, normal/scalar `FeatureTestRunner` dispatch, direct 8/10/12-bit predictor parity, zero-warning source builds, and 86 passing focused cases. Valid still-image syntax paths still contain explicit unsupported branches, so the independent AV1 decode matrix is not complete. - Local-warp fixture provenance is pinned to libavif `062e582e8afda88e6baf988fdcf046a801efa0f5` and libaom `03087864cf4bea6abb0d28f95cf7843511413d8f`. A deterministic two-frame limited-range YUV444 grid sequence was encoded with `avifenc -j 1 -s 0 -q 60 -a color:enable-warped-motion=1 -a color:enable-global-motion=0 -a color:enable-obmc=0`. SHA-256 values are `990BAC4AD443005C217B0DA4FCCFA9ADFB3AA147AD06C85F9A655A4433E9E8A7` for the AVIF, `984B2815CEE0C05FDE26430F150A21B5C993141E25BA1ED4FDE09372DB64AC13` for the pinned-libaom frame-1 Y4M, and `4490D62FB6679378E92CACA48427359091AD2106BE49FC1A3848F78BE03BEEB1` for the pinned-libavif frame-1 PNG. - Scaled-reference fixture provenance is pinned to libavif `062e582e8afda88e6baf988fdcf046a801efa0f5` and libaom `03087864cf4bea6abb0d28f95cf7843511413d8f`. The source is libavif `tests/data/webp_logo_animated.y4m`, encoded as two layered YUV444 inputs with `avifenc -j 1 -q 90 --layered --scaling-mode:u 1/2 --scaling-mode:u 1/1 `. The resulting item contains a 701-byte 40x40 base layer followed by a 1,494-byte 80x80 dependent layer. SHA-256 values are `EB239F31EC8DBF5E97AD6F52670FCA6497AE2A933822CFE724C75F66AAA2520B` for the AVIF, `B7E30E04A935414A517BAA2DF06AB756DA18BA7C291220D9D7C063297761AE82` for the 2,195-byte logical AV1 payload, `873DC1AB5623910FBF8053CBF684399F4F3BDDBA4110492E3E11DE86E01FEC1B` for the 4,800-byte pinned-libaom base YUV, `964F4D85A79FC10EEB3722E2C9DA3415083E1EB8C2611FE154946FA55E2171ED` for the pinned-libaom final Y4M, and `DC4C6DBE6BD92C5FCE1E3E23700AFA603EF04ED02EDD336213EBBA1E3BD84BA0` for the pinned-libavif final PNG. +- The selected-layer derivative adds one essential `lsel` association selecting spatial layer zero without changing the logical AV1 payload. Its absolute `iloc` extent offsets move by eleven bytes with the enlarged property containers, and pinned libavif linked to the same libaom commit presents the selected 40x40 base at the item's 80x80 `ispe` extent. The 2,505-byte AVIF has SHA-256 `E6E1F53F8DD558C5309E0C80B0635FA4EFCBB636541E4E38206985A91E0FBA66`; its exact pinned-libavif PNG has SHA-256 `678C5E5D4650EA6F0C590302E7DB9E3C6608851BC577453DA4A6837BDB4D3AF3`. - The AV1 transform matrix verifies scalar, `Vector128`, `Vector256`, and `Vector512` forward representations plus the production inverse tiers across every valid size/type combination and supported bit depth. All 511 focused forward and inverse cases pass in Release; `FeatureTestRunner` isolates hardware tiers, every two-dimensional configuration exercises production dispatch, and the complete-block benchmark records zero managed allocation. - Independent libavif fixtures cover primary, grid, auxiliary-alpha, ICC, metadata-skipping, and all-sync image-sequence presentation. Reference-dependent AV1 and HEVC sequence reconstruction and independent HEVC ICC sequence coverage remain open. - Focused decoder-option tests cover strict, ancillary-only, image-data, and metadata-skipping behavior for still items and sequence samples. Complete adversarial dimension, allocation, malformed-container, and resource-limit coverage remains open. @@ -587,8 +590,9 @@ Implement and verify in dependency order: - [x] Decode scaled references through exact Q14 scale factors, signed Q4-to-Q10 coordinate rounding, variable per-output phases and steps, regular/smooth/sharp/bilinear reduced and full filters, bit-depth-dependent two-pass rounding, scaled OBMC neighbors, and the normative 288-sample retained border without per-block allocation. The independently generated 40x40-to-80x80 layered fixture proves the base reference before retention and in the production retained session, then matches the final pinned-libaom Y/U/V planes and pinned-libavif RGBA presentation exactly. Active inter deblocking additionally verifies per-block reference/mode deltas and skipped inter/intra boundary decisions. Normal and scalar `FeatureTestRunner` runs, constrained contiguous planes, unique allocation identities, 8/10/12-bit production-block coverage, zero-warning `net10.0` and `net11.0` source builds, the zero-error test-project build with 1,013 pre-existing warnings, 87 focused cases, documentation review, and `git diff --check` pass. - [x] Decode local warped motion through pinned integer projection and allocation-free 8x8-tiled affine filtering across byte and high-bit-depth planes. Commit `0b2db86809e26757460e719a6b59c0fe58bb0586` records exact traced native/presentation evidence, constrained allocation, and normal/scalar dispatch. - [x] Decode non-translational global motion through reconstructed reference planes. Commit `c5637ea0187df35b385bf43e2fe85cd955f01099` records pinned single-reference rotation/zoom syntax and reconstruction, both compound reference workspaces, 8/10/12-bit storage, exact native and presentation output, constrained allocation, balanced ownership, and normal/scalar dispatch. - - [x] Decode all ordinary inter modes, motion modes, and regular/smooth/sharp dual-filter pairs from the official four-frame 352x288 YUV420 motion-vector sequence. The production path includes sub-8x8 chroma assembly and allocation-free no-round compound intermediates. Exact pinned-libaom native planes, normal/scalar dispatch, constrained tracked allocation, direct compound predictor parity, zero-warning source builds, the zero-error analyzer build, Roslyn compiler diagnostics, 3,985 focused cases, documentation review, and `git diff --check` pass. The checkpoint is complete locally and awaiting commit. - - [ ] **Queued until this checkpoint commits:** verify every remaining connected inter mode and filter with independently encoded bounded AV1 content and exact native-plane comparisons. + - [x] Decode all ordinary inter modes, motion modes, and regular/smooth/sharp dual-filter pairs from the official four-frame 352x288 YUV420 motion-vector sequence. The production path includes sub-8x8 chroma assembly and allocation-free no-round compound intermediates. Exact pinned-libaom native planes, normal/scalar dispatch, constrained tracked allocation, direct compound predictor parity, zero-warning source builds, the zero-error analyzer build, Roslyn compiler diagnostics, 3,985 focused cases, documentation review, and `git diff --check` pass. Commit `1c58d855f70b024170ced9eb0a7005f0f9c955ad` records the checkpoint. + - [x] Verify every connected inter mode and filter with exact independent content. Complete official coverage masks prove all ordinary inter modes, all motion modes, and every switchable regular/smooth/sharp filter pair, while the committed bounded AVIF fixtures independently prove every compound, inter-intra, OBMC, scaled, local-warp, and global-warp reconstruction branch. + - [x] Return an explicitly selected lower spatial layer at the item's presentation extent without rescaling retained references. The essential-`lsel` derivative selects the exact pinned-libaom 40x40 YUV444 base, scales native planes to 80x80 before color conversion, and matches pinned-libavif output. The allocator-owned presentation buffer uses group-safe rows, portable `Vector128` two-times filtering with scalar fallback, exact 8/12-bit libyuv-shaped vectors, constrained-allocation lifetime tracking, and leaves committed final-layer and display-only film-grain results unchanged. - [ ] Lossless and high-bit-depth reconstruction with correct clipping and intermediate precision. - [x] Route lossless 4x4 blocks through allocation-free reversible inverse Walsh-Hadamard reconstruction for 8/10/12-bit samples, including the DC-only specialization, `Vector128` production traversal, scalar fallback, exact clipping, and `FeatureTestRunner` parity. - [x] Verify lossless syntax, inverse quantization, prediction, and presented reconstruction with independently encoded 8/10/12-bit AVIF fixtures. The tests require coded residuals with palette and intra-block copy disabled, compare every native YUV sample with the pinned generic libaom-backed decoder, and compare every presented RGBA byte with pinned generic libavif exactly under normal hardware dispatch and the scalar fallback. diff --git a/src/ImageSharp/Formats/Heif/Av1/Av1Decoder.cs b/src/ImageSharp/Formats/Heif/Av1/Av1Decoder.cs index c964bd2fb..de881bce1 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Av1Decoder.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Av1Decoder.cs @@ -118,12 +118,14 @@ internal sealed class Av1Decoder : IAv1TileReader, IDisposable /// The item-associated AV1 codec configuration validated against the coded sequence header. /// /// The optional byte boundaries of a layered AV1 image item. + /// The requested item presentation size, or an empty size for the coded dimensions. /// The decoded image. public Image Decode( Span buffer, CicpProfile? containerColorProfile = null, Av1CodecConfiguration? codecConfiguration = null, - Av1LayeredImageIndex? layeredImageIndex = null) + Av1LayeredImageIndex? layeredImageIndex = null, + Size presentationSize = default) where TPixel : unmanaged, IPixel { ImageFrame frame = this.DecodeFrame( @@ -131,7 +133,8 @@ internal sealed class Av1Decoder : IAv1TileReader, IDisposable containerColorProfile, codecConfiguration, out CicpProfile effectiveColorProfile, - layeredImageIndex); + layeredImageIndex, + presentationSize); ImageMetadata metadata = new() { @@ -163,13 +166,15 @@ internal sealed class Av1Decoder : IAv1TileReader, IDisposable /// /// Receives the effective CICP description used for conversion. /// The optional byte boundaries of a layered AV1 image item. + /// The requested item presentation size, or an empty size for the coded dimensions. /// The decoded frame. Ownership transfers to the caller. public ImageFrame DecodeFrame( Span buffer, CicpProfile? containerColorProfile, Av1CodecConfiguration? codecConfiguration, out CicpProfile effectiveColorProfile, - Av1LayeredImageIndex? layeredImageIndex = null) + Av1LayeredImageIndex? layeredImageIndex = null, + Size presentationSize = default) where TPixel : unmanaged, IPixel { using Av1FrameBuffer frameBuffer = this.DecodeFrameBuffer( @@ -179,7 +184,7 @@ internal sealed class Av1Decoder : IAv1TileReader, IDisposable out effectiveColorProfile, layeredImageIndex); - return this.ConvertToFrame(frameBuffer, effectiveColorProfile); + return this.ConvertToFrame(frameBuffer, effectiveColorProfile, presentationSize); } /// @@ -268,20 +273,32 @@ internal sealed class Av1Decoder : IAv1TileReader, IDisposable /// The destination pixel type. /// The decoded native planes. /// The effective CICP description. + /// The requested item presentation size, or an empty size for the coded dimensions. /// The independently owned packed-pixel frame. private ImageFrame ConvertToFrame( Av1FrameBuffer frameBuffer, - CicpProfile effectiveColorProfile) + CicpProfile effectiveColorProfile, + Size presentationSize = default) where TPixel : unmanaged, IPixel { ImageFrame? resultFrame = null; try { - resultFrame = new ImageFrame( - this.configuration, + Size codedSize = new( this.FrameHeader!.FrameSize.SuperResolutionUpscaledWidth, this.FrameHeader.FrameSize.FrameHeight); + // A selected lower spatial layer can only be scaled upward to the image item's ispe extent here. + // Other item-size corrections keep using the shared packed-pixel presentation path after decoding. + Size outputSize = presentationSize.Width >= codedSize.Width && presentationSize.Height >= codedSize.Height + ? presentationSize + : codedSize; + + resultFrame = new ImageFrame( + this.configuration, + outputSize.Width, + outputSize.Height); + Av1YuvConverter.ConvertToRgb(this.configuration, frameBuffer, resultFrame); resultFrame.Metadata.CicpProfile = effectiveColorProfile.DeepClone(); return resultFrame; diff --git a/src/ImageSharp/Formats/Heif/Av1/Color/Av1PresentationSampleBuffer.cs b/src/ImageSharp/Formats/Heif/Av1/Color/Av1PresentationSampleBuffer.cs new file mode 100644 index 000000000..9a69c7285 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Color/Av1PresentationSampleBuffer.cs @@ -0,0 +1,745 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using System.Runtime.Intrinsics; +using SixLabors.ImageSharp.Common.Helpers; +using SixLabors.ImageSharp.Formats.Heif.Components; +using SixLabors.ImageSharp.Memory; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Color; + +/// +/// Owns AV1 component planes scaled to an image item's presentation extent. +/// +/// The native unsigned sample storage type. +/// The reconstructed AV1 plane adapter. +internal sealed class Av1PresentationSampleBuffer : IDisposable + where TSample : unmanaged + where TBuffer : struct, IHeifPlanarSampleBuffer +{ + /// + /// The allocator that owns the presentation planes and row workspace. + /// + private readonly MemoryAllocator memoryAllocator; + + /// + /// The scaled luma plane. + /// + private Buffer2D? luma; + + /// + /// The scaled blue-difference plane. + /// + private Buffer2D? chromaBlue; + + /// + /// The scaled red-difference plane. + /// + private Buffer2D? chromaRed; + + /// + /// Initializes a new instance of the class. + /// + /// The configuration providing allocator-owned plane storage. + /// The unscaled reconstructed AV1 planes. + /// The presented luma width. + /// The presented luma height. + public Av1PresentationSampleBuffer(Configuration configuration, TBuffer source, int width, int height) + { + this.memoryAllocator = configuration.MemoryAllocator; + this.Width = width; + this.Height = height; + this.LumaBitDepth = source.LumaBitDepth; + this.ChromaBitDepth = source.ChromaBitDepth; + this.IsMonochrome = source.IsMonochrome; + this.ChromaSubsamplingX = source.ChromaSubsamplingX; + this.ChromaSubsamplingY = source.ChromaSubsamplingY; + this.ChromaPositionX = source.ChromaPositionX; + this.ChromaPositionY = source.ChromaPositionY; + + int sourceChromaWidth = DivideCeiling(source.Width, 1 << source.ChromaSubsamplingX); + int sourceChromaHeight = DivideCeiling(source.Height, 1 << source.ChromaSubsamplingY); + int destinationChromaWidth = DivideCeiling(width, 1 << source.ChromaSubsamplingX); + int destinationChromaHeight = DivideCeiling(height, 1 << source.ChromaSubsamplingY); + + try + { + this.luma = this.memoryAllocator.Allocate2D(width, height); + this.ScalePlane(source, Av1Plane.Y, source.Width, source.Height, this.luma); + + if (!source.IsMonochrome) + { + this.chromaBlue = this.memoryAllocator.Allocate2D(destinationChromaWidth, destinationChromaHeight); + this.ScalePlane(source, Av1Plane.U, sourceChromaWidth, sourceChromaHeight, this.chromaBlue); + + this.chromaRed = this.memoryAllocator.Allocate2D(destinationChromaWidth, destinationChromaHeight); + this.ScalePlane(source, Av1Plane.V, sourceChromaWidth, sourceChromaHeight, this.chromaRed); + } + } + catch + { + this.Dispose(); + throw; + } + } + + /// + /// Gets the presented luma width. + /// + public int Width { get; } + + /// + /// Gets the presented luma height. + /// + public int Height { get; } + + /// + /// Gets the luma sample bit depth. + /// + public int LumaBitDepth { get; } + + /// + /// Gets the chroma sample bit depth. + /// + public int ChromaBitDepth { get; } + + /// + /// Gets a value indicating whether only luma is present. + /// + public bool IsMonochrome { get; } + + /// + /// Gets the horizontal chroma-subsampling shift. + /// + public int ChromaSubsamplingX { get; } + + /// + /// Gets the vertical chroma-subsampling shift. + /// + public int ChromaSubsamplingY { get; } + + /// + /// Gets the horizontal chroma position in half-luma-sample units. + /// + public int ChromaPositionX { get; } + + /// + /// Gets the vertical chroma position in half-luma-sample units. + /// + public int ChromaPositionY { get; } + + /// + /// Gets a borrowed adapter over the scaled planes. + /// + public Av1PresentationSampleBufferView View => new(this); + + /// + /// Releases the scaled planes. + /// + public void Dispose() + { + this.luma?.Dispose(); + this.chromaBlue?.Dispose(); + this.chromaRed?.Dispose(); + this.luma = null; + this.chromaBlue = null; + this.chromaRed = null; + } + + /// + /// Gets one scaled component row. + /// + /// The requested component plane. + /// The zero-based plane row. + /// The visible samples in the requested row. + public Span GetRowSpan(Av1Plane plane, int row) + => plane switch + { + Av1Plane.Y => this.luma!.DangerousGetRowSpan(row), + Av1Plane.U => this.chromaBlue!.DangerousGetRowSpan(row), + _ => this.chromaRed!.DangerousGetRowSpan(row) + }; + + /// + /// Scales one component plane with the native integer filter used by pinned libavif's libyuv backend. + /// + /// The reconstructed component planes. + /// The component plane to scale. + /// The source plane width. + /// The source plane height. + /// The scaled destination plane. + private void ScalePlane( + TBuffer source, + Av1Plane plane, + int sourceWidth, + int sourceHeight, + Buffer2D destination) + { + int destinationWidth = destination.Width; + int destinationHeight = destination.Height; + if (sourceWidth == destinationWidth && sourceHeight == destinationHeight) + { + for (int y = 0; y < sourceHeight; y++) + { + GetSourceRow(source, plane, y)[..sourceWidth].CopyTo(destination.DangerousGetRowSpan(y)); + } + + return; + } + + bool doublesWidth = (destinationWidth + 1) / 2 == sourceWidth; + bool doublesHeight = (destinationHeight + 1) / 2 == sourceHeight; + if (doublesWidth && doublesHeight) + { + ScaleUp2(source, plane, sourceWidth, sourceHeight, destination); + return; + } + + if (doublesWidth && sourceHeight == destinationHeight) + { + for (int y = 0; y < sourceHeight; y++) + { + ScaleRowUp2Linear( + GetSourceRow(source, plane, y)[..sourceWidth], + destination.DangerousGetRowSpan(y)); + } + + return; + } + + if (sourceHeight == destinationHeight) + { + int rowHorizontalStep = sourceWidth > 1 && destinationWidth > 1 + ? FixedDivideOne(sourceWidth, destinationWidth) + : 0; + + for (int y = 0; y < sourceHeight; y++) + { + ScaleHorizontal( + GetSourceRow(source, plane, y)[..sourceWidth], + destination.DangerousGetRowSpan(y), + rowHorizontalStep); + } + + return; + } + + // Layer selection presents a lower spatial layer at the full item extent, so both dimensions are monotonic. + // The general libyuv path maps destination centers in 16.16 fixed point and retains only two horizontally + // filtered rows. This avoids a second full-plane intermediate and remains group-safe under small allocators. + using Buffer2D horizontalRows = this.memoryAllocator.Allocate2D(destinationWidth, 2); + int horizontalStep = sourceWidth > 1 && destinationWidth > 1 + ? FixedDivideOne(sourceWidth, destinationWidth) + : 0; + + int verticalStep = sourceHeight > 1 && destinationHeight > 1 + ? FixedDivideOne(sourceHeight, destinationHeight) + : 0; + + int sourcePositionY = 0; + int firstSourceRow = -1; + int secondSourceRow = -1; + int firstSlot = 0; + int secondSlot = 1; + for (int y = 0; y < destinationHeight; y++) + { + int sourceRow = sourcePositionY >> 16; + int nextSourceRow = Math.Min(sourceRow + 1, sourceHeight - 1); + if (sourceRow == secondSourceRow) + { + (firstSourceRow, secondSourceRow) = (secondSourceRow, firstSourceRow); + (firstSlot, secondSlot) = (secondSlot, firstSlot); + } + + if (firstSourceRow != sourceRow) + { + ScaleHorizontal( + GetSourceRow(source, plane, sourceRow)[..sourceWidth], + horizontalRows.DangerousGetRowSpan(firstSlot), + horizontalStep); + + firstSourceRow = sourceRow; + } + + if (secondSourceRow != nextSourceRow) + { + ScaleHorizontal( + GetSourceRow(source, plane, nextSourceRow)[..sourceWidth], + horizontalRows.DangerousGetRowSpan(secondSlot), + horizontalStep); + + secondSourceRow = nextSourceRow; + } + + int verticalFraction = (sourcePositionY >> 8) & 255; + InterpolateRows( + horizontalRows.DangerousGetRowSpan(firstSlot), + horizontalRows.DangerousGetRowSpan(secondSlot), + destination.DangerousGetRowSpan(y), + verticalFraction); + + sourcePositionY += verticalStep; + } + } + + /// + /// Applies libyuv's edge-aware two-times bilinear kernel to one complete plane. + /// + /// The reconstructed component planes. + /// The component plane to scale. + /// The source plane width. + /// The source plane height. + /// The scaled destination plane. + private static void ScaleUp2( + TBuffer source, + Av1Plane plane, + int sourceWidth, + int sourceHeight, + Buffer2D destination) + { + Span firstSource = GetSourceRow(source, plane, 0)[..sourceWidth]; + Span firstDestination = destination.DangerousGetRowSpan(0); + ScaleRowUp2Bilinear(firstSource, firstSource, firstDestination, firstDestination); + + int destinationRow = 1; + for (int y = 0; y < sourceHeight - 1; y++) + { + ScaleRowUp2Bilinear( + GetSourceRow(source, plane, y)[..sourceWidth], + GetSourceRow(source, plane, y + 1)[..sourceWidth], + destination.DangerousGetRowSpan(destinationRow), + destination.DangerousGetRowSpan(destinationRow + 1)); + + destinationRow += 2; + } + + if ((destination.Height & 1) == 0) + { + Span lastSource = GetSourceRow(source, plane, sourceHeight - 1)[..sourceWidth]; + Span lastDestination = destination.DangerousGetRowSpan(destination.Height - 1); + ScaleRowUp2Bilinear(lastSource, lastSource, lastDestination, lastDestination); + } + } + + /// + /// Gets one visible source row without boxing the codec adapter. + /// + /// The reconstructed component planes. + /// The requested component plane. + /// The zero-based plane row. + /// The source row. + private static Span GetSourceRow(TBuffer source, Av1Plane plane, int row) + => plane switch + { + Av1Plane.Y => source.GetLumaRowSpan(row), + Av1Plane.U => source.GetChromaBlueRowSpan(row), + _ => source.GetChromaRedRowSpan(row) + }; + + /// + /// Applies the edge-aware two-times bilinear row kernel. + /// + /// The upper source row. + /// The lower source row. + /// The upper destination row. + /// The lower destination row. + private static void ScaleRowUp2Bilinear( + ReadOnlySpan topSource, + ReadOnlySpan bottomSource, + Span topDestination, + Span bottomDestination) + { + if (typeof(TSample) == typeof(byte)) + { + ScaleRowUp2BilinearByte( + MemoryMarshal.Cast(topSource), + MemoryMarshal.Cast(bottomSource), + MemoryMarshal.Cast(topDestination), + MemoryMarshal.Cast(bottomDestination)); + + return; + } + + ScaleRowUp2BilinearUInt16( + MemoryMarshal.Cast(topSource), + MemoryMarshal.Cast(bottomSource), + MemoryMarshal.Cast(topDestination), + MemoryMarshal.Cast(bottomDestination)); + } + + /// + /// Applies the byte two-times bilinear row kernel through portable 128-bit lanes and a scalar tail. + /// + /// The upper source row. + /// The lower source row. + /// The upper destination row. + /// The lower destination row. + private static void ScaleRowUp2BilinearByte( + ReadOnlySpan topSource, + ReadOnlySpan bottomSource, + Span topDestination, + Span bottomDestination) + { + int lastSource = topSource.Length - 1; + topDestination[0] = (byte)(((3 * topSource[0]) + bottomSource[0] + 2) >> 2); + bottomDestination[0] = (byte)((topSource[0] + (3 * bottomSource[0]) + 2) >> 2); + + int x = 0; + if (Vector128.IsHardwareAccelerated) + { + ref byte topSourceBase = ref MemoryMarshal.GetReference(topSource); + ref byte bottomSourceBase = ref MemoryMarshal.GetReference(bottomSource); + ref byte topDestinationBase = ref MemoryMarshal.GetReference(topDestination); + ref byte bottomDestinationBase = ref MemoryMarshal.GetReference(bottomDestination); + for (; x + 8 <= lastSource; x += 8) + { + Vector128 top0 = LoadEightBytes(ref topSourceBase, x); + Vector128 top1 = LoadEightBytes(ref topSourceBase, x + 1); + Vector128 bottom0 = LoadEightBytes(ref bottomSourceBase, x); + Vector128 bottom1 = LoadEightBytes(ref bottomSourceBase, x + 1); + CalculateBilinearPairs( + top0, + top1, + bottom0, + bottom1, + out Vector128 upperEven, + out Vector128 upperOdd, + out Vector128 lowerEven, + out Vector128 lowerOdd); + + StoreInterleavedBytes(upperEven, upperOdd, ref topDestinationBase, 1 + (2 * x)); + StoreInterleavedBytes(lowerEven, lowerOdd, ref bottomDestinationBase, 1 + (2 * x)); + } + } + + for (; x < lastSource; x++) + { + int top0 = topSource[x]; + int top1 = topSource[x + 1]; + int bottom0 = bottomSource[x]; + int bottom1 = bottomSource[x + 1]; + int destination = 1 + (2 * x); + topDestination[destination] = (byte)(((9 * top0) + (3 * top1) + (3 * bottom0) + bottom1 + 8) >> 4); + topDestination[destination + 1] = (byte)(((3 * top0) + (9 * top1) + bottom0 + (3 * bottom1) + 8) >> 4); + bottomDestination[destination] = (byte)(((3 * top0) + top1 + (9 * bottom0) + (3 * bottom1) + 8) >> 4); + bottomDestination[destination + 1] = (byte)((top0 + (3 * top1) + (3 * bottom0) + (9 * bottom1) + 8) >> 4); + } + + int lastDestination = topDestination.Length - 1; + topDestination[lastDestination] = (byte)(((3 * topSource[lastSource]) + bottomSource[lastSource] + 2) >> 2); + bottomDestination[lastDestination] = (byte)((topSource[lastSource] + (3 * bottomSource[lastSource]) + 2) >> 2); + } + + /// + /// Applies the unsigned 16-bit two-times bilinear row kernel through portable 128-bit lanes and a scalar tail. + /// + /// The upper source row. + /// The lower source row. + /// The upper destination row. + /// The lower destination row. + private static void ScaleRowUp2BilinearUInt16( + ReadOnlySpan topSource, + ReadOnlySpan bottomSource, + Span topDestination, + Span bottomDestination) + { + int lastSource = topSource.Length - 1; + topDestination[0] = (ushort)(((3 * topSource[0]) + bottomSource[0] + 2) >> 2); + bottomDestination[0] = (ushort)((topSource[0] + (3 * bottomSource[0]) + 2) >> 2); + + int x = 0; + if (Vector128.IsHardwareAccelerated) + { + ref ushort topSourceBase = ref MemoryMarshal.GetReference(topSource); + ref ushort bottomSourceBase = ref MemoryMarshal.GetReference(bottomSource); + ref ushort topDestinationBase = ref MemoryMarshal.GetReference(topDestination); + ref ushort bottomDestinationBase = ref MemoryMarshal.GetReference(bottomDestination); + for (; x + Vector128.Count <= lastSource; x += Vector128.Count) + { + Vector128 top0 = Vector128.LoadUnsafe(ref topSourceBase, (nuint)x); + Vector128 top1 = Vector128.LoadUnsafe(ref topSourceBase, (nuint)(x + 1)); + Vector128 bottom0 = Vector128.LoadUnsafe(ref bottomSourceBase, (nuint)x); + Vector128 bottom1 = Vector128.LoadUnsafe(ref bottomSourceBase, (nuint)(x + 1)); + CalculateBilinearPairs( + top0, + top1, + bottom0, + bottom1, + out Vector128 upperEven, + out Vector128 upperOdd, + out Vector128 lowerEven, + out Vector128 lowerOdd); + + StoreInterleavedUInt16(upperEven, upperOdd, ref topDestinationBase, 1 + (2 * x)); + StoreInterleavedUInt16(lowerEven, lowerOdd, ref bottomDestinationBase, 1 + (2 * x)); + } + } + + for (; x < lastSource; x++) + { + int top0 = topSource[x]; + int top1 = topSource[x + 1]; + int bottom0 = bottomSource[x]; + int bottom1 = bottomSource[x + 1]; + int destination = 1 + (2 * x); + topDestination[destination] = (ushort)(((9 * top0) + (3 * top1) + (3 * bottom0) + bottom1 + 8) >> 4); + topDestination[destination + 1] = (ushort)(((3 * top0) + (9 * top1) + bottom0 + (3 * bottom1) + 8) >> 4); + bottomDestination[destination] = (ushort)(((3 * top0) + top1 + (9 * bottom0) + (3 * bottom1) + 8) >> 4); + bottomDestination[destination + 1] = (ushort)((top0 + (3 * top1) + (3 * bottom0) + (9 * bottom1) + 8) >> 4); + } + + int lastDestination = topDestination.Length - 1; + topDestination[lastDestination] = (ushort)(((3 * topSource[lastSource]) + bottomSource[lastSource] + 2) >> 2); + bottomDestination[lastDestination] = (ushort)((topSource[lastSource] + (3 * bottomSource[lastSource]) + 2) >> 2); + } + + /// + /// Calculates the four interleaved bilinear products for eight source positions. + /// + /// The upper-left samples. + /// The upper-right samples. + /// The lower-left samples. + /// The lower-right samples. + /// Receives the upper left-biased samples. + /// Receives the upper right-biased samples. + /// Receives the lower left-biased samples. + /// Receives the lower right-biased samples. + private static void CalculateBilinearPairs( + Vector128 top0, + Vector128 top1, + Vector128 bottom0, + Vector128 bottom1, + out Vector128 upperEven, + out Vector128 upperOdd, + out Vector128 lowerEven, + out Vector128 lowerOdd) + { + Vector128 rounding = Vector128.Create((ushort)8); + + // The largest twelve-bit weighted sum is 16 * 4095 + 8, which remains within unsigned 16-bit lanes. + // Keeping eight independent source positions per vector therefore avoids widening and preserves libyuv's + // exact add-before-shift rounding for both byte and high-bit-depth presentation planes. + upperEven = (((top0 << 3) + top0) + ((top1 << 1) + top1) + ((bottom0 << 1) + bottom0) + bottom1 + rounding) >> 4; + upperOdd = (((top0 << 1) + top0) + ((top1 << 3) + top1) + bottom0 + ((bottom1 << 1) + bottom1) + rounding) >> 4; + lowerEven = (((top0 << 1) + top0) + top1 + ((bottom0 << 3) + bottom0) + ((bottom1 << 1) + bottom1) + rounding) >> 4; + lowerOdd = (top0 + ((top1 << 1) + top1) + ((bottom0 << 1) + bottom0) + ((bottom1 << 3) + bottom1) + rounding) >> 4; + } + + /// + /// Loads eight byte samples as unsigned 16-bit lanes. + /// + /// The first source byte. + /// The byte offset. + /// The widened samples. + private static Vector128 LoadEightBytes(ref byte source, int offset) + { + ulong packed = Unsafe.ReadUnaligned(ref Unsafe.Add(ref source, offset)); + return Vector128.WidenLower(Vector128.CreateScalarUnsafe(packed).AsByte()); + } + + /// + /// Interleaves and stores eight pairs of byte results. + /// + /// The left-biased results. + /// The right-biased results. + /// The first destination byte. + /// The destination byte offset. + private static void StoreInterleavedBytes( + Vector128 even, + Vector128 odd, + ref byte destination, + int offset) + { + Vector128 lower = Vector128_.UnpackLow(even.AsInt16(), odd.AsInt16()).AsUInt16(); + Vector128 upper = Vector128_.UnpackHigh(even.AsInt16(), odd.AsInt16()).AsUInt16(); + Vector128.Narrow(lower, upper).StoreUnsafe(ref destination, (nuint)offset); + } + + /// + /// Interleaves and stores eight pairs of unsigned 16-bit results. + /// + /// The left-biased results. + /// The right-biased results. + /// The first destination sample. + /// The destination sample offset. + private static void StoreInterleavedUInt16( + Vector128 even, + Vector128 odd, + ref ushort destination, + int offset) + { + Vector128_.UnpackLow(even.AsInt16(), odd.AsInt16()).AsUInt16().StoreUnsafe(ref destination, (nuint)offset); + Vector128_.UnpackHigh(even.AsInt16(), odd.AsInt16()).AsUInt16().StoreUnsafe( + ref destination, + (nuint)(offset + Vector128.Count)); + } + + /// + /// Applies libyuv's edge-aware horizontal two-times linear kernel. + /// + /// The source row. + /// The destination row. + private static void ScaleRowUp2Linear(ReadOnlySpan source, Span destination) + { + if (typeof(TSample) == typeof(byte)) + { + ScaleRowUp2LinearByte( + MemoryMarshal.Cast(source), + MemoryMarshal.Cast(destination)); + + return; + } + + ScaleRowUp2LinearUInt16( + MemoryMarshal.Cast(source), + MemoryMarshal.Cast(destination)); + } + + /// + /// Applies the byte horizontal two-times linear kernel. + /// + /// The source row. + /// The destination row. + private static void ScaleRowUp2LinearByte(ReadOnlySpan source, Span destination) + { + destination[0] = source[0]; + for (int x = 0; x < source.Length - 1; x++) + { + int destinationX = 1 + (2 * x); + destination[destinationX] = (byte)(((3 * source[x]) + source[x + 1] + 2) >> 2); + destination[destinationX + 1] = (byte)((source[x] + (3 * source[x + 1]) + 2) >> 2); + } + + destination[^1] = source[^1]; + } + + /// + /// Applies the unsigned 16-bit horizontal two-times linear kernel. + /// + /// The source row. + /// The destination row. + private static void ScaleRowUp2LinearUInt16(ReadOnlySpan source, Span destination) + { + destination[0] = source[0]; + for (int x = 0; x < source.Length - 1; x++) + { + int destinationX = 1 + (2 * x); + destination[destinationX] = (ushort)(((3 * source[x]) + source[x + 1] + 2) >> 2); + destination[destinationX + 1] = (ushort)((source[x] + (3 * source[x + 1]) + 2) >> 2); + } + + destination[^1] = source[^1]; + } + + /// + /// Horizontally maps one source row with libyuv's 16.16 fixed-point bilinear positions. + /// + /// The source row. + /// The destination row. + /// The 16.16 source-position increment. + private static void ScaleHorizontal(ReadOnlySpan source, Span destination, int step) + { + if (source.Length == destination.Length) + { + source.CopyTo(destination); + return; + } + + if (source.Length == 1) + { + destination.Fill(source[0]); + return; + } + + int sourcePosition = 0; + if (typeof(TSample) == typeof(byte)) + { + ReadOnlySpan sourceBytes = MemoryMarshal.Cast(source); + Span destinationBytes = MemoryMarshal.Cast(destination); + for (int x = 0; x < destinationBytes.Length; x++) + { + int sourceX = sourcePosition >> 16; + int fraction = (sourcePosition & 0xFFFF) >> 9; + int left = sourceBytes[sourceX]; + int right = sourceBytes[sourceX + 1]; + destinationBytes[x] = (byte)(left + (((fraction * (right - left)) + 0x40) >> 7)); + sourcePosition += step; + } + + return; + } + + ReadOnlySpan sourceWords = MemoryMarshal.Cast(source); + Span destinationWords = MemoryMarshal.Cast(destination); + for (int x = 0; x < destinationWords.Length; x++) + { + int sourceX = sourcePosition >> 16; + int fraction = sourcePosition & 0xFFFF; + int left = sourceWords[sourceX]; + int right = sourceWords[sourceX + 1]; + destinationWords[x] = (ushort)(left + ((((long)fraction * (right - left)) + 0x8000) >> 16)); + sourcePosition += step; + } + } + + /// + /// Vertically interpolates two horizontally scaled rows. + /// + /// The upper row. + /// The lower row. + /// The destination row. + /// The lower-row weight with a denominator of 256. + private static void InterpolateRows( + ReadOnlySpan top, + ReadOnlySpan bottom, + Span destination, + int bottomWeight) + { + if (bottomWeight == 0) + { + top.CopyTo(destination); + return; + } + + int topWeight = 256 - bottomWeight; + if (typeof(TSample) == typeof(byte)) + { + ReadOnlySpan topBytes = MemoryMarshal.Cast(top); + ReadOnlySpan bottomBytes = MemoryMarshal.Cast(bottom); + Span destinationBytes = MemoryMarshal.Cast(destination); + for (int x = 0; x < destinationBytes.Length; x++) + { + destinationBytes[x] = (byte)(((topBytes[x] * topWeight) + (bottomBytes[x] * bottomWeight) + 128) >> 8); + } + + return; + } + + ReadOnlySpan topWords = MemoryMarshal.Cast(top); + ReadOnlySpan bottomWords = MemoryMarshal.Cast(bottom); + Span destinationWords = MemoryMarshal.Cast(destination); + for (int x = 0; x < destinationWords.Length; x++) + { + destinationWords[x] = (ushort)(((topWords[x] * topWeight) + (bottomWords[x] * bottomWeight) + 128) >> 8); + } + } + + /// + /// Divides two decremented lengths into libyuv's 16.16 endpoint-preserving step. + /// + /// The source length. + /// The destination length. + /// The 16.16 source-position increment. + private static int FixedDivideOne(int sourceLength, int destinationLength) + => (int)((((long)sourceLength << 16) - 0x00010001) / (destinationLength - 1)); + + /// + /// Divides a positive value by a positive divisor with ceiling rounding. + /// + /// The value to divide. + /// The positive divisor. + /// The ceiling-rounded quotient. + private static int DivideCeiling(int value, int divisor) => (value + divisor - 1) / divisor; +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Color/Av1PresentationSampleBufferView.cs b/src/ImageSharp/Formats/Heif/Av1/Color/Av1PresentationSampleBufferView.cs new file mode 100644 index 000000000..d7f09520f --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Color/Av1PresentationSampleBufferView.cs @@ -0,0 +1,63 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using SixLabors.ImageSharp.Formats.Heif.Components; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Color; + +/// +/// Exposes borrowed rows from an owned AV1 presentation buffer to the shared HEIF color converter. +/// +/// The native unsigned sample storage type. +/// The reconstructed AV1 plane adapter owned by the presentation buffer. +internal readonly struct Av1PresentationSampleBufferView : IHeifPlanarSampleBuffer + where TSample : unmanaged + where TBuffer : struct, IHeifPlanarSampleBuffer +{ + /// + /// The owner that keeps all exposed rows alive. + /// + private readonly Av1PresentationSampleBuffer owner; + + /// + /// Initializes a new instance of the struct. + /// + /// The scaled plane owner. + public Av1PresentationSampleBufferView(Av1PresentationSampleBuffer owner) => this.owner = owner; + + /// + public int Width => this.owner.Width; + + /// + public int Height => this.owner.Height; + + /// + public int LumaBitDepth => this.owner.LumaBitDepth; + + /// + public int ChromaBitDepth => this.owner.ChromaBitDepth; + + /// + public bool IsMonochrome => this.owner.IsMonochrome; + + /// + public int ChromaSubsamplingX => this.owner.ChromaSubsamplingX; + + /// + public int ChromaSubsamplingY => this.owner.ChromaSubsamplingY; + + /// + public int ChromaPositionX => this.owner.ChromaPositionX; + + /// + public int ChromaPositionY => this.owner.ChromaPositionY; + + /// + public Span GetLumaRowSpan(int row) => this.owner.GetRowSpan(Av1Plane.Y, row); + + /// + public Span GetChromaBlueRowSpan(int row) => this.owner.GetRowSpan(Av1Plane.U, row); + + /// + public Span GetChromaRedRowSpan(int row) => this.owner.GetRowSpan(Av1Plane.V, row); +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Color/Av1YuvConverter.cs b/src/ImageSharp/Formats/Heif/Av1/Color/Av1YuvConverter.cs index 67ef7ec76..f6b0c50eb 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Color/Av1YuvConverter.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Color/Av1YuvConverter.cs @@ -28,6 +28,31 @@ internal static class Av1YuvConverter if (frameBuffer.BitDepth == Av1BitDepth.EightBit) { Av1PlanarSampleBuffer buffer = new(frameBuffer); + if (buffer.Width != image.Width || buffer.Height != image.Height) + { + // AVIF spatial-layer selection scales native YUV planes before color conversion. The retained + // reconstruction remains untouched because later dependent layers can still reference its coded + // dimensions, while this short-lived owner contains only the presented sample grid. + using Av1PresentationSampleBuffer> presentationBuffer = new( + configuration, + buffer, + image.Width, + image.Height); + + HeifPlanarColorConverter.ConvertToRgb< + TPixel, + Av1PresentationSampleBufferView>, + byte, + HeifByteSampleConverter>( + configuration, + presentationBuffer.View, + image, + in parameters, + mode); + + return; + } + HeifPlanarColorConverter.ConvertToRgb, byte, HeifByteSampleConverter>( configuration, buffer, @@ -39,6 +64,26 @@ internal static class Av1YuvConverter } Av1PlanarSampleBuffer highBitDepthBuffer = new(frameBuffer); + if (highBitDepthBuffer.Width != image.Width || highBitDepthBuffer.Height != image.Height) + { + using Av1PresentationSampleBuffer> presentationBuffer = new( + configuration, + highBitDepthBuffer, + image.Width, + image.Height); + + HeifPlanarColorConverter.ConvertToRgb< + TPixel, + Av1PresentationSampleBufferView>>( + configuration, + presentationBuffer.View, + image, + in parameters, + mode); + + return; + } + HeifPlanarColorConverter.ConvertToRgb>( configuration, highBitDepthBuffer, diff --git a/src/ImageSharp/Formats/Heif/Av1HeifItemDecoder.cs b/src/ImageSharp/Formats/Heif/Av1HeifItemDecoder.cs index bb74c3260..90c57e1ac 100644 --- a/src/ImageSharp/Formats/Heif/Av1HeifItemDecoder.cs +++ b/src/ImageSharp/Formats/Heif/Av1HeifItemDecoder.cs @@ -55,7 +55,13 @@ internal class Av1HeifItemDecoder : IHeifItemDecoder, IHeifAlpha byte operatingPointIndex = item.Av1OperatingPointSelector?.Index ?? 0; using Av1Decoder decoder = new(options.Configuration, operatingPointIndex); - Image image = decoder.Decode(itemData, colorProfile, codecConfiguration, item.Av1LayeredImageIndex); + Image image = decoder.Decode( + itemData, + colorProfile, + codecConfiguration, + item.Av1LayeredImageIndex, + item.Extent); + HeifMetadata metadata = image.Metadata.GetHeifMetadata(); metadata.CompressionMethod = this.CompressionMethod; metadata.BitDepth = codecConfiguration.BitDepth; diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReconstructionConformanceTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReconstructionConformanceTests.cs index b2e1a1709..cec60d1b8 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReconstructionConformanceTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReconstructionConformanceTests.cs @@ -473,6 +473,38 @@ public class Av1ReconstructionConformanceTests ValidateProgressiveSingleReferenceFixture(configuration, verifyPresentation: false); } + /// + /// Verifies that an essential lsel property returns the selected base spatial layer rather than the final + /// progressive layer, with exact pinned-libaom native planes and pinned-libavif presentation. + /// + [Fact] + public void DecodeSelectedProgressiveSpatialLayerMatchesPinnedReferences() + => FeatureTestRunner.RunWithHwIntrinsicsFeature( + ValidateSelectedProgressiveSpatialLayerWithDefaultConfiguration, + ReconstructionConfigurations); + + /// + /// Verifies selected-layer native reconstruction and public presentation with constrained tracked allocation. + /// + [Fact] + [ValidateDisposedMemoryAllocations] + public void DecodeSelectedProgressiveSpatialLayerWithConstrainedAllocator() + { + TestMemoryAllocator allocator = new() { BufferCapacityInBytes = 1_024 }; + allocator.EnableNonThreadSafeLogging(); + Configuration configuration = Configuration.Default.Clone(); + configuration.MemoryAllocator = allocator; + + ValidateSelectedProgressiveSpatialLayer(configuration); + + Assert.Equal(allocator.AllocationLog.Count, allocator.ReturnLog.Count); + Assert.All( + allocator.AllocationLog, + allocation => Assert.Single( + allocator.ReturnLog, + returned => returned.AllocationId == allocation.AllocationId)); + } + /// /// Verifies an independently encoded 40x40 retained layer scaled into an 80x80 dependent layer against exact /// pinned-libaom native planes and pinned-libavif presentation. @@ -1856,6 +1888,12 @@ public class Av1ReconstructionConformanceTests private static void ValidateProgressiveSingleReferenceFixtureWithDefaultConfiguration() => ValidateProgressiveSingleReferenceFixture(Configuration.Default, verifyPresentation: true); + /// + /// Runs the selected-spatial-layer native and presentation comparisons with the default configuration. + /// + private static void ValidateSelectedProgressiveSpatialLayerWithDefaultConfiguration() + => ValidateSelectedProgressiveSpatialLayer(Configuration.Default); + /// /// Runs the exact scaled-reference native and presentation comparison with the default configuration. /// @@ -1996,6 +2034,62 @@ public class Av1ReconstructionConformanceTests ImageComparer.Exact.VerifySimilarity(presentationReference, image); } + /// + /// Verifies the selected base spatial layer with the requested allocator. + /// + /// The decoder configuration. + private static void ValidateSelectedProgressiveSpatialLayer(Configuration configuration) + { + byte[] payload = TestFile.Create(TestImages.Heif.Av1ScaledReferencePayload).Bytes; + byte[] nativeReference = + TestFile.Create(TestImages.Heif.Av1ScaledReferenceBaseNativeReference).Bytes; + + Assert.Equal(ScaledReferenceBaseLayerSize * ScaledReferenceBaseLayerSize * 3, nativeReference.Length); + + Av1LayeredImageIndex layeredImageIndex = new(ScaledReferenceFirstLayerSize, 0, 0); + int selectedPayloadLength = layeredImageIndex.GetPayloadLength( + payload.Length, + new Av1LayerSelector(0)); + + Assert.Equal(ScaledReferenceFirstLayerSize, selectedPayloadLength); + + using Av1Decoder decoder = new(configuration); + using Av1FrameBuffer frameBuffer = decoder.DecodeFrameBuffer( + payload.AsSpan(0, selectedPayloadLength), + null, + null, + out _, + layeredImageIndex); + + Assert.Equal(ScaledReferenceBaseLayerSize, frameBuffer.Width); + Assert.Equal(ScaledReferenceBaseLayerSize, frameBuffer.Height); + Assert.Equal(Av1BitDepth.EightBit, frameBuffer.BitDepth); + Assert.Equal(Av1ColorFormat.Yuv444, frameBuffer.ColorFormat); + Assert.Equal(ObuFrameType.KeyFrame, Assert.IsType(decoder.FrameHeader).FrameType); + AssertNativePlanesEqual(decoder, frameBuffer, nativeReference); + + DecoderOptions options = new() { Configuration = configuration, MaxFrames = 1 }; + byte[] imageBytes = TestFile.Create(TestImages.Heif.Av1ScaledReferenceSelectedLayerAvif).Bytes; + byte[] presentationBytes = + TestFile.Create(TestImages.Heif.Av1ScaledReferenceSelectedLayerPresentationReference).Bytes; + + byte[] finalPresentationBytes = + TestFile.Create(TestImages.Heif.Av1ScaledReferencePresentationReference).Bytes; + + using Image image = Image.Load(options, imageBytes); + using Image presentationReference = Image.Load(presentationBytes); + using Image finalPresentationReference = Image.Load(finalPresentationBytes); + + // HEIF presents a selected lower-resolution spatial layer at the item's ispe extent. Exact comparison with + // libavif therefore proves both layer selection and the required 40x40-to-80x80 presentation scaling. + Assert.Equal(ScaledReferenceFixtureSize, image.Width); + Assert.Equal(ScaledReferenceFixtureSize, image.Height); + Assert.Single(image.Frames); + Assert.Equal(HeifBitDepth.Bit8, image.Metadata.GetHeifMetadata().BitDepth); + ImageComparer.Exact.VerifySimilarity(presentationReference, image); + Assert.NotEmpty(ImageComparer.Exact.CompareImages(finalPresentationReference, image)); + } + /// /// Verifies the final dependent layer with the requested allocator. /// diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1YuvConverterTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1YuvConverterTests.cs index 0ec3a7145..704230f40 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1YuvConverterTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1YuvConverterTests.cs @@ -253,12 +253,108 @@ public class Av1YuvConverterTests // Assert Assert.Equal(2, frameBuffer.BytesPerSample); - Assert.Equal(3 + 144, stride); + Assert.Equal(3 + (frameBuffer.OriginX * 2), stride); Assert.Equal(stride * 2, frameBuffer.BufferY!.Width); Assert.Equal(321, frameBuffer.GetHighBitDepthRowSpan(Av1Plane.Y, 0, 0, 0)[0]); Assert.Equal(2, chromaRow.Length); } + /// + /// Verifies libyuv's native two-times presentation filter at byte and twelve-bit precision across every + /// available intrinsic width and the scalar fallback. + /// + [Fact] + public void ScaleSelectedSpatialLayerMatchesPinnedLibyuvAcrossIntrinsicWidths() + => FeatureTestRunner.RunWithHwIntrinsicsFeature( + ValidateSelectedSpatialLayerScaling, + AlphaConfigurations); + + /// + /// Verifies the exact edge extension, quarter-sample weights, and rounding of the native presentation scaler. + /// + private static void ValidateSelectedSpatialLayerScaling() + { + byte[][] expectedByteRows = + [ + [0, 25, 75, 125, 175, 200], + [13, 38, 88, 138, 188, 213], + [38, 63, 113, 163, 213, 238], + [50, 75, 125, 175, 225, 250], + ]; + + ushort[][] expectedHighBitDepthRows = + [ + [0, 250, 750, 1250, 1750, 2000], + [125, 375, 875, 1375, 1875, 2125], + [375, 625, 1125, 1625, 2125, 2375], + [500, 750, 1250, 1750, 2250, 2500], + ]; + + ObuSequenceHeader byteSequenceHeader = CreateSequenceHeader( + 3, + 2, + colorFormat: Av1ColorFormat.Yuv400); + + using (Av1FrameBuffer frameBuffer = new( + Configuration.Default, + byteSequenceHeader, + Av1ColorFormat.Yuv400, + false)) + { + byte[] sourceSamples = [0, 100, 200, 50, 150, 250]; + for (int y = 0; y < byteSequenceHeader.MaxFrameHeight; y++) + { + sourceSamples.AsSpan(y * byteSequenceHeader.MaxFrameWidth, byteSequenceHeader.MaxFrameWidth).CopyTo( + frameBuffer.DeriveBlockPointer(Av1Plane.Y, 0, 0).DangerousGetRowSpan(y)); + } + + Av1PlanarSampleBuffer source = new(frameBuffer); + using Av1PresentationSampleBuffer> presentation = new( + Configuration.Default, + source, + 6, + 4); + + for (int y = 0; y < expectedByteRows.Length; y++) + { + Assert.True(presentation.View.GetLumaRowSpan(y).SequenceEqual(expectedByteRows[y])); + } + } + + ObuSequenceHeader highBitDepthSequenceHeader = CreateSequenceHeader( + 3, + 2, + colorFormat: Av1ColorFormat.Yuv400, + bitDepth: Av1BitDepth.TwelveBit); + + using Av1FrameBuffer highBitDepthFrameBuffer = new( + Configuration.Default, + highBitDepthSequenceHeader, + Av1ColorFormat.Yuv400, + false); + + ushort[] highBitDepthSourceSamples = [0, 1000, 2000, 500, 1500, 2500]; + for (int y = 0; y < highBitDepthSequenceHeader.MaxFrameHeight; y++) + { + highBitDepthSourceSamples.AsSpan( + y * highBitDepthSequenceHeader.MaxFrameWidth, + highBitDepthSequenceHeader.MaxFrameWidth).CopyTo( + highBitDepthFrameBuffer.GetHighBitDepthRowSpan(Av1Plane.Y, y, 0, 0)); + } + + Av1PlanarSampleBuffer highBitDepthSource = new(highBitDepthFrameBuffer); + using Av1PresentationSampleBuffer> highBitDepthPresentation = new( + Configuration.Default, + highBitDepthSource, + 6, + 4); + + for (int y = 0; y < expectedHighBitDepthRows.Length; y++) + { + Assert.True(highBitDepthPresentation.View.GetLumaRowSpan(y).SequenceEqual(expectedHighBitDepthRows[y])); + } + } + /// /// Verifies centered horizontal chroma reconstruction for a YUV 4:2:2 frame. /// diff --git a/tests/ImageSharp.Tests/TestImages.cs b/tests/ImageSharp.Tests/TestImages.cs index b24feddf7..73749cfd4 100644 --- a/tests/ImageSharp.Tests/TestImages.cs +++ b/tests/ImageSharp.Tests/TestImages.cs @@ -1358,10 +1358,12 @@ public static class TestImages public const string Av1Progressive8BitReference = "Heif/Av1/Conformance/libavif-progressive-draw-points-8b-libaom-y4m.yuv"; public const string Av1Progressive8BitPresentationReference = "Heif/Av1/Conformance/libavif-progressive-draw-points-8b.png"; public const string Av1ScaledReferenceAvif = "Heif/Av1/Conformance/libavif-webp-logo-scaled-reference.avif"; + public const string Av1ScaledReferenceSelectedLayerAvif = "Heif/Av1/Conformance/libavif-webp-logo-scaled-reference-lsel0.avif"; public const string Av1ScaledReferencePayload = "Heif/Av1/Conformance/libavif-webp-logo-scaled-reference.bit"; public const string Av1ScaledReferenceBaseNativeReference = "Heif/Av1/Conformance/libavif-webp-logo-scaled-reference-base-libaom.yuv"; public const string Av1ScaledReferenceNativeReference = "Heif/Av1/Conformance/libavif-webp-logo-scaled-reference-libaom.y4m"; public const string Av1ScaledReferencePresentationReference = "Heif/Av1/Conformance/libavif-webp-logo-scaled-reference.png"; + public const string Av1ScaledReferenceSelectedLayerPresentationReference = "Heif/Av1/Conformance/libavif-webp-logo-scaled-reference-lsel0-libavif.png"; public const string Av1AverageCompoundSequenceAvif = "Heif/Av1/Conformance/libavif-webp-logo-average-compound.avif"; public const string Av1AverageCompoundSequenceNativeReference = "Heif/Av1/Conformance/libavif-webp-logo-average-compound-libaom.y4m"; public const string Av1AverageCompoundSequencePresentationReference = "Heif/Av1/Conformance/libavif-webp-logo-average-compound-libavif.png"; diff --git a/tests/Images/Input/Heif/Av1/Conformance/libavif-webp-logo-scaled-reference-lsel0-libavif.png b/tests/Images/Input/Heif/Av1/Conformance/libavif-webp-logo-scaled-reference-lsel0-libavif.png new file mode 100644 index 000000000..1b0c1e476 --- /dev/null +++ b/tests/Images/Input/Heif/Av1/Conformance/libavif-webp-logo-scaled-reference-lsel0-libavif.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:678c5e5d4650ea6f0c590302e7db9e3c6608851bc577453da4a6837bdb4d3af3 +size 6453 diff --git a/tests/Images/Input/Heif/Av1/Conformance/libavif-webp-logo-scaled-reference-lsel0.avif b/tests/Images/Input/Heif/Av1/Conformance/libavif-webp-logo-scaled-reference-lsel0.avif new file mode 100644 index 000000000..6b8111e84 --- /dev/null +++ b/tests/Images/Input/Heif/Av1/Conformance/libavif-webp-logo-scaled-reference-lsel0.avif @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e6e1f53f8dd558c5309e0c80b0635fa4efcbb636541e4e38206985a91e0fba66 +size 2505