From 780189159b61818cc2d8245b25c165859c76eb73 Mon Sep 17 00:00:00 2001 From: James Jackson-South Date: Thu, 27 Aug 2026 05:04:07 +1000 Subject: [PATCH] Implement AV1 super-resolution filtering --- HEIF_IMPLEMENTATION_PLAN.md | 10 +- .../Av1LoopRestorationBoundary.cs | 84 ++-- .../Av1SuperResolutionDecoder.cs | 67 +-- .../Av1SuperResolutionFilter.cs | 416 ++++++++++++++++++ .../Av1SuperResolutionKernels.cs | 182 -------- .../Heif/Av1SuperResolutionBenchmarks.cs | 145 ++++++ .../Av1/Av1ReconstructionConformanceTests.cs | 78 +++- tests/ImageSharp.Tests/TestImages.cs | 6 + .../Input/Heif/Av1/Conformance/README.md | 4 + .../libaom-superres-cosmos-10b-libaom.yuv | 3 + .../libaom-superres-cosmos-10b.bit | 3 + .../libaom-superres-cosmos-12b-libaom.yuv | 3 + .../libaom-superres-cosmos-12b.bit | 3 + .../libaom-superres-kodim23-8b-libaom.yuv | 3 + .../libaom-superres-kodim23-8b.bit | 3 + 15 files changed, 752 insertions(+), 258 deletions(-) create mode 100644 src/ImageSharp/Formats/Heif/Av1/Pipeline/SuperResolution/Av1SuperResolutionFilter.cs delete mode 100644 src/ImageSharp/Formats/Heif/Av1/Pipeline/SuperResolution/Av1SuperResolutionKernels.cs create mode 100644 tests/ImageSharp.Benchmarks/Codecs/Heif/Av1SuperResolutionBenchmarks.cs create mode 100644 tests/Images/Input/Heif/Av1/Conformance/libaom-superres-cosmos-10b-libaom.yuv create mode 100644 tests/Images/Input/Heif/Av1/Conformance/libaom-superres-cosmos-10b.bit create mode 100644 tests/Images/Input/Heif/Av1/Conformance/libaom-superres-cosmos-12b-libaom.yuv create mode 100644 tests/Images/Input/Heif/Av1/Conformance/libaom-superres-cosmos-12b.bit create mode 100644 tests/Images/Input/Heif/Av1/Conformance/libaom-superres-kodim23-8b-libaom.yuv create mode 100644 tests/Images/Input/Heif/Av1/Conformance/libaom-superres-kodim23-8b.bit diff --git a/HEIF_IMPLEMENTATION_PLAN.md b/HEIF_IMPLEMENTATION_PLAN.md index aa374676da..f733d7b6cf 100644 --- a/HEIF_IMPLEMENTATION_PLAN.md +++ b/HEIF_IMPLEMENTATION_PLAN.md @@ -270,7 +270,7 @@ This snapshot pins or classifies the available references and failures; it does | `Av1TileReader` palette mode/color-map parsing, `Av1SymbolDecoder` palette distributions, `Av1BlockModeInfo` palette state, and `Av1PredictionDecoder` palette reconstruction | AV1 sections 5.11.46, 5.11.49, and 7.11.2 palette prediction | libaom `av1/decoder/decodemv.c`, `av1/decoder/detokenize.c`, `av1/decoder/decoder.h`, `av1/common/pred_common.c`, `av1/common/pred_common.h`, and `av1/common/entropymode.c` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Reuse the existing tile range decoder and frame-owned mode map while porting the normative palette cache merge, high-bit-depth color deltas, tile-adaptive mode/size/index distributions, diagonal color-map traversal, edge padding, and direct palette-sample reconstruction. This is AV1 still-image compression syntax and does not add retained video reference state or any ISO BMFF surface. | | `Av1DeblockingFilter`, `Av1LoopFilterContext`, and `Av1LoopFilterDecoder` | AV1 section 7.14 deblocking loop filter | libaom `aom_dsp/loopfilter.c` and `av1/common/av1_loopfilter.c` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Design the 8- and high-bit-depth edge layout and 4-, 6-, 8-, and 14-tap kernels for SIMD-first traversal, with sharpness thresholds, still-frame intra filter-level derivation, transform-edge selection, and plane traversal shared by the exact scalar fallback. Implement and verify both through the same contract before enabling the stage. This is normative AV1 image reconstruction and adds neither generic ISO BMFF models nor retained video reference state. | | `Av1CdefDecoder`, `Av1CdefFilter`, and CDEF-unit strength storage | AV1 sections 7.15.2 through 7.15.4 constrained directional enhancement filtering | libaom `av1/common/cdef.c`, `av1/common/cdef_block.c`, `av1/common/cdef.h`, and `av1/common/cdef_block.h` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Design direction search and constrained primary/secondary filtering around vector-friendly block storage and SIMD-first kernels, with variance adjustment, subsampling direction conversion, skipped-8x8 selection, and frame-edge sentinel behavior shared by the scalar fallback. Use a frame-owned source snapshot so filtering never consumes already modified samples. This is normative AV1 still-image reconstruction and introduces no ISO BMFF, track, timing, or sequence-playback surface. | -| `Av1SuperResolutionDecoder`, `Av1SuperResolutionKernels`, frame-size derivation, and decoded-image dimensions | AV1 section 7.16 normative super-resolution upscaling | libaom `av1/common/resize.c`, `av1/common/resize.h`, `av1/common/convolve.c`, and `aom_dsp/aom_filter.h` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Port the fixed 64-phase, 8-tap horizontal filter, phase/step derivation, replicated frame edges, chroma width rounding, signed rounding, and 8/10/12-bit clipping. Reuse ImageSharp's existing cross-platform `Vector128_.MultiplyAddAdjacent` helper for the exact eight-coefficient dot product with a scalar fallback. Generic image resizing is not normative AV1 super-resolution. This adds no track, timing, fragment, animation, or generic ISO BMFF model. | +| `Av1SuperResolutionDecoder`, `Av1SuperResolutionFilter`, frame-size derivation, and decoded-image dimensions | AV1 section 7.16 normative super-resolution upscaling | libaom `av1/common/resize.c`, `av1/common/resize.h`, `av1/common/convolve.c`, `av1/common/x86/av1_convolve_horiz_rs_sse4.c`, and `aom_dsp/aom_filter.h` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Port the fixed 64-phase, 8-tap horizontal filter, phase/step derivation, replicated frame edges, chroma width rounding, signed rounding, and 8/10/12-bit clipping. Follow libaom's four-output multiply/add, two-stage horizontal reduction, packed-store traversal with exact scalar fallback. Generic image resizing is not normative AV1 super-resolution. This adds no track, timing, fragment, animation, or generic ISO BMFF model. | | `Av1TileReader` loop-restoration unit syntax, `Av1SymbolDecoder` restoration distributions/subexponential codes, and `Av1FrameInfo` unit storage | AV1 section 5.11.57 `read_lr` and `read_lr_unit` syntax | libaom `av1/decoder/decodeframe.c`, `av1/common/restoration.c`, `av1/common/restoration.h`, `av1/common/entropymode.c`, `aom_dsp/binary_codes_reader.c`, and `aom_dsp/recenter.h` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Decode tile-local switchable/Wiener/self-guided selections, finite reference-subexponential coefficients, chroma Wiener windows, self-guided parameter sets, super-resolution-adjusted unit corners, and the AV1 nearest-unit-count rule into frame-owned per-plane grids. This is compressed still-image syntax and adds no movie, track, timing, fragment, audio, or sequence surface. | | `Av1WienerFilter` | AV1 sections 7.17.4 and 7.17.5 Wiener restoration filtering and coefficient derivation | libaom `av1/common/restoration.c`, `av1/common/restoration.h`, `av1/common/convolve.c`, and `av1/common/convolve.h` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Preserve the implicit center-sample contribution, separable horizontal/vertical rounding, bit-depth-dependent 16-bit intermediate range, and final 8/10/12-bit clipping. Reuse `Vector128_.MultiplyAddAdjacent` for the contiguous horizontal eight-tap product with an exact scalar fallback. Keep the restoration stage disabled until stripe boundaries and self-guided filtering are both complete. | | `Av1SelfGuidedFilter` | AV1 sections 7.17.2 and 7.17.3 self-guided and box-filter processes | libaom `av1/common/restoration.c` and `av1/common/restoration.h` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Design the window layout, caller-owned scratch, local statistics, and projection traversal from libaom's architecture-specific kernels so the production path is SIMD-first. Implement the equivalent scalar fallback through the same contract. Preserve the sixteen normative radius/variance parameter sets, local mean and variance normalization, alternating-row radius-two optimization, decoded projection-coefficient behavior, signed rounding, and 8/10/12-bit clipping. Keep this image-reconstruction stage disabled until restoration stripe boundaries are complete. | @@ -352,7 +352,7 @@ This assessment was reconciled with the source tree on 2026-08-26. Unless a resu - The single-still `Av1Decoder` path now parses tile state before allocating and reconstructing one independently decodable frame, and it disposes the reconstruction planes after pixel conversion. It deliberately does not retain animation/video reference frames or implement `show_existing_frame` playback state. - Transform coefficient entropy derivation and updates now address the above contexts relative to the tile column and the left contexts relative to the current superblock row, preserve luma coordinates independently of chroma subsampling, and test every packed context entry for the libaom any-nonzero rule. Extended vertical partition updates advance the mode-information column rather than the row. The existing multi-superblock 4:4:4 AVIF fixture now completes tile parsing; independent coefficient-context vectors across tile boundaries, chroma layouts, bit depths, and edge-clipped transforms remain required. -- The reconstruction pipeline now records plane-relative transform geometry, preserves tile-local delta-Q and delta-LF predictors, derives segmentation and reference-adjusted filter levels, and runs the exact AV1 4-, 6-, 8-, and 14-tap deblocking kernels in normative vertical-then-horizontal order. Deblocking uses the same closed edge-operator architecture as the HEVC filter, with operators specialized by sample storage and orientation, `Vector128` lanes representing the four rows or columns along an edge, and an allocation-free scalar fallback for disabled intrinsics. Exact native-plane comparison with pinned scalar libaom output now verifies active deblocking and complete reconstruction for real 8-bit 4:2:0, 10-bit 4:4:4, and 12-bit 4:4:4 content; genuine AVIF containers separately verify presentation and public bit-depth metadata. The pipeline then applies CDEF through one semantic filter architecture: paired AVX2 and single-block `Vector128` direction analysis, closed primary/secondary strength operators, packed 4x4/4x8/8x4/8x8 constrained filtering, byte/16-bit output operators, and an exact allocation-free scalar fallback. Decoder orchestration now owns the immutable plane snapshots and clean direction/variance maps through ImageSharp's memory allocator, widens 8-bit source rows with the same AVX2/128-bit/scalar tiers as libaom, lists each unit's non-skipped blocks in fixed inline storage, analyzes listed blocks in pairs, and writes filtered bytes or 16-bit samples directly to the frame planes. Independently encoded active-CDEF 8-bit 4:2:0 and 10/12-bit 4:4:4 streams match every visible native sample produced by pinned scalar libaom with restoration disabled. Independently encoded AVIF containers at the same three bit depths also match pinned scalar-libavif presentation exactly under normal, 256-bit, 128-bit, and scalar color-conversion dispatch. Active super-resolution derives the Appendix A bounded coded width and applies the exact 64-phase, 8-tap horizontal filter with aligned reconstruction-edge input, 8/10/12-bit clipping, and the existing cross-platform `Vector128_.MultiplyAddAdjacent` helper. Loop restoration follows super-resolution, preserves the required pre-CDEF deblocked context at internal stripes, and applies decoded Wiener or self-guided units from immutable plane snapshots. The visible still-image path then applies the complete self-contained film-grain parameter set after all in-loop filters. Independent 8-, 10-, and 12-bit vectors exercising super-resolution, restoration, and grain remain required before those stages have external pixel-level verification. +- The reconstruction pipeline now records plane-relative transform geometry, preserves tile-local delta-Q and delta-LF predictors, derives segmentation and reference-adjusted filter levels, and runs the exact AV1 4-, 6-, 8-, and 14-tap deblocking kernels in normative vertical-then-horizontal order. Deblocking uses the same closed edge-operator architecture as the HEVC filter, with operators specialized by sample storage and orientation, `Vector128` lanes representing the four rows or columns along an edge, and an allocation-free scalar fallback for disabled intrinsics. Exact native-plane comparison with pinned scalar libaom output now verifies active deblocking and complete reconstruction for real 8-bit 4:2:0, 10-bit 4:4:4, and 12-bit 4:4:4 content; genuine AVIF containers separately verify presentation and public bit-depth metadata. The pipeline then applies CDEF through one semantic filter architecture: paired AVX2 and single-block `Vector128` direction analysis, closed primary/secondary strength operators, packed 4x4/4x8/8x4/8x8 constrained filtering, byte/16-bit output operators, and an exact allocation-free scalar fallback. Decoder orchestration now owns the immutable plane snapshots and clean direction/variance maps through ImageSharp's memory allocator, widens 8-bit source rows with the same AVX2/128-bit/scalar tiers as libaom, lists each unit's non-skipped blocks in fixed inline storage, analyzes listed blocks in pairs, and writes filtered bytes or 16-bit samples directly to the frame planes. Independently encoded active-CDEF 8-bit 4:2:0 and 10/12-bit 4:4:4 streams match every visible native sample produced by pinned scalar libaom with restoration disabled. Independently encoded AVIF containers at the same three bit depths also match pinned scalar-libavif presentation exactly under normal, 256-bit, 128-bit, and scalar color-conversion dispatch. Active super-resolution derives the Appendix A bounded coded width and applies the exact 64-phase, 8-tap horizontal filter with aligned reconstruction-edge input, 8/10/12-bit clipping, and the existing cross-platform `Vector128_.MultiplyAddAdjacent` helper. Independently encoded active-super-resolution AV1 streams at 8, 10, and 12 bits match every visible native sample produced by pinned scalar libaom under normal and forced-scalar dispatch; equivalent AVIF presentation fixtures remain required. Loop restoration follows super-resolution, preserves the required pre-CDEF deblocked context at internal stripes, and applies decoded Wiener or self-guided units from immutable plane snapshots. The visible still-image path then applies the complete self-contained film-grain parameter set after all in-loop filters. Independent restoration and film-grain vectors remain required before those stages have external pixel-level verification. - Palette mode now reads the normative luma/chroma mode and size CDFs, neighbor color caches, high-bit-depth color syntax, diagonal color-index maps, and clipped-edge padding. Reconstruction retains libaom's exact indexed-palette contract while using repeated eight-entry tables and native lane-local shuffles across runtime-selected `Vector512`, `Vector256`, and `Vector128` paths before an exact scalar fallback. `FeatureTestRunner` verifies palette sizes 2-8, transform widths 4-64, padded rows, and 8/10/12-bit samples. This path is not production-complete until its syntax and reconstruction are verified with an independently encoded palette AVIF fixture. - Non-reduced still-image sequence parsing now consumes decoder-model operating parameters, temporal presentation fields, and OBU-layer-applicable buffer-removal fields only to preserve AV1 bit alignment. The scheduling values are not retained, and no movie, track, timing, playback, or generic ISO BMFF surface has been introduced. Existing focused sequence-header coverage exercises only reduced-still syntax, so an independent non-reduced still AVIF vector remains required. - Loop-restoration unit parsing records tile-local switchable/Wiener/self-guided filter selections and coefficients in frame-owned plane grids, including super-resolution-adjusted unit corners and the corrected conditional 64x64-superblock unit-size bit. The active restoration stage implements the normative unit geometry, striped deblocked boundaries, Wiener filtering, self-guided projection, and 8/10/12-bit clipping, while reusing the existing SIMD-backed super-resolution and adjacent multiply/add primitives. Independently encoded fixtures covering every parameter set, plane layout, bit depth, and frame-edge geometry are still required. Other normative independently decodable still-image syntax paths still contain `NotImplementedException` or equivalent unsupported branches. Tile-local palette CDF adaptation is present; the remaining still-image frame-context behavior requires a separate source audit without introducing sequence playback state. @@ -390,6 +390,8 @@ This assessment was reconciled with the source tree on 2026-08-26. Unless a resu Expose HEIC and AVIF as distinct public format identities backed by one internal HEIF/ISO BMFF container implementation. Provide format-appropriate decoders, encoders, metadata, options, MIME types, extensions, and generated save methods. Generic `.heif` and `.hif` input must be dispatched by brands and primary item type to HEVC, AV1, or legacy JPEG rather than by extension. Avoid a broad internal namespace rename until the container model is stable. +Name folders, namespaces, and types for the image-format behavior they own. SIMD width, ISA, sample storage, and bit depth remain internal operator or dispatch details; do not create `.Simd`, `.LowBitDepth`, width-specific, or equivalent implementation-mechanic namespaces, files, or type suffixes. + Generate the standard sync and async `SaveAsHeic` and `SaveAsAvif` overloads through the existing format templates. If a generic `SaveAsHeif` API is retained, its encoder options must require an explicit supported payload codec rather than infer one from an extension. Public XML documentation must describe observable behavior and option effects only. ### Parsed container model @@ -511,6 +513,9 @@ Implement and verify in dependency order: - [x] Verify CDEF syntax, selected nonzero strengths, unit/block traversal, subsampling, frame edges, and exact native reconstruction with independently encoded active-CDEF AV1 streams across 8/10/12-bit samples and pinned scalar-libaom output. - [x] Verify exact presented reconstruction with independently encoded active-CDEF AVIF containers across 8/10/12-bit samples and pinned scalar-libavif output. `FeatureTestRunner` covers normal dispatch, AVX-512 disabled, AVX disabled, and all hardware intrinsics disabled without a comparer tolerance. - [ ] Super-resolution scaling. + - [x] Replace the per-output dot product with the libaom-shaped four-output SIMD filter, exact scalar remainder, compile-time coefficient span, direct decoder-padding source rows, and one allocator-owned overlap row shared by every plane. + - [x] Verify exact 8/10/12-bit native reconstruction against independently encoded active-super-resolution AV1 streams under normal SIMD dispatch and with hardware intrinsics disabled. + - [ ] Verify exact presented reconstruction with independently packaged AVIF containers across 8/10/12-bit samples. - [ ] Loop restoration. - [ ] Frame padding and film-grain synthesis when signaled. @@ -650,6 +655,7 @@ Tasks: - [x] Add a permanent frame-wide AV1 palette-reconstruction benchmark. On .NET 10, runtime-selected hardware measured 20.40 and 62.99 microseconds per padded 1920x1088 frame at 8 and 12 bits, compared with forced-scalar timings of 488.96 and 509.58 microseconds: 24.0 and 8.1 times faster with zero managed allocations. Forced `Vector512` measured 20.81 and 62.70 microseconds, so production continues to honor the runtime's preferred vector width instead of overriding this machine's `Vector256` selection. - [x] Add a permanent frame-wide AV1 lossless inverse-transform benchmark. Dense four-by-four reconstruction across a padded 1920x1088 frame measured 595.6 microseconds at 8 bits and 512.4 microseconds at 12 bits, compared with forced-scalar timings of 2.074 and 2.270 milliseconds: 3.5 and 4.4 times faster with zero managed allocations. - [x] Add a permanent frame-wide AV1 deblocking benchmark. Complete vertical and horizontal fourteen-tap passes across a padded 1920x1088 frame measured 4.040 milliseconds at 8 bits and 4.184 milliseconds at 12 bits, compared with forced-scalar timings of 9.416 and 9.410 milliseconds: 2.33 and 2.25 times faster with zero managed allocations. + - [x] Add a permanent frame-wide AV1 super-resolution benchmark. Upscaling a 1280x1080 coded luma plane to 1920x1080 measured 2.052 milliseconds at 8 bits and 2.045 milliseconds at 12 bits, compared with forced-scalar timings of 10.627 and 10.634 milliseconds: 5.18 and 5.20 times faster with zero managed allocations. BenchmarkDotNet could not change the Windows power plan, so retain these figures as an uncontrolled local baseline rather than a release comparison. - [ ] Implement vector paths only for confirmed hot loops, using existing `Vector128`, `Vector256`, and `Vector512` helper and dispatch patterns where supported. - [ ] Prioritize shared color conversion and pixel packing, chroma upsampling, inverse-transform add-and-clip, intra predictors, HEVC deblock/SAO, AV1 loop filter/CDEF/restoration, and contiguous grid copies. - [ ] Benchmark the complete decode color pipeline on representative 8/10/12-bit AVIF and HEIC images with and without embedded ICC profiles. Report absolute end-to-end timings and allocations in addition to the isolated YUV/CICP and ICC stage costs. diff --git a/src/ImageSharp/Formats/Heif/Av1/Pipeline/LoopRestoration/Av1LoopRestorationBoundary.cs b/src/ImageSharp/Formats/Heif/Av1/Pipeline/LoopRestoration/Av1LoopRestorationBoundary.cs index 211f886b1e..ba5e9006a0 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Pipeline/LoopRestoration/Av1LoopRestorationBoundary.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Pipeline/LoopRestoration/Av1LoopRestorationBoundary.cs @@ -160,17 +160,7 @@ internal class Av1LoopRestorationBoundary : IDisposable int planeHeight = Av1Math.DivideLog2Ceiling(frameSize.FrameHeight, subsamplingY); int stripeHeight = ProcessingStripeSize >> subsamplingY; int stripeOffset = ProcessingStripeOffset >> subsamplingY; - int upscaleSourceLength = usesSuperResolution - ? reconstructedWidth + (Av1SuperResolutionKernels.SourceBorder * 2) - : 0; - - using IMemoryOwner? upscaleSourceOwner = usesSuperResolution - ? this.frameBuffer.MemoryAllocator.Allocate(upscaleSourceLength) - : null; - - Span upscaleSource = upscaleSourceOwner is not null - ? upscaleSourceOwner.Memory.Span[..upscaleSourceLength] - : Span.Empty; + int sourceBorder = usesSuperResolution ? Av1SuperResolutionFilter.SourceBorder : 0; Span lowBitDepthPlane = default; Span highBitDepthPlane = default; @@ -179,7 +169,7 @@ internal class Av1LoopRestorationBoundary : IDisposable { Span signedPlane = this.frameBuffer.DeriveBlockPointer16( plane, - Point.Empty, + new Point(-sourceBorder, 0), subsamplingX, subsamplingY, out sourceStride); @@ -190,18 +180,18 @@ internal class Av1LoopRestorationBoundary : IDisposable { lowBitDepthPlane = this.frameBuffer.DeriveBlockPointer( plane, - Point.Empty, + new Point(-sourceBorder, 0), subsamplingX, subsamplingY, out sourceStride); } int step = usesSuperResolution - ? Av1SuperResolutionKernels.GetConvolveStep(codedWidth, upscaledWidth) + ? Av1SuperResolutionFilter.GetConvolveStep(codedWidth, upscaledWidth) : 0; int initialSubpixel = usesSuperResolution - ? Av1SuperResolutionKernels.GetInitialSubpixel(codedWidth, upscaledWidth, step) + ? Av1SuperResolutionFilter.GetInitialSubpixel(codedWidth, upscaledWidth, step) : 0; for (int stripe = 0; stripe < this.stripeCounts[planeIndex]; stripe++) @@ -220,7 +210,7 @@ internal class Av1LoopRestorationBoundary : IDisposable reconstructedWidth, step, initialSubpixel, - upscaleSource, + sourceBorder, this.GetBoundaryRow(this.rowsAbove, planeIndex, stripe, 0)); this.SaveDeblockedRow( @@ -231,7 +221,7 @@ internal class Av1LoopRestorationBoundary : IDisposable reconstructedWidth, step, initialSubpixel, - upscaleSource, + sourceBorder, this.GetBoundaryRow(this.rowsAbove, planeIndex, stripe, 1)); } @@ -247,7 +237,7 @@ internal class Av1LoopRestorationBoundary : IDisposable reconstructedWidth, step, initialSubpixel, - upscaleSource, + sourceBorder, this.GetBoundaryRow(this.rowsBelow, planeIndex, stripe, 0)); this.SaveDeblockedRow( @@ -258,7 +248,7 @@ internal class Av1LoopRestorationBoundary : IDisposable reconstructedWidth, step, initialSubpixel, - upscaleSource, + sourceBorder, this.GetBoundaryRow(this.rowsBelow, planeIndex, stripe, 1)); } } @@ -310,7 +300,7 @@ internal class Av1LoopRestorationBoundary : IDisposable Span topRow0 = this.GetBoundaryRow(this.rowsAbove, planeIndex, 0, 0); Span topRow1 = this.GetBoundaryRow(this.rowsAbove, planeIndex, 0, 1); - CopyFrameRow(lowBitDepthPlane, highBitDepthPlane, sourceStride, 0, topRow0); + CopyFrameRow(lowBitDepthPlane, highBitDepthPlane, sourceStride, 0, 0, topRow0); // Frame boundaries use post-CDEF/post-super-resolution samples and replicate the outer row. topRow0.CopyTo(topRow1); @@ -318,7 +308,7 @@ internal class Av1LoopRestorationBoundary : IDisposable int lastStripe = stripeCount - 1; Span bottomRow0 = this.GetBoundaryRow(this.rowsBelow, planeIndex, lastStripe, 0); Span bottomRow1 = this.GetBoundaryRow(this.rowsBelow, planeIndex, lastStripe, 1); - CopyFrameRow(lowBitDepthPlane, highBitDepthPlane, sourceStride, planeHeight - 1, bottomRow0); + CopyFrameRow(lowBitDepthPlane, highBitDepthPlane, sourceStride, planeHeight - 1, 0, bottomRow0); bottomRow0.CopyTo(bottomRow1); } } @@ -353,39 +343,51 @@ internal class Av1LoopRestorationBoundary : IDisposable /// The mode-info-aligned width supplying super-resolution taps. /// The fixed-point super-resolution source-position increment, or zero when unscaled. /// The first super-resolution source position, or zero when unscaled. - /// The replicated-edge super-resolution work row. + /// The decoder-padding samples preceding the visible source row. /// The preserved upscaled boundary row. private void SaveDeblockedRow( - ReadOnlySpan lowBitDepthPlane, - ReadOnlySpan highBitDepthPlane, + Span lowBitDepthPlane, + Span highBitDepthPlane, int sourceStride, int row, int reconstructedWidth, int step, int initialSubpixel, - Span upscaleSource, + int sourceBorder, Span destination) { - if (upscaleSource.IsEmpty) + if (sourceBorder == 0) { - CopyFrameRow(lowBitDepthPlane, highBitDepthPlane, sourceStride, row, destination); + CopyFrameRow(lowBitDepthPlane, highBitDepthPlane, sourceStride, row, 0, destination); return; } - int sourceStart = Av1SuperResolutionKernels.SourceBorder; - Span reconstructedSamples = upscaleSource.Slice(sourceStart, reconstructedWidth); - CopyFrameRow(lowBitDepthPlane, highBitDepthPlane, sourceStride, row, reconstructedSamples); - // Boundary rows must follow the same continuous phase and aligned right-edge behavior as // the full-frame super-resolution stage or restoration would see different stripe context. - upscaleSource[..sourceStart].Fill(reconstructedSamples[0]); - upscaleSource[(sourceStart + reconstructedWidth)..].Fill(reconstructedSamples[^1]); - Av1SuperResolutionKernels.UpscaleRow( - upscaleSource, - destination, - step, - initialSubpixel, - this.frameBuffer.BitDepth.GetBitCount()); + int sourceOffset = sourceStride + (row * sourceStride); + if (!highBitDepthPlane.IsEmpty) + { + Span source = highBitDepthPlane.Slice(sourceOffset, reconstructedWidth + (sourceBorder * 2)); + Span reconstructedSamples = source.Slice(sourceBorder, reconstructedWidth); + + source[..sourceBorder].Fill(reconstructedSamples[0]); + source[(sourceBorder + reconstructedWidth)..].Fill(reconstructedSamples[^1]); + Av1SuperResolutionFilter.UpscaleRow( + source, + destination, + step, + initialSubpixel, + this.frameBuffer.BitDepth.GetBitCount()); + + return; + } + + Span lowBitDepthSource = lowBitDepthPlane.Slice(sourceOffset, reconstructedWidth + (sourceBorder * 2)); + Span lowBitDepthReconstructedSamples = lowBitDepthSource.Slice(sourceBorder, reconstructedWidth); + + lowBitDepthSource[..sourceBorder].Fill(lowBitDepthReconstructedSamples[0]); + lowBitDepthSource[(sourceBorder + reconstructedWidth)..].Fill(lowBitDepthReconstructedSamples[^1]); + Av1SuperResolutionFilter.UpscaleRow(lowBitDepthSource, destination, step, initialSubpixel); } /// @@ -395,15 +397,17 @@ internal class Av1LoopRestorationBoundary : IDisposable /// The native 16-bit plane when the frame uses high-bit-depth samples. /// The number of samples between reconstructed rows. /// The zero-based visible row index. + /// The decoder-padding samples preceding the visible source row. /// The destination row whose length determines the copied width. private static void CopyFrameRow( ReadOnlySpan lowBitDepthPlane, ReadOnlySpan highBitDepthPlane, int sourceStride, int row, + int sourceBorder, Span destination) { - int sourceOffset = sourceStride + (row * sourceStride); + int sourceOffset = sourceStride + (row * sourceStride) + sourceBorder; if (!highBitDepthPlane.IsEmpty) { highBitDepthPlane.Slice(sourceOffset, destination.Length).CopyTo(destination); diff --git a/src/ImageSharp/Formats/Heif/Av1/Pipeline/SuperResolution/Av1SuperResolutionDecoder.cs b/src/ImageSharp/Formats/Heif/Av1/Pipeline/SuperResolution/Av1SuperResolutionDecoder.cs index cfedb69dd6..6f4eff0d32 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Pipeline/SuperResolution/Av1SuperResolutionDecoder.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Pipeline/SuperResolution/Av1SuperResolutionDecoder.cs @@ -1,6 +1,7 @@ // Copyright (c) Six Labors. // Licensed under the Six Labors Split License. +using System.Buffers; using System.Runtime.InteropServices; using SixLabors.ImageSharp.Formats.Heif.Av1.OpenBitstreamUnit; @@ -52,13 +53,16 @@ internal class Av1SuperResolutionDecoder int upscaledWidth = frameSize.SuperResolutionUpscaledWidth; if (codedWidth != upscaledWidth) { + int outputLength = upscaledWidth * this.frameBuffer.BytesPerSample; + using IMemoryOwner outputOwner = this.frameBuffer.MemoryAllocator.Allocate(outputLength); + Span output = outputOwner.Memory.Span[..outputLength]; ObuColorConfig colorConfig = this.sequenceHeader.ColorConfig; for (int planeIndex = 0; planeIndex < colorConfig.PlaneCount; planeIndex++) { Av1Plane plane = (Av1Plane)planeIndex; int subsamplingX = plane != Av1Plane.Y && colorConfig.SubSamplingX ? 1 : 0; int subsamplingY = plane != Av1Plane.Y && colorConfig.SubSamplingY ? 1 : 0; - this.UpscalePlane(plane, subsamplingX, subsamplingY, codedWidth, upscaledWidth, frameSize.FrameHeight); + this.UpscalePlane(plane, subsamplingX, subsamplingY, codedWidth, upscaledWidth, frameSize.FrameHeight, output); } } @@ -77,22 +81,22 @@ internal class Av1SuperResolutionDecoder /// The coded luma width before upscaling. /// The luma width after upscaling. /// The unchanged luma height. + /// The allocator-owned row used to prevent source and destination overlap. private void UpscalePlane( Av1Plane plane, int subsamplingX, int subsamplingY, int codedLumaWidth, int upscaledLumaWidth, - int lumaHeight) + int lumaHeight, + Span outputBuffer) { int codedWidth = Av1Math.DivideLog2Ceiling(codedLumaWidth, subsamplingX); int upscaledWidth = Av1Math.DivideLog2Ceiling(upscaledLumaWidth, subsamplingX); int reconstructedWidth = this.frameHeader.ModeInfoColumnCount << (Av1Constants.ModeInfoSizeLog2 - subsamplingX); int height = Av1Math.DivideLog2Ceiling(lumaHeight, subsamplingY); - int step = Av1SuperResolutionKernels.GetConvolveStep(codedWidth, upscaledWidth); - int initialSubpixel = Av1SuperResolutionKernels.GetInitialSubpixel(codedWidth, upscaledWidth, step); - ushort[] sourceRow = new ushort[reconstructedWidth + (Av1SuperResolutionKernels.SourceBorder * 2)]; - ushort[] outputRow = new ushort[upscaledWidth]; + int step = Av1SuperResolutionFilter.GetConvolveStep(codedWidth, upscaledWidth); + int initialSubpixel = Av1SuperResolutionFilter.GetInitialSubpixel(codedWidth, upscaledWidth, step); Span lowBitDepthPlane = default; Span highBitDepthPlane = default; @@ -101,7 +105,7 @@ internal class Av1SuperResolutionDecoder { Span signedPlane = this.frameBuffer.DeriveBlockPointer16( plane, - Point.Empty, + new Point(-Av1SuperResolutionFilter.SourceBorder, 0), subsamplingX, subsamplingY, out stride); @@ -112,51 +116,48 @@ internal class Av1SuperResolutionDecoder { lowBitDepthPlane = this.frameBuffer.DeriveBlockPointer( plane, - Point.Empty, + new Point(-Av1SuperResolutionFilter.SourceBorder, 0), subsamplingX, subsamplingY, out stride); } - int sourceStart = Av1SuperResolutionKernels.SourceBorder; + int sourceStart = Av1SuperResolutionFilter.SourceBorder; int bitDepth = this.frameBuffer.BitDepth.GetBitCount(); // libaom partitions the same continuous phase progression by tile column but does not pad // internal boundaries. Filtering the complete row therefore produces the identical samples. for (int row = 0; row < height; row++) { - Span reconstructedSamples = sourceRow.AsSpan(sourceStart, reconstructedWidth); int planeOffset = stride + (row * stride); if (this.frameBuffer.BytesPerSample == 2) { - highBitDepthPlane.Slice(planeOffset, reconstructedWidth).CopyTo(reconstructedSamples); - } - else - { - Span input = lowBitDepthPlane.Slice(planeOffset, reconstructedWidth); - for (int column = 0; column < reconstructedWidth; column++) - { - reconstructedSamples[column] = input[column]; - } - } + Span sourceRow = highBitDepthPlane.Slice(planeOffset, reconstructedWidth + (sourceStart * 2)); + Span reconstructedSamples = sourceRow.Slice(sourceStart, reconstructedWidth); + Span destinationSamples = highBitDepthPlane.Slice(planeOffset + sourceStart, upscaledWidth); + Span outputRow = MemoryMarshal.Cast(outputBuffer)[..upscaledWidth]; - // Phase derives from the exact coded width, but taps consume reconstruction through the - // enclosing 8-sample mode-info edge. Only samples beyond that aligned edge are replicated. - sourceRow.AsSpan(0, sourceStart).Fill(reconstructedSamples[0]); - sourceRow.AsSpan(sourceStart + reconstructedWidth, sourceStart).Fill(reconstructedSamples[^1]); - Av1SuperResolutionKernels.UpscaleRow(sourceRow, outputRow, step, initialSubpixel, bitDepth); + // The frame allocation already reserves decoder padding. Replicating its edge samples in place avoids + // copying the reconstructed row into a second working buffer before every filter pass. + sourceRow[..sourceStart].Fill(reconstructedSamples[0]); + sourceRow[(sourceStart + reconstructedWidth)..].Fill(reconstructedSamples[^1]); + Av1SuperResolutionFilter.UpscaleRow(sourceRow, outputRow, step, initialSubpixel, bitDepth); - if (this.frameBuffer.BytesPerSample == 2) - { - outputRow.CopyTo(highBitDepthPlane.Slice(planeOffset, upscaledWidth)); + // Horizontal expansion cannot write directly over its input because later taps still address samples + // to the right. The single pooled row is copied back only after all of its source samples are consumed. + outputRow.CopyTo(destinationSamples); } else { - Span output = lowBitDepthPlane.Slice(planeOffset, upscaledWidth); - for (int column = 0; column < upscaledWidth; column++) - { - output[column] = (byte)outputRow[column]; - } + Span sourceRow = lowBitDepthPlane.Slice(planeOffset, reconstructedWidth + (sourceStart * 2)); + Span reconstructedSamples = sourceRow.Slice(sourceStart, reconstructedWidth); + Span destinationSamples = lowBitDepthPlane.Slice(planeOffset + sourceStart, upscaledWidth); + Span outputRow = outputBuffer[..upscaledWidth]; + + sourceRow[..sourceStart].Fill(reconstructedSamples[0]); + sourceRow[(sourceStart + reconstructedWidth)..].Fill(reconstructedSamples[^1]); + Av1SuperResolutionFilter.UpscaleRow(sourceRow, outputRow, step, initialSubpixel); + outputRow.CopyTo(destinationSamples); } } } diff --git a/src/ImageSharp/Formats/Heif/Av1/Pipeline/SuperResolution/Av1SuperResolutionFilter.cs b/src/ImageSharp/Formats/Heif/Av1/Pipeline/SuperResolution/Av1SuperResolutionFilter.cs new file mode 100644 index 0000000000..fc8aecda95 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Pipeline/SuperResolution/Av1SuperResolutionFilter.cs @@ -0,0 +1,416 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using System.Runtime.Intrinsics; +using System.Runtime.Intrinsics.Arm; +using System.Runtime.Intrinsics.X86; +using SixLabors.ImageSharp.Common.Helpers; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.SuperResolution; + +/// +/// Applies the normative horizontal filter used by AV1 super-resolution upscaling. +/// +internal static class Av1SuperResolutionFilter +{ + /// + /// The number of source samples consumed for each output sample. + /// + public const int TapCount = 8; + + /// + /// The number of replicated source samples required at each row edge. + /// + public const int SourceBorder = (TapCount / 2) + 1; + + /// + /// The number of output samples evaluated together by the vector path. + /// + private const int OutputGroupSize = 4; + + /// + /// The number of fractional bits represented by each filter coefficient. + /// + private const int FilterBits = 7; + + /// + /// The rounding offset applied before removing the filter-coefficient fractional bits. + /// + private const int FilterRounding = 1 << (FilterBits - 1); + + /// + /// The number of filter phases in the normative table, expressed as a base-two exponent. + /// + private const int SubpixelBits = 6; + + /// + /// The number of fractional bits used by the horizontal sample position. + /// + private const int ScaleSubpixelBits = 14; + + /// + /// The mask selecting the fractional sample position. + /// + private const int ScaleSubpixelMask = (1 << ScaleSubpixelBits) - 1; + + /// + /// The number of low position bits omitted when selecting a filter phase. + /// + private const int ScaleExtraBits = ScaleSubpixelBits - SubpixelBits; + + /// + /// The half-step bias applied before filter-phase selection. + /// + private const int ScaleExtraOffset = 1 << (ScaleExtraBits - 1); + + /// + /// Gets the flattened 64-phase, 8-tap normative AV1 super-resolution filter table. + /// + private static ReadOnlySpan Filters => + [ + 0, 0, 0, 128, 0, 0, 0, 0, 0, 0, -1, 128, 2, -1, 0, 0, + 0, 1, -3, 127, 4, -2, 1, 0, 0, 1, -4, 127, 6, -3, 1, 0, + 0, 2, -6, 126, 8, -3, 1, 0, 0, 2, -7, 125, 11, -4, 1, 0, + -1, 2, -8, 125, 13, -5, 2, 0, -1, 3, -9, 124, 15, -6, 2, 0, + -1, 3, -10, 123, 18, -6, 2, -1, -1, 3, -11, 122, 20, -7, 3, -1, + -1, 4, -12, 121, 22, -8, 3, -1, -1, 4, -13, 120, 25, -9, 3, -1, + -1, 4, -14, 118, 28, -9, 3, -1, -1, 4, -15, 117, 30, -10, 4, -1, + -1, 5, -16, 116, 32, -11, 4, -1, -1, 5, -16, 114, 35, -12, 4, -1, + -1, 5, -17, 112, 38, -12, 4, -1, -1, 5, -18, 111, 40, -13, 5, -1, + -1, 5, -18, 109, 43, -14, 5, -1, -1, 6, -19, 107, 45, -14, 5, -1, + -1, 6, -19, 105, 48, -15, 5, -1, -1, 6, -19, 103, 51, -16, 5, -1, + -1, 6, -20, 101, 53, -16, 6, -1, -1, 6, -20, 99, 56, -17, 6, -1, + -1, 6, -20, 97, 58, -17, 6, -1, -1, 6, -20, 95, 61, -18, 6, -1, + -2, 7, -20, 93, 64, -18, 6, -2, -2, 7, -20, 91, 66, -19, 6, -1, + -2, 7, -20, 88, 69, -19, 6, -1, -2, 7, -20, 86, 71, -19, 6, -1, + -2, 7, -20, 84, 74, -20, 7, -2, -2, 7, -20, 81, 76, -20, 7, -1, + -2, 7, -20, 79, 79, -20, 7, -2, -1, 7, -20, 76, 81, -20, 7, -2, + -2, 7, -20, 74, 84, -20, 7, -2, -1, 6, -19, 71, 86, -20, 7, -2, + -1, 6, -19, 69, 88, -20, 7, -2, -1, 6, -19, 66, 91, -20, 7, -2, + -2, 6, -18, 64, 93, -20, 7, -2, -1, 6, -18, 61, 95, -20, 6, -1, + -1, 6, -17, 58, 97, -20, 6, -1, -1, 6, -17, 56, 99, -20, 6, -1, + -1, 6, -16, 53, 101, -20, 6, -1, -1, 5, -16, 51, 103, -19, 6, -1, + -1, 5, -15, 48, 105, -19, 6, -1, -1, 5, -14, 45, 107, -19, 6, -1, + -1, 5, -14, 43, 109, -18, 5, -1, -1, 5, -13, 40, 111, -18, 5, -1, + -1, 4, -12, 38, 112, -17, 5, -1, -1, 4, -12, 35, 114, -16, 5, -1, + -1, 4, -11, 32, 116, -16, 5, -1, -1, 4, -10, 30, 117, -15, 4, -1, + -1, 3, -9, 28, 118, -14, 4, -1, -1, 3, -9, 25, 120, -13, 4, -1, + -1, 3, -8, 22, 121, -12, 4, -1, -1, 3, -7, 20, 122, -11, 3, -1, + -1, 2, -6, 18, 123, -10, 3, -1, 0, 2, -6, 15, 124, -9, 3, -1, + 0, 2, -5, 13, 125, -8, 2, -1, 0, 1, -4, 11, 125, -7, 2, 0, + 0, 1, -3, 8, 126, -6, 2, 0, 0, 1, -3, 6, 127, -4, 1, 0, + 0, 1, -2, 4, 127, -3, 1, 0, 0, 0, -1, 2, 128, -1, 0, 0 + ]; + + /// + /// Derives the fixed-point source-position increment for an output row. + /// + /// The coded plane width. + /// The upscaled plane width. + /// The source-position increment with 14 fractional bits. + public static int GetConvolveStep(int inputLength, int outputLength) + => ((inputLength << ScaleSubpixelBits) + (outputLength / 2)) / outputLength; + + /// + /// Derives the initial fixed-point source position for an output row. + /// + /// The coded plane width. + /// The upscaled plane width. + /// The source-position increment returned by . + /// The initial source position with 14 fractional bits. + public static int GetInitialSubpixel(int inputLength, int outputLength, int step) + { + int error = (outputLength * step) - (inputLength << ScaleSubpixelBits); + int initial = (-((outputLength - inputLength) << (ScaleSubpixelBits - 1)) + (outputLength / 2)) / outputLength; + + initial += ScaleExtraOffset - (error / 2); + return initial & ScaleSubpixelMask; + } + + /// + /// Upscales one replicated-edge eight-bit source row into eight-bit output samples. + /// + /// The coded row with replicated samples on each edge. + /// The upscaled destination row. + /// The fixed-point source-position increment. + /// The initial fixed-point source position. + public static void UpscaleRow(ReadOnlySpan source, Span destination, int step, int initialSubpixel) + { + ref byte sourceBase = ref MemoryMarshal.GetReference(source); + ref byte destinationBase = ref MemoryMarshal.GetReference(destination); + int sourcePosition = initialSubpixel; + int column = 0; + if (Vector128.IsHardwareAccelerated) + { + for (; column <= destination.Length - OutputGroupSize; column += OutputGroupSize) + { + Vector128 filtered = FilterFour(ref sourceBase, sourcePosition, step); + Vector128 samples16 = Vector128_.PackUnsignedSaturate(filtered, Vector128.Zero); + Vector128 samples8 = Vector128_.PackUnsignedSaturate(samples16.AsInt16(), Vector128.Zero); + + Unsafe.WriteUnaligned(ref Unsafe.Add(ref destinationBase, column), samples8.AsUInt32().GetElement(0)); + sourcePosition += step * OutputGroupSize; + } + } + + for (; column < destination.Length; column++) + { + Unsafe.Add(ref destinationBase, column) = (byte)FilterOne(ref sourceBase, sourcePosition, byte.MaxValue); + sourcePosition += step; + } + } + + /// + /// Upscales one replicated-edge eight-bit source row into 16-bit output storage. + /// + /// The coded row with replicated samples on each edge. + /// The upscaled destination row. + /// The fixed-point source-position increment. + /// The initial fixed-point source position. + public static void UpscaleRow(ReadOnlySpan source, Span destination, int step, int initialSubpixel) + { + ref byte sourceBase = ref MemoryMarshal.GetReference(source); + ref ushort destinationBase = ref MemoryMarshal.GetReference(destination); + int sourcePosition = initialSubpixel; + int column = 0; + if (Vector128.IsHardwareAccelerated) + { + for (; column <= destination.Length - OutputGroupSize; column += OutputGroupSize) + { + Vector128 filtered = FilterFour(ref sourceBase, sourcePosition, step); + Vector128 samples = Vector128_.PackUnsignedSaturate(filtered, Vector128.Zero); + + samples.GetLower().StoreUnsafe(ref destinationBase, (nuint)column); + sourcePosition += step * OutputGroupSize; + } + } + + for (; column < destination.Length; column++) + { + Unsafe.Add(ref destinationBase, column) = (ushort)FilterOne(ref sourceBase, sourcePosition, byte.MaxValue); + sourcePosition += step; + } + } + + /// + /// Upscales one replicated-edge high-bit-depth source row into 16-bit output storage. + /// + /// The coded row with replicated samples on each edge. + /// The upscaled destination row. + /// The fixed-point source-position increment. + /// The initial fixed-point source position. + /// The encoded sample bit depth. + public static void UpscaleRow(ReadOnlySpan source, Span destination, int step, int initialSubpixel, int bitDepth) + { + ref ushort sourceBase = ref MemoryMarshal.GetReference(source); + ref ushort destinationBase = ref MemoryMarshal.GetReference(destination); + int maximum = (1 << bitDepth) - 1; + int sourcePosition = initialSubpixel; + int column = 0; + if (Vector128.IsHardwareAccelerated) + { + Vector128 maximumVector = Vector128.Create((ushort)maximum); + for (; column <= destination.Length - OutputGroupSize; column += OutputGroupSize) + { + Vector128 filtered = FilterFour(ref sourceBase, sourcePosition, step); + Vector128 samples = Vector128.Min(Vector128_.PackUnsignedSaturate(filtered, Vector128.Zero), maximumVector); + + samples.GetLower().StoreUnsafe(ref destinationBase, (nuint)column); + sourcePosition += step * OutputGroupSize; + } + } + + for (; column < destination.Length; column++) + { + Unsafe.Add(ref destinationBase, column) = (ushort)FilterOne(ref sourceBase, sourcePosition, maximum); + sourcePosition += step; + } + } + + /// + /// Evaluates four consecutive output positions from an eight-bit source row. + /// + /// The first sample in the replicated-edge source row. + /// The first fixed-point source position. + /// The fixed-point increment between output samples. + /// The rounded filter results in output order. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector128 FilterFour(ref byte source, int sourcePosition, int step) + { + GetOffsets(sourcePosition, out int sourceOffset0, out int filterOffset0); + GetOffsets(sourcePosition + step, out int sourceOffset1, out int filterOffset1); + GetOffsets(sourcePosition + (step * 2), out int sourceOffset2, out int filterOffset2); + GetOffsets(sourcePosition + (step * 3), out int sourceOffset3, out int filterOffset3); + + ref short filter = ref MemoryMarshal.GetReference(Filters); + Vector128 samples0 = Vector128.WidenLower(Vector128.CreateScalarUnsafe(Unsafe.ReadUnaligned(ref Unsafe.Add(ref source, sourceOffset0))).AsByte()).AsInt16(); + Vector128 samples1 = Vector128.WidenLower(Vector128.CreateScalarUnsafe(Unsafe.ReadUnaligned(ref Unsafe.Add(ref source, sourceOffset1))).AsByte()).AsInt16(); + Vector128 samples2 = Vector128.WidenLower(Vector128.CreateScalarUnsafe(Unsafe.ReadUnaligned(ref Unsafe.Add(ref source, sourceOffset2))).AsByte()).AsInt16(); + Vector128 samples3 = Vector128.WidenLower(Vector128.CreateScalarUnsafe(Unsafe.ReadUnaligned(ref Unsafe.Add(ref source, sourceOffset3))).AsByte()).AsInt16(); + return FilterFour( + samples0, + samples1, + samples2, + samples3, + Vector128.LoadUnsafe(ref filter, (nuint)filterOffset0), + Vector128.LoadUnsafe(ref filter, (nuint)filterOffset1), + Vector128.LoadUnsafe(ref filter, (nuint)filterOffset2), + Vector128.LoadUnsafe(ref filter, (nuint)filterOffset3)); + } + + /// + /// Evaluates four consecutive output positions from a high-bit-depth source row. + /// + /// The first sample in the replicated-edge source row. + /// The first fixed-point source position. + /// The fixed-point increment between output samples. + /// The rounded filter results in output order. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector128 FilterFour(ref ushort source, int sourcePosition, int step) + { + GetOffsets(sourcePosition, out int sourceOffset0, out int filterOffset0); + GetOffsets(sourcePosition + step, out int sourceOffset1, out int filterOffset1); + GetOffsets(sourcePosition + (step * 2), out int sourceOffset2, out int filterOffset2); + GetOffsets(sourcePosition + (step * 3), out int sourceOffset3, out int filterOffset3); + + ref short filter = ref MemoryMarshal.GetReference(Filters); + return FilterFour( + Vector128.LoadUnsafe(ref source, (nuint)sourceOffset0).AsInt16(), + Vector128.LoadUnsafe(ref source, (nuint)sourceOffset1).AsInt16(), + Vector128.LoadUnsafe(ref source, (nuint)sourceOffset2).AsInt16(), + Vector128.LoadUnsafe(ref source, (nuint)sourceOffset3).AsInt16(), + Vector128.LoadUnsafe(ref filter, (nuint)filterOffset0), + Vector128.LoadUnsafe(ref filter, (nuint)filterOffset1), + Vector128.LoadUnsafe(ref filter, (nuint)filterOffset2), + Vector128.LoadUnsafe(ref filter, (nuint)filterOffset3)); + } + + /// + /// Multiplies and reduces four independent eight-tap filter inputs. + /// + /// The source samples for the first output. + /// The source samples for the second output. + /// The source samples for the third output. + /// The source samples for the fourth output. + /// The coefficients for the first output. + /// The coefficients for the second output. + /// The coefficients for the third output. + /// The coefficients for the fourth output. + /// The rounded filter results in output order. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector128 FilterFour( + Vector128 samples0, + Vector128 samples1, + Vector128 samples2, + Vector128 samples3, + Vector128 filter0, + Vector128 filter1, + Vector128 filter2, + Vector128 filter3) + { + Vector128 products0 = Vector128_.MultiplyAddAdjacent(samples0, filter0); + Vector128 products1 = Vector128_.MultiplyAddAdjacent(samples1, filter1); + Vector128 products2 = Vector128_.MultiplyAddAdjacent(samples2, filter2); + Vector128 products3 = Vector128_.MultiplyAddAdjacent(samples3, filter3); + + // libaom reduces four independent filters in two horizontal-add stages so the four complete sums occupy + // consecutive lanes. Keeping that arrangement also allows both destination forms to use one packed store. + Vector128 pairs01 = HorizontalAdd(products0, products1); + Vector128 pairs23 = HorizontalAdd(products2, products3); + return (HorizontalAdd(pairs01, pairs23) + Vector128.Create(FilterRounding)) >> FilterBits; + } + + /// + /// Horizontally adds adjacent 32-bit lanes from two filter-product vectors. + /// + /// The first filter-product vector. + /// The second filter-product vector. + /// The adjacent sums from followed by those from . + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector128 HorizontalAdd(Vector128 left, Vector128 right) + { + if (Ssse3.IsSupported) + { + return Ssse3.HorizontalAdd(left, right); + } + + if (AdvSimd.Arm64.IsSupported) + { + return AdvSimd.Arm64.AddPairwise(left, right); + } + + if (AdvSimd.IsSupported) + { + Vector64 leftPairs = AdvSimd.AddPairwise(left.GetLower(), left.GetUpper()); + Vector64 rightPairs = AdvSimd.AddPairwise(right.GetLower(), right.GetUpper()); + return Vector128.Create(leftPairs, rightPairs); + } + + Vector128 evenIndices = Vector128.Create(0, 2, 0, 2); + Vector128 oddIndices = Vector128.Create(1, 3, 1, 3); + Vector128 leftPairsFallback = Vector128.ShuffleNative(left, evenIndices) + Vector128.ShuffleNative(left, oddIndices); + Vector128 rightPairsFallback = Vector128.ShuffleNative(right, evenIndices) + Vector128.ShuffleNative(right, oddIndices); + return Vector128.Create(leftPairsFallback.GetLower(), rightPairsFallback.GetLower()); + } + + /// + /// Evaluates one eight-bit source position for the scalar remainder. + /// + /// The first sample in the replicated-edge source row. + /// The fixed-point source position. + /// The largest permitted output sample. + /// The rounded and clipped output sample. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static int FilterOne(ref byte source, int sourcePosition, int maximum) + { + GetOffsets(sourcePosition, out int sourceOffset, out int filterOffset); + + ref short filter = ref MemoryMarshal.GetReference(Filters); + int sum = 0; + for (int tap = 0; tap < TapCount; tap++) + { + sum += Unsafe.Add(ref source, sourceOffset + tap) * Unsafe.Add(ref filter, filterOffset + tap); + } + + return Av1Math.Clip3(0, maximum, (sum + FilterRounding) >> FilterBits); + } + + /// + /// Evaluates one high-bit-depth source position for the scalar remainder. + /// + /// The first sample in the replicated-edge source row. + /// The fixed-point source position. + /// The largest permitted output sample. + /// The rounded and clipped output sample. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static int FilterOne(ref ushort source, int sourcePosition, int maximum) + { + GetOffsets(sourcePosition, out int sourceOffset, out int filterOffset); + + ref short filter = ref MemoryMarshal.GetReference(Filters); + int sum = 0; + for (int tap = 0; tap < TapCount; tap++) + { + sum += Unsafe.Add(ref source, sourceOffset + tap) * Unsafe.Add(ref filter, filterOffset + tap); + } + + return Av1Math.Clip3(0, maximum, (sum + FilterRounding) >> FilterBits); + } + + /// + /// Resolves the source and filter-table offsets for one fixed-point position. + /// + /// The fixed-point source position. + /// Receives the first source tap relative to the replicated-edge row. + /// Receives the first coefficient for the selected filter phase. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void GetOffsets(int sourcePosition, out int sourceOffset, out int filterOffset) + { + int integerPosition = sourcePosition >> ScaleSubpixelBits; + int filterPhase = (sourcePosition & ScaleSubpixelMask) >> ScaleExtraBits; + sourceOffset = SourceBorder + integerPosition - (TapCount / 2); + filterOffset = filterPhase * TapCount; + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Pipeline/SuperResolution/Av1SuperResolutionKernels.cs b/src/ImageSharp/Formats/Heif/Av1/Pipeline/SuperResolution/Av1SuperResolutionKernels.cs deleted file mode 100644 index 3982c2eb36..0000000000 --- a/src/ImageSharp/Formats/Heif/Av1/Pipeline/SuperResolution/Av1SuperResolutionKernels.cs +++ /dev/null @@ -1,182 +0,0 @@ -// Copyright (c) Six Labors. -// Licensed under the Six Labors Split License. - -using System.Runtime.CompilerServices; -using System.Runtime.InteropServices; -using System.Runtime.Intrinsics; -using SixLabors.ImageSharp.Common.Helpers; - -namespace SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.SuperResolution; - -/// -/// Provides the normative horizontal filter operations used by AV1 super-resolution upscaling. -/// -internal static class Av1SuperResolutionKernels -{ - /// - /// The number of source samples consumed for each output sample. - /// - public const int FilterTapCount = 8; - - /// - /// The number of replicated source samples reserved at each row edge. - /// - public const int SourceBorder = (FilterTapCount / 2) + 1; - - /// - /// The number of fractional bits represented by each filter coefficient. - /// - private const int FilterBits = 7; - - /// - /// The number of filter phases in the normative table, expressed as a base-two exponent. - /// - private const int SubpixelBits = 6; - - /// - /// The number of fractional bits used by the horizontal sample position. - /// - private const int ScaleSubpixelBits = 14; - - /// - /// The mask selecting the fractional sample position. - /// - private const int ScaleSubpixelMask = (1 << ScaleSubpixelBits) - 1; - - /// - /// The number of low position bits omitted when selecting a filter phase. - /// - private const int ScaleExtraBits = ScaleSubpixelBits - SubpixelBits; - - /// - /// The half-step bias applied before filter-phase selection. - /// - private const int ScaleExtraOffset = 1 << (ScaleExtraBits - 1); - - /// - /// The flattened 64-phase, 8-tap normative AV1 super-resolution filter table. - /// - private static readonly short[] Filters = - [ - 0, 0, 0, 128, 0, 0, 0, 0, 0, 0, -1, 128, 2, -1, 0, 0, - 0, 1, -3, 127, 4, -2, 1, 0, 0, 1, -4, 127, 6, -3, 1, 0, - 0, 2, -6, 126, 8, -3, 1, 0, 0, 2, -7, 125, 11, -4, 1, 0, - -1, 2, -8, 125, 13, -5, 2, 0, -1, 3, -9, 124, 15, -6, 2, 0, - -1, 3, -10, 123, 18, -6, 2, -1, -1, 3, -11, 122, 20, -7, 3, -1, - -1, 4, -12, 121, 22, -8, 3, -1, -1, 4, -13, 120, 25, -9, 3, -1, - -1, 4, -14, 118, 28, -9, 3, -1, -1, 4, -15, 117, 30, -10, 4, -1, - -1, 5, -16, 116, 32, -11, 4, -1, -1, 5, -16, 114, 35, -12, 4, -1, - -1, 5, -17, 112, 38, -12, 4, -1, -1, 5, -18, 111, 40, -13, 5, -1, - -1, 5, -18, 109, 43, -14, 5, -1, -1, 6, -19, 107, 45, -14, 5, -1, - -1, 6, -19, 105, 48, -15, 5, -1, -1, 6, -19, 103, 51, -16, 5, -1, - -1, 6, -20, 101, 53, -16, 6, -1, -1, 6, -20, 99, 56, -17, 6, -1, - -1, 6, -20, 97, 58, -17, 6, -1, -1, 6, -20, 95, 61, -18, 6, -1, - -2, 7, -20, 93, 64, -18, 6, -2, -2, 7, -20, 91, 66, -19, 6, -1, - -2, 7, -20, 88, 69, -19, 6, -1, -2, 7, -20, 86, 71, -19, 6, -1, - -2, 7, -20, 84, 74, -20, 7, -2, -2, 7, -20, 81, 76, -20, 7, -1, - -2, 7, -20, 79, 79, -20, 7, -2, -1, 7, -20, 76, 81, -20, 7, -2, - -2, 7, -20, 74, 84, -20, 7, -2, -1, 6, -19, 71, 86, -20, 7, -2, - -1, 6, -19, 69, 88, -20, 7, -2, -1, 6, -19, 66, 91, -20, 7, -2, - -2, 6, -18, 64, 93, -20, 7, -2, -1, 6, -18, 61, 95, -20, 6, -1, - -1, 6, -17, 58, 97, -20, 6, -1, -1, 6, -17, 56, 99, -20, 6, -1, - -1, 6, -16, 53, 101, -20, 6, -1, -1, 5, -16, 51, 103, -19, 6, -1, - -1, 5, -15, 48, 105, -19, 6, -1, -1, 5, -14, 45, 107, -19, 6, -1, - -1, 5, -14, 43, 109, -18, 5, -1, -1, 5, -13, 40, 111, -18, 5, -1, - -1, 4, -12, 38, 112, -17, 5, -1, -1, 4, -12, 35, 114, -16, 5, -1, - -1, 4, -11, 32, 116, -16, 5, -1, -1, 4, -10, 30, 117, -15, 4, -1, - -1, 3, -9, 28, 118, -14, 4, -1, -1, 3, -9, 25, 120, -13, 4, -1, - -1, 3, -8, 22, 121, -12, 4, -1, -1, 3, -7, 20, 122, -11, 3, -1, - -1, 2, -6, 18, 123, -10, 3, -1, 0, 2, -6, 15, 124, -9, 3, -1, - 0, 2, -5, 13, 125, -8, 2, -1, 0, 1, -4, 11, 125, -7, 2, 0, - 0, 1, -3, 8, 126, -6, 2, 0, 0, 1, -3, 6, 127, -4, 1, 0, - 0, 1, -2, 4, 127, -3, 1, 0, 0, 0, -1, 2, 128, -1, 0, 0 - ]; - - /// - /// Derives the fixed-point source-position increment for an output row. - /// - /// The coded plane width. - /// The upscaled plane width. - /// The source-position increment with 14 fractional bits. - public static int GetConvolveStep(int inputLength, int outputLength) - => ((inputLength << ScaleSubpixelBits) + (outputLength / 2)) / outputLength; - - /// - /// Derives the initial fixed-point source position for an output row. - /// - /// The coded plane width. - /// The upscaled plane width. - /// The source-position increment returned by . - /// The initial source position with 14 fractional bits. - public static int GetInitialSubpixel(int inputLength, int outputLength, int step) - { - int error = (outputLength * step) - (inputLength << ScaleSubpixelBits); - int initial = (-((outputLength - inputLength) << (ScaleSubpixelBits - 1)) + (outputLength / 2)) / - outputLength; - - initial += ScaleExtraOffset - (error / 2); - return initial & ScaleSubpixelMask; - } - - /// - /// Upscales one replicated-edge source row with the normative AV1 filter. - /// - /// The coded row with replicated samples on each edge. - /// The upscaled destination row. - /// The fixed-point source-position increment. - /// The initial fixed-point source position. - /// The encoded sample bit depth. - public static void UpscaleRow( - ReadOnlySpan source, - Span destination, - int step, - int initialSubpixel, - int bitDepth) - { - int maximum = (1 << bitDepth) - 1; - int sourcePosition = initialSubpixel; - for (int column = 0; column < destination.Length; column++) - { - int integerPosition = sourcePosition >> ScaleSubpixelBits; - int filterPhase = (sourcePosition & ScaleSubpixelMask) >> ScaleExtraBits; - int sourceOffset = SourceBorder + integerPosition - (FilterTapCount / 2); - int filterOffset = filterPhase * FilterTapCount; - int sum = DotProduct(source, sourceOffset, filterOffset); - - destination[column] = (ushort)Av1Math.Clip3(0, maximum, (sum + (1 << (FilterBits - 1))) >> FilterBits); - sourcePosition += step; - } - } - - /// - /// Computes the exact signed eight-tap filter product for one output sample. - /// - /// The replicated-edge source row. - /// The first source sample consumed by the filter. - /// The first coefficient in the selected filter phase. - /// The unrounded filter sum. - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static int DotProduct(ReadOnlySpan source, int sourceOffset, int filterOffset) - { - if (Vector128.IsHardwareAccelerated) - { - ref ushort sourceReference = ref MemoryMarshal.GetReference(source); - ref short filterReference = ref MemoryMarshal.GetArrayDataReference(Filters); - Vector128 samples = Vector128.LoadUnsafe(ref sourceReference, (nuint)sourceOffset).AsInt16(); - Vector128 coefficients = Vector128.LoadUnsafe(ref filterReference, (nuint)filterOffset); - Vector128 pairSums = Vector128_.MultiplyAddAdjacent(samples, coefficients); - - // Reuse ImageSharp's architecture-specific adjacent multiply/add and finish its four - // 32-bit lanes scalarly; this remains cheaper than constructing a per-pixel shuffle tree. - return pairSums.GetElement(0) + pairSums.GetElement(1) + pairSums.GetElement(2) + pairSums.GetElement(3); - } - - int sum = 0; - for (int tap = 0; tap < FilterTapCount; tap++) - { - sum += source[sourceOffset + tap] * Filters[filterOffset + tap]; - } - - return sum; - } -} diff --git a/tests/ImageSharp.Benchmarks/Codecs/Heif/Av1SuperResolutionBenchmarks.cs b/tests/ImageSharp.Benchmarks/Codecs/Heif/Av1SuperResolutionBenchmarks.cs new file mode 100644 index 0000000000..84c6181b0e --- /dev/null +++ b/tests/ImageSharp.Benchmarks/Codecs/Heif/Av1SuperResolutionBenchmarks.cs @@ -0,0 +1,145 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Columns; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; +using SixLabors.ImageSharp.Formats.Heif.Av1.Pipeline.SuperResolution; + +namespace SixLabors.ImageSharp.Benchmarks.Codecs.Heif; + +/// +/// Measures normative AV1 super-resolution filtering across a full-HD luma plane. +/// +[Config(typeof(Configuration))] +[MemoryDiagnoser(displayGenColumns: false)] +[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)] +[CategoriesColumn] +public class Av1SuperResolutionBenchmarks +{ + /// + /// The displayed luma width after normative upscaling. + /// + private const int UpscaledWidth = 1920; + + /// + /// The coded luma width produced by a fixed super-resolution denominator of twelve. + /// + private const int CodedWidth = 1280; + + /// + /// The number of visible rows in the measured luma plane. + /// + private const int Height = 1080; + + /// + /// The fixed-point source-position increment for the measured scale ratio. + /// + private readonly int step = Av1SuperResolutionFilter.GetConvolveStep(CodedWidth, UpscaledWidth); + + /// + /// The initial fixed-point source position for the measured scale ratio. + /// + private readonly int initialSubpixel; + + /// + /// The replicated-edge eight-bit source row. + /// + private readonly byte[] source8 = new byte[CodedWidth + (Av1SuperResolutionFilter.SourceBorder * 2)]; + + /// + /// The filtered eight-bit output row. + /// + private readonly byte[] destination8 = new byte[UpscaledWidth]; + + /// + /// The replicated-edge twelve-bit source row. + /// + private readonly ushort[] source12 = new ushort[CodedWidth + (Av1SuperResolutionFilter.SourceBorder * 2)]; + + /// + /// The filtered twelve-bit output row. + /// + private readonly ushort[] destination12 = new ushort[UpscaledWidth]; + + /// + /// Initializes a new instance of the class. + /// + public Av1SuperResolutionBenchmarks() + { + this.initialSubpixel = Av1SuperResolutionFilter.GetInitialSubpixel(CodedWidth, UpscaledWidth, this.step); + } + + /// + /// Populates deterministic source samples and their replicated frame edges outside the measured traversal. + /// + [GlobalSetup] + public void Setup() + { + Span reconstructed8 = this.source8.AsSpan(Av1SuperResolutionFilter.SourceBorder, CodedWidth); + Span reconstructed12 = this.source12.AsSpan(Av1SuperResolutionFilter.SourceBorder, CodedWidth); + for (int column = 0; column < CodedWidth; column++) + { + int sample = ((column * 4051) + 127) & byte.MaxValue; + reconstructed8[column] = (byte)sample; + reconstructed12[column] = (ushort)(sample << 4); + } + + this.source8.AsSpan(0, Av1SuperResolutionFilter.SourceBorder).Fill(reconstructed8[0]); + this.source8.AsSpan(Av1SuperResolutionFilter.SourceBorder + CodedWidth).Fill(reconstructed8[^1]); + this.source12.AsSpan(0, Av1SuperResolutionFilter.SourceBorder).Fill(reconstructed12[0]); + this.source12.AsSpan(Av1SuperResolutionFilter.SourceBorder + CodedWidth).Fill(reconstructed12[^1]); + } + + /// + /// Measures normative eight-bit filtering for a full-HD luma plane. + /// + /// The final filtered sample, keeping the output observable. + [Benchmark] + [BenchmarkCategory("8Bit")] + public byte Filter8BitPlane() + { + for (int row = 0; row < Height; row++) + { + Av1SuperResolutionFilter.UpscaleRow(this.source8, this.destination8, this.step, this.initialSubpixel); + } + + return this.destination8[^1]; + } + + /// + /// Measures normative twelve-bit filtering for a full-HD luma plane. + /// + /// The final filtered sample, keeping the output observable. + [Benchmark] + [BenchmarkCategory("12Bit")] + public ushort Filter12BitPlane() + { + for (int row = 0; row < Height; row++) + { + Av1SuperResolutionFilter.UpscaleRow(this.source12, this.destination12, this.step, this.initialSubpixel, 12); + } + + return this.destination12[^1]; + } + + /// + /// Configures production-process measurements for hardware and scalar filtering. + /// + public sealed class Configuration : ManualConfig + { + /// + /// Initializes a new instance of the class. + /// + public Configuration() + { + this.AddJob(Job.ShortRun.WithId("Hardware").AsBaseline()); + + this.AddJob( + Job.ShortRun + .WithId("Scalar") + .WithEnvironmentVariable("DOTNET_EnableHWIntrinsic", "0")); + } + } +} diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReconstructionConformanceTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReconstructionConformanceTests.cs index a97c7d84c5..2589e31a66 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReconstructionConformanceTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReconstructionConformanceTests.cs @@ -86,6 +86,14 @@ public class Av1ReconstructionConformanceTests public void DecodeWithActiveCdefMatchesPinnedLibavifPresentation() => FeatureTestRunner.RunWithHwIntrinsicsFeature(ValidatePresentedFixtures, PresentationConfigurations); + /// + /// Verifies active normative super-resolution, chroma-width rounding, replicated edges, and exact native samples + /// against scalar libaom for independently encoded eight-, ten-, and twelve-bit still-picture streams. + /// + [Fact] + public void DecodeWithSuperResolutionMatchesPinnedLibaomReference() + => FeatureTestRunner.RunWithHwIntrinsicsFeature(ValidateSuperResolutionFixtures, ReconstructionConfigurations); + /// /// Validates every active-CDEF fixture under the hardware configuration selected by . /// @@ -144,6 +152,37 @@ public class Av1ReconstructionConformanceTests HeifBitDepth.Bit12); } + /// + /// Validates every active super-resolution fixture under the hardware configuration selected by + /// . + /// + private static void ValidateSuperResolutionFixtures() + { + ValidateSuperResolutionFixture( + TestImages.Heif.Av1SuperResolution8BitPayload, + TestImages.Heif.Av1SuperResolution8BitReference, + 768, + 512, + Av1BitDepth.EightBit, + Av1ColorFormat.Yuv420); + + ValidateSuperResolutionFixture( + TestImages.Heif.Av1SuperResolution10BitPayload, + TestImages.Heif.Av1SuperResolution10BitReference, + 1024, + 428, + Av1BitDepth.TenBit, + Av1ColorFormat.Yuv444); + + ValidateSuperResolutionFixture( + TestImages.Heif.Av1SuperResolution12BitPayload, + TestImages.Heif.Av1SuperResolution12BitReference, + 1024, + 428, + Av1BitDepth.TwelveBit, + Av1ColorFormat.Yuv444); + } + /// /// Validates one elementary-stream sample and its containing AVIF image. /// @@ -179,6 +218,7 @@ public class Av1ReconstructionConformanceTests /// The expected AV1 sample precision. /// The expected native chroma-sampling layout. /// Indicates whether the stream must signal and select nonzero CDEF strengths. + /// Indicates whether the stream must use normative horizontal upscaling. private static void ValidateNativeFixture( string payloadPath, string referencePath, @@ -186,7 +226,8 @@ public class Av1ReconstructionConformanceTests int height, Av1BitDepth bitDepth, Av1ColorFormat colorFormat, - bool requireActiveCdef) + bool requireActiveCdef, + bool requireSuperResolution = false) { byte[] payload = TestFile.Create(payloadPath).Bytes; byte[] reference = TestFile.Create(referencePath).Bytes; @@ -198,6 +239,15 @@ public class Av1ReconstructionConformanceTests Assert.Equal(bitDepth, frameBuffer.BitDepth); Assert.Equal(colorFormat, frameBuffer.ColorFormat); Assert.NotNull(decoder.FrameHeader); + + if (requireSuperResolution) + { + ObuFrameSize frameSize = decoder.FrameHeader.FrameSize; + Assert.True(frameSize.FrameWidth < frameSize.SuperResolutionUpscaledWidth); + Assert.Equal(width, frameSize.SuperResolutionUpscaledWidth); + Assert.False(decoder.FrameHeader.LoopRestorationParameters.UsesLoopRestoration); + } + ObuLoopFilterParameters filterParameters = decoder.FrameHeader.LoopFilterParameters; Assert.True( filterParameters.FilterLevel[0] != 0 @@ -261,6 +311,32 @@ public class Av1ReconstructionConformanceTests Av1ColorFormat colorFormat) => ValidateNativeFixture(payloadPath, referencePath, width, height, bitDepth, colorFormat, requireActiveCdef: true); + /// + /// Validates one independently encoded stream that activates normative super-resolution. + /// + /// The AV1 elementary-stream sample. + /// The native planar output produced by the pinned scalar libaom decoder. + /// The expected upscaled width. + /// The expected reconstructed height. + /// The expected AV1 sample precision. + /// The expected native chroma-sampling layout. + private static void ValidateSuperResolutionFixture( + string payloadPath, + string referencePath, + int width, + int height, + Av1BitDepth bitDepth, + Av1ColorFormat colorFormat) + => ValidateNativeFixture( + payloadPath, + referencePath, + width, + height, + bitDepth, + colorFormat, + requireActiveCdef: false, + requireSuperResolution: true); + /// /// Validates the public presentation and metadata produced from one complete AVIF container. /// diff --git a/tests/ImageSharp.Tests/TestImages.cs b/tests/ImageSharp.Tests/TestImages.cs index 4287fd07da..3679e7bc02 100644 --- a/tests/ImageSharp.Tests/TestImages.cs +++ b/tests/ImageSharp.Tests/TestImages.cs @@ -1365,6 +1365,12 @@ public static class TestImages public const string Av1Cdef12BitReference = "Heif/Av1/Conformance/libaom-cdef-cosmos-12b-libaom.yuv"; public const string Av1Cdef12BitAvif = "Heif/Av1/Conformance/libavif-cdef-cosmos-12b.avif"; public const string Av1Cdef12BitPresentationReference = "Heif/Av1/Conformance/libavif-cdef-cosmos-12b.png"; + public const string Av1SuperResolution8BitPayload = "Heif/Av1/Conformance/libaom-superres-kodim23-8b.bit"; + public const string Av1SuperResolution8BitReference = "Heif/Av1/Conformance/libaom-superres-kodim23-8b-libaom.yuv"; + public const string Av1SuperResolution10BitPayload = "Heif/Av1/Conformance/libaom-superres-cosmos-10b.bit"; + public const string Av1SuperResolution10BitReference = "Heif/Av1/Conformance/libaom-superres-cosmos-10b-libaom.yuv"; + public const string Av1SuperResolution12BitPayload = "Heif/Av1/Conformance/libaom-superres-cosmos-12b.bit"; + public const string Av1SuperResolution12BitReference = "Heif/Av1/Conformance/libaom-superres-cosmos-12b-libaom.yuv"; } public static class Ico diff --git a/tests/Images/Input/Heif/Av1/Conformance/README.md b/tests/Images/Input/Heif/Av1/Conformance/README.md index fbe35c0c42..c3351146ed 100644 --- a/tests/Images/Input/Heif/Av1/Conformance/README.md +++ b/tests/Images/Input/Heif/Av1/Conformance/README.md @@ -26,3 +26,7 @@ The native reference layouts are: - `libaom-cdef-cosmos-12b-libaom.yuv`: 1024x428, 12-bit YUV 4:4:4, planar Y/U/V with little-endian 16-bit samples. The conformance tests compare every visible reconstructed sample with these files. The deblocking corpus verifies nonzero loop-filter levels. The CDEF corpus additionally verifies sequence-level CDEF enablement, a selected nonzero frame strength, and disabled loop restoration, so a disabled or bypassed CDEF stage cannot satisfy the exact native-plane comparison accidentally. + +The `libaom-superres-*` streams were encoded from the same Kodak and Cosmos sources with the pinned generic libaom build. Their material options were `--usage=2 --passes=1 --limit=1 --obu --end-usage=q --cq-level=30 --cpu-used=4 --threads=1 --lag-in-frames=0 --full-still-picture-hdr --enable-cdef=0 --enable-restoration=0 --superres-mode=1 --superres-denominator=12 --superres-kf-denominator=12`, together with the matching input depth, output depth, and profile. Disabling CDEF and restoration isolates the normative horizontal upscaling result, while the tests separately require a coded width smaller than the displayed width so an unscaled stream cannot satisfy the reference comparison. + +The matching `libaom-superres-*-libaom.yuv` files were decoded by `aomdec --rawvideo` from that exact generic build. They retain the displayed 768x512 8-bit YUV 4:2:0 and 1024x428 10/12-bit YUV 4:4:4 layouts described above. diff --git a/tests/Images/Input/Heif/Av1/Conformance/libaom-superres-cosmos-10b-libaom.yuv b/tests/Images/Input/Heif/Av1/Conformance/libaom-superres-cosmos-10b-libaom.yuv new file mode 100644 index 0000000000..eb3ad9033b --- /dev/null +++ b/tests/Images/Input/Heif/Av1/Conformance/libaom-superres-cosmos-10b-libaom.yuv @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5ab5a53eccebfa727a249c04b8ff81486dc8473b1aaea39e07d91e6e53d97a10 +size 2629632 diff --git a/tests/Images/Input/Heif/Av1/Conformance/libaom-superres-cosmos-10b.bit b/tests/Images/Input/Heif/Av1/Conformance/libaom-superres-cosmos-10b.bit new file mode 100644 index 0000000000..22e946a332 --- /dev/null +++ b/tests/Images/Input/Heif/Av1/Conformance/libaom-superres-cosmos-10b.bit @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8ff2272ed9d8d40b925d9711f0a5af93638b979a8e7c4930336b583d9e9db837 +size 18427 diff --git a/tests/Images/Input/Heif/Av1/Conformance/libaom-superres-cosmos-12b-libaom.yuv b/tests/Images/Input/Heif/Av1/Conformance/libaom-superres-cosmos-12b-libaom.yuv new file mode 100644 index 0000000000..d2398caf0f --- /dev/null +++ b/tests/Images/Input/Heif/Av1/Conformance/libaom-superres-cosmos-12b-libaom.yuv @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6a50d88223eb46bb6079bb28032e682676fe4376a0cc11137ec1f6a56cdb85c7 +size 2629632 diff --git a/tests/Images/Input/Heif/Av1/Conformance/libaom-superres-cosmos-12b.bit b/tests/Images/Input/Heif/Av1/Conformance/libaom-superres-cosmos-12b.bit new file mode 100644 index 0000000000..e4b3fc6781 --- /dev/null +++ b/tests/Images/Input/Heif/Av1/Conformance/libaom-superres-cosmos-12b.bit @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5d5bb84dc57f28aae953196464729ba99ba83d6510f15ebaabe23f523722a82d +size 16085 diff --git a/tests/Images/Input/Heif/Av1/Conformance/libaom-superres-kodim23-8b-libaom.yuv b/tests/Images/Input/Heif/Av1/Conformance/libaom-superres-kodim23-8b-libaom.yuv new file mode 100644 index 0000000000..49515f6de6 --- /dev/null +++ b/tests/Images/Input/Heif/Av1/Conformance/libaom-superres-kodim23-8b-libaom.yuv @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6a6ae4b088d7e193dad3057e53da9664dd30bee6bbf2938a2caefee2da1c6b5b +size 589824 diff --git a/tests/Images/Input/Heif/Av1/Conformance/libaom-superres-kodim23-8b.bit b/tests/Images/Input/Heif/Av1/Conformance/libaom-superres-kodim23-8b.bit new file mode 100644 index 0000000000..0dd8351ca8 --- /dev/null +++ b/tests/Images/Input/Heif/Av1/Conformance/libaom-superres-kodim23-8b.bit @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:532bcf722b067a9583dc849c9a48a2404124e9357877c72284c61e9b54794272 +size 12302