diff --git a/HEIF_IMPLEMENTATION_PLAN.md b/HEIF_IMPLEMENTATION_PLAN.md index dc221f0cb..bef99942b 100644 --- a/HEIF_IMPLEMENTATION_PLAN.md +++ b/HEIF_IMPLEMENTATION_PLAN.md @@ -29,7 +29,7 @@ Checkboxes may be marked complete only when the implementation and the verificat ## Delivery dashboard -Last reconciled with the source tree on 2026-08-29 against production checkpoint `beb2ab86ce5e4052c667df30af8a48a860814031`. Committed checkpoints include the AV1 transform architecture, OBU framing, intra-block copy, 12-profile reconstruction matrix, layered-item properties, layered reference/header/CDF/motion-field state, inter-frame intra blocks, SIMD-first translational prediction, complete single-reference inter reconstruction, compound reference trees and modes, paired reference-MV derivation, reference-dependent bounded sequence decoding, allocation-free SIMD-first equal averaging, selected inter-intra prediction, selectable compound blending, OBMC, scaled-reference reconstruction, local warped prediction, non-translational global prediction, official motion-vector conformance, selected spatial-layer presentation, and progressive color and auxiliary-alpha conformance. This dashboard is the authoritative delivery order. The detailed phase checklists below provide subsystem evidence; they do not override the current-stage marker or permit work to skip ahead. +Last reconciled with the source tree on 2026-08-29 against production checkpoint `ef93d584511055f8e91e8d801662c44a5b0984d9`. Committed checkpoints include the AV1 transform architecture, OBU framing, intra-block copy, 12-profile reconstruction matrix, corrected twelve-bit inverse-transform SIMD arithmetic, layered-item properties, layered reference/header/CDF/motion-field state, inter-frame intra blocks, SIMD-first translational prediction, complete single-reference inter reconstruction, compound reference trees and modes, paired reference-MV derivation, reference-dependent bounded sequence decoding, allocation-free SIMD-first equal averaging, selected inter-intra prediction, selectable compound blending, OBMC, scaled-reference reconstruction, local warped prediction, non-translational global prediction, official motion-vector conformance, selected spatial-layer presentation, and progressive color and auxiliary-alpha conformance. This dashboard is the authoritative delivery order. The detailed phase checklists below provide subsystem evidence; they do not override the current-stage marker or permit work to skip ahead. Commit `1c58d855f70b024170ced9eb0a7005f0f9c955ad` records the complete official motion-vector conformance checkpoint. The official IVF has SHA-1 `F064290D7FCD3B3DE19020E8AEC6C43C88D3A505`, matching the pinned libaom test-data manifest, and SHA-256 `222A9050059B254DAB17CFB802FF829C778E3F93AF18961A622C8268576C1395`; its pinned-libaom Y4M has SHA-256 `D97AC78C81782CF1507549368047769DC677DBE205706458D1EE9C807DE6EC78`. Both source targets build with zero warnings and errors, the `net10.0` test-project analyzer build completes with zero errors and 1,014 pre-existing repository warnings, Roslynk reports zero compiler errors, 3,985 focused `net10.0` cases pass without failures or skips, and `git diff --check` is clean. @@ -37,7 +37,9 @@ Commit `019ac5648b4380a6ff4865072c96fde29ce09fad` records the complete selected- Commit `beb2ab86ce5e4052c667df30af8a48a860814031` records the complete progressive color and auxiliary-alpha conformance checkpoint. Both pinned progressive AVIF fixtures pass their exact native-plane and pinned-libavif presentation comparisons through normal dispatch, AVX-512-disabled, AVX-disabled, and scalar `FeatureTestRunner` configurations. The YUV444-alpha fixture also compares every composed alpha sample with the independent final native auxiliary plane. Both complete production presentation paths pass under a 1,024-byte constrained tracked allocator with every allocation returned exactly once. Both source targets build with zero warnings and errors, the `net10.0` test-project analyzer build completes with zero errors and 1,014 pre-existing repository warnings, Roslynk reports zero compiler errors, 156 focused reconstruction and color cases pass without failures or skips, and `git diff --check` is clean. -The working tree contains a complete, verified twelve-bit inverse-transform arithmetic checkpoint awaiting commit. ADST4 retains pinned libaom's signed 32-bit sine products and factorized sums and widens only its terminal rounding; Identity4 and Identity16 widen their fixed-point product and rounding bias only for the 20-bit twelve-bit row stage. The established 8/10-bit and twelve-bit column paths remain unchanged. Exact boundary vectors cover both 128-bit and 256-bit operators through normal, AVX-512-disabled, AVX-disabled, and scalar `FeatureTestRunner` configurations. Both source targets build with zero warnings and errors, the `net10.0` test-project analyzer build completes with zero errors and 1,014 pre-existing repository warnings, Roslynk reports zero compiler errors, 510 focused inverse-transform cases and 44 production reconstruction cases pass without failures or skips, and `git diff --check` is clean. +Commit `ef93d584511055f8e91e8d801662c44a5b0984d9` records the complete twelve-bit inverse-transform arithmetic checkpoint. ADST4 retains pinned libaom's signed 32-bit sine products and factorized sums and widens only its terminal rounding; Identity4 and Identity16 widen their fixed-point product and rounding bias only for the 20-bit twelve-bit row stage. The established 8/10-bit and twelve-bit column paths remain unchanged. Exact boundary vectors cover both 128-bit and 256-bit operators through normal, AVX-512-disabled, AVX-disabled, and scalar `FeatureTestRunner` configurations. Both source targets build with zero warnings and errors, the `net10.0` test-project analyzer build completes with zero errors and 1,014 pre-existing repository warnings, Roslynk reports zero compiler errors, 510 focused inverse-transform cases and 44 production reconstruction cases pass without failures or skips, and `git diff --check` is clean. + +The working tree contains a complete, verified official syntax-coverage and predictor-architecture checkpoint awaiting commit. The official all-intra, CDF-update, and temporal motion-field IVF files have SHA-1 `A9F7EA6312A533CC6426A6145EDD190D45813C37`, `AFCA5502A489692B0A3C120370B0F43B8FC572A1`, and `B48A717C7C003B8DD23C3C2CAED1AC673380FDB3`, exactly matching the pinned libaom manifest. Their SHA-256 values are `5FCD265FD9F9BDD0D3179340B4C4532F1422CA5E5D97741C7481B84CB5DC122F`, `14A3DBF537B6BF15EFC003182D9916D61438C93624A8BD26E6E3AE7EAF33EA82`, and `B59BF9586D8546DFDA81DFEC4EE4E32CEB502C9D22412AB0B63A2ABB534A1F14`; the pinned-libaom Y4M references have SHA-256 `1211EBEFBC9CCEF9ED19BE4CCE3F807D69FFFE338E95CCA1B5F4CA8023482175`, `4FBFF73FF0DE2D9084DAE557D1D4BD677B0486516525BF4D327D2D795D5A7779`, and `F7DB607694818C19E62FD9A27F53E1A3E2D00B72C39C0430C1B26399CC76777D`. Exact native-plane comparison covers all 39 all-intra frames, every intra mode, seven selected transform types, both tile-local and frame-end adaptive CDF publication, all four temporal motion-field frames under normal/scalar dispatch, and constrained tracked motion-field allocation with exactly-once returns. Every distinct AV1 predictor traversal now owns a family-named JPEG-style static operator contract and SIMD traversal instead of nesting separate predictors beneath broad intra/inter families. Compound reference convolution, equal averaging, distance weighting, alpha-mask blending, difference-weighted mask construction, and each intermediate reconstruction or mask traversal have separate family owners and matching `.Operator.cs` contracts; inter-intra mask construction has its own non-operator builder. Both Release source targets build with zero warnings and errors, Roslynk reports zero compiler errors, 179 focused prediction, reconstruction, reference, ownership, and lifecycle cases pass without failures or skips, and `git diff --check` is clean. Status meanings: @@ -49,7 +51,7 @@ Status meanings: Current development stage: **Stage 3 — complete AV1 still-image decoding.** The decoder retains reference/header/CDF/motion-field state, derives frame-level skip-mode references, consumes temporal segment prediction, decodes intra-coded blocks inside inter frames, and reconstructs translational single-reference, compound, inter-intra, OBMC, scaled-reference, local warped, and non-translational global prediction before residual traversal. Commit `1c58d855f70b024170ced9eb0a7005f0f9c955ad` adds exact official four-frame coverage through all ordinary inter modes, all three motion modes, every regular/smooth/sharp dual-filter pair, sub-8x8 chroma prediction, and no-round compound intermediates. Commit `019ac5648b4380a6ff4865072c96fde29ce09fad` adds exact selected-layer native reconstruction and scaled presentation. Neither AV1 nor HEVC production encoding is implemented. -Immediate checkpoint: **commit the verified twelve-bit inverse-transform arithmetic correction before advancing.** The corrected operators match pinned libaom at the exact overflow boundaries, preserve narrower stage paths, and pass the complete focused transform and production reconstruction evidence. +Immediate checkpoint: **inventory and remove the next remaining production-reachable unsupported valid AV1 still-image syntax path.** Trace every explicit rejection through the decoder, distinguish malformed or out-of-scope syntax from valid still-image behavior, select the first valid gap in source order, and close it with exact independent native-plane and presentation evidence before advancing. | Order | Delivery stage | State | Delivered state | Gate that remains open | | --- | --- | --- | --- | --- | @@ -107,7 +109,8 @@ Immediate checkpoint: **commit the verified twelve-bit inverse-transform arithme - [x] Return the explicitly selected spatial layer or the final displayed layer, keeping reference reconstruction separate from display-only film grain. The essential-`lsel` production fixture reconstructs the selected 40x40 YUV444 base exactly, scales native component planes to the 80x80 item extent with pinned-libyuv integer rounding, and matches pinned-libavif RGBA presentation under normal/scalar dispatch. The committed final-layer fixture and film-grain matrix remain exact. Constrained tracked allocation returns every short-lived presentation plane exactly once. Commit `019ac5648b4380a6ff4865072c96fde29ce09fad` records the checkpoint. - [x] Verify color and auxiliary-alpha output exactly against both pinned libavif progressive fixtures under normal SIMD dispatch and all required `FeatureTestRunner` fallbacks. Both fixtures pass exact public presentation under normal, AVX-512-disabled, AVX-disabled, and scalar dispatch; the alpha-bearing fixture additionally matches every composed alpha sample with its pinned native auxiliary plane. Both full production paths pass under a 1,024-byte constrained tracked allocator with exactly-once returns. Both Release source targets build with zero warnings and errors, the test-project analyzer build completes with zero errors and 1,014 pre-existing warnings, Roslynk reports zero compiler errors, and all 156 focused reconstruction and color cases pass without failures or skips. - [x] Correct the audited 12-bit inverse ADST4, Identity4, and Identity16 SIMD arithmetic by widening only the libaom-widened multiply/accumulate operations, with exact conformant-range vectors and `FeatureTestRunner` coverage. Both 128-bit and 256-bit operators match pinned outputs through every required hardware fallback; 510 focused inverse-transform cases, 44 production reconstruction cases, both zero-warning source builds, the zero-error analyzer build, zero Roslynk compiler errors, and clean `git diff --check` complete the verification. -- [ ] **Queued until the twelve-bit inverse-transform checkpoint commit:** continue inventorying and removing every remaining valid AV1 still-image unsupported branch, adding exact independent compression-tool fixtures to the profile-matrix regression gate. +- [x] Verify the official libaom all-intra, CDF-update, and temporal motion-field sequences through exact native output, selected syntax coverage, normal/scalar dispatch, constrained tracked allocation, and the family-owned JPEG-style predictor operator architecture. +- [ ] **Current:** continue inventorying and removing every remaining valid AV1 still-image unsupported branch, adding exact independent compression-tool fixtures to the profile-matrix regression gate. - [ ] Complete the remaining HEVC still-image profile and Range Extensions matrix with exact independent native-plane and presentation evidence. - [ ] Close shared decoded presentation, ICC, alpha, grid, transform, metadata, and animated AV1/HEVC decode gates. - [ ] Implement and independently verify real AV1/AVIF still encoding. @@ -819,6 +822,7 @@ No valid HEVC or AV1 color, compression, or bit-depth row may remain `unsupporte ## Working rules for implementation - Keep changes vertical and reviewable. A slice should add one behavior, its focused tests, independent evidence, and any required notice update. +- Keep every AV1 prediction family on the established JPEG color-converter operator architecture. Each distinct traversal contract owns a family-named predictor type; its `.Operator.cs` defines the static interface, and its family-named files own the closed generic widest-to-narrowest SIMD traversal. Semantic `readonly struct` operators implement scalar, `Vector128`, `Vector256`, and `Vector512` arithmetic through that contract. Only modes which share the same traversal and contract may share a predictor family; do not nest a separate predictor beneath a broad intra/inter family or create hardware-width-specific class hierarchies. - Design SIMD-suitable codec work SIMD-first. Establish vector-friendly storage, operator boundaries, scratch ownership, traversal, every applicable lane width, and benchmark-gated dispatch before implementing the equivalent scalar fallback; never build a scalar production architecture and bolt SIMD onto it later. - Inspect every owning method and upstream invariant before adding guards. Validate external file data at the parser/model boundary and rely on those established invariants internally. - Do not extract one-use helpers merely to label code. Extract shared primitives only when they have genuine reuse or remove substantial complexity. @@ -841,7 +845,7 @@ The dashboard and immediate execution queue define the remaining critical path. - [x] Implement and independently verify non-translational global prediction with a genuine traced bounded sequence, exact native and presentation comparisons, constrained allocation, normal/scalar dispatch, and direct 8/10/12-bit compound production coverage. Commit `c5637ea0187df35b385bf43e2fe85cd955f01099` records the checkpoint. - [x] Complete progressive color and auxiliary-alpha verification through exact native and presentation comparisons, every required `FeatureTestRunner` fallback, and constrained tracked allocation. - [x] Correct the audited 12-bit inverse ADST4, Identity4, and Identity16 arithmetic through exact pinned boundary vectors and SIMD/scalar feature isolation. -- [ ] **Queued until the twelve-bit inverse-transform checkpoint commit:** remove every other unsupported valid AV1 still-image syntax path and prove the complete AVIF decode matrix with independent inputs and scalar/SIMD parity. +- [ ] **Current:** remove every other unsupported valid AV1 still-image syntax path and prove the complete AVIF decode matrix with independent inputs and scalar/SIMD parity. - [ ] Close Phase 4 by completing the remaining HEVC profile and Range Extensions matrix with exact native-plane and presented-image evidence. - [ ] Close Phase 5 and the decode portion of the bounded sequence ledger: color, ICC, alpha, grids, presentation transforms, reference-dependent samples, and complete animated AVIF/HEIC decode. - [ ] Close the still-image portions of Phases 0, 1, and 2 that remain as release gates: documentation, provenance, public format boundaries, API review, parser hardening, and malformed-input coverage. diff --git a/shared-infrastructure b/shared-infrastructure index 03471c6b4..a835a9d74 160000 --- a/shared-infrastructure +++ b/shared-infrastructure @@ -1 +1 @@ -Subproject commit 03471c6b458a2c11a0b1df1f5bf2777c8c773b99 +Subproject commit a835a9d74e82b2d32b580a7902eb2699ebc47098 diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1DcIntraPredictor.Operator.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1DcIntraPredictor.Operator.cs new file mode 100644 index 000000000..8dedd0fc4 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1DcIntraPredictor.Operator.cs @@ -0,0 +1,434 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using System.Runtime.Intrinsics; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; + +/// +/// Defines reference reduction and rounded mean arithmetic for AV1 DC intra prediction. +/// +internal static class Av1DcIntraPredictor +{ + /// + /// Defines scalar and SIMD reference reduction for AV1 DC intra prediction. + /// + internal interface IDcPredictionOperator + { + /// + /// Sums one 8-bit reference sample. + /// + /// The reference sample. + /// The sample value. + public static abstract int Sum(byte sample); + + /// + /// Sums sixteen 8-bit reference samples. + /// + /// The reference samples. + /// The exact sum. + public static abstract int Sum(Vector128 samples); + + /// + /// Sums thirty-two 8-bit reference samples. + /// + /// The reference samples. + /// The exact sum. + public static abstract int Sum(Vector256 samples); + + /// + /// Sums sixty-four 8-bit reference samples. + /// + /// The reference samples. + /// The exact sum. + public static abstract int Sum(Vector512 samples); + + /// + /// Sums one high-bit-depth reference sample. + /// + /// The reference sample. + /// The sample value. + public static abstract int Sum(short sample); + + /// + /// Sums eight high-bit-depth reference samples. + /// + /// The reference samples. + /// The exact sum. + public static abstract int Sum(Vector128 samples); + + /// + /// Sums sixteen high-bit-depth reference samples. + /// + /// The reference samples. + /// The exact sum. + public static abstract int Sum(Vector256 samples); + + /// + /// Sums thirty-two high-bit-depth reference samples. + /// + /// The reference samples. + /// The exact sum. + public static abstract int Sum(Vector512 samples); + + /// + /// Calculates the 8-bit DC prediction. + /// + /// The sum of available reference samples. + /// The number of available reference samples. + /// The rounded DC prediction. + public static abstract byte Predict(int sum, int count); + + /// + /// Calculates the high-bit-depth DC prediction. + /// + /// The sum of available reference samples. + /// The number of available reference samples. + /// The reconstructed sample precision. + /// The rounded DC prediction. + public static abstract short Predict(int sum, int count, int bitDepth); + } + + /// + /// Predicts an 8-bit DC block. + /// + public static void Predict(bool hasLeft, bool hasAbove, Span destination, int destinationStride, ReadOnlySpan above, ReadOnlySpan left, int width, int height) + => Predictor.Predict(hasLeft, hasAbove, destination, destinationStride, above, left, width, height); + + /// + /// Predicts a high-bit-depth DC block. + /// + public static void Predict(bool hasLeft, bool hasAbove, Span destination, int destinationStride, ReadOnlySpan above, ReadOnlySpan left, int width, int height, int bitDepth) + => Predictor.Predict(hasLeft, hasAbove, destination, destinationStride, above, left, width, height, bitDepth); + + /// + /// Predicts an 8-bit DC block without hardware intrinsics. + /// + public static void PredictScalar(bool hasLeft, bool hasAbove, Span destination, int destinationStride, ReadOnlySpan above, ReadOnlySpan left, int width, int height) + => Predictor.PredictScalar(hasLeft, hasAbove, destination, destinationStride, above, left, width, height); + + /// + /// Predicts a high-bit-depth DC block without hardware intrinsics. + /// + public static void PredictScalar(bool hasLeft, bool hasAbove, Span destination, int destinationStride, ReadOnlySpan above, ReadOnlySpan left, int width, int height, int bitDepth) + => Predictor.PredictScalar(hasLeft, hasAbove, destination, destinationStride, above, left, width, height, bitDepth); + + /// + /// Calculates the DC value from the available neighboring samples. + /// + internal readonly struct DcOperator : IDcPredictionOperator + { + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int Sum(byte sample) => sample; + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int Sum(Vector128 samples) + { + (Vector128 lower, Vector128 upper) = Vector128.Widen(samples); + + return Vector128.Sum(lower) + Vector128.Sum(upper); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int Sum(Vector256 samples) + { + (Vector256 lower, Vector256 upper) = Vector256.Widen(samples); + + return Vector256.Sum(lower) + Vector256.Sum(upper); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int Sum(Vector512 samples) + { + (Vector512 lower, Vector512 upper) = Vector512.Widen(samples); + + return Vector512.Sum(lower) + Vector512.Sum(upper); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int Sum(short sample) => sample; + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int Sum(Vector128 samples) + { + (Vector128 lower, Vector128 upper) = Vector128.Widen(samples); + + return Vector128.Sum(lower) + Vector128.Sum(upper); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int Sum(Vector256 samples) + { + (Vector256 lower, Vector256 upper) = Vector256.Widen(samples); + + return Vector256.Sum(lower) + Vector256.Sum(upper); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int Sum(Vector512 samples) + { + (Vector512 lower, Vector512 upper) = Vector512.Widen(samples); + + return Vector512.Sum(lower) + Vector512.Sum(upper); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static byte Predict(int sum, int count) => count == 0 ? (byte)128 : (byte)((sum + (count >> 1)) / count); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static short Predict(int sum, int count, int bitDepth) + => count == 0 ? (short)(1 << (bitDepth - 1)) : (short)((sum + (count >> 1)) / count); + } + + /// + /// Reconstructs DC blocks through one closed reduction operator. + /// + /// The reference reduction and rounded mean arithmetic. + private static class Predictor + where TOperator : struct, IDcPredictionOperator + { + /// + /// Predicts an 8-bit DC block. + /// + /// Whether the left reference is available. + /// Whether the top reference is available. + /// The destination block. + /// The distance between destination rows. + /// The top reference. + /// The left reference. + /// The block width. + /// The block height. + public static void Predict(bool hasLeft, bool hasAbove, Span destination, int destinationStride, ReadOnlySpan above, ReadOnlySpan left, int width, int height) + { + int count = (hasAbove ? width : 0) + (hasLeft ? height : 0); + int sum = (hasAbove ? Sum(above[..width]) : 0) + (hasLeft ? Sum(left[..height]) : 0); + byte prediction = TOperator.Predict(sum, count); + + for (int row = 0; row < height; row++) + { + destination.Slice(row * destinationStride, width).Fill(prediction); + } + } + + /// + /// Predicts a high-bit-depth DC block. + /// + /// Whether the left reference is available. + /// Whether the top reference is available. + /// The destination block. + /// The distance between destination rows. + /// The top reference. + /// The left reference. + /// The block width. + /// The block height. + /// The reconstructed sample precision. + public static void Predict(bool hasLeft, bool hasAbove, Span destination, int destinationStride, ReadOnlySpan above, ReadOnlySpan left, int width, int height, int bitDepth) + { + int count = (hasAbove ? width : 0) + (hasLeft ? height : 0); + int sum = (hasAbove ? Sum(above[..width]) : 0) + (hasLeft ? Sum(left[..height]) : 0); + short prediction = TOperator.Predict(sum, count, bitDepth); + + for (int row = 0; row < height; row++) + { + destination.Slice(row * destinationStride, width).Fill(prediction); + } + } + + /// + /// Predicts an 8-bit DC block without hardware intrinsics. + /// + /// Whether the left reference is available. + /// Whether the top reference is available. + /// The destination block. + /// The distance between destination rows. + /// The top reference. + /// The left reference. + /// The block width. + /// The block height. + public static void PredictScalar(bool hasLeft, bool hasAbove, Span destination, int destinationStride, ReadOnlySpan above, ReadOnlySpan left, int width, int height) + { + int count = (hasAbove ? width : 0) + (hasLeft ? height : 0); + int sum = (hasAbove ? SumScalar(above[..width]) : 0) + (hasLeft ? SumScalar(left[..height]) : 0); + byte prediction = TOperator.Predict(sum, count); + + for (int row = 0; row < height; row++) + { + ref byte destinationRow = ref destination[row * destinationStride]; + for (int column = 0; column < width; column++) + { + Unsafe.Add(ref destinationRow, column) = prediction; + } + } + } + + /// + /// Predicts a high-bit-depth DC block without hardware intrinsics. + /// + /// Whether the left reference is available. + /// Whether the top reference is available. + /// The destination block. + /// The distance between destination rows. + /// The top reference. + /// The left reference. + /// The block width. + /// The block height. + /// The reconstructed sample precision. + public static void PredictScalar(bool hasLeft, bool hasAbove, Span destination, int destinationStride, ReadOnlySpan above, ReadOnlySpan left, int width, int height, int bitDepth) + { + int count = (hasAbove ? width : 0) + (hasLeft ? height : 0); + int sum = (hasAbove ? SumScalar(above[..width]) : 0) + (hasLeft ? SumScalar(left[..height]) : 0); + short prediction = TOperator.Predict(sum, count, bitDepth); + + for (int row = 0; row < height; row++) + { + ref short destinationRow = ref destination[row * destinationStride]; + for (int column = 0; column < width; column++) + { + Unsafe.Add(ref destinationRow, column) = prediction; + } + } + } + + /// + /// Sums 8-bit references through the widest available SIMD widths. + /// + /// The reference samples. + /// The exact sum. + private static int Sum(ReadOnlySpan samples) + { + ref byte samplesBase = ref MemoryMarshal.GetReference(samples); + int sum = 0; + int index = 0; + + // The shared index deliberately continues through narrower widths. This handles every legal AV1 edge + // length without a separate dispatch tree and leaves only an incomplete final vector to scalar code. + if (Vector512.IsHardwareAccelerated) + { + int oneVectorFromEnd = samples.Length - Vector512.Count; + for (; index <= oneVectorFromEnd; index += Vector512.Count) + { + sum += TOperator.Sum(Vector512.LoadUnsafe(ref samplesBase, (nuint)index)); + } + } + + if (Vector256.IsHardwareAccelerated) + { + int oneVectorFromEnd = samples.Length - Vector256.Count; + for (; index <= oneVectorFromEnd; index += Vector256.Count) + { + sum += TOperator.Sum(Vector256.LoadUnsafe(ref samplesBase, (nuint)index)); + } + } + + if (Vector128.IsHardwareAccelerated) + { + int oneVectorFromEnd = samples.Length - Vector128.Count; + for (; index <= oneVectorFromEnd; index += Vector128.Count) + { + sum += TOperator.Sum(Vector128.LoadUnsafe(ref samplesBase, (nuint)index)); + } + } + + for (; index < samples.Length; index++) + { + sum += TOperator.Sum(Unsafe.Add(ref samplesBase, index)); + } + + return sum; + } + + /// + /// Sums high-bit-depth references through the widest available SIMD widths. + /// + /// The reference samples. + /// The exact sum. + private static int Sum(ReadOnlySpan samples) + { + ref short samplesBase = ref MemoryMarshal.GetReference(samples); + int sum = 0; + int index = 0; + + if (Vector512.IsHardwareAccelerated) + { + int oneVectorFromEnd = samples.Length - Vector512.Count; + for (; index <= oneVectorFromEnd; index += Vector512.Count) + { + sum += TOperator.Sum(Vector512.LoadUnsafe(ref samplesBase, (nuint)index)); + } + } + + if (Vector256.IsHardwareAccelerated) + { + int oneVectorFromEnd = samples.Length - Vector256.Count; + for (; index <= oneVectorFromEnd; index += Vector256.Count) + { + sum += TOperator.Sum(Vector256.LoadUnsafe(ref samplesBase, (nuint)index)); + } + } + + if (Vector128.IsHardwareAccelerated) + { + int oneVectorFromEnd = samples.Length - Vector128.Count; + for (; index <= oneVectorFromEnd; index += Vector128.Count) + { + sum += TOperator.Sum(Vector128.LoadUnsafe(ref samplesBase, (nuint)index)); + } + } + + for (; index < samples.Length; index++) + { + sum += TOperator.Sum(Unsafe.Add(ref samplesBase, index)); + } + + return sum; + } + + /// + /// Sums 8-bit references without hardware intrinsics. + /// + /// The reference samples. + /// The exact sum. + private static int SumScalar(ReadOnlySpan samples) + { + ref byte samplesBase = ref MemoryMarshal.GetReference(samples); + int sum = 0; + + for (int index = 0; index < samples.Length; index++) + { + sum += TOperator.Sum(Unsafe.Add(ref samplesBase, index)); + } + + return sum; + } + + /// + /// Sums high-bit-depth references without hardware intrinsics. + /// + /// The reference samples. + /// The exact sum. + private static int SumScalar(ReadOnlySpan samples) + { + ref short samplesBase = ref MemoryMarshal.GetReference(samples); + int sum = 0; + + for (int index = 0; index < samples.Length; index++) + { + sum += TOperator.Sum(Unsafe.Add(ref samplesBase, index)); + } + + return sum; + } + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1DcIntraPredictor.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1DcIntraPredictor.cs deleted file mode 100644 index 7168c56b4..000000000 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1DcIntraPredictor.cs +++ /dev/null @@ -1,254 +0,0 @@ -// Copyright (c) Six Labors. -// Licensed under the Six Labors Split License. - -using System.Runtime.CompilerServices; -using System.Runtime.InteropServices; -using System.Runtime.Intrinsics; - -namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; - -/// -/// Reconstructs AV1 DC intra-prediction blocks from the available neighboring samples. -/// -/// -/// Reference reduction uses the widest available integer lanes, then one rounded scalar mean is broadcast across each -/// destination row. Row filling is delegated to span operations so the runtime supplies its native vectorized store; -/// explicit scalar entry points remain available for feature-disabled conformance tests. -/// -internal static class Av1DcIntraPredictor -{ - /// - /// Predicts an 8-bit DC block. - /// - /// Whether the prepared left reference is available. - /// Whether the prepared top reference is available. - /// The destination block origin. - /// The destination row stride in samples. - /// The prepared top reference. - /// The prepared left reference. - /// The block width in samples. - /// The block height in samples. - public static void Predict(bool hasLeft, bool hasAbove, Span destination, int destinationStride, ReadOnlySpan above, ReadOnlySpan left, int width, int height) - { - int count = (hasAbove ? width : 0) + (hasLeft ? height : 0); - int sum = (hasAbove ? Sum(above[..width]) : 0) + (hasLeft ? Sum(left[..height]) : 0); - - // Section 7.11.2.2 defines the unsigned midpoint when no reference is available. Otherwise adding half - // the reference count implements the specified rounded mean before extending it across the whole block. - byte prediction = count == 0 ? (byte)128 : (byte)((sum + (count >> 1)) / count); - for (int row = 0; row < height; row++) - { - destination.Slice(row * destinationStride, width).Fill(prediction); - } - } - - /// - /// Predicts a high-bit-depth DC block. - /// - /// Whether the prepared left reference is available. - /// Whether the prepared top reference is available. - /// The destination block origin. - /// The destination row stride in samples. - /// The prepared top reference. - /// The prepared left reference. - /// The block width in samples. - /// The block height in samples. - /// The reconstructed sample precision. - public static void Predict(bool hasLeft, bool hasAbove, Span destination, int destinationStride, ReadOnlySpan above, ReadOnlySpan left, int width, int height, int bitDepth) - { - int count = (hasAbove ? width : 0) + (hasLeft ? height : 0); - int sum = (hasAbove ? Sum(above[..width]) : 0) + (hasLeft ? Sum(left[..height]) : 0); - short prediction = count == 0 ? (short)(1 << (bitDepth - 1)) : (short)((sum + (count >> 1)) / count); - for (int row = 0; row < height; row++) - { - destination.Slice(row * destinationStride, width).Fill(prediction); - } - } - - /// - /// Predicts an 8-bit DC block without hardware intrinsics. - /// - /// Whether the prepared left reference is available. - /// Whether the prepared top reference is available. - /// The destination block origin. - /// The destination row stride in samples. - /// The prepared top reference. - /// The prepared left reference. - /// The block width in samples. - /// The block height in samples. - public static void PredictScalar(bool hasLeft, bool hasAbove, Span destination, int destinationStride, ReadOnlySpan above, ReadOnlySpan left, int width, int height) - { - int count = (hasAbove ? width : 0) + (hasLeft ? height : 0); - int sum = (hasAbove ? SumScalar(above[..width]) : 0) + (hasLeft ? SumScalar(left[..height]) : 0); - byte prediction = count == 0 ? (byte)128 : (byte)((sum + (count >> 1)) / count); - for (int row = 0; row < height; row++) - { - ref byte destinationRow = ref destination[row * destinationStride]; - for (int column = 0; column < width; column++) - { - Unsafe.Add(ref destinationRow, column) = prediction; - } - } - } - - /// - /// Predicts a high-bit-depth DC block without hardware intrinsics. - /// - /// Whether the prepared left reference is available. - /// Whether the prepared top reference is available. - /// The destination block origin. - /// The destination row stride in samples. - /// The prepared top reference. - /// The prepared left reference. - /// The block width in samples. - /// The block height in samples. - /// The reconstructed sample precision. - public static void PredictScalar(bool hasLeft, bool hasAbove, Span destination, int destinationStride, ReadOnlySpan above, ReadOnlySpan left, int width, int height, int bitDepth) - { - int count = (hasAbove ? width : 0) + (hasLeft ? height : 0); - int sum = (hasAbove ? SumScalar(above[..width]) : 0) + (hasLeft ? SumScalar(left[..height]) : 0); - short prediction = count == 0 ? (short)(1 << (bitDepth - 1)) : (short)((sum + (count >> 1)) / count); - for (int row = 0; row < height; row++) - { - ref short destinationRow = ref destination[row * destinationStride]; - for (int column = 0; column < width; column++) - { - Unsafe.Add(ref destinationRow, column) = prediction; - } - } - } - - /// - /// Sums 8-bit neighboring samples using the widest available SIMD width. - /// - /// The samples to sum. - /// The exact sum. - private static int Sum(ReadOnlySpan samples) - { - ref byte samplesBase = ref MemoryMarshal.GetReference(samples); - int sum = 0; - int index = 0; - - // Widening prevents the packed-byte reduction from overflowing before the horizontal sum. The shared index - // advances through every available vector width and leaves only the incomplete final group to the scalar loop. - if (Vector512.IsHardwareAccelerated) - { - int oneVectorFromEnd = samples.Length - Vector512.Count; - for (; index <= oneVectorFromEnd; index += Vector512.Count) - { - (Vector512 low, Vector512 high) = Vector512.Widen(Vector512.LoadUnsafe(ref samplesBase, (nuint)index)); - sum += Vector512.Sum(low) + Vector512.Sum(high); - } - } - - if (Vector256.IsHardwareAccelerated) - { - int oneVectorFromEnd = samples.Length - Vector256.Count; - for (; index <= oneVectorFromEnd; index += Vector256.Count) - { - (Vector256 low, Vector256 high) = Vector256.Widen(Vector256.LoadUnsafe(ref samplesBase, (nuint)index)); - sum += Vector256.Sum(low) + Vector256.Sum(high); - } - } - - if (Vector128.IsHardwareAccelerated) - { - int oneVectorFromEnd = samples.Length - Vector128.Count; - for (; index <= oneVectorFromEnd; index += Vector128.Count) - { - (Vector128 low, Vector128 high) = Vector128.Widen(Vector128.LoadUnsafe(ref samplesBase, (nuint)index)); - sum += Vector128.Sum(low) + Vector128.Sum(high); - } - } - - for (; index < samples.Length; index++) - { - sum += Unsafe.Add(ref samplesBase, index); - } - - return sum; - } - - /// - /// Sums high-bit-depth neighboring samples using the widest available SIMD width. - /// - /// The samples to sum. - /// The exact sum. - private static int Sum(ReadOnlySpan samples) - { - ref short samplesBase = ref MemoryMarshal.GetReference(samples); - int sum = 0; - int index = 0; - - // Signed 16-bit storage is nonnegative for supported bit depths. Widening to Int32 preserves the exact sum of - // the largest permitted reference edge before the rounded mean is calculated once outside this loop. - if (Vector512.IsHardwareAccelerated) - { - int oneVectorFromEnd = samples.Length - Vector512.Count; - for (; index <= oneVectorFromEnd; index += Vector512.Count) - { - (Vector512 low, Vector512 high) = Vector512.Widen(Vector512.LoadUnsafe(ref samplesBase, (nuint)index)); - sum += Vector512.Sum(low) + Vector512.Sum(high); - } - } - - if (Vector256.IsHardwareAccelerated) - { - int oneVectorFromEnd = samples.Length - Vector256.Count; - for (; index <= oneVectorFromEnd; index += Vector256.Count) - { - (Vector256 low, Vector256 high) = Vector256.Widen(Vector256.LoadUnsafe(ref samplesBase, (nuint)index)); - sum += Vector256.Sum(low) + Vector256.Sum(high); - } - } - - if (Vector128.IsHardwareAccelerated) - { - int oneVectorFromEnd = samples.Length - Vector128.Count; - for (; index <= oneVectorFromEnd; index += Vector128.Count) - { - (Vector128 low, Vector128 high) = Vector128.Widen(Vector128.LoadUnsafe(ref samplesBase, (nuint)index)); - sum += Vector128.Sum(low) + Vector128.Sum(high); - } - } - - for (; index < samples.Length; index++) - { - sum += Unsafe.Add(ref samplesBase, index); - } - - return sum; - } - - /// - /// Sums 8-bit neighboring samples without hardware intrinsics. - /// - /// The samples to sum. - /// The exact sum. - private static int SumScalar(ReadOnlySpan samples) - { - int sum = 0; - foreach (byte sample in samples) - { - sum += sample; - } - - return sum; - } - - /// - /// Sums high-bit-depth neighboring samples without hardware intrinsics. - /// - /// The samples to sum. - /// The exact sum. - private static int SumScalar(ReadOnlySpan samples) - { - int sum = 0; - foreach (short sample in samples) - { - sum += sample; - } - - return sum; - } -} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1DirectionalIntraPredictor.Operations.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1DirectionalIntraPredictor.Operations.cs index 26b8d8834..4c5f7e7fc 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1DirectionalIntraPredictor.Operations.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1DirectionalIntraPredictor.Operations.cs @@ -4,6 +4,7 @@ using System.Runtime.CompilerServices; using System.Runtime.InteropServices; using System.Runtime.Intrinsics; +using SixLabors.ImageSharp.Formats.Heif.Av1.Transform; namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; @@ -17,615 +18,933 @@ namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; internal static partial class Av1DirectionalIntraPredictor { /// - /// Gets the indices of even bytes in one upsampled reference vector. + /// Implements the directional traversal for one closed interpolation operator. /// - private static Vector128 EvenByteIndices => Vector128.Create((byte)0, 2, 4, 6, 8, 10, 12, 14, 0, 2, 4, 6, 8, 10, 12, 14); - - /// - /// Gets the indices of odd bytes in one upsampled reference vector. - /// - private static Vector128 OddByteIndices => Vector128.Create((byte)1, 3, 5, 7, 9, 11, 13, 15, 1, 3, 5, 7, 9, 11, 13, 15); - - /// - /// Gets the indices of even high-bit-depth samples in one upsampled reference vector. - /// - private static Vector128 EvenShortIndices => Vector128.Create((short)0, 2, 4, 6, 0, 2, 4, 6); - - /// - /// Gets the indices of odd high-bit-depth samples in one upsampled reference vector. - /// - private static Vector128 OddShortIndices => Vector128.Create((short)1, 3, 5, 7, 1, 3, 5, 7); - - /// - /// Predicts one 8-bit zone 1 block using contiguous SIMD projection rows. - /// - /// The destination block origin. - /// The destination row stride. - /// The projected top reference. - /// Whether the reference contains half-sample positions. - /// The Q8 projection derivative. - /// The block width. - /// The block height. - private static void PredictZone1(Span destination, int destinationStride, ReadOnlySpan above, bool upsample, int derivative, int width, int height) + private static partial class Predictor + where TOperator : struct, IDirectionalPredictionOperator { - int upsampleShift = upsample ? 1 : 0; - int maximumBasis = (width + height - 1) << upsampleShift; - int fractionBits = 6 - upsampleShift; - int projection = derivative; - - for (int row = 0; row < height; row++, projection += derivative) + /// + /// Gets the indices of even bytes in one upsampled reference vector. + /// + private static Vector128 EvenByteIndices => Vector128.Create((byte)0, 2, 4, 6, 8, 10, 12, 14, 0, 2, 4, 6, 8, 10, 12, 14); + + /// + /// Gets the indices of odd bytes in one upsampled reference vector. + /// + private static Vector128 OddByteIndices => Vector128.Create((byte)1, 3, 5, 7, 9, 11, 13, 15, 1, 3, 5, 7, 9, 11, 13, 15); + + /// + /// Gets the indices of even high-bit-depth samples in one upsampled reference vector. + /// + private static Vector128 EvenShortIndices => Vector128.Create((short)0, 2, 4, 6, 0, 2, 4, 6); + + /// + /// Gets the indices of odd high-bit-depth samples in one upsampled reference vector. + /// + private static Vector128 OddShortIndices => Vector128.Create((short)1, 3, 5, 7, 1, 3, 5, 7); + + /// + /// Predicts one 8-bit zone 1 block using contiguous SIMD projection rows. + /// + /// The destination block origin. + /// The destination row stride. + /// The projected top reference. + /// Whether the reference contains half-sample positions. + /// The Q8 projection derivative. + /// The block width. + /// The block height. + private static void PredictZone1(Span destination, int destinationStride, ReadOnlySpan above, bool upsample, int derivative, int width, int height) { - int basis = projection >> fractionBits; - int weight = ((projection << upsampleShift) & 0x3F) >> 1; - InterpolateRow(destination.Slice(row * destinationStride, width), above, basis, weight, upsample, maximumBasis); - } - } + int upsampleShift = upsample ? 1 : 0; + int maximumBasis = (width + height - 1) << upsampleShift; + int fractionBits = 6 - upsampleShift; + int projection = derivative; - /// - /// Predicts one high-bit-depth zone 1 block using contiguous SIMD projection rows. - /// - /// The destination block origin. - /// The destination row stride. - /// The projected top reference. - /// Whether the reference contains half-sample positions. - /// The Q8 projection derivative. - /// The block width. - /// The block height. - private static void PredictZone1(Span destination, int destinationStride, ReadOnlySpan above, bool upsample, int derivative, int width, int height) - { - int upsampleShift = upsample ? 1 : 0; - int maximumBasis = (width + height - 1) << upsampleShift; - int fractionBits = 6 - upsampleShift; - int projection = derivative; - - for (int row = 0; row < height; row++, projection += derivative) - { - int basis = projection >> fractionBits; - int weight = ((projection << upsampleShift) & 0x3F) >> 1; - InterpolateRow(destination.Slice(row * destinationStride, width), above, basis, weight, upsample, maximumBasis); + for (int row = 0; row < height; row++, projection += derivative) + { + int basis = projection >> fractionBits; + int weight = ((projection << upsampleShift) & 0x3F) >> 1; + InterpolateRow(destination.Slice(row * destinationStride, width), above, basis, weight, upsample, maximumBasis); + } } - } - /// - /// Predicts one 8-bit zone 2 block using vectorized left gathers and contiguous top projection rows. - /// - /// The destination block origin. - /// The destination row stride. - /// The projected top reference. - /// The projected left reference. - /// Whether the top reference contains half-sample positions. - /// Whether the left reference contains half-sample positions. - /// The horizontal Q8 derivative. - /// The vertical Q8 derivative. - /// The block width. - /// The block height. - private static void PredictZone2(Span destination, int destinationStride, ReadOnlySpan above, ReadOnlySpan left, bool upsampleAbove, bool upsampleLeft, int dx, int dy, int width, int height) - { - int aboveShift = upsampleAbove ? 1 : 0; - int minimumTopBasis = -(1 << aboveShift); - int topFractionBits = 6 - aboveShift; - int topBasisIncrement = 1 << aboveShift; - int topProjection = -dx; - - for (int row = 0; row < height; row++, topProjection -= dx) + /// + /// Predicts one high-bit-depth zone 1 block using contiguous SIMD projection rows. + /// + /// The destination block origin. + /// The destination row stride. + /// The projected top reference. + /// Whether the reference contains half-sample positions. + /// The Q8 projection derivative. + /// The block width. + /// The block height. + private static void PredictZone1(Span destination, int destinationStride, ReadOnlySpan above, bool upsample, int derivative, int width, int height) { - int topBasis = topProjection >> topFractionBits; - int leftCount = 0; - while (leftCount < width && topBasis < minimumTopBasis) + int upsampleShift = upsample ? 1 : 0; + int maximumBasis = (width + height - 1) << upsampleShift; + int fractionBits = 6 - upsampleShift; + int projection = derivative; + + for (int row = 0; row < height; row++, projection += derivative) { - leftCount++; - topBasis += topBasisIncrement; + int basis = projection >> fractionBits; + int weight = ((projection << upsampleShift) & 0x3F) >> 1; + InterpolateRow(destination.Slice(row * destinationStride, width), above, basis, weight, upsample, maximumBasis); } + } - Span destinationRow = destination.Slice(row * destinationStride, width); - int leftProjection = (row << 6) - dy; - InterpolateLeft(destinationRow[..leftCount], left, leftProjection, dy, upsampleLeft); + /// + /// Predicts one 8-bit zone 2 block using vectorized left gathers and contiguous top projection rows. + /// + /// The destination block origin. + /// The destination row stride. + /// The projected top reference. + /// The projected left reference. + /// Whether the top reference contains half-sample positions. + /// Whether the left reference contains half-sample positions. + /// The horizontal Q8 derivative. + /// The vertical Q8 derivative. + /// The block width. + /// The block height. + private static void PredictZone2(Span destination, int destinationStride, ReadOnlySpan above, ReadOnlySpan left, bool upsampleAbove, bool upsampleLeft, int dx, int dy, int width, int height) + { + int aboveShift = upsampleAbove ? 1 : 0; + int minimumTopBasis = -(1 << aboveShift); + int topFractionBits = 6 - aboveShift; + int topBasisIncrement = 1 << aboveShift; + int topProjection = -dx; - if (leftCount < width) + for (int row = 0; row < height; row++, topProjection -= dx) { - int topWeight = ((topProjection << aboveShift) & 0x3F) >> 1; - InterpolateRow(destinationRow[leftCount..], above, topBasis, topWeight, upsampleAbove, int.MaxValue); + int topBasis = topProjection >> topFractionBits; + int leftCount = 0; + while (leftCount < width && topBasis < minimumTopBasis) + { + leftCount++; + topBasis += topBasisIncrement; + } + + Span destinationRow = destination.Slice(row * destinationStride, width); + int leftProjection = (row << 6) - dy; + InterpolateLeft(destinationRow[..leftCount], left, leftProjection, dy, upsampleLeft); + + if (leftCount < width) + { + int topWeight = ((topProjection << aboveShift) & 0x3F) >> 1; + InterpolateRow(destinationRow[leftCount..], above, topBasis, topWeight, upsampleAbove, int.MaxValue); + } } } - } - - /// - /// Predicts one high-bit-depth zone 2 block using vectorized left gathers and contiguous top projection rows. - /// - /// The destination block origin. - /// The destination row stride. - /// The projected top reference. - /// The projected left reference. - /// Whether the top reference contains half-sample positions. - /// Whether the left reference contains half-sample positions. - /// The horizontal Q8 derivative. - /// The vertical Q8 derivative. - /// The block width. - /// The block height. - private static void PredictZone2(Span destination, int destinationStride, ReadOnlySpan above, ReadOnlySpan left, bool upsampleAbove, bool upsampleLeft, int dx, int dy, int width, int height) - { - int aboveShift = upsampleAbove ? 1 : 0; - int minimumTopBasis = -(1 << aboveShift); - int topFractionBits = 6 - aboveShift; - int topBasisIncrement = 1 << aboveShift; - int topProjection = -dx; - for (int row = 0; row < height; row++, topProjection -= dx) + /// + /// Predicts one high-bit-depth zone 2 block using vectorized left gathers and contiguous top projection rows. + /// + /// The destination block origin. + /// The destination row stride. + /// The projected top reference. + /// The projected left reference. + /// Whether the top reference contains half-sample positions. + /// Whether the left reference contains half-sample positions. + /// The horizontal Q8 derivative. + /// The vertical Q8 derivative. + /// The block width. + /// The block height. + private static void PredictZone2(Span destination, int destinationStride, ReadOnlySpan above, ReadOnlySpan left, bool upsampleAbove, bool upsampleLeft, int dx, int dy, int width, int height) { - int topBasis = topProjection >> topFractionBits; - int leftCount = 0; - while (leftCount < width && topBasis < minimumTopBasis) + int aboveShift = upsampleAbove ? 1 : 0; + int minimumTopBasis = -(1 << aboveShift); + int topFractionBits = 6 - aboveShift; + int topBasisIncrement = 1 << aboveShift; + int topProjection = -dx; + + for (int row = 0; row < height; row++, topProjection -= dx) { - leftCount++; - topBasis += topBasisIncrement; - } + int topBasis = topProjection >> topFractionBits; + int leftCount = 0; + while (leftCount < width && topBasis < minimumTopBasis) + { + leftCount++; + topBasis += topBasisIncrement; + } - Span destinationRow = destination.Slice(row * destinationStride, width); - int leftProjection = (row << 6) - dy; - InterpolateLeft(destinationRow[..leftCount], left, leftProjection, dy, upsampleLeft); + Span destinationRow = destination.Slice(row * destinationStride, width); + int leftProjection = (row << 6) - dy; + InterpolateLeft(destinationRow[..leftCount], left, leftProjection, dy, upsampleLeft); - if (leftCount < width) - { - int topWeight = ((topProjection << aboveShift) & 0x3F) >> 1; - InterpolateRow(destinationRow[leftCount..], above, topBasis, topWeight, upsampleAbove, int.MaxValue); + if (leftCount < width) + { + int topWeight = ((topProjection << aboveShift) & 0x3F) >> 1; + InterpolateRow(destinationRow[leftCount..], above, topBasis, topWeight, upsampleAbove, int.MaxValue); + } } } - } - /// - /// Interpolates one 8-bit projection row. - /// - /// The destination row. - /// The projected reference samples. - /// The first integral reference coordinate. - /// The right-sample interpolation weight. - /// Whether consecutive output samples advance two reference positions. - /// The final extended reference coordinate, or when the row cannot reach it. - private static void InterpolateRow(Span destination, ReadOnlySpan reference, int basis, int weight, bool upsample, int maximumBasis) - { - ref byte destinationBase = ref MemoryMarshal.GetReference(destination); - ref byte referenceBase = ref MemoryMarshal.GetReference(reference); - int basisIncrement = upsample ? 2 : 1; - int validCount = maximumBasis == int.MaxValue || basis >= maximumBasis - ? maximumBasis == int.MaxValue ? destination.Length : 0 - : Math.Min(destination.Length, ((maximumBasis - 1 - basis) / basisIncrement) + 1); - int index = 0; - - if (!upsample) + /// + /// Interpolates one 8-bit projection row. + /// + /// The destination row. + /// The projected reference samples. + /// The first integral reference coordinate. + /// The right-sample interpolation weight. + /// Whether consecutive output samples advance two reference positions. + /// The final extended reference coordinate, or when the row cannot reach it. + private static void InterpolateRow(Span destination, ReadOnlySpan reference, int basis, int weight, bool upsample, int maximumBasis) { - // A single index is advanced through all supported widths. A narrower path consumes only the remainder - // left by the wider path, so the row is written once without requiring padded destination storage. - if (Vector512.IsHardwareAccelerated) + ref byte destinationBase = ref MemoryMarshal.GetReference(destination); + ref byte referenceBase = ref MemoryMarshal.GetReference(reference); + int basisIncrement = upsample ? 2 : 1; + int validCount = maximumBasis == int.MaxValue || basis >= maximumBasis + ? maximumBasis == int.MaxValue ? destination.Length : 0 + : Math.Min(destination.Length, ((maximumBasis - 1 - basis) / basisIncrement) + 1); + int index = 0; + + if (!upsample) { - int oneVectorFromEnd = validCount - Vector512.Count; - for (; index <= oneVectorFromEnd; index += Vector512.Count) + // A single index is advanced through all supported widths. A narrower path consumes only the remainder + // left by the wider path, so the row is written once without requiring padded destination storage. + if (Vector512.IsHardwareAccelerated) { - Vector512 left = Vector512.LoadUnsafe(ref referenceBase, (nuint)(basis + index)); - Vector512 right = Vector512.LoadUnsafe(ref referenceBase, (nuint)(basis + index + 1)); - Interpolate(left, right, weight).StoreUnsafe(ref destinationBase, (nuint)index); + int oneVectorFromEnd = validCount - Vector512.Count; + for (; index <= oneVectorFromEnd; index += Vector512.Count) + { + Vector512 left = Vector512.LoadUnsafe(ref referenceBase, (nuint)(basis + index)); + Vector512 right = Vector512.LoadUnsafe(ref referenceBase, (nuint)(basis + index + 1)); + TOperator.Interpolate(left, right, weight).StoreUnsafe(ref destinationBase, (nuint)index); + } + } + + if (Vector256.IsHardwareAccelerated) + { + int oneVectorFromEnd = validCount - Vector256.Count; + for (; index <= oneVectorFromEnd; index += Vector256.Count) + { + Vector256 left = Vector256.LoadUnsafe(ref referenceBase, (nuint)(basis + index)); + Vector256 right = Vector256.LoadUnsafe(ref referenceBase, (nuint)(basis + index + 1)); + TOperator.Interpolate(left, right, weight).StoreUnsafe(ref destinationBase, (nuint)index); + } } - } - if (Vector256.IsHardwareAccelerated) + if (Vector128.IsHardwareAccelerated) + { + int oneVectorFromEnd = validCount - Vector128.Count; + for (; index <= oneVectorFromEnd; index += Vector128.Count) + { + Vector128 left = Vector128.LoadUnsafe(ref referenceBase, (nuint)(basis + index)); + Vector128 right = Vector128.LoadUnsafe(ref referenceBase, (nuint)(basis + index + 1)); + TOperator.Interpolate(left, right, weight).StoreUnsafe(ref destinationBase, (nuint)index); + } + } + } + else if (Vector128.IsHardwareAccelerated) { - int oneVectorFromEnd = validCount - Vector256.Count; - for (; index <= oneVectorFromEnd; index += Vector256.Count) + int oneVectorFromEnd = validCount - 8; + for (; index <= oneVectorFromEnd; index += 8) { - Vector256 left = Vector256.LoadUnsafe(ref referenceBase, (nuint)(basis + index)); - Vector256 right = Vector256.LoadUnsafe(ref referenceBase, (nuint)(basis + index + 1)); - Interpolate(left, right, weight).StoreUnsafe(ref destinationBase, (nuint)index); + Vector128 source = Vector128.LoadUnsafe(ref referenceBase, (nuint)(basis + (index * 2))); + + // ShuffleNative maps to byte-table lookup on AdvSimd and PSHUFB on x86. Eight output samples are + // gathered from sixteen half-sample positions without scalar lane construction. + Vector128 left = Vector128.ShuffleNative(source, EvenByteIndices); + Vector128 right = Vector128.ShuffleNative(source, OddByteIndices); + Vector128 prediction = TOperator.Interpolate(left, right, weight); + Unsafe.As(ref Unsafe.Add(ref destinationBase, index)) = prediction.AsUInt64().ToScalar(); } } if (Vector128.IsHardwareAccelerated) { - int oneVectorFromEnd = validCount - Vector128.Count; - for (; index <= oneVectorFromEnd; index += Vector128.Count) + // Four-lane construction covers both the final non-upsampled remainder and targets without a native + // gather. Each lane carries an independently projected coordinate but shares the row's interpolation weight. + int oneVectorFromEnd = validCount - 4; + for (; index <= oneVectorFromEnd; index += 4) { - Vector128 left = Vector128.LoadUnsafe(ref referenceBase, (nuint)(basis + index)); - Vector128 right = Vector128.LoadUnsafe(ref referenceBase, (nuint)(basis + index + 1)); - Interpolate(left, right, weight).StoreUnsafe(ref destinationBase, (nuint)index); + int source = basis + (index * basisIncrement); + Vector128 left = Vector128.Create( + (int)Unsafe.Add(ref referenceBase, source), + Unsafe.Add(ref referenceBase, source + basisIncrement), + Unsafe.Add(ref referenceBase, source + (2 * basisIncrement)), + Unsafe.Add(ref referenceBase, source + (3 * basisIncrement))); + + Vector128 right = Vector128.Create( + (int)Unsafe.Add(ref referenceBase, source + 1), + Unsafe.Add(ref referenceBase, source + basisIncrement + 1), + Unsafe.Add(ref referenceBase, source + (2 * basisIncrement) + 1), + Unsafe.Add(ref referenceBase, source + (3 * basisIncrement) + 1)); + + StoreFourBytes(TOperator.Interpolate(left, right, Vector128.Create(weight)), ref Unsafe.Add(ref destinationBase, index)); } } - } - else if (Vector128.IsHardwareAccelerated) - { - int oneVectorFromEnd = validCount - 8; - for (; index <= oneVectorFromEnd; index += 8) - { - Vector128 source = Vector128.LoadUnsafe(ref referenceBase, (nuint)(basis + (index * 2))); - - // ShuffleNative maps to byte-table lookup on AdvSimd and PSHUFB on x86. Eight output samples are - // gathered from sixteen half-sample positions without scalar lane construction. - Vector128 left = Vector128.ShuffleNative(source, EvenByteIndices); - Vector128 right = Vector128.ShuffleNative(source, OddByteIndices); - Vector128 prediction = Interpolate(left, right, weight); - Unsafe.As(ref Unsafe.Add(ref destinationBase, index)) = prediction.AsUInt64().GetElement(0); - } - } - if (Vector128.IsHardwareAccelerated) - { - // Four-lane construction covers both the final non-upsampled remainder and targets without a native - // gather. Each lane carries an independently projected coordinate but shares the row's interpolation weight. - int oneVectorFromEnd = validCount - 4; - for (; index <= oneVectorFromEnd; index += 4) + for (; index < validCount; index++) { int source = basis + (index * basisIncrement); - Vector128 left = Vector128.Create( - (int)Unsafe.Add(ref referenceBase, source), - Unsafe.Add(ref referenceBase, source + basisIncrement), - Unsafe.Add(ref referenceBase, source + (2 * basisIncrement)), - Unsafe.Add(ref referenceBase, source + (3 * basisIncrement))); - - Vector128 right = Vector128.Create( - (int)Unsafe.Add(ref referenceBase, source + 1), - Unsafe.Add(ref referenceBase, source + basisIncrement + 1), - Unsafe.Add(ref referenceBase, source + (2 * basisIncrement) + 1), - Unsafe.Add(ref referenceBase, source + (3 * basisIncrement) + 1)); + Unsafe.Add(ref destinationBase, index) = (byte)(((Unsafe.Add(ref referenceBase, source) * (32 - weight)) + (Unsafe.Add(ref referenceBase, source + 1) * weight) + 16) >> 5); + } - StoreFourBytes(Interpolate(left, right, Vector128.Create(weight)), ref Unsafe.Add(ref destinationBase, index)); + if (validCount < destination.Length) + { + destination[validCount..].Fill(Unsafe.Add(ref referenceBase, maximumBasis)); } } - for (; index < validCount; index++) + /// + /// Interpolates one high-bit-depth projection row. + /// + /// The destination row. + /// The projected reference samples. + /// The first integral reference coordinate. + /// The right-sample interpolation weight. + /// Whether consecutive output samples advance two reference positions. + /// The final extended reference coordinate, or when the row cannot reach it. + private static void InterpolateRow(Span destination, ReadOnlySpan reference, int basis, int weight, bool upsample, int maximumBasis) { - int source = basis + (index * basisIncrement); - Unsafe.Add(ref destinationBase, index) = (byte)(((Unsafe.Add(ref referenceBase, source) * (32 - weight)) + (Unsafe.Add(ref referenceBase, source + 1) * weight) + 16) >> 5); - } + ref short destinationBase = ref MemoryMarshal.GetReference(destination); + ref short referenceBase = ref MemoryMarshal.GetReference(reference); + int basisIncrement = upsample ? 2 : 1; + int validCount = maximumBasis == int.MaxValue || basis >= maximumBasis + ? maximumBasis == int.MaxValue ? destination.Length : 0 + : Math.Min(destination.Length, ((maximumBasis - 1 - basis) / basisIncrement) + 1); + int index = 0; + + if (!upsample) + { + // High-bit-depth samples stay in signed 16-bit storage, but interpolation widens to Int32 before the Q5 + // weighted sum. The largest supported 12-bit sample therefore cannot overflow an intermediate lane. + if (Vector512.IsHardwareAccelerated) + { + int oneVectorFromEnd = validCount - Vector512.Count; + for (; index <= oneVectorFromEnd; index += Vector512.Count) + { + Vector512 left = Vector512.LoadUnsafe(ref referenceBase, (nuint)(basis + index)); + Vector512 right = Vector512.LoadUnsafe(ref referenceBase, (nuint)(basis + index + 1)); + TOperator.Interpolate(left, right, weight).StoreUnsafe(ref destinationBase, (nuint)index); + } + } - if (validCount < destination.Length) - { - destination[validCount..].Fill(Unsafe.Add(ref referenceBase, maximumBasis)); - } - } + if (Vector256.IsHardwareAccelerated) + { + int oneVectorFromEnd = validCount - Vector256.Count; + for (; index <= oneVectorFromEnd; index += Vector256.Count) + { + Vector256 left = Vector256.LoadUnsafe(ref referenceBase, (nuint)(basis + index)); + Vector256 right = Vector256.LoadUnsafe(ref referenceBase, (nuint)(basis + index + 1)); + TOperator.Interpolate(left, right, weight).StoreUnsafe(ref destinationBase, (nuint)index); + } + } - /// - /// Interpolates one high-bit-depth projection row. - /// - /// The destination row. - /// The projected reference samples. - /// The first integral reference coordinate. - /// The right-sample interpolation weight. - /// Whether consecutive output samples advance two reference positions. - /// The final extended reference coordinate, or when the row cannot reach it. - private static void InterpolateRow(Span destination, ReadOnlySpan reference, int basis, int weight, bool upsample, int maximumBasis) - { - ref short destinationBase = ref MemoryMarshal.GetReference(destination); - ref short referenceBase = ref MemoryMarshal.GetReference(reference); - int basisIncrement = upsample ? 2 : 1; - int validCount = maximumBasis == int.MaxValue || basis >= maximumBasis - ? maximumBasis == int.MaxValue ? destination.Length : 0 - : Math.Min(destination.Length, ((maximumBasis - 1 - basis) / basisIncrement) + 1); - int index = 0; - - if (!upsample) - { - // High-bit-depth samples stay in signed 16-bit storage, but interpolation widens to Int32 before the Q5 - // weighted sum. The largest supported 12-bit sample therefore cannot overflow an intermediate lane. - if (Vector512.IsHardwareAccelerated) - { - int oneVectorFromEnd = validCount - Vector512.Count; - for (; index <= oneVectorFromEnd; index += Vector512.Count) + if (Vector128.IsHardwareAccelerated) { - Vector512 left = Vector512.LoadUnsafe(ref referenceBase, (nuint)(basis + index)); - Vector512 right = Vector512.LoadUnsafe(ref referenceBase, (nuint)(basis + index + 1)); - Interpolate(left, right, weight).StoreUnsafe(ref destinationBase, (nuint)index); + int oneVectorFromEnd = validCount - Vector128.Count; + for (; index <= oneVectorFromEnd; index += Vector128.Count) + { + Vector128 left = Vector128.LoadUnsafe(ref referenceBase, (nuint)(basis + index)); + Vector128 right = Vector128.LoadUnsafe(ref referenceBase, (nuint)(basis + index + 1)); + TOperator.Interpolate(left, right, weight).StoreUnsafe(ref destinationBase, (nuint)index); + } } } - - if (Vector256.IsHardwareAccelerated) + else if (Vector128.IsHardwareAccelerated) { - int oneVectorFromEnd = validCount - Vector256.Count; - for (; index <= oneVectorFromEnd; index += Vector256.Count) + int oneVectorFromEnd = validCount - 4; + for (; index <= oneVectorFromEnd; index += 4) { - Vector256 left = Vector256.LoadUnsafe(ref referenceBase, (nuint)(basis + index)); - Vector256 right = Vector256.LoadUnsafe(ref referenceBase, (nuint)(basis + index + 1)); - Interpolate(left, right, weight).StoreUnsafe(ref destinationBase, (nuint)index); + Vector128 source = Vector128.LoadUnsafe(ref referenceBase, (nuint)(basis + (index * 2))); + Vector128 left = Vector128.ShuffleNative(source, EvenShortIndices); + Vector128 right = Vector128.ShuffleNative(source, OddShortIndices); + Vector128 prediction = TOperator.Interpolate(left, right, weight); + Unsafe.As(ref Unsafe.Add(ref destinationBase, index)) = prediction.AsUInt64().ToScalar(); } } if (Vector128.IsHardwareAccelerated) { - int oneVectorFromEnd = validCount - Vector128.Count; - for (; index <= oneVectorFromEnd; index += Vector128.Count) + int oneVectorFromEnd = validCount - 4; + for (; index <= oneVectorFromEnd; index += 4) { - Vector128 left = Vector128.LoadUnsafe(ref referenceBase, (nuint)(basis + index)); - Vector128 right = Vector128.LoadUnsafe(ref referenceBase, (nuint)(basis + index + 1)); - Interpolate(left, right, weight).StoreUnsafe(ref destinationBase, (nuint)index); + int source = basis + (index * basisIncrement); + Vector128 left = Vector128.Create( + (int)Unsafe.Add(ref referenceBase, source), + Unsafe.Add(ref referenceBase, source + basisIncrement), + Unsafe.Add(ref referenceBase, source + (2 * basisIncrement)), + Unsafe.Add(ref referenceBase, source + (3 * basisIncrement))); + + Vector128 right = Vector128.Create( + (int)Unsafe.Add(ref referenceBase, source + 1), + Unsafe.Add(ref referenceBase, source + basisIncrement + 1), + Unsafe.Add(ref referenceBase, source + (2 * basisIncrement) + 1), + Unsafe.Add(ref referenceBase, source + (3 * basisIncrement) + 1)); + + StoreFourShorts(TOperator.Interpolate(left, right, Vector128.Create(weight)), ref Unsafe.Add(ref destinationBase, index)); } } + + for (; index < validCount; index++) + { + int source = basis + (index * basisIncrement); + Unsafe.Add(ref destinationBase, index) = (short)(((Unsafe.Add(ref referenceBase, source) * (32 - weight)) + (Unsafe.Add(ref referenceBase, source + 1) * weight) + 16) >> 5); + } + + if (validCount < destination.Length) + { + destination[validCount..].Fill(Unsafe.Add(ref referenceBase, maximumBasis)); + } } - else if (Vector128.IsHardwareAccelerated) + + /// + /// Interpolates the left-edge prefix of one 8-bit zone 2 row. + /// + /// The destination prefix. + /// The projected left reference. + /// The first Q6 left projection. + /// The Q8 derivative subtracted between columns. + /// Whether the left reference contains half-sample positions. + private static void InterpolateLeft(Span destination, ReadOnlySpan left, int projection, int derivative, bool upsample) { - int oneVectorFromEnd = validCount - 4; - for (; index <= oneVectorFromEnd; index += 4) + ref byte destinationBase = ref MemoryMarshal.GetReference(destination); + ref byte leftBase = ref MemoryMarshal.GetReference(left); + int upsampleShift = upsample ? 1 : 0; + int fractionBits = 6 - upsampleShift; + int index = 0; + + if (Vector128.IsHardwareAccelerated) + { + // Zone-two left references are not contiguous across output columns. Constructing the four source pairs + // directly avoids a temporary gather-index buffer and keeps the scalar continuation at the same offset. + int oneVectorFromEnd = destination.Length - 4; + for (; index <= oneVectorFromEnd; index += 4) + { + int projection0 = projection - (index * derivative); + int projection1 = projection0 - derivative; + int projection2 = projection1 - derivative; + int projection3 = projection2 - derivative; + int basis0 = projection0 >> fractionBits; + int basis1 = projection1 >> fractionBits; + int basis2 = projection2 >> fractionBits; + int basis3 = projection3 >> fractionBits; + Vector128 source0 = Vector128.Create((int)Unsafe.Add(ref leftBase, basis0), Unsafe.Add(ref leftBase, basis1), Unsafe.Add(ref leftBase, basis2), Unsafe.Add(ref leftBase, basis3)); + Vector128 source1 = Vector128.Create((int)Unsafe.Add(ref leftBase, basis0 + 1), Unsafe.Add(ref leftBase, basis1 + 1), Unsafe.Add(ref leftBase, basis2 + 1), Unsafe.Add(ref leftBase, basis3 + 1)); + Vector128 weights = Vector128.Create( + ((projection0 << upsampleShift) & 0x3F) >> 1, + ((projection1 << upsampleShift) & 0x3F) >> 1, + ((projection2 << upsampleShift) & 0x3F) >> 1, + ((projection3 << upsampleShift) & 0x3F) >> 1); + + StoreFourBytes(TOperator.Interpolate(source0, source1, weights), ref Unsafe.Add(ref destinationBase, index)); + } + } + + for (; index < destination.Length; index++) { - Vector128 source = Vector128.LoadUnsafe(ref referenceBase, (nuint)(basis + (index * 2))); - Vector128 left = Vector128.ShuffleNative(source, EvenShortIndices); - Vector128 right = Vector128.ShuffleNative(source, OddShortIndices); - Vector128 prediction = Interpolate(left, right, weight); - Unsafe.As(ref Unsafe.Add(ref destinationBase, index)) = prediction.AsUInt64().GetElement(0); + int currentProjection = projection - (index * derivative); + int basis = currentProjection >> fractionBits; + int weight = ((currentProjection << upsampleShift) & 0x3F) >> 1; + Unsafe.Add(ref destinationBase, index) = (byte)(((Unsafe.Add(ref leftBase, basis) * (32 - weight)) + (Unsafe.Add(ref leftBase, basis + 1) * weight) + 16) >> 5); } } - if (Vector128.IsHardwareAccelerated) + /// + /// Interpolates the left-edge prefix of one high-bit-depth zone 2 row. + /// + /// The destination prefix. + /// The projected left reference. + /// The first Q6 left projection. + /// The Q8 derivative subtracted between columns. + /// Whether the left reference contains half-sample positions. + private static void InterpolateLeft(Span destination, ReadOnlySpan left, int projection, int derivative, bool upsample) { - int oneVectorFromEnd = validCount - 4; - for (; index <= oneVectorFromEnd; index += 4) - { - int source = basis + (index * basisIncrement); - Vector128 left = Vector128.Create( - (int)Unsafe.Add(ref referenceBase, source), - Unsafe.Add(ref referenceBase, source + basisIncrement), - Unsafe.Add(ref referenceBase, source + (2 * basisIncrement)), - Unsafe.Add(ref referenceBase, source + (3 * basisIncrement))); + ref short destinationBase = ref MemoryMarshal.GetReference(destination); + ref short leftBase = ref MemoryMarshal.GetReference(left); + int upsampleShift = upsample ? 1 : 0; + int fractionBits = 6 - upsampleShift; + int index = 0; - Vector128 right = Vector128.Create( - (int)Unsafe.Add(ref referenceBase, source + 1), - Unsafe.Add(ref referenceBase, source + basisIncrement + 1), - Unsafe.Add(ref referenceBase, source + (2 * basisIncrement) + 1), - Unsafe.Add(ref referenceBase, source + (3 * basisIncrement) + 1)); + if (Vector128.IsHardwareAccelerated) + { + int oneVectorFromEnd = destination.Length - 4; + for (; index <= oneVectorFromEnd; index += 4) + { + int projection0 = projection - (index * derivative); + int projection1 = projection0 - derivative; + int projection2 = projection1 - derivative; + int projection3 = projection2 - derivative; + int basis0 = projection0 >> fractionBits; + int basis1 = projection1 >> fractionBits; + int basis2 = projection2 >> fractionBits; + int basis3 = projection3 >> fractionBits; + Vector128 source0 = Vector128.Create((int)Unsafe.Add(ref leftBase, basis0), Unsafe.Add(ref leftBase, basis1), Unsafe.Add(ref leftBase, basis2), Unsafe.Add(ref leftBase, basis3)); + Vector128 source1 = Vector128.Create((int)Unsafe.Add(ref leftBase, basis0 + 1), Unsafe.Add(ref leftBase, basis1 + 1), Unsafe.Add(ref leftBase, basis2 + 1), Unsafe.Add(ref leftBase, basis3 + 1)); + Vector128 weights = Vector128.Create( + ((projection0 << upsampleShift) & 0x3F) >> 1, + ((projection1 << upsampleShift) & 0x3F) >> 1, + ((projection2 << upsampleShift) & 0x3F) >> 1, + ((projection3 << upsampleShift) & 0x3F) >> 1); + + StoreFourShorts(TOperator.Interpolate(source0, source1, weights), ref Unsafe.Add(ref destinationBase, index)); + } + } - StoreFourShorts(Interpolate(left, right, Vector128.Create(weight)), ref Unsafe.Add(ref destinationBase, index)); + for (; index < destination.Length; index++) + { + int currentProjection = projection - (index * derivative); + int basis = currentProjection >> fractionBits; + int weight = ((currentProjection << upsampleShift) & 0x3F) >> 1; + Unsafe.Add(ref destinationBase, index) = (short)(((Unsafe.Add(ref leftBase, basis) * (32 - weight)) + (Unsafe.Add(ref leftBase, basis + 1) * weight) + 16) >> 5); } } - for (; index < validCount; index++) + /// + /// Stores four widened predictions as packed 8-bit samples. + /// + /// The widened predictions. + /// The first destination sample. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void StoreFourBytes(Vector128 prediction, ref byte destination) { - int source = basis + (index * basisIncrement); - Unsafe.Add(ref destinationBase, index) = (short)(((Unsafe.Add(ref referenceBase, source) * (32 - weight)) + (Unsafe.Add(ref referenceBase, source + 1) * weight) + 16) >> 5); + Vector128 narrowed16 = Vector128.Narrow(prediction.AsUInt32(), Vector128.Zero); + Vector128 narrowed8 = Vector128.Narrow(narrowed16, Vector128.Zero); + Unsafe.As(ref destination) = narrowed8.AsUInt32().GetElement(0); } - if (validCount < destination.Length) + /// + /// Stores four widened predictions as packed high-bit-depth samples. + /// + /// The widened predictions. + /// The first destination sample. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void StoreFourShorts(Vector128 prediction, ref short destination) { - destination[validCount..].Fill(Unsafe.Add(ref referenceBase, maximumBasis)); + Vector128 narrowed = Vector128.Narrow(prediction, Vector128.Zero); + Unsafe.As(ref destination) = narrowed.AsUInt64().ToScalar(); } } /// - /// Interpolates the left-edge prefix of one 8-bit zone 2 row. + /// Implements directional intra prediction through one closed interpolation operator. /// - /// The destination prefix. - /// The projected left reference. - /// The first Q6 left projection. - /// The Q8 derivative subtracted between columns. - /// Whether the left reference contains half-sample positions. - private static void InterpolateLeft(Span destination, ReadOnlySpan left, int projection, int derivative, bool upsample) + /// The directional interpolation arithmetic. + private static partial class Predictor + where TOperator : struct, IDirectionalPredictionOperator { - ref byte destinationBase = ref MemoryMarshal.GetReference(destination); - ref byte leftBase = ref MemoryMarshal.GetReference(left); - int upsampleShift = upsample ? 1 : 0; - int fractionBits = 6 - upsampleShift; - int index = 0; - - if (Vector128.IsHardwareAccelerated) + /// + /// Gets the Q8 directional derivatives indexed by acute prediction angle. + /// + private static ReadOnlySpan DirectionalIntraDerivative => + [ + + // Zero entries represent angles which AV1 never signals. Direct indexing avoids a search or division in + // each directional block while retaining the exact fixed-point projections from the normative table. + 0, 0, 0, 1023, 0, 0, 547, 0, 0, 372, 0, 0, 0, 0, 273, 0, 0, 215, 0, 0, 178, 0, 0, + 151, 0, 0, 132, 0, 0, 116, 0, 0, 102, 0, 0, 0, 90, 0, 0, 80, 0, 0, 71, 0, 0, 64, 0, 0, + 57, 0, 0, 51, 0, 0, 45, 0, 0, 0, 40, 0, 0, 35, 0, 0, 31, 0, 0, 27, 0, 0, 23, 0, 0, + 19, 0, 0, 15, 0, 0, 0, 0, 11, 0, 0, 7, 0, 0, 3, 0, 0, + ]; + + /// + /// Predicts an 8-bit directional block using the widest available SIMD path. + /// + /// The destination block origin. + /// The destination row stride in samples. + /// The predicted block dimensions. + /// The prepared top reference, including any required extension. + /// The prepared left reference, including any required extension. + /// Whether the top edge contains half-sample positions. + /// Whether the left edge contains half-sample positions. + /// The adjusted prediction angle. + /// The caller-owned block transposition workspace. + public static void Predict(Span destination, int destinationStride, Av1TransformSize transformSize, ReadOnlySpan above, ReadOnlySpan left, bool upsampleAbove, bool upsampleLeft, int angle, Span scratch) { - // Zone-two left references are not contiguous across output columns. Constructing the four source pairs - // directly avoids a temporary gather-index buffer and keeps the scalar continuation at the same offset. - int oneVectorFromEnd = destination.Length - 4; - for (; index <= oneVectorFromEnd; index += 4) - { - int projection0 = projection - (index * derivative); - int projection1 = projection0 - derivative; - int projection2 = projection1 - derivative; - int projection3 = projection2 - derivative; - int basis0 = projection0 >> fractionBits; - int basis1 = projection1 >> fractionBits; - int basis2 = projection2 >> fractionBits; - int basis3 = projection3 >> fractionBits; - Vector128 source0 = Vector128.Create((int)Unsafe.Add(ref leftBase, basis0), Unsafe.Add(ref leftBase, basis1), Unsafe.Add(ref leftBase, basis2), Unsafe.Add(ref leftBase, basis3)); - Vector128 source1 = Vector128.Create((int)Unsafe.Add(ref leftBase, basis0 + 1), Unsafe.Add(ref leftBase, basis1 + 1), Unsafe.Add(ref leftBase, basis2 + 1), Unsafe.Add(ref leftBase, basis3 + 1)); - Vector128 weights = Vector128.Create( - ((projection0 << upsampleShift) & 0x3F) >> 1, - ((projection1 << upsampleShift) & 0x3F) >> 1, - ((projection2 << upsampleShift) & 0x3F) >> 1, - ((projection3 << upsampleShift) & 0x3F) >> 1); - - StoreFourBytes(Interpolate(source0, source1, weights), ref Unsafe.Add(ref destinationBase, index)); + int width = transformSize.GetWidth(); + int height = transformSize.GetHeight(); + + if (angle is > 0 and < 90) + { + PredictZone1(destination, destinationStride, above, upsampleAbove, GetDeltaX(angle), width, height); + } + else if (angle is > 90 and < 180) + { + PredictZone2(destination, destinationStride, above, left, upsampleAbove, upsampleLeft, GetDeltaX(angle), GetDeltaY(angle), width, height); + } + else if (angle is > 180 and < 270) + { + // libaom computes zone 3 as a zone 1 block with swapped dimensions, then transposes it. This preserves + // contiguous reference reads and destination stores in both hot stages instead of scattering columns. + Span transposed = scratch[..(width * height)]; + PredictZone1(transposed, height, left, upsampleLeft, GetDeltaY(angle), height, width); + Transpose(transposed, destination, height, width, destinationStride); + } + else + { + Av1PredictionMode mode = angle == 90 ? Av1PredictionMode.Vertical : Av1PredictionMode.Horizontal; + Av1IntraPredictorBase.GetPredictor(mode).Predict(destination, destinationStride, above, left, width, height); } } - for (; index < destination.Length; index++) + /// + /// Predicts a high-bit-depth directional block using the widest available SIMD path. + /// + /// The destination block origin. + /// The destination row stride in samples. + /// The predicted block dimensions. + /// The prepared top reference, including any required extension. + /// The prepared left reference, including any required extension. + /// Whether the top edge contains half-sample positions. + /// Whether the left edge contains half-sample positions. + /// The adjusted prediction angle. + /// The caller-owned block transposition workspace. + public static void Predict(Span destination, int destinationStride, Av1TransformSize transformSize, ReadOnlySpan above, ReadOnlySpan left, bool upsampleAbove, bool upsampleLeft, int angle, Span scratch) { - int currentProjection = projection - (index * derivative); - int basis = currentProjection >> fractionBits; - int weight = ((currentProjection << upsampleShift) & 0x3F) >> 1; - Unsafe.Add(ref destinationBase, index) = (byte)(((Unsafe.Add(ref leftBase, basis) * (32 - weight)) + (Unsafe.Add(ref leftBase, basis + 1) * weight) + 16) >> 5); - } - } + int width = transformSize.GetWidth(); + int height = transformSize.GetHeight(); - /// - /// Interpolates the left-edge prefix of one high-bit-depth zone 2 row. - /// - /// The destination prefix. - /// The projected left reference. - /// The first Q6 left projection. - /// The Q8 derivative subtracted between columns. - /// Whether the left reference contains half-sample positions. - private static void InterpolateLeft(Span destination, ReadOnlySpan left, int projection, int derivative, bool upsample) - { - ref short destinationBase = ref MemoryMarshal.GetReference(destination); - ref short leftBase = ref MemoryMarshal.GetReference(left); - int upsampleShift = upsample ? 1 : 0; - int fractionBits = 6 - upsampleShift; - int index = 0; + if (angle is > 0 and < 90) + { + PredictZone1(destination, destinationStride, above, upsampleAbove, GetDeltaX(angle), width, height); + } + else if (angle is > 90 and < 180) + { + PredictZone2(destination, destinationStride, above, left, upsampleAbove, upsampleLeft, GetDeltaX(angle), GetDeltaY(angle), width, height); + } + else if (angle is > 180 and < 270) + { + Span transposed = scratch[..(width * height)]; + PredictZone1(transposed, height, left, upsampleLeft, GetDeltaY(angle), height, width); + Transpose(transposed, destination, height, width, destinationStride); + } + else + { + Av1PredictionMode mode = angle == 90 ? Av1PredictionMode.Vertical : Av1PredictionMode.Horizontal; + Av1IntraPredictorBase.GetPredictor(mode).Predict(destination, destinationStride, above, left, width, height); + } + } - if (Vector128.IsHardwareAccelerated) + /// + /// Predicts an 8-bit directional block without hardware intrinsics. + /// + /// The destination block origin. + /// The destination row stride in samples. + /// The predicted block dimensions. + /// The prepared top reference, including any required extension. + /// The prepared left reference, including any required extension. + /// Whether the top edge contains half-sample positions. + /// Whether the left edge contains half-sample positions. + /// The adjusted prediction angle. + public static void PredictScalar(Span destination, int destinationStride, Av1TransformSize transformSize, ReadOnlySpan above, ReadOnlySpan left, bool upsampleAbove, bool upsampleLeft, int angle) { - int oneVectorFromEnd = destination.Length - 4; - for (; index <= oneVectorFromEnd; index += 4) - { - int projection0 = projection - (index * derivative); - int projection1 = projection0 - derivative; - int projection2 = projection1 - derivative; - int projection3 = projection2 - derivative; - int basis0 = projection0 >> fractionBits; - int basis1 = projection1 >> fractionBits; - int basis2 = projection2 >> fractionBits; - int basis3 = projection3 >> fractionBits; - Vector128 source0 = Vector128.Create((int)Unsafe.Add(ref leftBase, basis0), Unsafe.Add(ref leftBase, basis1), Unsafe.Add(ref leftBase, basis2), Unsafe.Add(ref leftBase, basis3)); - Vector128 source1 = Vector128.Create((int)Unsafe.Add(ref leftBase, basis0 + 1), Unsafe.Add(ref leftBase, basis1 + 1), Unsafe.Add(ref leftBase, basis2 + 1), Unsafe.Add(ref leftBase, basis3 + 1)); - Vector128 weights = Vector128.Create( - ((projection0 << upsampleShift) & 0x3F) >> 1, - ((projection1 << upsampleShift) & 0x3F) >> 1, - ((projection2 << upsampleShift) & 0x3F) >> 1, - ((projection3 << upsampleShift) & 0x3F) >> 1); - - StoreFourShorts(Interpolate(source0, source1, weights), ref Unsafe.Add(ref destinationBase, index)); + int width = transformSize.GetWidth(); + int height = transformSize.GetHeight(); + + if (angle is > 0 and < 90) + { + PredictZone1Scalar(destination, destinationStride, above, upsampleAbove, GetDeltaX(angle), width, height); + } + else if (angle is > 90 and < 180) + { + PredictZone2Scalar(destination, destinationStride, above, left, upsampleAbove, upsampleLeft, GetDeltaX(angle), GetDeltaY(angle), width, height); + } + else if (angle is > 180 and < 270) + { + PredictZone3Scalar(destination, destinationStride, left, upsampleLeft, GetDeltaY(angle), width, height); + } + else + { + Av1PredictionMode mode = angle == 90 ? Av1PredictionMode.Vertical : Av1PredictionMode.Horizontal; + Av1IntraPredictorBase.GetPredictor(mode).PredictScalar(destination, destinationStride, above, left, width, height); } } - for (; index < destination.Length; index++) + /// + /// Predicts a high-bit-depth directional block without hardware intrinsics. + /// + /// The destination block origin. + /// The destination row stride in samples. + /// The predicted block dimensions. + /// The prepared top reference, including any required extension. + /// The prepared left reference, including any required extension. + /// Whether the top edge contains half-sample positions. + /// Whether the left edge contains half-sample positions. + /// The adjusted prediction angle. + public static void PredictScalar(Span destination, int destinationStride, Av1TransformSize transformSize, ReadOnlySpan above, ReadOnlySpan left, bool upsampleAbove, bool upsampleLeft, int angle) { - int currentProjection = projection - (index * derivative); - int basis = currentProjection >> fractionBits; - int weight = ((currentProjection << upsampleShift) & 0x3F) >> 1; - Unsafe.Add(ref destinationBase, index) = (short)(((Unsafe.Add(ref leftBase, basis) * (32 - weight)) + (Unsafe.Add(ref leftBase, basis + 1) * weight) + 16) >> 5); + int width = transformSize.GetWidth(); + int height = transformSize.GetHeight(); + + if (angle is > 0 and < 90) + { + PredictZone1Scalar(destination, destinationStride, above, upsampleAbove, GetDeltaX(angle), width, height); + } + else if (angle is > 90 and < 180) + { + PredictZone2Scalar(destination, destinationStride, above, left, upsampleAbove, upsampleLeft, GetDeltaX(angle), GetDeltaY(angle), width, height); + } + else if (angle is > 180 and < 270) + { + PredictZone3Scalar(destination, destinationStride, left, upsampleLeft, GetDeltaY(angle), width, height); + } + else + { + Av1PredictionMode mode = angle == 90 ? Av1PredictionMode.Vertical : Av1PredictionMode.Horizontal; + Av1IntraPredictorBase.GetPredictor(mode).PredictScalar(destination, destinationStride, above, left, width, height); + } } - } - /// - /// Interpolates sixty-four pairs of 8-bit references. - /// - /// The left interpolation samples. - /// The right interpolation samples. - /// The right-sample weight. - /// The rounded interpolated samples. - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector512 Interpolate(Vector512 left, Vector512 right, int weight) - { - (Vector512 leftLow, Vector512 leftHigh) = Vector512.Widen(left); - (Vector512 rightLow, Vector512 rightHigh) = Vector512.Widen(right); - Vector512 rounding = Vector512.Create((ushort)16); - Vector512 low = ((leftLow * (ushort)(32 - weight)) + (rightLow * (ushort)weight) + rounding) >> 5; - Vector512 high = ((leftHigh * (ushort)(32 - weight)) + (rightHigh * (ushort)weight) + rounding) >> 5; - return Vector512.Narrow(low, high); - } + /// + /// Gets the horizontal Q8 projection derivative for an adjusted angle. + /// + /// The adjusted prediction angle. + /// The horizontal derivative, or one when the selected zone does not consume it. + public static int GetDeltaX(int angle) + => angle switch + { + > 0 and < 90 => DirectionalIntraDerivative[angle], + > 90 and < 180 => DirectionalIntraDerivative[180 - angle], + _ => 1, + }; + + /// + /// Gets the vertical Q8 projection derivative for an adjusted angle. + /// + /// The adjusted prediction angle. + /// The vertical derivative, or one when the selected zone does not consume it. + public static int GetDeltaY(int angle) + => angle switch + { + > 90 and < 180 => DirectionalIntraDerivative[angle - 90], + > 180 and < 270 => DirectionalIntraDerivative[270 - angle], + _ => 1, + }; + + /// + /// Predicts one 8-bit zone 1 block without hardware intrinsics. + /// + /// The destination block origin. + /// The destination row stride. + /// The projected top reference. + /// Whether the reference contains half-sample positions. + /// The Q8 projection derivative. + /// The block width. + /// The block height. + private static void PredictZone1Scalar(Span destination, int destinationStride, ReadOnlySpan above, bool upsample, int derivative, int width, int height) + { + int upsampleShift = upsample ? 1 : 0; + int maximumBasis = (width + height - 1) << upsampleShift; + int fractionBits = 6 - upsampleShift; + int basisIncrement = 1 << upsampleShift; + int projection = derivative; + ref byte aboveBase = ref Unsafe.AsRef(in above[0]); + + for (int row = 0; row < height; row++, projection += derivative) + { + int basis = projection >> fractionBits; + int weight = ((projection << upsampleShift) & 0x3F) >> 1; + ref byte destinationRow = ref destination[row * destinationStride]; - /// - /// Interpolates thirty-two pairs of 8-bit references. - /// - /// The left interpolation samples. - /// The right interpolation samples. - /// The right-sample weight. - /// The rounded interpolated samples. - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector256 Interpolate(Vector256 left, Vector256 right, int weight) - { - (Vector256 leftLow, Vector256 leftHigh) = Vector256.Widen(left); - (Vector256 rightLow, Vector256 rightHigh) = Vector256.Widen(right); - Vector256 rounding = Vector256.Create((ushort)16); - Vector256 low = ((leftLow * (ushort)(32 - weight)) + (rightLow * (ushort)weight) + rounding) >> 5; - Vector256 high = ((leftHigh * (ushort)(32 - weight)) + (rightHigh * (ushort)weight) + rounding) >> 5; - return Vector256.Narrow(low, high); - } + for (int column = 0; column < width; column++, basis += basisIncrement) + { + Unsafe.Add(ref destinationRow, column) = basis < maximumBasis + ? TOperator.Interpolate(Unsafe.Add(ref aboveBase, basis), Unsafe.Add(ref aboveBase, basis + 1), weight) + : Unsafe.Add(ref aboveBase, maximumBasis); + } + } + } - /// - /// Interpolates sixteen pairs of 8-bit references. - /// - /// The left interpolation samples. - /// The right interpolation samples. - /// The right-sample weight. - /// The rounded interpolated samples. - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector128 Interpolate(Vector128 left, Vector128 right, int weight) - { - (Vector128 leftLow, Vector128 leftHigh) = Vector128.Widen(left); - (Vector128 rightLow, Vector128 rightHigh) = Vector128.Widen(right); - Vector128 rounding = Vector128.Create((ushort)16); - Vector128 low = ((leftLow * (ushort)(32 - weight)) + (rightLow * (ushort)weight) + rounding) >> 5; - Vector128 high = ((leftHigh * (ushort)(32 - weight)) + (rightHigh * (ushort)weight) + rounding) >> 5; - return Vector128.Narrow(low, high); - } + /// + /// Predicts one high-bit-depth zone 1 block without hardware intrinsics. + /// + /// The destination block origin. + /// The destination row stride. + /// The projected top reference. + /// Whether the reference contains half-sample positions. + /// The Q8 projection derivative. + /// The block width. + /// The block height. + private static void PredictZone1Scalar(Span destination, int destinationStride, ReadOnlySpan above, bool upsample, int derivative, int width, int height) + { + int upsampleShift = upsample ? 1 : 0; + int maximumBasis = (width + height - 1) << upsampleShift; + int fractionBits = 6 - upsampleShift; + int basisIncrement = 1 << upsampleShift; + int projection = derivative; + ref short aboveBase = ref Unsafe.AsRef(in above[0]); + + for (int row = 0; row < height; row++, projection += derivative) + { + int basis = projection >> fractionBits; + int weight = ((projection << upsampleShift) & 0x3F) >> 1; + ref short destinationRow = ref destination[row * destinationStride]; - /// - /// Interpolates thirty-two pairs of high-bit-depth references. - /// - /// The left interpolation samples. - /// The right interpolation samples. - /// The right-sample weight. - /// The rounded interpolated samples. - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector512 Interpolate(Vector512 left, Vector512 right, int weight) - { - (Vector512 leftLow, Vector512 leftHigh) = Vector512.Widen(left); - (Vector512 rightLow, Vector512 rightHigh) = Vector512.Widen(right); - Vector512 rounding = Vector512.Create(16); - Vector512 low = ((leftLow * (32 - weight)) + (rightLow * weight) + rounding) >> 5; - Vector512 high = ((leftHigh * (32 - weight)) + (rightHigh * weight) + rounding) >> 5; - return Vector512.Narrow(low, high); - } + for (int column = 0; column < width; column++, basis += basisIncrement) + { + Unsafe.Add(ref destinationRow, column) = basis < maximumBasis + ? TOperator.Interpolate(Unsafe.Add(ref aboveBase, basis), Unsafe.Add(ref aboveBase, basis + 1), weight) + : Unsafe.Add(ref aboveBase, maximumBasis); + } + } + } - /// - /// Interpolates sixteen pairs of high-bit-depth references. - /// - /// The left interpolation samples. - /// The right interpolation samples. - /// The right-sample weight. - /// The rounded interpolated samples. - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector256 Interpolate(Vector256 left, Vector256 right, int weight) - { - (Vector256 leftLow, Vector256 leftHigh) = Vector256.Widen(left); - (Vector256 rightLow, Vector256 rightHigh) = Vector256.Widen(right); - Vector256 rounding = Vector256.Create(16); - Vector256 low = ((leftLow * (32 - weight)) + (rightLow * weight) + rounding) >> 5; - Vector256 high = ((leftHigh * (32 - weight)) + (rightHigh * weight) + rounding) >> 5; - return Vector256.Narrow(low, high); - } + /// + /// Predicts one 8-bit zone 2 block without hardware intrinsics. + /// + /// The destination block origin. + /// The destination row stride. + /// The projected top reference. + /// The projected left reference. + /// Whether the top reference contains half-sample positions. + /// Whether the left reference contains half-sample positions. + /// The horizontal Q8 derivative. + /// The vertical Q8 derivative. + /// The block width. + /// The block height. + private static void PredictZone2Scalar(Span destination, int destinationStride, ReadOnlySpan above, ReadOnlySpan left, bool upsampleAbove, bool upsampleLeft, int dx, int dy, int width, int height) + { + int aboveShift = upsampleAbove ? 1 : 0; + int leftShift = upsampleLeft ? 1 : 0; + int minimumTopBasis = -(1 << aboveShift); + int topFractionBits = 6 - aboveShift; + int leftFractionBits = 6 - leftShift; + int topBasisIncrement = 1 << aboveShift; + int topProjection = -dx; + ref byte aboveBase = ref Unsafe.AsRef(in above[0]); + ref byte leftBase = ref Unsafe.AsRef(in left[0]); + + for (int row = 0; row < height; row++, topProjection -= dx) + { + int topBasis = topProjection >> topFractionBits; + int topWeight = ((topProjection << aboveShift) & 0x3F) >> 1; + int leftProjection = (row << 6) - dy; + ref byte destinationRow = ref destination[row * destinationStride]; - /// - /// Interpolates eight pairs of high-bit-depth references. - /// - /// The left interpolation samples. - /// The right interpolation samples. - /// The right-sample weight. - /// The rounded interpolated samples. - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector128 Interpolate(Vector128 left, Vector128 right, int weight) - { - (Vector128 leftLow, Vector128 leftHigh) = Vector128.Widen(left); - (Vector128 rightLow, Vector128 rightHigh) = Vector128.Widen(right); - Vector128 rounding = Vector128.Create(16); - Vector128 low = ((leftLow * (32 - weight)) + (rightLow * weight) + rounding) >> 5; - Vector128 high = ((leftHigh * (32 - weight)) + (rightHigh * weight) + rounding) >> 5; - return Vector128.Narrow(low, high); - } + for (int column = 0; column < width; column++, topBasis += topBasisIncrement, leftProjection -= dy) + { + byte prediction; + if (topBasis >= minimumTopBasis) + { + prediction = TOperator.Interpolate(Unsafe.Add(ref aboveBase, topBasis), Unsafe.Add(ref aboveBase, topBasis + 1), topWeight); + } + else + { + int leftBasis = leftProjection >> leftFractionBits; + int leftWeight = ((leftProjection << leftShift) & 0x3F) >> 1; + prediction = TOperator.Interpolate(Unsafe.Add(ref leftBase, leftBasis), Unsafe.Add(ref leftBase, leftBasis + 1), leftWeight); + } + + Unsafe.Add(ref destinationRow, column) = prediction; + } + } + } - /// - /// Interpolates four widened reference pairs with independent weights. - /// - /// The left interpolation samples. - /// The right interpolation samples. - /// The right-sample weights. - /// The rounded interpolated samples. - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector128 Interpolate(Vector128 left, Vector128 right, Vector128 weights) - => ((left * (Vector128.Create(32) - weights)) + (right * weights) + Vector128.Create(16)) >> 5; + /// + /// Predicts one high-bit-depth zone 2 block without hardware intrinsics. + /// + /// The destination block origin. + /// The destination row stride. + /// The projected top reference. + /// The projected left reference. + /// Whether the top reference contains half-sample positions. + /// Whether the left reference contains half-sample positions. + /// The horizontal Q8 derivative. + /// The vertical Q8 derivative. + /// The block width. + /// The block height. + private static void PredictZone2Scalar(Span destination, int destinationStride, ReadOnlySpan above, ReadOnlySpan left, bool upsampleAbove, bool upsampleLeft, int dx, int dy, int width, int height) + { + int aboveShift = upsampleAbove ? 1 : 0; + int leftShift = upsampleLeft ? 1 : 0; + int minimumTopBasis = -(1 << aboveShift); + int topFractionBits = 6 - aboveShift; + int leftFractionBits = 6 - leftShift; + int topBasisIncrement = 1 << aboveShift; + int topProjection = -dx; + ref short aboveBase = ref Unsafe.AsRef(in above[0]); + ref short leftBase = ref Unsafe.AsRef(in left[0]); + + for (int row = 0; row < height; row++, topProjection -= dx) + { + int topBasis = topProjection >> topFractionBits; + int topWeight = ((topProjection << aboveShift) & 0x3F) >> 1; + int leftProjection = (row << 6) - dy; + ref short destinationRow = ref destination[row * destinationStride]; - /// - /// Stores four widened predictions as packed 8-bit samples. - /// - /// The widened predictions. - /// The first destination sample. - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static void StoreFourBytes(Vector128 prediction, ref byte destination) - { - Vector128 narrowed16 = Vector128.Narrow(prediction.AsUInt32(), Vector128.Zero); - Vector128 narrowed8 = Vector128.Narrow(narrowed16, Vector128.Zero); - Unsafe.As(ref destination) = narrowed8.AsUInt32().GetElement(0); - } + for (int column = 0; column < width; column++, topBasis += topBasisIncrement, leftProjection -= dy) + { + short prediction; + if (topBasis >= minimumTopBasis) + { + prediction = TOperator.Interpolate(Unsafe.Add(ref aboveBase, topBasis), Unsafe.Add(ref aboveBase, topBasis + 1), topWeight); + } + else + { + int leftBasis = leftProjection >> leftFractionBits; + int leftWeight = ((leftProjection << leftShift) & 0x3F) >> 1; + prediction = TOperator.Interpolate(Unsafe.Add(ref leftBase, leftBasis), Unsafe.Add(ref leftBase, leftBasis + 1), leftWeight); + } + + Unsafe.Add(ref destinationRow, column) = prediction; + } + } + } - /// - /// Stores four widened predictions as packed high-bit-depth samples. - /// - /// The widened predictions. - /// The first destination sample. - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static void StoreFourShorts(Vector128 prediction, ref short destination) - { - Vector128 narrowed = Vector128.Narrow(prediction, Vector128.Zero); - Unsafe.As(ref destination) = narrowed.AsUInt64().GetElement(0); + /// + /// Predicts one 8-bit zone 3 block without hardware intrinsics. + /// + /// The destination block origin. + /// The destination row stride. + /// The projected left reference. + /// Whether the reference contains half-sample positions. + /// The Q8 projection derivative. + /// The block width. + /// The block height. + private static void PredictZone3Scalar(Span destination, int destinationStride, ReadOnlySpan left, bool upsample, int derivative, int width, int height) + { + int upsampleShift = upsample ? 1 : 0; + int maximumBasis = (width + height - 1) << upsampleShift; + int fractionBits = 6 - upsampleShift; + int basisIncrement = 1 << upsampleShift; + int projection = derivative; + ref byte leftBase = ref Unsafe.AsRef(in left[0]); + + for (int column = 0; column < width; column++, projection += derivative) + { + int basis = projection >> fractionBits; + int weight = ((projection << upsampleShift) & 0x3F) >> 1; + for (int row = 0; row < height; row++, basis += basisIncrement) + { + destination[(row * destinationStride) + column] = basis < maximumBasis + ? TOperator.Interpolate(Unsafe.Add(ref leftBase, basis), Unsafe.Add(ref leftBase, basis + 1), weight) + : Unsafe.Add(ref leftBase, maximumBasis); + } + } + } + + /// + /// Predicts one high-bit-depth zone 3 block without hardware intrinsics. + /// + /// The destination block origin. + /// The destination row stride. + /// The projected left reference. + /// Whether the reference contains half-sample positions. + /// The Q8 projection derivative. + /// The block width. + /// The block height. + private static void PredictZone3Scalar(Span destination, int destinationStride, ReadOnlySpan left, bool upsample, int derivative, int width, int height) + { + int upsampleShift = upsample ? 1 : 0; + int maximumBasis = (width + height - 1) << upsampleShift; + int fractionBits = 6 - upsampleShift; + int basisIncrement = 1 << upsampleShift; + int projection = derivative; + ref short leftBase = ref Unsafe.AsRef(in left[0]); + + for (int column = 0; column < width; column++, projection += derivative) + { + int basis = projection >> fractionBits; + int weight = ((projection << upsampleShift) & 0x3F) >> 1; + for (int row = 0; row < height; row++, basis += basisIncrement) + { + destination[(row * destinationStride) + column] = basis < maximumBasis + ? TOperator.Interpolate(Unsafe.Add(ref leftBase, basis), Unsafe.Add(ref leftBase, basis + 1), weight) + : Unsafe.Add(ref leftBase, maximumBasis); + } + } + } } } diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1DirectionalIntraPredictor.Operator.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1DirectionalIntraPredictor.Operator.cs new file mode 100644 index 000000000..681560339 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1DirectionalIntraPredictor.Operator.cs @@ -0,0 +1,239 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.Intrinsics; +using SixLabors.ImageSharp.Formats.Heif.Av1.Transform; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; + +/// +/// Defines Q5 interpolation for AV1 directional intra prediction. +/// +internal static partial class Av1DirectionalIntraPredictor +{ + /// + /// The largest number of samples required to transpose a directional prediction block. + /// + public const int ScratchLength = 64 * 64; + + /// + /// Defines scalar and SIMD interpolation for AV1 directional intra prediction. + /// + internal interface IDirectionalPredictionOperator + { + /// + /// Interpolates one 8-bit pair. + /// + /// The first reference sample. + /// The second reference sample. + /// The Q5 weight of . + /// The interpolated sample. + public static abstract byte Interpolate(byte left, byte right, int weight); + + /// + /// Interpolates one high-bit-depth pair. + /// + /// The first reference sample. + /// The second reference sample. + /// The Q5 weight of . + /// The interpolated sample. + public static abstract short Interpolate(short left, short right, int weight); + + /// + /// Interpolates sixteen 8-bit pairs. + /// + /// The first reference samples. + /// The second reference samples. + /// The Q5 weight of . + /// The interpolated samples. + public static abstract Vector128 Interpolate(Vector128 left, Vector128 right, int weight); + + /// + /// Interpolates thirty-two 8-bit pairs. + /// + /// The first reference samples. + /// The second reference samples. + /// The Q5 weight of . + /// The interpolated samples. + public static abstract Vector256 Interpolate(Vector256 left, Vector256 right, int weight); + + /// + /// Interpolates sixty-four 8-bit pairs. + /// + /// The first reference samples. + /// The second reference samples. + /// The Q5 weight of . + /// The interpolated samples. + public static abstract Vector512 Interpolate(Vector512 left, Vector512 right, int weight); + + /// + /// Interpolates eight high-bit-depth pairs. + /// + /// The first reference samples. + /// The second reference samples. + /// The Q5 weight of . + /// The interpolated samples. + public static abstract Vector128 Interpolate(Vector128 left, Vector128 right, int weight); + + /// + /// Interpolates sixteen high-bit-depth pairs. + /// + /// The first reference samples. + /// The second reference samples. + /// The Q5 weight of . + /// The interpolated samples. + public static abstract Vector256 Interpolate(Vector256 left, Vector256 right, int weight); + + /// + /// Interpolates thirty-two high-bit-depth pairs. + /// + /// The first reference samples. + /// The second reference samples. + /// The Q5 weight of . + /// The interpolated samples. + public static abstract Vector512 Interpolate(Vector512 left, Vector512 right, int weight); + + /// + /// Interpolates four widened pairs with independent weights. + /// + /// The first reference samples. + /// The second reference samples. + /// The Q5 weights of . + /// The interpolated samples. + public static abstract Vector128 Interpolate(Vector128 left, Vector128 right, Vector128 weights); + } + + /// + /// Gets the horizontal Q8 projection derivative for an adjusted angle. + /// + public static int GetDeltaX(int angle) => Predictor.GetDeltaX(angle); + + /// + /// Gets the vertical Q8 projection derivative for an adjusted angle. + /// + public static int GetDeltaY(int angle) => Predictor.GetDeltaY(angle); + + /// + /// Predicts an 8-bit directional block. + /// + public static void Predict(Span destination, int destinationStride, Av1TransformSize transformSize, ReadOnlySpan above, ReadOnlySpan left, bool upsampleAbove, bool upsampleLeft, int angle, Span scratch) + => Predictor.Predict(destination, destinationStride, transformSize, above, left, upsampleAbove, upsampleLeft, angle, scratch); + + /// + /// Predicts a high-bit-depth directional block. + /// + public static void Predict(Span destination, int destinationStride, Av1TransformSize transformSize, ReadOnlySpan above, ReadOnlySpan left, bool upsampleAbove, bool upsampleLeft, int angle, Span scratch) + => Predictor.Predict(destination, destinationStride, transformSize, above, left, upsampleAbove, upsampleLeft, angle, scratch); + + /// + /// Predicts an 8-bit directional block without hardware intrinsics. + /// + public static void PredictScalar(Span destination, int destinationStride, Av1TransformSize transformSize, ReadOnlySpan above, ReadOnlySpan left, bool upsampleAbove, bool upsampleLeft, int angle) + => Predictor.PredictScalar(destination, destinationStride, transformSize, above, left, upsampleAbove, upsampleLeft, angle); + + /// + /// Predicts a high-bit-depth directional block without hardware intrinsics. + /// + public static void PredictScalar(Span destination, int destinationStride, Av1TransformSize transformSize, ReadOnlySpan above, ReadOnlySpan left, bool upsampleAbove, bool upsampleLeft, int angle) + => Predictor.PredictScalar(destination, destinationStride, transformSize, above, left, upsampleAbove, upsampleLeft, angle); + + /// + /// Interpolates projected neighboring samples for all directional prediction zones. + /// + internal readonly struct DirectionalOperator : IDirectionalPredictionOperator + { + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static byte Interpolate(byte left, byte right, int weight) + => (byte)(((left * (32 - weight)) + (right * weight) + 16) >> 5); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static short Interpolate(short left, short right, int weight) + => (short)(((left * (32 - weight)) + (right * weight) + 16) >> 5); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector128 Interpolate(Vector128 left, Vector128 right, int weight) + { + (Vector128 leftLow, Vector128 leftHigh) = Vector128.Widen(left); + (Vector128 rightLow, Vector128 rightHigh) = Vector128.Widen(right); + Vector128 rounding = Vector128.Create((ushort)16); + Vector128 low = ((leftLow * (ushort)(32 - weight)) + (rightLow * (ushort)weight) + rounding) >> 5; + Vector128 high = ((leftHigh * (ushort)(32 - weight)) + (rightHigh * (ushort)weight) + rounding) >> 5; + + return Vector128.Narrow(low, high); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector256 Interpolate(Vector256 left, Vector256 right, int weight) + { + (Vector256 leftLow, Vector256 leftHigh) = Vector256.Widen(left); + (Vector256 rightLow, Vector256 rightHigh) = Vector256.Widen(right); + Vector256 rounding = Vector256.Create((ushort)16); + Vector256 low = ((leftLow * (ushort)(32 - weight)) + (rightLow * (ushort)weight) + rounding) >> 5; + Vector256 high = ((leftHigh * (ushort)(32 - weight)) + (rightHigh * (ushort)weight) + rounding) >> 5; + + return Vector256.Narrow(low, high); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector512 Interpolate(Vector512 left, Vector512 right, int weight) + { + (Vector512 leftLow, Vector512 leftHigh) = Vector512.Widen(left); + (Vector512 rightLow, Vector512 rightHigh) = Vector512.Widen(right); + Vector512 rounding = Vector512.Create((ushort)16); + Vector512 low = ((leftLow * (ushort)(32 - weight)) + (rightLow * (ushort)weight) + rounding) >> 5; + Vector512 high = ((leftHigh * (ushort)(32 - weight)) + (rightHigh * (ushort)weight) + rounding) >> 5; + + return Vector512.Narrow(low, high); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector128 Interpolate(Vector128 left, Vector128 right, int weight) + { + (Vector128 leftLow, Vector128 leftHigh) = Vector128.Widen(left); + (Vector128 rightLow, Vector128 rightHigh) = Vector128.Widen(right); + Vector128 rounding = Vector128.Create(16); + Vector128 low = ((leftLow * (32 - weight)) + (rightLow * weight) + rounding) >> 5; + Vector128 high = ((leftHigh * (32 - weight)) + (rightHigh * weight) + rounding) >> 5; + + return Vector128.Narrow(low, high); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector256 Interpolate(Vector256 left, Vector256 right, int weight) + { + (Vector256 leftLow, Vector256 leftHigh) = Vector256.Widen(left); + (Vector256 rightLow, Vector256 rightHigh) = Vector256.Widen(right); + Vector256 rounding = Vector256.Create(16); + Vector256 low = ((leftLow * (32 - weight)) + (rightLow * weight) + rounding) >> 5; + Vector256 high = ((leftHigh * (32 - weight)) + (rightHigh * weight) + rounding) >> 5; + + return Vector256.Narrow(low, high); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector512 Interpolate(Vector512 left, Vector512 right, int weight) + { + (Vector512 leftLow, Vector512 leftHigh) = Vector512.Widen(left); + (Vector512 rightLow, Vector512 rightHigh) = Vector512.Widen(right); + Vector512 rounding = Vector512.Create(16); + Vector512 low = ((leftLow * (32 - weight)) + (rightLow * weight) + rounding) >> 5; + Vector512 high = ((leftHigh * (32 - weight)) + (rightHigh * weight) + rounding) >> 5; + + return Vector512.Narrow(low, high); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector128 Interpolate(Vector128 left, Vector128 right, Vector128 weights) + => ((left * (Vector128.Create(32) - weights)) + (right * weights) + Vector128.Create(16)) >> 5; + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1DirectionalIntraPredictor.Transpose.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1DirectionalIntraPredictor.Transpose.cs index 06b6c530b..96e1b9b48 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1DirectionalIntraPredictor.Transpose.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1DirectionalIntraPredictor.Transpose.cs @@ -17,303 +17,310 @@ namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; internal static partial class Av1DirectionalIntraPredictor { /// - /// Transposes a contiguous 8-bit prediction block into the reconstructed destination. + /// Implements the directional traversal for one closed interpolation operator. /// - /// The contiguous source block. - /// The destination block origin. - /// The source row width. - /// The number of source rows. - /// The destination row stride. - private static void Transpose(ReadOnlySpan source, Span destination, int sourceWidth, int sourceHeight, int destinationStride) + private static partial class Predictor + where TOperator : struct, IDirectionalPredictionOperator { - if (Vector128.IsHardwareAccelerated) + /// + /// Transposes a contiguous 8-bit prediction block into the reconstructed destination. + /// + /// The contiguous source block. + /// The destination block origin. + /// The source row width. + /// The number of source rows. + /// The destination row stride. + private static void Transpose(ReadOnlySpan source, Span destination, int sourceWidth, int sourceHeight, int destinationStride) { - // Selecting one tile size for the complete block keeps both loop increments aligned with the AV1 block - // dimensions. No partial SIMD tile reaches a neighboring prediction block. - int tileSize = sourceWidth >= 8 && sourceHeight >= 8 ? 8 : 4; - for (int y = 0; y < sourceHeight; y += tileSize) + if (Vector128.IsHardwareAccelerated) { - for (int x = 0; x < sourceWidth; x += tileSize) + // Selecting one tile size for the complete block keeps both loop increments aligned with the AV1 block + // dimensions. No partial SIMD tile reaches a neighboring prediction block. + int tileSize = sourceWidth >= 8 && sourceHeight >= 8 ? 8 : 4; + for (int y = 0; y < sourceHeight; y += tileSize) { - if (tileSize == 8) + for (int x = 0; x < sourceWidth; x += tileSize) { - Transpose8x8(source, destination, sourceWidth, destinationStride, x, y); - } - else - { - Transpose4x4(source, destination, sourceWidth, destinationStride, x, y); + if (tileSize == 8) + { + Transpose8x8(source, destination, sourceWidth, destinationStride, x, y); + } + else + { + Transpose4x4(source, destination, sourceWidth, destinationStride, x, y); + } } } - } - return; - } + return; + } - for (int y = 0; y < sourceHeight; y++) - { - for (int x = 0; x < sourceWidth; x++) + for (int y = 0; y < sourceHeight; y++) { - destination[(x * destinationStride) + y] = source[(y * sourceWidth) + x]; + for (int x = 0; x < sourceWidth; x++) + { + destination[(x * destinationStride) + y] = source[(y * sourceWidth) + x]; + } } } - } - /// - /// Transposes a contiguous high-bit-depth prediction block into the reconstructed destination. - /// - /// The contiguous source block. - /// The destination block origin. - /// The source row width. - /// The number of source rows. - /// The destination row stride. - private static void Transpose(ReadOnlySpan source, Span destination, int sourceWidth, int sourceHeight, int destinationStride) - { - if (Vector128.IsHardwareAccelerated) + /// + /// Transposes a contiguous high-bit-depth prediction block into the reconstructed destination. + /// + /// The contiguous source block. + /// The destination block origin. + /// The source row width. + /// The number of source rows. + /// The destination row stride. + private static void Transpose(ReadOnlySpan source, Span destination, int sourceWidth, int sourceHeight, int destinationStride) { - // The same tiling invariant applies to two-byte samples; only the register unpack granularity differs. - int tileSize = sourceWidth >= 8 && sourceHeight >= 8 ? 8 : 4; - for (int y = 0; y < sourceHeight; y += tileSize) + if (Vector128.IsHardwareAccelerated) { - for (int x = 0; x < sourceWidth; x += tileSize) + // The same tiling invariant applies to two-byte samples; only the register unpack granularity differs. + int tileSize = sourceWidth >= 8 && sourceHeight >= 8 ? 8 : 4; + for (int y = 0; y < sourceHeight; y += tileSize) { - if (tileSize == 8) + for (int x = 0; x < sourceWidth; x += tileSize) { - Transpose8x8(source, destination, sourceWidth, destinationStride, x, y); - } - else - { - Transpose4x4(source, destination, sourceWidth, destinationStride, x, y); + if (tileSize == 8) + { + Transpose8x8(source, destination, sourceWidth, destinationStride, x, y); + } + else + { + Transpose4x4(source, destination, sourceWidth, destinationStride, x, y); + } } } - } - return; - } + return; + } - for (int y = 0; y < sourceHeight; y++) - { - for (int x = 0; x < sourceWidth; x++) + for (int y = 0; y < sourceHeight; y++) { - destination[(x * destinationStride) + y] = source[(y * sourceWidth) + x]; + for (int x = 0; x < sourceWidth; x++) + { + destination[(x * destinationStride) + y] = source[(y * sourceWidth) + x]; + } } } - } - /// - /// Transposes one eight-by-eight tile of 8-bit prediction samples. - /// - /// The contiguous source block. - /// The destination block origin. - /// The source row stride. - /// The destination row stride. - /// The tile X coordinate. - /// The tile Y coordinate. - private static void Transpose8x8(ReadOnlySpan source, Span destination, int sourceStride, int destinationStride, int x, int y) - { - ref byte sourceBase = ref MemoryMarshal.GetReference(source); - ref byte destinationBase = ref MemoryMarshal.GetReference(destination); - Vector128 row0 = LoadEightBytes(ref sourceBase, ((y + 0) * sourceStride) + x); - Vector128 row1 = LoadEightBytes(ref sourceBase, ((y + 1) * sourceStride) + x); - Vector128 row2 = LoadEightBytes(ref sourceBase, ((y + 2) * sourceStride) + x); - Vector128 row3 = LoadEightBytes(ref sourceBase, ((y + 3) * sourceStride) + x); - Vector128 row4 = LoadEightBytes(ref sourceBase, ((y + 4) * sourceStride) + x); - Vector128 row5 = LoadEightBytes(ref sourceBase, ((y + 5) * sourceStride) + x); - Vector128 row6 = LoadEightBytes(ref sourceBase, ((y + 6) * sourceStride) + x); - Vector128 row7 = LoadEightBytes(ref sourceBase, ((y + 7) * sourceStride) + x); + /// + /// Transposes one eight-by-eight tile of 8-bit prediction samples. + /// + /// The contiguous source block. + /// The destination block origin. + /// The source row stride. + /// The destination row stride. + /// The tile X coordinate. + /// The tile Y coordinate. + private static void Transpose8x8(ReadOnlySpan source, Span destination, int sourceStride, int destinationStride, int x, int y) + { + ref byte sourceBase = ref MemoryMarshal.GetReference(source); + ref byte destinationBase = ref MemoryMarshal.GetReference(destination); + Vector128 row0 = LoadEightBytes(ref sourceBase, ((y + 0) * sourceStride) + x); + Vector128 row1 = LoadEightBytes(ref sourceBase, ((y + 1) * sourceStride) + x); + Vector128 row2 = LoadEightBytes(ref sourceBase, ((y + 2) * sourceStride) + x); + Vector128 row3 = LoadEightBytes(ref sourceBase, ((y + 3) * sourceStride) + x); + Vector128 row4 = LoadEightBytes(ref sourceBase, ((y + 4) * sourceStride) + x); + Vector128 row5 = LoadEightBytes(ref sourceBase, ((y + 5) * sourceStride) + x); + Vector128 row6 = LoadEightBytes(ref sourceBase, ((y + 6) * sourceStride) + x); + Vector128 row7 = LoadEightBytes(ref sourceBase, ((y + 7) * sourceStride) + x); - // Three unpack stages exchange one, two, then four byte coordinates. Each final vector contains two - // complete source columns, which are written as two contiguous eight-byte destination rows. - Vector128 pair0 = Vector128_.UnpackLow(row0, row1); - Vector128 pair2 = Vector128_.UnpackLow(row2, row3); - Vector128 pair4 = Vector128_.UnpackLow(row4, row5); - Vector128 pair6 = Vector128_.UnpackLow(row6, row7); - Vector128 quad0 = Vector128_.UnpackLow(pair0.AsInt16(), pair2.AsInt16()); - Vector128 quad1 = Vector128_.UnpackHigh(pair0.AsInt16(), pair2.AsInt16()); - Vector128 quad4 = Vector128_.UnpackLow(pair4.AsInt16(), pair6.AsInt16()); - Vector128 quad5 = Vector128_.UnpackHigh(pair4.AsInt16(), pair6.AsInt16()); - Vector128 columns01 = Vector128_.UnpackLow(quad0.AsInt32(), quad4.AsInt32()); - Vector128 columns23 = Vector128_.UnpackHigh(quad0.AsInt32(), quad4.AsInt32()); - Vector128 columns45 = Vector128_.UnpackLow(quad1.AsInt32(), quad5.AsInt32()); - Vector128 columns67 = Vector128_.UnpackHigh(quad1.AsInt32(), quad5.AsInt32()); + // Three unpack stages exchange one, two, then four byte coordinates. Each final vector contains two + // complete source columns, which are written as two contiguous eight-byte destination rows. + Vector128 pair0 = Vector128_.UnpackLow(row0, row1); + Vector128 pair2 = Vector128_.UnpackLow(row2, row3); + Vector128 pair4 = Vector128_.UnpackLow(row4, row5); + Vector128 pair6 = Vector128_.UnpackLow(row6, row7); + Vector128 quad0 = Vector128_.UnpackLow(pair0.AsInt16(), pair2.AsInt16()); + Vector128 quad1 = Vector128_.UnpackHigh(pair0.AsInt16(), pair2.AsInt16()); + Vector128 quad4 = Vector128_.UnpackLow(pair4.AsInt16(), pair6.AsInt16()); + Vector128 quad5 = Vector128_.UnpackHigh(pair4.AsInt16(), pair6.AsInt16()); + Vector128 columns01 = Vector128_.UnpackLow(quad0.AsInt32(), quad4.AsInt32()); + Vector128 columns23 = Vector128_.UnpackHigh(quad0.AsInt32(), quad4.AsInt32()); + Vector128 columns45 = Vector128_.UnpackLow(quad1.AsInt32(), quad5.AsInt32()); + Vector128 columns67 = Vector128_.UnpackHigh(quad1.AsInt32(), quad5.AsInt32()); - StoreEightBytes(columns01.AsUInt64().GetElement(0), ref destinationBase, ((x + 0) * destinationStride) + y); - StoreEightBytes(columns01.AsUInt64().GetElement(1), ref destinationBase, ((x + 1) * destinationStride) + y); - StoreEightBytes(columns23.AsUInt64().GetElement(0), ref destinationBase, ((x + 2) * destinationStride) + y); - StoreEightBytes(columns23.AsUInt64().GetElement(1), ref destinationBase, ((x + 3) * destinationStride) + y); - StoreEightBytes(columns45.AsUInt64().GetElement(0), ref destinationBase, ((x + 4) * destinationStride) + y); - StoreEightBytes(columns45.AsUInt64().GetElement(1), ref destinationBase, ((x + 5) * destinationStride) + y); - StoreEightBytes(columns67.AsUInt64().GetElement(0), ref destinationBase, ((x + 6) * destinationStride) + y); - StoreEightBytes(columns67.AsUInt64().GetElement(1), ref destinationBase, ((x + 7) * destinationStride) + y); - } + StoreEightBytes(columns01.AsUInt64().ToScalar(), ref destinationBase, ((x + 0) * destinationStride) + y); + StoreEightBytes(columns01.AsUInt64().GetElement(1), ref destinationBase, ((x + 1) * destinationStride) + y); + StoreEightBytes(columns23.AsUInt64().ToScalar(), ref destinationBase, ((x + 2) * destinationStride) + y); + StoreEightBytes(columns23.AsUInt64().GetElement(1), ref destinationBase, ((x + 3) * destinationStride) + y); + StoreEightBytes(columns45.AsUInt64().ToScalar(), ref destinationBase, ((x + 4) * destinationStride) + y); + StoreEightBytes(columns45.AsUInt64().GetElement(1), ref destinationBase, ((x + 5) * destinationStride) + y); + StoreEightBytes(columns67.AsUInt64().ToScalar(), ref destinationBase, ((x + 6) * destinationStride) + y); + StoreEightBytes(columns67.AsUInt64().GetElement(1), ref destinationBase, ((x + 7) * destinationStride) + y); + } - /// - /// Transposes one four-by-four tile of 8-bit prediction samples. - /// - /// The contiguous source block. - /// The destination block origin. - /// The source row stride. - /// The destination row stride. - /// The tile X coordinate. - /// The tile Y coordinate. - private static void Transpose4x4(ReadOnlySpan source, Span destination, int sourceStride, int destinationStride, int x, int y) - { - ref byte sourceBase = ref MemoryMarshal.GetReference(source); - ref byte destinationBase = ref MemoryMarshal.GetReference(destination); - Vector128 row0 = LoadFourBytes(ref sourceBase, ((y + 0) * sourceStride) + x); - Vector128 row1 = LoadFourBytes(ref sourceBase, ((y + 1) * sourceStride) + x); - Vector128 row2 = LoadFourBytes(ref sourceBase, ((y + 2) * sourceStride) + x); - Vector128 row3 = LoadFourBytes(ref sourceBase, ((y + 3) * sourceStride) + x); - Vector128 pair0 = Vector128_.UnpackLow(row0, row1); - Vector128 pair1 = Vector128_.UnpackLow(row2, row3); - Vector128 columns = Vector128_.UnpackLow(pair0.AsInt16(), pair1.AsInt16()); + /// + /// Transposes one four-by-four tile of 8-bit prediction samples. + /// + /// The contiguous source block. + /// The destination block origin. + /// The source row stride. + /// The destination row stride. + /// The tile X coordinate. + /// The tile Y coordinate. + private static void Transpose4x4(ReadOnlySpan source, Span destination, int sourceStride, int destinationStride, int x, int y) + { + ref byte sourceBase = ref MemoryMarshal.GetReference(source); + ref byte destinationBase = ref MemoryMarshal.GetReference(destination); + Vector128 row0 = LoadFourBytes(ref sourceBase, ((y + 0) * sourceStride) + x); + Vector128 row1 = LoadFourBytes(ref sourceBase, ((y + 1) * sourceStride) + x); + Vector128 row2 = LoadFourBytes(ref sourceBase, ((y + 2) * sourceStride) + x); + Vector128 row3 = LoadFourBytes(ref sourceBase, ((y + 3) * sourceStride) + x); + Vector128 pair0 = Vector128_.UnpackLow(row0, row1); + Vector128 pair1 = Vector128_.UnpackLow(row2, row3); + Vector128 columns = Vector128_.UnpackLow(pair0.AsInt16(), pair1.AsInt16()); - Vector128 packedColumns = columns.AsUInt32(); - StoreFourBytes(packedColumns.GetElement(0), ref destinationBase, ((x + 0) * destinationStride) + y); - StoreFourBytes(packedColumns.GetElement(1), ref destinationBase, ((x + 1) * destinationStride) + y); - StoreFourBytes(packedColumns.GetElement(2), ref destinationBase, ((x + 2) * destinationStride) + y); - StoreFourBytes(packedColumns.GetElement(3), ref destinationBase, ((x + 3) * destinationStride) + y); - } + Vector128 packedColumns = columns.AsUInt32(); + StoreFourBytes(packedColumns.GetElement(0), ref destinationBase, ((x + 0) * destinationStride) + y); + StoreFourBytes(packedColumns.GetElement(1), ref destinationBase, ((x + 1) * destinationStride) + y); + StoreFourBytes(packedColumns.GetElement(2), ref destinationBase, ((x + 2) * destinationStride) + y); + StoreFourBytes(packedColumns.GetElement(3), ref destinationBase, ((x + 3) * destinationStride) + y); + } - /// - /// Transposes one eight-by-eight tile of high-bit-depth prediction samples. - /// - /// The contiguous source block. - /// The destination block origin. - /// The source row stride. - /// The destination row stride. - /// The tile X coordinate. - /// The tile Y coordinate. - private static void Transpose8x8(ReadOnlySpan source, Span destination, int sourceStride, int destinationStride, int x, int y) - { - ref short sourceBase = ref MemoryMarshal.GetReference(source); - ref short destinationBase = ref MemoryMarshal.GetReference(destination); - Vector128 row0 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 0) * sourceStride) + x)); - Vector128 row1 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 1) * sourceStride) + x)); - Vector128 row2 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 2) * sourceStride) + x)); - Vector128 row3 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 3) * sourceStride) + x)); - Vector128 row4 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 4) * sourceStride) + x)); - Vector128 row5 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 5) * sourceStride) + x)); - Vector128 row6 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 6) * sourceStride) + x)); - Vector128 row7 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 7) * sourceStride) + x)); - Vector128 pair0 = Vector128_.UnpackLow(row0, row1); - Vector128 pair1 = Vector128_.UnpackHigh(row0, row1); - Vector128 pair2 = Vector128_.UnpackLow(row2, row3); - Vector128 pair3 = Vector128_.UnpackHigh(row2, row3); - Vector128 pair4 = Vector128_.UnpackLow(row4, row5); - Vector128 pair5 = Vector128_.UnpackHigh(row4, row5); - Vector128 pair6 = Vector128_.UnpackLow(row6, row7); - Vector128 pair7 = Vector128_.UnpackHigh(row6, row7); - Vector128 quad0 = Vector128_.UnpackLow(pair0.AsInt32(), pair2.AsInt32()); - Vector128 quad1 = Vector128_.UnpackHigh(pair0.AsInt32(), pair2.AsInt32()); - Vector128 quad2 = Vector128_.UnpackLow(pair1.AsInt32(), pair3.AsInt32()); - Vector128 quad3 = Vector128_.UnpackHigh(pair1.AsInt32(), pair3.AsInt32()); - Vector128 quad4 = Vector128_.UnpackLow(pair4.AsInt32(), pair6.AsInt32()); - Vector128 quad5 = Vector128_.UnpackHigh(pair4.AsInt32(), pair6.AsInt32()); - Vector128 quad6 = Vector128_.UnpackLow(pair5.AsInt32(), pair7.AsInt32()); - Vector128 quad7 = Vector128_.UnpackHigh(pair5.AsInt32(), pair7.AsInt32()); - Vector128 column0 = Vector128_.UnpackLow(quad0.AsInt64(), quad4.AsInt64()).AsInt16(); - Vector128 column1 = Vector128_.UnpackHigh(quad0.AsInt64(), quad4.AsInt64()).AsInt16(); - Vector128 column2 = Vector128_.UnpackLow(quad1.AsInt64(), quad5.AsInt64()).AsInt16(); - Vector128 column3 = Vector128_.UnpackHigh(quad1.AsInt64(), quad5.AsInt64()).AsInt16(); - Vector128 column4 = Vector128_.UnpackLow(quad2.AsInt64(), quad6.AsInt64()).AsInt16(); - Vector128 column5 = Vector128_.UnpackHigh(quad2.AsInt64(), quad6.AsInt64()).AsInt16(); - Vector128 column6 = Vector128_.UnpackLow(quad3.AsInt64(), quad7.AsInt64()).AsInt16(); - Vector128 column7 = Vector128_.UnpackHigh(quad3.AsInt64(), quad7.AsInt64()).AsInt16(); - column0.StoreUnsafe(ref destinationBase, (nuint)(((x + 0) * destinationStride) + y)); - column1.StoreUnsafe(ref destinationBase, (nuint)(((x + 1) * destinationStride) + y)); - column2.StoreUnsafe(ref destinationBase, (nuint)(((x + 2) * destinationStride) + y)); - column3.StoreUnsafe(ref destinationBase, (nuint)(((x + 3) * destinationStride) + y)); - column4.StoreUnsafe(ref destinationBase, (nuint)(((x + 4) * destinationStride) + y)); - column5.StoreUnsafe(ref destinationBase, (nuint)(((x + 5) * destinationStride) + y)); - column6.StoreUnsafe(ref destinationBase, (nuint)(((x + 6) * destinationStride) + y)); - column7.StoreUnsafe(ref destinationBase, (nuint)(((x + 7) * destinationStride) + y)); - } + /// + /// Transposes one eight-by-eight tile of high-bit-depth prediction samples. + /// + /// The contiguous source block. + /// The destination block origin. + /// The source row stride. + /// The destination row stride. + /// The tile X coordinate. + /// The tile Y coordinate. + private static void Transpose8x8(ReadOnlySpan source, Span destination, int sourceStride, int destinationStride, int x, int y) + { + ref short sourceBase = ref MemoryMarshal.GetReference(source); + ref short destinationBase = ref MemoryMarshal.GetReference(destination); + Vector128 row0 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 0) * sourceStride) + x)); + Vector128 row1 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 1) * sourceStride) + x)); + Vector128 row2 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 2) * sourceStride) + x)); + Vector128 row3 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 3) * sourceStride) + x)); + Vector128 row4 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 4) * sourceStride) + x)); + Vector128 row5 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 5) * sourceStride) + x)); + Vector128 row6 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 6) * sourceStride) + x)); + Vector128 row7 = Vector128.LoadUnsafe(ref sourceBase, (nuint)(((y + 7) * sourceStride) + x)); + Vector128 pair0 = Vector128_.UnpackLow(row0, row1); + Vector128 pair1 = Vector128_.UnpackHigh(row0, row1); + Vector128 pair2 = Vector128_.UnpackLow(row2, row3); + Vector128 pair3 = Vector128_.UnpackHigh(row2, row3); + Vector128 pair4 = Vector128_.UnpackLow(row4, row5); + Vector128 pair5 = Vector128_.UnpackHigh(row4, row5); + Vector128 pair6 = Vector128_.UnpackLow(row6, row7); + Vector128 pair7 = Vector128_.UnpackHigh(row6, row7); + Vector128 quad0 = Vector128_.UnpackLow(pair0.AsInt32(), pair2.AsInt32()); + Vector128 quad1 = Vector128_.UnpackHigh(pair0.AsInt32(), pair2.AsInt32()); + Vector128 quad2 = Vector128_.UnpackLow(pair1.AsInt32(), pair3.AsInt32()); + Vector128 quad3 = Vector128_.UnpackHigh(pair1.AsInt32(), pair3.AsInt32()); + Vector128 quad4 = Vector128_.UnpackLow(pair4.AsInt32(), pair6.AsInt32()); + Vector128 quad5 = Vector128_.UnpackHigh(pair4.AsInt32(), pair6.AsInt32()); + Vector128 quad6 = Vector128_.UnpackLow(pair5.AsInt32(), pair7.AsInt32()); + Vector128 quad7 = Vector128_.UnpackHigh(pair5.AsInt32(), pair7.AsInt32()); + Vector128 column0 = Vector128_.UnpackLow(quad0.AsInt64(), quad4.AsInt64()).AsInt16(); + Vector128 column1 = Vector128_.UnpackHigh(quad0.AsInt64(), quad4.AsInt64()).AsInt16(); + Vector128 column2 = Vector128_.UnpackLow(quad1.AsInt64(), quad5.AsInt64()).AsInt16(); + Vector128 column3 = Vector128_.UnpackHigh(quad1.AsInt64(), quad5.AsInt64()).AsInt16(); + Vector128 column4 = Vector128_.UnpackLow(quad2.AsInt64(), quad6.AsInt64()).AsInt16(); + Vector128 column5 = Vector128_.UnpackHigh(quad2.AsInt64(), quad6.AsInt64()).AsInt16(); + Vector128 column6 = Vector128_.UnpackLow(quad3.AsInt64(), quad7.AsInt64()).AsInt16(); + Vector128 column7 = Vector128_.UnpackHigh(quad3.AsInt64(), quad7.AsInt64()).AsInt16(); + column0.StoreUnsafe(ref destinationBase, (nuint)(((x + 0) * destinationStride) + y)); + column1.StoreUnsafe(ref destinationBase, (nuint)(((x + 1) * destinationStride) + y)); + column2.StoreUnsafe(ref destinationBase, (nuint)(((x + 2) * destinationStride) + y)); + column3.StoreUnsafe(ref destinationBase, (nuint)(((x + 3) * destinationStride) + y)); + column4.StoreUnsafe(ref destinationBase, (nuint)(((x + 4) * destinationStride) + y)); + column5.StoreUnsafe(ref destinationBase, (nuint)(((x + 5) * destinationStride) + y)); + column6.StoreUnsafe(ref destinationBase, (nuint)(((x + 6) * destinationStride) + y)); + column7.StoreUnsafe(ref destinationBase, (nuint)(((x + 7) * destinationStride) + y)); + } - /// - /// Transposes one four-by-four tile of high-bit-depth prediction samples. - /// - /// The contiguous source block. - /// The destination block origin. - /// The source row stride. - /// The destination row stride. - /// The tile X coordinate. - /// The tile Y coordinate. - private static void Transpose4x4(ReadOnlySpan source, Span destination, int sourceStride, int destinationStride, int x, int y) - { - ref short sourceBase = ref MemoryMarshal.GetReference(source); - ref short destinationBase = ref MemoryMarshal.GetReference(destination); - Vector128 row0 = LoadFourShorts(ref sourceBase, ((y + 0) * sourceStride) + x); - Vector128 row1 = LoadFourShorts(ref sourceBase, ((y + 1) * sourceStride) + x); - Vector128 row2 = LoadFourShorts(ref sourceBase, ((y + 2) * sourceStride) + x); - Vector128 row3 = LoadFourShorts(ref sourceBase, ((y + 3) * sourceStride) + x); - Vector128 pair0 = Vector128_.UnpackLow(row0, row1); - Vector128 pair1 = Vector128_.UnpackLow(row2, row3); - Vector128 columns01 = Vector128_.UnpackLow(pair0.AsInt32(), pair1.AsInt32()); - Vector128 columns23 = Vector128_.UnpackHigh(pair0.AsInt32(), pair1.AsInt32()); + /// + /// Transposes one four-by-four tile of high-bit-depth prediction samples. + /// + /// The contiguous source block. + /// The destination block origin. + /// The source row stride. + /// The destination row stride. + /// The tile X coordinate. + /// The tile Y coordinate. + private static void Transpose4x4(ReadOnlySpan source, Span destination, int sourceStride, int destinationStride, int x, int y) + { + ref short sourceBase = ref MemoryMarshal.GetReference(source); + ref short destinationBase = ref MemoryMarshal.GetReference(destination); + Vector128 row0 = LoadFourShorts(ref sourceBase, ((y + 0) * sourceStride) + x); + Vector128 row1 = LoadFourShorts(ref sourceBase, ((y + 1) * sourceStride) + x); + Vector128 row2 = LoadFourShorts(ref sourceBase, ((y + 2) * sourceStride) + x); + Vector128 row3 = LoadFourShorts(ref sourceBase, ((y + 3) * sourceStride) + x); + Vector128 pair0 = Vector128_.UnpackLow(row0, row1); + Vector128 pair1 = Vector128_.UnpackLow(row2, row3); + Vector128 columns01 = Vector128_.UnpackLow(pair0.AsInt32(), pair1.AsInt32()); + Vector128 columns23 = Vector128_.UnpackHigh(pair0.AsInt32(), pair1.AsInt32()); - StoreFourShorts(columns01.AsUInt64().GetElement(0), ref destinationBase, ((x + 0) * destinationStride) + y); - StoreFourShorts(columns01.AsUInt64().GetElement(1), ref destinationBase, ((x + 1) * destinationStride) + y); - StoreFourShorts(columns23.AsUInt64().GetElement(0), ref destinationBase, ((x + 2) * destinationStride) + y); - StoreFourShorts(columns23.AsUInt64().GetElement(1), ref destinationBase, ((x + 3) * destinationStride) + y); - } + StoreFourShorts(columns01.AsUInt64().ToScalar(), ref destinationBase, ((x + 0) * destinationStride) + y); + StoreFourShorts(columns01.AsUInt64().GetElement(1), ref destinationBase, ((x + 1) * destinationStride) + y); + StoreFourShorts(columns23.AsUInt64().ToScalar(), ref destinationBase, ((x + 2) * destinationStride) + y); + StoreFourShorts(columns23.AsUInt64().GetElement(1), ref destinationBase, ((x + 3) * destinationStride) + y); + } - /// - /// Loads eight bytes into the lower half of a vector without reading past a tile row. - /// - /// The source buffer origin. - /// The source offset. - /// The loaded bytes. - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector128 LoadEightBytes(ref byte source, int offset) - => Vector128.Create(Unsafe.As(ref Unsafe.Add(ref source, offset)), 0UL).AsByte(); + /// + /// Loads eight bytes into the lower half of a vector without reading past a tile row. + /// + /// The source buffer origin. + /// The source offset. + /// The loaded bytes. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector128 LoadEightBytes(ref byte source, int offset) + => Vector128.Create(Unsafe.As(ref Unsafe.Add(ref source, offset)), 0UL).AsByte(); - /// - /// Loads four bytes into the low vector lanes without reading past a tile row. - /// - /// The source buffer origin. - /// The source offset. - /// The loaded bytes. - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector128 LoadFourBytes(ref byte source, int offset) - => Vector128.Create(Unsafe.As(ref Unsafe.Add(ref source, offset)), 0U, 0U, 0U).AsByte(); + /// + /// Loads four bytes into the low vector lanes without reading past a tile row. + /// + /// The source buffer origin. + /// The source offset. + /// The loaded bytes. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector128 LoadFourBytes(ref byte source, int offset) + => Vector128.Create(Unsafe.As(ref Unsafe.Add(ref source, offset)), 0U, 0U, 0U).AsByte(); - /// - /// Loads four high-bit-depth samples into the lower half of a vector without reading past a tile row. - /// - /// The source buffer origin. - /// The source offset. - /// The loaded samples. - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector128 LoadFourShorts(ref short source, int offset) - => Vector128.Create(Unsafe.As(ref Unsafe.Add(ref source, offset)), 0UL).AsInt16(); + /// + /// Loads four high-bit-depth samples into the lower half of a vector without reading past a tile row. + /// + /// The source buffer origin. + /// The source offset. + /// The loaded samples. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector128 LoadFourShorts(ref short source, int offset) + => Vector128.Create(Unsafe.As(ref Unsafe.Add(ref source, offset)), 0UL).AsInt16(); - /// - /// Stores the lower eight bytes of a vector. - /// - /// The packed bytes. - /// The destination buffer origin. - /// The destination offset. - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static void StoreEightBytes(ulong source, ref byte destination, int offset) - => Unsafe.As(ref Unsafe.Add(ref destination, offset)) = source; + /// + /// Stores the lower eight bytes of a vector. + /// + /// The packed bytes. + /// The destination buffer origin. + /// The destination offset. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void StoreEightBytes(ulong source, ref byte destination, int offset) + => Unsafe.As(ref Unsafe.Add(ref destination, offset)) = source; - /// - /// Stores four bytes from the low vector lanes. - /// - /// The packed bytes. - /// The destination buffer origin. - /// The destination offset. - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static void StoreFourBytes(uint source, ref byte destination, int offset) - => Unsafe.As(ref Unsafe.Add(ref destination, offset)) = source; + /// + /// Stores four bytes from the low vector lanes. + /// + /// The packed bytes. + /// The destination buffer origin. + /// The destination offset. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void StoreFourBytes(uint source, ref byte destination, int offset) + => Unsafe.As(ref Unsafe.Add(ref destination, offset)) = source; - /// - /// Stores four high-bit-depth samples from the low vector lanes. - /// - /// The packed samples. - /// The destination buffer origin. - /// The destination offset. - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static void StoreFourShorts(ulong source, ref short destination, int offset) - => Unsafe.As(ref Unsafe.Add(ref destination, offset)) = source; + /// + /// Stores four high-bit-depth samples from the low vector lanes. + /// + /// The packed samples. + /// The destination buffer origin. + /// The destination offset. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void StoreFourShorts(ulong source, ref short destination, int offset) + => Unsafe.As(ref Unsafe.Add(ref destination, offset)) = source; + } } diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1DirectionalIntraPredictor.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1DirectionalIntraPredictor.cs deleted file mode 100644 index 1f07788da..000000000 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1DirectionalIntraPredictor.cs +++ /dev/null @@ -1,443 +0,0 @@ -// Copyright (c) Six Labors. -// Licensed under the Six Labors Split License. - -using System.Runtime.CompilerServices; -using SixLabors.ImageSharp.Formats.Heif.Av1.Transform; - -namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; - -/// -/// Reconstructs AV1 directional intra-prediction blocks from prepared neighboring samples. -/// -/// -/// The three projection zones implement the directional prediction process in section 7.11.2.4 of the AV1 specification. -/// -internal static partial class Av1DirectionalIntraPredictor -{ - /// - /// The largest number of samples required to transpose a directional prediction block. - /// - public const int ScratchLength = 64 * 64; - - /// - /// Gets the Q8 directional derivatives indexed by acute prediction angle. - /// - private static ReadOnlySpan DirectionalIntraDerivative => - [ - - // Zero entries represent angles which AV1 never signals. Direct indexing avoids a search or division in - // each directional block while retaining the exact fixed-point projections from the normative table. - 0, 0, 0, 1023, 0, 0, 547, 0, 0, 372, 0, 0, 0, 0, 273, 0, 0, 215, 0, 0, 178, 0, 0, - 151, 0, 0, 132, 0, 0, 116, 0, 0, 102, 0, 0, 0, 90, 0, 0, 80, 0, 0, 71, 0, 0, 64, 0, 0, - 57, 0, 0, 51, 0, 0, 45, 0, 0, 0, 40, 0, 0, 35, 0, 0, 31, 0, 0, 27, 0, 0, 23, 0, 0, - 19, 0, 0, 15, 0, 0, 0, 0, 11, 0, 0, 7, 0, 0, 3, 0, 0, - ]; - - /// - /// Predicts an 8-bit directional block using the widest available SIMD path. - /// - /// The destination block origin. - /// The destination row stride in samples. - /// The predicted block dimensions. - /// The prepared top reference, including any required extension. - /// The prepared left reference, including any required extension. - /// Whether the top edge contains half-sample positions. - /// Whether the left edge contains half-sample positions. - /// The adjusted prediction angle. - /// The caller-owned block transposition workspace. - public static void Predict(Span destination, int destinationStride, Av1TransformSize transformSize, ReadOnlySpan above, ReadOnlySpan left, bool upsampleAbove, bool upsampleLeft, int angle, Span scratch) - { - int width = transformSize.GetWidth(); - int height = transformSize.GetHeight(); - - if (angle is > 0 and < 90) - { - PredictZone1(destination, destinationStride, above, upsampleAbove, GetDeltaX(angle), width, height); - } - else if (angle is > 90 and < 180) - { - PredictZone2(destination, destinationStride, above, left, upsampleAbove, upsampleLeft, GetDeltaX(angle), GetDeltaY(angle), width, height); - } - else if (angle is > 180 and < 270) - { - // libaom computes zone 3 as a zone 1 block with swapped dimensions, then transposes it. This preserves - // contiguous reference reads and destination stores in both hot stages instead of scattering columns. - Span transposed = scratch[..(width * height)]; - PredictZone1(transposed, height, left, upsampleLeft, GetDeltaY(angle), height, width); - Transpose(transposed, destination, height, width, destinationStride); - } - else - { - Av1PredictionMode mode = angle == 90 ? Av1PredictionMode.Vertical : Av1PredictionMode.Horizontal; - Av1IntraPredictorBase.GetPredictor(mode).Predict(destination, destinationStride, above, left, width, height); - } - } - - /// - /// Predicts a high-bit-depth directional block using the widest available SIMD path. - /// - /// The destination block origin. - /// The destination row stride in samples. - /// The predicted block dimensions. - /// The prepared top reference, including any required extension. - /// The prepared left reference, including any required extension. - /// Whether the top edge contains half-sample positions. - /// Whether the left edge contains half-sample positions. - /// The adjusted prediction angle. - /// The caller-owned block transposition workspace. - public static void Predict(Span destination, int destinationStride, Av1TransformSize transformSize, ReadOnlySpan above, ReadOnlySpan left, bool upsampleAbove, bool upsampleLeft, int angle, Span scratch) - { - int width = transformSize.GetWidth(); - int height = transformSize.GetHeight(); - - if (angle is > 0 and < 90) - { - PredictZone1(destination, destinationStride, above, upsampleAbove, GetDeltaX(angle), width, height); - } - else if (angle is > 90 and < 180) - { - PredictZone2(destination, destinationStride, above, left, upsampleAbove, upsampleLeft, GetDeltaX(angle), GetDeltaY(angle), width, height); - } - else if (angle is > 180 and < 270) - { - Span transposed = scratch[..(width * height)]; - PredictZone1(transposed, height, left, upsampleLeft, GetDeltaY(angle), height, width); - Transpose(transposed, destination, height, width, destinationStride); - } - else - { - Av1PredictionMode mode = angle == 90 ? Av1PredictionMode.Vertical : Av1PredictionMode.Horizontal; - Av1IntraPredictorBase.GetPredictor(mode).Predict(destination, destinationStride, above, left, width, height); - } - } - - /// - /// Predicts an 8-bit directional block without hardware intrinsics. - /// - /// The destination block origin. - /// The destination row stride in samples. - /// The predicted block dimensions. - /// The prepared top reference, including any required extension. - /// The prepared left reference, including any required extension. - /// Whether the top edge contains half-sample positions. - /// Whether the left edge contains half-sample positions. - /// The adjusted prediction angle. - public static void PredictScalar(Span destination, int destinationStride, Av1TransformSize transformSize, ReadOnlySpan above, ReadOnlySpan left, bool upsampleAbove, bool upsampleLeft, int angle) - { - int width = transformSize.GetWidth(); - int height = transformSize.GetHeight(); - - if (angle is > 0 and < 90) - { - PredictZone1Scalar(destination, destinationStride, above, upsampleAbove, GetDeltaX(angle), width, height); - } - else if (angle is > 90 and < 180) - { - PredictZone2Scalar(destination, destinationStride, above, left, upsampleAbove, upsampleLeft, GetDeltaX(angle), GetDeltaY(angle), width, height); - } - else if (angle is > 180 and < 270) - { - PredictZone3Scalar(destination, destinationStride, left, upsampleLeft, GetDeltaY(angle), width, height); - } - else - { - Av1PredictionMode mode = angle == 90 ? Av1PredictionMode.Vertical : Av1PredictionMode.Horizontal; - Av1IntraPredictorBase.GetPredictor(mode).PredictScalar(destination, destinationStride, above, left, width, height); - } - } - - /// - /// Predicts a high-bit-depth directional block without hardware intrinsics. - /// - /// The destination block origin. - /// The destination row stride in samples. - /// The predicted block dimensions. - /// The prepared top reference, including any required extension. - /// The prepared left reference, including any required extension. - /// Whether the top edge contains half-sample positions. - /// Whether the left edge contains half-sample positions. - /// The adjusted prediction angle. - public static void PredictScalar(Span destination, int destinationStride, Av1TransformSize transformSize, ReadOnlySpan above, ReadOnlySpan left, bool upsampleAbove, bool upsampleLeft, int angle) - { - int width = transformSize.GetWidth(); - int height = transformSize.GetHeight(); - - if (angle is > 0 and < 90) - { - PredictZone1Scalar(destination, destinationStride, above, upsampleAbove, GetDeltaX(angle), width, height); - } - else if (angle is > 90 and < 180) - { - PredictZone2Scalar(destination, destinationStride, above, left, upsampleAbove, upsampleLeft, GetDeltaX(angle), GetDeltaY(angle), width, height); - } - else if (angle is > 180 and < 270) - { - PredictZone3Scalar(destination, destinationStride, left, upsampleLeft, GetDeltaY(angle), width, height); - } - else - { - Av1PredictionMode mode = angle == 90 ? Av1PredictionMode.Vertical : Av1PredictionMode.Horizontal; - Av1IntraPredictorBase.GetPredictor(mode).PredictScalar(destination, destinationStride, above, left, width, height); - } - } - - /// - /// Gets the horizontal Q8 projection derivative for an adjusted angle. - /// - /// The adjusted prediction angle. - /// The horizontal derivative, or one when the selected zone does not consume it. - public static int GetDeltaX(int angle) - => angle switch - { - > 0 and < 90 => DirectionalIntraDerivative[angle], - > 90 and < 180 => DirectionalIntraDerivative[180 - angle], - _ => 1, - }; - - /// - /// Gets the vertical Q8 projection derivative for an adjusted angle. - /// - /// The adjusted prediction angle. - /// The vertical derivative, or one when the selected zone does not consume it. - public static int GetDeltaY(int angle) - => angle switch - { - > 90 and < 180 => DirectionalIntraDerivative[angle - 90], - > 180 and < 270 => DirectionalIntraDerivative[270 - angle], - _ => 1, - }; - - /// - /// Predicts one 8-bit zone 1 block without hardware intrinsics. - /// - /// The destination block origin. - /// The destination row stride. - /// The projected top reference. - /// Whether the reference contains half-sample positions. - /// The Q8 projection derivative. - /// The block width. - /// The block height. - private static void PredictZone1Scalar(Span destination, int destinationStride, ReadOnlySpan above, bool upsample, int derivative, int width, int height) - { - int upsampleShift = upsample ? 1 : 0; - int maximumBasis = (width + height - 1) << upsampleShift; - int fractionBits = 6 - upsampleShift; - int basisIncrement = 1 << upsampleShift; - int projection = derivative; - ref byte aboveBase = ref Unsafe.AsRef(in above[0]); - - for (int row = 0; row < height; row++, projection += derivative) - { - int basis = projection >> fractionBits; - int weight = ((projection << upsampleShift) & 0x3F) >> 1; - ref byte destinationRow = ref destination[row * destinationStride]; - - for (int column = 0; column < width; column++, basis += basisIncrement) - { - Unsafe.Add(ref destinationRow, column) = basis < maximumBasis - ? (byte)(((Unsafe.Add(ref aboveBase, basis) * (32 - weight)) + (Unsafe.Add(ref aboveBase, basis + 1) * weight) + 16) >> 5) - : Unsafe.Add(ref aboveBase, maximumBasis); - } - } - } - - /// - /// Predicts one high-bit-depth zone 1 block without hardware intrinsics. - /// - /// The destination block origin. - /// The destination row stride. - /// The projected top reference. - /// Whether the reference contains half-sample positions. - /// The Q8 projection derivative. - /// The block width. - /// The block height. - private static void PredictZone1Scalar(Span destination, int destinationStride, ReadOnlySpan above, bool upsample, int derivative, int width, int height) - { - int upsampleShift = upsample ? 1 : 0; - int maximumBasis = (width + height - 1) << upsampleShift; - int fractionBits = 6 - upsampleShift; - int basisIncrement = 1 << upsampleShift; - int projection = derivative; - ref short aboveBase = ref Unsafe.AsRef(in above[0]); - - for (int row = 0; row < height; row++, projection += derivative) - { - int basis = projection >> fractionBits; - int weight = ((projection << upsampleShift) & 0x3F) >> 1; - ref short destinationRow = ref destination[row * destinationStride]; - - for (int column = 0; column < width; column++, basis += basisIncrement) - { - Unsafe.Add(ref destinationRow, column) = basis < maximumBasis - ? (short)(((Unsafe.Add(ref aboveBase, basis) * (32 - weight)) + (Unsafe.Add(ref aboveBase, basis + 1) * weight) + 16) >> 5) - : Unsafe.Add(ref aboveBase, maximumBasis); - } - } - } - - /// - /// Predicts one 8-bit zone 2 block without hardware intrinsics. - /// - /// The destination block origin. - /// The destination row stride. - /// The projected top reference. - /// The projected left reference. - /// Whether the top reference contains half-sample positions. - /// Whether the left reference contains half-sample positions. - /// The horizontal Q8 derivative. - /// The vertical Q8 derivative. - /// The block width. - /// The block height. - private static void PredictZone2Scalar(Span destination, int destinationStride, ReadOnlySpan above, ReadOnlySpan left, bool upsampleAbove, bool upsampleLeft, int dx, int dy, int width, int height) - { - int aboveShift = upsampleAbove ? 1 : 0; - int leftShift = upsampleLeft ? 1 : 0; - int minimumTopBasis = -(1 << aboveShift); - int topFractionBits = 6 - aboveShift; - int leftFractionBits = 6 - leftShift; - int topBasisIncrement = 1 << aboveShift; - int topProjection = -dx; - ref byte aboveBase = ref Unsafe.AsRef(in above[0]); - ref byte leftBase = ref Unsafe.AsRef(in left[0]); - - for (int row = 0; row < height; row++, topProjection -= dx) - { - int topBasis = topProjection >> topFractionBits; - int topWeight = ((topProjection << aboveShift) & 0x3F) >> 1; - int leftProjection = (row << 6) - dy; - ref byte destinationRow = ref destination[row * destinationStride]; - - for (int column = 0; column < width; column++, topBasis += topBasisIncrement, leftProjection -= dy) - { - int prediction; - if (topBasis >= minimumTopBasis) - { - prediction = (Unsafe.Add(ref aboveBase, topBasis) * (32 - topWeight)) + (Unsafe.Add(ref aboveBase, topBasis + 1) * topWeight); - } - else - { - int leftBasis = leftProjection >> leftFractionBits; - int leftWeight = ((leftProjection << leftShift) & 0x3F) >> 1; - prediction = (Unsafe.Add(ref leftBase, leftBasis) * (32 - leftWeight)) + (Unsafe.Add(ref leftBase, leftBasis + 1) * leftWeight); - } - - Unsafe.Add(ref destinationRow, column) = (byte)((prediction + 16) >> 5); - } - } - } - - /// - /// Predicts one high-bit-depth zone 2 block without hardware intrinsics. - /// - /// The destination block origin. - /// The destination row stride. - /// The projected top reference. - /// The projected left reference. - /// Whether the top reference contains half-sample positions. - /// Whether the left reference contains half-sample positions. - /// The horizontal Q8 derivative. - /// The vertical Q8 derivative. - /// The block width. - /// The block height. - private static void PredictZone2Scalar(Span destination, int destinationStride, ReadOnlySpan above, ReadOnlySpan left, bool upsampleAbove, bool upsampleLeft, int dx, int dy, int width, int height) - { - int aboveShift = upsampleAbove ? 1 : 0; - int leftShift = upsampleLeft ? 1 : 0; - int minimumTopBasis = -(1 << aboveShift); - int topFractionBits = 6 - aboveShift; - int leftFractionBits = 6 - leftShift; - int topBasisIncrement = 1 << aboveShift; - int topProjection = -dx; - ref short aboveBase = ref Unsafe.AsRef(in above[0]); - ref short leftBase = ref Unsafe.AsRef(in left[0]); - - for (int row = 0; row < height; row++, topProjection -= dx) - { - int topBasis = topProjection >> topFractionBits; - int topWeight = ((topProjection << aboveShift) & 0x3F) >> 1; - int leftProjection = (row << 6) - dy; - ref short destinationRow = ref destination[row * destinationStride]; - - for (int column = 0; column < width; column++, topBasis += topBasisIncrement, leftProjection -= dy) - { - int prediction; - if (topBasis >= minimumTopBasis) - { - prediction = (Unsafe.Add(ref aboveBase, topBasis) * (32 - topWeight)) + (Unsafe.Add(ref aboveBase, topBasis + 1) * topWeight); - } - else - { - int leftBasis = leftProjection >> leftFractionBits; - int leftWeight = ((leftProjection << leftShift) & 0x3F) >> 1; - prediction = (Unsafe.Add(ref leftBase, leftBasis) * (32 - leftWeight)) + (Unsafe.Add(ref leftBase, leftBasis + 1) * leftWeight); - } - - Unsafe.Add(ref destinationRow, column) = (short)((prediction + 16) >> 5); - } - } - } - - /// - /// Predicts one 8-bit zone 3 block without hardware intrinsics. - /// - /// The destination block origin. - /// The destination row stride. - /// The projected left reference. - /// Whether the reference contains half-sample positions. - /// The Q8 projection derivative. - /// The block width. - /// The block height. - private static void PredictZone3Scalar(Span destination, int destinationStride, ReadOnlySpan left, bool upsample, int derivative, int width, int height) - { - int upsampleShift = upsample ? 1 : 0; - int maximumBasis = (width + height - 1) << upsampleShift; - int fractionBits = 6 - upsampleShift; - int basisIncrement = 1 << upsampleShift; - int projection = derivative; - ref byte leftBase = ref Unsafe.AsRef(in left[0]); - - for (int column = 0; column < width; column++, projection += derivative) - { - int basis = projection >> fractionBits; - int weight = ((projection << upsampleShift) & 0x3F) >> 1; - for (int row = 0; row < height; row++, basis += basisIncrement) - { - destination[(row * destinationStride) + column] = basis < maximumBasis - ? (byte)(((Unsafe.Add(ref leftBase, basis) * (32 - weight)) + (Unsafe.Add(ref leftBase, basis + 1) * weight) + 16) >> 5) - : Unsafe.Add(ref leftBase, maximumBasis); - } - } - } - - /// - /// Predicts one high-bit-depth zone 3 block without hardware intrinsics. - /// - /// The destination block origin. - /// The destination row stride. - /// The projected left reference. - /// Whether the reference contains half-sample positions. - /// The Q8 projection derivative. - /// The block width. - /// The block height. - private static void PredictZone3Scalar(Span destination, int destinationStride, ReadOnlySpan left, bool upsample, int derivative, int width, int height) - { - int upsampleShift = upsample ? 1 : 0; - int maximumBasis = (width + height - 1) << upsampleShift; - int fractionBits = 6 - upsampleShift; - int basisIncrement = 1 << upsampleShift; - int projection = derivative; - ref short leftBase = ref Unsafe.AsRef(in left[0]); - - for (int column = 0; column < width; column++, projection += derivative) - { - int basis = projection >> fractionBits; - int weight = ((projection << upsampleShift) & 0x3F) >> 1; - for (int row = 0; row < height; row++, basis += basisIncrement) - { - destination[(row * destinationStride) + column] = basis < maximumBasis - ? (short)(((Unsafe.Add(ref leftBase, basis) * (32 - weight)) + (Unsafe.Add(ref leftBase, basis + 1) * weight) + 16) >> 5) - : Unsafe.Add(ref leftBase, maximumBasis); - } - } - } -} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1FilterIntraPredictor.Operator.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1FilterIntraPredictor.Operator.cs index f3ea62a65..c573f3166 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1FilterIntraPredictor.Operator.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1FilterIntraPredictor.Operator.cs @@ -452,7 +452,7 @@ internal abstract partial class Av1FilterIntraPredictorBase private static void StoreEightShorts(Vector256 prediction, ref short buffer, int firstRowOffset, int secondRowOffset) { Vector256 narrowed = Vector256.Narrow(prediction, Vector256.Zero); - Unsafe.As(ref Unsafe.Add(ref buffer, firstRowOffset)) = narrowed.AsUInt64().GetElement(0); + Unsafe.As(ref Unsafe.Add(ref buffer, firstRowOffset)) = narrowed.AsUInt64().ToScalar(); Unsafe.As(ref Unsafe.Add(ref buffer, secondRowOffset)) = narrowed.AsUInt64().GetElement(1); } @@ -466,7 +466,7 @@ internal abstract partial class Av1FilterIntraPredictorBase private static void StoreFourShorts(Vector128 prediction, ref short buffer, int offset) { Vector128 narrowed = Vector128.Narrow(prediction, Vector128.Zero); - Unsafe.As(ref Unsafe.Add(ref buffer, offset)) = narrowed.AsUInt64().GetElement(0); + Unsafe.As(ref Unsafe.Add(ref buffer, offset)) = narrowed.AsUInt64().ToScalar(); } /// diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1IntraPredictor.Operator.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1IntraPredictor.Operator.cs index 7c6cb0c20..2ddfa692e 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1IntraPredictor.Operator.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1IntraPredictor.Operator.cs @@ -4,6 +4,7 @@ using System.Runtime.CompilerServices; using System.Runtime.InteropServices; using System.Runtime.Intrinsics; +using SixLabors.ImageSharp.Formats.Heif.Av1.Transform; namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1IntraPredictorBase.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1IntraPredictorBase.cs index 9bb6bc64c..e79b5b3a8 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1IntraPredictorBase.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1IntraPredictorBase.cs @@ -1,6 +1,8 @@ // Copyright (c) Six Labors. // Licensed under the Six Labors Split License. +using SixLabors.ImageSharp.Formats.Heif.Av1.Transform; + namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; /// diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1PalettePredictor.Operator.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1PalettePredictor.Operator.cs new file mode 100644 index 000000000..ca500cbf3 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1PalettePredictor.Operator.cs @@ -0,0 +1,356 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using System.Runtime.Intrinsics; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; + +/// +/// Defines the closed scalar/SIMD operator contract and traversal for AV1 palette prediction. +/// +internal static class Av1PalettePredictor +{ + /// + /// Multiplies a palette index by two to select the low byte of a high-bit-depth entry. + /// + private const ushort PaletteByteOffsetMultiplier = 0x0202; + + /// + /// Adds one to each odd control byte so each shuffled high-bit-depth sample retains both bytes. + /// + private const ushort PaletteHighByteOffset = 0x0100; + + /// + /// Defines scalar and SIMD palette-index lookup. + /// + private interface IPaletteOperator + { + /// + /// Predicts one 8-bit sample. + /// + /// The first palette entry. + /// The palette index. + /// The selected sample. + public static abstract byte Predict(ref byte palette, byte index); + + /// + /// Predicts sixteen 8-bit samples. + /// + /// The palette entries repeated in each 128-bit lane. + /// The palette indices. + /// The selected samples. + public static abstract Vector128 Predict(Vector128 palette, Vector128 indices); + + /// + /// Predicts thirty-two 8-bit samples. + /// + /// The palette entries repeated in each 128-bit lane. + /// The palette indices. + /// The selected samples. + public static abstract Vector256 Predict(Vector256 palette, Vector256 indices); + + /// + /// Predicts sixty-four 8-bit samples. + /// + /// The palette entries repeated in each 128-bit lane. + /// The palette indices. + /// The selected samples. + public static abstract Vector512 Predict(Vector512 palette, Vector512 indices); + + /// + /// Predicts one high-bit-depth sample. + /// + /// The first palette entry. + /// The palette index. + /// The selected sample. + public static abstract short Predict(ref ushort palette, byte index); + + /// + /// Predicts eight high-bit-depth samples. + /// + /// The palette bytes repeated in each 128-bit lane. + /// The palette indices. + /// The selected samples. + public static abstract Vector128 Predict(Vector128 palette, Vector128 indices); + + /// + /// Predicts sixteen high-bit-depth samples. + /// + /// The palette bytes repeated in each 128-bit lane. + /// The palette indices. + /// The selected samples. + public static abstract Vector256 Predict(Vector256 palette, Vector256 indices); + + /// + /// Predicts thirty-two high-bit-depth samples. + /// + /// The palette bytes repeated in each 128-bit lane. + /// The palette indices. + /// The selected samples. + public static abstract Vector512 Predict(Vector512 palette, Vector512 indices); + } + + /// + /// Reconstructs an 8-bit palette-predicted block. + /// + public static void Predict( + ReadOnlySpan paletteColors, + ReadOnlySpan colorIndexMap, + int colorIndexMapStride, + Span destination, + int destinationStride, + int width, + int height) + => Predictor.Predict(paletteColors, colorIndexMap, colorIndexMapStride, destination, destinationStride, width, height); + + /// + /// Reconstructs a high-bit-depth palette-predicted block. + /// + public static void Predict( + ReadOnlySpan paletteColors, + ReadOnlySpan colorIndexMap, + int colorIndexMapStride, + Span destination, + int destinationStride, + int width, + int height) + => Predictor.Predict(paletteColors, colorIndexMap, colorIndexMapStride, destination, destinationStride, width, height); + + /// + /// Maps decoded palette indices to reconstructed samples. + /// + private readonly struct PaletteOperator : IPaletteOperator + { + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static byte Predict(ref byte palette, byte index) => Unsafe.Add(ref palette, index); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector128 Predict(Vector128 palette, Vector128 indices) + => Vector128.ShuffleNative(palette, indices); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector256 Predict(Vector256 palette, Vector256 indices) + => Vector256.ShuffleNative(palette, indices); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector512 Predict(Vector512 palette, Vector512 indices) + => Vector512.ShuffleNative(palette, indices); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static short Predict(ref ushort palette, byte index) => (short)Unsafe.Add(ref palette, index); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector128 Predict(Vector128 palette, Vector128 indices) + { + Vector128 controls = (indices * Vector128.Create(PaletteByteOffsetMultiplier)) + Vector128.Create(PaletteHighByteOffset); + + return Vector128.ShuffleNative(palette, controls.AsByte()).AsInt16(); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector256 Predict(Vector256 palette, Vector256 indices) + { + Vector256 controls = (indices * Vector256.Create(PaletteByteOffsetMultiplier)) + Vector256.Create(PaletteHighByteOffset); + + return Vector256.ShuffleNative(palette, controls.AsByte()).AsInt16(); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector512 Predict(Vector512 palette, Vector512 indices) + { + Vector512 controls = (indices * Vector512.Create(PaletteByteOffsetMultiplier)) + Vector512.Create(PaletteHighByteOffset); + + return Vector512.ShuffleNative(palette, controls.AsByte()).AsInt16(); + } + } + + /// + /// Traverses palette blocks through one closed lookup operator. + /// + /// The palette lookup arithmetic. + private static class Predictor + where TOperator : struct, IPaletteOperator + { + /// + /// Reconstructs an 8-bit palette block. + /// + public static void Predict( + ReadOnlySpan paletteColors, + ReadOnlySpan colorIndexMap, + int colorIndexMapStride, + Span destination, + int destinationStride, + int width, + int height) + { + ref byte mapBase = ref MemoryMarshal.GetReference(colorIndexMap); + ref byte destinationBase = ref MemoryMarshal.GetReference(destination); + + // AV1 palettes contain at most eight colors. Repeating all eight entries in every 128-bit lane keeps native + // table lookup lane-local at every SIMD width and removes palette bounds work from the reconstruction loop. + ulong packedPalette = 0; + for (int index = 0; index < paletteColors.Length; index++) + { + packedPalette |= (ulong)(byte)paletteColors[index] << (index * 8); + } + + ref byte paletteBase = ref Unsafe.As(ref packedPalette); + Vector128 palette128 = Vector128.Create(packedPalette, packedPalette).AsByte(); + + for (int row = 0; row < height; row++) + { + ref byte mapRow = ref Unsafe.Add(ref mapBase, row * colorIndexMapStride); + ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); + int column = 0; + + if (Vector512.IsHardwareAccelerated) + { + Vector256 palette256 = Vector256.Create(palette128, palette128); + Vector512 palette512 = Vector512.Create(palette256, palette256); + int oneVectorFromEnd = width - Vector512.Count; + + for (; column <= oneVectorFromEnd; column += Vector512.Count) + { + Vector512 indices = Vector512.LoadUnsafe(ref mapRow, (nuint)column); + TOperator.Predict(palette512, indices).StoreUnsafe(ref destinationRow, (nuint)column); + } + } + + if (Vector256.IsHardwareAccelerated) + { + Vector256 palette256 = Vector256.Create(palette128, palette128); + int oneVectorFromEnd = width - Vector256.Count; + + for (; column <= oneVectorFromEnd; column += Vector256.Count) + { + Vector256 indices = Vector256.LoadUnsafe(ref mapRow, (nuint)column); + TOperator.Predict(palette256, indices).StoreUnsafe(ref destinationRow, (nuint)column); + } + } + + if (Vector128.IsHardwareAccelerated) + { + int oneVectorFromEnd = width - Vector128.Count; + for (; column <= oneVectorFromEnd; column += Vector128.Count) + { + Vector128 indices = Vector128.LoadUnsafe(ref mapRow, (nuint)column); + TOperator.Predict(palette128, indices).StoreUnsafe(ref destinationRow, (nuint)column); + } + + int remaining = width - column; + if (remaining >= 8) + { + ulong packedIndices = Unsafe.ReadUnaligned(ref Unsafe.Add(ref mapRow, column)); + Vector128 prediction = TOperator.Predict(palette128, Vector128.CreateScalarUnsafe(packedIndices).AsByte()); + Unsafe.WriteUnaligned(ref Unsafe.Add(ref destinationRow, column), prediction.AsUInt64().ToScalar()); + column += 8; + remaining -= 8; + } + + if (remaining >= 4) + { + uint packedIndices = Unsafe.ReadUnaligned(ref Unsafe.Add(ref mapRow, column)); + Vector128 prediction = TOperator.Predict(palette128, Vector128.CreateScalarUnsafe(packedIndices).AsByte()); + Unsafe.WriteUnaligned(ref Unsafe.Add(ref destinationRow, column), prediction.AsUInt32().ToScalar()); + column += 4; + } + } + + for (; column < width; column++) + { + Unsafe.Add(ref destinationRow, column) = TOperator.Predict(ref paletteBase, Unsafe.Add(ref mapRow, column)); + } + } + } + + /// + /// Reconstructs a high-bit-depth palette block. + /// + public static void Predict( + ReadOnlySpan paletteColors, + ReadOnlySpan colorIndexMap, + int colorIndexMapStride, + Span destination, + int destinationStride, + int width, + int height) + { + ref byte mapBase = ref MemoryMarshal.GetReference(colorIndexMap); + ref short destinationBase = ref MemoryMarshal.GetReference(destination); + InlineArray8 paletteStorage = default; + paletteColors.CopyTo(paletteStorage); + + ref ushort paletteBase = ref paletteStorage[0]; + Vector128 palette128 = Vector128.LoadUnsafe(ref paletteBase).AsByte(); + + for (int row = 0; row < height; row++) + { + ref byte mapRow = ref Unsafe.Add(ref mapBase, row * colorIndexMapStride); + ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); + int column = 0; + + if (Vector512.IsHardwareAccelerated) + { + Vector256 palette256 = Vector256.Create(palette128, palette128); + Vector512 palette512 = Vector512.Create(palette256, palette256); + int oneVectorFromEnd = width - Vector512.Count; + + for (; column <= oneVectorFromEnd; column += Vector512.Count) + { + (Vector256 lower, Vector256 upper) = Vector256.Widen(Vector256.LoadUnsafe(ref mapRow, (nuint)column)); + Vector512 indices = Vector512.Create(lower, upper); + TOperator.Predict(palette512, indices).StoreUnsafe(ref destinationRow, (nuint)column); + } + } + + if (Vector256.IsHardwareAccelerated) + { + Vector256 palette256 = Vector256.Create(palette128, palette128); + int oneVectorFromEnd = width - Vector256.Count; + + for (; column <= oneVectorFromEnd; column += Vector256.Count) + { + (Vector128 lower, Vector128 upper) = Vector128.Widen(Vector128.LoadUnsafe(ref mapRow, (nuint)column)); + Vector256 indices = Vector256.Create(lower, upper); + TOperator.Predict(palette256, indices).StoreUnsafe(ref destinationRow, (nuint)column); + } + } + + if (Vector128.IsHardwareAccelerated) + { + int oneVectorFromEnd = width - Vector128.Count; + for (; column <= oneVectorFromEnd; column += Vector128.Count) + { + ulong packedIndices = Unsafe.ReadUnaligned(ref Unsafe.Add(ref mapRow, column)); + Vector128 indices = Vector128.WidenLower(Vector128.CreateScalarUnsafe(packedIndices).AsByte()); + TOperator.Predict(palette128, indices).StoreUnsafe(ref destinationRow, (nuint)column); + } + + if (width - column >= 4) + { + uint packedIndices = Unsafe.ReadUnaligned(ref Unsafe.Add(ref mapRow, column)); + Vector128 indices = Vector128.WidenLower(Vector128.CreateScalarUnsafe(packedIndices).AsByte()); + Vector128 prediction = TOperator.Predict(palette128, indices); + Unsafe.WriteUnaligned(ref Unsafe.As(ref Unsafe.Add(ref destinationRow, column)), prediction.AsUInt64().ToScalar()); + column += 4; + } + } + + for (; column < width; column++) + { + Unsafe.Add(ref destinationRow, column) = TOperator.Predict(ref paletteBase, Unsafe.Add(ref mapRow, column)); + } + } + } + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1PalettePredictor.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1PalettePredictor.cs deleted file mode 100644 index 21d648a3e..000000000 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1PalettePredictor.cs +++ /dev/null @@ -1,274 +0,0 @@ -// Copyright (c) Six Labors. -// Licensed under the Six Labors Split License. - -using System.Runtime.CompilerServices; -using System.Runtime.InteropServices; -using System.Runtime.Intrinsics; - -namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; - -/// -/// Reconstructs AV1 palette-predicted sample blocks from decoded color-index maps. -/// -/// -/// AV1 palettes contain at most eight entries, so a complete 8-bit palette fits in each native shuffle lane and a -/// complete high-bit-depth palette fits as sixteen bytes. Color indices become byte-shuffle controls; replicating the -/// table per 128-bit lane keeps every lookup lane-local at 128, 256, and 512 bits. Exact-width tail loads and stores -/// avoid requiring writable padding around small transform blocks. -/// -internal static class Av1PalettePredictor -{ - /// - /// The repeated byte offsets that select both bytes of eight 16-bit palette entries. - /// - private const ushort PaletteByteOffsetMultiplier = 0x0202; - - /// - /// The high-byte increment that selects the second byte of each 16-bit palette entry. - /// - private const ushort PaletteHighByteOffset = 0x0100; - - /// - /// Reconstructs an 8-bit palette-predicted block. - /// - /// The decoded palette colors in prediction-index order. - /// The color-index map beginning at the prediction block origin. - /// The distance, in indices, between map rows. - /// The destination beginning at the prediction block origin. - /// The distance, in samples, between destination rows. - /// The prediction width in samples. - /// The prediction height in samples. - public static void Predict( - ReadOnlySpan paletteColors, - ReadOnlySpan colorIndexMap, - int colorIndexMapStride, - Span destination, - int destinationStride, - int width, - int height) - { - ref byte mapBase = ref MemoryMarshal.GetReference(colorIndexMap); - ref byte destinationBase = ref MemoryMarshal.GetReference(destination); - - // An AV1 palette contains at most eight colors. Packing it once into the low 64 bits and repeating it in - // every 128-bit lane turns reconstruction into the lane-local table lookup implemented by ShuffleNative. - ulong packedPalette = 0; - for (int index = 0; index < paletteColors.Length; index++) - { - packedPalette |= (ulong)(byte)paletteColors[index] << (index * 8); - } - - Vector128 palette128 = Vector128.Create(packedPalette, packedPalette).AsByte(); - - if (Vector512.IsHardwareAccelerated && width >= Vector512.Count) - { - Vector256 palette256 = Vector256.Create(palette128, palette128); - Vector512 palette512 = Vector512.Create(palette256, palette256); - - for (int row = 0; row < height; row++) - { - ref byte mapRow = ref Unsafe.Add(ref mapBase, row * colorIndexMapStride); - ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); - - for (int column = 0; column < width; column += Vector512.Count) - { - Vector512 indices = Vector512.LoadUnsafe(ref mapRow, (nuint)column); - Vector512.ShuffleNative(palette512, indices).StoreUnsafe(ref destinationRow, (nuint)column); - } - } - - return; - } - - if (Vector256.IsHardwareAccelerated && width >= Vector256.Count) - { - Vector256 palette256 = Vector256.Create(palette128, palette128); - for (int row = 0; row < height; row++) - { - ref byte mapRow = ref Unsafe.Add(ref mapBase, row * colorIndexMapStride); - ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); - - for (int column = 0; column < width; column += Vector256.Count) - { - Vector256 indices = Vector256.LoadUnsafe(ref mapRow, (nuint)column); - Vector256.ShuffleNative(palette256, indices).StoreUnsafe(ref destinationRow, (nuint)column); - } - } - - return; - } - - if (Vector128.IsHardwareAccelerated) - { - for (int row = 0; row < height; row++) - { - ref byte mapRow = ref Unsafe.Add(ref mapBase, row * colorIndexMapStride); - ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); - int column = 0; - - for (; column <= width - Vector128.Count; column += Vector128.Count) - { - Vector128 indices = Vector128.LoadUnsafe(ref mapRow, (nuint)column); - Vector128.ShuffleNative(palette128, indices).StoreUnsafe(ref destinationRow, (nuint)column); - } - - if (column < width) - { - // Transform widths are powers of two. After complete vector chunks, only a four- or eight-byte - // row tail remains, so an exact-width load and store keeps neighboring transforms untouched. - int remaining = width - column; - ulong packedIndices = remaining == 4 - ? Unsafe.ReadUnaligned(ref Unsafe.Add(ref mapRow, column)) - : Unsafe.ReadUnaligned(ref Unsafe.Add(ref mapRow, column)); - - Vector128 result = Vector128.ShuffleNative(palette128, Vector128.CreateScalarUnsafe(packedIndices).AsByte()); - - if (remaining == 4) - { - Unsafe.WriteUnaligned(ref Unsafe.Add(ref destinationRow, column), result.AsUInt32().ToScalar()); - } - else - { - Unsafe.WriteUnaligned(ref Unsafe.Add(ref destinationRow, column), result.AsUInt64().ToScalar()); - } - - column = width; - } - } - - return; - } - - for (int row = 0; row < height; row++) - { - ref byte mapRow = ref Unsafe.Add(ref mapBase, row * colorIndexMapStride); - ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); - for (int column = 0; column < width; column++) - { - Unsafe.Add(ref destinationRow, column) = (byte)paletteColors[Unsafe.Add(ref mapRow, column)]; - } - } - } - - /// - /// Reconstructs a high-bit-depth palette-predicted block. - /// - /// The decoded palette colors in prediction-index order. - /// The color-index map beginning at the prediction block origin. - /// The distance, in indices, between map rows. - /// The destination beginning at the prediction block origin. - /// The distance, in samples, between destination rows. - /// The prediction width in samples. - /// The prediction height in samples. - public static void Predict( - ReadOnlySpan paletteColors, - ReadOnlySpan colorIndexMap, - int colorIndexMapStride, - Span destination, - int destinationStride, - int width, - int height) - { - ref byte mapBase = ref MemoryMarshal.GetReference(colorIndexMap); - ref short destinationBase = ref MemoryMarshal.GetReference(destination); - InlineArray8 paletteStorage = default; - - paletteColors.CopyTo(paletteStorage); - - // Each index is expanded to the byte offsets 2n and 2n+1. Repeating the complete 16-byte palette in every - // 128-bit lane then permits the same native byte-table shuffle on x86, Arm, and WebAssembly. - Vector128 palette128 = Vector128.LoadUnsafe(ref paletteStorage[0]).AsByte(); - - if (Vector512.IsHardwareAccelerated && width >= Vector512.Count) - { - Vector256 palette256 = Vector256.Create(palette128, palette128); - Vector512 palette512 = Vector512.Create(palette256, palette256); - Vector512 multiplier = Vector512.Create(PaletteByteOffsetMultiplier); - Vector512 increment = Vector512.Create(PaletteHighByteOffset); - - for (int row = 0; row < height; row++) - { - ref byte mapRow = ref Unsafe.Add(ref mapBase, row * colorIndexMapStride); - ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); - - for (int column = 0; column < width; column += Vector512.Count) - { - (Vector256 lower, Vector256 upper) = Vector256.Widen(Vector256.LoadUnsafe(ref mapRow, (nuint)column)); - - Vector512 indices = Vector512.Create(lower, upper); - Vector512 controls = ((indices * multiplier) + increment).AsByte(); - Vector512.ShuffleNative(palette512, controls).AsInt16().StoreUnsafe(ref destinationRow, (nuint)column); - } - } - - return; - } - - if (Vector256.IsHardwareAccelerated && width >= Vector256.Count) - { - Vector256 palette256 = Vector256.Create(palette128, palette128); - Vector256 multiplier = Vector256.Create(PaletteByteOffsetMultiplier); - Vector256 increment = Vector256.Create(PaletteHighByteOffset); - - for (int row = 0; row < height; row++) - { - ref byte mapRow = ref Unsafe.Add(ref mapBase, row * colorIndexMapStride); - ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); - - for (int column = 0; column < width; column += Vector256.Count) - { - (Vector128 lower, Vector128 upper) = Vector128.Widen(Vector128.LoadUnsafe(ref mapRow, (nuint)column)); - - Vector256 indices = Vector256.Create(lower, upper); - Vector256 controls = ((indices * multiplier) + increment).AsByte(); - Vector256.ShuffleNative(palette256, controls).AsInt16().StoreUnsafe(ref destinationRow, (nuint)column); - } - } - - return; - } - - if (Vector128.IsHardwareAccelerated) - { - Vector128 multiplier = Vector128.Create(PaletteByteOffsetMultiplier); - Vector128 increment = Vector128.Create(PaletteHighByteOffset); - - for (int row = 0; row < height; row++) - { - ref byte mapRow = ref Unsafe.Add(ref mapBase, row * colorIndexMapStride); - ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); - int column = 0; - - for (; column <= width - Vector128.Count; column += Vector128.Count) - { - ulong packedIndices = Unsafe.ReadUnaligned(ref Unsafe.Add(ref mapRow, column)); - Vector128 indices = Vector128.WidenLower(Vector128.CreateScalarUnsafe(packedIndices).AsByte()); - Vector128 controls = ((indices * multiplier) + increment).AsByte(); - Vector128.ShuffleNative(palette128, controls).AsInt16().StoreUnsafe(ref destinationRow, (nuint)column); - } - - if (column < width) - { - uint packedIndices = Unsafe.ReadUnaligned(ref Unsafe.Add(ref mapRow, column)); - Vector128 indices = Vector128.WidenLower(Vector128.CreateScalarUnsafe(packedIndices).AsByte()); - Vector128 controls = ((indices * multiplier) + increment).AsByte(); - Vector128 result = Vector128.ShuffleNative(palette128, controls).AsInt16(); - Unsafe.WriteUnaligned(ref Unsafe.As(ref Unsafe.Add(ref destinationRow, column)), result.AsUInt64().ToScalar()); - column += 4; - } - } - - return; - } - - for (int row = 0; row < height; row++) - { - ref byte mapRow = ref Unsafe.Add(ref mapBase, row * colorIndexMapStride); - ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); - for (int column = 0; column < width; column++) - { - Unsafe.Add(ref destinationRow, column) = (short)paletteColors[Unsafe.Add(ref mapRow, column)]; - } - } - } -} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1PredictionDecoder.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1PredictionDecoder.cs index 4ded4d591..de7872cb6 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1PredictionDecoder.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1PredictionDecoder.cs @@ -1390,7 +1390,7 @@ internal class Av1PredictionDecoder Vector128 interpolated = InterpolateEightBytes(ref inputBase, i); Vector128 originals = Vector128.LoadUnsafe(ref inputBase, (nuint)(i + 2)); Vector128 interleaved = Vector128_.UnpackLow(interpolated, originals); - Unsafe.As(ref Unsafe.Add(ref bufferBase, (2 * i) - 1)) = interleaved.AsUInt64().GetElement(0); + Unsafe.As(ref Unsafe.Add(ref bufferBase, (2 * i) - 1)) = interleaved.AsUInt64().ToScalar(); i += 4; } } @@ -1610,7 +1610,7 @@ internal class Av1PredictionDecoder Vector128 source1 = WidenLower(Vector128.LoadUnsafe(ref edge, (nuint)(processed + 2))); Vector128 source2 = WidenLower(Vector128.LoadUnsafe(ref edge, (nuint)(processed + 3))); Vector128 result = Vector128.Narrow(FilterEdgeStrength1(source0, source1, source2), Vector128.Zero); - Unsafe.As(ref Unsafe.Add(ref buffer, processed + 1)) = result.AsUInt64().GetElement(0); + Unsafe.As(ref Unsafe.Add(ref buffer, processed + 1)) = result.AsUInt64().ToScalar(); } break; @@ -1621,7 +1621,7 @@ internal class Av1PredictionDecoder Vector128 source1 = WidenLower(Vector128.LoadUnsafe(ref edge, (nuint)(processed + 2))); Vector128 source2 = WidenLower(Vector128.LoadUnsafe(ref edge, (nuint)(processed + 3))); Vector128 result = Vector128.Narrow(FilterEdgeStrength2(source0, source1, source2), Vector128.Zero); - Unsafe.As(ref Unsafe.Add(ref buffer, processed + 1)) = result.AsUInt64().GetElement(0); + Unsafe.As(ref Unsafe.Add(ref buffer, processed + 1)) = result.AsUInt64().ToScalar(); } break; @@ -1634,7 +1634,7 @@ internal class Av1PredictionDecoder Vector128 source3 = WidenLower(Vector128.LoadUnsafe(ref edge, (nuint)(processed + 3))); Vector128 source4 = WidenLower(Vector128.LoadUnsafe(ref edge, (nuint)(processed + 4))); Vector128 result = Vector128.Narrow(FilterEdgeStrength3(source0, source1, source2, source3, source4), Vector128.Zero); - Unsafe.As(ref Unsafe.Add(ref buffer, processed + 1)) = result.AsUInt64().GetElement(0); + Unsafe.As(ref Unsafe.Add(ref buffer, processed + 1)) = result.AsUInt64().ToScalar(); } break; diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/ChromaFromLuma/Av1ChromaFromLumaPredictor.Operator.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/ChromaFromLuma/Av1ChromaFromLumaPredictor.Operator.cs new file mode 100644 index 000000000..23dfca502 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/ChromaFromLuma/Av1ChromaFromLumaPredictor.Operator.cs @@ -0,0 +1,324 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using System.Runtime.Intrinsics; +using System.Runtime.Intrinsics.Arm; +using System.Runtime.Intrinsics.X86; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.ChromaFromLuma; + +/// +/// Defines the closed scalar/SIMD operator contract and traversal for AV1 chroma-from-luma prediction. +/// +internal static partial class Av1ChromaFromLumaPredictor +{ + /// + /// The fixed row stride of the AV1 chroma-from-luma scratch buffer. + /// + private const int BufferLine = 32; + + /// + /// Defines scalar and SIMD signed Q3 arithmetic for AV1 chroma-from-luma prediction. + /// + private interface IChromaFromLumaOperator + { + /// + /// Predicts one chroma sample. + /// + /// The zero-mean Q3 luma sample. + /// The chroma DC prediction. + /// The signed Q3 chroma scaling factor. + /// The maximum sample value. + /// The predicted chroma sample. + public static abstract short Predict(short lumaQ3, short dc, int alphaQ3, short maximum); + + /// + /// Predicts eight chroma samples. + /// + /// The zero-mean Q3 luma samples. + /// The chroma DC prediction. + /// The signed Q3 chroma scaling factor. + /// The maximum sample value. + /// The predicted chroma samples. + public static abstract Vector128 Predict(Vector128 lumaQ3, short dc, int alphaQ3, short maximum); + + /// + /// Predicts sixteen chroma samples. + /// + /// The zero-mean Q3 luma samples. + /// The chroma DC prediction. + /// The signed Q3 chroma scaling factor. + /// The maximum sample value. + /// The predicted chroma samples. + public static abstract Vector256 Predict(Vector256 lumaQ3, short dc, int alphaQ3, short maximum); + + /// + /// Predicts thirty-two chroma samples. + /// + /// The zero-mean Q3 luma samples. + /// The chroma DC prediction. + /// The signed Q3 chroma scaling factor. + /// The maximum sample value. + /// The predicted chroma samples. + public static abstract Vector512 Predict(Vector512 lumaQ3, short dc, int alphaQ3, short maximum); + } + + /// + /// Applies chroma-from-luma prediction to an 8-bit chroma block. + /// + /// The zero-mean Q3 luma surface. + /// The DC-predicted chroma block that receives the luma adjustment. + /// The distance, in samples, between destination rows. + /// The signed Q3 chroma scaling factor. + /// The block width in samples. + /// The block height in samples. + public static void Predict(ReadOnlySpan lumaQ3, Span destination, int destinationStride, int alphaQ3, int width, int height) + => Predictor.Predict(lumaQ3, destination, destinationStride, alphaQ3, width, height); + + /// + /// Applies chroma-from-luma prediction to a high-bit-depth chroma block. + /// + /// The zero-mean Q3 luma surface. + /// The DC-predicted chroma block that receives the luma adjustment. + /// The distance, in samples, between destination rows. + /// The signed Q3 chroma scaling factor. + /// The number of bits used to represent each sample. + /// The block width in samples. + /// The block height in samples. + public static void Predict(ReadOnlySpan lumaQ3, Span destination, int destinationStride, int alphaQ3, int bitDepth, int width, int height) + => Predictor.Predict(lumaQ3, destination, destinationStride, alphaQ3, bitDepth, width, height); + + /// + /// Applies the decoded chroma scaling factor to zero-mean luma samples. + /// + private readonly struct ChromaFromLumaOperator : IChromaFromLumaOperator + { + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static short Predict(short lumaQ3, short dc, int alphaQ3, short maximum) + { + int scaledLumaQ0 = Av1Math.RoundPowerOf2Signed(alphaQ3 * lumaQ3, 6); + + return (short)Math.Clamp(dc + scaledLumaQ0, (short)0, maximum); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector128 Predict(Vector128 lumaQ3, short dc, int alphaQ3, short maximum) + { + Vector128 dcVector = Vector128.Create(dc); + Vector128 scaledLumaQ0; + + if (Ssse3.IsSupported) + { + Vector128 alphaSign = Vector128.Create((short)alphaQ3); + Vector128 alphaQ12 = Vector128.Create((short)(Math.Abs(alphaQ3) << 9)); + scaledLumaQ0 = Ssse3.MultiplyHighRoundScale(Ssse3.Abs(lumaQ3).AsInt16(), alphaQ12); + Vector128 signMask = (lumaQ3 ^ alphaSign) >> 15; + scaledLumaQ0 = (scaledLumaQ0 ^ signMask) - signMask; + } + else if (AdvSimd.IsSupported) + { + Vector128 alphaSign = Vector128.Create((short)alphaQ3); + Vector128 alphaQ12 = Vector128.Create((short)(Math.Abs(alphaQ3) << 9)); + scaledLumaQ0 = AdvSimd.MultiplyRoundedDoublingSaturateHigh(Vector128.Abs(lumaQ3), alphaQ12); + Vector128 signMask = (lumaQ3 ^ alphaSign) >> 15; + scaledLumaQ0 = (scaledLumaQ0 ^ signMask) - signMask; + } + else + { + // WebAssembly and other Vector128 targets do not expose packed rounded-high multiply. Widening keeps + // the same signed rounding rule without introducing a second scalar traversal. + (Vector128 lower, Vector128 upper) = Vector128.Widen(lumaQ3); + Vector128 alpha = Vector128.Create(alphaQ3); + lower *= alpha; + upper *= alpha; + lower = (lower + Vector128.Create(32) + (lower >> 31)) >> 6; + upper = (upper + Vector128.Create(32) + (upper >> 31)) >> 6; + scaledLumaQ0 = Vector128.Narrow(lower, upper); + } + + return Vector128.Clamp(scaledLumaQ0 + dcVector, Vector128.Zero, Vector128.Create(maximum)); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector256 Predict(Vector256 lumaQ3, short dc, int alphaQ3, short maximum) + { + Vector256 dcVector = Vector256.Create(dc); + Vector256 scaledLumaQ0; + + if (Avx2.IsSupported) + { + Vector256 alphaSign = Vector256.Create((short)alphaQ3); + Vector256 alphaQ12 = Vector256.Create((short)(Math.Abs(alphaQ3) << 9)); + scaledLumaQ0 = Avx2.MultiplyHighRoundScale(Avx2.Abs(lumaQ3).AsInt16(), alphaQ12); + Vector256 signMask = (lumaQ3 ^ alphaSign) >> 15; + scaledLumaQ0 = (scaledLumaQ0 ^ signMask) - signMask; + } + else + { + (Vector256 lower, Vector256 upper) = Vector256.Widen(lumaQ3); + Vector256 alpha = Vector256.Create(alphaQ3); + lower *= alpha; + upper *= alpha; + lower = (lower + Vector256.Create(32) + (lower >> 31)) >> 6; + upper = (upper + Vector256.Create(32) + (upper >> 31)) >> 6; + scaledLumaQ0 = Vector256.Narrow(lower, upper); + } + + return Vector256.Clamp(scaledLumaQ0 + dcVector, Vector256.Zero, Vector256.Create(maximum)); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector512 Predict(Vector512 lumaQ3, short dc, int alphaQ3, short maximum) + { + (Vector512 lower, Vector512 upper) = Vector512.Widen(lumaQ3); + Vector512 alpha = Vector512.Create(alphaQ3); + lower *= alpha; + upper *= alpha; + lower = (lower + Vector512.Create(32) + (lower >> 31)) >> 6; + upper = (upper + Vector512.Create(32) + (upper >> 31)) >> 6; + Vector512 scaledLumaQ0 = Vector512.Narrow(lower, upper); + + return Vector512.Clamp(scaledLumaQ0 + Vector512.Create(dc), Vector512.Zero, Vector512.Create(maximum)); + } + } + + /// + /// Traverses a chroma block through one closed prediction operator. + /// + /// The signed Q3 prediction arithmetic. + private static class Predictor + where TOperator : struct, IChromaFromLumaOperator + { + /// + /// Applies chroma-from-luma prediction to an 8-bit block. + /// + public static void Predict(ReadOnlySpan lumaQ3, Span destination, int destinationStride, int alphaQ3, int width, int height) + { + ref short lumaBase = ref MemoryMarshal.GetReference(lumaQ3); + ref byte destinationBase = ref MemoryMarshal.GetReference(destination); + short dc = destinationBase; + + // CfL follows DC prediction, so one sample supplies the base value for the complete block. The fixed + // scratch stride also makes exact-width Vector128 loads safe for the four-sample AV1 tail. + for (int row = 0; row < height; row++) + { + int lumaRowOffset = row * BufferLine; + int destinationRowOffset = row * destinationStride; + ref short lumaRow = ref Unsafe.Add(ref lumaBase, lumaRowOffset); + ref byte destinationRow = ref Unsafe.Add(ref destinationBase, destinationRowOffset); + int column = 0; + + if (Vector512.IsHardwareAccelerated) + { + int oneVectorFromEnd = width - Vector512.Count; + for (; column <= oneVectorFromEnd; column += Vector512.Count) + { + Vector512 prediction = TOperator.Predict(Vector512.LoadUnsafe(ref lumaRow, (nuint)column), dc, alphaQ3, byte.MaxValue); + Vector256 packed = Vector512.Narrow(prediction.AsUInt16(), Vector512.Zero).GetLower(); + packed.StoreUnsafe(ref destinationRow, (nuint)column); + } + } + + if (Vector256.IsHardwareAccelerated) + { + int oneVectorFromEnd = width - Vector256.Count; + for (; column <= oneVectorFromEnd; column += Vector256.Count) + { + Vector256 prediction = TOperator.Predict(Vector256.LoadUnsafe(ref lumaRow, (nuint)column), dc, alphaQ3, byte.MaxValue); + Vector128 packed = Vector256.Narrow(prediction.AsUInt16(), Vector256.Zero).GetLower(); + packed.StoreUnsafe(ref destinationRow, (nuint)column); + } + } + + if (Vector128.IsHardwareAccelerated) + { + int oneVectorFromEnd = width - Vector128.Count; + for (; column <= oneVectorFromEnd; column += Vector128.Count) + { + Vector128 prediction = TOperator.Predict(Vector128.LoadUnsafe(ref lumaRow, (nuint)column), dc, alphaQ3, byte.MaxValue); + Vector64 packed = Vector128.Narrow(prediction.AsUInt16(), Vector128.Zero).GetLower(); + packed.StoreUnsafe(ref destinationRow, (nuint)column); + } + + if (width - column >= 4) + { + Vector128 prediction = TOperator.Predict(Vector128.LoadUnsafe(ref lumaRow, (nuint)column), dc, alphaQ3, byte.MaxValue); + Vector64 packed = Vector128.Narrow(prediction.AsUInt16(), Vector128.Zero).GetLower(); + Unsafe.WriteUnaligned(ref Unsafe.Add(ref destinationRow, column), packed.AsUInt32().ToScalar()); + column += 4; + } + } + + for (; column < width; column++) + { + Unsafe.Add(ref destinationRow, column) = (byte)TOperator.Predict(Unsafe.Add(ref lumaRow, column), dc, alphaQ3, byte.MaxValue); + } + } + } + + /// + /// Applies chroma-from-luma prediction to a high-bit-depth block. + /// + public static void Predict(ReadOnlySpan lumaQ3, Span destination, int destinationStride, int alphaQ3, int bitDepth, int width, int height) + { + ref short lumaBase = ref MemoryMarshal.GetReference(lumaQ3); + ref short destinationBase = ref MemoryMarshal.GetReference(destination); + short dc = destinationBase; + short maximum = (short)((1 << bitDepth) - 1); + + for (int row = 0; row < height; row++) + { + int lumaRowOffset = row * BufferLine; + int destinationRowOffset = row * destinationStride; + ref short lumaRow = ref Unsafe.Add(ref lumaBase, lumaRowOffset); + ref short destinationRow = ref Unsafe.Add(ref destinationBase, destinationRowOffset); + int column = 0; + + if (Vector512.IsHardwareAccelerated) + { + int oneVectorFromEnd = width - Vector512.Count; + for (; column <= oneVectorFromEnd; column += Vector512.Count) + { + TOperator.Predict(Vector512.LoadUnsafe(ref lumaRow, (nuint)column), dc, alphaQ3, maximum).StoreUnsafe(ref destinationRow, (nuint)column); + } + } + + if (Vector256.IsHardwareAccelerated) + { + int oneVectorFromEnd = width - Vector256.Count; + for (; column <= oneVectorFromEnd; column += Vector256.Count) + { + TOperator.Predict(Vector256.LoadUnsafe(ref lumaRow, (nuint)column), dc, alphaQ3, maximum).StoreUnsafe(ref destinationRow, (nuint)column); + } + } + + if (Vector128.IsHardwareAccelerated) + { + int oneVectorFromEnd = width - Vector128.Count; + for (; column <= oneVectorFromEnd; column += Vector128.Count) + { + TOperator.Predict(Vector128.LoadUnsafe(ref lumaRow, (nuint)column), dc, alphaQ3, maximum).StoreUnsafe(ref destinationRow, (nuint)column); + } + + if (width - column >= 4) + { + Vector128 prediction = TOperator.Predict(Vector128.LoadUnsafe(ref lumaRow, (nuint)column), dc, alphaQ3, maximum); + Unsafe.WriteUnaligned(ref Unsafe.As(ref Unsafe.Add(ref destinationRow, column)), prediction.AsUInt64().ToScalar()); + column += 4; + } + } + + for (; column < width; column++) + { + Unsafe.Add(ref destinationRow, column) = TOperator.Predict(Unsafe.Add(ref lumaRow, column), dc, alphaQ3, maximum); + } + } + } + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/ChromaFromLuma/Av1ChromaFromLumaPredictor.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/ChromaFromLuma/Av1ChromaFromLumaPredictor.cs deleted file mode 100644 index 5a6465477..000000000 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/ChromaFromLuma/Av1ChromaFromLumaPredictor.cs +++ /dev/null @@ -1,237 +0,0 @@ -// Copyright (c) Six Labors. -// Licensed under the Six Labors Split License. - -using System.Runtime.CompilerServices; -using System.Runtime.InteropServices; -using System.Runtime.Intrinsics; -using System.Runtime.Intrinsics.Arm; -using System.Runtime.Intrinsics.X86; - -namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.ChromaFromLuma; - -/// -/// Applies an AV1 chroma-from-luma residual to a DC-predicted chroma block. -/// -/// -/// Each signed 16-bit lane contains one zero-mean Q3 luma value. Packed rounded-high multiplication converts the product -/// with the Q3 alpha parameter directly to a signed integer adjustment; the alpha/luma sign mask restores the product -/// sign after the magnitude operation. The common DC prediction is broadcast, then results are clipped and narrowed to -/// the destination sample representation. -/// -internal static class Av1ChromaFromLumaPredictor -{ - /// - /// The fixed row stride of the AV1 chroma-from-luma scratch buffer. - /// - private const int BufferLine = 32; - - /// - /// Applies chroma-from-luma prediction to an 8-bit chroma block. - /// - /// The zero-mean Q3 luma surface. - /// The DC-predicted chroma block that receives the luma adjustment. - /// The distance, in samples, between destination rows. - /// The signed Q3 chroma scaling factor. - /// The block width in samples. - /// The block height in samples. - public static void Predict(ReadOnlySpan lumaQ3, Span destination, int destinationStride, int alphaQ3, int width, int height) - { - ref short lumaBase = ref MemoryMarshal.GetReference(lumaQ3); - ref byte destinationBase = ref MemoryMarshal.GetReference(destination); - short dc = destinationBase; - - // CfL always follows DC prediction, so the first sample is the common base value for every lane. This - // mirrors libaom and avoids loading a block that is known to contain a single repeated prediction value. - if (Avx2.IsSupported && width >= Vector256.Count) - { - Vector256 alphaSign = Vector256.Create((short)alphaQ3); - Vector256 alphaQ12 = Vector256.Create((short)(Math.Abs(alphaQ3) << 9)); - Vector256 dcVector = Vector256.Create(dc); - Vector256 maximum = Vector256.Create((short)byte.MaxValue); - - for (int row = 0; row < height; row++) - { - int lumaRowOffset = row * BufferLine; - int destinationRowOffset = row * destinationStride; - - for (int column = 0; column < width; column += Vector256.Count) - { - Vector256 prediction = Predict(Vector256.LoadUnsafe(ref lumaBase, (nuint)(lumaRowOffset + column)), alphaSign, alphaQ12, dcVector); - prediction = Vector256.Clamp(prediction, Vector256.Zero, maximum); - Vector256.Narrow(prediction.AsUInt16(), Vector256.Zero).GetLower().StoreUnsafe(ref destinationBase, (nuint)(destinationRowOffset + column)); - } - } - - return; - } - - if (Vector128.IsHardwareAccelerated) - { - Vector128 alphaSign = Vector128.Create((short)alphaQ3); - Vector128 alphaQ12 = Vector128.Create((short)(Math.Abs(alphaQ3) << 9)); - Vector128 dcVector = Vector128.Create(dc); - Vector128 maximum = Vector128.Create((short)byte.MaxValue); - - for (int row = 0; row < height; row++) - { - int lumaRowOffset = row * BufferLine; - int destinationRowOffset = row * destinationStride; - int column = 0; - - for (; column <= width - Vector128.Count; column += Vector128.Count) - { - Vector128 prediction = Predict(Vector128.LoadUnsafe(ref lumaBase, (nuint)(lumaRowOffset + column)), alphaSign, alphaQ12, dcVector, alphaQ3); - prediction = Vector128.Clamp(prediction, Vector128.Zero, maximum); - Vector128.Narrow(prediction.AsUInt16(), Vector128.Zero).GetLower().StoreUnsafe(ref destinationBase, (nuint)(destinationRowOffset + column)); - } - - // Four-wide CfL blocks still have a complete padded scratch row, so reading eight residuals is - // valid. Only the four active predictions are stored to the image buffer. - if (column < width) - { - Vector128 prediction = Predict(Vector128.LoadUnsafe(ref lumaBase, (nuint)(lumaRowOffset + column)), alphaSign, alphaQ12, dcVector, alphaQ3); - prediction = Vector128.Clamp(prediction, Vector128.Zero, maximum); - Vector64 packed = Vector128.Narrow(prediction.AsUInt16(), Vector128.Zero).GetLower(); - Unsafe.WriteUnaligned(ref Unsafe.Add(ref destinationBase, destinationRowOffset + column), packed.AsUInt32().ToScalar()); - } - } - - return; - } - - for (int row = 0; row < height; row++) - { - int lumaRowOffset = row * BufferLine; - int destinationRowOffset = row * destinationStride; - for (int column = 0; column < width; column++) - { - int scaledLumaQ0 = Av1Math.RoundPowerOf2Signed(alphaQ3 * Unsafe.Add(ref lumaBase, lumaRowOffset + column), 6); - Unsafe.Add(ref destinationBase, destinationRowOffset + column) = (byte)Math.Clamp(dc + scaledLumaQ0, byte.MinValue, byte.MaxValue); - } - } - } - - /// - /// Applies chroma-from-luma prediction to a high-bit-depth chroma block. - /// - /// The zero-mean Q3 luma surface. - /// The DC-predicted chroma block that receives the luma adjustment. - /// The distance, in samples, between destination rows. - /// The signed Q3 chroma scaling factor. - /// The number of bits used to represent each sample. - /// The block width in samples. - /// The block height in samples. - public static void Predict(ReadOnlySpan lumaQ3, Span destination, int destinationStride, int alphaQ3, int bitDepth, int width, int height) - { - ref short lumaBase = ref MemoryMarshal.GetReference(lumaQ3); - ref short destinationBase = ref MemoryMarshal.GetReference(destination); - short dc = destinationBase; - short maximum = (short)((1 << bitDepth) - 1); - - if (Avx2.IsSupported && width >= Vector256.Count) - { - Vector256 alphaSign = Vector256.Create((short)alphaQ3); - Vector256 alphaQ12 = Vector256.Create((short)(Math.Abs(alphaQ3) << 9)); - Vector256 dcVector = Vector256.Create(dc); - Vector256 maximumVector = Vector256.Create(maximum); - - for (int row = 0; row < height; row++) - { - int lumaRowOffset = row * BufferLine; - int destinationRowOffset = row * destinationStride; - - for (int column = 0; column < width; column += Vector256.Count) - { - Vector256 prediction = Predict(Vector256.LoadUnsafe(ref lumaBase, (nuint)(lumaRowOffset + column)), alphaSign, alphaQ12, dcVector); - Vector256.Clamp(prediction, Vector256.Zero, maximumVector).StoreUnsafe(ref destinationBase, (nuint)(destinationRowOffset + column)); - } - } - - return; - } - - if (Vector128.IsHardwareAccelerated) - { - Vector128 alphaSign = Vector128.Create((short)alphaQ3); - Vector128 alphaQ12 = Vector128.Create((short)(Math.Abs(alphaQ3) << 9)); - Vector128 dcVector = Vector128.Create(dc); - Vector128 maximumVector = Vector128.Create(maximum); - - for (int row = 0; row < height; row++) - { - int lumaRowOffset = row * BufferLine; - int destinationRowOffset = row * destinationStride; - int column = 0; - - for (; column <= width - Vector128.Count; column += Vector128.Count) - { - Vector128 prediction = Predict(Vector128.LoadUnsafe(ref lumaBase, (nuint)(lumaRowOffset + column)), alphaSign, alphaQ12, dcVector, alphaQ3); - Vector128.Clamp(prediction, Vector128.Zero, maximumVector).StoreUnsafe(ref destinationBase, (nuint)(destinationRowOffset + column)); - } - - if (column < width) - { - Vector128 prediction = Predict(Vector128.LoadUnsafe(ref lumaBase, (nuint)(lumaRowOffset + column)), alphaSign, alphaQ12, dcVector, alphaQ3); - prediction = Vector128.Clamp(prediction, Vector128.Zero, maximumVector); - Unsafe.WriteUnaligned(ref Unsafe.As(ref Unsafe.Add(ref destinationBase, destinationRowOffset + column)), prediction.AsUInt64().ToScalar()); - } - } - - return; - } - - for (int row = 0; row < height; row++) - { - int lumaRowOffset = row * BufferLine; - int destinationRowOffset = row * destinationStride; - for (int column = 0; column < width; column++) - { - int scaledLumaQ0 = Av1Math.RoundPowerOf2Signed(alphaQ3 * Unsafe.Add(ref lumaBase, lumaRowOffset + column), 6); - Unsafe.Add(ref destinationBase, destinationRowOffset + column) = (short)Math.Clamp(dc + scaledLumaQ0, 0, maximum); - } - } - } - - /// - /// Calculates sixteen chroma predictions using the packed Q3 arithmetic defined by AV1. - /// - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector256 Predict(Vector256 lumaQ3, Vector256 alphaSign, Vector256 alphaQ12, Vector256 dc) - { - Vector256 scaledLumaQ0 = Avx2.MultiplyHighRoundScale(Avx2.Abs(lumaQ3).AsInt16(), alphaQ12); - Vector256 signMask = (lumaQ3 ^ alphaSign) >> 15; - return ((scaledLumaQ0 ^ signMask) - signMask) + dc; - } - - /// - /// Calculates eight chroma predictions using the packed Q3 arithmetic defined by AV1. - /// - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector128 Predict(Vector128 lumaQ3, Vector128 alphaSign, Vector128 alphaQ12, Vector128 dc, int alphaQ3) - { - Vector128 scaledLumaQ0; - if (Ssse3.IsSupported) - { - scaledLumaQ0 = Ssse3.MultiplyHighRoundScale(Ssse3.Abs(lumaQ3).AsInt16(), alphaQ12); - } - else if (AdvSimd.IsSupported) - { - scaledLumaQ0 = AdvSimd.MultiplyRoundedDoublingSaturateHigh(Vector128.Abs(lumaQ3), alphaQ12); - } - else - { - // WebAssembly and other Vector128 targets do not expose packed rounded-high multiply. Widening retains - // SIMD traversal while reproducing the same signed nearest-integer result in ordinary integer lanes. - (Vector128 lower, Vector128 upper) = Vector128.Widen(lumaQ3); - Vector128 alpha = Vector128.Create(alphaQ3); - lower *= alpha; - upper *= alpha; - lower = (lower + Vector128.Create(32) + (lower >> 31)) >> 6; - upper = (upper + Vector128.Create(32) + (upper >> 31)) >> 6; - return Vector128.Narrow(lower, upper) + dc; - } - - Vector128 signMask = (lumaQ3 ^ alphaSign) >> 15; - return ((scaledLumaQ0 ^ signMask) - signMask) + dc; - } -} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundAveragePredictor.Operator.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundAveragePredictor.Operator.cs new file mode 100644 index 000000000..c14a44155 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundAveragePredictor.Operator.cs @@ -0,0 +1,132 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.Intrinsics; + +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; + +/// +/// Defines equal-weight compound prediction arithmetic. +/// +internal static partial class Av1CompoundAveragePredictor +{ + /// + /// Defines equal-weight compound averaging for scalar and SIMD lane groups. + /// + private interface IAv1CompoundAverageOperator + { + /// + /// Averages two 8-bit samples. + /// + /// The first sample. + /// The second sample. + /// The rounded average. + public static abstract byte Blend(byte first, byte second); + + /// + /// Averages two high-bit-depth samples. + /// + /// The first sample. + /// The second sample. + /// The rounded average. + public static abstract ushort Blend(ushort first, ushort second); + + /// + /// Averages two 128-bit vectors of 8-bit samples. + /// + /// The first samples. + /// The second samples. + /// The rounded averages. + public static abstract Vector128 Blend(Vector128 first, Vector128 second); + + /// + /// Averages two 256-bit vectors of 8-bit samples. + /// + /// The first samples. + /// The second samples. + /// The rounded averages. + public static abstract Vector256 Blend(Vector256 first, Vector256 second); + + /// + /// Averages two 512-bit vectors of 8-bit samples. + /// + /// The first samples. + /// The second samples. + /// The rounded averages. + public static abstract Vector512 Blend(Vector512 first, Vector512 second); + + /// + /// Averages two 128-bit vectors of high-bit-depth samples. + /// + /// The first samples. + /// The second samples. + /// The rounded averages. + public static abstract Vector128 Blend(Vector128 first, Vector128 second); + + /// + /// Averages two 256-bit vectors of high-bit-depth samples. + /// + /// The first samples. + /// The second samples. + /// The rounded averages. + public static abstract Vector256 Blend(Vector256 first, Vector256 second); + + /// + /// Averages two 512-bit vectors of high-bit-depth samples. + /// + /// The first samples. + /// The second samples. + /// The rounded averages. + public static abstract Vector512 Blend(Vector512 first, Vector512 second); + } + + /// + /// Implements equal-weight rounded averaging for scalar and SIMD lane groups. + /// + private readonly struct CompoundAverageOperator : IAv1CompoundAverageOperator + { + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static byte Blend(byte first, byte second) => (byte)((first + second + 1) >> 1); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static ushort Blend(ushort first, ushort second) => (ushort)((first + second + 1) >> 1); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector128 Blend(Vector128 first, Vector128 second) + => (first | second) - ((first ^ second) >> 1); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector256 Blend(Vector256 first, Vector256 second) + => (first | second) - ((first ^ second) >> 1); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector512 Blend(Vector512 first, Vector512 second) + { + // This identity is exactly (a + b + 1) >> 1 but cannot overflow unsigned lanes at any SIMD width. + return (first | second) - ((first ^ second) >> 1); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector128 Blend(Vector128 first, Vector128 second) + => (first | second) - ((first ^ second) >> 1); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector256 Blend(Vector256 first, Vector256 second) + => (first | second) - ((first ^ second) >> 1); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector512 Blend(Vector512 first, Vector512 second) + => (first | second) - ((first ^ second) >> 1); + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundAveragePredictor.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundAveragePredictor.cs new file mode 100644 index 000000000..1b3e17536 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundAveragePredictor.cs @@ -0,0 +1,281 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.InteropServices; +using System.Runtime.Intrinsics; + +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; + +/// +/// Combines two AV1 inter predictors with equal-weight rounded averaging. +/// +internal static partial class Av1CompoundAveragePredictor +{ + /// + /// Averages an 8-bit predictor into an existing prediction block. + /// + /// The first predictor and combined output. + /// The distance between destination rows in samples. + /// The second predictor. + /// The distance between second-predictor rows in samples. + /// The active block width. + /// The active block height. + public static void Average( + Span destination, + int destinationStride, + ReadOnlySpan second, + int secondStride, + int width, + int height) + => Average(destination, destinationStride, second, secondStride, width, height); + + /// + /// Averages an 8-bit predictor through one closed compound operator. + /// + /// The equal-weight averaging arithmetic. + /// The first predictor and combined output. + /// The distance between destination rows. + /// The second predictor. + /// The distance between second-predictor rows. + /// The active block width. + /// The active block height. + private static void Average( + Span destination, + int destinationStride, + ReadOnlySpan second, + int secondStride, + int width, + int height) + where TOperator : struct, IAv1CompoundAverageOperator + { + for (int row = 0; row < height; row++) + { + Span destinationRow = destination.Slice(row * destinationStride, width); + ReadOnlySpan secondRow = second.Slice(row * secondStride, width); + ref byte destinationReference = ref MemoryMarshal.GetReference(destinationRow); + ref byte secondReference = ref MemoryMarshal.GetReference(secondRow); + int column = 0; + + if (Vector512.IsHardwareAccelerated) + { + int vectorEnd = width - Vector512.Count; + for (; column <= vectorEnd; column += Vector512.Count) + { + Vector512 firstVector = Vector512.LoadUnsafe(ref destinationReference, (nuint)column); + Vector512 secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column); + TOperator.Blend(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + if (Vector256.IsHardwareAccelerated) + { + int vectorEnd = width - Vector256.Count; + for (; column <= vectorEnd; column += Vector256.Count) + { + Vector256 firstVector = Vector256.LoadUnsafe(ref destinationReference, (nuint)column); + Vector256 secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column); + TOperator.Blend(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + if (Vector128.IsHardwareAccelerated) + { + int vectorEnd = width - Vector128.Count; + for (; column <= vectorEnd; column += Vector128.Count) + { + Vector128 firstVector = Vector128.LoadUnsafe(ref destinationReference, (nuint)column); + Vector128 secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column); + TOperator.Blend(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + for (; column < width; column++) + { + destinationRow[column] = TOperator.Blend(destinationRow[column], secondRow[column]); + } + } + } + + /// + /// Averages a high-bit-depth predictor into an existing prediction block. + /// + /// The first predictor and combined output. + /// The distance between destination rows in samples. + /// The second predictor. + /// The distance between second-predictor rows in samples. + /// The active block width. + /// The active block height. + public static void Average( + Span destination, + int destinationStride, + ReadOnlySpan second, + int secondStride, + int width, + int height) + => Average(destination, destinationStride, second, secondStride, width, height); + + /// + /// Averages a high-bit-depth predictor through one closed compound operator. + /// + /// The equal-weight averaging arithmetic. + /// The first predictor and combined output. + /// The distance between destination rows. + /// The second predictor. + /// The distance between second-predictor rows. + /// The active block width. + /// The active block height. + private static void Average( + Span destination, + int destinationStride, + ReadOnlySpan second, + int secondStride, + int width, + int height) + where TOperator : struct, IAv1CompoundAverageOperator + { + for (int row = 0; row < height; row++) + { + Span destinationRow = destination.Slice(row * destinationStride, width); + ReadOnlySpan secondRow = second.Slice(row * secondStride, width); + ref ushort destinationReference = ref MemoryMarshal.GetReference(destinationRow); + ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow); + int column = 0; + + if (Vector512.IsHardwareAccelerated) + { + int vectorEnd = width - Vector512.Count; + for (; column <= vectorEnd; column += Vector512.Count) + { + Vector512 firstVector = Vector512.LoadUnsafe(ref destinationReference, (nuint)column); + Vector512 secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column); + TOperator.Blend(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + if (Vector256.IsHardwareAccelerated) + { + int vectorEnd = width - Vector256.Count; + for (; column <= vectorEnd; column += Vector256.Count) + { + Vector256 firstVector = Vector256.LoadUnsafe(ref destinationReference, (nuint)column); + Vector256 secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column); + TOperator.Blend(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + if (Vector128.IsHardwareAccelerated) + { + int vectorEnd = width - Vector128.Count; + for (; column <= vectorEnd; column += Vector128.Count) + { + Vector128 firstVector = Vector128.LoadUnsafe(ref destinationReference, (nuint)column); + Vector128 secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column); + TOperator.Blend(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + for (; column < width; column++) + { + destinationRow[column] = TOperator.Blend(destinationRow[column], secondRow[column]); + } + } + } + + /// + /// Averages an 8-bit predictor without explicit hardware intrinsics. + /// + /// The first predictor and combined output. + /// The distance between destination rows in samples. + /// The second predictor. + /// The distance between second-predictor rows in samples. + /// The active block width. + /// The active block height. + public static void AverageScalar( + Span destination, + int destinationStride, + ReadOnlySpan second, + int secondStride, + int width, + int height) + => AverageScalar(destination, destinationStride, second, secondStride, width, height); + + /// + /// Averages an 8-bit predictor through one closed scalar compound operator. + /// + /// The equal-weight averaging arithmetic. + /// The first predictor and combined output. + /// The distance between destination rows. + /// The second predictor. + /// The distance between second-predictor rows. + /// The active block width. + /// The active block height. + private static void AverageScalar( + Span destination, + int destinationStride, + ReadOnlySpan second, + int secondStride, + int width, + int height) + where TOperator : struct, IAv1CompoundAverageOperator + { + for (int row = 0; row < height; row++) + { + Span destinationRow = destination.Slice(row * destinationStride, width); + ReadOnlySpan secondRow = second.Slice(row * secondStride, width); + for (int column = 0; column < width; column++) + { + destinationRow[column] = TOperator.Blend(destinationRow[column], secondRow[column]); + } + } + } + + /// + /// Averages a high-bit-depth predictor without explicit hardware intrinsics. + /// + /// The first predictor and combined output. + /// The distance between destination rows in samples. + /// The second predictor. + /// The distance between second-predictor rows in samples. + /// The active block width. + /// The active block height. + public static void AverageScalar( + Span destination, + int destinationStride, + ReadOnlySpan second, + int secondStride, + int width, + int height) + => AverageScalar(destination, destinationStride, second, secondStride, width, height); + + /// + /// Averages a high-bit-depth predictor through one closed scalar compound operator. + /// + /// The equal-weight averaging arithmetic. + /// The first predictor and combined output. + /// The distance between destination rows. + /// The second predictor. + /// The distance between second-predictor rows. + /// The active block width. + /// The active block height. + private static void AverageScalar( + Span destination, + int destinationStride, + ReadOnlySpan second, + int secondStride, + int width, + int height) + where TOperator : struct, IAv1CompoundAverageOperator + { + for (int row = 0; row < height; row++) + { + Span destinationRow = destination.Slice(row * destinationStride, width); + ReadOnlySpan secondRow = second.Slice(row * secondStride, width); + for (int column = 0; column < width; column++) + { + destinationRow[column] = TOperator.Blend(destinationRow[column], secondRow[column]); + } + } + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundDistanceWeightedPredictor.Operator.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundDistanceWeightedPredictor.Operator.cs new file mode 100644 index 000000000..5cd6247cf --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundDistanceWeightedPredictor.Operator.cs @@ -0,0 +1,211 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.Intrinsics; + +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor; +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; + +/// +/// Defines distance-weighted compound prediction arithmetic. +/// +internal static partial class Av1CompoundDistanceWeightedPredictor +{ + /// + /// Defines distance-weighted compound blending for scalar and SIMD lane groups. + /// + private interface IAv1CompoundDistanceWeightedOperator + { + /// + /// Blends two 8-bit samples with display-distance weights. + /// + /// The first sample. + /// The second sample. + /// The first predictor weight. + /// The second predictor weight. + /// The weighted sample. + public static abstract byte Blend(byte first, byte second, int firstWeight, int secondWeight); + + /// + /// Blends two high-bit-depth samples with display-distance weights. + /// + /// The first sample. + /// The second sample. + /// The first predictor weight. + /// The second predictor weight. + /// The weighted sample. + public static abstract ushort Blend(ushort first, ushort second, int firstWeight, int secondWeight); + + /// + /// Blends 128-bit vectors of 8-bit samples with display-distance weights. + /// + /// The first samples. + /// The second samples. + /// The first predictor weight. + /// The second predictor weight. + /// The weighted samples. + public static abstract Vector128 Blend(Vector128 first, Vector128 second, int firstWeight, int secondWeight); + + /// + /// Blends 256-bit vectors of 8-bit samples with display-distance weights. + /// + /// The first samples. + /// The second samples. + /// The first predictor weight. + /// The second predictor weight. + /// The weighted samples. + public static abstract Vector256 Blend(Vector256 first, Vector256 second, int firstWeight, int secondWeight); + + /// + /// Blends 512-bit vectors of 8-bit samples with display-distance weights. + /// + /// The first samples. + /// The second samples. + /// The first predictor weight. + /// The second predictor weight. + /// The weighted samples. + public static abstract Vector512 Blend(Vector512 first, Vector512 second, int firstWeight, int secondWeight); + + /// + /// Blends 128-bit vectors of high-bit-depth samples with display-distance weights. + /// + /// The first samples. + /// The second samples. + /// The first predictor weight. + /// The second predictor weight. + /// The weighted samples. + public static abstract Vector128 Blend(Vector128 first, Vector128 second, int firstWeight, int secondWeight); + + /// + /// Blends 256-bit vectors of high-bit-depth samples with display-distance weights. + /// + /// The first samples. + /// The second samples. + /// The first predictor weight. + /// The second predictor weight. + /// The weighted samples. + public static abstract Vector256 Blend(Vector256 first, Vector256 second, int firstWeight, int secondWeight); + + /// + /// Blends 512-bit vectors of high-bit-depth samples with display-distance weights. + /// + /// The first samples. + /// The second samples. + /// The first predictor weight. + /// The second predictor weight. + /// The weighted samples. + public static abstract Vector512 Blend(Vector512 first, Vector512 second, int firstWeight, int secondWeight); + } + + /// + /// Implements display-distance-weighted blending for scalar and SIMD lane groups. + /// + private readonly struct CompoundDistanceWeightedOperator : IAv1CompoundDistanceWeightedOperator + { + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static byte Blend(byte first, byte second, int firstWeight, int secondWeight) + => (byte)(((first * firstWeight) + (second * secondWeight) + 8) >> DistanceWeightBits); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static ushort Blend(ushort first, ushort second, int firstWeight, int secondWeight) + => (ushort)(((first * firstWeight) + (second * secondWeight) + 8) >> DistanceWeightBits); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector128 Blend(Vector128 first, Vector128 second, int firstWeight, int secondWeight) + { + Av1IntraPredictorBase.Widen(first, out Vector128 first0, out Vector128 first1, out Vector128 first2, out Vector128 first3); + Av1IntraPredictorBase.Widen(second, out Vector128 second0, out Vector128 second1, out Vector128 second2, out Vector128 second3); + return Av1IntraPredictorBase.Narrow( + Blend(first0, second0, firstWeight, secondWeight), + Blend(first1, second1, firstWeight, secondWeight), + Blend(first2, second2, firstWeight, secondWeight), + Blend(first3, second3, firstWeight, secondWeight)); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector256 Blend(Vector256 first, Vector256 second, int firstWeight, int secondWeight) + { + Av1IntraPredictorBase.Widen(first, out Vector256 first0, out Vector256 first1, out Vector256 first2, out Vector256 first3); + Av1IntraPredictorBase.Widen(second, out Vector256 second0, out Vector256 second1, out Vector256 second2, out Vector256 second3); + return Av1IntraPredictorBase.Narrow( + Blend(first0, second0, firstWeight, secondWeight), + Blend(first1, second1, firstWeight, secondWeight), + Blend(first2, second2, firstWeight, secondWeight), + Blend(first3, second3, firstWeight, secondWeight)); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector512 Blend(Vector512 first, Vector512 second, int firstWeight, int secondWeight) + { + Av1IntraPredictorBase.Widen(first, out Vector512 first0, out Vector512 first1, out Vector512 first2, out Vector512 first3); + Av1IntraPredictorBase.Widen(second, out Vector512 second0, out Vector512 second1, out Vector512 second2, out Vector512 second3); + return Av1IntraPredictorBase.Narrow( + Blend(first0, second0, firstWeight, secondWeight), + Blend(first1, second1, firstWeight, secondWeight), + Blend(first2, second2, firstWeight, secondWeight), + Blend(first3, second3, firstWeight, secondWeight)); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector128 Blend(Vector128 first, Vector128 second, int firstWeight, int secondWeight) + { + Av1IntraPredictorBase.Widen(first.AsInt16(), out Vector128 first0, out Vector128 first1); + Av1IntraPredictorBase.Widen(second.AsInt16(), out Vector128 second0, out Vector128 second1); + return Av1IntraPredictorBase.Narrow( + Blend(first0, second0, firstWeight, secondWeight), + Blend(first1, second1, firstWeight, secondWeight)).AsUInt16(); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector256 Blend(Vector256 first, Vector256 second, int firstWeight, int secondWeight) + { + Av1IntraPredictorBase.Widen(first.AsInt16(), out Vector256 first0, out Vector256 first1); + Av1IntraPredictorBase.Widen(second.AsInt16(), out Vector256 second0, out Vector256 second1); + return Av1IntraPredictorBase.Narrow( + Blend(first0, second0, firstWeight, secondWeight), + Blend(first1, second1, firstWeight, secondWeight)).AsUInt16(); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector512 Blend(Vector512 first, Vector512 second, int firstWeight, int secondWeight) + { + Av1IntraPredictorBase.Widen(first.AsInt16(), out Vector512 first0, out Vector512 first1); + Av1IntraPredictorBase.Widen(second.AsInt16(), out Vector512 second0, out Vector512 second1); + return Av1IntraPredictorBase.Narrow( + Blend(first0, second0, firstWeight, secondWeight), + Blend(first1, second1, firstWeight, secondWeight)).AsUInt16(); + } + + /// + /// Applies display-distance weighting to 128-bit vectors of widened samples. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector128 Blend(Vector128 first, Vector128 second, int firstWeight, int secondWeight) + => ((first * Vector128.Create(firstWeight)) + (second * Vector128.Create(secondWeight)) + Vector128.Create(8)) >> DistanceWeightBits; + + /// + /// Applies display-distance weighting to 256-bit vectors of widened samples. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector256 Blend(Vector256 first, Vector256 second, int firstWeight, int secondWeight) + => ((first * Vector256.Create(firstWeight)) + (second * Vector256.Create(secondWeight)) + Vector256.Create(8)) >> DistanceWeightBits; + + /// + /// Applies display-distance weighting to 512-bit vectors of widened samples. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector512 Blend(Vector512 first, Vector512 second, int firstWeight, int secondWeight) + => ((first * Vector512.Create(firstWeight)) + (second * Vector512.Create(secondWeight)) + Vector512.Create(8)) >> DistanceWeightBits; + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundDistanceWeightedPredictor.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundDistanceWeightedPredictor.cs new file mode 100644 index 000000000..c315d161e --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundDistanceWeightedPredictor.cs @@ -0,0 +1,235 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using System.Runtime.Intrinsics; + +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor; +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; + +/// +/// Reconstructs display-distance-weighted compound prediction. +/// +internal static partial class Av1CompoundDistanceWeightedPredictor +{ + /// + /// Combines two 8-bit predictors with AV1 display-distance weights. + /// + public static void DistanceWeighted( + Span destination, + int destinationStride, + ReadOnlySpan second, + int secondStride, + int width, + int height, + int firstWeight, + int secondWeight) + => DistanceWeighted( + destination, + destinationStride, + second, + secondStride, + width, + height, + firstWeight, + secondWeight); + + /// + /// Executes one closed 8-bit distance-weighted compound operator. + /// + /// The compound arithmetic operator. + private static void DistanceWeighted( + Span destination, + int destinationStride, + ReadOnlySpan second, + int secondStride, + int width, + int height, + int firstWeight, + int secondWeight) + where TOperator : struct, IAv1CompoundDistanceWeightedOperator + { + for (int row = 0; row < height; row++) + { + Span destinationRow = destination.Slice(row * destinationStride, width); + ReadOnlySpan secondRow = second.Slice(row * secondStride, width); + ref byte destinationReference = ref MemoryMarshal.GetReference(destinationRow); + ref byte secondReference = ref MemoryMarshal.GetReference(secondRow); + int column = 0; + + if (Vector512.IsHardwareAccelerated) + { + int vectorEnd = width - Vector512.Count; + for (; column <= vectorEnd; column += Vector512.Count) + { + Vector512 firstVector = Vector512.LoadUnsafe(ref destinationReference, (nuint)column); + Vector512 secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column); + TOperator.Blend(firstVector, secondVector, firstWeight, secondWeight).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + if (Vector256.IsHardwareAccelerated) + { + int vectorEnd = width - Vector256.Count; + for (; column <= vectorEnd; column += Vector256.Count) + { + Vector256 firstVector = Vector256.LoadUnsafe(ref destinationReference, (nuint)column); + Vector256 secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column); + TOperator.Blend(firstVector, secondVector, firstWeight, secondWeight).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + if (Vector128.IsHardwareAccelerated) + { + int vectorEnd = width - Vector128.Count; + for (; column <= vectorEnd; column += Vector128.Count) + { + Vector128 firstVector = Vector128.LoadUnsafe(ref destinationReference, (nuint)column); + Vector128 secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column); + TOperator.Blend(firstVector, secondVector, firstWeight, secondWeight).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + for (; column < width; column++) + { + destinationRow[column] = TOperator.Blend(destinationRow[column], secondRow[column], firstWeight, secondWeight); + } + } + } + + /// + /// Combines two high-bit-depth predictors with AV1 display-distance weights. + /// + public static void DistanceWeighted( + Span destination, + int destinationStride, + ReadOnlySpan second, + int secondStride, + int width, + int height, + int firstWeight, + int secondWeight) + => DistanceWeighted( + destination, + destinationStride, + second, + secondStride, + width, + height, + firstWeight, + secondWeight); + + /// + /// Executes one closed high-bit-depth distance-weighted compound operator. + /// + /// The compound arithmetic operator. + private static void DistanceWeighted( + Span destination, + int destinationStride, + ReadOnlySpan second, + int secondStride, + int width, + int height, + int firstWeight, + int secondWeight) + where TOperator : struct, IAv1CompoundDistanceWeightedOperator + { + for (int row = 0; row < height; row++) + { + Span destinationRow = destination.Slice(row * destinationStride, width); + ReadOnlySpan secondRow = second.Slice(row * secondStride, width); + ref ushort destinationReference = ref MemoryMarshal.GetReference(destinationRow); + ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow); + int column = 0; + + if (Vector512.IsHardwareAccelerated) + { + int vectorEnd = width - Vector512.Count; + for (; column <= vectorEnd; column += Vector512.Count) + { + Vector512 firstVector = Vector512.LoadUnsafe(ref destinationReference, (nuint)column); + Vector512 secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column); + TOperator.Blend(firstVector, secondVector, firstWeight, secondWeight).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + if (Vector256.IsHardwareAccelerated) + { + int vectorEnd = width - Vector256.Count; + for (; column <= vectorEnd; column += Vector256.Count) + { + Vector256 firstVector = Vector256.LoadUnsafe(ref destinationReference, (nuint)column); + Vector256 secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column); + TOperator.Blend(firstVector, secondVector, firstWeight, secondWeight).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + if (Vector128.IsHardwareAccelerated) + { + int vectorEnd = width - Vector128.Count; + for (; column <= vectorEnd; column += Vector128.Count) + { + Vector128 firstVector = Vector128.LoadUnsafe(ref destinationReference, (nuint)column); + Vector128 secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column); + TOperator.Blend(firstVector, secondVector, firstWeight, secondWeight).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + for (; column < width; column++) + { + destinationRow[column] = TOperator.Blend(destinationRow[column], secondRow[column], firstWeight, secondWeight); + } + } + } + + /// + /// Combines two 8-bit predictors with display-distance weights without explicit hardware intrinsics. + /// + public static void DistanceWeightedScalar( + Span destination, + int destinationStride, + ReadOnlySpan second, + int secondStride, + int width, + int height, + int firstWeight, + int secondWeight) + => DistanceWeightedScalar( + destination, + destinationStride, + second, + secondStride, + width, + height, + firstWeight, + secondWeight); + + /// + /// Executes one closed 8-bit distance-weighted compound operator without explicit hardware intrinsics. + /// + /// The compound arithmetic operator. + private static void DistanceWeightedScalar( + Span destination, + int destinationStride, + ReadOnlySpan second, + int secondStride, + int width, + int height, + int firstWeight, + int secondWeight) + where TOperator : struct, IAv1CompoundDistanceWeightedOperator + { + for (int row = 0; row < height; row++) + { + Span destinationRow = destination.Slice(row * destinationStride, width); + ReadOnlySpan secondRow = second.Slice(row * secondStride, width); + for (int column = 0; column < width; column++) + { + destinationRow[column] = TOperator.Blend(destinationRow[column], secondRow[column], firstWeight, secondWeight); + } + } + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.Arithmetic.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.Arithmetic.cs new file mode 100644 index 000000000..59b6e9ef2 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.Arithmetic.cs @@ -0,0 +1,112 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.Intrinsics; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; + +/// +/// Provides shared final-rounding arithmetic for compound intermediate reconstruction. +/// +internal static partial class Av1CompoundInterPredictor +{ + /// + /// Derives the bias and remaining fractional precision of a compound intermediate. + /// + public static void GetIntermediateRounding(int bitDepth, out int roundBits, out int roundOffset) + { + int intermediateRange = bitDepth + 7 - 3 + 2; + int round0 = 3 + Math.Max(intermediateRange - 16, 0); + int offsetBits = bitDepth + 14 - round0; + roundBits = 14 - round0 - CompoundRound1Bits; + roundOffset = (1 << (offsetBits - CompoundRound1Bits)) + + (1 << (offsetBits - CompoundRound1Bits - 1)); + } + + /// + /// Removes the compound bias and final fractional precision from 128-bit unsigned lanes. + /// + public static Vector128 FinalizeIntermediate(Vector128 value, int roundBits, int roundOffset) + { + Vector128 result = (value - Vector128.Create((ushort)roundOffset)).AsInt16(); + if (roundBits != 0) + { + result = (result + Vector128.Create((short)(1 << (roundBits - 1)))) >> roundBits; + } + + result = Vector128.Max(Vector128.Zero, Vector128.Min(Vector128.Create((short)byte.MaxValue), result)); + return result.AsUInt16(); + } + + /// + /// Removes the compound bias and final fractional precision from 256-bit unsigned lanes. + /// + public static Vector256 FinalizeIntermediate(Vector256 value, int roundBits, int roundOffset) + { + Vector256 result = (value - Vector256.Create((ushort)roundOffset)).AsInt16(); + if (roundBits != 0) + { + result = (result + Vector256.Create((short)(1 << (roundBits - 1)))) >> roundBits; + } + + result = Vector256.Max(Vector256.Zero, Vector256.Min(Vector256.Create((short)byte.MaxValue), result)); + return result.AsUInt16(); + } + + /// + /// Removes the compound bias and final fractional precision from 512-bit unsigned lanes. + /// + public static Vector512 FinalizeIntermediate(Vector512 value, int roundBits, int roundOffset) + { + Vector512 result = (value - Vector512.Create((ushort)roundOffset)).AsInt16(); + if (roundBits != 0) + { + result = (result + Vector512.Create((short)(1 << (roundBits - 1)))) >> roundBits; + } + + result = Vector512.Max(Vector512.Zero, Vector512.Min(Vector512.Create((short)byte.MaxValue), result)); + return result.AsUInt16(); + } + + /// + /// Removes the compound bias and final fractional precision from 128-bit widened lanes. + /// + public static Vector128 FinalizeIntermediate(Vector128 value, int roundBits, int roundOffset) + { + Vector128 result = value - Vector128.Create(roundOffset); + if (roundBits != 0) + { + result = (result + Vector128.Create(1 << (roundBits - 1))) >> roundBits; + } + + return Vector128.Max(Vector128.Zero, Vector128.Min(Vector128.Create((int)byte.MaxValue), result)); + } + + /// + /// Removes the compound bias and final fractional precision from 256-bit widened lanes. + /// + public static Vector256 FinalizeIntermediate(Vector256 value, int roundBits, int roundOffset) + { + Vector256 result = value - Vector256.Create(roundOffset); + if (roundBits != 0) + { + result = (result + Vector256.Create(1 << (roundBits - 1))) >> roundBits; + } + + return Vector256.Max(Vector256.Zero, Vector256.Min(Vector256.Create((int)byte.MaxValue), result)); + } + + /// + /// Removes the compound bias and final fractional precision from 512-bit widened lanes. + /// + public static Vector512 FinalizeIntermediate(Vector512 value, int roundBits, int roundOffset) + { + Vector512 result = value - Vector512.Create(roundOffset); + if (roundBits != 0) + { + result = (result + Vector512.Create(1 << (roundBits - 1))) >> roundBits; + } + + return Vector512.Max(Vector512.Zero, Vector512.Min(Vector512.Create((int)byte.MaxValue), result)); + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.Blending.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.Blending.cs deleted file mode 100644 index e61a3d86e..000000000 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.Blending.cs +++ /dev/null @@ -1,501 +0,0 @@ -// Copyright (c) Six Labors. -// Licensed under the Six Labors Split License. - -using System.Runtime.CompilerServices; -using System.Runtime.InteropServices; -using System.Runtime.Intrinsics; - -namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; - -/// -/// Provides distance-weighted and per-sample masked compound blending. -/// -internal static partial class Av1CompoundInterPredictor -{ - private const int DistanceWeightBits = 4; - private const int MaskWeightBits = 6; - private const int MaximumMaskAlpha = 1 << MaskWeightBits; - - /// - /// Combines two 8-bit predictors with AV1 display-distance weights. - /// - public static void DistanceWeighted( - Span destination, - int destinationStride, - ReadOnlySpan second, - int secondStride, - int width, - int height, - int firstWeight, - int secondWeight) - { - for (int row = 0; row < height; row++) - { - Span destinationRow = destination.Slice(row * destinationStride, width); - ReadOnlySpan secondRow = second.Slice(row * secondStride, width); - ref byte destinationReference = ref MemoryMarshal.GetReference(destinationRow); - ref byte secondReference = ref MemoryMarshal.GetReference(secondRow); - int column = 0; - - if (Vector512.IsHardwareAccelerated) - { - int vectorEnd = width - Vector512.Count; - for (; column <= vectorEnd; column += Vector512.Count) - { - Vector512 firstVector = Vector512.LoadUnsafe(ref destinationReference, (nuint)column); - Vector512 secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column); - DistanceWeighted(firstVector, secondVector, firstWeight, secondWeight).StoreUnsafe(ref destinationReference, (nuint)column); - } - } - - if (Vector256.IsHardwareAccelerated) - { - int vectorEnd = width - Vector256.Count; - for (; column <= vectorEnd; column += Vector256.Count) - { - Vector256 firstVector = Vector256.LoadUnsafe(ref destinationReference, (nuint)column); - Vector256 secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column); - DistanceWeighted(firstVector, secondVector, firstWeight, secondWeight).StoreUnsafe(ref destinationReference, (nuint)column); - } - } - - if (Vector128.IsHardwareAccelerated) - { - int vectorEnd = width - Vector128.Count; - for (; column <= vectorEnd; column += Vector128.Count) - { - Vector128 firstVector = Vector128.LoadUnsafe(ref destinationReference, (nuint)column); - Vector128 secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column); - DistanceWeighted(firstVector, secondVector, firstWeight, secondWeight).StoreUnsafe(ref destinationReference, (nuint)column); - } - } - - for (; column < width; column++) - { - destinationRow[column] = (byte)(((destinationRow[column] * firstWeight) + (secondRow[column] * secondWeight) + 8) >> DistanceWeightBits); - } - } - } - - /// - /// Combines two high-bit-depth predictors with AV1 display-distance weights. - /// - public static void DistanceWeighted( - Span destination, - int destinationStride, - ReadOnlySpan second, - int secondStride, - int width, - int height, - int firstWeight, - int secondWeight) - { - for (int row = 0; row < height; row++) - { - Span destinationRow = destination.Slice(row * destinationStride, width); - ReadOnlySpan secondRow = second.Slice(row * secondStride, width); - ref ushort destinationReference = ref MemoryMarshal.GetReference(destinationRow); - ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow); - int column = 0; - - if (Vector512.IsHardwareAccelerated) - { - int vectorEnd = width - Vector512.Count; - for (; column <= vectorEnd; column += Vector512.Count) - { - Vector512 firstVector = Vector512.LoadUnsafe(ref destinationReference, (nuint)column); - Vector512 secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column); - DistanceWeighted(firstVector, secondVector, firstWeight, secondWeight).StoreUnsafe(ref destinationReference, (nuint)column); - } - } - - if (Vector256.IsHardwareAccelerated) - { - int vectorEnd = width - Vector256.Count; - for (; column <= vectorEnd; column += Vector256.Count) - { - Vector256 firstVector = Vector256.LoadUnsafe(ref destinationReference, (nuint)column); - Vector256 secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column); - DistanceWeighted(firstVector, secondVector, firstWeight, secondWeight).StoreUnsafe(ref destinationReference, (nuint)column); - } - } - - if (Vector128.IsHardwareAccelerated) - { - int vectorEnd = width - Vector128.Count; - for (; column <= vectorEnd; column += Vector128.Count) - { - Vector128 firstVector = Vector128.LoadUnsafe(ref destinationReference, (nuint)column); - Vector128 secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column); - DistanceWeighted(firstVector, secondVector, firstWeight, secondWeight).StoreUnsafe(ref destinationReference, (nuint)column); - } - } - - for (; column < width; column++) - { - destinationRow[column] = (ushort)(((destinationRow[column] * firstWeight) + (secondRow[column] * secondWeight) + 8) >> DistanceWeightBits); - } - } - } - - /// - /// Blends two 8-bit predictors through a contiguous AV1 alpha mask. - /// - public static void Blend( - Span destination, - int destinationStride, - ReadOnlySpan second, - int secondStride, - ReadOnlySpan mask, - int maskStride, - int width, - int height) - { - for (int row = 0; row < height; row++) - { - Span destinationRow = destination.Slice(row * destinationStride, width); - ReadOnlySpan secondRow = second.Slice(row * secondStride, width); - ReadOnlySpan maskRow = mask.Slice(row * maskStride, width); - ref byte destinationReference = ref MemoryMarshal.GetReference(destinationRow); - ref byte secondReference = ref MemoryMarshal.GetReference(secondRow); - ref byte maskReference = ref MemoryMarshal.GetReference(maskRow); - int column = 0; - - if (Vector512.IsHardwareAccelerated) - { - int vectorEnd = width - Vector512.Count; - for (; column <= vectorEnd; column += Vector512.Count) - { - Vector512 firstVector = Vector512.LoadUnsafe(ref destinationReference, (nuint)column); - Vector512 secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column); - Vector512 maskVector = Vector512.LoadUnsafe(ref maskReference, (nuint)column); - Blend(firstVector, secondVector, maskVector).StoreUnsafe(ref destinationReference, (nuint)column); - } - } - - if (Vector256.IsHardwareAccelerated) - { - int vectorEnd = width - Vector256.Count; - for (; column <= vectorEnd; column += Vector256.Count) - { - Vector256 firstVector = Vector256.LoadUnsafe(ref destinationReference, (nuint)column); - Vector256 secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column); - Vector256 maskVector = Vector256.LoadUnsafe(ref maskReference, (nuint)column); - Blend(firstVector, secondVector, maskVector).StoreUnsafe(ref destinationReference, (nuint)column); - } - } - - if (Vector128.IsHardwareAccelerated) - { - int vectorEnd = width - Vector128.Count; - for (; column <= vectorEnd; column += Vector128.Count) - { - Vector128 firstVector = Vector128.LoadUnsafe(ref destinationReference, (nuint)column); - Vector128 secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column); - Vector128 maskVector = Vector128.LoadUnsafe(ref maskReference, (nuint)column); - Blend(firstVector, secondVector, maskVector).StoreUnsafe(ref destinationReference, (nuint)column); - } - } - - for (; column < width; column++) - { - int alpha = maskRow[column]; - destinationRow[column] = (byte)(((alpha * destinationRow[column]) + ((MaximumMaskAlpha - alpha) * secondRow[column]) + 32) >> MaskWeightBits); - } - } - } - - /// - /// Blends two high-bit-depth predictors through a contiguous AV1 alpha mask. - /// - public static void Blend( - Span destination, - int destinationStride, - ReadOnlySpan second, - int secondStride, - ReadOnlySpan mask, - int maskStride, - int width, - int height) - { - for (int row = 0; row < height; row++) - { - Span destinationRow = destination.Slice(row * destinationStride, width); - ReadOnlySpan secondRow = second.Slice(row * secondStride, width); - ReadOnlySpan maskRow = mask.Slice(row * maskStride, width); - ref ushort destinationReference = ref MemoryMarshal.GetReference(destinationRow); - ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow); - ref byte maskReference = ref MemoryMarshal.GetReference(maskRow); - int column = 0; - - if (Vector512.IsHardwareAccelerated) - { - int vectorEnd = width - Vector512.Count; - for (; column <= vectorEnd; column += Vector512.Count) - { - Vector512 firstVector = Vector512.LoadUnsafe(ref destinationReference, (nuint)column); - Vector512 secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column); - Vector512 maskVector = LoadMask512(ref maskReference, column); - Blend(firstVector, secondVector, maskVector).StoreUnsafe(ref destinationReference, (nuint)column); - } - } - - if (Vector256.IsHardwareAccelerated) - { - int vectorEnd = width - Vector256.Count; - for (; column <= vectorEnd; column += Vector256.Count) - { - Vector256 firstVector = Vector256.LoadUnsafe(ref destinationReference, (nuint)column); - Vector256 secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column); - Vector256 maskVector = LoadMask256(ref maskReference, column); - Blend(firstVector, secondVector, maskVector).StoreUnsafe(ref destinationReference, (nuint)column); - } - } - - if (Vector128.IsHardwareAccelerated) - { - int vectorEnd = width - Vector128.Count; - for (; column <= vectorEnd; column += Vector128.Count) - { - Vector128 firstVector = Vector128.LoadUnsafe(ref destinationReference, (nuint)column); - Vector128 secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column); - Vector128 maskVector = LoadMask128(ref maskReference, column); - Blend(firstVector, secondVector, maskVector).StoreUnsafe(ref destinationReference, (nuint)column); - } - } - - for (; column < width; column++) - { - int alpha = maskRow[column]; - destinationRow[column] = (ushort)(((alpha * destinationRow[column]) + ((MaximumMaskAlpha - alpha) * secondRow[column]) + 32) >> MaskWeightBits); - } - } - } - - /// - /// Combines two 8-bit predictors with display-distance weights without explicit hardware intrinsics. - /// - public static void DistanceWeightedScalar( - Span destination, - int destinationStride, - ReadOnlySpan second, - int secondStride, - int width, - int height, - int firstWeight, - int secondWeight) - { - for (int row = 0; row < height; row++) - { - Span destinationRow = destination.Slice(row * destinationStride, width); - ReadOnlySpan secondRow = second.Slice(row * secondStride, width); - for (int column = 0; column < width; column++) - { - destinationRow[column] = (byte)(((destinationRow[column] * firstWeight) + (secondRow[column] * secondWeight) + 8) >> DistanceWeightBits); - } - } - } - - /// - /// Blends two 8-bit predictors through an alpha mask without explicit hardware intrinsics. - /// - public static void BlendScalar( - Span destination, - int destinationStride, - ReadOnlySpan second, - int secondStride, - ReadOnlySpan mask, - int maskStride, - int width, - int height) - { - for (int row = 0; row < height; row++) - { - Span destinationRow = destination.Slice(row * destinationStride, width); - ReadOnlySpan secondRow = second.Slice(row * secondStride, width); - ReadOnlySpan maskRow = mask.Slice(row * maskStride, width); - for (int column = 0; column < width; column++) - { - int alpha = maskRow[column]; - destinationRow[column] = (byte)(((alpha * destinationRow[column]) + ((MaximumMaskAlpha - alpha) * secondRow[column]) + 32) >> MaskWeightBits); - } - } - } - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector128 DistanceWeighted(Vector128 first, Vector128 second, int firstWeight, int secondWeight) - { - Av1IntraPredictorBase.Widen(first, out Vector128 first0, out Vector128 first1, out Vector128 first2, out Vector128 first3); - Av1IntraPredictorBase.Widen(second, out Vector128 second0, out Vector128 second1, out Vector128 second2, out Vector128 second3); - return Av1IntraPredictorBase.Narrow( - DistanceWeighted(first0, second0, firstWeight, secondWeight), - DistanceWeighted(first1, second1, firstWeight, secondWeight), - DistanceWeighted(first2, second2, firstWeight, secondWeight), - DistanceWeighted(first3, second3, firstWeight, secondWeight)); - } - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector256 DistanceWeighted(Vector256 first, Vector256 second, int firstWeight, int secondWeight) - { - Av1IntraPredictorBase.Widen(first, out Vector256 first0, out Vector256 first1, out Vector256 first2, out Vector256 first3); - Av1IntraPredictorBase.Widen(second, out Vector256 second0, out Vector256 second1, out Vector256 second2, out Vector256 second3); - return Av1IntraPredictorBase.Narrow( - DistanceWeighted(first0, second0, firstWeight, secondWeight), - DistanceWeighted(first1, second1, firstWeight, secondWeight), - DistanceWeighted(first2, second2, firstWeight, secondWeight), - DistanceWeighted(first3, second3, firstWeight, secondWeight)); - } - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector512 DistanceWeighted(Vector512 first, Vector512 second, int firstWeight, int secondWeight) - { - Av1IntraPredictorBase.Widen(first, out Vector512 first0, out Vector512 first1, out Vector512 first2, out Vector512 first3); - Av1IntraPredictorBase.Widen(second, out Vector512 second0, out Vector512 second1, out Vector512 second2, out Vector512 second3); - return Av1IntraPredictorBase.Narrow( - DistanceWeighted(first0, second0, firstWeight, secondWeight), - DistanceWeighted(first1, second1, firstWeight, secondWeight), - DistanceWeighted(first2, second2, firstWeight, secondWeight), - DistanceWeighted(first3, second3, firstWeight, secondWeight)); - } - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector128 DistanceWeighted(Vector128 first, Vector128 second, int firstWeight, int secondWeight) - { - Av1IntraPredictorBase.Widen(first.AsInt16(), out Vector128 first0, out Vector128 first1); - Av1IntraPredictorBase.Widen(second.AsInt16(), out Vector128 second0, out Vector128 second1); - return Av1IntraPredictorBase.Narrow( - DistanceWeighted(first0, second0, firstWeight, secondWeight), - DistanceWeighted(first1, second1, firstWeight, secondWeight)).AsUInt16(); - } - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector256 DistanceWeighted(Vector256 first, Vector256 second, int firstWeight, int secondWeight) - { - Av1IntraPredictorBase.Widen(first.AsInt16(), out Vector256 first0, out Vector256 first1); - Av1IntraPredictorBase.Widen(second.AsInt16(), out Vector256 second0, out Vector256 second1); - return Av1IntraPredictorBase.Narrow( - DistanceWeighted(first0, second0, firstWeight, secondWeight), - DistanceWeighted(first1, second1, firstWeight, secondWeight)).AsUInt16(); - } - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector512 DistanceWeighted(Vector512 first, Vector512 second, int firstWeight, int secondWeight) - { - Av1IntraPredictorBase.Widen(first.AsInt16(), out Vector512 first0, out Vector512 first1); - Av1IntraPredictorBase.Widen(second.AsInt16(), out Vector512 second0, out Vector512 second1); - return Av1IntraPredictorBase.Narrow( - DistanceWeighted(first0, second0, firstWeight, secondWeight), - DistanceWeighted(first1, second1, firstWeight, secondWeight)).AsUInt16(); - } - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector128 DistanceWeighted(Vector128 first, Vector128 second, int firstWeight, int secondWeight) - => ((first * Vector128.Create(firstWeight)) + (second * Vector128.Create(secondWeight)) + Vector128.Create(8)) >> DistanceWeightBits; - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector256 DistanceWeighted(Vector256 first, Vector256 second, int firstWeight, int secondWeight) - => ((first * Vector256.Create(firstWeight)) + (second * Vector256.Create(secondWeight)) + Vector256.Create(8)) >> DistanceWeightBits; - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector512 DistanceWeighted(Vector512 first, Vector512 second, int firstWeight, int secondWeight) - => ((first * Vector512.Create(firstWeight)) + (second * Vector512.Create(secondWeight)) + Vector512.Create(8)) >> DistanceWeightBits; - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector128 Blend(Vector128 first, Vector128 second, Vector128 mask) - { - Av1IntraPredictorBase.Widen(first, out Vector128 first0, out Vector128 first1, out Vector128 first2, out Vector128 first3); - Av1IntraPredictorBase.Widen(second, out Vector128 second0, out Vector128 second1, out Vector128 second2, out Vector128 second3); - Av1IntraPredictorBase.Widen(mask, out Vector128 mask0, out Vector128 mask1, out Vector128 mask2, out Vector128 mask3); - return Av1IntraPredictorBase.Narrow( - Blend(first0, second0, mask0), - Blend(first1, second1, mask1), - Blend(first2, second2, mask2), - Blend(first3, second3, mask3)); - } - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector256 Blend(Vector256 first, Vector256 second, Vector256 mask) - { - Av1IntraPredictorBase.Widen(first, out Vector256 first0, out Vector256 first1, out Vector256 first2, out Vector256 first3); - Av1IntraPredictorBase.Widen(second, out Vector256 second0, out Vector256 second1, out Vector256 second2, out Vector256 second3); - Av1IntraPredictorBase.Widen(mask, out Vector256 mask0, out Vector256 mask1, out Vector256 mask2, out Vector256 mask3); - return Av1IntraPredictorBase.Narrow( - Blend(first0, second0, mask0), - Blend(first1, second1, mask1), - Blend(first2, second2, mask2), - Blend(first3, second3, mask3)); - } - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector512 Blend(Vector512 first, Vector512 second, Vector512 mask) - { - Av1IntraPredictorBase.Widen(first, out Vector512 first0, out Vector512 first1, out Vector512 first2, out Vector512 first3); - Av1IntraPredictorBase.Widen(second, out Vector512 second0, out Vector512 second1, out Vector512 second2, out Vector512 second3); - Av1IntraPredictorBase.Widen(mask, out Vector512 mask0, out Vector512 mask1, out Vector512 mask2, out Vector512 mask3); - return Av1IntraPredictorBase.Narrow( - Blend(first0, second0, mask0), - Blend(first1, second1, mask1), - Blend(first2, second2, mask2), - Blend(first3, second3, mask3)); - } - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector128 Blend(Vector128 first, Vector128 second, Vector128 mask) - { - Av1IntraPredictorBase.Widen(first.AsInt16(), out Vector128 first0, out Vector128 first1); - Av1IntraPredictorBase.Widen(second.AsInt16(), out Vector128 second0, out Vector128 second1); - Av1IntraPredictorBase.Widen(mask.AsInt16(), out Vector128 mask0, out Vector128 mask1); - return Av1IntraPredictorBase.Narrow(Blend(first0, second0, mask0), Blend(first1, second1, mask1)).AsUInt16(); - } - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector256 Blend(Vector256 first, Vector256 second, Vector256 mask) - { - Av1IntraPredictorBase.Widen(first.AsInt16(), out Vector256 first0, out Vector256 first1); - Av1IntraPredictorBase.Widen(second.AsInt16(), out Vector256 second0, out Vector256 second1); - Av1IntraPredictorBase.Widen(mask.AsInt16(), out Vector256 mask0, out Vector256 mask1); - return Av1IntraPredictorBase.Narrow(Blend(first0, second0, mask0), Blend(first1, second1, mask1)).AsUInt16(); - } - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector512 Blend(Vector512 first, Vector512 second, Vector512 mask) - { - Av1IntraPredictorBase.Widen(first.AsInt16(), out Vector512 first0, out Vector512 first1); - Av1IntraPredictorBase.Widen(second.AsInt16(), out Vector512 second0, out Vector512 second1); - Av1IntraPredictorBase.Widen(mask.AsInt16(), out Vector512 mask0, out Vector512 mask1); - return Av1IntraPredictorBase.Narrow(Blend(first0, second0, mask0), Blend(first1, second1, mask1)).AsUInt16(); - } - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector128 Blend(Vector128 first, Vector128 second, Vector128 mask) - => ((mask * first) + ((Vector128.Create(MaximumMaskAlpha) - mask) * second) + Vector128.Create(32)) >> MaskWeightBits; - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector256 Blend(Vector256 first, Vector256 second, Vector256 mask) - => ((mask * first) + ((Vector256.Create(MaximumMaskAlpha) - mask) * second) + Vector256.Create(32)) >> MaskWeightBits; - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector512 Blend(Vector512 first, Vector512 second, Vector512 mask) - => ((mask * first) + ((Vector512.Create(MaximumMaskAlpha) - mask) * second) + Vector512.Create(32)) >> MaskWeightBits; - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector128 LoadMask128(ref byte source, int offset) - { - Vector64 packed = Unsafe.As>(ref Unsafe.Add(ref source, offset)); - return Vector128.WidenLower(Vector128.Create(packed, Vector64.Zero)); - } - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector256 LoadMask256(ref byte source, int offset) - { - Vector128 packed = Vector128.LoadUnsafe(ref source, (nuint)offset); - return Vector256.WidenLower(Vector256.Create(packed, Vector128.Zero)); - } - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector512 LoadMask512(ref byte source, int offset) - { - Vector256 packed = Vector256.LoadUnsafe(ref source, (nuint)offset); - return Vector512.WidenLower(Vector512.Create(packed, Vector256.Zero)); - } -} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.Intermediate.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.Intermediate.cs deleted file mode 100644 index 0e126bbef..000000000 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.Intermediate.cs +++ /dev/null @@ -1,529 +0,0 @@ -// Copyright (c) Six Labors. -// Licensed under the Six Labors Split License. - -using System.Runtime.InteropServices; -using System.Runtime.Intrinsics; -using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; - -namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; - -/// -/// Combines high-precision compound convolution intermediates into reconstructed samples. -/// -internal static partial class Av1CompoundInterPredictor -{ - /// - /// Combines two compound intermediates by equal averaging. - /// - public static void AverageIntermediate( - Span destination, - int destinationStride, - ReadOnlySpan first, - int firstStride, - ReadOnlySpan second, - int secondStride, - int width, - int height, - int bitDepth) - { - GetIntermediateRounding(bitDepth, out int roundBits, out int roundOffset); - for (int row = 0; row < height; row++) - { - Span destinationRow = destination.Slice(row * destinationStride, width); - ReadOnlySpan firstRow = first.Slice(row * firstStride, width); - ReadOnlySpan secondRow = second.Slice(row * secondStride, width); - ref byte destinationReference = ref MemoryMarshal.GetReference(destinationRow); - ref ushort firstReference = ref MemoryMarshal.GetReference(firstRow); - ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow); - int column = 0; - - if (Vector128.IsHardwareAccelerated) - { - int vectorEnd = width - Vector128.Count; - for (; column <= vectorEnd; column += Vector128.Count) - { - Vector128 first0 = Vector128.LoadUnsafe(ref firstReference, (nuint)column); - Vector128 first1 = Vector128.LoadUnsafe( - ref firstReference, - (nuint)(column + Vector128.Count)); - - Vector128 second0 = Vector128.LoadUnsafe(ref secondReference, (nuint)column); - Vector128 second1 = Vector128.LoadUnsafe( - ref secondReference, - (nuint)(column + Vector128.Count)); - - AverageIntermediate(first0, first1, second0, second1, roundBits, roundOffset).StoreUnsafe( - ref destinationReference, - (nuint)column); - } - } - - for (; column < width; column++) - { - // The reference average deliberately truncates here. The sole rounding step follows bias removal, - // preventing the double rounding that occurs when each reference is first converted to pixels. - int result = ((firstRow[column] + secondRow[column]) >> 1) - roundOffset; - destinationRow[column] = (byte)Math.Clamp(RoundPowerOfTwo(result, roundBits), 0, byte.MaxValue); - } - } - } - - /// - /// Combines two compound intermediates using the decoded temporal-distance weights. - /// - public static void DistanceWeightedIntermediate( - Span destination, - int destinationStride, - ReadOnlySpan first, - int firstStride, - ReadOnlySpan second, - int secondStride, - int width, - int height, - int firstWeight, - int secondWeight, - int bitDepth) - { - GetIntermediateRounding(bitDepth, out int roundBits, out int roundOffset); - for (int row = 0; row < height; row++) - { - Span destinationRow = destination.Slice(row * destinationStride, width); - ReadOnlySpan firstRow = first.Slice(row * firstStride, width); - ReadOnlySpan secondRow = second.Slice(row * secondStride, width); - ref byte destinationReference = ref MemoryMarshal.GetReference(destinationRow); - ref ushort firstReference = ref MemoryMarshal.GetReference(firstRow); - ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow); - int column = 0; - - if (Vector128.IsHardwareAccelerated) - { - int vectorEnd = width - Vector128.Count; - for (; column <= vectorEnd; column += Vector128.Count) - { - Vector128 first0 = Vector128.LoadUnsafe(ref firstReference, (nuint)column); - Vector128 first1 = Vector128.LoadUnsafe( - ref firstReference, - (nuint)(column + Vector128.Count)); - - Vector128 second0 = Vector128.LoadUnsafe(ref secondReference, (nuint)column); - Vector128 second1 = Vector128.LoadUnsafe( - ref secondReference, - (nuint)(column + Vector128.Count)); - - DistanceWeightedIntermediate( - first0, - first1, - second0, - second1, - firstWeight, - secondWeight, - roundBits, - roundOffset).StoreUnsafe(ref destinationReference, (nuint)column); - } - } - - for (; column < width; column++) - { - int result = ((firstRow[column] * firstWeight) + (secondRow[column] * secondWeight)) >> DistanceWeightBits; - result -= roundOffset; - destinationRow[column] = (byte)Math.Clamp(RoundPowerOfTwo(result, roundBits), 0, byte.MaxValue); - } - } - } - - /// - /// Fills a luma-resolution difference-weighted mask from compound intermediates. - /// - public static void FillDifferenceWeightedIntermediateMask( - Span mask, - int maskStride, - ReadOnlySpan first, - int firstStride, - ReadOnlySpan second, - int secondStride, - int width, - int height, - int bitDepth, - Av1DifferenceWeightedMaskType maskType) - { - bool invert = maskType == Av1DifferenceWeightedMaskType.Type38Inverse; - GetIntermediateRounding(bitDepth, out int roundBits, out _); - int differenceRound = roundBits + bitDepth - 8; - for (int row = 0; row < height; row++) - { - Span maskRow = mask.Slice(row * maskStride, width); - ReadOnlySpan firstRow = first.Slice(row * firstStride, width); - ReadOnlySpan secondRow = second.Slice(row * secondStride, width); - ref byte maskReference = ref MemoryMarshal.GetReference(maskRow); - ref ushort firstReference = ref MemoryMarshal.GetReference(firstRow); - ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow); - int column = 0; - - if (Vector128.IsHardwareAccelerated) - { - int vectorEnd = width - Vector128.Count; - for (; column <= vectorEnd; column += Vector128.Count) - { - Vector128 first0 = Vector128.LoadUnsafe(ref firstReference, (nuint)column); - Vector128 first1 = Vector128.LoadUnsafe( - ref firstReference, - (nuint)(column + Vector128.Count)); - - Vector128 second0 = Vector128.LoadUnsafe(ref secondReference, (nuint)column); - Vector128 second1 = Vector128.LoadUnsafe( - ref secondReference, - (nuint)(column + Vector128.Count)); - - DifferenceWeightedIntermediate( - first0, - first1, - second0, - second1, - differenceRound, - invert).StoreUnsafe(ref maskReference, (nuint)column); - } - } - - for (; column < width; column++) - { - int difference = Math.Abs(firstRow[column] - secondRow[column]); - difference = RoundPowerOfTwo(difference, differenceRound); - int alpha = Math.Min(MaximumMaskAlpha, 38 + (difference >> 4)); - maskRow[column] = (byte)(invert ? MaximumMaskAlpha - alpha : alpha); - } - } - } - - /// - /// Blends two compound intermediates through a luma-resolution mask. - /// - public static void BlendIntermediate( - Span destination, - int destinationStride, - ReadOnlySpan first, - int firstStride, - ReadOnlySpan second, - int secondStride, - ReadOnlySpan mask, - int maskStride, - int width, - int height, - int subX, - int subY, - int bitDepth) - { - GetIntermediateRounding(bitDepth, out int roundBits, out int roundOffset); - for (int row = 0; row < height; row++) - { - Span destinationRow = destination.Slice(row * destinationStride, width); - ReadOnlySpan firstRow = first.Slice(row * firstStride, width); - ReadOnlySpan secondRow = second.Slice(row * secondStride, width); - ref byte destinationReference = ref MemoryMarshal.GetReference(destinationRow); - ref ushort firstReference = ref MemoryMarshal.GetReference(firstRow); - ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow); - int column = 0; - - if (Vector128.IsHardwareAccelerated && subX == 0 && subY == 0) - { - ref byte maskReference = ref MemoryMarshal.GetReference(mask); - int maskRowOffset = row * maskStride; - int vectorEnd = width - Vector128.Count; - for (; column <= vectorEnd; column += Vector128.Count) - { - Vector128 first0 = Vector128.LoadUnsafe(ref firstReference, (nuint)column); - Vector128 first1 = Vector128.LoadUnsafe( - ref firstReference, - (nuint)(column + Vector128.Count)); - - Vector128 second0 = Vector128.LoadUnsafe(ref secondReference, (nuint)column); - Vector128 second1 = Vector128.LoadUnsafe( - ref secondReference, - (nuint)(column + Vector128.Count)); - - Vector128 alpha = Vector128.LoadUnsafe( - ref maskReference, - (nuint)(maskRowOffset + column)); - - BlendIntermediate( - first0, - first1, - second0, - second1, - alpha, - roundBits, - roundOffset).StoreUnsafe(ref destinationReference, (nuint)column); - } - } - - for (; column < width; column++) - { - int alpha = GetSubsampledMaskAlpha(mask, maskStride, row, column, subX, subY); - - // Mask blending also truncates its Q6 result because final pixel rounding is still pending. Adding - // a half-unit here would produce a second rounding step and diverge from pinned libaom. - int result = ((alpha * firstRow[column]) + ((MaximumMaskAlpha - alpha) * secondRow[column])) >> MaskWeightBits; - result -= roundOffset; - destinationRow[column] = (byte)Math.Clamp(RoundPowerOfTwo(result, roundBits), 0, byte.MaxValue); - } - } - } - - /// - /// Equal-averages sixteen compound lanes and converts them to final 8-bit samples. - /// - private static Vector128 AverageIntermediate( - Vector128 first0, - Vector128 first1, - Vector128 second0, - Vector128 second1, - int roundBits, - int roundOffset) - => Vector128.Narrow( - FinalizeIntermediate( - (first0 & second0) + ((first0 ^ second0) >> 1), - roundBits, - roundOffset), - FinalizeIntermediate( - (first1 & second1) + ((first1 ^ second1) >> 1), - roundBits, - roundOffset)); - - /// - /// Distance-weights sixteen compound lanes and converts them to final 8-bit samples. - /// - private static Vector128 DistanceWeightedIntermediate( - Vector128 first0, - Vector128 first1, - Vector128 second0, - Vector128 second1, - int firstWeight, - int secondWeight, - int roundBits, - int roundOffset) - => Vector128.Narrow( - DistanceWeightedIntermediate(first0, second0, firstWeight, secondWeight, roundBits, roundOffset), - DistanceWeightedIntermediate(first1, second1, firstWeight, secondWeight, roundBits, roundOffset)); - - /// - /// Distance-weights eight compound lanes without overflowing the unsigned intermediate range. - /// - private static Vector128 DistanceWeightedIntermediate( - Vector128 first, - Vector128 second, - int firstWeight, - int secondWeight, - int roundBits, - int roundOffset) - { - Vector128 firstLower = Vector128.WidenLower(first).AsInt32(); - Vector128 firstUpper = Vector128.WidenUpper(first).AsInt32(); - Vector128 secondLower = Vector128.WidenLower(second).AsInt32(); - Vector128 secondUpper = Vector128.WidenUpper(second).AsInt32(); - Vector128 lower = - ((firstLower * firstWeight) + (secondLower * secondWeight)) >> DistanceWeightBits; - - Vector128 upper = - ((firstUpper * firstWeight) + (secondUpper * secondWeight)) >> DistanceWeightBits; - - return Vector128.Narrow( - FinalizeIntermediate(lower, roundBits, roundOffset), - FinalizeIntermediate(upper, roundBits, roundOffset)).AsUInt16(); - } - - /// - /// Creates sixteen difference-weighted mask values from compound intermediates. - /// - private static Vector128 DifferenceWeightedIntermediate( - Vector128 first0, - Vector128 first1, - Vector128 second0, - Vector128 second1, - int differenceRound, - bool invert) - => Vector128.Narrow( - DifferenceWeightedIntermediate(first0, second0, differenceRound, invert), - DifferenceWeightedIntermediate(first1, second1, differenceRound, invert)); - - /// - /// Creates eight difference-weighted mask values without losing the required pre-alpha rounding. - /// - private static Vector128 DifferenceWeightedIntermediate( - Vector128 first, - Vector128 second, - int differenceRound, - bool invert) - { - Vector128 difference = Vector128.Max(first, second) - Vector128.Min(first, second); - Vector128 lower = DifferenceWeightedIntermediate( - Vector128.WidenLower(difference).AsInt32(), - differenceRound, - invert); - - Vector128 upper = DifferenceWeightedIntermediate( - Vector128.WidenUpper(difference).AsInt32(), - differenceRound, - invert); - - return Vector128.Narrow(lower, upper).AsUInt16(); - } - - /// - /// Converts four intermediate differences to the decoded type-38 mask range. - /// - private static Vector128 DifferenceWeightedIntermediate( - Vector128 difference, - int differenceRound, - bool invert) - { - if (differenceRound != 0) - { - difference = (difference + Vector128.Create(1 << (differenceRound - 1))) >> differenceRound; - } - - Vector128 maximum = Vector128.Create(MaximumMaskAlpha); - Vector128 alpha = Vector128.Min(maximum, (difference >> 4) + Vector128.Create(38)); - return invert ? maximum - alpha : alpha; - } - - /// - /// Mask-blends sixteen compound lanes and converts them to final 8-bit samples. - /// - private static Vector128 BlendIntermediate( - Vector128 first0, - Vector128 first1, - Vector128 second0, - Vector128 second1, - Vector128 alpha, - int roundBits, - int roundOffset) - => Vector128.Narrow( - BlendIntermediate( - first0, - second0, - Vector128.WidenLower(alpha), - roundBits, - roundOffset), - BlendIntermediate( - first1, - second1, - Vector128.WidenUpper(alpha), - roundBits, - roundOffset)); - - /// - /// Mask-blends eight compound lanes after widening every product to signed 32-bit precision. - /// - private static Vector128 BlendIntermediate( - Vector128 first, - Vector128 second, - Vector128 alpha, - int roundBits, - int roundOffset) - { - Vector128 firstLower = Vector128.WidenLower(first).AsInt32(); - Vector128 firstUpper = Vector128.WidenUpper(first).AsInt32(); - Vector128 secondLower = Vector128.WidenLower(second).AsInt32(); - Vector128 secondUpper = Vector128.WidenUpper(second).AsInt32(); - Vector128 alphaLower = Vector128.WidenLower(alpha).AsInt32(); - Vector128 alphaUpper = Vector128.WidenUpper(alpha).AsInt32(); - Vector128 maximum = Vector128.Create(MaximumMaskAlpha); - Vector128 lower = - ((alphaLower * firstLower) + ((maximum - alphaLower) * secondLower)) >> MaskWeightBits; - - Vector128 upper = - ((alphaUpper * firstUpper) + ((maximum - alphaUpper) * secondUpper)) >> MaskWeightBits; - - return Vector128.Narrow( - FinalizeIntermediate(lower, roundBits, roundOffset), - FinalizeIntermediate(upper, roundBits, roundOffset)).AsUInt16(); - } - - /// - /// Removes the compound bias and final fractional precision from eight unsigned lanes. - /// - private static Vector128 FinalizeIntermediate( - Vector128 value, - int roundBits, - int roundOffset) - { - Vector128 result = (value - Vector128.Create((ushort)roundOffset)).AsInt16(); - if (roundBits != 0) - { - result = (result + Vector128.Create((short)(1 << (roundBits - 1)))) >> roundBits; - } - - result = Vector128.Max(Vector128.Zero, Vector128.Min(Vector128.Create((short)byte.MaxValue), result)); - return result.AsUInt16(); - } - - /// - /// Removes the compound bias and final fractional precision from four widened lanes. - /// - private static Vector128 FinalizeIntermediate( - Vector128 value, - int roundBits, - int roundOffset) - { - Vector128 result = value - Vector128.Create(roundOffset); - if (roundBits != 0) - { - result = (result + Vector128.Create(1 << (roundBits - 1))) >> roundBits; - } - - return Vector128.Max(Vector128.Zero, Vector128.Min(Vector128.Create((int)byte.MaxValue), result)); - } - - /// - /// Gets the mask alpha for one plane sample, averaging its two or four luma samples when required. - /// - private static int GetSubsampledMaskAlpha( - ReadOnlySpan mask, - int maskStride, - int row, - int column, - int subX, - int subY) - { - int maskRow = row << subY; - int maskColumn = column << subX; - int alpha = mask[(maskRow * maskStride) + maskColumn]; - if (subX != 0) - { - alpha += mask[(maskRow * maskStride) + maskColumn + 1]; - } - - if (subY != 0) - { - int lowerOffset = ((maskRow + 1) * maskStride) + maskColumn; - alpha += mask[lowerOffset]; - if (subX != 0) - { - alpha += mask[lowerOffset + 1]; - } - } - - int sampleCountShift = subX + subY; - return sampleCountShift == 0 - ? alpha - : RoundPowerOfTwo(alpha, sampleCountShift); - } - - /// - /// Derives the bias and remaining fractional precision of a compound intermediate. - /// - private static void GetIntermediateRounding(int bitDepth, out int roundBits, out int roundOffset) - { - int intermediateRange = bitDepth + 7 - 3 + 2; - int round0 = 3 + Math.Max(intermediateRange - 16, 0); - int offsetBits = bitDepth + 14 - round0; - roundBits = 14 - round0 - Av1InterPredictor.CompoundRound1Bits; - roundOffset = (1 << (offsetBits - Av1InterPredictor.CompoundRound1Bits)) + - (1 << (offsetBits - Av1InterPredictor.CompoundRound1Bits - 1)); - } - - /// - /// Applies AV1's positive power-of-two rounding rule. - /// - private static int RoundPowerOfTwo(int value, int bits) - => bits == 0 ? value : (value + (1 << (bits - 1))) >> bits; -} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.Masks.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.Masks.cs deleted file mode 100644 index d1224cf3d..000000000 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.Masks.cs +++ /dev/null @@ -1,294 +0,0 @@ -// Copyright (c) Six Labors. -// Licensed under the Six Labors Split License. - -using System.Runtime.CompilerServices; -using System.Runtime.InteropServices; -using System.Runtime.Intrinsics; -using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; - -namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; - -/// -/// Produces the smooth inter-intra and predictor-difference masks used by compound blending. -/// -internal static partial class Av1CompoundInterPredictor -{ - /// - /// Gets libaom's one-dimensional inter-intra alpha curve. - /// - private static ReadOnlySpan InterIntraWeights => - [ - 60, 58, 56, 54, 52, 50, 48, 47, 45, 44, 42, 41, 39, 38, 37, 35, - 34, 33, 32, 31, 30, 29, 28, 27, 26, 25, 24, 23, 22, 22, 21, 20, - 19, 19, 18, 18, 17, 16, 16, 15, 15, 14, 14, 13, 13, 12, 12, 12, - 11, 11, 10, 10, 10, 9, 9, 9, 8, 8, 8, 8, 7, 7, 7, 7, - 6, 6, 6, 6, 6, 5, 5, 5, 5, 5, 4, 4, 4, 4, 4, 4, - 4, 4, 4, 4, 3, 3, 3, 3, 3, 3, 3, 3, 3, 2, 2, 2, - 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, - 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, - ]; - - /// - /// Fills a smooth inter-intra mask for one plane. - /// - public static void FillInterIntraMask( - Span mask, - int maskStride, - int width, - int height, - Av1InterIntraMode mode, - bool invert) - { - int sizeScale = 128 / Math.Max(width, height); - for (int row = 0; row < height; row++) - { - Span maskRow = mask.Slice(row * maskStride, width); - for (int column = 0; column < width; column++) - { - int alpha = mode switch - { - Av1InterIntraMode.Vertical => InterIntraWeights[row * sizeScale], - Av1InterIntraMode.Horizontal => InterIntraWeights[column * sizeScale], - Av1InterIntraMode.Smooth => InterIntraWeights[Math.Min(row, column) * sizeScale], - _ => 32, - }; - - maskRow[column] = (byte)(invert ? MaximumMaskAlpha - alpha : alpha); - } - } - } - - /// - /// Fills an 8-bit difference-weighted compound mask. - /// - public static void FillDifferenceWeightedMask( - Span mask, - int maskStride, - ReadOnlySpan first, - int firstStride, - ReadOnlySpan second, - int secondStride, - int width, - int height, - Av1DifferenceWeightedMaskType maskType) - { - bool invert = maskType == Av1DifferenceWeightedMaskType.Type38Inverse; - for (int row = 0; row < height; row++) - { - Span maskRow = mask.Slice(row * maskStride, width); - ReadOnlySpan firstRow = first.Slice(row * firstStride, width); - ReadOnlySpan secondRow = second.Slice(row * secondStride, width); - ref byte maskReference = ref MemoryMarshal.GetReference(maskRow); - ref byte firstReference = ref MemoryMarshal.GetReference(firstRow); - ref byte secondReference = ref MemoryMarshal.GetReference(secondRow); - int column = 0; - - if (Vector512.IsHardwareAccelerated) - { - int vectorEnd = width - Vector512.Count; - for (; column <= vectorEnd; column += Vector512.Count) - { - Vector512 firstVector = Vector512.LoadUnsafe(ref firstReference, (nuint)column); - Vector512 secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column); - DifferenceWeighted(firstVector, secondVector, 4, invert).StoreUnsafe(ref maskReference, (nuint)column); - } - } - - if (Vector256.IsHardwareAccelerated) - { - int vectorEnd = width - Vector256.Count; - for (; column <= vectorEnd; column += Vector256.Count) - { - Vector256 firstVector = Vector256.LoadUnsafe(ref firstReference, (nuint)column); - Vector256 secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column); - DifferenceWeighted(firstVector, secondVector, 4, invert).StoreUnsafe(ref maskReference, (nuint)column); - } - } - - if (Vector128.IsHardwareAccelerated) - { - int vectorEnd = width - Vector128.Count; - for (; column <= vectorEnd; column += Vector128.Count) - { - Vector128 firstVector = Vector128.LoadUnsafe(ref firstReference, (nuint)column); - Vector128 secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column); - DifferenceWeighted(firstVector, secondVector, 4, invert).StoreUnsafe(ref maskReference, (nuint)column); - } - } - - for (; column < width; column++) - { - int difference = Math.Abs(firstRow[column] - secondRow[column]) >> 4; - int alpha = Math.Min(MaximumMaskAlpha, 38 + difference); - maskRow[column] = (byte)(invert ? MaximumMaskAlpha - alpha : alpha); - } - } - } - - /// - /// Fills a high-bit-depth difference-weighted compound mask. - /// - public static void FillDifferenceWeightedMask( - Span mask, - int maskStride, - ReadOnlySpan first, - int firstStride, - ReadOnlySpan second, - int secondStride, - int width, - int height, - int bitDepth, - Av1DifferenceWeightedMaskType maskType) - { - bool invert = maskType == Av1DifferenceWeightedMaskType.Type38Inverse; - int differenceShift = bitDepth - 8 + 4; - for (int row = 0; row < height; row++) - { - Span maskRow = mask.Slice(row * maskStride, width); - ReadOnlySpan firstRow = first.Slice(row * firstStride, width); - ReadOnlySpan secondRow = second.Slice(row * secondStride, width); - ref byte maskReference = ref MemoryMarshal.GetReference(maskRow); - ref ushort firstReference = ref MemoryMarshal.GetReference(firstRow); - ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow); - int column = 0; - - // Two input vectors narrow to one packed byte mask. This keeps mask construction contiguous and avoids - // temporary buffers before the following vector blend consumes the complete plane block. - if (Vector512.IsHardwareAccelerated) - { - int vectorEnd = width - Vector512.Count; - for (; column <= vectorEnd; column += Vector512.Count) - { - Vector512 first0 = Vector512.LoadUnsafe(ref firstReference, (nuint)column); - Vector512 first1 = Vector512.LoadUnsafe(ref firstReference, (nuint)(column + Vector512.Count)); - Vector512 second0 = Vector512.LoadUnsafe(ref secondReference, (nuint)column); - Vector512 second1 = Vector512.LoadUnsafe(ref secondReference, (nuint)(column + Vector512.Count)); - DifferenceWeighted(first0, first1, second0, second1, differenceShift, invert) - .StoreUnsafe(ref maskReference, (nuint)column); - } - } - - if (Vector256.IsHardwareAccelerated) - { - int vectorEnd = width - Vector256.Count; - for (; column <= vectorEnd; column += Vector256.Count) - { - Vector256 first0 = Vector256.LoadUnsafe(ref firstReference, (nuint)column); - Vector256 first1 = Vector256.LoadUnsafe(ref firstReference, (nuint)(column + Vector256.Count)); - Vector256 second0 = Vector256.LoadUnsafe(ref secondReference, (nuint)column); - Vector256 second1 = Vector256.LoadUnsafe(ref secondReference, (nuint)(column + Vector256.Count)); - DifferenceWeighted(first0, first1, second0, second1, differenceShift, invert) - .StoreUnsafe(ref maskReference, (nuint)column); - } - } - - if (Vector128.IsHardwareAccelerated) - { - int vectorEnd = width - Vector128.Count; - for (; column <= vectorEnd; column += Vector128.Count) - { - Vector128 first0 = Vector128.LoadUnsafe(ref firstReference, (nuint)column); - Vector128 first1 = Vector128.LoadUnsafe(ref firstReference, (nuint)(column + Vector128.Count)); - Vector128 second0 = Vector128.LoadUnsafe(ref secondReference, (nuint)column); - Vector128 second1 = Vector128.LoadUnsafe(ref secondReference, (nuint)(column + Vector128.Count)); - DifferenceWeighted(first0, first1, second0, second1, differenceShift, invert) - .StoreUnsafe(ref maskReference, (nuint)column); - } - } - - for (; column < width; column++) - { - int difference = Math.Abs(firstRow[column] - secondRow[column]) >> differenceShift; - int alpha = Math.Min(MaximumMaskAlpha, 38 + difference); - maskRow[column] = (byte)(invert ? MaximumMaskAlpha - alpha : alpha); - } - } - } - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector128 DifferenceWeighted(Vector128 first, Vector128 second, int shift, bool invert) - { - Vector128 difference = Vector128.Max(first, second) - Vector128.Min(first, second); - Vector128 lower = DifferenceWeightedAlpha(Vector128.WidenLower(difference), shift, invert); - Vector128 upper = DifferenceWeightedAlpha(Vector128.WidenUpper(difference), shift, invert); - return Vector128.Narrow(lower, upper); - } - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector256 DifferenceWeighted(Vector256 first, Vector256 second, int shift, bool invert) - { - Vector256 difference = Vector256.Max(first, second) - Vector256.Min(first, second); - Vector256 lower = DifferenceWeightedAlpha(Vector256.WidenLower(difference), shift, invert); - Vector256 upper = DifferenceWeightedAlpha(Vector256.WidenUpper(difference), shift, invert); - return Vector256.Narrow(lower, upper); - } - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector512 DifferenceWeighted(Vector512 first, Vector512 second, int shift, bool invert) - { - Vector512 difference = Vector512.Max(first, second) - Vector512.Min(first, second); - Vector512 lower = DifferenceWeightedAlpha(Vector512.WidenLower(difference), shift, invert); - Vector512 upper = DifferenceWeightedAlpha(Vector512.WidenUpper(difference), shift, invert); - return Vector512.Narrow(lower, upper); - } - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector128 DifferenceWeighted( - Vector128 first0, - Vector128 first1, - Vector128 second0, - Vector128 second1, - int shift, - bool invert) - => Vector128.Narrow( - DifferenceWeightedAlpha(Vector128.Max(first0, second0) - Vector128.Min(first0, second0), shift, invert), - DifferenceWeightedAlpha(Vector128.Max(first1, second1) - Vector128.Min(first1, second1), shift, invert)); - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector256 DifferenceWeighted( - Vector256 first0, - Vector256 first1, - Vector256 second0, - Vector256 second1, - int shift, - bool invert) - => Vector256.Narrow( - DifferenceWeightedAlpha(Vector256.Max(first0, second0) - Vector256.Min(first0, second0), shift, invert), - DifferenceWeightedAlpha(Vector256.Max(first1, second1) - Vector256.Min(first1, second1), shift, invert)); - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector512 DifferenceWeighted( - Vector512 first0, - Vector512 first1, - Vector512 second0, - Vector512 second1, - int shift, - bool invert) - => Vector512.Narrow( - DifferenceWeightedAlpha(Vector512.Max(first0, second0) - Vector512.Min(first0, second0), shift, invert), - DifferenceWeightedAlpha(Vector512.Max(first1, second1) - Vector512.Min(first1, second1), shift, invert)); - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector128 DifferenceWeightedAlpha(Vector128 difference, int shift, bool invert) - { - Vector128 maximum = Vector128.Create((ushort)MaximumMaskAlpha); - Vector128 alpha = Vector128.Min(maximum, (difference >> shift) + Vector128.Create((ushort)38)); - return invert ? maximum - alpha : alpha; - } - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector256 DifferenceWeightedAlpha(Vector256 difference, int shift, bool invert) - { - Vector256 maximum = Vector256.Create((ushort)MaximumMaskAlpha); - Vector256 alpha = Vector256.Min(maximum, (difference >> shift) + Vector256.Create((ushort)38)); - return invert ? maximum - alpha : alpha; - } - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector512 DifferenceWeightedAlpha(Vector512 difference, int shift, bool invert) - { - Vector512 maximum = Vector512.Create((ushort)MaximumMaskAlpha); - Vector512 alpha = Vector512.Min(maximum, (difference >> shift) + Vector512.Create((ushort)38)); - return invert ? maximum - alpha : alpha; - } -} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.Operator.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.Operator.cs new file mode 100644 index 000000000..72704cc80 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.Operator.cs @@ -0,0 +1,344 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.Intrinsics; + +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; + +/// +/// Defines biased compound-prediction conversion arithmetic. +/// +internal static partial class Av1CompoundInterPredictor +{ + /// + /// Defines biased compound-prediction conversion for scalar and SIMD lane groups. + /// + private interface IAv1CompoundPredictionOperator + { + /// + /// Converts one integer-position sample to the compound intermediate representation. + /// + /// The source sample. + /// The final reconstruction shift. + /// The compound intermediate bias. + /// The biased compound intermediate. + public static abstract ushort Copy(byte sample, int roundBits, int roundOffset); + + /// + /// Converts 128 bits of integer-position samples to compound intermediates. + /// + /// The source samples. + /// The final reconstruction shift. + /// The compound intermediate bias. + /// Receives the lower widened intermediates. + /// Receives the upper widened intermediates. + public static abstract void Copy( + Vector128 samples, + int roundBits, + int roundOffset, + out Vector128 lower, + out Vector128 upper); + + /// + /// Converts 256 bits of integer-position samples to compound intermediates. + /// + /// The source samples. + /// The final reconstruction shift. + /// The compound intermediate bias. + /// Receives the lower widened intermediates. + /// Receives the upper widened intermediates. + public static abstract void Copy( + Vector256 samples, + int roundBits, + int roundOffset, + out Vector256 lower, + out Vector256 upper); + + /// + /// Converts 512 bits of integer-position samples to compound intermediates. + /// + /// The source samples. + /// The final reconstruction shift. + /// The compound intermediate bias. + /// Receives the lower widened intermediates. + /// Receives the upper widened intermediates. + public static abstract void Copy( + Vector512 samples, + int roundBits, + int roundOffset, + out Vector512 lower, + out Vector512 upper); + + /// + /// Applies direct-filter rounding and bias to one convolution result. + /// + /// The convolution result. + /// The shift applied before rounding. + /// The rounding shift. + /// The compound intermediate bias. + /// The biased compound intermediate. + public static abstract ushort PrepareDirect(int result, int preShift, int round, int roundOffset); + + /// + /// Applies direct-filter rounding and bias to 128-bit widened convolution results. + /// + /// The lower convolution results. + /// The upper convolution results. + /// The shift applied before rounding. + /// The rounding shift. + /// The compound intermediate bias. + /// The biased compound intermediates. + public static abstract Vector128 PrepareDirect( + Vector128 lower, + Vector128 upper, + int preShift, + int round, + int roundOffset); + + /// + /// Applies direct-filter rounding and bias to 256-bit widened convolution results. + /// + /// The lower convolution results. + /// The upper convolution results. + /// The shift applied before rounding. + /// The rounding shift. + /// The compound intermediate bias. + /// The biased compound intermediates. + public static abstract Vector256 PrepareDirect( + Vector256 lower, + Vector256 upper, + int preShift, + int round, + int roundOffset); + + /// + /// Applies direct-filter rounding and bias to 512-bit widened convolution results. + /// + /// The lower convolution results. + /// The upper convolution results. + /// The shift applied before rounding. + /// The rounding shift. + /// The compound intermediate bias. + /// The biased compound intermediates. + public static abstract Vector512 PrepareDirect( + Vector512 lower, + Vector512 upper, + int preShift, + int round, + int roundOffset); + + /// + /// Applies first-pass compound rounding to one biased horizontal convolution result. + /// + /// The biased horizontal convolution result. + /// The rounded intermediate. + public static abstract short PrepareHorizontal(int result); + + /// + /// Applies first-pass compound rounding to 128-bit widened horizontal convolution results. + /// + /// The lower convolution results. + /// The upper convolution results. + /// The rounded intermediates. + public static abstract Vector128 PrepareHorizontal(Vector128 lower, Vector128 upper); + + /// + /// Applies first-pass compound rounding to 256-bit widened horizontal convolution results. + /// + /// The lower convolution results. + /// The upper convolution results. + /// The rounded intermediates. + public static abstract Vector256 PrepareHorizontal(Vector256 lower, Vector256 upper); + + /// + /// Applies first-pass compound rounding to 512-bit widened horizontal convolution results. + /// + /// The lower convolution results. + /// The upper convolution results. + /// The rounded intermediates. + public static abstract Vector512 PrepareHorizontal(Vector512 lower, Vector512 upper); + + /// + /// Applies second-pass compound rounding to one biased vertical convolution result. + /// + /// The biased vertical convolution result. + /// The compound intermediate. + public static abstract ushort PrepareVertical(int result); + + /// + /// Applies second-pass compound rounding to 128-bit widened vertical convolution results. + /// + /// The lower convolution results. + /// The upper convolution results. + /// The compound intermediates. + public static abstract Vector128 PrepareVertical(Vector128 lower, Vector128 upper); + + /// + /// Applies second-pass compound rounding to 256-bit widened vertical convolution results. + /// + /// The lower convolution results. + /// The upper convolution results. + /// The compound intermediates. + public static abstract Vector256 PrepareVertical(Vector256 lower, Vector256 upper); + + /// + /// Applies second-pass compound rounding to 512-bit widened vertical convolution results. + /// + /// The lower convolution results. + /// The upper convolution results. + /// The compound intermediates. + public static abstract Vector512 PrepareVertical(Vector512 lower, Vector512 upper); + } + + /// + /// Implements AV1 compound-prediction conversion for scalar and SIMD lane groups. + /// + private readonly struct CompoundPredictionOperator : IAv1CompoundPredictionOperator + { + private const int HorizontalBias = 1 << (8 + FilterBits - 1); + private const int VerticalBias = 1 << (8 + (2 * FilterBits) - Round0Bits); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static ushort Copy(byte sample, int roundBits, int roundOffset) + => (ushort)((sample << roundBits) + roundOffset); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Copy( + Vector128 samples, + int roundBits, + int roundOffset, + out Vector128 lower, + out Vector128 upper) + { + Vector128 offset = Vector128.Create((ushort)roundOffset); + lower = (Vector128.WidenLower(samples) << roundBits) + offset; + upper = (Vector128.WidenUpper(samples) << roundBits) + offset; + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Copy( + Vector256 samples, + int roundBits, + int roundOffset, + out Vector256 lower, + out Vector256 upper) + { + Vector256 offset = Vector256.Create((ushort)roundOffset); + lower = (Vector256.WidenLower(samples) << roundBits) + offset; + upper = (Vector256.WidenUpper(samples) << roundBits) + offset; + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Copy( + Vector512 samples, + int roundBits, + int roundOffset, + out Vector512 lower, + out Vector512 upper) + { + Vector512 offset = Vector512.Create((ushort)roundOffset); + lower = (Vector512.WidenLower(samples) << roundBits) + offset; + upper = (Vector512.WidenUpper(samples) << roundBits) + offset; + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static ushort PrepareDirect(int result, int preShift, int round, int roundOffset) + => (ushort)(RoundPowerOfTwo(result << preShift, round) + roundOffset); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector128 PrepareDirect( + Vector128 lower, + Vector128 upper, + int preShift, + int round, + int roundOffset) + => Av1IntraPredictorBase.Narrow( + RoundPowerOfTwo(lower << preShift, round) + Vector128.Create(roundOffset), + RoundPowerOfTwo(upper << preShift, round) + Vector128.Create(roundOffset)).AsUInt16(); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector256 PrepareDirect( + Vector256 lower, + Vector256 upper, + int preShift, + int round, + int roundOffset) + => Av1IntraPredictorBase.Narrow( + RoundPowerOfTwo(lower << preShift, round) + Vector256.Create(roundOffset), + RoundPowerOfTwo(upper << preShift, round) + Vector256.Create(roundOffset)).AsUInt16(); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector512 PrepareDirect( + Vector512 lower, + Vector512 upper, + int preShift, + int round, + int roundOffset) + => Av1IntraPredictorBase.Narrow( + RoundPowerOfTwo(lower << preShift, round) + Vector512.Create(roundOffset), + RoundPowerOfTwo(upper << preShift, round) + Vector512.Create(roundOffset)).AsUInt16(); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static short PrepareHorizontal(int result) + => (short)RoundPowerOfTwo(HorizontalBias + result, Round0Bits); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector128 PrepareHorizontal(Vector128 lower, Vector128 upper) + => Av1IntraPredictorBase.Narrow( + RoundPowerOfTwo(lower + Vector128.Create(HorizontalBias), Round0Bits), + RoundPowerOfTwo(upper + Vector128.Create(HorizontalBias), Round0Bits)); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector256 PrepareHorizontal(Vector256 lower, Vector256 upper) + => Av1IntraPredictorBase.Narrow( + RoundPowerOfTwo(lower + Vector256.Create(HorizontalBias), Round0Bits), + RoundPowerOfTwo(upper + Vector256.Create(HorizontalBias), Round0Bits)); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector512 PrepareHorizontal(Vector512 lower, Vector512 upper) + => Av1IntraPredictorBase.Narrow( + RoundPowerOfTwo(lower + Vector512.Create(HorizontalBias), Round0Bits), + RoundPowerOfTwo(upper + Vector512.Create(HorizontalBias), Round0Bits)); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static ushort PrepareVertical(int result) + => (ushort)RoundPowerOfTwo(VerticalBias + result, CompoundRound1Bits); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector128 PrepareVertical(Vector128 lower, Vector128 upper) + => Av1IntraPredictorBase.Narrow( + RoundPowerOfTwo(lower + Vector128.Create(VerticalBias), CompoundRound1Bits), + RoundPowerOfTwo(upper + Vector128.Create(VerticalBias), CompoundRound1Bits)).AsUInt16(); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector256 PrepareVertical(Vector256 lower, Vector256 upper) + => Av1IntraPredictorBase.Narrow( + RoundPowerOfTwo(lower + Vector256.Create(VerticalBias), CompoundRound1Bits), + RoundPowerOfTwo(upper + Vector256.Create(VerticalBias), CompoundRound1Bits)).AsUInt16(); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector512 PrepareVertical(Vector512 lower, Vector512 upper) + => Av1IntraPredictorBase.Narrow( + RoundPowerOfTwo(lower + Vector512.Create(VerticalBias), CompoundRound1Bits), + RoundPowerOfTwo(upper + Vector512.Create(VerticalBias), CompoundRound1Bits)).AsUInt16(); + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.cs index 25685890a..ede6a8eed 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundInterPredictor.cs @@ -5,241 +5,606 @@ using System.Runtime.CompilerServices; using System.Runtime.InteropServices; using System.Runtime.Intrinsics; +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor; + namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; -/// -/// Combines two AV1 inter predictors with equal-weight rounded averaging. -/// +/// +/// Produces the biased high-precision intermediates required by compound inter prediction. +/// internal static partial class Av1CompoundInterPredictor { /// - /// Averages an 8-bit predictor into an existing prediction block. + /// The second-round shift retained by every compound convolution path. + /// + internal const int CompoundRound1Bits = 7; + + /// + /// The fixed-point precision used by AV1 distance weights. + /// + internal const int DistanceWeightBits = 4; + + /// + /// The fixed-point precision used by AV1 compound masks. + /// + internal const int MaskWeightBits = 6; + + /// + /// The inclusive upper bound for an AV1 compound-mask alpha value. + /// + internal const int MaximumMaskAlpha = 1 << MaskWeightBits; + + /// + /// Reconstructs one 8-bit translational reference into AV1's unsigned compound intermediate format. + /// + public static void PredictCompound( + ReadOnlySpan source, + int sourceStride, + int sourceOrigin, + Span destination, + int destinationStride, + int width, + int height, + Av1InterpolationFilter horizontalFilter, + Av1InterpolationFilter verticalFilter, + int horizontalPhase, + int verticalPhase, + Span scratch) + => PredictCompound( + source, + sourceStride, + sourceOrigin, + destination, + destinationStride, + width, + height, + horizontalFilter, + verticalFilter, + horizontalPhase, + verticalPhase, + scratch, + useSimd: true); + + /// + /// Executes one closed compound-prediction conversion operator. + /// + /// The compound-prediction conversion operator. + private static void PredictCompound( + ReadOnlySpan source, + int sourceStride, + int sourceOrigin, + Span destination, + int destinationStride, + int width, + int height, + Av1InterpolationFilter horizontalFilter, + Av1InterpolationFilter verticalFilter, + int horizontalPhase, + int verticalPhase, + Span scratch, + bool useSimd) + where TOperator : struct, IAv1CompoundPredictionOperator + { + ReadOnlySpan horizontalCoefficients = GetCompoundCoefficients(horizontalFilter, horizontalPhase, width <= 4); + ReadOnlySpan verticalCoefficients = GetCompoundCoefficients(verticalFilter, verticalPhase, height <= 4); + int roundBits = (2 * FilterBits) - Round0Bits - CompoundRound1Bits; + int offsetBits = 8 + (2 * FilterBits) - Round0Bits; + int roundOffset = (1 << (offsetBits - CompoundRound1Bits)) + + (1 << (offsetBits - CompoundRound1Bits - 1)); + + if (horizontalPhase == 0 && verticalPhase == 0) + { + CopyCompound( + source, + sourceStride, + sourceOrigin, + destination, + destinationStride, + width, + height, + roundBits, + roundOffset, + useSimd); + + return; + } + + if (verticalPhase == 0) + { + GetEffectiveKernel(horizontalCoefficients, out int firstCoefficient, out int tapCount); + FilterCompoundDirect( + source, + sourceStride, + sourceOrigin, + destination, + destinationStride, + width, + height, + horizontalCoefficients[firstCoefficient..], + tapCount, + firstCoefficient - 3, + tapStride: 1, + preShift: 0, + round: Round0Bits, + roundOffset, + useSimd); + + return; + } + + if (horizontalPhase == 0) + { + GetEffectiveKernel(verticalCoefficients, out int firstCoefficient, out int tapCount); + FilterCompoundDirect( + source, + sourceStride, + sourceOrigin, + destination, + destinationStride, + width, + height, + verticalCoefficients[firstCoefficient..], + tapCount, + (firstCoefficient - 3) * sourceStride, + sourceStride, + FilterBits - Round0Bits, + CompoundRound1Bits, + roundOffset, + useSimd); + + return; + } + + GetEffectiveKernel(horizontalCoefficients, out int firstHorizontalCoefficient, out int horizontalTapCount); + GetEffectiveKernel(verticalCoefficients, out int firstVerticalCoefficient, out int verticalTapCount); + FilterCompound2D( + source, + sourceStride, + sourceOrigin, + destination, + destinationStride, + width, + height, + horizontalCoefficients[firstHorizontalCoefficient..], + horizontalTapCount, + firstHorizontalCoefficient - 3, + verticalCoefficients[firstVerticalCoefficient..], + verticalTapCount, + firstVerticalCoefficient - 3, + scratch, + useSimd); + } + + /// + /// Reconstructs one compound intermediate without explicit hardware intrinsics. /// - /// The first predictor and combined output. - /// The distance between destination rows in samples. - /// The second predictor. - /// The distance between second-predictor rows in samples. - /// The active block width. - /// The active block height. - public static void Average( - Span destination, + public static void PredictCompoundScalar( + ReadOnlySpan source, + int sourceStride, + int sourceOrigin, + Span destination, int destinationStride, - ReadOnlySpan second, - int secondStride, int width, - int height) + int height, + Av1InterpolationFilter horizontalFilter, + Av1InterpolationFilter verticalFilter, + int horizontalPhase, + int verticalPhase, + Span scratch) + => PredictCompound( + source, + sourceStride, + sourceOrigin, + destination, + destinationStride, + width, + height, + horizontalFilter, + verticalFilter, + horizontalPhase, + verticalPhase, + scratch, + useSimd: false); + + /// + /// Copies integer-position samples through one closed compound-prediction operator. + /// + /// The compound-prediction conversion operator. + private static void CopyCompound( + ReadOnlySpan source, + int sourceStride, + int sourceOrigin, + Span destination, + int destinationStride, + int width, + int height, + int roundBits, + int roundOffset, + bool useSimd) + where TOperator : struct, IAv1CompoundPredictionOperator { + ref byte sourceBase = ref Unsafe.Add(ref MemoryMarshal.GetReference(source), sourceOrigin); + ref ushort destinationBase = ref MemoryMarshal.GetReference(destination); + for (int row = 0; row < height; row++) { - Span destinationRow = destination.Slice(row * destinationStride, width); - ReadOnlySpan secondRow = second.Slice(row * secondStride, width); - ref byte destinationReference = ref MemoryMarshal.GetReference(destinationRow); - ref byte secondReference = ref MemoryMarshal.GetReference(secondRow); + ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride); + ref ushort destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); int column = 0; - if (Vector512.IsHardwareAccelerated) + if (useSimd && Vector512.IsHardwareAccelerated) { int vectorEnd = width - Vector512.Count; for (; column <= vectorEnd; column += Vector512.Count) { - Vector512 firstVector = Vector512.LoadUnsafe(ref destinationReference, (nuint)column); - Vector512 secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column); - Average(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column); + Vector512 samples = Vector512.LoadUnsafe(ref sourceRow, (nuint)column); + TOperator.Copy(samples, roundBits, roundOffset, out Vector512 lower, out Vector512 upper); + lower.StoreUnsafe(ref destinationRow, (nuint)column); + upper.StoreUnsafe(ref destinationRow, (nuint)(column + Vector512.Count)); } } - if (Vector256.IsHardwareAccelerated) + if (useSimd && Vector256.IsHardwareAccelerated) { int vectorEnd = width - Vector256.Count; for (; column <= vectorEnd; column += Vector256.Count) { - Vector256 firstVector = Vector256.LoadUnsafe(ref destinationReference, (nuint)column); - Vector256 secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column); - Average(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column); + Vector256 samples = Vector256.LoadUnsafe(ref sourceRow, (nuint)column); + TOperator.Copy(samples, roundBits, roundOffset, out Vector256 lower, out Vector256 upper); + lower.StoreUnsafe(ref destinationRow, (nuint)column); + upper.StoreUnsafe(ref destinationRow, (nuint)(column + Vector256.Count)); } } - if (Vector128.IsHardwareAccelerated) + if (useSimd && Vector128.IsHardwareAccelerated) { int vectorEnd = width - Vector128.Count; for (; column <= vectorEnd; column += Vector128.Count) { - Vector128 firstVector = Vector128.LoadUnsafe(ref destinationReference, (nuint)column); - Vector128 secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column); - Average(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column); + Vector128 samples = Vector128.LoadUnsafe(ref sourceRow, (nuint)column); + TOperator.Copy(samples, roundBits, roundOffset, out Vector128 lower, out Vector128 upper); + lower.StoreUnsafe(ref destinationRow, (nuint)column); + upper.StoreUnsafe(ref destinationRow, (nuint)(column + Vector128.Count)); } } for (; column < width; column++) { - destinationRow[column] = (byte)((destinationRow[column] + secondRow[column] + 1) >> 1); + Unsafe.Add(ref destinationRow, column) = TOperator.Copy(Unsafe.Add(ref sourceRow, column), roundBits, roundOffset); } } } /// - /// Averages a high-bit-depth predictor into an existing prediction block. + /// Applies one compound convolution direction through one closed conversion operator. /// - /// The first predictor and combined output. - /// The distance between destination rows in samples. - /// The second predictor. - /// The distance between second-predictor rows in samples. - /// The active block width. - /// The active block height. - public static void Average( + /// The compound-prediction conversion operator. + private static void FilterCompoundDirect( + ReadOnlySpan source, + int sourceStride, + int sourceOrigin, Span destination, int destinationStride, - ReadOnlySpan second, - int secondStride, int width, - int height) + int height, + ReadOnlySpan coefficients, + int tapCount, + int sourceOffset, + int tapStride, + int preShift, + int round, + int roundOffset, + bool useSimd) + where TOperator : struct, IAv1CompoundPredictionOperator { + ref byte sourceBase = ref Unsafe.Add(ref MemoryMarshal.GetReference(source), sourceOrigin); + ref ushort destinationBase = ref MemoryMarshal.GetReference(destination); + ref short coefficientBase = ref MemoryMarshal.GetReference(coefficients); + for (int row = 0; row < height; row++) { - Span destinationRow = destination.Slice(row * destinationStride, width); - ReadOnlySpan secondRow = second.Slice(row * secondStride, width); - ref ushort destinationReference = ref MemoryMarshal.GetReference(destinationRow); - ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow); + ref byte sourceRow = ref Unsafe.Add(ref sourceBase, (row * sourceStride) + sourceOffset); + ref ushort destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); int column = 0; - if (Vector512.IsHardwareAccelerated) + if (useSimd && Vector512.IsHardwareAccelerated) { - int vectorEnd = width - Vector512.Count; - for (; column <= vectorEnd; column += Vector512.Count) + int vectorEnd = width - Vector512.Count; + for (; column <= vectorEnd; column += Vector512.Count) { - Vector512 firstVector = Vector512.LoadUnsafe(ref destinationReference, (nuint)column); - Vector512 secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column); - Average(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column); + Convolve( + ref sourceRow, + tapStride, + (nuint)column, + ref coefficientBase, + tapCount, + Vector512.Zero, + out Vector512 result0, + out Vector512 result1, + out Vector512 result2, + out Vector512 result3); + + TOperator.PrepareDirect(result0, result1, preShift, round, roundOffset) + .StoreUnsafe(ref destinationRow, (nuint)column); + + TOperator.PrepareDirect(result2, result3, preShift, round, roundOffset) + .StoreUnsafe(ref destinationRow, (nuint)(column + Vector512.Count)); } } - if (Vector256.IsHardwareAccelerated) + if (useSimd && Vector256.IsHardwareAccelerated) { - int vectorEnd = width - Vector256.Count; - for (; column <= vectorEnd; column += Vector256.Count) + int vectorEnd = width - Vector256.Count; + for (; column <= vectorEnd; column += Vector256.Count) { - Vector256 firstVector = Vector256.LoadUnsafe(ref destinationReference, (nuint)column); - Vector256 secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column); - Average(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column); + Convolve( + ref sourceRow, + tapStride, + (nuint)column, + ref coefficientBase, + tapCount, + Vector256.Zero, + out Vector256 result0, + out Vector256 result1, + out Vector256 result2, + out Vector256 result3); + + TOperator.PrepareDirect(result0, result1, preShift, round, roundOffset) + .StoreUnsafe(ref destinationRow, (nuint)column); + + TOperator.PrepareDirect(result2, result3, preShift, round, roundOffset) + .StoreUnsafe(ref destinationRow, (nuint)(column + Vector256.Count)); } } - if (Vector128.IsHardwareAccelerated) + if (useSimd && Vector128.IsHardwareAccelerated) { - int vectorEnd = width - Vector128.Count; - for (; column <= vectorEnd; column += Vector128.Count) + int vectorEnd = width - Vector128.Count; + for (; column <= vectorEnd; column += Vector128.Count) { - Vector128 firstVector = Vector128.LoadUnsafe(ref destinationReference, (nuint)column); - Vector128 secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column); - Average(firstVector, secondVector).StoreUnsafe(ref destinationReference, (nuint)column); + Convolve( + ref sourceRow, + tapStride, + (nuint)column, + ref coefficientBase, + tapCount, + Vector128.Zero, + out Vector128 result0, + out Vector128 result1, + out Vector128 result2, + out Vector128 result3); + + TOperator.PrepareDirect(result0, result1, preShift, round, roundOffset) + .StoreUnsafe(ref destinationRow, (nuint)column); + + TOperator.PrepareDirect(result2, result3, preShift, round, roundOffset) + .StoreUnsafe(ref destinationRow, (nuint)(column + Vector128.Count)); } } for (; column < width; column++) { - destinationRow[column] = (ushort)((destinationRow[column] + secondRow[column] + 1) >> 1); + int result = ConvolveScalar( + ref Unsafe.Add(ref sourceRow, column), + tapStride, + ref coefficientBase, + tapCount); + + Unsafe.Add(ref destinationRow, column) = TOperator.PrepareDirect(result, preShift, round, roundOffset); } } } /// - /// Averages an 8-bit predictor without explicit hardware intrinsics. + /// Applies separable compound convolution through caller-owned signed scratch. /// - /// The first predictor and combined output. - /// The distance between destination rows in samples. - /// The second predictor. - /// The distance between second-predictor rows in samples. - /// The active block width. - /// The active block height. - public static void AverageScalar( - Span destination, + /// The compound-prediction conversion operator. + private static void FilterCompound2D( + ReadOnlySpan source, + int sourceStride, + int sourceOrigin, + Span destination, int destinationStride, - ReadOnlySpan second, - int secondStride, int width, - int height) + int height, + ReadOnlySpan horizontalCoefficients, + int horizontalTapCount, + int horizontalSourceOffset, + ReadOnlySpan verticalCoefficients, + int verticalTapCount, + int verticalSourceOffset, + Span scratch, + bool useSimd) + where TOperator : struct, IAv1CompoundPredictionOperator { - for (int row = 0; row < height; row++) + ref byte sourceBase = ref Unsafe.Add(ref MemoryMarshal.GetReference(source), sourceOrigin); + ref ushort destinationBase = ref MemoryMarshal.GetReference(destination); + ref short scratchBase = ref MemoryMarshal.GetReference(scratch); + ref short horizontalCoefficientBase = ref MemoryMarshal.GetReference(horizontalCoefficients); + ref short verticalCoefficientBase = ref MemoryMarshal.GetReference(verticalCoefficients); + int scratchStride = Math.Max(width, MinimumScratchStride); + int intermediateHeight = height + verticalTapCount - 1; + + // The horizontal pass retains Q7 precision in signed scratch. Each SIMD stage continues at the shared + // column offset so mixed-width rows need no padding stores and never cross the logical block edge. + for (int row = 0; row < intermediateHeight; row++) { - Span destinationRow = destination.Slice(row * destinationStride, width); - ReadOnlySpan secondRow = second.Slice(row * secondStride, width); - for (int column = 0; column < width; column++) + ref byte sourceRow = ref Unsafe.Add( + ref sourceBase, + ((row + verticalSourceOffset) * sourceStride) + horizontalSourceOffset); + + ref short scratchRow = ref Unsafe.Add(ref scratchBase, row * scratchStride); + int column = 0; + + if (useSimd && Vector512.IsHardwareAccelerated) { - destinationRow[column] = (byte)((destinationRow[column] + secondRow[column] + 1) >> 1); + int vectorEnd = width - Vector512.Count; + for (; column <= vectorEnd; column += Vector512.Count) + { + Convolve( + ref sourceRow, + 1, + (nuint)column, + ref horizontalCoefficientBase, + horizontalTapCount, + Vector512.Zero, + out Vector512 result0, + out Vector512 result1, + out Vector512 result2, + out Vector512 result3); + + TOperator.PrepareHorizontal(result0, result1).StoreUnsafe(ref scratchRow, (nuint)column); + TOperator.PrepareHorizontal(result2, result3) + .StoreUnsafe(ref scratchRow, (nuint)(column + Vector512.Count)); + } + } + + if (useSimd && Vector256.IsHardwareAccelerated) + { + int vectorEnd = width - Vector256.Count; + for (; column <= vectorEnd; column += Vector256.Count) + { + Convolve( + ref sourceRow, + 1, + (nuint)column, + ref horizontalCoefficientBase, + horizontalTapCount, + Vector256.Zero, + out Vector256 result0, + out Vector256 result1, + out Vector256 result2, + out Vector256 result3); + + TOperator.PrepareHorizontal(result0, result1).StoreUnsafe(ref scratchRow, (nuint)column); + TOperator.PrepareHorizontal(result2, result3) + .StoreUnsafe(ref scratchRow, (nuint)(column + Vector256.Count)); + } + } + + if (useSimd && Vector128.IsHardwareAccelerated) + { + int vectorEnd = width - Vector128.Count; + for (; column <= vectorEnd; column += Vector128.Count) + { + Convolve( + ref sourceRow, + 1, + (nuint)column, + ref horizontalCoefficientBase, + horizontalTapCount, + Vector128.Zero, + out Vector128 result0, + out Vector128 result1, + out Vector128 result2, + out Vector128 result3); + + TOperator.PrepareHorizontal(result0, result1).StoreUnsafe(ref scratchRow, (nuint)column); + TOperator.PrepareHorizontal(result2, result3) + .StoreUnsafe(ref scratchRow, (nuint)(column + Vector128.Count)); + } + } + + for (; column < width; column++) + { + int result = ConvolveScalar( + ref Unsafe.Add(ref sourceRow, column), + 1, + ref horizontalCoefficientBase, + horizontalTapCount); + + Unsafe.Add(ref scratchRow, column) = TOperator.PrepareHorizontal(result); } } - } - /// - /// Averages a high-bit-depth predictor without explicit hardware intrinsics. - /// - /// The first predictor and combined output. - /// The distance between destination rows in samples. - /// The second predictor. - /// The distance between second-predictor rows in samples. - /// The active block width. - /// The active block height. - public static void AverageScalar( - Span destination, - int destinationStride, - ReadOnlySpan second, - int secondStride, - int width, - int height) - { for (int row = 0; row < height; row++) { - Span destinationRow = destination.Slice(row * destinationStride, width); - ReadOnlySpan secondRow = second.Slice(row * secondStride, width); - for (int column = 0; column < width; column++) + ref short scratchRow = ref Unsafe.Add(ref scratchBase, row * scratchStride); + ref ushort destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); + int column = 0; + + if (useSimd && Vector512.IsHardwareAccelerated) { - destinationRow[column] = (ushort)((destinationRow[column] + secondRow[column] + 1) >> 1); + int vectorEnd = width - Vector512.Count; + for (; column <= vectorEnd; column += Vector512.Count) + { + Convolve( + ref scratchRow, + scratchStride, + (nuint)column, + ref verticalCoefficientBase, + verticalTapCount, + Vector512.Zero, + out Vector512 lower, + out Vector512 upper); + + TOperator.PrepareVertical(lower, upper).StoreUnsafe(ref destinationRow, (nuint)column); + } } - } - } - /// - /// Computes rounded unsigned averages without widening either input vector. - /// - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector512 Average(Vector512 first, Vector512 second) - { - // (a | b) - ((a ^ b) >> 1) is exactly (a + b + 1) >> 1 and cannot overflow an unsigned lane. - return (first | second) - ((first ^ second) >> 1); - } + if (useSimd && Vector256.IsHardwareAccelerated) + { + int vectorEnd = width - Vector256.Count; + for (; column <= vectorEnd; column += Vector256.Count) + { + Convolve( + ref scratchRow, + scratchStride, + (nuint)column, + ref verticalCoefficientBase, + verticalTapCount, + Vector256.Zero, + out Vector256 lower, + out Vector256 upper); - /// - /// Computes rounded unsigned averages without widening either input vector. - /// - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector256 Average(Vector256 first, Vector256 second) - => (first | second) - ((first ^ second) >> 1); + TOperator.PrepareVertical(lower, upper).StoreUnsafe(ref destinationRow, (nuint)column); + } + } - /// - /// Computes rounded unsigned averages without widening either input vector. - /// - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector128 Average(Vector128 first, Vector128 second) - => (first | second) - ((first ^ second) >> 1); + if (useSimd && Vector128.IsHardwareAccelerated) + { + int vectorEnd = width - Vector128.Count; + for (; column <= vectorEnd; column += Vector128.Count) + { + Convolve( + ref scratchRow, + scratchStride, + (nuint)column, + ref verticalCoefficientBase, + verticalTapCount, + Vector128.Zero, + out Vector128 lower, + out Vector128 upper); - /// - /// Computes rounded unsigned averages without widening either input vector. - /// - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector512 Average(Vector512 first, Vector512 second) - => (first | second) - ((first ^ second) >> 1); + TOperator.PrepareVertical(lower, upper).StoreUnsafe(ref destinationRow, (nuint)column); + } + } - /// - /// Computes rounded unsigned averages without widening either input vector. - /// - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector256 Average(Vector256 first, Vector256 second) - => (first | second) - ((first ^ second) >> 1); + for (; column < width; column++) + { + int result = ConvolveScalar( + ref Unsafe.Add(ref scratchRow, column), + scratchStride, + ref verticalCoefficientBase, + verticalTapCount); + + Unsafe.Add(ref destinationRow, column) = TOperator.PrepareVertical(result); + } + } + } /// - /// Computes rounded unsigned averages without widening either input vector. + /// Gets the selected interpolation kernel for compound traversal. /// - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector128 Average(Vector128 first, Vector128 second) - => (first | second) - ((first ^ second) >> 1); + private static ReadOnlySpan GetCompoundCoefficients( + Av1InterpolationFilter filter, + int phase, + bool useReducedFilter) + => filter switch + { + Av1InterpolationFilter.Regular => RegularOperator.GetCoefficients(phase, useReducedFilter), + Av1InterpolationFilter.Smooth => SmoothOperator.GetCoefficients(phase, useReducedFilter), + Av1InterpolationFilter.Sharp => SharpOperator.GetCoefficients(phase, useReducedFilter), + _ => BilinearOperator.GetCoefficients(phase, useReducedFilter), + }; } diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateAveragePredictor.Operator.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateAveragePredictor.Operator.cs new file mode 100644 index 000000000..7696e76fa --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateAveragePredictor.Operator.cs @@ -0,0 +1,140 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.Intrinsics; + +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor; +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; + +/// +/// Defines final equal-average compound-intermediate reconstruction. +/// +internal static partial class Av1CompoundIntermediateAveragePredictor +{ + /// + /// Defines equal-average finalization for scalar and SIMD lane groups. + /// + private interface IAv1CompoundIntermediateAverageOperator + { + /// + /// Equal-averages and finalizes one pair of compound intermediate samples. + /// + /// The first compound intermediate. + /// The second compound intermediate. + /// The final reconstruction shift. + /// The compound intermediate bias. + /// The reconstructed sample. + public static abstract byte Average(ushort first, ushort second, int roundBits, int roundOffset); + + /// + /// Equal-averages and finalizes 128 bits of compound intermediate samples. + /// + /// The lower first-predictor intermediates. + /// The upper first-predictor intermediates. + /// The lower second-predictor intermediates. + /// The upper second-predictor intermediates. + /// The final reconstruction shift. + /// The compound intermediate bias. + /// The reconstructed samples. + public static abstract Vector128 Average( + Vector128 first0, + Vector128 first1, + Vector128 second0, + Vector128 second1, + int roundBits, + int roundOffset); + + /// + /// Equal-averages and finalizes 256 bits of compound intermediate samples. + /// + /// The lower first-predictor intermediates. + /// The upper first-predictor intermediates. + /// The lower second-predictor intermediates. + /// The upper second-predictor intermediates. + /// The final reconstruction shift. + /// The compound intermediate bias. + /// The reconstructed samples. + public static abstract Vector256 Average( + Vector256 first0, + Vector256 first1, + Vector256 second0, + Vector256 second1, + int roundBits, + int roundOffset); + + /// + /// Equal-averages and finalizes 512 bits of compound intermediate samples. + /// + /// The lower first-predictor intermediates. + /// The upper first-predictor intermediates. + /// The lower second-predictor intermediates. + /// The upper second-predictor intermediates. + /// The final reconstruction shift. + /// The compound intermediate bias. + /// The reconstructed samples. + public static abstract Vector512 Average( + Vector512 first0, + Vector512 first1, + Vector512 second0, + Vector512 second1, + int roundBits, + int roundOffset); + } + + /// + /// Implements equal-average finalization for scalar and SIMD lane groups. + /// + private readonly struct CompoundIntermediateAverageOperator : IAv1CompoundIntermediateAverageOperator + { + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static byte Average(ushort first, ushort second, int roundBits, int roundOffset) + { + // The reference average deliberately truncates here. Finalization performs the sole rounding step. + int result = ((first + second) >> 1) - roundOffset; + return (byte)Math.Clamp(RoundPowerOfTwo(result, roundBits), 0, byte.MaxValue); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector128 Average( + Vector128 first0, + Vector128 first1, + Vector128 second0, + Vector128 second1, + int roundBits, + int roundOffset) + => Vector128.Narrow( + FinalizeIntermediate((first0 & second0) + ((first0 ^ second0) >> 1), roundBits, roundOffset), + FinalizeIntermediate((first1 & second1) + ((first1 ^ second1) >> 1), roundBits, roundOffset)); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector256 Average( + Vector256 first0, + Vector256 first1, + Vector256 second0, + Vector256 second1, + int roundBits, + int roundOffset) + => Vector256.Narrow( + FinalizeIntermediate((first0 & second0) + ((first0 ^ second0) >> 1), roundBits, roundOffset), + FinalizeIntermediate((first1 & second1) + ((first1 ^ second1) >> 1), roundBits, roundOffset)); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector512 Average( + Vector512 first0, + Vector512 first1, + Vector512 second0, + Vector512 second1, + int roundBits, + int roundOffset) + => Vector512.Narrow( + FinalizeIntermediate((first0 & second0) + ((first0 ^ second0) >> 1), roundBits, roundOffset), + FinalizeIntermediate((first1 & second1) + ((first1 ^ second1) >> 1), roundBits, roundOffset)); + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateAveragePredictor.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateAveragePredictor.cs new file mode 100644 index 000000000..bab8c7aa8 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateAveragePredictor.cs @@ -0,0 +1,124 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using System.Runtime.Intrinsics; + +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor; +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; + +/// +/// Reconstructs final samples by equal-averaging compound intermediates. +/// +internal static partial class Av1CompoundIntermediateAveragePredictor +{ + /// + /// Combines two compound intermediates by equal averaging. + /// + public static void AverageIntermediate( + Span destination, + int destinationStride, + ReadOnlySpan first, + int firstStride, + ReadOnlySpan second, + int secondStride, + int width, + int height, + int bitDepth) + => AverageIntermediate( + destination, + destinationStride, + first, + firstStride, + second, + secondStride, + width, + height, + bitDepth); + + /// + /// Executes one closed equal-average compound-intermediate operator. + /// + /// The compound-intermediate operator. + private static void AverageIntermediate( + Span destination, + int destinationStride, + ReadOnlySpan first, + int firstStride, + ReadOnlySpan second, + int secondStride, + int width, + int height, + int bitDepth) + where TOperator : struct, IAv1CompoundIntermediateAverageOperator + { + GetIntermediateRounding(bitDepth, out int roundBits, out int roundOffset); + for (int row = 0; row < height; row++) + { + Span destinationRow = destination.Slice(row * destinationStride, width); + ReadOnlySpan firstRow = first.Slice(row * firstStride, width); + ReadOnlySpan secondRow = second.Slice(row * secondStride, width); + ref byte destinationReference = ref MemoryMarshal.GetReference(destinationRow); + ref ushort firstReference = ref MemoryMarshal.GetReference(firstRow); + ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow); + int column = 0; + + if (Vector512.IsHardwareAccelerated) + { + int vectorEnd = width - Vector512.Count; + for (; column <= vectorEnd; column += Vector512.Count) + { + Vector512 first0 = Vector512.LoadUnsafe(ref firstReference, (nuint)column); + Vector512 first1 = Vector512.LoadUnsafe(ref firstReference, (nuint)(column + Vector512.Count)); + Vector512 second0 = Vector512.LoadUnsafe(ref secondReference, (nuint)column); + Vector512 second1 = Vector512.LoadUnsafe(ref secondReference, (nuint)(column + Vector512.Count)); + TOperator.Average(first0, first1, second0, second1, roundBits, roundOffset) + .StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + if (Vector256.IsHardwareAccelerated) + { + int vectorEnd = width - Vector256.Count; + for (; column <= vectorEnd; column += Vector256.Count) + { + Vector256 first0 = Vector256.LoadUnsafe(ref firstReference, (nuint)column); + Vector256 first1 = Vector256.LoadUnsafe(ref firstReference, (nuint)(column + Vector256.Count)); + Vector256 second0 = Vector256.LoadUnsafe(ref secondReference, (nuint)column); + Vector256 second1 = Vector256.LoadUnsafe(ref secondReference, (nuint)(column + Vector256.Count)); + TOperator.Average(first0, first1, second0, second1, roundBits, roundOffset) + .StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + if (Vector128.IsHardwareAccelerated) + { + int vectorEnd = width - Vector128.Count; + for (; column <= vectorEnd; column += Vector128.Count) + { + Vector128 first0 = Vector128.LoadUnsafe(ref firstReference, (nuint)column); + Vector128 first1 = Vector128.LoadUnsafe( + ref firstReference, + (nuint)(column + Vector128.Count)); + + Vector128 second0 = Vector128.LoadUnsafe(ref secondReference, (nuint)column); + Vector128 second1 = Vector128.LoadUnsafe( + ref secondReference, + (nuint)(column + Vector128.Count)); + + TOperator.Average(first0, first1, second0, second1, roundBits, roundOffset).StoreUnsafe( + ref destinationReference, + (nuint)column); + } + } + + for (; column < width; column++) + { + destinationRow[column] = TOperator.Average(firstRow[column], secondRow[column], roundBits, roundOffset); + } + } + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateDifferenceWeightedMaskBuilder.Operator.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateDifferenceWeightedMaskBuilder.Operator.cs new file mode 100644 index 000000000..17a7a4c7f --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateDifferenceWeightedMaskBuilder.Operator.cs @@ -0,0 +1,218 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.Intrinsics; + +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor; +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; + +/// +/// Defines difference-weighted mask generation from compound intermediates. +/// +internal static partial class Av1CompoundIntermediateDifferenceWeightedMaskBuilder +{ + /// + /// Defines difference-weighted mask generation for scalar and SIMD lane groups. + /// + private interface IAv1CompoundIntermediateDifferenceWeightedMaskOperator + { + /// + /// Creates one difference-weighted mask value from compound intermediate samples. + /// + /// The first compound intermediate. + /// The second compound intermediate. + /// The difference scaling shift. + /// Whether to invert the selected predictor. + /// The AV1 mask value. + public static abstract byte CreateMask(ushort first, ushort second, int differenceRound, bool invert); + + /// + /// Creates 128 bits of difference-weighted mask values from compound intermediate samples. + /// + /// The lower first-predictor intermediates. + /// The upper first-predictor intermediates. + /// The lower second-predictor intermediates. + /// The upper second-predictor intermediates. + /// The difference scaling shift. + /// Whether to invert the selected predictor. + /// The packed AV1 mask values. + public static abstract Vector128 CreateMask( + Vector128 first0, + Vector128 first1, + Vector128 second0, + Vector128 second1, + int differenceRound, + bool invert); + + /// + /// Creates 256 bits of difference-weighted mask values from compound intermediate samples. + /// + /// The lower first-predictor intermediates. + /// The upper first-predictor intermediates. + /// The lower second-predictor intermediates. + /// The upper second-predictor intermediates. + /// The difference scaling shift. + /// Whether to invert the selected predictor. + /// The packed AV1 mask values. + public static abstract Vector256 CreateMask( + Vector256 first0, + Vector256 first1, + Vector256 second0, + Vector256 second1, + int differenceRound, + bool invert); + + /// + /// Creates 512 bits of difference-weighted mask values from compound intermediate samples. + /// + /// The lower first-predictor intermediates. + /// The upper first-predictor intermediates. + /// The lower second-predictor intermediates. + /// The upper second-predictor intermediates. + /// The difference scaling shift. + /// Whether to invert the selected predictor. + /// The packed AV1 mask values. + public static abstract Vector512 CreateMask( + Vector512 first0, + Vector512 first1, + Vector512 second0, + Vector512 second1, + int differenceRound, + bool invert); + } + + /// + /// Implements difference-weighted mask generation for scalar and SIMD lane groups. + /// + private readonly struct CompoundIntermediateDifferenceWeightedMaskOperator : IAv1CompoundIntermediateDifferenceWeightedMaskOperator + { + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static byte CreateMask(ushort first, ushort second, int differenceRound, bool invert) + { + int difference = RoundPowerOfTwo(Math.Abs(first - second), differenceRound); + int alpha = Math.Min(MaximumMaskAlpha, 38 + (difference >> 4)); + return (byte)(invert ? MaximumMaskAlpha - alpha : alpha); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector128 CreateMask( + Vector128 first0, + Vector128 first1, + Vector128 second0, + Vector128 second1, + int differenceRound, + bool invert) + => Vector128.Narrow( + CreateMask(first0, second0, differenceRound, invert), + CreateMask(first1, second1, differenceRound, invert)); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector256 CreateMask( + Vector256 first0, + Vector256 first1, + Vector256 second0, + Vector256 second1, + int differenceRound, + bool invert) + => Vector256.Narrow( + CreateMask(first0, second0, differenceRound, invert), + CreateMask(first1, second1, differenceRound, invert)); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector512 CreateMask( + Vector512 first0, + Vector512 first1, + Vector512 second0, + Vector512 second1, + int differenceRound, + bool invert) + => Vector512.Narrow( + CreateMask(first0, second0, differenceRound, invert), + CreateMask(first1, second1, differenceRound, invert)); + + /// + /// Creates 128-bit unpacked mask values without losing the required pre-alpha rounding. + /// + private static Vector128 CreateMask(Vector128 first, Vector128 second, int differenceRound, bool invert) + { + Vector128 difference = Vector128.Max(first, second) - Vector128.Min(first, second); + Vector128 lower = CreateMaskAlpha(Vector128.WidenLower(difference).AsInt32(), differenceRound, invert); + Vector128 upper = CreateMaskAlpha(Vector128.WidenUpper(difference).AsInt32(), differenceRound, invert); + return Vector128.Narrow(lower, upper).AsUInt16(); + } + + /// + /// Creates 256-bit unpacked mask values without losing the required pre-alpha rounding. + /// + private static Vector256 CreateMask(Vector256 first, Vector256 second, int differenceRound, bool invert) + { + Vector256 difference = Vector256.Max(first, second) - Vector256.Min(first, second); + Vector256 lower = CreateMaskAlpha(Vector256.WidenLower(difference).AsInt32(), differenceRound, invert); + Vector256 upper = CreateMaskAlpha(Vector256.WidenUpper(difference).AsInt32(), differenceRound, invert); + return Vector256.Narrow(lower, upper).AsUInt16(); + } + + /// + /// Creates 512-bit unpacked mask values without losing the required pre-alpha rounding. + /// + private static Vector512 CreateMask(Vector512 first, Vector512 second, int differenceRound, bool invert) + { + Vector512 difference = Vector512.Max(first, second) - Vector512.Min(first, second); + Vector512 lower = CreateMaskAlpha(Vector512.WidenLower(difference).AsInt32(), differenceRound, invert); + Vector512 upper = CreateMaskAlpha(Vector512.WidenUpper(difference).AsInt32(), differenceRound, invert); + return Vector512.Narrow(lower, upper).AsUInt16(); + } + + /// + /// Converts 128-bit intermediate differences to the decoded type-38 mask range. + /// + private static Vector128 CreateMaskAlpha(Vector128 difference, int differenceRound, bool invert) + { + if (differenceRound != 0) + { + difference = (difference + Vector128.Create(1 << (differenceRound - 1))) >> differenceRound; + } + + Vector128 maximum = Vector128.Create(MaximumMaskAlpha); + Vector128 alpha = Vector128.Min(maximum, (difference >> 4) + Vector128.Create(38)); + return invert ? maximum - alpha : alpha; + } + + /// + /// Converts 256-bit intermediate differences to the decoded type-38 mask range. + /// + private static Vector256 CreateMaskAlpha(Vector256 difference, int differenceRound, bool invert) + { + if (differenceRound != 0) + { + difference = (difference + Vector256.Create(1 << (differenceRound - 1))) >> differenceRound; + } + + Vector256 maximum = Vector256.Create(MaximumMaskAlpha); + Vector256 alpha = Vector256.Min(maximum, (difference >> 4) + Vector256.Create(38)); + return invert ? maximum - alpha : alpha; + } + + /// + /// Converts 512-bit intermediate differences to the decoded type-38 mask range. + /// + private static Vector512 CreateMaskAlpha(Vector512 difference, int differenceRound, bool invert) + { + if (differenceRound != 0) + { + difference = (difference + Vector512.Create(1 << (differenceRound - 1))) >> differenceRound; + } + + Vector512 maximum = Vector512.Create(MaximumMaskAlpha); + Vector512 alpha = Vector512.Min(maximum, (difference >> 4) + Vector512.Create(38)); + return invert ? maximum - alpha : alpha; + } + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateDifferenceWeightedMaskBuilder.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateDifferenceWeightedMaskBuilder.cs new file mode 100644 index 000000000..128689f85 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateDifferenceWeightedMaskBuilder.cs @@ -0,0 +1,134 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using System.Runtime.Intrinsics; +using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; + +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor; +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; + +/// +/// Builds difference-weighted masks from compound intermediates. +/// +internal static partial class Av1CompoundIntermediateDifferenceWeightedMaskBuilder +{ + /// + /// Fills a luma-resolution difference-weighted mask from compound intermediates. + /// + public static void FillDifferenceWeightedIntermediateMask( + Span mask, + int maskStride, + ReadOnlySpan first, + int firstStride, + ReadOnlySpan second, + int secondStride, + int width, + int height, + int bitDepth, + Av1DifferenceWeightedMaskType maskType) + => FillDifferenceWeightedIntermediateMask( + mask, + maskStride, + first, + firstStride, + second, + secondStride, + width, + height, + bitDepth, + maskType); + + /// + /// Executes one closed difference-mask compound-intermediate operator. + /// + /// The compound-intermediate operator. + private static void FillDifferenceWeightedIntermediateMask( + Span mask, + int maskStride, + ReadOnlySpan first, + int firstStride, + ReadOnlySpan second, + int secondStride, + int width, + int height, + int bitDepth, + Av1DifferenceWeightedMaskType maskType) + where TOperator : struct, IAv1CompoundIntermediateDifferenceWeightedMaskOperator + { + bool invert = maskType == Av1DifferenceWeightedMaskType.Type38Inverse; + GetIntermediateRounding(bitDepth, out int roundBits, out _); + int differenceRound = roundBits + bitDepth - 8; + for (int row = 0; row < height; row++) + { + Span maskRow = mask.Slice(row * maskStride, width); + ReadOnlySpan firstRow = first.Slice(row * firstStride, width); + ReadOnlySpan secondRow = second.Slice(row * secondStride, width); + ref byte maskReference = ref MemoryMarshal.GetReference(maskRow); + ref ushort firstReference = ref MemoryMarshal.GetReference(firstRow); + ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow); + int column = 0; + + if (Vector512.IsHardwareAccelerated) + { + int vectorEnd = width - Vector512.Count; + for (; column <= vectorEnd; column += Vector512.Count) + { + Vector512 first0 = Vector512.LoadUnsafe(ref firstReference, (nuint)column); + Vector512 first1 = Vector512.LoadUnsafe(ref firstReference, (nuint)(column + Vector512.Count)); + Vector512 second0 = Vector512.LoadUnsafe(ref secondReference, (nuint)column); + Vector512 second1 = Vector512.LoadUnsafe(ref secondReference, (nuint)(column + Vector512.Count)); + TOperator.CreateMask(first0, first1, second0, second1, differenceRound, invert) + .StoreUnsafe(ref maskReference, (nuint)column); + } + } + + if (Vector256.IsHardwareAccelerated) + { + int vectorEnd = width - Vector256.Count; + for (; column <= vectorEnd; column += Vector256.Count) + { + Vector256 first0 = Vector256.LoadUnsafe(ref firstReference, (nuint)column); + Vector256 first1 = Vector256.LoadUnsafe(ref firstReference, (nuint)(column + Vector256.Count)); + Vector256 second0 = Vector256.LoadUnsafe(ref secondReference, (nuint)column); + Vector256 second1 = Vector256.LoadUnsafe(ref secondReference, (nuint)(column + Vector256.Count)); + TOperator.CreateMask(first0, first1, second0, second1, differenceRound, invert) + .StoreUnsafe(ref maskReference, (nuint)column); + } + } + + if (Vector128.IsHardwareAccelerated) + { + int vectorEnd = width - Vector128.Count; + for (; column <= vectorEnd; column += Vector128.Count) + { + Vector128 first0 = Vector128.LoadUnsafe(ref firstReference, (nuint)column); + Vector128 first1 = Vector128.LoadUnsafe( + ref firstReference, + (nuint)(column + Vector128.Count)); + + Vector128 second0 = Vector128.LoadUnsafe(ref secondReference, (nuint)column); + Vector128 second1 = Vector128.LoadUnsafe( + ref secondReference, + (nuint)(column + Vector128.Count)); + + TOperator.CreateMask( + first0, + first1, + second0, + second1, + differenceRound, + invert).StoreUnsafe(ref maskReference, (nuint)column); + } + } + + for (; column < width; column++) + { + maskRow[column] = TOperator.CreateMask(firstRow[column], secondRow[column], differenceRound, invert); + } + } + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateDistanceWeightedPredictor.Operator.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateDistanceWeightedPredictor.Operator.cs new file mode 100644 index 000000000..f211f3052 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateDistanceWeightedPredictor.Operator.cs @@ -0,0 +1,232 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.Intrinsics; + +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor; +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; + +/// +/// Defines final distance-weighted compound-intermediate reconstruction. +/// +internal static partial class Av1CompoundIntermediateDistanceWeightedPredictor +{ + /// + /// Defines distance-weighted finalization for scalar and SIMD lane groups. + /// + private interface IAv1CompoundIntermediateDistanceWeightedOperator + { + /// + /// Distance-weights and finalizes one pair of compound intermediate samples. + /// + /// The first compound intermediate. + /// The second compound intermediate. + /// The first predictor weight. + /// The second predictor weight. + /// The final reconstruction shift. + /// The compound intermediate bias. + /// The reconstructed sample. + public static abstract byte DistanceWeighted( + ushort first, + ushort second, + int firstWeight, + int secondWeight, + int roundBits, + int roundOffset); + + /// + /// Distance-weights and finalizes 128 bits of compound intermediate samples. + /// + /// The lower first-predictor intermediates. + /// The upper first-predictor intermediates. + /// The lower second-predictor intermediates. + /// The upper second-predictor intermediates. + /// The first predictor weight. + /// The second predictor weight. + /// The final reconstruction shift. + /// The compound intermediate bias. + /// The reconstructed samples. + public static abstract Vector128 DistanceWeighted( + Vector128 first0, + Vector128 first1, + Vector128 second0, + Vector128 second1, + int firstWeight, + int secondWeight, + int roundBits, + int roundOffset); + + /// + /// Distance-weights and finalizes 256 bits of compound intermediate samples. + /// + /// The lower first-predictor intermediates. + /// The upper first-predictor intermediates. + /// The lower second-predictor intermediates. + /// The upper second-predictor intermediates. + /// The first predictor weight. + /// The second predictor weight. + /// The final reconstruction shift. + /// The compound intermediate bias. + /// The reconstructed samples. + public static abstract Vector256 DistanceWeighted( + Vector256 first0, + Vector256 first1, + Vector256 second0, + Vector256 second1, + int firstWeight, + int secondWeight, + int roundBits, + int roundOffset); + + /// + /// Distance-weights and finalizes 512 bits of compound intermediate samples. + /// + /// The lower first-predictor intermediates. + /// The upper first-predictor intermediates. + /// The lower second-predictor intermediates. + /// The upper second-predictor intermediates. + /// The first predictor weight. + /// The second predictor weight. + /// The final reconstruction shift. + /// The compound intermediate bias. + /// The reconstructed samples. + public static abstract Vector512 DistanceWeighted( + Vector512 first0, + Vector512 first1, + Vector512 second0, + Vector512 second1, + int firstWeight, + int secondWeight, + int roundBits, + int roundOffset); + } + + /// + /// Implements distance-weighted finalization for scalar and SIMD lane groups. + /// + private readonly struct CompoundIntermediateDistanceWeightedOperator : IAv1CompoundIntermediateDistanceWeightedOperator + { + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static byte DistanceWeighted( + ushort first, + ushort second, + int firstWeight, + int secondWeight, + int roundBits, + int roundOffset) + { + int result = ((first * firstWeight) + (second * secondWeight)) >> DistanceWeightBits; + result -= roundOffset; + return (byte)Math.Clamp(RoundPowerOfTwo(result, roundBits), 0, byte.MaxValue); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector128 DistanceWeighted( + Vector128 first0, + Vector128 first1, + Vector128 second0, + Vector128 second1, + int firstWeight, + int secondWeight, + int roundBits, + int roundOffset) + => Vector128.Narrow( + DistanceWeighted(first0, second0, firstWeight, secondWeight, roundBits, roundOffset), + DistanceWeighted(first1, second1, firstWeight, secondWeight, roundBits, roundOffset)); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector256 DistanceWeighted( + Vector256 first0, + Vector256 first1, + Vector256 second0, + Vector256 second1, + int firstWeight, + int secondWeight, + int roundBits, + int roundOffset) + => Vector256.Narrow( + DistanceWeighted(first0, second0, firstWeight, secondWeight, roundBits, roundOffset), + DistanceWeighted(first1, second1, firstWeight, secondWeight, roundBits, roundOffset)); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector512 DistanceWeighted( + Vector512 first0, + Vector512 first1, + Vector512 second0, + Vector512 second1, + int firstWeight, + int secondWeight, + int roundBits, + int roundOffset) + => Vector512.Narrow( + DistanceWeighted(first0, second0, firstWeight, secondWeight, roundBits, roundOffset), + DistanceWeighted(first1, second1, firstWeight, secondWeight, roundBits, roundOffset)); + + /// + /// Distance-weights 128-bit lanes without overflowing the unsigned intermediate range. + /// + private static Vector128 DistanceWeighted( + Vector128 first, + Vector128 second, + int firstWeight, + int secondWeight, + int roundBits, + int roundOffset) + { + Vector128 firstLower = Vector128.WidenLower(first).AsInt32(); + Vector128 firstUpper = Vector128.WidenUpper(first).AsInt32(); + Vector128 secondLower = Vector128.WidenLower(second).AsInt32(); + Vector128 secondUpper = Vector128.WidenUpper(second).AsInt32(); + Vector128 lower = ((firstLower * firstWeight) + (secondLower * secondWeight)) >> DistanceWeightBits; + Vector128 upper = ((firstUpper * firstWeight) + (secondUpper * secondWeight)) >> DistanceWeightBits; + return Vector128.Narrow(FinalizeIntermediate(lower, roundBits, roundOffset), FinalizeIntermediate(upper, roundBits, roundOffset)).AsUInt16(); + } + + /// + /// Distance-weights 256-bit lanes without overflowing the unsigned intermediate range. + /// + private static Vector256 DistanceWeighted( + Vector256 first, + Vector256 second, + int firstWeight, + int secondWeight, + int roundBits, + int roundOffset) + { + Vector256 firstLower = Vector256.WidenLower(first).AsInt32(); + Vector256 firstUpper = Vector256.WidenUpper(first).AsInt32(); + Vector256 secondLower = Vector256.WidenLower(second).AsInt32(); + Vector256 secondUpper = Vector256.WidenUpper(second).AsInt32(); + Vector256 lower = ((firstLower * firstWeight) + (secondLower * secondWeight)) >> DistanceWeightBits; + Vector256 upper = ((firstUpper * firstWeight) + (secondUpper * secondWeight)) >> DistanceWeightBits; + return Vector256.Narrow(FinalizeIntermediate(lower, roundBits, roundOffset), FinalizeIntermediate(upper, roundBits, roundOffset)).AsUInt16(); + } + + /// + /// Distance-weights 512-bit lanes without overflowing the unsigned intermediate range. + /// + private static Vector512 DistanceWeighted( + Vector512 first, + Vector512 second, + int firstWeight, + int secondWeight, + int roundBits, + int roundOffset) + { + Vector512 firstLower = Vector512.WidenLower(first).AsInt32(); + Vector512 firstUpper = Vector512.WidenUpper(first).AsInt32(); + Vector512 secondLower = Vector512.WidenLower(second).AsInt32(); + Vector512 secondUpper = Vector512.WidenUpper(second).AsInt32(); + Vector512 lower = ((firstLower * firstWeight) + (secondLower * secondWeight)) >> DistanceWeightBits; + Vector512 upper = ((firstUpper * firstWeight) + (secondUpper * secondWeight)) >> DistanceWeightBits; + return Vector512.Narrow(FinalizeIntermediate(lower, roundBits, roundOffset), FinalizeIntermediate(upper, roundBits, roundOffset)).AsUInt16(); + } + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateDistanceWeightedPredictor.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateDistanceWeightedPredictor.cs new file mode 100644 index 000000000..c8479bbc6 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateDistanceWeightedPredictor.cs @@ -0,0 +1,156 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using System.Runtime.Intrinsics; + +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor; +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; + +/// +/// Reconstructs final samples by distance-weighting compound intermediates. +/// +internal static partial class Av1CompoundIntermediateDistanceWeightedPredictor +{ + /// + /// Combines two compound intermediates using the decoded temporal-distance weights. + /// + public static void DistanceWeightedIntermediate( + Span destination, + int destinationStride, + ReadOnlySpan first, + int firstStride, + ReadOnlySpan second, + int secondStride, + int width, + int height, + int firstWeight, + int secondWeight, + int bitDepth) + => DistanceWeightedIntermediate( + destination, + destinationStride, + first, + firstStride, + second, + secondStride, + width, + height, + firstWeight, + secondWeight, + bitDepth); + + /// + /// Executes one closed distance-weighted compound-intermediate operator. + /// + /// The compound-intermediate operator. + private static void DistanceWeightedIntermediate( + Span destination, + int destinationStride, + ReadOnlySpan first, + int firstStride, + ReadOnlySpan second, + int secondStride, + int width, + int height, + int firstWeight, + int secondWeight, + int bitDepth) + where TOperator : struct, IAv1CompoundIntermediateDistanceWeightedOperator + { + GetIntermediateRounding(bitDepth, out int roundBits, out int roundOffset); + for (int row = 0; row < height; row++) + { + Span destinationRow = destination.Slice(row * destinationStride, width); + ReadOnlySpan firstRow = first.Slice(row * firstStride, width); + ReadOnlySpan secondRow = second.Slice(row * secondStride, width); + ref byte destinationReference = ref MemoryMarshal.GetReference(destinationRow); + ref ushort firstReference = ref MemoryMarshal.GetReference(firstRow); + ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow); + int column = 0; + + if (Vector512.IsHardwareAccelerated) + { + int vectorEnd = width - Vector512.Count; + for (; column <= vectorEnd; column += Vector512.Count) + { + Vector512 first0 = Vector512.LoadUnsafe(ref firstReference, (nuint)column); + Vector512 first1 = Vector512.LoadUnsafe(ref firstReference, (nuint)(column + Vector512.Count)); + Vector512 second0 = Vector512.LoadUnsafe(ref secondReference, (nuint)column); + Vector512 second1 = Vector512.LoadUnsafe(ref secondReference, (nuint)(column + Vector512.Count)); + TOperator.DistanceWeighted( + first0, + first1, + second0, + second1, + firstWeight, + secondWeight, + roundBits, + roundOffset).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + if (Vector256.IsHardwareAccelerated) + { + int vectorEnd = width - Vector256.Count; + for (; column <= vectorEnd; column += Vector256.Count) + { + Vector256 first0 = Vector256.LoadUnsafe(ref firstReference, (nuint)column); + Vector256 first1 = Vector256.LoadUnsafe(ref firstReference, (nuint)(column + Vector256.Count)); + Vector256 second0 = Vector256.LoadUnsafe(ref secondReference, (nuint)column); + Vector256 second1 = Vector256.LoadUnsafe(ref secondReference, (nuint)(column + Vector256.Count)); + TOperator.DistanceWeighted( + first0, + first1, + second0, + second1, + firstWeight, + secondWeight, + roundBits, + roundOffset).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + if (Vector128.IsHardwareAccelerated) + { + int vectorEnd = width - Vector128.Count; + for (; column <= vectorEnd; column += Vector128.Count) + { + Vector128 first0 = Vector128.LoadUnsafe(ref firstReference, (nuint)column); + Vector128 first1 = Vector128.LoadUnsafe( + ref firstReference, + (nuint)(column + Vector128.Count)); + + Vector128 second0 = Vector128.LoadUnsafe(ref secondReference, (nuint)column); + Vector128 second1 = Vector128.LoadUnsafe( + ref secondReference, + (nuint)(column + Vector128.Count)); + + TOperator.DistanceWeighted( + first0, + first1, + second0, + second1, + firstWeight, + secondWeight, + roundBits, + roundOffset).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + for (; column < width; column++) + { + destinationRow[column] = TOperator.DistanceWeighted( + firstRow[column], + secondRow[column], + firstWeight, + secondWeight, + roundBits, + roundOffset); + } + } + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateMaskBlendPredictor.Operator.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateMaskBlendPredictor.Operator.cs new file mode 100644 index 000000000..93be9b072 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateMaskBlendPredictor.Operator.cs @@ -0,0 +1,217 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.Intrinsics; + +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor; +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; + +/// +/// Defines final masked compound-intermediate reconstruction. +/// +internal static partial class Av1CompoundIntermediateMaskBlendPredictor +{ + /// + /// Defines masked compound-intermediate finalization for scalar and SIMD lane groups. + /// + private interface IAv1CompoundIntermediateMaskBlendOperator + { + /// + /// Alpha-blends and finalizes one pair of compound intermediate samples. + /// + /// The first compound intermediate. + /// The second compound intermediate. + /// The first-predictor weight in the AV1 mask range. + /// The final reconstruction shift. + /// The compound intermediate bias. + /// The reconstructed sample. + public static abstract byte Blend(ushort first, ushort second, byte alpha, int roundBits, int roundOffset); + + /// + /// Alpha-blends and finalizes 128 bits of compound intermediate samples. + /// + /// The lower first-predictor intermediates. + /// The upper first-predictor intermediates. + /// The lower second-predictor intermediates. + /// The upper second-predictor intermediates. + /// The first-predictor weights in the AV1 mask range. + /// The final reconstruction shift. + /// The compound intermediate bias. + /// The reconstructed samples. + public static abstract Vector128 Blend( + Vector128 first0, + Vector128 first1, + Vector128 second0, + Vector128 second1, + Vector128 alpha, + int roundBits, + int roundOffset); + + /// + /// Alpha-blends and finalizes 256 bits of compound intermediate samples. + /// + /// The lower first-predictor intermediates. + /// The upper first-predictor intermediates. + /// The lower second-predictor intermediates. + /// The upper second-predictor intermediates. + /// The first-predictor weights in the AV1 mask range. + /// The final reconstruction shift. + /// The compound intermediate bias. + /// The reconstructed samples. + public static abstract Vector256 Blend( + Vector256 first0, + Vector256 first1, + Vector256 second0, + Vector256 second1, + Vector256 alpha, + int roundBits, + int roundOffset); + + /// + /// Alpha-blends and finalizes 512 bits of compound intermediate samples. + /// + /// The lower first-predictor intermediates. + /// The upper first-predictor intermediates. + /// The lower second-predictor intermediates. + /// The upper second-predictor intermediates. + /// The first-predictor weights in the AV1 mask range. + /// The final reconstruction shift. + /// The compound intermediate bias. + /// The reconstructed samples. + public static abstract Vector512 Blend( + Vector512 first0, + Vector512 first1, + Vector512 second0, + Vector512 second1, + Vector512 alpha, + int roundBits, + int roundOffset); + } + + /// + /// Implements masked compound-intermediate finalization for scalar and SIMD lane groups. + /// + private readonly struct CompoundIntermediateMaskBlendOperator : IAv1CompoundIntermediateMaskBlendOperator + { + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static byte Blend(ushort first, ushort second, byte alpha, int roundBits, int roundOffset) + { + // The Q6 blend truncates because final pixel rounding is still pending after bias removal. + int result = ((alpha * first) + ((MaximumMaskAlpha - alpha) * second)) >> MaskWeightBits; + result -= roundOffset; + return (byte)Math.Clamp(RoundPowerOfTwo(result, roundBits), 0, byte.MaxValue); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector128 Blend( + Vector128 first0, + Vector128 first1, + Vector128 second0, + Vector128 second1, + Vector128 alpha, + int roundBits, + int roundOffset) + => Vector128.Narrow( + Blend(first0, second0, Vector128.WidenLower(alpha), roundBits, roundOffset), + Blend(first1, second1, Vector128.WidenUpper(alpha), roundBits, roundOffset)); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector256 Blend( + Vector256 first0, + Vector256 first1, + Vector256 second0, + Vector256 second1, + Vector256 alpha, + int roundBits, + int roundOffset) + => Vector256.Narrow( + Blend(first0, second0, Vector256.WidenLower(alpha), roundBits, roundOffset), + Blend(first1, second1, Vector256.WidenUpper(alpha), roundBits, roundOffset)); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector512 Blend( + Vector512 first0, + Vector512 first1, + Vector512 second0, + Vector512 second1, + Vector512 alpha, + int roundBits, + int roundOffset) + => Vector512.Narrow( + Blend(first0, second0, Vector512.WidenLower(alpha), roundBits, roundOffset), + Blend(first1, second1, Vector512.WidenUpper(alpha), roundBits, roundOffset)); + + /// + /// Alpha-blends 128-bit lanes after widening every product to signed 32-bit precision. + /// + private static Vector128 Blend( + Vector128 first, + Vector128 second, + Vector128 alpha, + int roundBits, + int roundOffset) + { + Vector128 firstLower = Vector128.WidenLower(first).AsInt32(); + Vector128 firstUpper = Vector128.WidenUpper(first).AsInt32(); + Vector128 secondLower = Vector128.WidenLower(second).AsInt32(); + Vector128 secondUpper = Vector128.WidenUpper(second).AsInt32(); + Vector128 alphaLower = Vector128.WidenLower(alpha).AsInt32(); + Vector128 alphaUpper = Vector128.WidenUpper(alpha).AsInt32(); + Vector128 maximum = Vector128.Create(MaximumMaskAlpha); + Vector128 lower = ((alphaLower * firstLower) + ((maximum - alphaLower) * secondLower)) >> MaskWeightBits; + Vector128 upper = ((alphaUpper * firstUpper) + ((maximum - alphaUpper) * secondUpper)) >> MaskWeightBits; + return Vector128.Narrow(FinalizeIntermediate(lower, roundBits, roundOffset), FinalizeIntermediate(upper, roundBits, roundOffset)).AsUInt16(); + } + + /// + /// Alpha-blends 256-bit lanes after widening every product to signed 32-bit precision. + /// + private static Vector256 Blend( + Vector256 first, + Vector256 second, + Vector256 alpha, + int roundBits, + int roundOffset) + { + Vector256 firstLower = Vector256.WidenLower(first).AsInt32(); + Vector256 firstUpper = Vector256.WidenUpper(first).AsInt32(); + Vector256 secondLower = Vector256.WidenLower(second).AsInt32(); + Vector256 secondUpper = Vector256.WidenUpper(second).AsInt32(); + Vector256 alphaLower = Vector256.WidenLower(alpha).AsInt32(); + Vector256 alphaUpper = Vector256.WidenUpper(alpha).AsInt32(); + Vector256 maximum = Vector256.Create(MaximumMaskAlpha); + Vector256 lower = ((alphaLower * firstLower) + ((maximum - alphaLower) * secondLower)) >> MaskWeightBits; + Vector256 upper = ((alphaUpper * firstUpper) + ((maximum - alphaUpper) * secondUpper)) >> MaskWeightBits; + return Vector256.Narrow(FinalizeIntermediate(lower, roundBits, roundOffset), FinalizeIntermediate(upper, roundBits, roundOffset)).AsUInt16(); + } + + /// + /// Alpha-blends 512-bit lanes after widening every product to signed 32-bit precision. + /// + private static Vector512 Blend( + Vector512 first, + Vector512 second, + Vector512 alpha, + int roundBits, + int roundOffset) + { + Vector512 firstLower = Vector512.WidenLower(first).AsInt32(); + Vector512 firstUpper = Vector512.WidenUpper(first).AsInt32(); + Vector512 secondLower = Vector512.WidenLower(second).AsInt32(); + Vector512 secondUpper = Vector512.WidenUpper(second).AsInt32(); + Vector512 alphaLower = Vector512.WidenLower(alpha).AsInt32(); + Vector512 alphaUpper = Vector512.WidenUpper(alpha).AsInt32(); + Vector512 maximum = Vector512.Create(MaximumMaskAlpha); + Vector512 lower = ((alphaLower * firstLower) + ((maximum - alphaLower) * secondLower)) >> MaskWeightBits; + Vector512 upper = ((alphaUpper * firstUpper) + ((maximum - alphaUpper) * secondUpper)) >> MaskWeightBits; + return Vector512.Narrow(FinalizeIntermediate(lower, roundBits, roundOffset), FinalizeIntermediate(upper, roundBits, roundOffset)).AsUInt16(); + } + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateMaskBlendPredictor.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateMaskBlendPredictor.cs new file mode 100644 index 000000000..3c98a060a --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundIntermediateMaskBlendPredictor.cs @@ -0,0 +1,189 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using System.Runtime.Intrinsics; + +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor; +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; + +/// +/// Reconstructs final samples by alpha-blending compound intermediates. +/// +internal static partial class Av1CompoundIntermediateMaskBlendPredictor +{ + /// + /// Blends two compound intermediates through a luma-resolution mask. + /// + public static void BlendIntermediate( + Span destination, + int destinationStride, + ReadOnlySpan first, + int firstStride, + ReadOnlySpan second, + int secondStride, + ReadOnlySpan mask, + int maskStride, + int width, + int height, + int subX, + int subY, + int bitDepth) + => BlendIntermediate( + destination, + destinationStride, + first, + firstStride, + second, + secondStride, + mask, + maskStride, + width, + height, + subX, + subY, + bitDepth); + + /// + /// Executes one closed alpha-blend compound-intermediate operator. + /// + /// The compound-intermediate operator. + private static void BlendIntermediate( + Span destination, + int destinationStride, + ReadOnlySpan first, + int firstStride, + ReadOnlySpan second, + int secondStride, + ReadOnlySpan mask, + int maskStride, + int width, + int height, + int subX, + int subY, + int bitDepth) + where TOperator : struct, IAv1CompoundIntermediateMaskBlendOperator + { + GetIntermediateRounding(bitDepth, out int roundBits, out int roundOffset); + for (int row = 0; row < height; row++) + { + Span destinationRow = destination.Slice(row * destinationStride, width); + ReadOnlySpan firstRow = first.Slice(row * firstStride, width); + ReadOnlySpan secondRow = second.Slice(row * secondStride, width); + ref byte destinationReference = ref MemoryMarshal.GetReference(destinationRow); + ref ushort firstReference = ref MemoryMarshal.GetReference(firstRow); + ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow); + int column = 0; + + if (Vector512.IsHardwareAccelerated && subX == 0 && subY == 0) + { + ref byte maskReference = ref MemoryMarshal.GetReference(mask); + int maskRowOffset = row * maskStride; + int vectorEnd = width - Vector512.Count; + for (; column <= vectorEnd; column += Vector512.Count) + { + Vector512 first0 = Vector512.LoadUnsafe(ref firstReference, (nuint)column); + Vector512 first1 = Vector512.LoadUnsafe(ref firstReference, (nuint)(column + Vector512.Count)); + Vector512 second0 = Vector512.LoadUnsafe(ref secondReference, (nuint)column); + Vector512 second1 = Vector512.LoadUnsafe(ref secondReference, (nuint)(column + Vector512.Count)); + Vector512 alpha = Vector512.LoadUnsafe(ref maskReference, (nuint)(maskRowOffset + column)); + TOperator.Blend(first0, first1, second0, second1, alpha, roundBits, roundOffset) + .StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + if (Vector256.IsHardwareAccelerated && subX == 0 && subY == 0) + { + ref byte maskReference = ref MemoryMarshal.GetReference(mask); + int maskRowOffset = row * maskStride; + int vectorEnd = width - Vector256.Count; + for (; column <= vectorEnd; column += Vector256.Count) + { + Vector256 first0 = Vector256.LoadUnsafe(ref firstReference, (nuint)column); + Vector256 first1 = Vector256.LoadUnsafe(ref firstReference, (nuint)(column + Vector256.Count)); + Vector256 second0 = Vector256.LoadUnsafe(ref secondReference, (nuint)column); + Vector256 second1 = Vector256.LoadUnsafe(ref secondReference, (nuint)(column + Vector256.Count)); + Vector256 alpha = Vector256.LoadUnsafe(ref maskReference, (nuint)(maskRowOffset + column)); + TOperator.Blend(first0, first1, second0, second1, alpha, roundBits, roundOffset) + .StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + if (Vector128.IsHardwareAccelerated && subX == 0 && subY == 0) + { + ref byte maskReference = ref MemoryMarshal.GetReference(mask); + int maskRowOffset = row * maskStride; + int vectorEnd = width - Vector128.Count; + for (; column <= vectorEnd; column += Vector128.Count) + { + Vector128 first0 = Vector128.LoadUnsafe(ref firstReference, (nuint)column); + Vector128 first1 = Vector128.LoadUnsafe( + ref firstReference, + (nuint)(column + Vector128.Count)); + + Vector128 second0 = Vector128.LoadUnsafe(ref secondReference, (nuint)column); + Vector128 second1 = Vector128.LoadUnsafe( + ref secondReference, + (nuint)(column + Vector128.Count)); + + Vector128 alpha = Vector128.LoadUnsafe( + ref maskReference, + (nuint)(maskRowOffset + column)); + + TOperator.Blend( + first0, + first1, + second0, + second1, + alpha, + roundBits, + roundOffset).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + for (; column < width; column++) + { + byte alpha = (byte)GetSubsampledMaskAlpha(mask, maskStride, row, column, subX, subY); + destinationRow[column] = TOperator.Blend(firstRow[column], secondRow[column], alpha, roundBits, roundOffset); + } + } + } + + /// + /// Gets the mask alpha for one plane sample, averaging its two or four luma samples when required. + /// + private static int GetSubsampledMaskAlpha( + ReadOnlySpan mask, + int maskStride, + int row, + int column, + int subX, + int subY) + { + int maskRow = row << subY; + int maskColumn = column << subX; + int alpha = mask[(maskRow * maskStride) + maskColumn]; + if (subX != 0) + { + alpha += mask[(maskRow * maskStride) + maskColumn + 1]; + } + + if (subY != 0) + { + int lowerOffset = ((maskRow + 1) * maskStride) + maskColumn; + alpha += mask[lowerOffset]; + if (subX != 0) + { + alpha += mask[lowerOffset + 1]; + } + } + + int sampleCountShift = subX + subY; + return sampleCountShift == 0 + ? alpha + : RoundPowerOfTwo(alpha, sampleCountShift); + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundMaskBlendPredictor.Operator.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundMaskBlendPredictor.Operator.cs new file mode 100644 index 000000000..43889c979 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundMaskBlendPredictor.Operator.cs @@ -0,0 +1,203 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.Intrinsics; + +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor; +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; + +/// +/// Defines alpha-masked compound prediction arithmetic. +/// +internal static partial class Av1CompoundMaskBlendPredictor +{ + /// + /// Defines alpha-masked compound blending for scalar and SIMD lane groups. + /// + private interface IAv1CompoundMaskBlendOperator + { + /// + /// Blends two 8-bit samples through an AV1 alpha value. + /// + /// The first sample. + /// The second sample. + /// The first-sample weight in the AV1 mask range. + /// The blended sample. + public static abstract byte Blend(byte first, byte second, byte alpha); + + /// + /// Blends two high-bit-depth samples through an AV1 alpha value. + /// + /// The first sample. + /// The second sample. + /// The first-sample weight in the AV1 mask range. + /// The blended sample. + public static abstract ushort Blend(ushort first, ushort second, byte alpha); + + /// + /// Blends 128-bit vectors of 8-bit samples through AV1 alpha values. + /// + /// The first samples. + /// The second samples. + /// The first-sample weights in the AV1 mask range. + /// The blended samples. + public static abstract Vector128 Blend(Vector128 first, Vector128 second, Vector128 alpha); + + /// + /// Blends 256-bit vectors of 8-bit samples through AV1 alpha values. + /// + /// The first samples. + /// The second samples. + /// The first-sample weights in the AV1 mask range. + /// The blended samples. + public static abstract Vector256 Blend(Vector256 first, Vector256 second, Vector256 alpha); + + /// + /// Blends 512-bit vectors of 8-bit samples through AV1 alpha values. + /// + /// The first samples. + /// The second samples. + /// The first-sample weights in the AV1 mask range. + /// The blended samples. + public static abstract Vector512 Blend(Vector512 first, Vector512 second, Vector512 alpha); + + /// + /// Blends 128-bit vectors of high-bit-depth samples through AV1 alpha values. + /// + /// The first samples. + /// The second samples. + /// The first-sample weights in the AV1 mask range. + /// The blended samples. + public static abstract Vector128 Blend(Vector128 first, Vector128 second, Vector128 alpha); + + /// + /// Blends 256-bit vectors of high-bit-depth samples through AV1 alpha values. + /// + /// The first samples. + /// The second samples. + /// The first-sample weights in the AV1 mask range. + /// The blended samples. + public static abstract Vector256 Blend(Vector256 first, Vector256 second, Vector256 alpha); + + /// + /// Blends 512-bit vectors of high-bit-depth samples through AV1 alpha values. + /// + /// The first samples. + /// The second samples. + /// The first-sample weights in the AV1 mask range. + /// The blended samples. + public static abstract Vector512 Blend(Vector512 first, Vector512 second, Vector512 alpha); + } + + /// + /// Implements AV1 alpha-mask blending for scalar and SIMD lane groups. + /// + private readonly struct CompoundMaskBlendOperator : IAv1CompoundMaskBlendOperator + { + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static byte Blend(byte first, byte second, byte alpha) + => (byte)(((alpha * first) + ((MaximumMaskAlpha - alpha) * second) + 32) >> MaskWeightBits); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static ushort Blend(ushort first, ushort second, byte alpha) + => (ushort)(((alpha * first) + ((MaximumMaskAlpha - alpha) * second) + 32) >> MaskWeightBits); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector128 Blend(Vector128 first, Vector128 second, Vector128 alpha) + { + Av1IntraPredictorBase.Widen(first, out Vector128 first0, out Vector128 first1, out Vector128 first2, out Vector128 first3); + Av1IntraPredictorBase.Widen(second, out Vector128 second0, out Vector128 second1, out Vector128 second2, out Vector128 second3); + Av1IntraPredictorBase.Widen(alpha, out Vector128 alpha0, out Vector128 alpha1, out Vector128 alpha2, out Vector128 alpha3); + return Av1IntraPredictorBase.Narrow( + Blend(first0, second0, alpha0), + Blend(first1, second1, alpha1), + Blend(first2, second2, alpha2), + Blend(first3, second3, alpha3)); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector256 Blend(Vector256 first, Vector256 second, Vector256 alpha) + { + Av1IntraPredictorBase.Widen(first, out Vector256 first0, out Vector256 first1, out Vector256 first2, out Vector256 first3); + Av1IntraPredictorBase.Widen(second, out Vector256 second0, out Vector256 second1, out Vector256 second2, out Vector256 second3); + Av1IntraPredictorBase.Widen(alpha, out Vector256 alpha0, out Vector256 alpha1, out Vector256 alpha2, out Vector256 alpha3); + return Av1IntraPredictorBase.Narrow( + Blend(first0, second0, alpha0), + Blend(first1, second1, alpha1), + Blend(first2, second2, alpha2), + Blend(first3, second3, alpha3)); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector512 Blend(Vector512 first, Vector512 second, Vector512 alpha) + { + Av1IntraPredictorBase.Widen(first, out Vector512 first0, out Vector512 first1, out Vector512 first2, out Vector512 first3); + Av1IntraPredictorBase.Widen(second, out Vector512 second0, out Vector512 second1, out Vector512 second2, out Vector512 second3); + Av1IntraPredictorBase.Widen(alpha, out Vector512 alpha0, out Vector512 alpha1, out Vector512 alpha2, out Vector512 alpha3); + return Av1IntraPredictorBase.Narrow( + Blend(first0, second0, alpha0), + Blend(first1, second1, alpha1), + Blend(first2, second2, alpha2), + Blend(first3, second3, alpha3)); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector128 Blend(Vector128 first, Vector128 second, Vector128 alpha) + { + Av1IntraPredictorBase.Widen(first.AsInt16(), out Vector128 first0, out Vector128 first1); + Av1IntraPredictorBase.Widen(second.AsInt16(), out Vector128 second0, out Vector128 second1); + Av1IntraPredictorBase.Widen(alpha.AsInt16(), out Vector128 alpha0, out Vector128 alpha1); + return Av1IntraPredictorBase.Narrow(Blend(first0, second0, alpha0), Blend(first1, second1, alpha1)).AsUInt16(); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector256 Blend(Vector256 first, Vector256 second, Vector256 alpha) + { + Av1IntraPredictorBase.Widen(first.AsInt16(), out Vector256 first0, out Vector256 first1); + Av1IntraPredictorBase.Widen(second.AsInt16(), out Vector256 second0, out Vector256 second1); + Av1IntraPredictorBase.Widen(alpha.AsInt16(), out Vector256 alpha0, out Vector256 alpha1); + return Av1IntraPredictorBase.Narrow(Blend(first0, second0, alpha0), Blend(first1, second1, alpha1)).AsUInt16(); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector512 Blend(Vector512 first, Vector512 second, Vector512 alpha) + { + Av1IntraPredictorBase.Widen(first.AsInt16(), out Vector512 first0, out Vector512 first1); + Av1IntraPredictorBase.Widen(second.AsInt16(), out Vector512 second0, out Vector512 second1); + Av1IntraPredictorBase.Widen(alpha.AsInt16(), out Vector512 alpha0, out Vector512 alpha1); + return Av1IntraPredictorBase.Narrow(Blend(first0, second0, alpha0), Blend(first1, second1, alpha1)).AsUInt16(); + } + + /// + /// Applies alpha-mask blending to 128-bit vectors of widened samples. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector128 Blend(Vector128 first, Vector128 second, Vector128 alpha) + => ((alpha * first) + ((Vector128.Create(MaximumMaskAlpha) - alpha) * second) + Vector128.Create(32)) >> MaskWeightBits; + + /// + /// Applies alpha-mask blending to 256-bit vectors of widened samples. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector256 Blend(Vector256 first, Vector256 second, Vector256 alpha) + => ((alpha * first) + ((Vector256.Create(MaximumMaskAlpha) - alpha) * second) + Vector256.Create(32)) >> MaskWeightBits; + + /// + /// Applies alpha-mask blending to 512-bit vectors of widened samples. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector512 Blend(Vector512 first, Vector512 second, Vector512 alpha) + => ((alpha * first) + ((Vector512.Create(MaximumMaskAlpha) - alpha) * second) + Vector512.Create(32)) >> MaskWeightBits; + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundMaskBlendPredictor.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundMaskBlendPredictor.cs new file mode 100644 index 000000000..38ce40a50 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1CompoundMaskBlendPredictor.cs @@ -0,0 +1,267 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using System.Runtime.Intrinsics; + +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor; +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; + +/// +/// Reconstructs alpha-masked compound prediction. +/// +internal static partial class Av1CompoundMaskBlendPredictor +{ + /// + /// Blends two 8-bit predictors through a contiguous AV1 alpha mask. + /// + public static void Blend( + Span destination, + int destinationStride, + ReadOnlySpan second, + int secondStride, + ReadOnlySpan mask, + int maskStride, + int width, + int height) + => Blend( + destination, + destinationStride, + second, + secondStride, + mask, + maskStride, + width, + height); + + /// + /// Executes one closed 8-bit masked compound operator. + /// + /// The compound arithmetic operator. + private static void Blend( + Span destination, + int destinationStride, + ReadOnlySpan second, + int secondStride, + ReadOnlySpan mask, + int maskStride, + int width, + int height) + where TOperator : struct, IAv1CompoundMaskBlendOperator + { + for (int row = 0; row < height; row++) + { + Span destinationRow = destination.Slice(row * destinationStride, width); + ReadOnlySpan secondRow = second.Slice(row * secondStride, width); + ReadOnlySpan maskRow = mask.Slice(row * maskStride, width); + ref byte destinationReference = ref MemoryMarshal.GetReference(destinationRow); + ref byte secondReference = ref MemoryMarshal.GetReference(secondRow); + ref byte maskReference = ref MemoryMarshal.GetReference(maskRow); + int column = 0; + + if (Vector512.IsHardwareAccelerated) + { + int vectorEnd = width - Vector512.Count; + for (; column <= vectorEnd; column += Vector512.Count) + { + Vector512 firstVector = Vector512.LoadUnsafe(ref destinationReference, (nuint)column); + Vector512 secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column); + Vector512 maskVector = Vector512.LoadUnsafe(ref maskReference, (nuint)column); + TOperator.Blend(firstVector, secondVector, maskVector).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + if (Vector256.IsHardwareAccelerated) + { + int vectorEnd = width - Vector256.Count; + for (; column <= vectorEnd; column += Vector256.Count) + { + Vector256 firstVector = Vector256.LoadUnsafe(ref destinationReference, (nuint)column); + Vector256 secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column); + Vector256 maskVector = Vector256.LoadUnsafe(ref maskReference, (nuint)column); + TOperator.Blend(firstVector, secondVector, maskVector).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + if (Vector128.IsHardwareAccelerated) + { + int vectorEnd = width - Vector128.Count; + for (; column <= vectorEnd; column += Vector128.Count) + { + Vector128 firstVector = Vector128.LoadUnsafe(ref destinationReference, (nuint)column); + Vector128 secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column); + Vector128 maskVector = Vector128.LoadUnsafe(ref maskReference, (nuint)column); + TOperator.Blend(firstVector, secondVector, maskVector).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + for (; column < width; column++) + { + destinationRow[column] = TOperator.Blend(destinationRow[column], secondRow[column], maskRow[column]); + } + } + } + + /// + /// Blends two high-bit-depth predictors through a contiguous AV1 alpha mask. + /// + public static void Blend( + Span destination, + int destinationStride, + ReadOnlySpan second, + int secondStride, + ReadOnlySpan mask, + int maskStride, + int width, + int height) + => Blend( + destination, + destinationStride, + second, + secondStride, + mask, + maskStride, + width, + height); + + /// + /// Executes one closed high-bit-depth masked compound operator. + /// + /// The compound arithmetic operator. + private static void Blend( + Span destination, + int destinationStride, + ReadOnlySpan second, + int secondStride, + ReadOnlySpan mask, + int maskStride, + int width, + int height) + where TOperator : struct, IAv1CompoundMaskBlendOperator + { + for (int row = 0; row < height; row++) + { + Span destinationRow = destination.Slice(row * destinationStride, width); + ReadOnlySpan secondRow = second.Slice(row * secondStride, width); + ReadOnlySpan maskRow = mask.Slice(row * maskStride, width); + ref ushort destinationReference = ref MemoryMarshal.GetReference(destinationRow); + ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow); + ref byte maskReference = ref MemoryMarshal.GetReference(maskRow); + int column = 0; + + if (Vector512.IsHardwareAccelerated) + { + int vectorEnd = width - Vector512.Count; + for (; column <= vectorEnd; column += Vector512.Count) + { + Vector512 firstVector = Vector512.LoadUnsafe(ref destinationReference, (nuint)column); + Vector512 secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column); + Vector512 maskVector = LoadMask512(ref maskReference, column); + TOperator.Blend(firstVector, secondVector, maskVector).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + if (Vector256.IsHardwareAccelerated) + { + int vectorEnd = width - Vector256.Count; + for (; column <= vectorEnd; column += Vector256.Count) + { + Vector256 firstVector = Vector256.LoadUnsafe(ref destinationReference, (nuint)column); + Vector256 secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column); + Vector256 maskVector = LoadMask256(ref maskReference, column); + TOperator.Blend(firstVector, secondVector, maskVector).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + if (Vector128.IsHardwareAccelerated) + { + int vectorEnd = width - Vector128.Count; + for (; column <= vectorEnd; column += Vector128.Count) + { + Vector128 firstVector = Vector128.LoadUnsafe(ref destinationReference, (nuint)column); + Vector128 secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column); + Vector128 maskVector = LoadMask128(ref maskReference, column); + TOperator.Blend(firstVector, secondVector, maskVector).StoreUnsafe(ref destinationReference, (nuint)column); + } + } + + for (; column < width; column++) + { + destinationRow[column] = TOperator.Blend(destinationRow[column], secondRow[column], maskRow[column]); + } + } + } + + /// + /// Blends two 8-bit predictors through an alpha mask without explicit hardware intrinsics. + /// + public static void BlendScalar( + Span destination, + int destinationStride, + ReadOnlySpan second, + int secondStride, + ReadOnlySpan mask, + int maskStride, + int width, + int height) + => BlendScalar( + destination, + destinationStride, + second, + secondStride, + mask, + maskStride, + width, + height); + + /// + /// Executes one closed 8-bit masked compound operator without explicit hardware intrinsics. + /// + /// The compound arithmetic operator. + private static void BlendScalar( + Span destination, + int destinationStride, + ReadOnlySpan second, + int secondStride, + ReadOnlySpan mask, + int maskStride, + int width, + int height) + where TOperator : struct, IAv1CompoundMaskBlendOperator + { + for (int row = 0; row < height; row++) + { + Span destinationRow = destination.Slice(row * destinationStride, width); + ReadOnlySpan secondRow = second.Slice(row * secondStride, width); + ReadOnlySpan maskRow = mask.Slice(row * maskStride, width); + for (int column = 0; column < width; column++) + { + destinationRow[column] = TOperator.Blend(destinationRow[column], secondRow[column], maskRow[column]); + } + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector128 LoadMask128(ref byte source, int offset) + { + Vector64 packed = Unsafe.As>(ref Unsafe.Add(ref source, offset)); + return Vector128.WidenLower(Vector128.Create(packed, Vector64.Zero)); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector256 LoadMask256(ref byte source, int offset) + { + Vector128 packed = Vector128.LoadUnsafe(ref source, (nuint)offset); + return Vector256.WidenLower(Vector256.Create(packed, Vector128.Zero)); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector512 LoadMask512(ref byte source, int offset) + { + Vector256 packed = Vector256.LoadUnsafe(ref source, (nuint)offset); + return Vector512.WidenLower(Vector512.Create(packed, Vector256.Zero)); + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1DifferenceWeightedMaskBuilder.Operator.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1DifferenceWeightedMaskBuilder.Operator.cs new file mode 100644 index 000000000..5143c2fa3 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1DifferenceWeightedMaskBuilder.Operator.cs @@ -0,0 +1,254 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.Intrinsics; + +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor; +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; + +/// +/// Defines difference-weighted compound mask arithmetic. +/// +internal static partial class Av1DifferenceWeightedMaskBuilder +{ + /// + /// Defines difference-weighted compound mask generation for scalar and SIMD lane groups. + /// + private interface IAv1DifferenceWeightedMaskOperator + { + /// + /// Creates one mask value from two 8-bit predictor samples. + /// + /// The first predictor sample. + /// The second predictor sample. + /// The difference scaling shift. + /// Whether to invert the selected predictor. + /// The AV1 mask value. + public static abstract byte Create(byte first, byte second, int shift, bool invert); + + /// + /// Creates one mask value from two high-bit-depth predictor samples. + /// + /// The first predictor sample. + /// The second predictor sample. + /// The difference scaling shift. + /// Whether to invert the selected predictor. + /// The AV1 mask value. + public static abstract byte Create(ushort first, ushort second, int shift, bool invert); + + /// + /// Creates 128 bits of mask values from 8-bit predictor samples. + /// + /// The first predictor samples. + /// The second predictor samples. + /// The difference scaling shift. + /// Whether to invert the selected predictor. + /// The AV1 mask values. + public static abstract Vector128 Create(Vector128 first, Vector128 second, int shift, bool invert); + + /// + /// Creates 256 bits of mask values from 8-bit predictor samples. + /// + /// The first predictor samples. + /// The second predictor samples. + /// The difference scaling shift. + /// Whether to invert the selected predictor. + /// The AV1 mask values. + public static abstract Vector256 Create(Vector256 first, Vector256 second, int shift, bool invert); + + /// + /// Creates 512 bits of mask values from 8-bit predictor samples. + /// + /// The first predictor samples. + /// The second predictor samples. + /// The difference scaling shift. + /// Whether to invert the selected predictor. + /// The AV1 mask values. + public static abstract Vector512 Create(Vector512 first, Vector512 second, int shift, bool invert); + + /// + /// Creates 128 bits of packed mask values from high-bit-depth predictor samples. + /// + /// The lower first-predictor samples. + /// The upper first-predictor samples. + /// The lower second-predictor samples. + /// The upper second-predictor samples. + /// The difference scaling shift. + /// Whether to invert the selected predictor. + /// The packed AV1 mask values. + public static abstract Vector128 Create( + Vector128 first0, + Vector128 first1, + Vector128 second0, + Vector128 second1, + int shift, + bool invert); + + /// + /// Creates 256 bits of packed mask values from high-bit-depth predictor samples. + /// + /// The lower first-predictor samples. + /// The upper first-predictor samples. + /// The lower second-predictor samples. + /// The upper second-predictor samples. + /// The difference scaling shift. + /// Whether to invert the selected predictor. + /// The packed AV1 mask values. + public static abstract Vector256 Create( + Vector256 first0, + Vector256 first1, + Vector256 second0, + Vector256 second1, + int shift, + bool invert); + + /// + /// Creates 512 bits of packed mask values from high-bit-depth predictor samples. + /// + /// The lower first-predictor samples. + /// The upper first-predictor samples. + /// The lower second-predictor samples. + /// The upper second-predictor samples. + /// The difference scaling shift. + /// Whether to invert the selected predictor. + /// The packed AV1 mask values. + public static abstract Vector512 Create( + Vector512 first0, + Vector512 first1, + Vector512 second0, + Vector512 second1, + int shift, + bool invert); + } + + /// + /// Implements AV1 difference-weighted mask generation for scalar and SIMD lane groups. + /// + private readonly struct DifferenceWeightedMaskOperator : IAv1DifferenceWeightedMaskOperator + { + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static byte Create(byte first, byte second, int shift, bool invert) + => Create((ushort)Math.Abs(first - second), shift, invert); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static byte Create(ushort first, ushort second, int shift, bool invert) + => Create((ushort)Math.Abs(first - second), shift, invert); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector128 Create(Vector128 first, Vector128 second, int shift, bool invert) + { + Vector128 difference = Vector128.Max(first, second) - Vector128.Min(first, second); + Vector128 lower = CreateAlpha(Vector128.WidenLower(difference), shift, invert); + Vector128 upper = CreateAlpha(Vector128.WidenUpper(difference), shift, invert); + return Vector128.Narrow(lower, upper); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector256 Create(Vector256 first, Vector256 second, int shift, bool invert) + { + Vector256 difference = Vector256.Max(first, second) - Vector256.Min(first, second); + Vector256 lower = CreateAlpha(Vector256.WidenLower(difference), shift, invert); + Vector256 upper = CreateAlpha(Vector256.WidenUpper(difference), shift, invert); + return Vector256.Narrow(lower, upper); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector512 Create(Vector512 first, Vector512 second, int shift, bool invert) + { + Vector512 difference = Vector512.Max(first, second) - Vector512.Min(first, second); + Vector512 lower = CreateAlpha(Vector512.WidenLower(difference), shift, invert); + Vector512 upper = CreateAlpha(Vector512.WidenUpper(difference), shift, invert); + return Vector512.Narrow(lower, upper); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector128 Create( + Vector128 first0, + Vector128 first1, + Vector128 second0, + Vector128 second1, + int shift, + bool invert) + => Vector128.Narrow( + CreateAlpha(Vector128.Max(first0, second0) - Vector128.Min(first0, second0), shift, invert), + CreateAlpha(Vector128.Max(first1, second1) - Vector128.Min(first1, second1), shift, invert)); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector256 Create( + Vector256 first0, + Vector256 first1, + Vector256 second0, + Vector256 second1, + int shift, + bool invert) + => Vector256.Narrow( + CreateAlpha(Vector256.Max(first0, second0) - Vector256.Min(first0, second0), shift, invert), + CreateAlpha(Vector256.Max(first1, second1) - Vector256.Min(first1, second1), shift, invert)); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector512 Create( + Vector512 first0, + Vector512 first1, + Vector512 second0, + Vector512 second1, + int shift, + bool invert) + => Vector512.Narrow( + CreateAlpha(Vector512.Max(first0, second0) - Vector512.Min(first0, second0), shift, invert), + CreateAlpha(Vector512.Max(first1, second1) - Vector512.Min(first1, second1), shift, invert)); + + /// + /// Creates one mask value from an absolute predictor difference. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static byte Create(ushort difference, int shift, bool invert) + { + int alpha = Math.Min(MaximumMaskAlpha, 38 + (difference >> shift)); + return (byte)(invert ? MaximumMaskAlpha - alpha : alpha); + } + + /// + /// Creates 128-bit vectors of unpacked mask values from absolute predictor differences. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector128 CreateAlpha(Vector128 difference, int shift, bool invert) + { + Vector128 maximum = Vector128.Create((ushort)MaximumMaskAlpha); + Vector128 alpha = Vector128.Min(maximum, (difference >> shift) + Vector128.Create((ushort)38)); + return invert ? maximum - alpha : alpha; + } + + /// + /// Creates 256-bit vectors of unpacked mask values from absolute predictor differences. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector256 CreateAlpha(Vector256 difference, int shift, bool invert) + { + Vector256 maximum = Vector256.Create((ushort)MaximumMaskAlpha); + Vector256 alpha = Vector256.Min(maximum, (difference >> shift) + Vector256.Create((ushort)38)); + return invert ? maximum - alpha : alpha; + } + + /// + /// Creates 512-bit vectors of unpacked mask values from absolute predictor differences. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector512 CreateAlpha(Vector512 difference, int shift, bool invert) + { + Vector512 maximum = Vector512.Create((ushort)MaximumMaskAlpha); + Vector512 alpha = Vector512.Min(maximum, (difference >> shift) + Vector512.Create((ushort)38)); + return invert ? maximum - alpha : alpha; + } + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1DifferenceWeightedMaskBuilder.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1DifferenceWeightedMaskBuilder.cs new file mode 100644 index 000000000..f7339b6a1 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1DifferenceWeightedMaskBuilder.cs @@ -0,0 +1,214 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.InteropServices; +using System.Runtime.Intrinsics; +using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; + +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor; +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; + +/// +/// Builds AV1 difference-weighted compound masks. +/// +internal static partial class Av1DifferenceWeightedMaskBuilder +{ + /// + /// Fills an 8-bit difference-weighted compound mask. + /// + public static void FillDifferenceWeightedMask( + Span mask, + int maskStride, + ReadOnlySpan first, + int firstStride, + ReadOnlySpan second, + int secondStride, + int width, + int height, + Av1DifferenceWeightedMaskType maskType) + => FillDifferenceWeightedMask( + mask, + maskStride, + first, + firstStride, + second, + secondStride, + width, + height, + maskType); + + /// + /// Executes one closed 8-bit difference-weighted mask operator. + /// + /// The difference-weighted mask operator. + private static void FillDifferenceWeightedMask( + Span mask, + int maskStride, + ReadOnlySpan first, + int firstStride, + ReadOnlySpan second, + int secondStride, + int width, + int height, + Av1DifferenceWeightedMaskType maskType) + where TOperator : struct, IAv1DifferenceWeightedMaskOperator + { + bool invert = maskType == Av1DifferenceWeightedMaskType.Type38Inverse; + for (int row = 0; row < height; row++) + { + Span maskRow = mask.Slice(row * maskStride, width); + ReadOnlySpan firstRow = first.Slice(row * firstStride, width); + ReadOnlySpan secondRow = second.Slice(row * secondStride, width); + ref byte maskReference = ref MemoryMarshal.GetReference(maskRow); + ref byte firstReference = ref MemoryMarshal.GetReference(firstRow); + ref byte secondReference = ref MemoryMarshal.GetReference(secondRow); + int column = 0; + + if (Vector512.IsHardwareAccelerated) + { + int vectorEnd = width - Vector512.Count; + for (; column <= vectorEnd; column += Vector512.Count) + { + Vector512 firstVector = Vector512.LoadUnsafe(ref firstReference, (nuint)column); + Vector512 secondVector = Vector512.LoadUnsafe(ref secondReference, (nuint)column); + TOperator.Create(firstVector, secondVector, 4, invert).StoreUnsafe(ref maskReference, (nuint)column); + } + } + + if (Vector256.IsHardwareAccelerated) + { + int vectorEnd = width - Vector256.Count; + for (; column <= vectorEnd; column += Vector256.Count) + { + Vector256 firstVector = Vector256.LoadUnsafe(ref firstReference, (nuint)column); + Vector256 secondVector = Vector256.LoadUnsafe(ref secondReference, (nuint)column); + TOperator.Create(firstVector, secondVector, 4, invert).StoreUnsafe(ref maskReference, (nuint)column); + } + } + + if (Vector128.IsHardwareAccelerated) + { + int vectorEnd = width - Vector128.Count; + for (; column <= vectorEnd; column += Vector128.Count) + { + Vector128 firstVector = Vector128.LoadUnsafe(ref firstReference, (nuint)column); + Vector128 secondVector = Vector128.LoadUnsafe(ref secondReference, (nuint)column); + TOperator.Create(firstVector, secondVector, 4, invert).StoreUnsafe(ref maskReference, (nuint)column); + } + } + + for (; column < width; column++) + { + maskRow[column] = TOperator.Create(firstRow[column], secondRow[column], 4, invert); + } + } + } + + /// + /// Fills a high-bit-depth difference-weighted compound mask. + /// + public static void FillDifferenceWeightedMask( + Span mask, + int maskStride, + ReadOnlySpan first, + int firstStride, + ReadOnlySpan second, + int secondStride, + int width, + int height, + int bitDepth, + Av1DifferenceWeightedMaskType maskType) + => FillDifferenceWeightedMask( + mask, + maskStride, + first, + firstStride, + second, + secondStride, + width, + height, + bitDepth, + maskType); + + /// + /// Executes one closed high-bit-depth difference-weighted mask operator. + /// + /// The difference-weighted mask operator. + private static void FillDifferenceWeightedMask( + Span mask, + int maskStride, + ReadOnlySpan first, + int firstStride, + ReadOnlySpan second, + int secondStride, + int width, + int height, + int bitDepth, + Av1DifferenceWeightedMaskType maskType) + where TOperator : struct, IAv1DifferenceWeightedMaskOperator + { + bool invert = maskType == Av1DifferenceWeightedMaskType.Type38Inverse; + int differenceShift = bitDepth - 8 + 4; + for (int row = 0; row < height; row++) + { + Span maskRow = mask.Slice(row * maskStride, width); + ReadOnlySpan firstRow = first.Slice(row * firstStride, width); + ReadOnlySpan secondRow = second.Slice(row * secondStride, width); + ref byte maskReference = ref MemoryMarshal.GetReference(maskRow); + ref ushort firstReference = ref MemoryMarshal.GetReference(firstRow); + ref ushort secondReference = ref MemoryMarshal.GetReference(secondRow); + int column = 0; + + // Two input vectors narrow to one packed byte mask. This keeps mask construction contiguous and avoids + // temporary buffers before the following vector blend consumes the complete plane block. + if (Vector512.IsHardwareAccelerated) + { + int vectorEnd = width - Vector512.Count; + for (; column <= vectorEnd; column += Vector512.Count) + { + Vector512 first0 = Vector512.LoadUnsafe(ref firstReference, (nuint)column); + Vector512 first1 = Vector512.LoadUnsafe(ref firstReference, (nuint)(column + Vector512.Count)); + Vector512 second0 = Vector512.LoadUnsafe(ref secondReference, (nuint)column); + Vector512 second1 = Vector512.LoadUnsafe(ref secondReference, (nuint)(column + Vector512.Count)); + TOperator.Create(first0, first1, second0, second1, differenceShift, invert) + .StoreUnsafe(ref maskReference, (nuint)column); + } + } + + if (Vector256.IsHardwareAccelerated) + { + int vectorEnd = width - Vector256.Count; + for (; column <= vectorEnd; column += Vector256.Count) + { + Vector256 first0 = Vector256.LoadUnsafe(ref firstReference, (nuint)column); + Vector256 first1 = Vector256.LoadUnsafe(ref firstReference, (nuint)(column + Vector256.Count)); + Vector256 second0 = Vector256.LoadUnsafe(ref secondReference, (nuint)column); + Vector256 second1 = Vector256.LoadUnsafe(ref secondReference, (nuint)(column + Vector256.Count)); + TOperator.Create(first0, first1, second0, second1, differenceShift, invert) + .StoreUnsafe(ref maskReference, (nuint)column); + } + } + + if (Vector128.IsHardwareAccelerated) + { + int vectorEnd = width - Vector128.Count; + for (; column <= vectorEnd; column += Vector128.Count) + { + Vector128 first0 = Vector128.LoadUnsafe(ref firstReference, (nuint)column); + Vector128 first1 = Vector128.LoadUnsafe(ref firstReference, (nuint)(column + Vector128.Count)); + Vector128 second0 = Vector128.LoadUnsafe(ref secondReference, (nuint)column); + Vector128 second1 = Vector128.LoadUnsafe(ref secondReference, (nuint)(column + Vector128.Count)); + TOperator.Create(first0, first1, second0, second1, differenceShift, invert) + .StoreUnsafe(ref maskReference, (nuint)column); + } + } + + for (; column < width; column++) + { + maskRow[column] = TOperator.Create(firstRow[column], secondRow[column], differenceShift, invert); + } + } + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterIntraMaskBuilder.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterIntraMaskBuilder.cs new file mode 100644 index 000000000..5f7244f7f --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterIntraMaskBuilder.cs @@ -0,0 +1,62 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.InteropServices; +using System.Runtime.Intrinsics; +using SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; + +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1CompoundInterPredictor; +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; + +/// +/// Builds AV1 inter-intra prediction masks. +/// +internal static partial class Av1InterIntraMaskBuilder +{ + /// + /// Gets libaom's one-dimensional inter-intra alpha curve. + /// + private static ReadOnlySpan InterIntraWeights => + [ + 60, 58, 56, 54, 52, 50, 48, 47, 45, 44, 42, 41, 39, 38, 37, 35, + 34, 33, 32, 31, 30, 29, 28, 27, 26, 25, 24, 23, 22, 22, 21, 20, + 19, 19, 18, 18, 17, 16, 16, 15, 15, 14, 14, 13, 13, 12, 12, 12, + 11, 11, 10, 10, 10, 9, 9, 9, 8, 8, 8, 8, 7, 7, 7, 7, + 6, 6, 6, 6, 6, 5, 5, 5, 5, 5, 4, 4, 4, 4, 4, 4, + 4, 4, 3, 3, 3, 3, 3, 3, 3, 3, 3, 2, 2, 2, 2, 2, + 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, + ]; + + /// + /// Fills a smooth inter-intra mask for one plane. + /// + public static void FillInterIntraMask( + Span mask, + int maskStride, + int width, + int height, + Av1InterIntraMode mode, + bool invert) + { + int sizeScale = 128 / Math.Max(width, height); + for (int row = 0; row < height; row++) + { + Span maskRow = mask.Slice(row * maskStride, width); + for (int column = 0; column < width; column++) + { + int alpha = mode switch + { + Av1InterIntraMode.Vertical => InterIntraWeights[row * sizeScale], + Av1InterIntraMode.Horizontal => InterIntraWeights[column * sizeScale], + Av1InterIntraMode.Smooth => InterIntraWeights[Math.Min(row, column) * sizeScale], + _ => 32, + }; + + maskRow[column] = (byte)(invert ? MaximumMaskAlpha - alpha : alpha); + } + } + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Arithmetic.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Arithmetic.cs index 81d83db1e..7f355d14a 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Arithmetic.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Arithmetic.cs @@ -15,7 +15,7 @@ internal static partial class Av1InterPredictor /// Convolves sixteen adjacent 8-bit samples into four signed 32-bit accumulator vectors. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static void Convolve( + internal static void Convolve( ref byte source, int tapStride, nuint column, @@ -51,7 +51,7 @@ internal static partial class Av1InterPredictor /// Convolves thirty-two adjacent 8-bit samples into four signed 32-bit accumulator vectors. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static void Convolve( + internal static void Convolve( ref byte source, int tapStride, nuint column, @@ -85,7 +85,7 @@ internal static partial class Av1InterPredictor /// Convolves sixty-four adjacent 8-bit samples into four signed 32-bit accumulator vectors. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static void Convolve( + internal static void Convolve( ref byte source, int tapStride, nuint column, @@ -119,7 +119,7 @@ internal static partial class Av1InterPredictor /// Convolves eight adjacent nonnegative 16-bit samples into two signed 32-bit accumulator vectors. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static void Convolve( + internal static void Convolve( ref short source, int tapStride, nuint column, @@ -149,7 +149,7 @@ internal static partial class Av1InterPredictor /// Convolves sixteen adjacent nonnegative 16-bit samples into two signed 32-bit accumulator vectors. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static void Convolve( + internal static void Convolve( ref short source, int tapStride, nuint column, @@ -176,7 +176,7 @@ internal static partial class Av1InterPredictor /// Convolves thirty-two adjacent nonnegative 16-bit samples into two signed 32-bit accumulator vectors. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static void Convolve( + internal static void Convolve( ref short source, int tapStride, nuint column, @@ -203,28 +203,28 @@ internal static partial class Av1InterPredictor /// Applies AV1 power-of-two rounding to four-lane signed accumulators. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector128 RoundPowerOfTwo(Vector128 value, int bits) + internal static Vector128 RoundPowerOfTwo(Vector128 value, int bits) => bits == 0 ? value : (value + Vector128.Create(1 << (bits - 1))) >> bits; /// /// Applies AV1 power-of-two rounding to eight-lane signed accumulators. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector256 RoundPowerOfTwo(Vector256 value, int bits) + internal static Vector256 RoundPowerOfTwo(Vector256 value, int bits) => bits == 0 ? value : (value + Vector256.Create(1 << (bits - 1))) >> bits; /// /// Applies AV1 power-of-two rounding to sixteen-lane signed accumulators. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector512 RoundPowerOfTwo(Vector512 value, int bits) + internal static Vector512 RoundPowerOfTwo(Vector512 value, int bits) => bits == 0 ? value : (value + Vector512.Create(1 << (bits - 1))) >> bits; /// /// Clips and packs sixteen signed accumulators into 8-bit samples. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector128 PackBytes(Vector128 result0, Vector128 result1, Vector128 result2, Vector128 result3) + internal static Vector128 PackBytes(Vector128 result0, Vector128 result1, Vector128 result2, Vector128 result3) { Vector128 maximum = Vector128.Create((int)byte.MaxValue); result0 = Vector128.Clamp(result0, Vector128.Zero, maximum); @@ -238,7 +238,7 @@ internal static partial class Av1InterPredictor /// Clips and packs thirty-two signed accumulators into 8-bit samples. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector256 PackBytes(Vector256 result0, Vector256 result1, Vector256 result2, Vector256 result3) + internal static Vector256 PackBytes(Vector256 result0, Vector256 result1, Vector256 result2, Vector256 result3) { Vector256 maximum = Vector256.Create((int)byte.MaxValue); result0 = Vector256.Clamp(result0, Vector256.Zero, maximum); @@ -252,7 +252,7 @@ internal static partial class Av1InterPredictor /// Clips and packs sixty-four signed accumulators into 8-bit samples. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector512 PackBytes(Vector512 result0, Vector512 result1, Vector512 result2, Vector512 result3) + internal static Vector512 PackBytes(Vector512 result0, Vector512 result1, Vector512 result2, Vector512 result3) { Vector512 maximum = Vector512.Create((int)byte.MaxValue); result0 = Vector512.Clamp(result0, Vector512.Zero, maximum); @@ -266,7 +266,7 @@ internal static partial class Av1InterPredictor /// Clips and packs eight signed accumulators into high-bit-depth samples. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector128 PackHighBitDepth(Vector128 result0, Vector128 result1, int maximumValue) + internal static Vector128 PackHighBitDepth(Vector128 result0, Vector128 result1, int maximumValue) { Vector128 maximum = Vector128.Create(maximumValue); result0 = Vector128.Clamp(result0, Vector128.Zero, maximum); @@ -278,7 +278,7 @@ internal static partial class Av1InterPredictor /// Clips and packs sixteen signed accumulators into high-bit-depth samples. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector256 PackHighBitDepth(Vector256 result0, Vector256 result1, int maximumValue) + internal static Vector256 PackHighBitDepth(Vector256 result0, Vector256 result1, int maximumValue) { Vector256 maximum = Vector256.Create(maximumValue); result0 = Vector256.Clamp(result0, Vector256.Zero, maximum); @@ -290,7 +290,7 @@ internal static partial class Av1InterPredictor /// Clips and packs thirty-two signed accumulators into high-bit-depth samples. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector512 PackHighBitDepth(Vector512 result0, Vector512 result1, int maximumValue) + internal static Vector512 PackHighBitDepth(Vector512 result0, Vector512 result1, int maximumValue) { Vector512 maximum = Vector512.Create(maximumValue); result0 = Vector512.Clamp(result0, Vector512.Zero, maximum); @@ -301,7 +301,7 @@ internal static partial class Av1InterPredictor /// /// Computes one signed Q7 convolution sum from 8-bit samples. /// - private static int ConvolveScalar(ref byte source, int sourceStride, ref short coefficients, int tapCount) + internal static int ConvolveScalar(ref byte source, int sourceStride, ref short coefficients, int tapCount) { int sum = 0; for (int tap = 0; tap < tapCount; tap++) @@ -315,7 +315,7 @@ internal static partial class Av1InterPredictor /// /// Computes one signed Q7 convolution sum from high-bit-depth samples. /// - private static int ConvolveScalar(ref ushort source, int sourceStride, ref short coefficients, int tapCount) + internal static int ConvolveScalar(ref ushort source, int sourceStride, ref short coefficients, int tapCount) { int sum = 0; for (int tap = 0; tap < tapCount; tap++) @@ -329,7 +329,7 @@ internal static partial class Av1InterPredictor /// /// Computes one signed Q7 convolution sum from biased intermediate samples. /// - private static int ConvolveScalar(ref short source, int sourceStride, ref short coefficients, int tapCount) + internal static int ConvolveScalar(ref short source, int sourceStride, ref short coefficients, int tapCount) { int sum = 0; for (int tap = 0; tap < tapCount; tap++) @@ -343,5 +343,5 @@ internal static partial class Av1InterPredictor /// /// Rounds an integer after division by a power of two using AV1's unsigned-bias rule. /// - private static int RoundPowerOfTwo(int value, int bits) => bits == 0 ? value : (value + (1 << (bits - 1))) >> bits; + internal static int RoundPowerOfTwo(int value, int bits) => bits == 0 ? value : (value + (1 << (bits - 1))) >> bits; } diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Compound.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Compound.cs deleted file mode 100644 index 7327a2c34..000000000 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Compound.cs +++ /dev/null @@ -1,732 +0,0 @@ -// Copyright (c) Six Labors. -// Licensed under the Six Labors Split License. - -using System.Runtime.CompilerServices; -using System.Runtime.InteropServices; -using System.Runtime.Intrinsics; - -namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; - -/// -/// Produces the biased high-precision intermediates required by compound inter prediction. -/// -internal static partial class Av1InterPredictor -{ - /// - /// The second-round shift retained by every compound convolution path. - /// - internal const int CompoundRound1Bits = 7; - - /// - /// Reconstructs one 8-bit translational reference into AV1's unsigned compound intermediate format. - /// - public static void PredictCompound( - ReadOnlySpan source, - int sourceStride, - int sourceOrigin, - Span destination, - int destinationStride, - int width, - int height, - Av1InterpolationFilter horizontalFilter, - Av1InterpolationFilter verticalFilter, - int horizontalPhase, - int verticalPhase, - Span scratch) - { - if (Vector128.IsHardwareAccelerated) - { - PredictCompoundVector128( - source, - sourceStride, - sourceOrigin, - destination, - destinationStride, - width, - height, - horizontalFilter, - verticalFilter, - horizontalPhase, - verticalPhase, - scratch); - - return; - } - - PredictCompoundScalar( - source, - sourceStride, - sourceOrigin, - destination, - destinationStride, - width, - height, - horizontalFilter, - verticalFilter, - horizontalPhase, - verticalPhase, - scratch); - } - - /// - /// Reconstructs one compound intermediate without explicit hardware intrinsics. - /// - public static void PredictCompoundScalar( - ReadOnlySpan source, - int sourceStride, - int sourceOrigin, - Span destination, - int destinationStride, - int width, - int height, - Av1InterpolationFilter horizontalFilter, - Av1InterpolationFilter verticalFilter, - int horizontalPhase, - int verticalPhase, - Span scratch) - { - ReadOnlySpan horizontalCoefficients = GetCompoundCoefficients(horizontalFilter, horizontalPhase, width <= 4); - ReadOnlySpan verticalCoefficients = GetCompoundCoefficients(verticalFilter, verticalPhase, height <= 4); - int roundBits = (2 * FilterBits) - Round0Bits - CompoundRound1Bits; - int offsetBits = 8 + (2 * FilterBits) - Round0Bits; - int roundOffset = (1 << (offsetBits - CompoundRound1Bits)) + - (1 << (offsetBits - CompoundRound1Bits - 1)); - - ref byte sourceBase = ref Unsafe.Add(ref MemoryMarshal.GetReference(source), sourceOrigin); - ref ushort destinationBase = ref MemoryMarshal.GetReference(destination); - - if (horizontalPhase == 0 && verticalPhase == 0) - { - for (int row = 0; row < height; row++) - { - ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride); - ref ushort destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); - for (int column = 0; column < width; column++) - { - Unsafe.Add(ref destinationRow, column) = - (ushort)((Unsafe.Add(ref sourceRow, column) << roundBits) + roundOffset); - } - } - - return; - } - - if (verticalPhase == 0) - { - GetEffectiveKernel(horizontalCoefficients, out int firstCoefficient, out int tapCount); - ref short coefficientBase = ref Unsafe.Add( - ref MemoryMarshal.GetReference(horizontalCoefficients), - firstCoefficient); - - int sourceOffset = firstCoefficient - 3; - for (int row = 0; row < height; row++) - { - ref byte sourceRow = ref Unsafe.Add(ref sourceBase, (row * sourceStride) + sourceOffset); - ref ushort destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); - for (int column = 0; column < width; column++) - { - int sum = ConvolveScalar(ref Unsafe.Add(ref sourceRow, column), 1, ref coefficientBase, tapCount); - int result = (RoundPowerOfTwo(sum, Round0Bits) << (FilterBits - CompoundRound1Bits)) + roundOffset; - Unsafe.Add(ref destinationRow, column) = (ushort)result; - } - } - - return; - } - - if (horizontalPhase == 0) - { - GetEffectiveKernel(verticalCoefficients, out int firstCoefficient, out int tapCount); - ref short coefficientBase = ref Unsafe.Add( - ref MemoryMarshal.GetReference(verticalCoefficients), - firstCoefficient); - - int sourceOffset = firstCoefficient - 3; - int firstPassBits = FilterBits - Round0Bits; - for (int row = 0; row < height; row++) - { - ref byte sourceRow = ref Unsafe.Add(ref sourceBase, (row + sourceOffset) * sourceStride); - ref ushort destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); - for (int column = 0; column < width; column++) - { - int sum = ConvolveScalar( - ref Unsafe.Add(ref sourceRow, column), - sourceStride, - ref coefficientBase, - tapCount); - - int result = RoundPowerOfTwo(sum << firstPassBits, CompoundRound1Bits) + roundOffset; - Unsafe.Add(ref destinationRow, column) = (ushort)result; - } - } - - return; - } - - GetEffectiveKernel(horizontalCoefficients, out int firstHorizontalCoefficient, out int horizontalTapCount); - GetEffectiveKernel(verticalCoefficients, out int firstVerticalCoefficient, out int verticalTapCount); - ref short horizontalCoefficientBase = ref Unsafe.Add( - ref MemoryMarshal.GetReference(horizontalCoefficients), - firstHorizontalCoefficient); - - ref short verticalCoefficientBase = ref Unsafe.Add( - ref MemoryMarshal.GetReference(verticalCoefficients), - firstVerticalCoefficient); - - int scratchStride = Math.Max(width, MinimumScratchStride); - int intermediateHeight = height + verticalTapCount - 1; - int horizontalSourceOffset = firstHorizontalCoefficient - 3; - int verticalSourceOffset = firstVerticalCoefficient - 3; - int horizontalBias = 1 << (8 + FilterBits - 1); - ref short scratchBase = ref MemoryMarshal.GetReference(scratch); - - // The Q7 horizontal pass keeps enough precision for the vertical pass while the positive bias makes every - // intermediate representable by signed 16-bit scratch. This is the same no-round compound shape as libaom. - for (int row = 0; row < intermediateHeight; row++) - { - ref byte sourceRow = ref Unsafe.Add( - ref sourceBase, - ((row + verticalSourceOffset) * sourceStride) + horizontalSourceOffset); - - ref short scratchRow = ref Unsafe.Add(ref scratchBase, row * scratchStride); - for (int column = 0; column < width; column++) - { - int sum = horizontalBias + ConvolveScalar( - ref Unsafe.Add(ref sourceRow, column), - 1, - ref horizontalCoefficientBase, - horizontalTapCount); - - Unsafe.Add(ref scratchRow, column) = (short)RoundPowerOfTwo(sum, Round0Bits); - } - } - - int verticalBias = 1 << offsetBits; - for (int row = 0; row < height; row++) - { - ref short scratchRow = ref Unsafe.Add(ref scratchBase, row * scratchStride); - ref ushort destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); - for (int column = 0; column < width; column++) - { - int sum = verticalBias + ConvolveScalar( - ref Unsafe.Add(ref scratchRow, column), - scratchStride, - ref verticalCoefficientBase, - verticalTapCount); - - Unsafe.Add(ref destinationRow, column) = (ushort)RoundPowerOfTwo(sum, CompoundRound1Bits); - } - } - } - - /// - /// Reconstructs one compound intermediate through the 128-bit convolution tier. - /// - private static void PredictCompoundVector128( - ReadOnlySpan source, - int sourceStride, - int sourceOrigin, - Span destination, - int destinationStride, - int width, - int height, - Av1InterpolationFilter horizontalFilter, - Av1InterpolationFilter verticalFilter, - int horizontalPhase, - int verticalPhase, - Span scratch) - { - ReadOnlySpan horizontalCoefficients = GetCompoundCoefficients(horizontalFilter, horizontalPhase, width <= 4); - ReadOnlySpan verticalCoefficients = GetCompoundCoefficients(verticalFilter, verticalPhase, height <= 4); - int roundBits = (2 * FilterBits) - Round0Bits - CompoundRound1Bits; - int offsetBits = 8 + (2 * FilterBits) - Round0Bits; - int roundOffset = (1 << (offsetBits - CompoundRound1Bits)) + - (1 << (offsetBits - CompoundRound1Bits - 1)); - - if (horizontalPhase == 0 && verticalPhase == 0) - { - CopyCompoundVector128( - source, - sourceStride, - sourceOrigin, - destination, - destinationStride, - width, - height, - roundBits, - roundOffset); - - return; - } - - if (verticalPhase == 0) - { - GetEffectiveKernel(horizontalCoefficients, out int firstCoefficient, out int tapCount); - FilterCompoundDirectVector128( - source, - sourceStride, - sourceOrigin, - destination, - destinationStride, - width, - height, - horizontalCoefficients[firstCoefficient..], - tapCount, - firstCoefficient - 3, - tapStride: 1, - preShift: 0, - round: Round0Bits, - roundOffset); - - return; - } - - if (horizontalPhase == 0) - { - GetEffectiveKernel(verticalCoefficients, out int firstCoefficient, out int tapCount); - FilterCompoundDirectVector128( - source, - sourceStride, - sourceOrigin, - destination, - destinationStride, - width, - height, - verticalCoefficients[firstCoefficient..], - tapCount, - (firstCoefficient - 3) * sourceStride, - sourceStride, - FilterBits - Round0Bits, - CompoundRound1Bits, - roundOffset); - - return; - } - - GetEffectiveKernel(horizontalCoefficients, out int firstHorizontalCoefficient, out int horizontalTapCount); - GetEffectiveKernel(verticalCoefficients, out int firstVerticalCoefficient, out int verticalTapCount); - FilterCompound2DVector128( - source, - sourceStride, - sourceOrigin, - destination, - destinationStride, - width, - height, - horizontalCoefficients[firstHorizontalCoefficient..], - horizontalTapCount, - firstHorizontalCoefficient - 3, - verticalCoefficients[firstVerticalCoefficient..], - verticalTapCount, - firstVerticalCoefficient - 3, - scratch); - } - - /// - /// Copies integer-position samples into biased compound intermediates in sixteen-sample groups. - /// - private static void CopyCompoundVector128( - ReadOnlySpan source, - int sourceStride, - int sourceOrigin, - Span destination, - int destinationStride, - int width, - int height, - int roundBits, - int roundOffset) - { - ref byte sourceBase = ref Unsafe.Add(ref MemoryMarshal.GetReference(source), sourceOrigin); - ref ushort destinationBase = ref MemoryMarshal.GetReference(destination); - Vector128 offset = Vector128.Create((ushort)roundOffset); - - for (int row = 0; row < height; row++) - { - ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride); - ref ushort destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); - int column = 0; - int vectorEnd = width - Vector128.Count; - for (; column <= vectorEnd; column += Vector128.Count) - { - Vector128 samples = Vector128.LoadUnsafe(ref sourceRow, (nuint)column); - ((Vector128.WidenLower(samples) << roundBits) + offset).StoreUnsafe( - ref destinationRow, - (nuint)column); - - ((Vector128.WidenUpper(samples) << roundBits) + offset).StoreUnsafe( - ref destinationRow, - (nuint)(column + Vector128.Count)); - } - - if (column == 0) - { - // Narrow AV1 blocks still use the SIMD load; the width-specific stores preserve the adjacent block. - Vector128 samples = Vector128.LoadUnsafe(ref sourceRow); - StoreCompoundVectors( - (Vector128.WidenLower(samples) << roundBits) + offset, - (Vector128.WidenUpper(samples) << roundBits) + offset, - ref destinationRow, - width); - - continue; - } - - for (; column < width; column++) - { - Unsafe.Add(ref destinationRow, column) = - (ushort)((Unsafe.Add(ref sourceRow, column) << roundBits) + roundOffset); - } - } - } - - /// - /// Applies one compound convolution direction in sixteen-sample groups. - /// - private static void FilterCompoundDirectVector128( - ReadOnlySpan source, - int sourceStride, - int sourceOrigin, - Span destination, - int destinationStride, - int width, - int height, - ReadOnlySpan coefficients, - int tapCount, - int sourceOffset, - int tapStride, - int preShift, - int round, - int roundOffset) - { - ref byte sourceBase = ref Unsafe.Add(ref MemoryMarshal.GetReference(source), sourceOrigin); - ref ushort destinationBase = ref MemoryMarshal.GetReference(destination); - ref short coefficientBase = ref MemoryMarshal.GetReference(coefficients); - Vector128 offset = Vector128.Create(roundOffset); - - for (int row = 0; row < height; row++) - { - ref byte sourceRow = ref Unsafe.Add(ref sourceBase, (row * sourceStride) + sourceOffset); - ref ushort destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); - int column = 0; - int vectorEnd = width - Vector128.Count; - for (; column <= vectorEnd; column += Vector128.Count) - { - Convolve( - ref sourceRow, - tapStride, - (nuint)column, - ref coefficientBase, - tapCount, - Vector128.Zero, - out Vector128 result0, - out Vector128 result1, - out Vector128 result2, - out Vector128 result3); - - PrepareCompoundResults( - ref result0, - ref result1, - ref result2, - ref result3, - preShift, - round, - offset); - - StoreCompoundVectors(result0, result1, result2, result3, ref destinationRow, column, Vector128.Count); - } - - if (column == 0) - { - Convolve( - ref sourceRow, - tapStride, - 0, - ref coefficientBase, - tapCount, - Vector128.Zero, - out Vector128 result0, - out Vector128 result1, - out Vector128 result2, - out Vector128 result3); - - PrepareCompoundResults( - ref result0, - ref result1, - ref result2, - ref result3, - preShift, - round, - offset); - - StoreCompoundVectors(result0, result1, result2, result3, ref destinationRow, 0, width); - continue; - } - - for (; column < width; column++) - { - int sum = ConvolveScalar( - ref Unsafe.Add(ref sourceRow, column), - tapStride, - ref coefficientBase, - tapCount); - - sum = RoundPowerOfTwo(sum << preShift, round) + roundOffset; - Unsafe.Add(ref destinationRow, column) = (ushort)sum; - } - } - } - - /// - /// Applies separable compound convolution through caller-owned signed scratch. - /// - private static void FilterCompound2DVector128( - ReadOnlySpan source, - int sourceStride, - int sourceOrigin, - Span destination, - int destinationStride, - int width, - int height, - ReadOnlySpan horizontalCoefficients, - int horizontalTapCount, - int horizontalSourceOffset, - ReadOnlySpan verticalCoefficients, - int verticalTapCount, - int verticalSourceOffset, - Span scratch) - { - ref byte sourceBase = ref Unsafe.Add(ref MemoryMarshal.GetReference(source), sourceOrigin); - ref ushort destinationBase = ref MemoryMarshal.GetReference(destination); - ref short scratchBase = ref MemoryMarshal.GetReference(scratch); - ref short horizontalCoefficientBase = ref MemoryMarshal.GetReference(horizontalCoefficients); - ref short verticalCoefficientBase = ref MemoryMarshal.GetReference(verticalCoefficients); - int scratchStride = Math.Max(width, MinimumScratchStride); - int intermediateHeight = height + verticalTapCount - 1; - Vector128 horizontalBias = Vector128.Create(1 << (8 + FilterBits - 1)); - - // The complete narrow-block vector is retained in scratch because the vertical pass consumes the same lanes. - // Wider blocks use one vector per sixteen output samples and finish any nonstandard tail scalarly. - for (int row = 0; row < intermediateHeight; row++) - { - ref byte sourceRow = ref Unsafe.Add( - ref sourceBase, - ((row + verticalSourceOffset) * sourceStride) + horizontalSourceOffset); - - ref short scratchRow = ref Unsafe.Add(ref scratchBase, row * scratchStride); - int column = 0; - int vectorEnd = width - Vector128.Count; - for (; column <= vectorEnd; column += Vector128.Count) - { - Convolve( - ref sourceRow, - 1, - (nuint)column, - ref horizontalCoefficientBase, - horizontalTapCount, - horizontalBias, - out Vector128 result0, - out Vector128 result1, - out Vector128 result2, - out Vector128 result3); - - Av1IntraPredictorBase.Narrow( - RoundPowerOfTwo(result0, Round0Bits), - RoundPowerOfTwo(result1, Round0Bits)).StoreUnsafe(ref scratchRow, (nuint)column); - - Av1IntraPredictorBase.Narrow( - RoundPowerOfTwo(result2, Round0Bits), - RoundPowerOfTwo(result3, Round0Bits)).StoreUnsafe( - ref scratchRow, - (nuint)(column + Vector128.Count)); - } - - if (column == 0) - { - Convolve( - ref sourceRow, - 1, - 0, - ref horizontalCoefficientBase, - horizontalTapCount, - horizontalBias, - out Vector128 result0, - out Vector128 result1, - out Vector128 result2, - out Vector128 result3); - - Av1IntraPredictorBase.Narrow( - RoundPowerOfTwo(result0, Round0Bits), - RoundPowerOfTwo(result1, Round0Bits)).StoreUnsafe(ref scratchRow); - - Av1IntraPredictorBase.Narrow( - RoundPowerOfTwo(result2, Round0Bits), - RoundPowerOfTwo(result3, Round0Bits)).StoreUnsafe( - ref scratchRow, - (nuint)Vector128.Count); - - continue; - } - - for (; column < width; column++) - { - int sum = (1 << (8 + FilterBits - 1)) + ConvolveScalar( - ref Unsafe.Add(ref sourceRow, column), - 1, - ref horizontalCoefficientBase, - horizontalTapCount); - - Unsafe.Add(ref scratchRow, column) = (short)RoundPowerOfTwo(sum, Round0Bits); - } - } - - Vector128 verticalBias = Vector128.Create(1 << (8 + (2 * FilterBits) - Round0Bits)); - for (int row = 0; row < height; row++) - { - ref short scratchRow = ref Unsafe.Add(ref scratchBase, row * scratchStride); - ref ushort destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); - int column = 0; - int vectorEnd = width - Vector128.Count; - for (; column <= vectorEnd; column += Vector128.Count) - { - Convolve( - ref scratchRow, - scratchStride, - (nuint)column, - ref verticalCoefficientBase, - verticalTapCount, - verticalBias, - out Vector128 result0, - out Vector128 result1); - - Convolve( - ref scratchRow, - scratchStride, - (nuint)(column + Vector128.Count), - ref verticalCoefficientBase, - verticalTapCount, - verticalBias, - out Vector128 result2, - out Vector128 result3); - - result0 = RoundPowerOfTwo(result0, CompoundRound1Bits); - result1 = RoundPowerOfTwo(result1, CompoundRound1Bits); - result2 = RoundPowerOfTwo(result2, CompoundRound1Bits); - result3 = RoundPowerOfTwo(result3, CompoundRound1Bits); - StoreCompoundVectors(result0, result1, result2, result3, ref destinationRow, column, Vector128.Count); - } - - if (column == 0) - { - Convolve( - ref scratchRow, - scratchStride, - 0, - ref verticalCoefficientBase, - verticalTapCount, - verticalBias, - out Vector128 result0, - out Vector128 result1); - - Convolve( - ref Unsafe.Add(ref scratchRow, Vector128.Count), - scratchStride, - 0, - ref verticalCoefficientBase, - verticalTapCount, - verticalBias, - out Vector128 result2, - out Vector128 result3); - - result0 = RoundPowerOfTwo(result0, CompoundRound1Bits); - result1 = RoundPowerOfTwo(result1, CompoundRound1Bits); - result2 = RoundPowerOfTwo(result2, CompoundRound1Bits); - result3 = RoundPowerOfTwo(result3, CompoundRound1Bits); - StoreCompoundVectors(result0, result1, result2, result3, ref destinationRow, 0, width); - continue; - } - - for (; column < width; column++) - { - int sum = (1 << (8 + (2 * FilterBits) - Round0Bits)) + ConvolveScalar( - ref Unsafe.Add(ref scratchRow, column), - scratchStride, - ref verticalCoefficientBase, - verticalTapCount); - - Unsafe.Add(ref destinationRow, column) = - (ushort)RoundPowerOfTwo(sum, CompoundRound1Bits); - } - } - } - - /// - /// Applies the compound direct-filter shifts and bias to sixteen convolution results. - /// - private static void PrepareCompoundResults( - ref Vector128 result0, - ref Vector128 result1, - ref Vector128 result2, - ref Vector128 result3, - int preShift, - int round, - Vector128 offset) - { - result0 = RoundPowerOfTwo(result0 << preShift, round) + offset; - result1 = RoundPowerOfTwo(result1 << preShift, round) + offset; - result2 = RoundPowerOfTwo(result2 << preShift, round) + offset; - result3 = RoundPowerOfTwo(result3 << preShift, round) + offset; - } - - /// - /// Packs and stores up to sixteen unsigned compound results. - /// - private static void StoreCompoundVectors( - Vector128 result0, - Vector128 result1, - Vector128 result2, - Vector128 result3, - ref ushort destination, - int destinationOffset, - int width) - { - Vector128 lower = Av1IntraPredictorBase.Narrow(result0, result1).AsUInt16(); - Vector128 upper = Av1IntraPredictorBase.Narrow(result2, result3).AsUInt16(); - ref ushort destinationStart = ref Unsafe.Add(ref destination, destinationOffset); - StoreCompoundVectors(lower, upper, ref destinationStart, width); - } - - /// - /// Stores up to sixteen packed compound results without crossing the logical block edge. - /// - private static void StoreCompoundVectors( - Vector128 lower, - Vector128 upper, - ref ushort destination, - int width) - { - int lowerWidth = Math.Min(width, Vector128.Count); - StorePartial(lower, ref destination, lowerWidth); - if (width > Vector128.Count) - { - StorePartial( - upper, - ref Unsafe.Add(ref destination, Vector128.Count), - width - Vector128.Count); - } - } - - /// - /// Gets the selected interpolation kernel for compound traversal. - /// - private static ReadOnlySpan GetCompoundCoefficients( - Av1InterpolationFilter filter, - int phase, - bool useReducedFilter) - => filter switch - { - Av1InterpolationFilter.Regular => RegularOperator.GetCoefficients(phase, useReducedFilter), - Av1InterpolationFilter.Smooth => SmoothOperator.GetCoefficients(phase, useReducedFilter), - Av1InterpolationFilter.Sharp => SharpOperator.GetCoefficients(phase, useReducedFilter), - _ => BilinearOperator.GetCoefficients(phase, useReducedFilter), - }; -} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Filters.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Filters.cs index eba9293c5..1e4c9e99c 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Filters.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Filters.cs @@ -11,7 +11,7 @@ internal static partial class Av1InterPredictor /// /// The number of stored coefficient positions in every decoder interpolation kernel. /// - private const int FilterCoefficientCount = 8; + internal const int FilterCoefficientCount = 8; /// /// Gets the regular eight-tap kernels for the sixteen subpixel phases. diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Operator.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Operator.cs index c5a8649da..c298b3b5e 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Operator.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Operator.cs @@ -1,6 +1,8 @@ // Copyright (c) Six Labors. // Licensed under the Six Labors Split License. +using System.Runtime.Intrinsics; + namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; /// @@ -753,7 +755,7 @@ internal static partial class Av1InterPredictor /// The selected Q7 phase kernel. /// Receives the first coefficient used by the effective kernel. /// Receives the effective two-, four-, six-, or eight-tap length. - private static void GetEffectiveKernel(ReadOnlySpan coefficients, out int firstCoefficient, out int tapCount) + internal static void GetEffectiveKernel(ReadOnlySpan coefficients, out int firstCoefficient, out int tapCount) { // This matches libaom's get_filter_tap decision. Reducing symmetric zero endpoints avoids source loads and // multiply-adds while retaining the original tap-to-source alignment through firstCoefficient. diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.ScaledOperator.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.ScaledOperator.cs deleted file mode 100644 index 593621499..000000000 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.ScaledOperator.cs +++ /dev/null @@ -1,105 +0,0 @@ -// Copyright (c) Six Labors. -// Licensed under the Six Labors Split License. - -using System.Runtime.CompilerServices; -using System.Runtime.Intrinsics; - -namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; - -/// -/// Defines sample-storage operators for reference-scaled prediction. -/// -internal static partial class Av1InterPredictor -{ - /// - /// Supplies sample loading, clipping, and storage for a scaled predictor pipeline. - /// - /// The native sample storage type. - private interface IScaledSampleOperator - where T : unmanaged - { - /// - /// Loads one source sample as a signed accumulator value. - /// - /// The first source sample. - /// The sample offset. - /// The widened sample value. - public static abstract int Load(ref T source, int index); - - /// - /// Clips and stores eight completed vector lanes. - /// - /// The first destination sample. - /// The output offset. - /// The first four completed lanes. - /// The second four completed lanes. - /// The decoded sample precision. - public static abstract void StoreVector( - ref T destination, - int index, - Vector128 result0, - Vector128 result1, - int bitDepth); - - /// - /// Clips and stores one completed scalar value. - /// - /// The first destination sample. - /// The output offset. - /// The completed sample value. - /// The decoded sample precision. - public static abstract void StoreScalar(ref T destination, int index, int value, int bitDepth); - } - - /// - /// Implements scaled prediction storage for 8-bit samples. - /// - private readonly struct ScaledByteOperator : IScaledSampleOperator - { - /// - [MethodImpl(MethodImplOptions.AggressiveInlining)] - public static int Load(ref byte source, int index) => Unsafe.Add(ref source, index); - - /// - [MethodImpl(MethodImplOptions.AggressiveInlining)] - public static void StoreVector( - ref byte destination, - int index, - Vector128 result0, - Vector128 result1, - int bitDepth) - => PackBytes(result0, result1, Vector128.Zero, Vector128.Zero) - .GetLower() - .StoreUnsafe(ref destination, (nuint)index); - - /// - [MethodImpl(MethodImplOptions.AggressiveInlining)] - public static void StoreScalar(ref byte destination, int index, int value, int bitDepth) - => Unsafe.Add(ref destination, index) = (byte)Math.Clamp(value, byte.MinValue, byte.MaxValue); - } - - /// - /// Implements scaled prediction storage for 8-, 10-, and 12-bit samples. - /// - private readonly struct ScaledUInt16Operator : IScaledSampleOperator - { - /// - [MethodImpl(MethodImplOptions.AggressiveInlining)] - public static int Load(ref ushort source, int index) => Unsafe.Add(ref source, index); - - /// - [MethodImpl(MethodImplOptions.AggressiveInlining)] - public static void StoreVector( - ref ushort destination, - int index, - Vector128 result0, - Vector128 result1, - int bitDepth) - => PackHighBitDepth(result0, result1, (1 << bitDepth) - 1).StoreUnsafe(ref destination, (nuint)index); - - /// - [MethodImpl(MethodImplOptions.AggressiveInlining)] - public static void StoreScalar(ref ushort destination, int index, int value, int bitDepth) - => Unsafe.Add(ref destination, index) = (ushort)Math.Clamp(value, 0, (1 << bitDepth) - 1); - } -} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Warped.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Warped.cs deleted file mode 100644 index fcf50b2bb..000000000 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Warped.cs +++ /dev/null @@ -1,730 +0,0 @@ -// Copyright (c) Six Labors. -// Licensed under the Six Labors Split License. - -using System.Runtime.CompilerServices; -using System.Runtime.InteropServices; -using System.Runtime.Intrinsics; -using SixLabors.ImageSharp.Formats.Heif.Av1.Motion; - -namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; - -/// -/// Reconstructs local and global affine warped-motion prediction blocks. -/// -internal static partial class Av1InterPredictor -{ - /// - /// The number of rows in one warped filter's horizontal intermediate tile. - /// - private const int WarpedIntermediateRows = 15; - - /// - /// The number of columns in one warped filter tile. - /// - private const int WarpedTileSize = 8; - - /// - /// The number of low model bits removed when addressing the warped filter table. - /// - private const int WarpedDifferencePrecisionBits = 10; - - /// - /// The number of fractional positions in one warped pixel. - /// - private const int WarpedPixelPrecisionShifts = 64; - - /// - /// Gets the number of signed 16-bit elements required by warped prediction. - /// - public const int WarpedScratchLength = WarpedIntermediateRows * WarpedTileSize; - - /// - /// Supplies the horizontal and vertical eight-tap dot products for one execution width. - /// - private interface IWarpedConvolution - { - /// - /// Convolves eight adjacent unsigned byte samples. - /// - /// The first source sample. - /// The first signed Q7 coefficient. - /// The sum of the eight sample-coefficient products. - public static abstract int Convolve(ref byte source, ref short coefficients); - - /// - /// Convolves eight adjacent unsigned high-bit-depth samples. - /// - /// The first source sample. - /// The first signed Q7 coefficient. - /// The sum of the eight sample-coefficient products. - public static abstract int Convolve(ref ushort source, ref short coefficients); - - /// - /// Convolves eight vertically strided unsigned intermediate samples. - /// - /// The first intermediate sample. - /// The distance between intermediate rows. - /// The first signed Q7 coefficient. - /// The sum of the eight sample-coefficient products. - public static abstract int ConvolveVertical(ref ushort source, int stride, ref short coefficients); - } - - /// - /// Reconstructs an 8-bit affine warped prediction using the widest supported convolution operator. - /// - public static void PredictWarped( - ReadOnlySpan source, - int sourceStride, - Point sourceOrigin, - int sourceWidth, - int sourceHeight, - Span destination, - int destinationStride, - Point destinationPosition, - int width, - int height, - int subsamplingX, - int subsamplingY, - Av1GlobalMotionParameters parameters, - Span scratch) - { - if (Vector128.IsHardwareAccelerated) - { - PredictWarped( - source, - sourceStride, - sourceOrigin, - sourceWidth, - sourceHeight, - destination, - destinationStride, - destinationPosition, - width, - height, - subsamplingX, - subsamplingY, - parameters, - scratch); - - return; - } - - PredictWarped( - source, - sourceStride, - sourceOrigin, - sourceWidth, - sourceHeight, - destination, - destinationStride, - destinationPosition, - width, - height, - subsamplingX, - subsamplingY, - parameters, - scratch); - } - - /// - /// Reconstructs an 8-bit affine warped reference into AV1's unsigned compound intermediate format. - /// - public static void PredictWarpedCompound( - ReadOnlySpan source, - int sourceStride, - Point sourceOrigin, - int sourceWidth, - int sourceHeight, - Span destination, - int destinationStride, - Point destinationPosition, - int width, - int height, - int subsamplingX, - int subsamplingY, - Av1GlobalMotionParameters parameters, - Span scratch) - { - if (Vector128.IsHardwareAccelerated) - { - PredictWarpedCompound( - source, - sourceStride, - sourceOrigin, - sourceWidth, - sourceHeight, - destination, - destinationStride, - destinationPosition, - width, - height, - subsamplingX, - subsamplingY, - parameters, - scratch); - - return; - } - - PredictWarpedCompound( - source, - sourceStride, - sourceOrigin, - sourceWidth, - sourceHeight, - destination, - destinationStride, - destinationPosition, - width, - height, - subsamplingX, - subsamplingY, - parameters, - scratch); - } - - /// - /// Reconstructs a high-bit-depth affine warped prediction using the widest supported convolution operator. - /// - public static void PredictWarped( - ReadOnlySpan source, - int sourceStride, - Point sourceOrigin, - int sourceWidth, - int sourceHeight, - Span destination, - int destinationStride, - Point destinationPosition, - int width, - int height, - int subsamplingX, - int subsamplingY, - int bitDepth, - Av1GlobalMotionParameters parameters, - Span scratch) - { - if (Vector128.IsHardwareAccelerated) - { - PredictWarped( - source, - sourceStride, - sourceOrigin, - sourceWidth, - sourceHeight, - destination, - destinationStride, - destinationPosition, - width, - height, - subsamplingX, - subsamplingY, - bitDepth, - parameters, - scratch); - - return; - } - - PredictWarped( - source, - sourceStride, - sourceOrigin, - sourceWidth, - sourceHeight, - destination, - destinationStride, - destinationPosition, - width, - height, - subsamplingX, - subsamplingY, - bitDepth, - parameters, - scratch); - } - - /// - /// Reconstructs an 8-bit affine warped prediction without explicit hardware intrinsics. - /// - public static void PredictWarpedScalar( - ReadOnlySpan source, - int sourceStride, - Point sourceOrigin, - int sourceWidth, - int sourceHeight, - Span destination, - int destinationStride, - Point destinationPosition, - int width, - int height, - int subsamplingX, - int subsamplingY, - Av1GlobalMotionParameters parameters, - Span scratch) - => PredictWarped( - source, - sourceStride, - sourceOrigin, - sourceWidth, - sourceHeight, - destination, - destinationStride, - destinationPosition, - width, - height, - subsamplingX, - subsamplingY, - parameters, - scratch); - - /// - /// Reconstructs an 8-bit affine warped reference into compound intermediates without explicit hardware intrinsics. - /// - public static void PredictWarpedCompoundScalar( - ReadOnlySpan source, - int sourceStride, - Point sourceOrigin, - int sourceWidth, - int sourceHeight, - Span destination, - int destinationStride, - Point destinationPosition, - int width, - int height, - int subsamplingX, - int subsamplingY, - Av1GlobalMotionParameters parameters, - Span scratch) - => PredictWarpedCompound( - source, - sourceStride, - sourceOrigin, - sourceWidth, - sourceHeight, - destination, - destinationStride, - destinationPosition, - width, - height, - subsamplingX, - subsamplingY, - parameters, - scratch); - - /// - /// Reconstructs a high-bit-depth affine warped prediction without explicit hardware intrinsics. - /// - public static void PredictWarpedScalar( - ReadOnlySpan source, - int sourceStride, - Point sourceOrigin, - int sourceWidth, - int sourceHeight, - Span destination, - int destinationStride, - Point destinationPosition, - int width, - int height, - int subsamplingX, - int subsamplingY, - int bitDepth, - Av1GlobalMotionParameters parameters, - Span scratch) - => PredictWarped( - source, - sourceStride, - sourceOrigin, - sourceWidth, - sourceHeight, - destination, - destinationStride, - destinationPosition, - width, - height, - subsamplingX, - subsamplingY, - bitDepth, - parameters, - scratch); - - /// - /// Reconstructs one 8-bit warped block through a closed convolution operator. - /// - private static void PredictWarped( - ReadOnlySpan source, - int sourceStride, - Point sourceOrigin, - int sourceWidth, - int sourceHeight, - Span destination, - int destinationStride, - Point destinationPosition, - int width, - int height, - int subsamplingX, - int subsamplingY, - Av1GlobalMotionParameters parameters, - Span scratch) - where TConvolution : struct, IWarpedConvolution - { - ref byte sourceBase = ref MemoryMarshal.GetReference(source); - ref byte destinationBase = ref MemoryMarshal.GetReference(destination); - Span intermediate = MemoryMarshal.Cast(scratch)[..WarpedScratchLength]; - int horizontalBias = 1 << (8 + FilterBits - 1); - int verticalBias = 1 << (8 + (2 * FilterBits) - Round0Bits); - int verticalRound = (2 * FilterBits) - Round0Bits; - - for (int tileRow = destinationPosition.Y; tileRow < destinationPosition.Y + height; tileRow += WarpedTileSize) - { - for (int tileColumn = destinationPosition.X; tileColumn < destinationPosition.X + width; tileColumn += WarpedTileSize) - { - DeriveWarpedTilePosition( - parameters, - tileColumn, - tileRow, - subsamplingX, - subsamplingY, - out int integerX, - out int integerY, - out int phaseX, - out int phaseY); - - for (int row = -7; row < 8; row++) - { - int sourceY = Math.Clamp(integerY + row, 0, sourceHeight - 1); - int phase = phaseX + (parameters.Beta * (row + 4)); - for (int column = -4; column < 4; column++) - { - int sourceX = integerX + column - 3; - int sourceIndex = ((sourceOrigin.Y + sourceY) * sourceStride) + sourceOrigin.X + sourceX; - ref short coefficients = ref GetWarpedFilterReference(phase); - int sum = horizontalBias + TConvolution.Convolve(ref Unsafe.Add(ref sourceBase, sourceIndex), ref coefficients); - intermediate[((row + 7) * WarpedTileSize) + column + 4] = (ushort)RoundPowerOfTwoScalar(sum, Round0Bits); - phase += parameters.Alpha; - } - } - - int tileHeight = Math.Min(WarpedTileSize, destinationPosition.Y + height - tileRow); - int tileWidth = Math.Min(WarpedTileSize, destinationPosition.X + width - tileColumn); - for (int row = 0; row < tileHeight; row++) - { - int phase = phaseY + (parameters.Delta * row); - int destinationRowOffset = (tileRow - destinationPosition.Y + row) * destinationStride; - for (int column = 0; column < tileWidth; column++) - { - ref ushort intermediateSource = ref intermediate[(row * WarpedTileSize) + column]; - ref short coefficients = ref GetWarpedFilterReference(phase); - int sum = verticalBias + TConvolution.ConvolveVertical(ref intermediateSource, WarpedTileSize, ref coefficients); - int value = RoundPowerOfTwoScalar(sum, verticalRound) - (1 << 7) - (1 << 8); - Unsafe.Add(ref destinationBase, destinationRowOffset + tileColumn - destinationPosition.X + column) = - (byte)Math.Clamp(value, byte.MinValue, byte.MaxValue); - - phase += parameters.Gamma; - } - } - } - } - } - - /// - /// Reconstructs one 8-bit warped reference without discarding the compound convolution precision. - /// - private static void PredictWarpedCompound( - ReadOnlySpan source, - int sourceStride, - Point sourceOrigin, - int sourceWidth, - int sourceHeight, - Span destination, - int destinationStride, - Point destinationPosition, - int width, - int height, - int subsamplingX, - int subsamplingY, - Av1GlobalMotionParameters parameters, - Span scratch) - where TConvolution : struct, IWarpedConvolution - { - ref byte sourceBase = ref MemoryMarshal.GetReference(source); - ref ushort destinationBase = ref MemoryMarshal.GetReference(destination); - Span intermediate = MemoryMarshal.Cast(scratch)[..WarpedScratchLength]; - int horizontalBias = 1 << (8 + FilterBits - 1); - int verticalBias = 1 << (8 + (2 * FilterBits) - Round0Bits); - - for (int tileRow = destinationPosition.Y; tileRow < destinationPosition.Y + height; tileRow += WarpedTileSize) - { - for (int tileColumn = destinationPosition.X; tileColumn < destinationPosition.X + width; tileColumn += WarpedTileSize) - { - DeriveWarpedTilePosition( - parameters, - tileColumn, - tileRow, - subsamplingX, - subsamplingY, - out int integerX, - out int integerY, - out int phaseX, - out int phaseY); - - for (int row = -7; row < 8; row++) - { - int sourceY = Math.Clamp(integerY + row, 0, sourceHeight - 1); - int phase = phaseX + (parameters.Beta * (row + 4)); - for (int column = -4; column < 4; column++) - { - int sourceX = integerX + column - 3; - int sourceIndex = ((sourceOrigin.Y + sourceY) * sourceStride) + sourceOrigin.X + sourceX; - ref short coefficients = ref GetWarpedFilterReference(phase); - int sum = horizontalBias + TConvolution.Convolve(ref Unsafe.Add(ref sourceBase, sourceIndex), ref coefficients); - intermediate[((row + 7) * WarpedTileSize) + column + 4] = - (ushort)RoundPowerOfTwoScalar(sum, Round0Bits); - - phase += parameters.Alpha; - } - } - - int tileHeight = Math.Min(WarpedTileSize, destinationPosition.Y + height - tileRow); - int tileWidth = Math.Min(WarpedTileSize, destinationPosition.X + width - tileColumn); - for (int row = 0; row < tileHeight; row++) - { - int phase = phaseY + (parameters.Delta * row); - int destinationRowOffset = (tileRow - destinationPosition.Y + row) * destinationStride; - for (int column = 0; column < tileWidth; column++) - { - ref ushort intermediateSource = ref intermediate[(row * WarpedTileSize) + column]; - ref short coefficients = ref GetWarpedFilterReference(phase); - int sum = verticalBias + TConvolution.ConvolveVertical(ref intermediateSource, WarpedTileSize, ref coefficients); - Unsafe.Add(ref destinationBase, destinationRowOffset + tileColumn - destinationPosition.X + column) = - (ushort)RoundPowerOfTwoScalar(sum, CompoundRound1Bits); - - phase += parameters.Gamma; - } - } - } - } - } - - /// - /// Reconstructs one high-bit-depth warped block through a closed convolution operator. - /// - private static void PredictWarped( - ReadOnlySpan source, - int sourceStride, - Point sourceOrigin, - int sourceWidth, - int sourceHeight, - Span destination, - int destinationStride, - Point destinationPosition, - int width, - int height, - int subsamplingX, - int subsamplingY, - int bitDepth, - Av1GlobalMotionParameters parameters, - Span scratch) - where TConvolution : struct, IWarpedConvolution - { - ref ushort sourceBase = ref MemoryMarshal.GetReference(source); - ref ushort destinationBase = ref MemoryMarshal.GetReference(destination); - Span intermediate = MemoryMarshal.Cast(scratch)[..WarpedScratchLength]; - - // Twelve-bit prediction increases round0 by two so the biased horizontal intermediate remains representable - // in sixteen bits. Reducing round1 by the same amount preserves the complete normative Q14 shift. - int intermediateRange = bitDepth + FilterBits - Round0Bits + 2; - int round0 = Round0Bits + Math.Max(intermediateRange - 16, 0); - int verticalRound = (2 * FilterBits) - round0; - int horizontalBias = 1 << (bitDepth + FilterBits - 1); - int verticalBias = 1 << (bitDepth + (2 * FilterBits) - round0); - int maximum = (1 << bitDepth) - 1; - - for (int tileRow = destinationPosition.Y; tileRow < destinationPosition.Y + height; tileRow += WarpedTileSize) - { - for (int tileColumn = destinationPosition.X; tileColumn < destinationPosition.X + width; tileColumn += WarpedTileSize) - { - DeriveWarpedTilePosition( - parameters, - tileColumn, - tileRow, - subsamplingX, - subsamplingY, - out int integerX, - out int integerY, - out int phaseX, - out int phaseY); - - for (int row = -7; row < 8; row++) - { - int sourceY = Math.Clamp(integerY + row, 0, sourceHeight - 1); - int phase = phaseX + (parameters.Beta * (row + 4)); - for (int column = -4; column < 4; column++) - { - int sourceX = integerX + column - 3; - int sourceIndex = ((sourceOrigin.Y + sourceY) * sourceStride) + sourceOrigin.X + sourceX; - ref short coefficients = ref GetWarpedFilterReference(phase); - int sum = horizontalBias + TConvolution.Convolve(ref Unsafe.Add(ref sourceBase, sourceIndex), ref coefficients); - intermediate[((row + 7) * WarpedTileSize) + column + 4] = (ushort)RoundPowerOfTwoScalar(sum, round0); - phase += parameters.Alpha; - } - } - - int tileHeight = Math.Min(WarpedTileSize, destinationPosition.Y + height - tileRow); - int tileWidth = Math.Min(WarpedTileSize, destinationPosition.X + width - tileColumn); - for (int row = 0; row < tileHeight; row++) - { - int phase = phaseY + (parameters.Delta * row); - int destinationRowOffset = (tileRow - destinationPosition.Y + row) * destinationStride; - for (int column = 0; column < tileWidth; column++) - { - ref ushort intermediateSource = ref intermediate[(row * WarpedTileSize) + column]; - ref short coefficients = ref GetWarpedFilterReference(phase); - int sum = verticalBias + TConvolution.ConvolveVertical(ref intermediateSource, WarpedTileSize, ref coefficients); - int value = RoundPowerOfTwoScalar(sum, verticalRound) - (1 << (bitDepth - 1)) - (1 << bitDepth); - Unsafe.Add(ref destinationBase, destinationRowOffset + tileColumn - destinationPosition.X + column) = - (ushort)Math.Clamp(value, 0, maximum); - - phase += parameters.Gamma; - } - } - } - } - } - - /// - /// Projects the center of one 8x8 output tile and derives its integer source position and reduced phases. - /// - private static void DeriveWarpedTilePosition( - Av1GlobalMotionParameters parameters, - int tileColumn, - int tileRow, - int subsamplingX, - int subsamplingY, - out int integerX, - out int integerY, - out int phaseX, - out int phaseY) - { - int sourceX = (tileColumn + 4) << subsamplingX; - int sourceY = (tileRow + 4) << subsamplingY; - long projectedX = ((long)parameters[2] * sourceX) + ((long)parameters[3] * sourceY) + parameters[0]; - long projectedY = ((long)parameters[4] * sourceX) + ((long)parameters[5] * sourceY) + parameters[1]; - long planeX = projectedX >> subsamplingX; - long planeY = projectedY >> subsamplingY; - integerX = (int)(planeX >> Av1GlobalMotionParameters.ModelPrecisionBits); - integerY = (int)(planeY >> Av1GlobalMotionParameters.ModelPrecisionBits); - phaseX = (int)planeX & (Av1GlobalMotionParameters.ModelScale - 1); - phaseY = (int)planeY & (Av1GlobalMotionParameters.ModelScale - 1); - phaseX += (-4 * parameters.Alpha) + (-4 * parameters.Beta); - phaseY += (-4 * parameters.Gamma) + (-4 * parameters.Delta); - - // Shear parameters are quantized to 64-model-unit steps. Clearing the same low bits after the tile-center - // projection keeps negative and positive phases on the exact filter-table grid used by the bitstream model. - phaseX &= -1 << 6; - phaseY &= -1 << 6; - } - - /// - /// Gets a reference to the first coefficient for one reduced warped-filter phase. - /// - private static ref short GetWarpedFilterReference(int phase) - { - int filterIndex = ((phase + (1 << (WarpedDifferencePrecisionBits - 1))) >> WarpedDifferencePrecisionBits) + - WarpedPixelPrecisionShifts; - - return ref Unsafe.Add(ref MemoryMarshal.GetReference(WarpedFilter), filterIndex * FilterCoefficientCount); - } - - /// - /// Divides a nonnegative value by a power of two with nearest-integer rounding. - /// - private static int RoundPowerOfTwoScalar(int value, int bitCount) - => (value + (1 << (bitCount - 1))) >> bitCount; - - /// - /// Executes warped dot products with portable 128-bit SIMD. - /// - private readonly struct WarpedVector128Convolution : IWarpedConvolution - { - /// - public static int Convolve(ref byte source, ref short coefficients) - { - Vector128 packed = Vector128.LoadUnsafe(ref source); - (Vector128 samples, _) = Vector128.Widen(packed); - return MultiplyAndSum(samples, Vector128.LoadUnsafe(ref coefficients)); - } - - /// - public static int Convolve(ref ushort source, ref short coefficients) - => MultiplyAndSum(Vector128.LoadUnsafe(ref source), Vector128.LoadUnsafe(ref coefficients)); - - /// - public static int ConvolveVertical(ref ushort source, int stride, ref short coefficients) - { - Vector128 samples = Vector128.Create( - source, - Unsafe.Add(ref source, stride), - Unsafe.Add(ref source, stride * 2), - Unsafe.Add(ref source, stride * 3), - Unsafe.Add(ref source, stride * 4), - Unsafe.Add(ref source, stride * 5), - Unsafe.Add(ref source, stride * 6), - Unsafe.Add(ref source, stride * 7)); - - return MultiplyAndSum(samples, Vector128.LoadUnsafe(ref coefficients)); - } - - /// - /// Widens unsigned samples and signed coefficients before accumulating their exact 32-bit products. - /// - private static int MultiplyAndSum(Vector128 samples, Vector128 coefficients) - { - (Vector128 sampleLower, Vector128 sampleUpper) = Vector128.Widen(samples); - (Vector128 coefficientLower, Vector128 coefficientUpper) = Vector128.Widen(coefficients); - return Vector128.Sum(sampleLower.AsInt32() * coefficientLower) + - Vector128.Sum(sampleUpper.AsInt32() * coefficientUpper); - } - } - - /// - /// Executes warped dot products without explicit hardware intrinsics. - /// - private readonly struct WarpedScalarConvolution : IWarpedConvolution - { - /// - public static int Convolve(ref byte source, ref short coefficients) - { - int sum = 0; - for (nuint index = 0; index < FilterCoefficientCount; index++) - { - sum += Unsafe.Add(ref source, index) * Unsafe.Add(ref coefficients, index); - } - - return sum; - } - - /// - public static int Convolve(ref ushort source, ref short coefficients) - { - int sum = 0; - for (nuint index = 0; index < FilterCoefficientCount; index++) - { - sum += Unsafe.Add(ref source, index) * Unsafe.Add(ref coefficients, index); - } - - return sum; - } - - /// - public static int ConvolveVertical(ref ushort source, int stride, ref short coefficients) - { - int sum = 0; - for (int index = 0; index < FilterCoefficientCount; index++) - { - sum += Unsafe.Add(ref source, index * stride) * Unsafe.Add(ref coefficients, index); - } - - return sum; - } - } -} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.cs index a5af87360..3a58297e3 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.cs @@ -25,12 +25,12 @@ internal static partial class Av1InterPredictor /// /// The number of fractional bits in each interpolation coefficient. /// - private const int FilterBits = 7; + internal const int FilterBits = 7; /// /// The normal first-round shift used by libaom single-reference convolution. /// - private const int Round0Bits = 3; + internal const int Round0Bits = 3; /// /// The maximum number of source rows added by an eight-tap vertical filter. @@ -40,7 +40,7 @@ internal static partial class Av1InterPredictor /// /// The minimum scratch stride that lets a 128-bit byte kernel handle four- and eight-sample blocks. /// - private const int MinimumScratchStride = 16; + internal const int MinimumScratchStride = 16; /// /// Gets the maximum number of signed 16-bit elements required for one two-dimensional prediction block. diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1ScaledInterPredictor.Operator.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1ScaledInterPredictor.Operator.cs new file mode 100644 index 000000000..9215ae9bd --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1ScaledInterPredictor.Operator.cs @@ -0,0 +1,364 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.Intrinsics; + +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; + +/// +/// Defines variable-phase reference-scaled prediction arithmetic. +/// +internal static partial class Av1ScaledInterPredictor +{ + /// + /// Defines variable-phase scaled prediction for one native sample storage type. + /// + private interface IAv1ScaledPredictionOperator + { + /// + /// Loads one native source sample as a signed accumulator value. + /// + /// The native sample storage type. + /// The first native source sample. + /// The source sample offset. + /// The widened sample. + public static abstract int Load(ref T source, int index) + where T : unmanaged; + + /// + /// Accumulates one sample-coefficient product. + /// + /// The current convolution sum. + /// The source sample. + /// The signed Q7 coefficient. + /// The updated convolution sum. + public static abstract int MultiplyAdd(int accumulator, int sample, int coefficient); + + /// + /// Accumulates four independent sample-coefficient products. + /// + /// The current convolution sums. + /// The source samples. + /// The signed Q7 coefficients. + /// The updated convolution sums. + public static abstract Vector128 MultiplyAdd(Vector128 accumulator, Vector128 samples, Vector128 coefficients); + + /// + /// Accumulates eight independent sample-coefficient products. + /// + /// The current convolution sums. + /// The source samples. + /// The signed Q7 coefficients. + /// The updated convolution sums. + public static abstract Vector256 MultiplyAdd(Vector256 accumulator, Vector256 samples, Vector256 coefficients); + + /// + /// Accumulates sixteen independent sample-coefficient products. + /// + /// The current convolution sums. + /// The source samples. + /// The signed Q7 coefficients. + /// The updated convolution sums. + public static abstract Vector512 MultiplyAdd(Vector512 accumulator, Vector512 samples, Vector512 coefficients); + + /// + /// Convolves one intermediate sample column without hardware intrinsics. + /// + /// The first intermediate sample. + /// The distance between intermediate rows. + /// The first signed Q7 coefficient. + /// The number of active coefficients. + /// The exact convolution sum. + public static abstract int Convolve(ref short source, int sourceStride, ref short coefficients, int coefficientCount); + + /// + /// Convolves eight adjacent intermediate samples through a 128-bit lane group. + /// + /// The first intermediate sample. + /// The distance between intermediate rows. + /// The first column offset. + /// The first signed Q7 coefficient. + /// The number of active coefficients. + /// The initial convolution bias. + /// Receives the first four completed sums. + /// Receives the next four completed sums. + public static abstract void Convolve( + ref short source, + int sourceStride, + nuint sourceOffset, + ref short coefficients, + int coefficientCount, + Vector128 initial, + out Vector128 result0, + out Vector128 result1); + + /// + /// Convolves sixteen adjacent intermediate samples through a 256-bit lane group. + /// + /// The first intermediate sample. + /// The distance between intermediate rows. + /// The first column offset. + /// The first signed Q7 coefficient. + /// The number of active coefficients. + /// The initial convolution bias. + /// Receives the first eight completed sums. + /// Receives the next eight completed sums. + public static abstract void Convolve( + ref short source, + int sourceStride, + nuint sourceOffset, + ref short coefficients, + int coefficientCount, + Vector256 initial, + out Vector256 result0, + out Vector256 result1); + + /// + /// Convolves thirty-two adjacent intermediate samples through a 512-bit lane group. + /// + /// The first intermediate sample. + /// The distance between intermediate rows. + /// The first column offset. + /// The first signed Q7 coefficient. + /// The number of active coefficients. + /// The initial convolution bias. + /// Receives the first sixteen completed sums. + /// Receives the next sixteen completed sums. + public static abstract void Convolve( + ref short source, + int sourceStride, + nuint sourceOffset, + ref short coefficients, + int coefficientCount, + Vector512 initial, + out Vector512 result0, + out Vector512 result1); + + /// + /// Clips and stores one completed prediction. + /// + /// The native sample storage type. + /// The first destination sample. + /// The destination offset. + /// The completed prediction. + /// The decoded sample precision. + public static abstract void Store(ref T destination, int index, int value, int bitDepth) + where T : unmanaged; + + /// + /// Clips and stores eight completed predictions. + /// + /// The native sample storage type. + /// The first destination sample. + /// The destination offset. + /// The first four completed predictions. + /// The next four completed predictions. + /// The decoded sample precision. + public static abstract void Store(ref T destination, int index, Vector128 result0, Vector128 result1, int bitDepth) + where T : unmanaged; + + /// + /// Clips and stores sixteen completed predictions. + /// + /// The native sample storage type. + /// The first destination sample. + /// The destination offset. + /// The first eight completed predictions. + /// The next eight completed predictions. + /// The decoded sample precision. + public static abstract void Store(ref T destination, int index, Vector256 result0, Vector256 result1, int bitDepth) + where T : unmanaged; + + /// + /// Clips and stores thirty-two completed predictions. + /// + /// The native sample storage type. + /// The first destination sample. + /// The destination offset. + /// The first sixteen completed predictions. + /// The next sixteen completed predictions. + /// The decoded sample precision. + public static abstract void Store(ref T destination, int index, Vector512 result0, Vector512 result1, int bitDepth) + where T : unmanaged; + } + + /// + /// Implements variable-phase scaled prediction for scalar and SIMD lane groups. + /// + private readonly struct ScaledOperator : IAv1ScaledPredictionOperator + { + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int Load(ref T source, int index) + where T : unmanaged + { + // The only closed forms are byte and ushort. The JIT removes this storage choice from each specialization, + // leaving the shared variable-phase traversal free of duplicate 8-bit and high-bit-depth implementations. + if (typeof(T) == typeof(byte)) + { + return Unsafe.Add(ref Unsafe.As(ref source), index); + } + + return Unsafe.Add(ref Unsafe.As(ref source), index); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int MultiplyAdd(int accumulator, int sample, int coefficient) + => accumulator + (sample * coefficient); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector128 MultiplyAdd(Vector128 accumulator, Vector128 samples, Vector128 coefficients) + => accumulator + (samples * coefficients); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector256 MultiplyAdd(Vector256 accumulator, Vector256 samples, Vector256 coefficients) + => accumulator + (samples * coefficients); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector512 MultiplyAdd(Vector512 accumulator, Vector512 samples, Vector512 coefficients) + => accumulator + (samples * coefficients); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int Convolve(ref short source, int sourceStride, ref short coefficients, int coefficientCount) + => ConvolveScalar(ref source, sourceStride, ref coefficients, coefficientCount); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Convolve( + ref short source, + int sourceStride, + nuint sourceOffset, + ref short coefficients, + int coefficientCount, + Vector128 initial, + out Vector128 result0, + out Vector128 result1) + => Av1InterPredictor.Convolve( + ref source, + sourceStride, + sourceOffset, + ref coefficients, + coefficientCount, + initial, + out result0, + out result1); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Convolve( + ref short source, + int sourceStride, + nuint sourceOffset, + ref short coefficients, + int coefficientCount, + Vector256 initial, + out Vector256 result0, + out Vector256 result1) + => Av1InterPredictor.Convolve( + ref source, + sourceStride, + sourceOffset, + ref coefficients, + coefficientCount, + initial, + out result0, + out result1); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Convolve( + ref short source, + int sourceStride, + nuint sourceOffset, + ref short coefficients, + int coefficientCount, + Vector512 initial, + out Vector512 result0, + out Vector512 result1) + => Av1InterPredictor.Convolve( + ref source, + sourceStride, + sourceOffset, + ref coefficients, + coefficientCount, + initial, + out result0, + out result1); + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Store(ref T destination, int index, int value, int bitDepth) + where T : unmanaged + { + if (typeof(T) == typeof(byte)) + { + Unsafe.Add(ref Unsafe.As(ref destination), index) = (byte)Math.Clamp(value, byte.MinValue, byte.MaxValue); + return; + } + + Unsafe.Add(ref Unsafe.As(ref destination), index) = (ushort)Math.Clamp(value, 0, (1 << bitDepth) - 1); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Store(ref T destination, int index, Vector128 result0, Vector128 result1, int bitDepth) + where T : unmanaged + { + if (typeof(T) == typeof(byte)) + { + PackBytes(result0, result1, Vector128.Zero, Vector128.Zero) + .GetLower() + .StoreUnsafe(ref Unsafe.As(ref destination), (nuint)index); + + return; + } + + PackHighBitDepth(result0, result1, (1 << bitDepth) - 1) + .StoreUnsafe(ref Unsafe.As(ref destination), (nuint)index); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Store(ref T destination, int index, Vector256 result0, Vector256 result1, int bitDepth) + where T : unmanaged + { + if (typeof(T) == typeof(byte)) + { + PackBytes(result0, result1, Vector256.Zero, Vector256.Zero) + .GetLower() + .StoreUnsafe(ref Unsafe.As(ref destination), (nuint)index); + + return; + } + + PackHighBitDepth(result0, result1, (1 << bitDepth) - 1) + .StoreUnsafe(ref Unsafe.As(ref destination), (nuint)index); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Store(ref T destination, int index, Vector512 result0, Vector512 result1, int bitDepth) + where T : unmanaged + { + if (typeof(T) == typeof(byte)) + { + PackBytes(result0, result1, Vector512.Zero, Vector512.Zero) + .GetLower() + .StoreUnsafe(ref Unsafe.As(ref destination), (nuint)index); + + return; + } + + PackHighBitDepth(result0, result1, (1 << bitDepth) - 1) + .StoreUnsafe(ref Unsafe.As(ref destination), (nuint)index); + } + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Scaled.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1ScaledInterPredictor.cs similarity index 53% rename from src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Scaled.cs rename to src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1ScaledInterPredictor.cs index 4fefd2a0a..0c6142f04 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.Scaled.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1ScaledInterPredictor.cs @@ -5,12 +5,14 @@ using System.Runtime.CompilerServices; using System.Runtime.InteropServices; using System.Runtime.Intrinsics; +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor; + namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; /// /// Reconstructs reference-scaled inter prediction through variable-phase separable convolution. /// -internal static partial class Av1InterPredictor +internal static partial class Av1ScaledInterPredictor { /// /// Gets the scratch capacity required by one scaled prediction block. @@ -45,7 +47,7 @@ internal static partial class Av1InterPredictor int verticalPhase, int verticalStep, Span scratch) - => DispatchScaled( + => DispatchScaled( source, sourceStride, sourceOrigin, @@ -81,7 +83,7 @@ internal static partial class Av1InterPredictor int verticalStep, int bitDepth, Span scratch) - => DispatchScaled( + => DispatchScaled( source, sourceStride, sourceOrigin, @@ -118,7 +120,7 @@ internal static partial class Av1InterPredictor int bitDepth, Span scratch) where T : unmanaged - where TSample : struct, IScaledSampleOperator + where TSample : struct, IAv1ScaledPredictionOperator { switch (horizontalFilter) { @@ -216,7 +218,7 @@ internal static partial class Av1InterPredictor int bitDepth, Span scratch) where T : unmanaged - where TSample : struct, IScaledSampleOperator + where TSample : struct, IAv1ScaledPredictionOperator where THorizontal : struct, IAv1InterPredictorOperator { switch (verticalFilter) @@ -310,7 +312,7 @@ internal static partial class Av1InterPredictor int bitDepth, Span scratch) where T : unmanaged - where TSample : struct, IScaledSampleOperator + where TSample : struct, IAv1ScaledPredictionOperator where THorizontal : struct, IAv1InterPredictorOperator where TVertical : struct, IAv1InterPredictorOperator { @@ -332,45 +334,63 @@ internal static partial class Av1InterPredictor ref T sourceRow = ref Unsafe.Add(ref sourceBase, (row - 3) * sourceStride); ref short scratchRow = ref Unsafe.Add(ref scratchBase, row * scratchStride); int column = 0; + + if (Vector512.IsHardwareAccelerated) + { + int oneVectorFromEnd = width - Vector512.Count; + for (; column <= oneVectorFromEnd; column += Vector512.Count) + { + Vector512 result = FilterScaledHorizontalVector512( + ref sourceRow, + horizontalPhase, + horizontalStep, + column, + useReducedHorizontalFilter, + horizontalBias, + round0); + + Av1IntraPredictorBase.Narrow(result, Vector512.Zero) + .GetLower() + .StoreUnsafe(ref scratchRow, (nuint)column); + } + } + + if (Vector256.IsHardwareAccelerated) + { + int oneVectorFromEnd = width - Vector256.Count; + for (; column <= oneVectorFromEnd; column += Vector256.Count) + { + Vector256 result = FilterScaledHorizontalVector256( + ref sourceRow, + horizontalPhase, + horizontalStep, + column, + useReducedHorizontalFilter, + horizontalBias, + round0); + + Av1IntraPredictorBase.Narrow(result, Vector256.Zero) + .GetLower() + .StoreUnsafe(ref scratchRow, (nuint)column); + } + } + if (Vector128.IsHardwareAccelerated) { for (; column <= width - Vector128.Count; column += Vector128.Count) { - int position0 = horizontalPhase + (column * horizontalStep); - int position1 = position0 + horizontalStep; - int position2 = position1 + horizontalStep; - int position3 = position2 + horizontalStep; - int source0 = (position0 >> Av1ReferenceScale.SubpixelBits) - 3; - int source1 = (position1 >> Av1ReferenceScale.SubpixelBits) - 3; - int source2 = (position2 >> Av1ReferenceScale.SubpixelBits) - 3; - int source3 = (position3 >> Av1ReferenceScale.SubpixelBits) - 3; - ReadOnlySpan coefficients0 = THorizontal.GetCoefficients((position0 & Av1ReferenceScale.SubpixelMask) >> 6, useReducedHorizontalFilter); - ReadOnlySpan coefficients1 = THorizontal.GetCoefficients((position1 & Av1ReferenceScale.SubpixelMask) >> 6, useReducedHorizontalFilter); - ReadOnlySpan coefficients2 = THorizontal.GetCoefficients((position2 & Av1ReferenceScale.SubpixelMask) >> 6, useReducedHorizontalFilter); - ReadOnlySpan coefficients3 = THorizontal.GetCoefficients((position3 & Av1ReferenceScale.SubpixelMask) >> 6, useReducedHorizontalFilter); - Vector128 result = Vector128.Create(horizontalBias); - for (int tap = 0; tap < FilterCoefficientCount; tap++) - { - Vector128 samples = Vector128.Create( - TSample.Load(ref sourceRow, source0 + tap), - TSample.Load(ref sourceRow, source1 + tap), - TSample.Load(ref sourceRow, source2 + tap), - TSample.Load(ref sourceRow, source3 + tap)); - - Vector128 coefficients = Vector128.Create( - (int)coefficients0[tap], - coefficients1[tap], - coefficients2[tap], - coefficients3[tap]); - - result += samples * coefficients; - } - - Vector64 intermediate = Av1IntraPredictorBase.Narrow( - RoundPowerOfTwo(result, round0), - Vector128.Zero).GetLower(); - - intermediate.StoreUnsafe(ref scratchRow, (nuint)column); + Vector128 result = FilterScaledHorizontalVector128( + ref sourceRow, + horizontalPhase, + horizontalStep, + column, + useReducedHorizontalFilter, + horizontalBias, + round0); + + Av1IntraPredictorBase.Narrow(result, Vector128.Zero) + .GetLower() + .StoreUnsafe(ref scratchRow, (nuint)column); } } @@ -385,7 +405,7 @@ internal static partial class Av1InterPredictor int sum = horizontalBias; for (int tap = 0; tap < FilterCoefficientCount; tap++) { - sum += coefficients[tap] * TSample.Load(ref sourceRow, sourceColumn + tap); + sum = TSample.MultiplyAdd(sum, TSample.Load(ref sourceRow, sourceColumn + tap), coefficients[tap]); } Unsafe.Add(ref scratchRow, column) = (short)RoundPowerOfTwo(sum, round0); @@ -408,13 +428,61 @@ internal static partial class Av1InterPredictor ref short coefficientBase = ref MemoryMarshal.GetReference(coefficients); ref T destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); int column = 0; + + if (Vector512.IsHardwareAccelerated) + { + Vector512 initial = Vector512.Create(verticalBias); + Vector512 offset = Vector512.Create(roundOffset); + int oneVectorFromEnd = width - (Vector512.Count * 2); + for (; column <= oneVectorFromEnd; column += Vector512.Count * 2) + { + TSample.Convolve( + ref scratchRow, + scratchStride, + (nuint)column, + ref coefficientBase, + FilterCoefficientCount, + initial, + out Vector512 result0, + out Vector512 result1); + + result0 = RoundPowerOfTwo(result0, round1) - offset; + result1 = RoundPowerOfTwo(result1, round1) - offset; + TSample.Store(ref destinationRow, column, result0, result1, bitDepth); + } + } + + if (Vector256.IsHardwareAccelerated) + { + Vector256 initial = Vector256.Create(verticalBias); + Vector256 offset = Vector256.Create(roundOffset); + int oneVectorFromEnd = width - (Vector256.Count * 2); + for (; column <= oneVectorFromEnd; column += Vector256.Count * 2) + { + TSample.Convolve( + ref scratchRow, + scratchStride, + (nuint)column, + ref coefficientBase, + FilterCoefficientCount, + initial, + out Vector256 result0, + out Vector256 result1); + + result0 = RoundPowerOfTwo(result0, round1) - offset; + result1 = RoundPowerOfTwo(result1, round1) - offset; + TSample.Store(ref destinationRow, column, result0, result1, bitDepth); + } + } + if (Vector128.IsHardwareAccelerated) { Vector128 initial = Vector128.Create(verticalBias); Vector128 offset = Vector128.Create(roundOffset); - for (; column <= width - Vector128.Count; column += Vector128.Count) + int oneVectorFromEnd = width - (Vector128.Count * 2); + for (; column <= oneVectorFromEnd; column += Vector128.Count * 2) { - Convolve( + TSample.Convolve( ref scratchRow, scratchStride, (nuint)column, @@ -426,19 +494,19 @@ internal static partial class Av1InterPredictor result0 = RoundPowerOfTwo(result0, round1) - offset; result1 = RoundPowerOfTwo(result1, round1) - offset; - TSample.StoreVector(ref destinationRow, column, result0, result1, bitDepth); + TSample.Store(ref destinationRow, column, result0, result1, bitDepth); } } for (; column < width; column++) { - int sum = verticalBias + ConvolveScalar( + int sum = verticalBias + TSample.Convolve( ref Unsafe.Add(ref scratchRow, column), scratchStride, ref coefficientBase, FilterCoefficientCount); - TSample.StoreScalar( + TSample.Store( ref destinationRow, column, RoundPowerOfTwo(sum, round1) - roundOffset, @@ -446,4 +514,163 @@ internal static partial class Av1InterPredictor } } } + + /// + /// Filters four independently positioned horizontal samples through a closed scaled-prediction operator. + /// + private static Vector128 FilterScaledHorizontalVector128( + ref T source, + int phase, + int step, + int column, + bool useReducedFilter, + int bias, + int round) + where T : unmanaged + where TOperator : struct, IAv1ScaledPredictionOperator + where TFilter : struct, IAv1InterPredictorOperator + { + Vector128 result = Vector128.Create(bias); + for (int tap = 0; tap < FilterCoefficientCount; tap++) + { + result = TOperator.MultiplyAdd( + result, + LoadScaledSamplesVector128(ref source, phase, step, column, tap), + LoadScaledCoefficientsVector128(phase, step, column, tap, useReducedFilter)); + } + + return RoundPowerOfTwo(result, round); + } + + /// + /// Filters eight independently positioned horizontal samples through a closed scaled-prediction operator. + /// + private static Vector256 FilterScaledHorizontalVector256( + ref T source, + int phase, + int step, + int column, + bool useReducedFilter, + int bias, + int round) + where T : unmanaged + where TOperator : struct, IAv1ScaledPredictionOperator + where TFilter : struct, IAv1InterPredictorOperator + { + Vector256 result = Vector256.Create(bias); + for (int tap = 0; tap < FilterCoefficientCount; tap++) + { + Vector256 samples = Vector256.Create( + LoadScaledSamplesVector128(ref source, phase, step, column, tap), + LoadScaledSamplesVector128(ref source, phase, step, column + Vector128.Count, tap)); + + Vector256 coefficients = Vector256.Create( + LoadScaledCoefficientsVector128(phase, step, column, tap, useReducedFilter), + LoadScaledCoefficientsVector128(phase, step, column + Vector128.Count, tap, useReducedFilter)); + + result = TOperator.MultiplyAdd(result, samples, coefficients); + } + + return RoundPowerOfTwo(result, round); + } + + /// + /// Filters sixteen independently positioned horizontal samples through a closed scaled-prediction operator. + /// + private static Vector512 FilterScaledHorizontalVector512( + ref T source, + int phase, + int step, + int column, + bool useReducedFilter, + int bias, + int round) + where T : unmanaged + where TOperator : struct, IAv1ScaledPredictionOperator + where TFilter : struct, IAv1InterPredictorOperator + { + Vector512 result = Vector512.Create(bias); + for (int tap = 0; tap < FilterCoefficientCount; tap++) + { + Vector256 sampleLower = Vector256.Create( + LoadScaledSamplesVector128(ref source, phase, step, column, tap), + LoadScaledSamplesVector128(ref source, phase, step, column + Vector128.Count, tap)); + + Vector256 sampleUpper = Vector256.Create( + LoadScaledSamplesVector128(ref source, phase, step, column + Vector256.Count, tap), + LoadScaledSamplesVector128(ref source, phase, step, column + Vector256.Count + Vector128.Count, tap)); + + Vector256 coefficientLower = Vector256.Create( + LoadScaledCoefficientsVector128(phase, step, column, tap, useReducedFilter), + LoadScaledCoefficientsVector128(phase, step, column + Vector128.Count, tap, useReducedFilter)); + + Vector256 coefficientUpper = Vector256.Create( + LoadScaledCoefficientsVector128(phase, step, column + Vector256.Count, tap, useReducedFilter), + LoadScaledCoefficientsVector128(phase, step, column + Vector256.Count + Vector128.Count, tap, useReducedFilter)); + + result = TOperator.MultiplyAdd( + result, + Vector512.Create(sampleLower, sampleUpper), + Vector512.Create(coefficientLower, coefficientUpper)); + } + + return RoundPowerOfTwo(result, round); + } + + /// + /// Gathers four variable-position source samples for one horizontal filter tap. + /// + private static Vector128 LoadScaledSamplesVector128( + ref T source, + int phase, + int step, + int column, + int tap) + where T : unmanaged + where TOperator : struct, IAv1ScaledPredictionOperator + => Vector128.Create( + LoadScaledSample(ref source, phase, step, column, tap), + LoadScaledSample(ref source, phase, step, column + 1, tap), + LoadScaledSample(ref source, phase, step, column + 2, tap), + LoadScaledSample(ref source, phase, step, column + 3, tap)); + + /// + /// Gathers four variable-phase coefficients for one horizontal filter tap. + /// + private static Vector128 LoadScaledCoefficientsVector128( + int phase, + int step, + int column, + int tap, + bool useReducedFilter) + where TFilter : struct, IAv1InterPredictorOperator + => Vector128.Create( + LoadScaledCoefficient(phase, step, column, tap, useReducedFilter), + LoadScaledCoefficient(phase, step, column + 1, tap, useReducedFilter), + LoadScaledCoefficient(phase, step, column + 2, tap, useReducedFilter), + LoadScaledCoefficient(phase, step, column + 3, tap, useReducedFilter)); + + /// + /// Loads one variable-position source sample for a horizontal filter tap. + /// + private static int LoadScaledSample(ref T source, int phase, int step, int column, int tap) + where T : unmanaged + where TOperator : struct, IAv1ScaledPredictionOperator + { + int position = phase + (column * step); + int sourceColumn = (position >> Av1ReferenceScale.SubpixelBits) - 3; + return TOperator.Load(ref source, sourceColumn + tap); + } + + /// + /// Loads one variable-phase horizontal filter coefficient. + /// + private static int LoadScaledCoefficient(int phase, int step, int column, int tap, bool useReducedFilter) + where TFilter : struct, IAv1InterPredictorOperator + { + int position = phase + (column * step); + return TFilter.GetCoefficients( + (position & Av1ReferenceScale.SubpixelMask) >> 6, + useReducedFilter)[tap]; + } } diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.WarpedFilters.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1WarpedInterPredictor.Filters.cs similarity index 98% rename from src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.WarpedFilters.cs rename to src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1WarpedInterPredictor.Filters.cs index 5db4b2dbd..da0a5cc45 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1InterPredictor.WarpedFilters.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1WarpedInterPredictor.Filters.cs @@ -1,12 +1,14 @@ // Copyright (c) Six Labors. // Licensed under the Six Labors Split License. +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor; + namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; /// /// Provides the normative Q7 filter kernels for affine warped-motion prediction. /// -internal static partial class Av1InterPredictor +internal static partial class Av1WarpedInterPredictor { /// /// Gets the 193 consecutive eight-tap warped-filter phases spanning fractional positions [-1, 2]. diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1WarpedInterPredictor.Operator.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1WarpedInterPredictor.Operator.cs new file mode 100644 index 000000000..4a882d843 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1WarpedInterPredictor.Operator.cs @@ -0,0 +1,141 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.Intrinsics; + +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; + +/// +/// Defines affine warped-motion prediction arithmetic. +/// +internal static partial class Av1WarpedInterPredictor +{ + /// + /// Defines the affine warped-motion dot product for scalar and SIMD lane groups. + /// + /// + /// Each SIMD overload contains consecutive independent eight-tap filters. The generic warped traversal + /// gathers source windows and coefficient phases, while the closed operator owns the exact multiply-and-sum arithmetic. + /// + private interface IAv1WarpedPredictionOperator + { + /// + /// Convolves one eight-sample window without hardware intrinsics. + /// + /// The first source sample. + /// The distance between source samples. + /// The first signed Q7 coefficient. + /// The exact dot product. + public static abstract int Convolve(ref byte source, int sourceStride, ref short coefficients); + + /// + /// Convolves one high-bit-depth eight-sample window without hardware intrinsics. + /// + /// The first source sample. + /// The distance between source samples. + /// The first signed Q7 coefficient. + /// The exact dot product. + public static abstract int Convolve(ref ushort source, int sourceStride, ref short coefficients); + + /// + /// Convolves one packed eight-sample window. + /// + /// The unsigned samples. + /// The signed Q7 coefficients. + /// The exact dot product. + public static abstract int Convolve(Vector128 samples, Vector128 coefficients); + + /// + /// Convolves two packed eight-sample windows. + /// + /// The two unsigned sample windows. + /// The two signed Q7 coefficient windows. + /// The two exact dot products in the low lanes. + public static abstract Vector128 Convolve(Vector256 samples, Vector256 coefficients); + + /// + /// Convolves four packed eight-sample windows. + /// + /// The four unsigned sample windows. + /// The four signed Q7 coefficient windows. + /// The four exact dot products. + public static abstract Vector128 Convolve(Vector512 samples, Vector512 coefficients); + } + + /// + /// Implements the affine warped-motion dot product for scalar and SIMD lane groups. + /// + private readonly struct WarpedOperator : IAv1WarpedPredictionOperator + { + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int Convolve(ref byte source, int sourceStride, ref short coefficients) + { + int sum = 0; + for (int index = 0; index < FilterCoefficientCount; index++) + { + sum += Unsafe.Add(ref source, index * sourceStride) * Unsafe.Add(ref coefficients, index); + } + + return sum; + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int Convolve(ref ushort source, int sourceStride, ref short coefficients) + { + int sum = 0; + for (int index = 0; index < FilterCoefficientCount; index++) + { + sum += Unsafe.Add(ref source, index * sourceStride) * Unsafe.Add(ref coefficients, index); + } + + return sum; + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int Convolve(Vector128 samples, Vector128 coefficients) + { + (Vector128 sampleLower, Vector128 sampleUpper) = Vector128.Widen(samples); + (Vector128 coefficientLower, Vector128 coefficientUpper) = Vector128.Widen(coefficients); + return Vector128.Sum(sampleLower.AsInt32() * coefficientLower) + + Vector128.Sum(sampleUpper.AsInt32() * coefficientUpper); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector128 Convolve(Vector256 samples, Vector256 coefficients) + { + // Widen preserves the two eight-tap windows as separate 256-bit results. Reducing each product vector + // therefore produces the two independent predictions without horizontal lane shuffles. + (Vector256 sample0, Vector256 sample1) = Vector256.Widen(samples); + (Vector256 coefficient0, Vector256 coefficient1) = Vector256.Widen(coefficients); + return Vector128.Create( + Vector256.Sum(sample0.AsInt32() * coefficient0), + Vector256.Sum(sample1.AsInt32() * coefficient1), + 0, + 0); + } + + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Vector128 Convolve(Vector512 samples, Vector512 coefficients) + { + // The four eight-tap windows occupy four consecutive 256-bit quarters after widening. Multiplication + // remains 512-bit; quarter reductions recover the four independent scalar dot products in output order. + (Vector512 sampleLower, Vector512 sampleUpper) = Vector512.Widen(samples); + (Vector512 coefficientLower, Vector512 coefficientUpper) = Vector512.Widen(coefficients); + Vector512 productLower = sampleLower.AsInt32() * coefficientLower; + Vector512 productUpper = sampleUpper.AsInt32() * coefficientUpper; + return Vector128.Create( + Vector256.Sum(productLower.GetLower()), + Vector256.Sum(productLower.GetUpper()), + Vector256.Sum(productUpper.GetLower()), + Vector256.Sum(productUpper.GetUpper())); + } + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1WarpedInterPredictor.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1WarpedInterPredictor.cs new file mode 100644 index 000000000..7848f8b2c --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Inter/Av1WarpedInterPredictor.cs @@ -0,0 +1,1055 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using System.Runtime.Intrinsics; +using SixLabors.ImageSharp.Formats.Heif.Av1.Motion; + +using static SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter.Av1InterPredictor; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.Inter; + +/// +/// Reconstructs local and global affine warped-motion prediction blocks. +/// +internal static partial class Av1WarpedInterPredictor +{ + /// + /// The number of rows in one warped filter's horizontal intermediate tile. + /// + private const int WarpedIntermediateRows = 15; + + /// + /// The number of columns in one warped filter tile. + /// + private const int WarpedTileSize = 8; + + /// + /// The number of low model bits removed when addressing the warped filter table. + /// + private const int WarpedDifferencePrecisionBits = 10; + + /// + /// The number of fractional positions in one warped pixel. + /// + private const int WarpedPixelPrecisionShifts = 64; + + /// + /// Gets the number of signed 16-bit elements required by warped prediction. + /// + public const int WarpedScratchLength = WarpedIntermediateRows * WarpedTileSize; + + /// + /// Reconstructs an 8-bit affine warped prediction using the widest supported convolution operator. + /// + public static void PredictWarped( + ReadOnlySpan source, + int sourceStride, + Point sourceOrigin, + int sourceWidth, + int sourceHeight, + Span destination, + int destinationStride, + Point destinationPosition, + int width, + int height, + int subsamplingX, + int subsamplingY, + Av1GlobalMotionParameters parameters, + Span scratch) + => PredictWarped( + source, + sourceStride, + sourceOrigin, + sourceWidth, + sourceHeight, + destination, + destinationStride, + destinationPosition, + width, + height, + subsamplingX, + subsamplingY, + parameters, + scratch, + useHardwareIntrinsics: true); + + /// + /// Reconstructs an 8-bit affine warped reference into AV1's unsigned compound intermediate format. + /// + public static void PredictWarpedCompound( + ReadOnlySpan source, + int sourceStride, + Point sourceOrigin, + int sourceWidth, + int sourceHeight, + Span destination, + int destinationStride, + Point destinationPosition, + int width, + int height, + int subsamplingX, + int subsamplingY, + Av1GlobalMotionParameters parameters, + Span scratch) + => PredictWarpedCompound( + source, + sourceStride, + sourceOrigin, + sourceWidth, + sourceHeight, + destination, + destinationStride, + destinationPosition, + width, + height, + subsamplingX, + subsamplingY, + parameters, + scratch, + useHardwareIntrinsics: true); + + /// + /// Reconstructs a high-bit-depth affine warped prediction using the widest supported convolution operator. + /// + public static void PredictWarped( + ReadOnlySpan source, + int sourceStride, + Point sourceOrigin, + int sourceWidth, + int sourceHeight, + Span destination, + int destinationStride, + Point destinationPosition, + int width, + int height, + int subsamplingX, + int subsamplingY, + int bitDepth, + Av1GlobalMotionParameters parameters, + Span scratch) + => PredictWarped( + source, + sourceStride, + sourceOrigin, + sourceWidth, + sourceHeight, + destination, + destinationStride, + destinationPosition, + width, + height, + subsamplingX, + subsamplingY, + bitDepth, + parameters, + scratch, + useHardwareIntrinsics: true); + + /// + /// Reconstructs an 8-bit affine warped prediction without explicit hardware intrinsics. + /// + public static void PredictWarpedScalar( + ReadOnlySpan source, + int sourceStride, + Point sourceOrigin, + int sourceWidth, + int sourceHeight, + Span destination, + int destinationStride, + Point destinationPosition, + int width, + int height, + int subsamplingX, + int subsamplingY, + Av1GlobalMotionParameters parameters, + Span scratch) + => PredictWarped( + source, + sourceStride, + sourceOrigin, + sourceWidth, + sourceHeight, + destination, + destinationStride, + destinationPosition, + width, + height, + subsamplingX, + subsamplingY, + parameters, + scratch, + useHardwareIntrinsics: false); + + /// + /// Reconstructs an 8-bit affine warped reference into compound intermediates without explicit hardware intrinsics. + /// + public static void PredictWarpedCompoundScalar( + ReadOnlySpan source, + int sourceStride, + Point sourceOrigin, + int sourceWidth, + int sourceHeight, + Span destination, + int destinationStride, + Point destinationPosition, + int width, + int height, + int subsamplingX, + int subsamplingY, + Av1GlobalMotionParameters parameters, + Span scratch) + => PredictWarpedCompound( + source, + sourceStride, + sourceOrigin, + sourceWidth, + sourceHeight, + destination, + destinationStride, + destinationPosition, + width, + height, + subsamplingX, + subsamplingY, + parameters, + scratch, + useHardwareIntrinsics: false); + + /// + /// Reconstructs a high-bit-depth affine warped prediction without explicit hardware intrinsics. + /// + public static void PredictWarpedScalar( + ReadOnlySpan source, + int sourceStride, + Point sourceOrigin, + int sourceWidth, + int sourceHeight, + Span destination, + int destinationStride, + Point destinationPosition, + int width, + int height, + int subsamplingX, + int subsamplingY, + int bitDepth, + Av1GlobalMotionParameters parameters, + Span scratch) + => PredictWarped( + source, + sourceStride, + sourceOrigin, + sourceWidth, + sourceHeight, + destination, + destinationStride, + destinationPosition, + width, + height, + subsamplingX, + subsamplingY, + bitDepth, + parameters, + scratch, + useHardwareIntrinsics: false); + + /// + /// Reconstructs one 8-bit warped block through a closed convolution operator. + /// + private static void PredictWarped( + ReadOnlySpan source, + int sourceStride, + Point sourceOrigin, + int sourceWidth, + int sourceHeight, + Span destination, + int destinationStride, + Point destinationPosition, + int width, + int height, + int subsamplingX, + int subsamplingY, + Av1GlobalMotionParameters parameters, + Span scratch, + bool useHardwareIntrinsics) + where TOperator : struct, IAv1WarpedPredictionOperator + { + ref byte sourceBase = ref MemoryMarshal.GetReference(source); + ref byte destinationBase = ref MemoryMarshal.GetReference(destination); + Span intermediate = MemoryMarshal.Cast(scratch)[..WarpedScratchLength]; + int horizontalBias = 1 << (8 + FilterBits - 1); + int verticalBias = 1 << (8 + (2 * FilterBits) - Round0Bits); + int verticalRound = (2 * FilterBits) - Round0Bits; + + for (int tileRow = destinationPosition.Y; tileRow < destinationPosition.Y + height; tileRow += WarpedTileSize) + { + for (int tileColumn = destinationPosition.X; tileColumn < destinationPosition.X + width; tileColumn += WarpedTileSize) + { + DeriveWarpedTilePosition( + parameters, + tileColumn, + tileRow, + subsamplingX, + subsamplingY, + out int integerX, + out int integerY, + out int phaseX, + out int phaseY); + + FilterWarpedHorizontal( + ref sourceBase, + sourceStride, + sourceOrigin, + sourceHeight, + integerX, + integerY, + phaseX, + parameters, + intermediate, + horizontalBias, + Round0Bits, + useHardwareIntrinsics); + + int tileHeight = Math.Min(WarpedTileSize, destinationPosition.Y + height - tileRow); + int tileWidth = Math.Min(WarpedTileSize, destinationPosition.X + width - tileColumn); + for (int row = 0; row < tileHeight; row++) + { + int phase = phaseY + (parameters.Delta * row); + int destinationRowOffset = (tileRow - destinationPosition.Y + row) * destinationStride; + ref ushort intermediateSource = ref intermediate[row * WarpedTileSize]; + ref byte destinationRow = ref Unsafe.Add( + ref destinationBase, + destinationRowOffset + tileColumn - destinationPosition.X); + + FilterWarpedVertical( + ref intermediateSource, + ref destinationRow, + tileWidth, + phase, + parameters.Gamma, + verticalBias, + verticalRound, + useHardwareIntrinsics); + } + } + } + } + + /// + /// Produces the unsigned horizontal intermediate tile for an 8-bit source plane. + /// + private static void FilterWarpedHorizontal( + ref byte sourceBase, + int sourceStride, + Point sourceOrigin, + int sourceHeight, + int integerX, + int integerY, + int phaseX, + Av1GlobalMotionParameters parameters, + Span intermediate, + int bias, + int round, + bool useHardwareIntrinsics) + where TOperator : struct, IAv1WarpedPredictionOperator + { + for (int row = -7; row < 8; row++) + { + int sourceY = Math.Clamp(integerY + row, 0, sourceHeight - 1); + int sourceIndex = ((sourceOrigin.Y + sourceY) * sourceStride) + sourceOrigin.X + integerX - 7; + ref byte sourceRow = ref Unsafe.Add(ref sourceBase, sourceIndex); + ref ushort intermediateRow = ref intermediate[(row + 7) * WarpedTileSize]; + int phase = phaseX + (parameters.Beta * (row + 4)); + int column = 0; + + // Eight neighboring windows use different warped phases. Packing complete eight-tap windows into + // descending SIMD widths preserves those independent coefficients while leaving no per-row buffers. + if (useHardwareIntrinsics && Vector512.IsHardwareAccelerated) + { + int oneVectorFromEnd = WarpedTileSize - 4; + for (; column <= oneVectorFromEnd; column += 4) + { + Vector128 sums = ConvolveWarpedVector512( + ref Unsafe.Add(ref sourceRow, column), + phase + (column * parameters.Alpha), + parameters.Alpha); + + for (int lane = 0; lane < 4; lane++) + { + Unsafe.Add(ref intermediateRow, column + lane) = + (ushort)RoundPowerOfTwoScalar(bias + sums.GetElement(lane), round); + } + } + } + + if (useHardwareIntrinsics && Vector256.IsHardwareAccelerated) + { + int oneVectorFromEnd = WarpedTileSize - 2; + for (; column <= oneVectorFromEnd; column += 2) + { + Vector128 sums = ConvolveWarpedVector256( + ref Unsafe.Add(ref sourceRow, column), + phase + (column * parameters.Alpha), + parameters.Alpha); + + Unsafe.Add(ref intermediateRow, column) = (ushort)RoundPowerOfTwoScalar(bias + sums.GetElement(0), round); + Unsafe.Add(ref intermediateRow, column + 1) = (ushort)RoundPowerOfTwoScalar(bias + sums.GetElement(1), round); + } + } + + if (useHardwareIntrinsics && Vector128.IsHardwareAccelerated) + { + for (; column < WarpedTileSize; column++) + { + int sum = ConvolveWarpedVector128( + ref Unsafe.Add(ref sourceRow, column), + phase + (column * parameters.Alpha)); + + Unsafe.Add(ref intermediateRow, column) = (ushort)RoundPowerOfTwoScalar(bias + sum, round); + } + } + + for (; column < WarpedTileSize; column++) + { + ref short coefficients = ref GetWarpedFilterReference(phase + (column * parameters.Alpha)); + int sum = TOperator.Convolve(ref Unsafe.Add(ref sourceRow, column), 1, ref coefficients); + Unsafe.Add(ref intermediateRow, column) = (ushort)RoundPowerOfTwoScalar(bias + sum, round); + } + } + } + + /// + /// Produces the unsigned horizontal intermediate tile for a high-bit-depth source plane. + /// + private static void FilterWarpedHorizontal( + ref ushort sourceBase, + int sourceStride, + Point sourceOrigin, + int sourceHeight, + int integerX, + int integerY, + int phaseX, + Av1GlobalMotionParameters parameters, + Span intermediate, + int bias, + int round, + bool useHardwareIntrinsics) + where TOperator : struct, IAv1WarpedPredictionOperator + { + for (int row = -7; row < 8; row++) + { + int sourceY = Math.Clamp(integerY + row, 0, sourceHeight - 1); + int sourceIndex = ((sourceOrigin.Y + sourceY) * sourceStride) + sourceOrigin.X + integerX - 7; + ref ushort sourceRow = ref Unsafe.Add(ref sourceBase, sourceIndex); + ref ushort intermediateRow = ref intermediate[(row + 7) * WarpedTileSize]; + int phase = phaseX + (parameters.Beta * (row + 4)); + int column = 0; + + if (useHardwareIntrinsics && Vector512.IsHardwareAccelerated) + { + int oneVectorFromEnd = WarpedTileSize - 4; + for (; column <= oneVectorFromEnd; column += 4) + { + Vector128 sums = ConvolveWarpedVector512( + ref Unsafe.Add(ref sourceRow, column), + 1, + phase + (column * parameters.Alpha), + parameters.Alpha); + + for (int lane = 0; lane < 4; lane++) + { + Unsafe.Add(ref intermediateRow, column + lane) = + (ushort)RoundPowerOfTwoScalar(bias + sums.GetElement(lane), round); + } + } + } + + if (useHardwareIntrinsics && Vector256.IsHardwareAccelerated) + { + int oneVectorFromEnd = WarpedTileSize - 2; + for (; column <= oneVectorFromEnd; column += 2) + { + Vector128 sums = ConvolveWarpedVector256( + ref Unsafe.Add(ref sourceRow, column), + 1, + phase + (column * parameters.Alpha), + parameters.Alpha); + + Unsafe.Add(ref intermediateRow, column) = (ushort)RoundPowerOfTwoScalar(bias + sums.GetElement(0), round); + Unsafe.Add(ref intermediateRow, column + 1) = (ushort)RoundPowerOfTwoScalar(bias + sums.GetElement(1), round); + } + } + + if (useHardwareIntrinsics && Vector128.IsHardwareAccelerated) + { + for (; column < WarpedTileSize; column++) + { + int sum = ConvolveWarpedVector128( + ref Unsafe.Add(ref sourceRow, column), + 1, + phase + (column * parameters.Alpha)); + + Unsafe.Add(ref intermediateRow, column) = (ushort)RoundPowerOfTwoScalar(bias + sum, round); + } + } + + for (; column < WarpedTileSize; column++) + { + ref short coefficients = ref GetWarpedFilterReference(phase + (column * parameters.Alpha)); + int sum = TOperator.Convolve(ref Unsafe.Add(ref sourceRow, column), 1, ref coefficients); + Unsafe.Add(ref intermediateRow, column) = (ushort)RoundPowerOfTwoScalar(bias + sum, round); + } + } + } + + /// + /// Completes one 8-bit vertical warped-filter row. + /// + private static void FilterWarpedVertical( + ref ushort source, + ref byte destination, + int width, + int phase, + int phaseStep, + int bias, + int round, + bool useHardwareIntrinsics) + where TOperator : struct, IAv1WarpedPredictionOperator + { + int column = 0; + if (useHardwareIntrinsics && Vector512.IsHardwareAccelerated) + { + int oneVectorFromEnd = width - 4; + for (; column <= oneVectorFromEnd; column += 4) + { + Vector128 sums = ConvolveWarpedVector512( + ref Unsafe.Add(ref source, column), + WarpedTileSize, + phase + (column * phaseStep), + phaseStep); + + for (int lane = 0; lane < 4; lane++) + { + Unsafe.Add(ref destination, column + lane) = FinishWarpedByte(sums.GetElement(lane), bias, round); + } + } + } + + if (useHardwareIntrinsics && Vector256.IsHardwareAccelerated) + { + int oneVectorFromEnd = width - 2; + for (; column <= oneVectorFromEnd; column += 2) + { + Vector128 sums = ConvolveWarpedVector256( + ref Unsafe.Add(ref source, column), + WarpedTileSize, + phase + (column * phaseStep), + phaseStep); + + Unsafe.Add(ref destination, column) = FinishWarpedByte(sums.GetElement(0), bias, round); + Unsafe.Add(ref destination, column + 1) = FinishWarpedByte(sums.GetElement(1), bias, round); + } + } + + if (useHardwareIntrinsics && Vector128.IsHardwareAccelerated) + { + for (; column < width; column++) + { + int sum = ConvolveWarpedVector128( + ref Unsafe.Add(ref source, column), + WarpedTileSize, + phase + (column * phaseStep)); + + Unsafe.Add(ref destination, column) = FinishWarpedByte(sum, bias, round); + } + } + + for (; column < width; column++) + { + ref short coefficients = ref GetWarpedFilterReference(phase + (column * phaseStep)); + int sum = TOperator.Convolve(ref Unsafe.Add(ref source, column), WarpedTileSize, ref coefficients); + Unsafe.Add(ref destination, column) = FinishWarpedByte(sum, bias, round); + } + } + + /// + /// Completes one compound-intermediate vertical warped-filter row. + /// + private static void FilterWarpedCompoundVertical( + ref ushort source, + ref ushort destination, + int width, + int phase, + int phaseStep, + int bias, + bool useHardwareIntrinsics) + where TOperator : struct, IAv1WarpedPredictionOperator + { + int column = 0; + if (useHardwareIntrinsics && Vector512.IsHardwareAccelerated) + { + int oneVectorFromEnd = width - 4; + for (; column <= oneVectorFromEnd; column += 4) + { + Vector128 sums = ConvolveWarpedVector512( + ref Unsafe.Add(ref source, column), + WarpedTileSize, + phase + (column * phaseStep), + phaseStep); + + for (int lane = 0; lane < 4; lane++) + { + Unsafe.Add(ref destination, column + lane) = FinishWarpedCompound(sums.GetElement(lane), bias); + } + } + } + + if (useHardwareIntrinsics && Vector256.IsHardwareAccelerated) + { + int oneVectorFromEnd = width - 2; + for (; column <= oneVectorFromEnd; column += 2) + { + Vector128 sums = ConvolveWarpedVector256( + ref Unsafe.Add(ref source, column), + WarpedTileSize, + phase + (column * phaseStep), + phaseStep); + + Unsafe.Add(ref destination, column) = FinishWarpedCompound(sums.GetElement(0), bias); + Unsafe.Add(ref destination, column + 1) = FinishWarpedCompound(sums.GetElement(1), bias); + } + } + + if (useHardwareIntrinsics && Vector128.IsHardwareAccelerated) + { + for (; column < width; column++) + { + int sum = ConvolveWarpedVector128( + ref Unsafe.Add(ref source, column), + WarpedTileSize, + phase + (column * phaseStep)); + + Unsafe.Add(ref destination, column) = FinishWarpedCompound(sum, bias); + } + } + + for (; column < width; column++) + { + ref short coefficients = ref GetWarpedFilterReference(phase + (column * phaseStep)); + int sum = TOperator.Convolve(ref Unsafe.Add(ref source, column), WarpedTileSize, ref coefficients); + Unsafe.Add(ref destination, column) = FinishWarpedCompound(sum, bias); + } + } + + /// + /// Completes one high-bit-depth vertical warped-filter row. + /// + private static void FilterWarpedVertical( + ref ushort source, + ref ushort destination, + int width, + int phase, + int phaseStep, + int bias, + int round, + int bitDepth, + int maximum, + bool useHardwareIntrinsics) + where TOperator : struct, IAv1WarpedPredictionOperator + { + int column = 0; + if (useHardwareIntrinsics && Vector512.IsHardwareAccelerated) + { + int oneVectorFromEnd = width - 4; + for (; column <= oneVectorFromEnd; column += 4) + { + Vector128 sums = ConvolveWarpedVector512( + ref Unsafe.Add(ref source, column), + WarpedTileSize, + phase + (column * phaseStep), + phaseStep); + + for (int lane = 0; lane < 4; lane++) + { + Unsafe.Add(ref destination, column + lane) = FinishWarpedHighBitDepth(sums.GetElement(lane), bias, round, bitDepth, maximum); + } + } + } + + if (useHardwareIntrinsics && Vector256.IsHardwareAccelerated) + { + int oneVectorFromEnd = width - 2; + for (; column <= oneVectorFromEnd; column += 2) + { + Vector128 sums = ConvolveWarpedVector256( + ref Unsafe.Add(ref source, column), + WarpedTileSize, + phase + (column * phaseStep), + phaseStep); + + Unsafe.Add(ref destination, column) = FinishWarpedHighBitDepth(sums.GetElement(0), bias, round, bitDepth, maximum); + Unsafe.Add(ref destination, column + 1) = FinishWarpedHighBitDepth(sums.GetElement(1), bias, round, bitDepth, maximum); + } + } + + if (useHardwareIntrinsics && Vector128.IsHardwareAccelerated) + { + for (; column < width; column++) + { + int sum = ConvolveWarpedVector128( + ref Unsafe.Add(ref source, column), + WarpedTileSize, + phase + (column * phaseStep)); + + Unsafe.Add(ref destination, column) = FinishWarpedHighBitDepth(sum, bias, round, bitDepth, maximum); + } + } + + for (; column < width; column++) + { + ref short coefficients = ref GetWarpedFilterReference(phase + (column * phaseStep)); + int sum = TOperator.Convolve(ref Unsafe.Add(ref source, column), WarpedTileSize, ref coefficients); + Unsafe.Add(ref destination, column) = FinishWarpedHighBitDepth(sum, bias, round, bitDepth, maximum); + } + } + + /// + /// Convolves four adjacent 8-bit source windows with independent phases. + /// + private static Vector128 ConvolveWarpedVector512(ref byte source, int phase, int phaseStep) + where TOperator : struct, IAv1WarpedPredictionOperator + { + Vector256 sampleLower = Vector256.Create(LoadWarpedWindow(ref source), LoadWarpedWindow(ref Unsafe.Add(ref source, 1))); + Vector256 sampleUpper = Vector256.Create(LoadWarpedWindow(ref Unsafe.Add(ref source, 2)), LoadWarpedWindow(ref Unsafe.Add(ref source, 3))); + Vector256 coefficientLower = Vector256.Create(LoadWarpedCoefficients(phase), LoadWarpedCoefficients(phase + phaseStep)); + Vector256 coefficientUpper = Vector256.Create(LoadWarpedCoefficients(phase + (2 * phaseStep)), LoadWarpedCoefficients(phase + (3 * phaseStep))); + return TOperator.Convolve(Vector512.Create(sampleLower, sampleUpper), Vector512.Create(coefficientLower, coefficientUpper)); + } + + /// + /// Convolves two adjacent 8-bit source windows with independent phases. + /// + private static Vector128 ConvolveWarpedVector256(ref byte source, int phase, int phaseStep) + where TOperator : struct, IAv1WarpedPredictionOperator + => TOperator.Convolve( + Vector256.Create(LoadWarpedWindow(ref source), LoadWarpedWindow(ref Unsafe.Add(ref source, 1))), + Vector256.Create(LoadWarpedCoefficients(phase), LoadWarpedCoefficients(phase + phaseStep))); + + /// + /// Convolves one 8-bit source window. + /// + private static int ConvolveWarpedVector128(ref byte source, int phase) + where TOperator : struct, IAv1WarpedPredictionOperator + => TOperator.Convolve(LoadWarpedWindow(ref source), LoadWarpedCoefficients(phase)); + + /// + /// Convolves four adjacent high-bit-depth source windows with independent phases. + /// + private static Vector128 ConvolveWarpedVector512(ref ushort source, int sourceStride, int phase, int phaseStep) + where TOperator : struct, IAv1WarpedPredictionOperator + { + Vector256 sampleLower = Vector256.Create(LoadWarpedWindow(ref source, sourceStride), LoadWarpedWindow(ref Unsafe.Add(ref source, 1), sourceStride)); + Vector256 sampleUpper = Vector256.Create(LoadWarpedWindow(ref Unsafe.Add(ref source, 2), sourceStride), LoadWarpedWindow(ref Unsafe.Add(ref source, 3), sourceStride)); + Vector256 coefficientLower = Vector256.Create(LoadWarpedCoefficients(phase), LoadWarpedCoefficients(phase + phaseStep)); + Vector256 coefficientUpper = Vector256.Create(LoadWarpedCoefficients(phase + (2 * phaseStep)), LoadWarpedCoefficients(phase + (3 * phaseStep))); + return TOperator.Convolve(Vector512.Create(sampleLower, sampleUpper), Vector512.Create(coefficientLower, coefficientUpper)); + } + + /// + /// Convolves two adjacent high-bit-depth source windows with independent phases. + /// + private static Vector128 ConvolveWarpedVector256(ref ushort source, int sourceStride, int phase, int phaseStep) + where TOperator : struct, IAv1WarpedPredictionOperator + => TOperator.Convolve( + Vector256.Create(LoadWarpedWindow(ref source, sourceStride), LoadWarpedWindow(ref Unsafe.Add(ref source, 1), sourceStride)), + Vector256.Create(LoadWarpedCoefficients(phase), LoadWarpedCoefficients(phase + phaseStep))); + + /// + /// Convolves one high-bit-depth source window. + /// + private static int ConvolveWarpedVector128(ref ushort source, int sourceStride, int phase) + where TOperator : struct, IAv1WarpedPredictionOperator + => TOperator.Convolve(LoadWarpedWindow(ref source, sourceStride), LoadWarpedCoefficients(phase)); + + /// + /// Loads eight adjacent unsigned byte samples as unsigned 16-bit lanes. + /// + private static Vector128 LoadWarpedWindow(ref byte source) + { + Vector128 packed = Vector128.LoadUnsafe(ref source); + return Vector128.Widen(packed).Lower; + } + + /// + /// Loads eight unsigned high-bit-depth samples with the supplied spacing. + /// + private static Vector128 LoadWarpedWindow(ref ushort source, int sourceStride) + { + if (sourceStride == 1) + { + return Vector128.LoadUnsafe(ref source); + } + + return Vector128.Create( + source, + Unsafe.Add(ref source, sourceStride), + Unsafe.Add(ref source, sourceStride * 2), + Unsafe.Add(ref source, sourceStride * 3), + Unsafe.Add(ref source, sourceStride * 4), + Unsafe.Add(ref source, sourceStride * 5), + Unsafe.Add(ref source, sourceStride * 6), + Unsafe.Add(ref source, sourceStride * 7)); + } + + /// + /// Loads one signed Q7 warped-filter phase. + /// + private static Vector128 LoadWarpedCoefficients(int phase) + => Vector128.LoadUnsafe(ref GetWarpedFilterReference(phase)); + + /// + /// Applies the final 8-bit warped-prediction rounding and clipping. + /// + private static byte FinishWarpedByte(int sum, int bias, int round) + { + int value = RoundPowerOfTwoScalar(bias + sum, round) - (1 << 7) - (1 << 8); + return (byte)Math.Clamp(value, byte.MinValue, byte.MaxValue); + } + + /// + /// Applies the compound-intermediate warped-prediction rounding. + /// + private static ushort FinishWarpedCompound(int sum, int bias) + => (ushort)RoundPowerOfTwoScalar(bias + sum, Av1CompoundInterPredictor.CompoundRound1Bits); + + /// + /// Applies the final high-bit-depth warped-prediction rounding and clipping. + /// + private static ushort FinishWarpedHighBitDepth(int sum, int bias, int round, int bitDepth, int maximum) + { + int value = RoundPowerOfTwoScalar(bias + sum, round) - (1 << (bitDepth - 1)) - (1 << bitDepth); + return (ushort)Math.Clamp(value, 0, maximum); + } + + /// + /// Reconstructs one 8-bit warped reference without discarding the compound convolution precision. + /// + private static void PredictWarpedCompound( + ReadOnlySpan source, + int sourceStride, + Point sourceOrigin, + int sourceWidth, + int sourceHeight, + Span destination, + int destinationStride, + Point destinationPosition, + int width, + int height, + int subsamplingX, + int subsamplingY, + Av1GlobalMotionParameters parameters, + Span scratch, + bool useHardwareIntrinsics) + where TOperator : struct, IAv1WarpedPredictionOperator + { + ref byte sourceBase = ref MemoryMarshal.GetReference(source); + ref ushort destinationBase = ref MemoryMarshal.GetReference(destination); + Span intermediate = MemoryMarshal.Cast(scratch)[..WarpedScratchLength]; + int horizontalBias = 1 << (8 + FilterBits - 1); + int verticalBias = 1 << (8 + (2 * FilterBits) - Round0Bits); + + for (int tileRow = destinationPosition.Y; tileRow < destinationPosition.Y + height; tileRow += WarpedTileSize) + { + for (int tileColumn = destinationPosition.X; tileColumn < destinationPosition.X + width; tileColumn += WarpedTileSize) + { + DeriveWarpedTilePosition( + parameters, + tileColumn, + tileRow, + subsamplingX, + subsamplingY, + out int integerX, + out int integerY, + out int phaseX, + out int phaseY); + + FilterWarpedHorizontal( + ref sourceBase, + sourceStride, + sourceOrigin, + sourceHeight, + integerX, + integerY, + phaseX, + parameters, + intermediate, + horizontalBias, + Round0Bits, + useHardwareIntrinsics); + + int tileHeight = Math.Min(WarpedTileSize, destinationPosition.Y + height - tileRow); + int tileWidth = Math.Min(WarpedTileSize, destinationPosition.X + width - tileColumn); + for (int row = 0; row < tileHeight; row++) + { + int phase = phaseY + (parameters.Delta * row); + int destinationRowOffset = (tileRow - destinationPosition.Y + row) * destinationStride; + ref ushort intermediateSource = ref intermediate[row * WarpedTileSize]; + ref ushort destinationRow = ref Unsafe.Add( + ref destinationBase, + destinationRowOffset + tileColumn - destinationPosition.X); + + FilterWarpedCompoundVertical( + ref intermediateSource, + ref destinationRow, + tileWidth, + phase, + parameters.Gamma, + verticalBias, + useHardwareIntrinsics); + } + } + } + } + + /// + /// Reconstructs one high-bit-depth warped block through a closed convolution operator. + /// + private static void PredictWarped( + ReadOnlySpan source, + int sourceStride, + Point sourceOrigin, + int sourceWidth, + int sourceHeight, + Span destination, + int destinationStride, + Point destinationPosition, + int width, + int height, + int subsamplingX, + int subsamplingY, + int bitDepth, + Av1GlobalMotionParameters parameters, + Span scratch, + bool useHardwareIntrinsics) + where TOperator : struct, IAv1WarpedPredictionOperator + { + ref ushort sourceBase = ref MemoryMarshal.GetReference(source); + ref ushort destinationBase = ref MemoryMarshal.GetReference(destination); + Span intermediate = MemoryMarshal.Cast(scratch)[..WarpedScratchLength]; + + // Twelve-bit prediction increases round0 by two so the biased horizontal intermediate remains representable + // in sixteen bits. Reducing round1 by the same amount preserves the complete normative Q14 shift. + int intermediateRange = bitDepth + FilterBits - Round0Bits + 2; + int round0 = Round0Bits + Math.Max(intermediateRange - 16, 0); + int verticalRound = (2 * FilterBits) - round0; + int horizontalBias = 1 << (bitDepth + FilterBits - 1); + int verticalBias = 1 << (bitDepth + (2 * FilterBits) - round0); + int maximum = (1 << bitDepth) - 1; + + for (int tileRow = destinationPosition.Y; tileRow < destinationPosition.Y + height; tileRow += WarpedTileSize) + { + for (int tileColumn = destinationPosition.X; tileColumn < destinationPosition.X + width; tileColumn += WarpedTileSize) + { + DeriveWarpedTilePosition( + parameters, + tileColumn, + tileRow, + subsamplingX, + subsamplingY, + out int integerX, + out int integerY, + out int phaseX, + out int phaseY); + + FilterWarpedHorizontal( + ref sourceBase, + sourceStride, + sourceOrigin, + sourceHeight, + integerX, + integerY, + phaseX, + parameters, + intermediate, + horizontalBias, + round0, + useHardwareIntrinsics); + + int tileHeight = Math.Min(WarpedTileSize, destinationPosition.Y + height - tileRow); + int tileWidth = Math.Min(WarpedTileSize, destinationPosition.X + width - tileColumn); + for (int row = 0; row < tileHeight; row++) + { + int phase = phaseY + (parameters.Delta * row); + int destinationRowOffset = (tileRow - destinationPosition.Y + row) * destinationStride; + ref ushort intermediateSource = ref intermediate[row * WarpedTileSize]; + ref ushort destinationRow = ref Unsafe.Add( + ref destinationBase, + destinationRowOffset + tileColumn - destinationPosition.X); + + FilterWarpedVertical( + ref intermediateSource, + ref destinationRow, + tileWidth, + phase, + parameters.Gamma, + verticalBias, + verticalRound, + bitDepth, + maximum, + useHardwareIntrinsics); + } + } + } + } + + /// + /// Projects the center of one 8x8 output tile and derives its integer source position and reduced phases. + /// + private static void DeriveWarpedTilePosition( + Av1GlobalMotionParameters parameters, + int tileColumn, + int tileRow, + int subsamplingX, + int subsamplingY, + out int integerX, + out int integerY, + out int phaseX, + out int phaseY) + { + int sourceX = (tileColumn + 4) << subsamplingX; + int sourceY = (tileRow + 4) << subsamplingY; + long projectedX = ((long)parameters[2] * sourceX) + ((long)parameters[3] * sourceY) + parameters[0]; + long projectedY = ((long)parameters[4] * sourceX) + ((long)parameters[5] * sourceY) + parameters[1]; + long planeX = projectedX >> subsamplingX; + long planeY = projectedY >> subsamplingY; + integerX = (int)(planeX >> Av1GlobalMotionParameters.ModelPrecisionBits); + integerY = (int)(planeY >> Av1GlobalMotionParameters.ModelPrecisionBits); + phaseX = (int)planeX & (Av1GlobalMotionParameters.ModelScale - 1); + phaseY = (int)planeY & (Av1GlobalMotionParameters.ModelScale - 1); + phaseX += (-4 * parameters.Alpha) + (-4 * parameters.Beta); + phaseY += (-4 * parameters.Gamma) + (-4 * parameters.Delta); + + // Shear parameters are quantized to 64-model-unit steps. Clearing the same low bits after the tile-center + // projection keeps negative and positive phases on the exact filter-table grid used by the bitstream model. + phaseX &= -1 << 6; + phaseY &= -1 << 6; + } + + /// + /// Gets a reference to the first coefficient for one reduced warped-filter phase. + /// + private static ref short GetWarpedFilterReference(int phase) + { + int filterIndex = ((phase + (1 << (WarpedDifferencePrecisionBits - 1))) >> WarpedDifferencePrecisionBits) + + WarpedPixelPrecisionShifts; + + return ref Unsafe.Add(ref MemoryMarshal.GetReference(WarpedFilter), filterIndex * FilterCoefficientCount); + } + + /// + /// Divides a nonnegative value by a power of two with nearest-integer rounding. + /// + private static int RoundPowerOfTwoScalar(int value, int bitCount) + => (value + (1 << (bitCount - 1))) >> bitCount; +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyPredictor.BilinearOperator.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyBilinearPredictor.Operator.cs similarity index 56% rename from src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyPredictor.BilinearOperator.cs rename to src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyBilinearPredictor.Operator.cs index 2e2fa0d78..20b9ba0ac 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyPredictor.BilinearOperator.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyBilinearPredictor.Operator.cs @@ -5,8 +5,121 @@ using System.Runtime.Intrinsics; namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.IntraBlockCopy; -internal static partial class Av1IntraBlockCopyPredictor +/// +/// Defines the closed bilinear intra-block-copy interpolation operator. +/// +internal static partial class Av1IntraBlockCopyBilinearPredictor { + /// + /// Defines bilinear intra-block-copy filtering for scalar and SIMD lane groups. + /// + private interface IAv1IntraBlockCopyBilinearOperator + { + /// + /// Filters one 8-bit sample. + /// + /// The integer-position source sample. + /// The source sample one column to the right. + /// The source sample one row below. + /// The source sample one row below and one column to the right. + /// The filtered 8-bit sample. + public static abstract byte Filter(byte topLeft, byte topRight, byte bottomLeft, byte bottomRight); + + /// + /// Filters sixteen 8-bit samples in parallel. + /// + /// The integer-position source samples. + /// The source samples one column to the right. + /// The source samples one row below. + /// The source samples one row below and one column to the right. + /// The filtered 8-bit samples. + public static abstract Vector128 Filter( + Vector128 topLeft, + Vector128 topRight, + Vector128 bottomLeft, + Vector128 bottomRight); + + /// + /// Filters thirty-two 8-bit samples in parallel. + /// + /// The integer-position source samples. + /// The source samples one column to the right. + /// The source samples one row below. + /// The source samples one row below and one column to the right. + /// The filtered 8-bit samples. + public static abstract Vector256 Filter( + Vector256 topLeft, + Vector256 topRight, + Vector256 bottomLeft, + Vector256 bottomRight); + + /// + /// Filters sixty-four 8-bit samples in parallel. + /// + /// The integer-position source samples. + /// The source samples one column to the right. + /// The source samples one row below. + /// The source samples one row below and one column to the right. + /// The filtered 8-bit samples. + public static abstract Vector512 Filter( + Vector512 topLeft, + Vector512 topRight, + Vector512 bottomLeft, + Vector512 bottomRight); + + /// + /// Filters one high-bit-depth sample. + /// + /// The integer-position source sample. + /// The source sample one column to the right. + /// The source sample one row below. + /// The source sample one row below and one column to the right. + /// The filtered high-bit-depth sample. + public static abstract short Filter(short topLeft, short topRight, short bottomLeft, short bottomRight); + + /// + /// Filters eight high-bit-depth samples in parallel. + /// + /// The integer-position source samples. + /// The source samples one column to the right. + /// The source samples one row below. + /// The source samples one row below and one column to the right. + /// The filtered high-bit-depth samples. + public static abstract Vector128 Filter( + Vector128 topLeft, + Vector128 topRight, + Vector128 bottomLeft, + Vector128 bottomRight); + + /// + /// Filters sixteen high-bit-depth samples in parallel. + /// + /// The integer-position source samples. + /// The source samples one column to the right. + /// The source samples one row below. + /// The source samples one row below and one column to the right. + /// The filtered high-bit-depth samples. + public static abstract Vector256 Filter( + Vector256 topLeft, + Vector256 topRight, + Vector256 bottomLeft, + Vector256 bottomRight); + + /// + /// Filters thirty-two high-bit-depth samples in parallel. + /// + /// The integer-position source samples. + /// The source samples one column to the right. + /// The source samples one row below. + /// The source samples one row below and one column to the right. + /// The filtered high-bit-depth samples. + public static abstract Vector512 Filter( + Vector512 topLeft, + Vector512 topRight, + Vector512 bottomLeft, + Vector512 bottomRight); + } + /// /// Applies the separable two-dimensional interpolation required when both source axes have a half-sample phase. /// @@ -19,14 +132,8 @@ internal static partial class Av1IntraBlockCopyPredictor /// unsigned 32-bit halves. Narrowing recombines those halves in source-column order after the rounded result has /// returned to the original sample range. /// - private readonly struct BilinearOperator : IAv1IntraBlockCopyOperator + private readonly struct IntraBlockCopyBilinearOperator : IAv1IntraBlockCopyBilinearOperator { - /// - public static bool UsesRight => true; - - /// - public static bool UsesBottom => true; - /// public static byte Filter(byte topLeft, byte topRight, byte bottomLeft, byte bottomRight) => (byte)((topLeft + topRight + bottomLeft + bottomRight + 2) >> 2); diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyPredictor.Operator.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyBilinearPredictor.cs similarity index 50% rename from src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyPredictor.Operator.cs rename to src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyBilinearPredictor.cs index 013cedd14..b4f4321c8 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyPredictor.Operator.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyBilinearPredictor.cs @@ -8,134 +8,57 @@ using System.Runtime.Intrinsics; namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.IntraBlockCopy; /// -/// Defines the scalar and SIMD contract for closed intra-block-copy filter operators, and provides their shared -/// width-progressive SIMD traversal. +/// Provides the family-owned scalar and width-progressive SIMD traversal for bilinear intra-block-copy prediction. /// -internal static partial class Av1IntraBlockCopyPredictor +internal static partial class Av1IntraBlockCopyBilinearPredictor { /// - /// Defines lane-wise arithmetic for one intra-block-copy filter phase. + /// Reconstructs an 8-bit filtered intra-block-copy prediction. /// - /// - /// Every SIMD lane corresponds to one output column. The generic traversal supplies the integer source sample and - /// its right, lower, and lower-right neighbors; closed operator types allow the JIT to remove unused source loads. - /// - private interface IAv1IntraBlockCopyOperator - { - /// - /// Gets a value indicating whether the operator consumes the source sample to the right. - /// - public static abstract bool UsesRight { get; } - - /// - /// Gets a value indicating whether the operator consumes the source sample on the following row. - /// - public static abstract bool UsesBottom { get; } - - /// - /// Filters one 8-bit sample. - /// - /// The integer-position source sample. - /// The source sample one column to the right. - /// The source sample one row below. - /// The source sample one row below and one column to the right. - /// The filtered 8-bit sample. - public static abstract byte Filter(byte topLeft, byte topRight, byte bottomLeft, byte bottomRight); - - /// - /// Filters sixteen 8-bit samples in parallel. - /// - /// The integer-position source samples. - /// The source samples one column to the right. - /// The source samples one row below. - /// The source samples one row below and one column to the right. - /// The filtered 8-bit samples. - public static abstract Vector128 Filter( - Vector128 topLeft, - Vector128 topRight, - Vector128 bottomLeft, - Vector128 bottomRight); - - /// - /// Filters thirty-two 8-bit samples in parallel. - /// - /// The integer-position source samples. - /// The source samples one column to the right. - /// The source samples one row below. - /// The source samples one row below and one column to the right. - /// The filtered 8-bit samples. - public static abstract Vector256 Filter( - Vector256 topLeft, - Vector256 topRight, - Vector256 bottomLeft, - Vector256 bottomRight); - - /// - /// Filters sixty-four 8-bit samples in parallel. - /// - /// The integer-position source samples. - /// The source samples one column to the right. - /// The source samples one row below. - /// The source samples one row below and one column to the right. - /// The filtered 8-bit samples. - public static abstract Vector512 Filter( - Vector512 topLeft, - Vector512 topRight, - Vector512 bottomLeft, - Vector512 bottomRight); - - /// - /// Filters one high-bit-depth sample. - /// - /// The integer-position source sample. - /// The source sample one column to the right. - /// The source sample one row below. - /// The source sample one row below and one column to the right. - /// The filtered high-bit-depth sample. - public static abstract short Filter(short topLeft, short topRight, short bottomLeft, short bottomRight); - - /// - /// Filters eight high-bit-depth samples in parallel. - /// - /// The integer-position source samples. - /// The source samples one column to the right. - /// The source samples one row below. - /// The source samples one row below and one column to the right. - /// The filtered high-bit-depth samples. - public static abstract Vector128 Filter( - Vector128 topLeft, - Vector128 topRight, - Vector128 bottomLeft, - Vector128 bottomRight); - - /// - /// Filters sixteen high-bit-depth samples in parallel. - /// - /// The integer-position source samples. - /// The source samples one column to the right. - /// The source samples one row below. - /// The source samples one row below and one column to the right. - /// The filtered high-bit-depth samples. - public static abstract Vector256 Filter( - Vector256 topLeft, - Vector256 topRight, - Vector256 bottomLeft, - Vector256 bottomRight); - - /// - /// Filters thirty-two high-bit-depth samples in parallel. - /// - /// The integer-position source samples. - /// The source samples one column to the right. - /// The source samples one row below. - /// The source samples one row below and one column to the right. - /// The filtered high-bit-depth samples. - public static abstract Vector512 Filter( - Vector512 topLeft, - Vector512 topRight, - Vector512 bottomLeft, - Vector512 bottomRight); - } + public static void Predict( + ReadOnlySpan source, + int sourceStride, + Span destination, + int destinationStride, + int width, + int height) + => Predict(source, sourceStride, destination, destinationStride, width, height); + + /// + /// Reconstructs a high-bit-depth filtered intra-block-copy prediction. + /// + public static void Predict( + ReadOnlySpan source, + int sourceStride, + Span destination, + int destinationStride, + int width, + int height) + => Predict(source, sourceStride, destination, destinationStride, width, height); + + /// + /// Reconstructs an 8-bit filtered intra-block-copy prediction without explicit hardware intrinsics. + /// + public static void PredictScalar( + ReadOnlySpan source, + int sourceStride, + Span destination, + int destinationStride, + int width, + int height) + => PredictScalar(source, sourceStride, destination, destinationStride, width, height); + + /// + /// Reconstructs a high-bit-depth filtered intra-block-copy prediction without explicit hardware intrinsics. + /// + public static void PredictScalar( + ReadOnlySpan source, + int sourceStride, + Span destination, + int destinationStride, + int width, + int height) + => PredictScalar(source, sourceStride, destination, destinationStride, width, height); /// /// Applies one closed interpolation operator to an 8-bit source block. @@ -148,7 +71,7 @@ internal static partial class Av1IntraBlockCopyPredictor int destinationStride, int width, int height) - where TOperator : struct, IAv1IntraBlockCopyOperator + where TOperator : struct, IAv1IntraBlockCopyBilinearOperator { ref byte sourceBase = ref MemoryMarshal.GetReference(source); ref byte destinationBase = ref MemoryMarshal.GetReference(destination); @@ -163,11 +86,9 @@ internal static partial class Av1IntraBlockCopyPredictor ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride); ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); Vector128 topLeft = Vector128.LoadUnsafe(ref sourceRow); - Vector128 topRight = TOperator.UsesRight ? Vector128.LoadUnsafe(ref sourceRow, 1) : default; - Vector128 bottomLeft = TOperator.UsesBottom ? Vector128.LoadUnsafe(ref sourceRow, (nuint)sourceStride) : default; - Vector128 bottomRight = TOperator.UsesRight && TOperator.UsesBottom - ? Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + 1)) - : default; + Vector128 topRight = Vector128.LoadUnsafe(ref sourceRow, 1); + Vector128 bottomLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)sourceStride); + Vector128 bottomRight = Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + 1)); Vector128 prediction = TOperator.Filter(topLeft, topRight, bottomLeft, bottomRight); if (width == 8) @@ -200,13 +121,9 @@ internal static partial class Av1IntraBlockCopyPredictor for (int column = 0; column < vectorizedColumns; column += Vector512.Count) { Vector512 topLeft = Vector512.LoadUnsafe(ref sourceRow, (nuint)column); - Vector512 topRight = TOperator.UsesRight ? Vector512.LoadUnsafe(ref sourceRow, (nuint)(column + 1)) : default; - Vector512 bottomLeft = TOperator.UsesBottom - ? Vector512.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column)) - : default; - Vector512 bottomRight = TOperator.UsesRight && TOperator.UsesBottom - ? Vector512.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column + 1)) - : default; + Vector512 topRight = Vector512.LoadUnsafe(ref sourceRow, (nuint)(column + 1)); + Vector512 bottomLeft = Vector512.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column)); + Vector512 bottomRight = Vector512.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column + 1)); TOperator.Filter(topLeft, topRight, bottomLeft, bottomRight).StoreUnsafe(ref destinationRow, (nuint)column); } @@ -230,13 +147,9 @@ internal static partial class Av1IntraBlockCopyPredictor for (int column = processedColumns; column < endColumn; column += Vector256.Count) { Vector256 topLeft = Vector256.LoadUnsafe(ref sourceRow, (nuint)column); - Vector256 topRight = TOperator.UsesRight ? Vector256.LoadUnsafe(ref sourceRow, (nuint)(column + 1)) : default; - Vector256 bottomLeft = TOperator.UsesBottom - ? Vector256.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column)) - : default; - Vector256 bottomRight = TOperator.UsesRight && TOperator.UsesBottom - ? Vector256.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column + 1)) - : default; + Vector256 topRight = Vector256.LoadUnsafe(ref sourceRow, (nuint)(column + 1)); + Vector256 bottomLeft = Vector256.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column)); + Vector256 bottomRight = Vector256.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column + 1)); TOperator.Filter(topLeft, topRight, bottomLeft, bottomRight).StoreUnsafe(ref destinationRow, (nuint)column); } @@ -259,13 +172,9 @@ internal static partial class Av1IntraBlockCopyPredictor for (int column = processedColumns; column < endColumn; column += Vector128.Count) { Vector128 topLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)column); - Vector128 topRight = TOperator.UsesRight ? Vector128.LoadUnsafe(ref sourceRow, (nuint)(column + 1)) : default; - Vector128 bottomLeft = TOperator.UsesBottom - ? Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column)) - : default; - Vector128 bottomRight = TOperator.UsesRight && TOperator.UsesBottom - ? Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column + 1)) - : default; + Vector128 topRight = Vector128.LoadUnsafe(ref sourceRow, (nuint)(column + 1)); + Vector128 bottomLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column)); + Vector128 bottomRight = Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column + 1)); TOperator.Filter(topLeft, topRight, bottomLeft, bottomRight).StoreUnsafe(ref destinationRow, (nuint)column); } @@ -284,11 +193,9 @@ internal static partial class Av1IntraBlockCopyPredictor for (int column = processedColumns; column < width; column++) { byte topLeft = Unsafe.Add(ref sourceRow, column); - byte topRight = TOperator.UsesRight ? Unsafe.Add(ref sourceRow, column + 1) : default; - byte bottomLeft = TOperator.UsesBottom ? Unsafe.Add(ref sourceRow, sourceStride + column) : default; - byte bottomRight = TOperator.UsesRight && TOperator.UsesBottom - ? Unsafe.Add(ref sourceRow, sourceStride + column + 1) - : default; + byte topRight = Unsafe.Add(ref sourceRow, column + 1); + byte bottomLeft = Unsafe.Add(ref sourceRow, sourceStride + column); + byte bottomRight = Unsafe.Add(ref sourceRow, sourceStride + column + 1); Unsafe.Add(ref destinationRow, column) = TOperator.Filter(topLeft, topRight, bottomLeft, bottomRight); } @@ -306,7 +213,7 @@ internal static partial class Av1IntraBlockCopyPredictor int destinationStride, int width, int height) - where TOperator : struct, IAv1IntraBlockCopyOperator + where TOperator : struct, IAv1IntraBlockCopyBilinearOperator { ref short sourceBase = ref MemoryMarshal.GetReference(source); ref short destinationBase = ref MemoryMarshal.GetReference(destination); @@ -321,11 +228,9 @@ internal static partial class Av1IntraBlockCopyPredictor ref short sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride); ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); Vector128 topLeft = Vector128.LoadUnsafe(ref sourceRow); - Vector128 topRight = TOperator.UsesRight ? Vector128.LoadUnsafe(ref sourceRow, 1) : default; - Vector128 bottomLeft = TOperator.UsesBottom ? Vector128.LoadUnsafe(ref sourceRow, (nuint)sourceStride) : default; - Vector128 bottomRight = TOperator.UsesRight && TOperator.UsesBottom - ? Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + 1)) - : default; + Vector128 topRight = Vector128.LoadUnsafe(ref sourceRow, 1); + Vector128 bottomLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)sourceStride); + Vector128 bottomRight = Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + 1)); TOperator.Filter(topLeft, topRight, bottomLeft, bottomRight).GetLower().StoreUnsafe(ref destinationRow); } @@ -350,13 +255,9 @@ internal static partial class Av1IntraBlockCopyPredictor for (int column = 0; column < vectorizedColumns; column += Vector512.Count) { Vector512 topLeft = Vector512.LoadUnsafe(ref sourceRow, (nuint)column); - Vector512 topRight = TOperator.UsesRight ? Vector512.LoadUnsafe(ref sourceRow, (nuint)(column + 1)) : default; - Vector512 bottomLeft = TOperator.UsesBottom - ? Vector512.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column)) - : default; - Vector512 bottomRight = TOperator.UsesRight && TOperator.UsesBottom - ? Vector512.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column + 1)) - : default; + Vector512 topRight = Vector512.LoadUnsafe(ref sourceRow, (nuint)(column + 1)); + Vector512 bottomLeft = Vector512.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column)); + Vector512 bottomRight = Vector512.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column + 1)); TOperator.Filter(topLeft, topRight, bottomLeft, bottomRight).StoreUnsafe(ref destinationRow, (nuint)column); } @@ -380,13 +281,9 @@ internal static partial class Av1IntraBlockCopyPredictor for (int column = processedColumns; column < endColumn; column += Vector256.Count) { Vector256 topLeft = Vector256.LoadUnsafe(ref sourceRow, (nuint)column); - Vector256 topRight = TOperator.UsesRight ? Vector256.LoadUnsafe(ref sourceRow, (nuint)(column + 1)) : default; - Vector256 bottomLeft = TOperator.UsesBottom - ? Vector256.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column)) - : default; - Vector256 bottomRight = TOperator.UsesRight && TOperator.UsesBottom - ? Vector256.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column + 1)) - : default; + Vector256 topRight = Vector256.LoadUnsafe(ref sourceRow, (nuint)(column + 1)); + Vector256 bottomLeft = Vector256.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column)); + Vector256 bottomRight = Vector256.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column + 1)); TOperator.Filter(topLeft, topRight, bottomLeft, bottomRight).StoreUnsafe(ref destinationRow, (nuint)column); } @@ -409,13 +306,9 @@ internal static partial class Av1IntraBlockCopyPredictor for (int column = processedColumns; column < endColumn; column += Vector128.Count) { Vector128 topLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)column); - Vector128 topRight = TOperator.UsesRight ? Vector128.LoadUnsafe(ref sourceRow, (nuint)(column + 1)) : default; - Vector128 bottomLeft = TOperator.UsesBottom - ? Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column)) - : default; - Vector128 bottomRight = TOperator.UsesRight && TOperator.UsesBottom - ? Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column + 1)) - : default; + Vector128 topRight = Vector128.LoadUnsafe(ref sourceRow, (nuint)(column + 1)); + Vector128 bottomLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column)); + Vector128 bottomRight = Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column + 1)); TOperator.Filter(topLeft, topRight, bottomLeft, bottomRight).StoreUnsafe(ref destinationRow, (nuint)column); } @@ -432,11 +325,9 @@ internal static partial class Av1IntraBlockCopyPredictor for (int column = processedColumns; column < width; column++) { short topLeft = Unsafe.Add(ref sourceRow, column); - short topRight = TOperator.UsesRight ? Unsafe.Add(ref sourceRow, column + 1) : default; - short bottomLeft = TOperator.UsesBottom ? Unsafe.Add(ref sourceRow, sourceStride + column) : default; - short bottomRight = TOperator.UsesRight && TOperator.UsesBottom - ? Unsafe.Add(ref sourceRow, sourceStride + column + 1) - : default; + short topRight = Unsafe.Add(ref sourceRow, column + 1); + short bottomLeft = Unsafe.Add(ref sourceRow, sourceStride + column); + short bottomRight = Unsafe.Add(ref sourceRow, sourceStride + column + 1); Unsafe.Add(ref destinationRow, column) = TOperator.Filter(topLeft, topRight, bottomLeft, bottomRight); } @@ -454,7 +345,7 @@ internal static partial class Av1IntraBlockCopyPredictor int destinationStride, int width, int height) - where TOperator : struct, IAv1IntraBlockCopyOperator + where TOperator : struct, IAv1IntraBlockCopyBilinearOperator { for (int row = 0; row < height; row++) { @@ -464,9 +355,9 @@ internal static partial class Av1IntraBlockCopyPredictor for (int column = 0; column < width; column++) { byte topLeft = source[sourceRow + column]; - byte topRight = TOperator.UsesRight ? source[sourceRow + column + 1] : default; - byte bottomLeft = TOperator.UsesBottom ? source[sourceRow + sourceStride + column] : default; - byte bottomRight = TOperator.UsesRight && TOperator.UsesBottom ? source[sourceRow + sourceStride + column + 1] : default; + byte topRight = source[sourceRow + column + 1]; + byte bottomLeft = source[sourceRow + sourceStride + column]; + byte bottomRight = source[sourceRow + sourceStride + column + 1]; destination[destinationRow + column] = TOperator.Filter(topLeft, topRight, bottomLeft, bottomRight); } } @@ -483,7 +374,7 @@ internal static partial class Av1IntraBlockCopyPredictor int destinationStride, int width, int height) - where TOperator : struct, IAv1IntraBlockCopyOperator + where TOperator : struct, IAv1IntraBlockCopyBilinearOperator { for (int row = 0; row < height; row++) { @@ -493,9 +384,9 @@ internal static partial class Av1IntraBlockCopyPredictor for (int column = 0; column < width; column++) { short topLeft = source[sourceRow + column]; - short topRight = TOperator.UsesRight ? source[sourceRow + column + 1] : default; - short bottomLeft = TOperator.UsesBottom ? source[sourceRow + sourceStride + column] : default; - short bottomRight = TOperator.UsesRight && TOperator.UsesBottom ? source[sourceRow + sourceStride + column + 1] : default; + short topRight = source[sourceRow + column + 1]; + short bottomLeft = source[sourceRow + sourceStride + column]; + short bottomRight = source[sourceRow + sourceStride + column + 1]; destination[destinationRow + column] = TOperator.Filter(topLeft, topRight, bottomLeft, bottomRight); } } diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyHorizontalPredictor.Operator.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyHorizontalPredictor.Operator.cs new file mode 100644 index 000000000..a9a4d4304 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyHorizontalPredictor.Operator.cs @@ -0,0 +1,180 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.Intrinsics; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.IntraBlockCopy; + +/// +/// Defines the closed horizontal intra-block-copy interpolation operator. +/// +internal static partial class Av1IntraBlockCopyHorizontalPredictor +{ + /// + /// Defines horizontal intra-block-copy filtering for scalar and SIMD lane groups. + /// + private interface IAv1IntraBlockCopyHorizontalOperator + { + /// + /// Filters one 8-bit sample. + /// + /// The integer-position source sample. + /// The source sample one column to the right. + /// The filtered 8-bit sample. + public static abstract byte Filter(byte left, byte right); + + /// + /// Filters sixteen 8-bit samples in parallel. + /// + /// The integer-position source samples. + /// The source samples one column to the right. + /// The filtered 8-bit samples. + public static abstract Vector128 Filter( + Vector128 left, + Vector128 right); + + /// + /// Filters thirty-two 8-bit samples in parallel. + /// + /// The integer-position source samples. + /// The source samples one column to the right. + /// The filtered 8-bit samples. + public static abstract Vector256 Filter( + Vector256 left, + Vector256 right); + + /// + /// Filters sixty-four 8-bit samples in parallel. + /// + /// The integer-position source samples. + /// The source samples one column to the right. + /// The filtered 8-bit samples. + public static abstract Vector512 Filter( + Vector512 left, + Vector512 right); + + /// + /// Filters one high-bit-depth sample. + /// + /// The integer-position source sample. + /// The source sample one column to the right. + /// The filtered high-bit-depth sample. + public static abstract short Filter(short left, short right); + + /// + /// Filters eight high-bit-depth samples in parallel. + /// + /// The integer-position source samples. + /// The source samples one column to the right. + /// The filtered high-bit-depth samples. + public static abstract Vector128 Filter( + Vector128 left, + Vector128 right); + + /// + /// Filters sixteen high-bit-depth samples in parallel. + /// + /// The integer-position source samples. + /// The source samples one column to the right. + /// The filtered high-bit-depth samples. + public static abstract Vector256 Filter( + Vector256 left, + Vector256 right); + + /// + /// Filters thirty-two high-bit-depth samples in parallel. + /// + /// The integer-position source samples. + /// The source samples one column to the right. + /// The filtered high-bit-depth samples. + public static abstract Vector512 Filter( + Vector512 left, + Vector512 right); + } + + /// + /// Averages horizontally adjacent source samples for a half-sample horizontal phase. + /// + private readonly struct IntraBlockCopyHorizontalOperator : IAv1IntraBlockCopyHorizontalOperator + { + /// + public static byte Filter(byte left, byte right) => (byte)((left + right + 1) >> 1); + + /// + public static Vector128 Filter(Vector128 left, Vector128 right) + => AverageRounded(left, right); + + /// + public static Vector256 Filter(Vector256 left, Vector256 right) + => AverageRounded(left, right); + + /// + public static Vector512 Filter(Vector512 left, Vector512 right) + => AverageRounded(left, right); + + /// + public static short Filter(short left, short right) => (short)((left + right + 1) >> 1); + + /// + public static Vector128 Filter(Vector128 left, Vector128 right) + => AverageRounded(left, right); + + /// + public static Vector256 Filter(Vector256 left, Vector256 right) + => AverageRounded(left, right); + + /// + public static Vector512 Filter(Vector512 left, Vector512 right) + => AverageRounded(left, right); + + /// + /// Computes a rounded average without overflowing unsigned byte lanes. + /// + private static Vector128 AverageRounded(Vector128 left, Vector128 right) + => (left | right) - ((left ^ right) >> 1); + + /// + /// Computes a rounded average without overflowing unsigned byte lanes. + /// + private static Vector256 AverageRounded(Vector256 left, Vector256 right) + => (left | right) - ((left ^ right) >> 1); + + /// + /// Computes a rounded average without overflowing unsigned byte lanes. + /// + private static Vector512 AverageRounded(Vector512 left, Vector512 right) + => (left | right) - ((left ^ right) >> 1); + + /// + /// Computes a rounded average without overflowing nonnegative high-bit-depth lanes. + /// + private static Vector128 AverageRounded(Vector128 left, Vector128 right) + { + Vector128 leftUnsigned = left.AsUInt16(); + Vector128 rightUnsigned = right.AsUInt16(); + + // This identity computes ceil((a + b) / 2) without an overflowing lane-wise addition. + return ((leftUnsigned | rightUnsigned) - ((leftUnsigned ^ rightUnsigned) >> 1)).AsInt16(); + } + + /// + /// Computes a rounded average without overflowing nonnegative high-bit-depth lanes. + /// + private static Vector256 AverageRounded(Vector256 left, Vector256 right) + { + Vector256 leftUnsigned = left.AsUInt16(); + Vector256 rightUnsigned = right.AsUInt16(); + return ((leftUnsigned | rightUnsigned) - ((leftUnsigned ^ rightUnsigned) >> 1)).AsInt16(); + } + + /// + /// Computes a rounded average without overflowing nonnegative high-bit-depth lanes. + /// + private static Vector512 AverageRounded(Vector512 left, Vector512 right) + { + Vector512 leftUnsigned = left.AsUInt16(); + Vector512 rightUnsigned = right.AsUInt16(); + return ((leftUnsigned | rightUnsigned) - ((leftUnsigned ^ rightUnsigned) >> 1)).AsInt16(); + } + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyHorizontalPredictor.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyHorizontalPredictor.cs new file mode 100644 index 000000000..437f11074 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyHorizontalPredictor.cs @@ -0,0 +1,370 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using System.Runtime.Intrinsics; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.IntraBlockCopy; + +/// +/// Provides the family-owned scalar and width-progressive SIMD traversal for horizontal intra-block-copy prediction. +/// +internal static partial class Av1IntraBlockCopyHorizontalPredictor +{ + /// + /// Reconstructs an 8-bit filtered intra-block-copy prediction. + /// + public static void Predict( + ReadOnlySpan source, + int sourceStride, + Span destination, + int destinationStride, + int width, + int height) + => Predict(source, sourceStride, destination, destinationStride, width, height); + + /// + /// Reconstructs a high-bit-depth filtered intra-block-copy prediction. + /// + public static void Predict( + ReadOnlySpan source, + int sourceStride, + Span destination, + int destinationStride, + int width, + int height) + => Predict(source, sourceStride, destination, destinationStride, width, height); + + /// + /// Reconstructs an 8-bit filtered intra-block-copy prediction without explicit hardware intrinsics. + /// + public static void PredictScalar( + ReadOnlySpan source, + int sourceStride, + Span destination, + int destinationStride, + int width, + int height) + => PredictScalar(source, sourceStride, destination, destinationStride, width, height); + + /// + /// Reconstructs a high-bit-depth filtered intra-block-copy prediction without explicit hardware intrinsics. + /// + public static void PredictScalar( + ReadOnlySpan source, + int sourceStride, + Span destination, + int destinationStride, + int width, + int height) + => PredictScalar(source, sourceStride, destination, destinationStride, width, height); + + /// + /// Applies one closed interpolation operator to an 8-bit source block. + /// + /// The source-phase-specific interpolation arithmetic. + private static void Predict( + ReadOnlySpan source, + int sourceStride, + Span destination, + int destinationStride, + int width, + int height) + where TOperator : struct, IAv1IntraBlockCopyHorizontalOperator + { + ref byte sourceBase = ref MemoryMarshal.GetReference(source); + ref byte destinationBase = ref MemoryMarshal.GetReference(destination); + + if (Vector128.IsHardwareAccelerated && width is 4 or 8) + { + // AV1 permits 4- and 8-sample transform widths, both smaller than a byte Vector128. The frame allocation's + // 72-sample prediction border makes each full source load readable; exact-width stores avoid touching + // destination padding. + for (int row = 0; row < height; row++) + { + ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride); + ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); + Vector128 topLeft = Vector128.LoadUnsafe(ref sourceRow); + Vector128 topRight = Vector128.LoadUnsafe(ref sourceRow, 1); + + Vector128 prediction = TOperator.Filter(topLeft, topRight); + if (width == 8) + { + prediction.GetLower().StoreUnsafe(ref destinationRow); + } + else + { + Unsafe.As(ref destinationRow) = prediction.AsUInt32().GetElement(0); + } + } + + return; + } + + int processedColumns = 0; + + // AV1 transform widths are powers of two. The widest supported tier normally consumes the complete row; the + // cumulative narrower tiers preserve the same contract for future legal widths without over-reading a tail. + if (Vector512.IsHardwareAccelerated) + { + int vectorizedColumns = width - (width % Vector512.Count); + if (vectorizedColumns > 0) + { + for (int row = 0; row < height; row++) + { + ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride); + ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); + + for (int column = 0; column < vectorizedColumns; column += Vector512.Count) + { + Vector512 topLeft = Vector512.LoadUnsafe(ref sourceRow, (nuint)column); + Vector512 topRight = Vector512.LoadUnsafe(ref sourceRow, (nuint)(column + 1)); + + TOperator.Filter(topLeft, topRight).StoreUnsafe(ref destinationRow, (nuint)column); + } + } + + processedColumns = vectorizedColumns; + } + } + + if (Vector256.IsHardwareAccelerated) + { + int remainingColumns = width - processedColumns; + int vectorizedColumns = remainingColumns - (remainingColumns % Vector256.Count); + int endColumn = processedColumns + vectorizedColumns; + + for (int row = 0; row < height; row++) + { + ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride); + ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); + + for (int column = processedColumns; column < endColumn; column += Vector256.Count) + { + Vector256 topLeft = Vector256.LoadUnsafe(ref sourceRow, (nuint)column); + Vector256 topRight = Vector256.LoadUnsafe(ref sourceRow, (nuint)(column + 1)); + + TOperator.Filter(topLeft, topRight).StoreUnsafe(ref destinationRow, (nuint)column); + } + } + + processedColumns = endColumn; + } + + if (Vector128.IsHardwareAccelerated) + { + int remainingColumns = width - processedColumns; + int vectorizedColumns = remainingColumns - (remainingColumns % Vector128.Count); + int endColumn = processedColumns + vectorizedColumns; + + for (int row = 0; row < height; row++) + { + ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride); + ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); + + for (int column = processedColumns; column < endColumn; column += Vector128.Count) + { + Vector128 topLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)column); + Vector128 topRight = Vector128.LoadUnsafe(ref sourceRow, (nuint)(column + 1)); + + TOperator.Filter(topLeft, topRight).StoreUnsafe(ref destinationRow, (nuint)column); + } + } + + processedColumns = endColumn; + } + + // FeatureTestRunner can disable every intrinsic tier. Keeping the scalar continuation in the same traversal + // proves the fallback without changing source addressing or the normative rounding performed by the operator. + for (int row = 0; row < height; row++) + { + ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride); + ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); + + for (int column = processedColumns; column < width; column++) + { + byte topLeft = Unsafe.Add(ref sourceRow, column); + byte topRight = Unsafe.Add(ref sourceRow, column + 1); + + Unsafe.Add(ref destinationRow, column) = TOperator.Filter(topLeft, topRight); + } + } + } + + /// + /// Applies one closed interpolation operator to a high-bit-depth source block. + /// + /// The source-phase-specific interpolation arithmetic. + private static void Predict( + ReadOnlySpan source, + int sourceStride, + Span destination, + int destinationStride, + int width, + int height) + where TOperator : struct, IAv1IntraBlockCopyHorizontalOperator + { + ref short sourceBase = ref MemoryMarshal.GetReference(source); + ref short destinationBase = ref MemoryMarshal.GetReference(destination); + + if (Vector128.IsHardwareAccelerated && width == 4) + { + // Four high-bit-depth samples occupy the lower half of a Vector128. The frame allocation's prediction + // border makes the full source load readable; storing only the lower four lanes avoids relying on writable + // samples beyond the transform boundary. + for (int row = 0; row < height; row++) + { + ref short sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride); + ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); + Vector128 topLeft = Vector128.LoadUnsafe(ref sourceRow); + Vector128 topRight = Vector128.LoadUnsafe(ref sourceRow, 1); + + TOperator.Filter(topLeft, topRight).GetLower().StoreUnsafe(ref destinationRow); + } + + return; + } + + int processedColumns = 0; + + // High-bit-depth lanes hold half as many samples, but retain the same descending-width traversal and one scalar + // continuation as the byte path. + if (Vector512.IsHardwareAccelerated) + { + int vectorizedColumns = width - (width % Vector512.Count); + if (vectorizedColumns > 0) + { + for (int row = 0; row < height; row++) + { + ref short sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride); + ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); + + for (int column = 0; column < vectorizedColumns; column += Vector512.Count) + { + Vector512 topLeft = Vector512.LoadUnsafe(ref sourceRow, (nuint)column); + Vector512 topRight = Vector512.LoadUnsafe(ref sourceRow, (nuint)(column + 1)); + + TOperator.Filter(topLeft, topRight).StoreUnsafe(ref destinationRow, (nuint)column); + } + } + + processedColumns = vectorizedColumns; + } + } + + if (Vector256.IsHardwareAccelerated) + { + int remainingColumns = width - processedColumns; + int vectorizedColumns = remainingColumns - (remainingColumns % Vector256.Count); + int endColumn = processedColumns + vectorizedColumns; + + for (int row = 0; row < height; row++) + { + ref short sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride); + ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); + + for (int column = processedColumns; column < endColumn; column += Vector256.Count) + { + Vector256 topLeft = Vector256.LoadUnsafe(ref sourceRow, (nuint)column); + Vector256 topRight = Vector256.LoadUnsafe(ref sourceRow, (nuint)(column + 1)); + + TOperator.Filter(topLeft, topRight).StoreUnsafe(ref destinationRow, (nuint)column); + } + } + + processedColumns = endColumn; + } + + if (Vector128.IsHardwareAccelerated) + { + int remainingColumns = width - processedColumns; + int vectorizedColumns = remainingColumns - (remainingColumns % Vector128.Count); + int endColumn = processedColumns + vectorizedColumns; + + for (int row = 0; row < height; row++) + { + ref short sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride); + ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); + + for (int column = processedColumns; column < endColumn; column += Vector128.Count) + { + Vector128 topLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)column); + Vector128 topRight = Vector128.LoadUnsafe(ref sourceRow, (nuint)(column + 1)); + + TOperator.Filter(topLeft, topRight).StoreUnsafe(ref destinationRow, (nuint)column); + } + } + + processedColumns = endColumn; + } + + for (int row = 0; row < height; row++) + { + ref short sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride); + ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); + + for (int column = processedColumns; column < width; column++) + { + short topLeft = Unsafe.Add(ref sourceRow, column); + short topRight = Unsafe.Add(ref sourceRow, column + 1); + + Unsafe.Add(ref destinationRow, column) = TOperator.Filter(topLeft, topRight); + } + } + } + + /// + /// Applies one closed interpolation operator to an 8-bit source block without explicit hardware intrinsics. + /// + /// The source-phase-specific interpolation arithmetic. + private static void PredictScalar( + ReadOnlySpan source, + int sourceStride, + Span destination, + int destinationStride, + int width, + int height) + where TOperator : struct, IAv1IntraBlockCopyHorizontalOperator + { + for (int row = 0; row < height; row++) + { + int sourceRow = row * sourceStride; + int destinationRow = row * destinationStride; + + for (int column = 0; column < width; column++) + { + byte topLeft = source[sourceRow + column]; + byte topRight = source[sourceRow + column + 1]; + destination[destinationRow + column] = TOperator.Filter(topLeft, topRight); + } + } + } + + /// + /// Applies one closed interpolation operator to a high-bit-depth source block without explicit hardware intrinsics. + /// + /// The source-phase-specific interpolation arithmetic. + private static void PredictScalar( + ReadOnlySpan source, + int sourceStride, + Span destination, + int destinationStride, + int width, + int height) + where TOperator : struct, IAv1IntraBlockCopyHorizontalOperator + { + for (int row = 0; row < height; row++) + { + int sourceRow = row * sourceStride; + int destinationRow = row * destinationStride; + + for (int column = 0; column < width; column++) + { + short topLeft = source[sourceRow + column]; + short topRight = source[sourceRow + column + 1]; + destination[destinationRow + column] = TOperator.Filter(topLeft, topRight); + } + } + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyPredictor.Arithmetic.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyPredictor.Arithmetic.cs deleted file mode 100644 index 222df8a33..000000000 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyPredictor.Arithmetic.cs +++ /dev/null @@ -1,80 +0,0 @@ -// Copyright (c) Six Labors. -// Licensed under the Six Labors Split License. - -using System.Runtime.Intrinsics; - -namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.IntraBlockCopy; - -/// -/// Provides the overflow-free rounded-average arithmetic shared by the interpolation operators. -/// -internal static partial class Av1IntraBlockCopyPredictor -{ - /// - /// Computes the AV1 rounded average of two unsigned 8-bit vectors without widening their lanes. - /// - /// The first source vector. - /// The second source vector. - /// The lane-wise rounded averages. - private static Vector128 AverageRounded(Vector128 left, Vector128 right) - => (left | right) - ((left ^ right) >> 1); - - /// - /// Computes the AV1 rounded average of two unsigned 8-bit vectors without widening their lanes. - /// - /// The first source vector. - /// The second source vector. - /// The lane-wise rounded averages. - private static Vector256 AverageRounded(Vector256 left, Vector256 right) - => (left | right) - ((left ^ right) >> 1); - - /// - /// Computes the AV1 rounded average of two unsigned 8-bit vectors without widening their lanes. - /// - /// The first source vector. - /// The second source vector. - /// The lane-wise rounded averages. - private static Vector512 AverageRounded(Vector512 left, Vector512 right) - => (left | right) - ((left ^ right) >> 1); - - /// - /// Computes the AV1 rounded average of two nonnegative high-bit-depth vectors without widening their lanes. - /// - /// The first source vector. - /// The second source vector. - /// The lane-wise rounded averages. - private static Vector128 AverageRounded(Vector128 left, Vector128 right) - { - Vector128 leftUnsigned = left.AsUInt16(); - Vector128 rightUnsigned = right.AsUInt16(); - - // (a | b) - ((a ^ b) >> 1) is ceil((a + b) / 2) without an overflowing lane-wise addition. - return ((leftUnsigned | rightUnsigned) - ((leftUnsigned ^ rightUnsigned) >> 1)).AsInt16(); - } - - /// - /// Computes the AV1 rounded average of two nonnegative high-bit-depth vectors without widening their lanes. - /// - /// The first source vector. - /// The second source vector. - /// The lane-wise rounded averages. - private static Vector256 AverageRounded(Vector256 left, Vector256 right) - { - Vector256 leftUnsigned = left.AsUInt16(); - Vector256 rightUnsigned = right.AsUInt16(); - return ((leftUnsigned | rightUnsigned) - ((leftUnsigned ^ rightUnsigned) >> 1)).AsInt16(); - } - - /// - /// Computes the AV1 rounded average of two nonnegative high-bit-depth vectors without widening their lanes. - /// - /// The first source vector. - /// The second source vector. - /// The lane-wise rounded averages. - private static Vector512 AverageRounded(Vector512 left, Vector512 right) - { - Vector512 leftUnsigned = left.AsUInt16(); - Vector512 rightUnsigned = right.AsUInt16(); - return ((leftUnsigned | rightUnsigned) - ((leftUnsigned ^ rightUnsigned) >> 1)).AsInt16(); - } -} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyPredictor.HorizontalOperator.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyPredictor.HorizontalOperator.cs deleted file mode 100644 index ae9d2994c..000000000 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyPredictor.HorizontalOperator.cs +++ /dev/null @@ -1,78 +0,0 @@ -// Copyright (c) Six Labors. -// Licensed under the Six Labors Split License. - -using System.Runtime.Intrinsics; - -namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.IntraBlockCopy; - -/// -/// Defines the closed interpolation operators used by intra-block-copy prediction. -/// -internal static partial class Av1IntraBlockCopyPredictor -{ - /// - /// Averages horizontally adjacent source samples for a half-sample horizontal phase. - /// - private readonly struct HorizontalOperator : IAv1IntraBlockCopyOperator - { - /// - public static bool UsesRight => true; - - /// - public static bool UsesBottom => false; - - /// - public static byte Filter(byte topLeft, byte topRight, byte bottomLeft, byte bottomRight) => (byte)((topLeft + topRight + 1) >> 1); - - /// - public static Vector128 Filter( - Vector128 topLeft, - Vector128 topRight, - Vector128 bottomLeft, - Vector128 bottomRight) - => AverageRounded(topLeft, topRight); - - /// - public static Vector256 Filter( - Vector256 topLeft, - Vector256 topRight, - Vector256 bottomLeft, - Vector256 bottomRight) - => AverageRounded(topLeft, topRight); - - /// - public static Vector512 Filter( - Vector512 topLeft, - Vector512 topRight, - Vector512 bottomLeft, - Vector512 bottomRight) - => AverageRounded(topLeft, topRight); - - /// - public static short Filter(short topLeft, short topRight, short bottomLeft, short bottomRight) => (short)((topLeft + topRight + 1) >> 1); - - /// - public static Vector128 Filter( - Vector128 topLeft, - Vector128 topRight, - Vector128 bottomLeft, - Vector128 bottomRight) - => AverageRounded(topLeft, topRight); - - /// - public static Vector256 Filter( - Vector256 topLeft, - Vector256 topRight, - Vector256 bottomLeft, - Vector256 bottomRight) - => AverageRounded(topLeft, topRight); - - /// - public static Vector512 Filter( - Vector512 topLeft, - Vector512 topRight, - Vector512 bottomLeft, - Vector512 bottomRight) - => AverageRounded(topLeft, topRight); - } -} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyPredictor.VerticalOperator.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyPredictor.VerticalOperator.cs deleted file mode 100644 index 27088106a..000000000 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyPredictor.VerticalOperator.cs +++ /dev/null @@ -1,75 +0,0 @@ -// Copyright (c) Six Labors. -// Licensed under the Six Labors Split License. - -using System.Runtime.Intrinsics; - -namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.IntraBlockCopy; - -internal static partial class Av1IntraBlockCopyPredictor -{ - /// - /// Averages vertically adjacent source samples for a half-sample vertical phase. - /// - private readonly struct VerticalOperator : IAv1IntraBlockCopyOperator - { - /// - public static bool UsesRight => false; - - /// - public static bool UsesBottom => true; - - /// - public static byte Filter(byte topLeft, byte topRight, byte bottomLeft, byte bottomRight) => (byte)((topLeft + bottomLeft + 1) >> 1); - - /// - public static Vector128 Filter( - Vector128 topLeft, - Vector128 topRight, - Vector128 bottomLeft, - Vector128 bottomRight) - => AverageRounded(topLeft, bottomLeft); - - /// - public static Vector256 Filter( - Vector256 topLeft, - Vector256 topRight, - Vector256 bottomLeft, - Vector256 bottomRight) - => AverageRounded(topLeft, bottomLeft); - - /// - public static Vector512 Filter( - Vector512 topLeft, - Vector512 topRight, - Vector512 bottomLeft, - Vector512 bottomRight) - => AverageRounded(topLeft, bottomLeft); - - /// - public static short Filter(short topLeft, short topRight, short bottomLeft, short bottomRight) => (short)((topLeft + bottomLeft + 1) >> 1); - - /// - public static Vector128 Filter( - Vector128 topLeft, - Vector128 topRight, - Vector128 bottomLeft, - Vector128 bottomRight) - => AverageRounded(topLeft, bottomLeft); - - /// - public static Vector256 Filter( - Vector256 topLeft, - Vector256 topRight, - Vector256 bottomLeft, - Vector256 bottomRight) - => AverageRounded(topLeft, bottomLeft); - - /// - public static Vector512 Filter( - Vector512 topLeft, - Vector512 topRight, - Vector512 bottomLeft, - Vector512 bottomRight) - => AverageRounded(topLeft, bottomLeft); - } -} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyPredictor.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyPredictor.cs index 7dbdbde4a..8e75b3573 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyPredictor.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyPredictor.cs @@ -42,15 +42,15 @@ internal static partial class Av1IntraBlockCopyPredictor } else if (halfX && halfY) { - Predict(source, sourceStride, destination, destinationStride, width, height); + Av1IntraBlockCopyBilinearPredictor.Predict(source, sourceStride, destination, destinationStride, width, height); } else if (halfX) { - Predict(source, sourceStride, destination, destinationStride, width, height); + Av1IntraBlockCopyHorizontalPredictor.Predict(source, sourceStride, destination, destinationStride, width, height); } else { - Predict(source, sourceStride, destination, destinationStride, width, height); + Av1IntraBlockCopyVerticalPredictor.Predict(source, sourceStride, destination, destinationStride, width, height); } } @@ -81,15 +81,15 @@ internal static partial class Av1IntraBlockCopyPredictor } else if (halfX && halfY) { - Predict(source, sourceStride, destination, destinationStride, width, height); + Av1IntraBlockCopyBilinearPredictor.Predict(source, sourceStride, destination, destinationStride, width, height); } else if (halfX) { - Predict(source, sourceStride, destination, destinationStride, width, height); + Av1IntraBlockCopyHorizontalPredictor.Predict(source, sourceStride, destination, destinationStride, width, height); } else { - Predict(source, sourceStride, destination, destinationStride, width, height); + Av1IntraBlockCopyVerticalPredictor.Predict(source, sourceStride, destination, destinationStride, width, height); } } @@ -120,15 +120,15 @@ internal static partial class Av1IntraBlockCopyPredictor } else if (halfX && halfY) { - PredictScalar(source, sourceStride, destination, destinationStride, width, height); + Av1IntraBlockCopyBilinearPredictor.PredictScalar(source, sourceStride, destination, destinationStride, width, height); } else if (halfX) { - PredictScalar(source, sourceStride, destination, destinationStride, width, height); + Av1IntraBlockCopyHorizontalPredictor.PredictScalar(source, sourceStride, destination, destinationStride, width, height); } else { - PredictScalar(source, sourceStride, destination, destinationStride, width, height); + Av1IntraBlockCopyVerticalPredictor.PredictScalar(source, sourceStride, destination, destinationStride, width, height); } } @@ -159,15 +159,15 @@ internal static partial class Av1IntraBlockCopyPredictor } else if (halfX && halfY) { - PredictScalar(source, sourceStride, destination, destinationStride, width, height); + Av1IntraBlockCopyBilinearPredictor.PredictScalar(source, sourceStride, destination, destinationStride, width, height); } else if (halfX) { - PredictScalar(source, sourceStride, destination, destinationStride, width, height); + Av1IntraBlockCopyHorizontalPredictor.PredictScalar(source, sourceStride, destination, destinationStride, width, height); } else { - PredictScalar(source, sourceStride, destination, destinationStride, width, height); + Av1IntraBlockCopyVerticalPredictor.PredictScalar(source, sourceStride, destination, destinationStride, width, height); } } diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyVerticalPredictor.Operator.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyVerticalPredictor.Operator.cs new file mode 100644 index 000000000..2757c6e39 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyVerticalPredictor.Operator.cs @@ -0,0 +1,180 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.Intrinsics; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.IntraBlockCopy; + +/// +/// Defines the closed vertical intra-block-copy interpolation operator. +/// +internal static partial class Av1IntraBlockCopyVerticalPredictor +{ + /// + /// Defines vertical intra-block-copy filtering for scalar and SIMD lane groups. + /// + private interface IAv1IntraBlockCopyVerticalOperator + { + /// + /// Filters one 8-bit sample. + /// + /// The integer-position source sample. + /// The source sample one row below. + /// The filtered 8-bit sample. + public static abstract byte Filter(byte top, byte bottom); + + /// + /// Filters sixteen 8-bit samples in parallel. + /// + /// The integer-position source samples. + /// The source samples one row below. + /// The filtered 8-bit samples. + public static abstract Vector128 Filter( + Vector128 top, + Vector128 bottom); + + /// + /// Filters thirty-two 8-bit samples in parallel. + /// + /// The integer-position source samples. + /// The source samples one row below. + /// The filtered 8-bit samples. + public static abstract Vector256 Filter( + Vector256 top, + Vector256 bottom); + + /// + /// Filters sixty-four 8-bit samples in parallel. + /// + /// The integer-position source samples. + /// The source samples one row below. + /// The filtered 8-bit samples. + public static abstract Vector512 Filter( + Vector512 top, + Vector512 bottom); + + /// + /// Filters one high-bit-depth sample. + /// + /// The integer-position source sample. + /// The source sample one row below. + /// The filtered high-bit-depth sample. + public static abstract short Filter(short top, short bottom); + + /// + /// Filters eight high-bit-depth samples in parallel. + /// + /// The integer-position source samples. + /// The source samples one row below. + /// The filtered high-bit-depth samples. + public static abstract Vector128 Filter( + Vector128 top, + Vector128 bottom); + + /// + /// Filters sixteen high-bit-depth samples in parallel. + /// + /// The integer-position source samples. + /// The source samples one row below. + /// The filtered high-bit-depth samples. + public static abstract Vector256 Filter( + Vector256 top, + Vector256 bottom); + + /// + /// Filters thirty-two high-bit-depth samples in parallel. + /// + /// The integer-position source samples. + /// The source samples one row below. + /// The filtered high-bit-depth samples. + public static abstract Vector512 Filter( + Vector512 top, + Vector512 bottom); + } + + /// + /// Averages vertically adjacent source samples for a half-sample vertical phase. + /// + private readonly struct IntraBlockCopyVerticalOperator : IAv1IntraBlockCopyVerticalOperator + { + /// + public static byte Filter(byte top, byte bottom) => (byte)((top + bottom + 1) >> 1); + + /// + public static Vector128 Filter(Vector128 top, Vector128 bottom) + => AverageRounded(top, bottom); + + /// + public static Vector256 Filter(Vector256 top, Vector256 bottom) + => AverageRounded(top, bottom); + + /// + public static Vector512 Filter(Vector512 top, Vector512 bottom) + => AverageRounded(top, bottom); + + /// + public static short Filter(short top, short bottom) => (short)((top + bottom + 1) >> 1); + + /// + public static Vector128 Filter(Vector128 top, Vector128 bottom) + => AverageRounded(top, bottom); + + /// + public static Vector256 Filter(Vector256 top, Vector256 bottom) + => AverageRounded(top, bottom); + + /// + public static Vector512 Filter(Vector512 top, Vector512 bottom) + => AverageRounded(top, bottom); + + /// + /// Computes a rounded average without overflowing unsigned byte lanes. + /// + private static Vector128 AverageRounded(Vector128 left, Vector128 right) + => (left | right) - ((left ^ right) >> 1); + + /// + /// Computes a rounded average without overflowing unsigned byte lanes. + /// + private static Vector256 AverageRounded(Vector256 left, Vector256 right) + => (left | right) - ((left ^ right) >> 1); + + /// + /// Computes a rounded average without overflowing unsigned byte lanes. + /// + private static Vector512 AverageRounded(Vector512 left, Vector512 right) + => (left | right) - ((left ^ right) >> 1); + + /// + /// Computes a rounded average without overflowing nonnegative high-bit-depth lanes. + /// + private static Vector128 AverageRounded(Vector128 left, Vector128 right) + { + Vector128 leftUnsigned = left.AsUInt16(); + Vector128 rightUnsigned = right.AsUInt16(); + + // This identity computes ceil((a + b) / 2) without an overflowing lane-wise addition. + return ((leftUnsigned | rightUnsigned) - ((leftUnsigned ^ rightUnsigned) >> 1)).AsInt16(); + } + + /// + /// Computes a rounded average without overflowing nonnegative high-bit-depth lanes. + /// + private static Vector256 AverageRounded(Vector256 left, Vector256 right) + { + Vector256 leftUnsigned = left.AsUInt16(); + Vector256 rightUnsigned = right.AsUInt16(); + return ((leftUnsigned | rightUnsigned) - ((leftUnsigned ^ rightUnsigned) >> 1)).AsInt16(); + } + + /// + /// Computes a rounded average without overflowing nonnegative high-bit-depth lanes. + /// + private static Vector512 AverageRounded(Vector512 left, Vector512 right) + { + Vector512 leftUnsigned = left.AsUInt16(); + Vector512 rightUnsigned = right.AsUInt16(); + return ((leftUnsigned | rightUnsigned) - ((leftUnsigned ^ rightUnsigned) >> 1)).AsInt16(); + } + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyVerticalPredictor.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyVerticalPredictor.cs new file mode 100644 index 000000000..565020766 --- /dev/null +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/IntraBlockCopy/Av1IntraBlockCopyVerticalPredictor.cs @@ -0,0 +1,370 @@ +// Copyright (c) Six Labors. +// Licensed under the Six Labors Split License. + +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using System.Runtime.Intrinsics; + +namespace SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.IntraBlockCopy; + +/// +/// Provides the family-owned scalar and width-progressive SIMD traversal for vertical intra-block-copy prediction. +/// +internal static partial class Av1IntraBlockCopyVerticalPredictor +{ + /// + /// Reconstructs an 8-bit filtered intra-block-copy prediction. + /// + public static void Predict( + ReadOnlySpan source, + int sourceStride, + Span destination, + int destinationStride, + int width, + int height) + => Predict(source, sourceStride, destination, destinationStride, width, height); + + /// + /// Reconstructs a high-bit-depth filtered intra-block-copy prediction. + /// + public static void Predict( + ReadOnlySpan source, + int sourceStride, + Span destination, + int destinationStride, + int width, + int height) + => Predict(source, sourceStride, destination, destinationStride, width, height); + + /// + /// Reconstructs an 8-bit filtered intra-block-copy prediction without explicit hardware intrinsics. + /// + public static void PredictScalar( + ReadOnlySpan source, + int sourceStride, + Span destination, + int destinationStride, + int width, + int height) + => PredictScalar(source, sourceStride, destination, destinationStride, width, height); + + /// + /// Reconstructs a high-bit-depth filtered intra-block-copy prediction without explicit hardware intrinsics. + /// + public static void PredictScalar( + ReadOnlySpan source, + int sourceStride, + Span destination, + int destinationStride, + int width, + int height) + => PredictScalar(source, sourceStride, destination, destinationStride, width, height); + + /// + /// Applies one closed interpolation operator to an 8-bit source block. + /// + /// The source-phase-specific interpolation arithmetic. + private static void Predict( + ReadOnlySpan source, + int sourceStride, + Span destination, + int destinationStride, + int width, + int height) + where TOperator : struct, IAv1IntraBlockCopyVerticalOperator + { + ref byte sourceBase = ref MemoryMarshal.GetReference(source); + ref byte destinationBase = ref MemoryMarshal.GetReference(destination); + + if (Vector128.IsHardwareAccelerated && width is 4 or 8) + { + // AV1 permits 4- and 8-sample transform widths, both smaller than a byte Vector128. The frame allocation's + // 72-sample prediction border makes each full source load readable; exact-width stores avoid touching + // destination padding. + for (int row = 0; row < height; row++) + { + ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride); + ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); + Vector128 topLeft = Vector128.LoadUnsafe(ref sourceRow); + Vector128 bottomLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)sourceStride); + + Vector128 prediction = TOperator.Filter(topLeft, bottomLeft); + if (width == 8) + { + prediction.GetLower().StoreUnsafe(ref destinationRow); + } + else + { + Unsafe.As(ref destinationRow) = prediction.AsUInt32().GetElement(0); + } + } + + return; + } + + int processedColumns = 0; + + // AV1 transform widths are powers of two. The widest supported tier normally consumes the complete row; the + // cumulative narrower tiers preserve the same contract for future legal widths without over-reading a tail. + if (Vector512.IsHardwareAccelerated) + { + int vectorizedColumns = width - (width % Vector512.Count); + if (vectorizedColumns > 0) + { + for (int row = 0; row < height; row++) + { + ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride); + ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); + + for (int column = 0; column < vectorizedColumns; column += Vector512.Count) + { + Vector512 topLeft = Vector512.LoadUnsafe(ref sourceRow, (nuint)column); + Vector512 bottomLeft = Vector512.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column)); + + TOperator.Filter(topLeft, bottomLeft).StoreUnsafe(ref destinationRow, (nuint)column); + } + } + + processedColumns = vectorizedColumns; + } + } + + if (Vector256.IsHardwareAccelerated) + { + int remainingColumns = width - processedColumns; + int vectorizedColumns = remainingColumns - (remainingColumns % Vector256.Count); + int endColumn = processedColumns + vectorizedColumns; + + for (int row = 0; row < height; row++) + { + ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride); + ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); + + for (int column = processedColumns; column < endColumn; column += Vector256.Count) + { + Vector256 topLeft = Vector256.LoadUnsafe(ref sourceRow, (nuint)column); + Vector256 bottomLeft = Vector256.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column)); + + TOperator.Filter(topLeft, bottomLeft).StoreUnsafe(ref destinationRow, (nuint)column); + } + } + + processedColumns = endColumn; + } + + if (Vector128.IsHardwareAccelerated) + { + int remainingColumns = width - processedColumns; + int vectorizedColumns = remainingColumns - (remainingColumns % Vector128.Count); + int endColumn = processedColumns + vectorizedColumns; + + for (int row = 0; row < height; row++) + { + ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride); + ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); + + for (int column = processedColumns; column < endColumn; column += Vector128.Count) + { + Vector128 topLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)column); + Vector128 bottomLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column)); + + TOperator.Filter(topLeft, bottomLeft).StoreUnsafe(ref destinationRow, (nuint)column); + } + } + + processedColumns = endColumn; + } + + // FeatureTestRunner can disable every intrinsic tier. Keeping the scalar continuation in the same traversal + // proves the fallback without changing source addressing or the normative rounding performed by the operator. + for (int row = 0; row < height; row++) + { + ref byte sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride); + ref byte destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); + + for (int column = processedColumns; column < width; column++) + { + byte topLeft = Unsafe.Add(ref sourceRow, column); + byte bottomLeft = Unsafe.Add(ref sourceRow, sourceStride + column); + + Unsafe.Add(ref destinationRow, column) = TOperator.Filter(topLeft, bottomLeft); + } + } + } + + /// + /// Applies one closed interpolation operator to a high-bit-depth source block. + /// + /// The source-phase-specific interpolation arithmetic. + private static void Predict( + ReadOnlySpan source, + int sourceStride, + Span destination, + int destinationStride, + int width, + int height) + where TOperator : struct, IAv1IntraBlockCopyVerticalOperator + { + ref short sourceBase = ref MemoryMarshal.GetReference(source); + ref short destinationBase = ref MemoryMarshal.GetReference(destination); + + if (Vector128.IsHardwareAccelerated && width == 4) + { + // Four high-bit-depth samples occupy the lower half of a Vector128. The frame allocation's prediction + // border makes the full source load readable; storing only the lower four lanes avoids relying on writable + // samples beyond the transform boundary. + for (int row = 0; row < height; row++) + { + ref short sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride); + ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); + Vector128 topLeft = Vector128.LoadUnsafe(ref sourceRow); + Vector128 bottomLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)sourceStride); + + TOperator.Filter(topLeft, bottomLeft).GetLower().StoreUnsafe(ref destinationRow); + } + + return; + } + + int processedColumns = 0; + + // High-bit-depth lanes hold half as many samples, but retain the same descending-width traversal and one scalar + // continuation as the byte path. + if (Vector512.IsHardwareAccelerated) + { + int vectorizedColumns = width - (width % Vector512.Count); + if (vectorizedColumns > 0) + { + for (int row = 0; row < height; row++) + { + ref short sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride); + ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); + + for (int column = 0; column < vectorizedColumns; column += Vector512.Count) + { + Vector512 topLeft = Vector512.LoadUnsafe(ref sourceRow, (nuint)column); + Vector512 bottomLeft = Vector512.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column)); + + TOperator.Filter(topLeft, bottomLeft).StoreUnsafe(ref destinationRow, (nuint)column); + } + } + + processedColumns = vectorizedColumns; + } + } + + if (Vector256.IsHardwareAccelerated) + { + int remainingColumns = width - processedColumns; + int vectorizedColumns = remainingColumns - (remainingColumns % Vector256.Count); + int endColumn = processedColumns + vectorizedColumns; + + for (int row = 0; row < height; row++) + { + ref short sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride); + ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); + + for (int column = processedColumns; column < endColumn; column += Vector256.Count) + { + Vector256 topLeft = Vector256.LoadUnsafe(ref sourceRow, (nuint)column); + Vector256 bottomLeft = Vector256.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column)); + + TOperator.Filter(topLeft, bottomLeft).StoreUnsafe(ref destinationRow, (nuint)column); + } + } + + processedColumns = endColumn; + } + + if (Vector128.IsHardwareAccelerated) + { + int remainingColumns = width - processedColumns; + int vectorizedColumns = remainingColumns - (remainingColumns % Vector128.Count); + int endColumn = processedColumns + vectorizedColumns; + + for (int row = 0; row < height; row++) + { + ref short sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride); + ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); + + for (int column = processedColumns; column < endColumn; column += Vector128.Count) + { + Vector128 topLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)column); + Vector128 bottomLeft = Vector128.LoadUnsafe(ref sourceRow, (nuint)(sourceStride + column)); + + TOperator.Filter(topLeft, bottomLeft).StoreUnsafe(ref destinationRow, (nuint)column); + } + } + + processedColumns = endColumn; + } + + for (int row = 0; row < height; row++) + { + ref short sourceRow = ref Unsafe.Add(ref sourceBase, row * sourceStride); + ref short destinationRow = ref Unsafe.Add(ref destinationBase, row * destinationStride); + + for (int column = processedColumns; column < width; column++) + { + short topLeft = Unsafe.Add(ref sourceRow, column); + short bottomLeft = Unsafe.Add(ref sourceRow, sourceStride + column); + + Unsafe.Add(ref destinationRow, column) = TOperator.Filter(topLeft, bottomLeft); + } + } + } + + /// + /// Applies one closed interpolation operator to an 8-bit source block without explicit hardware intrinsics. + /// + /// The source-phase-specific interpolation arithmetic. + private static void PredictScalar( + ReadOnlySpan source, + int sourceStride, + Span destination, + int destinationStride, + int width, + int height) + where TOperator : struct, IAv1IntraBlockCopyVerticalOperator + { + for (int row = 0; row < height; row++) + { + int sourceRow = row * sourceStride; + int destinationRow = row * destinationStride; + + for (int column = 0; column < width; column++) + { + byte topLeft = source[sourceRow + column]; + byte bottomLeft = source[sourceRow + sourceStride + column]; + destination[destinationRow + column] = TOperator.Filter(topLeft, bottomLeft); + } + } + } + + /// + /// Applies one closed interpolation operator to a high-bit-depth source block without explicit hardware intrinsics. + /// + /// The source-phase-specific interpolation arithmetic. + private static void PredictScalar( + ReadOnlySpan source, + int sourceStride, + Span destination, + int destinationStride, + int width, + int height) + where TOperator : struct, IAv1IntraBlockCopyVerticalOperator + { + for (int row = 0; row < height; row++) + { + int sourceRow = row * sourceStride; + int destinationRow = row * destinationStride; + + for (int column = 0; column < width; column++) + { + short topLeft = source[sourceRow + column]; + short bottomLeft = source[sourceRow + sourceStride + column]; + destination[destinationRow + column] = TOperator.Filter(topLeft, bottomLeft); + } + } + } +} diff --git a/src/ImageSharp/Formats/Heif/Av1/Transform/Av1BlockDecoder.cs b/src/ImageSharp/Formats/Heif/Av1/Transform/Av1BlockDecoder.cs index e9abf004b..1366a8b8e 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Transform/Av1BlockDecoder.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Transform/Av1BlockDecoder.cs @@ -131,7 +131,7 @@ internal sealed class Av1BlockDecoder : IDisposable Av1PredictionDecoder.ScratchLength, Math.Max( Av1InterPredictor.GetScratchLength(maximumBlockLength, maximumBlockLength), - Av1InterPredictor.GetMaximumScaledScratchLength(maximumBlockLength, maximumBlockLength))); + Av1ScaledInterPredictor.GetMaximumScaledScratchLength(maximumBlockLength, maximumBlockLength))); int compoundMaskLength = (maximumBlockArea + 1) >> 1; @@ -444,7 +444,7 @@ internal sealed class Av1BlockDecoder : IDisposable ? highBitDepthBlockReconstructionBuffer[reconstructionStride..] : secondPredictionStorage); - Av1InterPredictor.PredictWarped( + Av1WarpedInterPredictor.PredictWarped( source, sourceStride, sourceOrigin, @@ -476,7 +476,7 @@ internal sealed class Av1BlockDecoder : IDisposable ? firstCompoundPrediction : highBitDepthSecondPrediction; - Av1InterPredictor.PredictWarpedCompound( + Av1WarpedInterPredictor.PredictWarpedCompound( source, sourceStride, sourceOrigin, @@ -498,7 +498,7 @@ internal sealed class Av1BlockDecoder : IDisposable ? blockReconstructionBuffer[reconstructionStride..] : secondPrediction; - Av1InterPredictor.PredictWarped( + Av1WarpedInterPredictor.PredictWarped( source, sourceStride, sourceOrigin, @@ -635,7 +635,7 @@ internal sealed class Av1BlockDecoder : IDisposable ? firstCompoundPrediction : highBitDepthSecondPrediction; - Av1InterPredictor.PredictCompound( + Av1CompoundInterPredictor.PredictCompound( source, sourceStride, sourceIndex, @@ -681,7 +681,7 @@ internal sealed class Av1BlockDecoder : IDisposable switch (modeInfo.CompoundType) { case Av1CompoundType.Average: - Av1CompoundInterPredictor.AverageIntermediate( + Av1CompoundIntermediateAveragePredictor.AverageIntermediate( destination, reconstructionStride, first, @@ -694,7 +694,7 @@ internal sealed class Av1BlockDecoder : IDisposable break; case Av1CompoundType.DistanceWeighted: - Av1CompoundInterPredictor.DistanceWeightedIntermediate( + Av1CompoundIntermediateDistanceWeightedPredictor.DistanceWeightedIntermediate( destination, reconstructionStride, first, @@ -719,7 +719,7 @@ internal sealed class Av1BlockDecoder : IDisposable subY, invert: false); - Av1CompoundInterPredictor.BlendIntermediate( + Av1CompoundIntermediateMaskBlendPredictor.BlendIntermediate( destination, reconstructionStride, first, @@ -739,7 +739,7 @@ internal sealed class Av1BlockDecoder : IDisposable int lumaWidth = blockSize.GetWidth(); if (plane == 0) { - Av1CompoundInterPredictor.FillDifferenceWeightedIntermediateMask( + Av1CompoundIntermediateDifferenceWeightedMaskBuilder.FillDifferenceWeightedIntermediateMask( compoundMask, lumaWidth, first, @@ -752,7 +752,7 @@ internal sealed class Av1BlockDecoder : IDisposable modeInfo.DifferenceWeightedMaskType); } - Av1CompoundInterPredictor.BlendIntermediate( + Av1CompoundIntermediateMaskBlendPredictor.BlendIntermediate( destination, reconstructionStride, first, @@ -778,7 +778,7 @@ internal sealed class Av1BlockDecoder : IDisposable switch (modeInfo.CompoundType) { case Av1CompoundType.Average: - Av1CompoundInterPredictor.Average( + Av1CompoundAveragePredictor.Average( destination, reconstructionStride, highBitDepthSecondPrediction, @@ -788,7 +788,7 @@ internal sealed class Av1BlockDecoder : IDisposable break; case Av1CompoundType.DistanceWeighted: - Av1CompoundInterPredictor.DistanceWeighted( + Av1CompoundDistanceWeightedPredictor.DistanceWeighted( destination, reconstructionStride, highBitDepthSecondPrediction, @@ -810,7 +810,7 @@ internal sealed class Av1BlockDecoder : IDisposable subY, invert: false); - Av1CompoundInterPredictor.Blend( + Av1CompoundMaskBlendPredictor.Blend( destination, reconstructionStride, highBitDepthSecondPrediction, @@ -822,7 +822,7 @@ internal sealed class Av1BlockDecoder : IDisposable break; default: - Av1CompoundInterPredictor.FillDifferenceWeightedMask( + Av1DifferenceWeightedMaskBuilder.FillDifferenceWeightedMask( compoundMask, predictionWidth, destination, @@ -834,7 +834,7 @@ internal sealed class Av1BlockDecoder : IDisposable this.frameBuffer.BitDepth.GetBitCount(), modeInfo.DifferenceWeightedMaskType); - Av1CompoundInterPredictor.Blend( + Av1CompoundMaskBlendPredictor.Blend( destination, reconstructionStride, highBitDepthSecondPrediction, @@ -853,7 +853,7 @@ internal sealed class Av1BlockDecoder : IDisposable switch (modeInfo.CompoundType) { case Av1CompoundType.Average: - Av1CompoundInterPredictor.Average( + Av1CompoundAveragePredictor.Average( destination, reconstructionStride, secondPrediction, @@ -863,7 +863,7 @@ internal sealed class Av1BlockDecoder : IDisposable break; case Av1CompoundType.DistanceWeighted: - Av1CompoundInterPredictor.DistanceWeighted( + Av1CompoundDistanceWeightedPredictor.DistanceWeighted( destination, reconstructionStride, secondPrediction, @@ -885,7 +885,7 @@ internal sealed class Av1BlockDecoder : IDisposable subY, invert: false); - Av1CompoundInterPredictor.Blend( + Av1CompoundMaskBlendPredictor.Blend( destination, reconstructionStride, secondPrediction, @@ -897,7 +897,7 @@ internal sealed class Av1BlockDecoder : IDisposable break; default: - Av1CompoundInterPredictor.FillDifferenceWeightedMask( + Av1DifferenceWeightedMaskBuilder.FillDifferenceWeightedMask( compoundMask, predictionWidth, destination, @@ -908,7 +908,7 @@ internal sealed class Av1BlockDecoder : IDisposable predictionHeight, modeInfo.DifferenceWeightedMaskType); - Av1CompoundInterPredictor.Blend( + Av1CompoundMaskBlendPredictor.Blend( destination, reconstructionStride, secondPrediction, @@ -950,7 +950,7 @@ internal sealed class Av1BlockDecoder : IDisposable } else { - Av1CompoundInterPredictor.FillInterIntraMask( + Av1InterIntraMaskBuilder.FillInterIntraMask( compoundMask, predictionWidth, predictionWidth, @@ -959,7 +959,7 @@ internal sealed class Av1BlockDecoder : IDisposable invert: true); } - Av1CompoundInterPredictor.Blend( + Av1CompoundMaskBlendPredictor.Blend( MemoryMarshal.Cast(highBitDepthBlockReconstructionBuffer[reconstructionStride..]), reconstructionStride, highBitDepthSecondPrediction, @@ -995,7 +995,7 @@ internal sealed class Av1BlockDecoder : IDisposable } else { - Av1CompoundInterPredictor.FillInterIntraMask( + Av1InterIntraMaskBuilder.FillInterIntraMask( compoundMask, predictionWidth, predictionWidth, @@ -1004,7 +1004,7 @@ internal sealed class Av1BlockDecoder : IDisposable invert: true); } - Av1CompoundInterPredictor.Blend( + Av1CompoundMaskBlendPredictor.Blend( blockReconstructionBuffer[reconstructionStride..], reconstructionStride, secondPrediction, @@ -1528,7 +1528,7 @@ internal sealed class Av1BlockDecoder : IDisposable sourceOrigin.X + (sourceColumnQ10 >> Av1ReferenceScale.SubpixelBits); - Av1InterPredictor.PredictScaled( + Av1ScaledInterPredictor.PredictScaled( source, sourceStride, sourceIndex, @@ -1559,7 +1559,7 @@ internal sealed class Av1BlockDecoder : IDisposable sourceOrigin.X + (sourceColumnQ10 >> Av1ReferenceScale.SubpixelBits); - Av1InterPredictor.PredictScaled( + Av1ScaledInterPredictor.PredictScaled( source, sourceStride, sourceIndex, @@ -1664,7 +1664,7 @@ internal sealed class Av1BlockDecoder : IDisposable if (highBitDepth) { - Av1CompoundInterPredictor.Blend( + Av1CompoundMaskBlendPredictor.Blend( MemoryMarshal.Cast(highBitDepthBlockReconstructionBuffer[reconstructionStride..])[destinationColumn..], reconstructionStride, highBitDepthNeighborPrediction, @@ -1676,7 +1676,7 @@ internal sealed class Av1BlockDecoder : IDisposable } else { - Av1CompoundInterPredictor.Blend( + Av1CompoundMaskBlendPredictor.Blend( blockReconstructionBuffer[reconstructionStride..][destinationColumn..], reconstructionStride, neighborPrediction, @@ -1742,7 +1742,7 @@ internal sealed class Av1BlockDecoder : IDisposable ReadOnlySpan horizontalMask = Av1ObmcMask.Get(overlapWidth); if (highBitDepth) { - Av1CompoundInterPredictor.Blend( + Av1CompoundMaskBlendPredictor.Blend( MemoryMarshal.Cast(highBitDepthBlockReconstructionBuffer[reconstructionStride..])[(destinationRow * reconstructionStride)..], reconstructionStride, highBitDepthNeighborPrediction, @@ -1754,7 +1754,7 @@ internal sealed class Av1BlockDecoder : IDisposable } else { - Av1CompoundInterPredictor.Blend( + Av1CompoundMaskBlendPredictor.Blend( blockReconstructionBuffer[reconstructionStride..][(destinationRow * reconstructionStride)..], reconstructionStride, neighborPrediction, diff --git a/src/ImageSharp/Formats/Heif/Av1/Transform/Av1Transform2dOperations.cs b/src/ImageSharp/Formats/Heif/Av1/Transform/Av1Transform2dOperations.cs index 954c5a86b..91717a67f 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Transform/Av1Transform2dOperations.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Transform/Av1Transform2dOperations.cs @@ -202,9 +202,9 @@ internal static class Av1Transform2dOperations Vector128 columns01 = Vector128_.UnpackLow(pair0.AsInt32(), pair1.AsInt32()); Vector128 columns23 = Vector128_.UnpackHigh(pair0.AsInt32(), pair1.AsInt32()); - Store4Int16(columns01.AsUInt64().GetElement(0), ref destination); + Store4Int16(columns01.AsUInt64().ToScalar(), ref destination); Store4Int16(columns01.AsUInt64().GetElement(1), ref Unsafe.Add(ref destination, destinationStride)); - Store4Int16(columns23.AsUInt64().GetElement(0), ref Unsafe.Add(ref destination, 2 * destinationStride)); + Store4Int16(columns23.AsUInt64().ToScalar(), ref Unsafe.Add(ref destination, 2 * destinationStride)); Store4Int16(columns23.AsUInt64().GetElement(1), ref Unsafe.Add(ref destination, 3 * destinationStride)); } diff --git a/src/ImageSharp/Formats/Heif/Components/ColorConverters/HeifSampleConversion.cs b/src/ImageSharp/Formats/Heif/Components/ColorConverters/HeifSampleConversion.cs index a511604c5..9817cad90 100644 --- a/src/ImageSharp/Formats/Heif/Components/ColorConverters/HeifSampleConversion.cs +++ b/src/ImageSharp/Formats/Heif/Components/ColorConverters/HeifSampleConversion.cs @@ -884,5 +884,4 @@ internal static class HeifSampleConversion Unsafe.As>(ref destination) = lower; Unsafe.As>(ref Unsafe.Add(ref destination, 2)) = upper; } - } diff --git a/src/ImageSharp/Formats/Heif/Components/ColorConverters/HeifYuv420ToRgb8Converter.Parameters.cs b/src/ImageSharp/Formats/Heif/Components/ColorConverters/HeifYuv420ToRgb8Converter.Parameters.cs index 3dd279c42..7584e9b00 100644 --- a/src/ImageSharp/Formats/Heif/Components/ColorConverters/HeifYuv420ToRgb8Converter.Parameters.cs +++ b/src/ImageSharp/Formats/Heif/Components/ColorConverters/HeifYuv420ToRgb8Converter.Parameters.cs @@ -2,6 +2,7 @@ // Licensed under the Six Labors Split License. using System.Runtime.Intrinsics; + namespace SixLabors.ImageSharp.Formats.Heif.Components; /// diff --git a/tests/ImageSharp.Benchmarks/Codecs/Heif/Av1ChromaFromLumaBenchmarks.cs b/tests/ImageSharp.Benchmarks/Codecs/Heif/Av1ChromaFromLumaBenchmarks.cs index ea40b3144..c1b621cef 100644 --- a/tests/ImageSharp.Benchmarks/Codecs/Heif/Av1ChromaFromLumaBenchmarks.cs +++ b/tests/ImageSharp.Benchmarks/Codecs/Heif/Av1ChromaFromLumaBenchmarks.cs @@ -5,6 +5,7 @@ using BenchmarkDotNet.Attributes; using BenchmarkDotNet.Columns; using BenchmarkDotNet.Configs; using BenchmarkDotNet.Jobs; +using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction; using SixLabors.ImageSharp.Formats.Heif.Av1.Prediction.ChromaFromLuma; namespace SixLabors.ImageSharp.Benchmarks.Codecs.Heif; diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundBlockDecoderTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundBlockDecoderTests.cs index 18c73a78b..e2880ca3f 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundBlockDecoderTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundBlockDecoderTests.cs @@ -671,8 +671,8 @@ public class Av1CompoundBlockDecoderTests Av1FrameBuffer secondReference = referenceFrames.Resolve(1)!.FrameBuffer; ushort[] firstHighBitDepthPrediction = new ushort[blockSize * blockSize]; ushort[] secondHighBitDepthPrediction = new ushort[blockSize * blockSize]; - short[] firstScratch = new short[Av1InterPredictor.WarpedScratchLength]; - short[] secondScratch = new short[Av1InterPredictor.WarpedScratchLength]; + short[] firstScratch = new short[Av1WarpedInterPredictor.WarpedScratchLength]; + short[] secondScratch = new short[Av1WarpedInterPredictor.WarpedScratchLength]; Point blockPosition = new(blockOrigin, blockOrigin); if (bitDepth == Av1BitDepth.EightBit) { @@ -690,7 +690,7 @@ public class Av1CompoundBlockDecoderTests out int secondStride, out Point secondOrigin); - Av1InterPredictor.PredictWarpedCompoundScalar( + Av1WarpedInterPredictor.PredictWarpedCompoundScalar( firstSource, firstStride, firstOrigin, @@ -706,7 +706,7 @@ public class Av1CompoundBlockDecoderTests globalMotionParameters, firstScratch); - Av1InterPredictor.PredictWarpedCompoundScalar( + Av1WarpedInterPredictor.PredictWarpedCompoundScalar( secondSource, secondStride, secondOrigin, @@ -739,7 +739,7 @@ public class Av1CompoundBlockDecoderTests out Point secondOrigin); int bitDepthValue = bitDepth.GetBitCount(); - Av1InterPredictor.PredictWarpedScalar( + Av1WarpedInterPredictor.PredictWarpedScalar( firstSource, firstStride, firstOrigin, @@ -756,7 +756,7 @@ public class Av1CompoundBlockDecoderTests globalMotionParameters, firstScratch); - Av1InterPredictor.PredictWarpedScalar( + Av1WarpedInterPredictor.PredictWarpedScalar( secondSource, secondStride, secondOrigin, diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundInterPredictorTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundInterPredictorTests.cs index 89a27e1cb..b0109eec4 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundInterPredictorTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CompoundInterPredictorTests.cs @@ -8,7 +8,7 @@ using SixLabors.ImageSharp.Tests.TestUtilities; namespace SixLabors.ImageSharp.Tests.Formats.Heif.Av1; /// -/// Verifies equal-weight AV1 compound prediction across every hardware-intrinsic tier. +/// Verifies AV1 compound prediction and blending across every hardware-intrinsic tier. /// [Trait("Format", "Avif")] public class Av1CompoundInterPredictorTests @@ -65,8 +65,8 @@ public class Av1CompoundInterPredictorTests mask.AsSpan().Fill(0xA5); inverted.AsSpan().Fill(0xA5); - Av1CompoundInterPredictor.FillInterIntraMask(mask, stride, width, height, mode, invert: false); - Av1CompoundInterPredictor.FillInterIntraMask(inverted, stride, width, height, mode, invert: true); + Av1InterIntraMaskBuilder.FillInterIntraMask(mask, stride, width, height, mode, invert: false); + Av1InterIntraMaskBuilder.FillInterIntraMask(inverted, stride, width, height, mode, invert: true); for (int row = 0; row < height; row++) { @@ -93,6 +93,24 @@ public class Av1CompoundInterPredictorTests } } + /// + /// Verifies the pinned horizontal curve at the index exercised by a 32-by-16 inter-intra block. + /// + [Fact] + public void HorizontalInterIntraMaskMatchesPinnedThirtyTwoWideCurve() + { + const int width = 32; + const int height = 16; + byte[] mask = new byte[width * height]; + byte[] inverted = new byte[width * height]; + + Av1InterIntraMaskBuilder.FillInterIntraMask(mask, width, width, height, Av1InterIntraMode.Horizontal, invert: false); + Av1InterIntraMaskBuilder.FillInterIntraMask(inverted, width, width, height, Av1InterIntraMode.Horizontal, invert: true); + + Assert.Equal(2, mask[23]); + Assert.Equal(62, inverted[23]); + } + /// /// Verifies both difference-mask orientations at each supported bit depth. /// @@ -136,7 +154,7 @@ public class Av1CompoundInterPredictorTests { byte[] firstByte = Array.ConvertAll(first, value => (byte)value); byte[] secondByte = Array.ConvertAll(second, value => (byte)value); - Av1CompoundInterPredictor.FillDifferenceWeightedMask( + Av1DifferenceWeightedMaskBuilder.FillDifferenceWeightedMask( actual, maskStride, firstByte, @@ -149,7 +167,7 @@ public class Av1CompoundInterPredictorTests } else { - Av1CompoundInterPredictor.FillDifferenceWeightedMask( + Av1DifferenceWeightedMaskBuilder.FillDifferenceWeightedMask( actual, maskStride, first, @@ -214,8 +232,8 @@ public class Av1CompoundInterPredictorTests } } - Av1CompoundInterPredictor.Average(actual, destinationStride, second, secondStride, width, height); - Av1CompoundInterPredictor.AverageScalar(scalar, destinationStride, second, secondStride, width, height); + Av1CompoundAveragePredictor.Average(actual, destinationStride, second, secondStride, width, height); + Av1CompoundAveragePredictor.AverageScalar(scalar, destinationStride, second, secondStride, width, height); Assert.Equal(expected, actual); Assert.Equal(expected, scalar); @@ -255,8 +273,8 @@ public class Av1CompoundInterPredictorTests } } - Av1CompoundInterPredictor.Average(actual, destinationStride, second, secondStride, width, height); - Av1CompoundInterPredictor.AverageScalar(scalar, destinationStride, second, secondStride, width, height); + Av1CompoundAveragePredictor.Average(actual, destinationStride, second, secondStride, width, height); + Av1CompoundAveragePredictor.AverageScalar(scalar, destinationStride, second, secondStride, width, height); Assert.Equal(expected, actual); Assert.Equal(expected, scalar); @@ -303,7 +321,7 @@ public class Av1CompoundInterPredictorTests } } - Av1CompoundInterPredictor.DistanceWeighted( + Av1CompoundDistanceWeightedPredictor.DistanceWeighted( actual, destinationStride, second, @@ -330,7 +348,7 @@ public class Av1CompoundInterPredictorTests } } - Av1CompoundInterPredictor.Blend( + Av1CompoundMaskBlendPredictor.Blend( maskedActual, destinationStride, second, @@ -385,7 +403,7 @@ public class Av1CompoundInterPredictorTests } } - Av1CompoundInterPredictor.DistanceWeighted( + Av1CompoundDistanceWeightedPredictor.DistanceWeighted( actual, destinationStride, second, @@ -412,7 +430,7 @@ public class Av1CompoundInterPredictorTests } } - Av1CompoundInterPredictor.Blend( + Av1CompoundMaskBlendPredictor.Blend( maskedActual, destinationStride, second, diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterPredictorTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterPredictorTests.cs index 27a59f5ef..253967a42 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterPredictorTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1InterPredictorTests.cs @@ -186,7 +186,7 @@ public class Av1InterPredictorTests short[] simdScratch = CreateScratch(testCase); short[] scalarScratch = CreateScratch(testCase); - Av1InterPredictor.PredictCompoundScalar( + Av1CompoundInterPredictor.PredictCompoundScalar( source, sourceStride, sourceOrigin, @@ -200,7 +200,7 @@ public class Av1InterPredictorTests testCase.VerticalPhase, scalarScratch); - Av1InterPredictor.PredictCompound( + Av1CompoundInterPredictor.PredictCompound( source, sourceStride, sourceOrigin, diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReconstructionConformanceTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReconstructionConformanceTests.cs index 8929239ad..c874c82a4 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReconstructionConformanceTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ReconstructionConformanceTests.cs @@ -2,6 +2,7 @@ // Licensed under the Six Labors Split License. using System.Buffers.Binary; +using System.Globalization; using System.Text; using SixLabors.ImageSharp.Formats; using SixLabors.ImageSharp.Formats.Heif; @@ -155,6 +156,53 @@ public class Av1ReconstructionConformanceTests /// private const int AverageCompoundFixtureFrameCount = 19; + /// + /// The number of shown frames in the official libaom all-intra sequence. + /// + private const int OfficialAllIntraFixtureFrameCount = 39; + + /// + /// The number of shown frames in the official libaom CDF-update sequence. + /// + private const int OfficialCdfUpdateFixtureFrameCount = 2; + + /// + /// The number of shown frames in the official libaom temporal motion-field sequence. + /// + private const int OfficialMotionFieldFixtureFrameCount = 4; + + /// + /// The coverage bit representing tile-local adaptive CDF updates. + /// + private const int TileCdfUpdateCoverage = 1 << 0; + + /// + /// The coverage bit representing publication of the selected frame-end CDF. + /// + private const int FrameEndCdfUpdateCoverage = 1 << 1; + + /// + /// The coverage bit representing temporal reference-motion-vector projection. + /// + private const int ReferenceFrameMotionVectorCoverage = 1 << 2; + + /// + /// The bit mask containing every intra prediction mode. + /// + private const int RequiredIntraModeCoverage = (1 << (int)Av1PredictionMode.IntraModes) - 1; + + /// + /// The transform types selected by the official all-intra conformance sequence. + /// + private const int RequiredAllIntraTransformTypeCoverage = + (1 << (int)Av1TransformType.DctDct) | + (1 << (int)Av1TransformType.AdstDct) | + (1 << (int)Av1TransformType.DctAdst) | + (1 << (int)Av1TransformType.AdstAdst) | + (1 << (int)Av1TransformType.Identity) | + (1 << (int)Av1TransformType.VerticalDct) | + (1 << (int)Av1TransformType.HorizontalDct); + /// /// The displayed width of the official libaom motion-vector sequence. /// @@ -898,6 +946,255 @@ public class Av1ReconstructionConformanceTests fixtureSize: 256, visibleFrameCount: 2); + /// + /// Verifies every intra prediction mode and the fixture's seven transform types against the official + /// pinned-libaom all-intra conformance sequence and its exact native output. + /// + [Fact] + public void DecodeOfficialAllIntraSequenceMatchesPinnedLibaomReference() => ValidateOfficialAllIntraFixture(); + + /// + /// Decodes every all-intra IVF sample in one session, compares each frame exactly, and records the syntax + /// selections that make the fixture authoritative for prediction and transform coverage. + /// + private static void ValidateOfficialAllIntraFixture() + { + byte[] ivf = TestFile.Create(TestImages.Heif.Av1OfficialAllIntraSequence).Bytes; + byte[] nativeReference = TestFile.Create(TestImages.Heif.Av1OfficialAllIntraSequenceNativeReference).Bytes; + ReadOnlySpan y4mFileHeader = "YUV4MPEG2 W352 H288 F3:1 Ip C420jpeg\n"u8; + ReadOnlySpan y4mFrameHeader = "FRAME\n"u8; + + Assert.True(ivf.AsSpan(0, 4).SequenceEqual("DKIF"u8)); + Assert.Equal(0, BinaryPrimitives.ReadUInt16LittleEndian(ivf.AsSpan(4, 2))); + Assert.Equal(32, BinaryPrimitives.ReadUInt16LittleEndian(ivf.AsSpan(6, 2))); + Assert.True(ivf.AsSpan(8, 4).SequenceEqual("AV01"u8)); + Assert.Equal(OfficialMotionVectorFixtureWidth, BinaryPrimitives.ReadUInt16LittleEndian(ivf.AsSpan(12, 2))); + Assert.Equal(OfficialMotionVectorFixtureHeight, BinaryPrimitives.ReadUInt16LittleEndian(ivf.AsSpan(14, 2))); + Assert.Equal( + OfficialAllIntraFixtureFrameCount, + checked((int)BinaryPrimitives.ReadUInt32LittleEndian(ivf.AsSpan(24, 4)))); + + Assert.True(nativeReference.AsSpan().StartsWith(y4mFileHeader)); + + int ivfOffset = 32; + int nativeOffset = y4mFileHeader.Length; + int nativeFrameLength = + (OfficialMotionVectorFixtureWidth * OfficialMotionVectorFixtureHeight) + + (2 * (OfficialMotionVectorFixtureWidth >> 1) * (OfficialMotionVectorFixtureHeight >> 1)); + + int intraModeCoverage = 0; + int transformTypeCoverage = 0; + using Av1Decoder decoder = new(Configuration.Default); + for (int frameIndex = 0; frameIndex < OfficialAllIntraFixtureFrameCount; frameIndex++) + { + int payloadLength = checked((int)BinaryPrimitives.ReadUInt32LittleEndian(ivf.AsSpan(ivfOffset, 4))); + ivfOffset += 12; + using ImageFrame frame = decoder.DecodeSequenceFrame( + ivf.AsSpan(ivfOffset, payloadLength), + null, + null); + + ivfOffset += payloadLength; + Assert.Equal(OfficialMotionVectorFixtureWidth, frame.Width); + Assert.Equal(OfficialMotionVectorFixtureHeight, frame.Height); + Assert.True(nativeReference.AsSpan(nativeOffset).StartsWith(y4mFrameHeader)); + nativeOffset += y4mFrameHeader.Length; + + Av1FrameBuffer frameBuffer = Assert.IsType>(decoder.FrameBuffer); + Assert.Equal(OfficialMotionVectorFixtureWidth, frameBuffer.Width); + Assert.Equal(OfficialMotionVectorFixtureHeight, frameBuffer.Height); + Assert.Equal(Av1BitDepth.EightBit, frameBuffer.BitDepth); + Assert.Equal(Av1ColorFormat.Yuv420, frameBuffer.ColorFormat); + AssertNativePlanesEqual( + decoder, + frameBuffer, + nativeReference.AsSpan(nativeOffset, nativeFrameLength)); + + nativeOffset += nativeFrameLength; + + ObuSequenceHeader sequenceHeader = Assert.IsType(decoder.SequenceHeader); + Av1FrameInfo frameInfo = Assert.IsType(decoder.FrameInfo); + int superblockColumnCount = Av1Math.AlignPowerOf2(sequenceHeader.MaxFrameWidth, sequenceHeader.SuperblockSizeLog2) + >> sequenceHeader.SuperblockSizeLog2; + int superblockRowCount = Av1Math.AlignPowerOf2(sequenceHeader.MaxFrameHeight, sequenceHeader.SuperblockSizeLog2) + >> sequenceHeader.SuperblockSizeLog2; + + for (int superblockRow = 0; superblockRow < superblockRowCount; superblockRow++) + { + for (int superblockColumn = 0; superblockColumn < superblockColumnCount; superblockColumn++) + { + Av1SuperblockInfo superblockInfo = frameInfo.GetSuperblock(new Point(superblockColumn, superblockRow)); + foreach (Av1BlockModeInfo modeInfo in superblockInfo.GetModeInfos()) + { + if (modeInfo.YMode is >= Av1PredictionMode.IntraModeStart and < Av1PredictionMode.IntraModeEnd) + { + intraModeCoverage |= 1 << ((int)modeInfo.YMode - (int)Av1PredictionMode.IntraModeStart); + } + + int firstTransformLocation = modeInfo.GetFirstTransformLocation(Av1Plane.Y); + int transformUnitCount = modeInfo.GetTransformUnitCount(Av1Plane.Y); + foreach (Av1TransformInfo transformInfo in + superblockInfo.GetTransformInfoY().Slice(firstTransformLocation, transformUnitCount)) + { + transformTypeCoverage |= 1 << (int)transformInfo.Type; + } + } + } + } + } + + Assert.Equal(ivf.Length, ivfOffset); + Assert.Equal(nativeReference.Length, nativeOffset); + Assert.Equal(RequiredIntraModeCoverage, intraModeCoverage); + Assert.Equal(RequiredAllIntraTransformTypeCoverage, transformTypeCoverage); + } + + /// + /// Verifies adaptive tile and frame-end CDF updates against the official pinned-libaom sequence and exact native + /// output under normal and scalar dispatch. + /// + [Fact] + public void DecodeOfficialCdfUpdateSequenceMatchesPinnedLibaomReference() + => FeatureTestRunner.RunWithHwIntrinsicsFeature( + ValidateOfficialCdfUpdateFixture, + ReconstructionConfigurations); + + /// + /// Verifies temporal reference-motion-vector projection against the official pinned-libaom sequence and exact + /// native output under normal and scalar dispatch. + /// + [Fact] + public void DecodeOfficialMotionFieldSequenceMatchesPinnedLibaomReference() + => FeatureTestRunner.RunWithHwIntrinsicsFeature( + ValidateOfficialMotionFieldFixture, + ReconstructionConfigurations); + + /// + /// Verifies exact temporal motion-field reconstruction and balanced ownership with a constrained allocator. + /// + [Fact] + [ValidateDisposedMemoryAllocations] + public void DecodeOfficialMotionFieldSequenceWithConstrainedAllocator() + { + TestMemoryAllocator allocator = new() { BufferCapacityInBytes = 2_048 }; + allocator.EnableNonThreadSafeLogging(); + Configuration configuration = Configuration.Default.Clone(); + configuration.MemoryAllocator = allocator; + + int coverage = ValidateOfficialCompactSequence( + configuration, + TestImages.Heif.Av1OfficialMotionFieldSequence, + TestImages.Heif.Av1OfficialMotionFieldSequenceNativeReference, + OfficialMotionFieldFixtureFrameCount); + + Assert.NotEqual(0, coverage & ReferenceFrameMotionVectorCoverage); + Assert.Contains(allocator.AllocationLog, request => request.ElementType.Name == "RetainedMotionFieldEntry"); + Assert.Contains(allocator.AllocationLog, request => request.ElementType.Name == "TemporalMotionFieldEntry"); + Assert.Equal(allocator.AllocationLog.Count, allocator.ReturnLog.Count); + Assert.All( + allocator.AllocationLog, + allocation => Assert.Single( + allocator.ReturnLog, + returned => returned.AllocationId == allocation.AllocationId)); + } + + /// + /// Validates that the official CDF-update fixture selects both adaptive update boundaries. + /// + private static void ValidateOfficialCdfUpdateFixture() + { + int coverage = ValidateOfficialCompactSequence( + Configuration.Default, + TestImages.Heif.Av1OfficialCdfUpdateSequence, + TestImages.Heif.Av1OfficialCdfUpdateSequenceNativeReference, + OfficialCdfUpdateFixtureFrameCount); + + Assert.Equal(TileCdfUpdateCoverage | FrameEndCdfUpdateCoverage, coverage & 3); + } + + /// + /// Validates that the official temporal motion-field fixture enables projected reference motion vectors. + /// + private static void ValidateOfficialMotionFieldFixture() + { + int coverage = ValidateOfficialCompactSequence( + Configuration.Default, + TestImages.Heif.Av1OfficialMotionFieldSequence, + TestImages.Heif.Av1OfficialMotionFieldSequenceNativeReference, + OfficialMotionFieldFixtureFrameCount); + + Assert.NotEqual(0, coverage & ReferenceFrameMotionVectorCoverage); + } + + /// + /// Decodes one compact official IVF sequence, compares every native sample, and returns its active frame-state + /// coverage mask. + /// + private static int ValidateOfficialCompactSequence( + Configuration configuration, + string fixturePath, + string nativeReferencePath, + int expectedFrameCount) + { + byte[] ivf = TestFile.Create(fixturePath).Bytes; + byte[] nativeReference = TestFile.Create(nativeReferencePath).Bytes; + ReadOnlySpan y4mFileHeader = "YUV4MPEG2 W352 H288 F30:1 Ip C420jpeg\n"u8; + ReadOnlySpan y4mFrameHeader = "FRAME\n"u8; + + Assert.True(ivf.AsSpan(0, 4).SequenceEqual("DKIF"u8)); + Assert.Equal(0, BinaryPrimitives.ReadUInt16LittleEndian(ivf.AsSpan(4, 2))); + Assert.Equal(32, BinaryPrimitives.ReadUInt16LittleEndian(ivf.AsSpan(6, 2))); + Assert.True(ivf.AsSpan(8, 4).SequenceEqual("AV01"u8)); + Assert.Equal(OfficialMotionVectorFixtureWidth, BinaryPrimitives.ReadUInt16LittleEndian(ivf.AsSpan(12, 2))); + Assert.Equal(OfficialMotionVectorFixtureHeight, BinaryPrimitives.ReadUInt16LittleEndian(ivf.AsSpan(14, 2))); + Assert.Equal( + expectedFrameCount, + checked((int)BinaryPrimitives.ReadUInt32LittleEndian(ivf.AsSpan(24, 4)))); + + Assert.True(nativeReference.AsSpan().StartsWith(y4mFileHeader)); + + int ivfOffset = 32; + int nativeOffset = y4mFileHeader.Length; + int nativeFrameLength = + (OfficialMotionVectorFixtureWidth * OfficialMotionVectorFixtureHeight) + + (2 * (OfficialMotionVectorFixtureWidth >> 1) * (OfficialMotionVectorFixtureHeight >> 1)); + + int coverage = 0; + using Av1Decoder decoder = new(configuration); + for (int frameIndex = 0; frameIndex < expectedFrameCount; frameIndex++) + { + int payloadLength = checked((int)BinaryPrimitives.ReadUInt32LittleEndian(ivf.AsSpan(ivfOffset, 4))); + ivfOffset += 12; + using ImageFrame frame = decoder.DecodeSequenceFrame( + ivf.AsSpan(ivfOffset, payloadLength), + null, + null); + + ivfOffset += payloadLength; + Assert.Equal(OfficialMotionVectorFixtureWidth, frame.Width); + Assert.Equal(OfficialMotionVectorFixtureHeight, frame.Height); + Assert.True(nativeReference.AsSpan(nativeOffset).StartsWith(y4mFrameHeader)); + nativeOffset += y4mFrameHeader.Length; + + Av1FrameBuffer frameBuffer = Assert.IsType>(decoder.FrameBuffer); + AssertNativePlanesEqual( + decoder, + frameBuffer, + nativeReference.AsSpan(nativeOffset, nativeFrameLength)); + + nativeOffset += nativeFrameLength; + + ObuFrameHeader frameHeader = Assert.IsType(decoder.FrameHeader); + coverage |= frameHeader.DisableCdfUpdate ? 0 : TileCdfUpdateCoverage; + coverage |= frameHeader.DisableFrameEndUpdateCdf ? 0 : FrameEndCdfUpdateCoverage; + coverage |= frameHeader.UseReferenceFrameMotionVectors ? ReferenceFrameMotionVectorCoverage : 0; + } + + Assert.Equal(ivf.Length, ivfOffset); + Assert.Equal(nativeReference.Length, nativeOffset); + return coverage; + } + /// /// Verifies every ordinary inter mode, motion mode, and switchable dual-filter pair against the official /// pinned-libaom motion-vector conformance sequence and its exact native output. @@ -3219,7 +3516,7 @@ public class Av1ReconstructionConformanceTests if (mismatchCount < 16) { mismatchDescription ??= new StringBuilder(); - mismatchDescription.Append($" {plane}({x},{y})={expectedRow[x]}/{actualRow[x]}"); + mismatchDescription.Append(CultureInfo.InvariantCulture, $" {plane}({x},{y})={expectedRow[x]}/{actualRow[x]}"); } if (mismatchCount == 0 || Math.Abs(expectedRow[x] - actualRow[x]) > Math.Abs(largestExpected - largestActual)) @@ -3252,7 +3549,7 @@ public class Av1ReconstructionConformanceTests if (mismatchCount < 16) { mismatchDescription ??= new StringBuilder(); - mismatchDescription.Append($" {plane}({x},{y})={expected}/{actualRow[x]}"); + mismatchDescription.Append(CultureInfo.InvariantCulture, $" {plane}({x},{y})={expected}/{actualRow[x]}"); } if (mismatchCount == 0 || Math.Abs(expected - actualRow[x]) > Math.Abs(largestExpected - largestActual)) @@ -3374,7 +3671,7 @@ public class Av1ReconstructionConformanceTests if (containingTransform.CodeBlockFlag) { int coefficientCount = superblockCoefficients[coefficientOffset]; - coefficientDescription.Append($", quantized-coefficients={coefficientCount}:["); + coefficientDescription.Append(CultureInfo.InvariantCulture, $", quantized-coefficients={coefficientCount}:["); for (int coefficientIndex = 0; coefficientIndex < coefficientCount; coefficientIndex++) { if (coefficientIndex != 0) diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ScaledInterPredictorTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ScaledInterPredictorTests.cs index 3fc6ae274..04f2c4ddc 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ScaledInterPredictorTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1ScaledInterPredictorTests.cs @@ -118,7 +118,7 @@ public class Av1ScaledInterPredictorTests byte[] expected = CreateByteDestination(testCase, destinationStride); byte[] actual = (byte[])expected.Clone(); short[] scratch = new short[ - Av1InterPredictor.GetScaledScratchLength( + Av1ScaledInterPredictor.GetScaledScratchLength( testCase.Width, testCase.Height, testCase.VerticalPhase, @@ -126,7 +126,7 @@ public class Av1ScaledInterPredictorTests ApplyReference(source, sourceStride, sourceOrigin, expected, destinationStride, testCase, 8); - Av1InterPredictor.PredictScaled( + Av1ScaledInterPredictor.PredictScaled( source, sourceStride, sourceOrigin, @@ -161,7 +161,7 @@ public class Av1ScaledInterPredictorTests ushort[] expected = CreateUInt16Destination(testCase, destinationStride); ushort[] actual = (ushort[])expected.Clone(); short[] scratch = new short[ - Av1InterPredictor.GetScaledScratchLength( + Av1ScaledInterPredictor.GetScaledScratchLength( testCase.Width, testCase.Height, testCase.VerticalPhase, @@ -169,7 +169,7 @@ public class Av1ScaledInterPredictorTests ApplyReference(source, sourceStride, sourceOrigin, expected, destinationStride, testCase, bitDepth); - Av1InterPredictor.PredictScaled( + Av1ScaledInterPredictor.PredictScaled( source, sourceStride, sourceOrigin, @@ -201,6 +201,7 @@ public class Av1ScaledInterPredictorTests new("fixture-regular-8x4", 8, 4, Av1InterpolationFilter.Regular, Av1InterpolationFilter.Regular, 800, 512, 800, 512), new("bilinear-variable-phase", 13, 9, Av1InterpolationFilter.Bilinear, Av1InterpolationFilter.Bilinear, 192, 1536, 512, 640), new("regular-smooth-wide", 20, 8, Av1InterpolationFilter.Regular, Av1InterpolationFilter.Smooth, 64, 2048, 448, 2048), + new("regular-sharp-all-widths", 37, 7, Av1InterpolationFilter.Regular, Av1InterpolationFilter.Sharp, 64, 2048, 512, 2048), new("sharp-bilinear-tail", 12, 5, Av1InterpolationFilter.Sharp, Av1InterpolationFilter.Bilinear, 512, 2048, 192, 2048), new("reduced-regular", 4, 8, Av1InterpolationFilter.Regular, Av1InterpolationFilter.Smooth, 192, 2048, 448, 2048), new("reduced-sharp-maps-to-regular", 4, 8, Av1InterpolationFilter.Sharp, Av1InterpolationFilter.Smooth, 192, 2048, 448, 2048), diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1WarpedInterPredictorTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1WarpedInterPredictorTests.cs index d137b223e..b7507578b 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1WarpedInterPredictorTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1WarpedInterPredictorTests.cs @@ -59,11 +59,11 @@ public class Av1WarpedInterPredictorTests byte[] actual = new byte[destinationStride * height]; Array.Fill(expected, (byte)0xD3); Array.Fill(actual, (byte)0xD3); - short[] expectedScratch = new short[Av1InterPredictor.WarpedScratchLength]; - short[] actualScratch = new short[Av1InterPredictor.WarpedScratchLength]; + short[] expectedScratch = new short[Av1WarpedInterPredictor.WarpedScratchLength]; + short[] actualScratch = new short[Av1WarpedInterPredictor.WarpedScratchLength]; Point destinationPosition = subsampling == 0 ? new Point(32, 24) : new Point(16, 12); - Av1InterPredictor.PredictWarpedScalar( + Av1WarpedInterPredictor.PredictWarpedScalar( source, sourceStride, new Point(padding, padding), @@ -79,7 +79,7 @@ public class Av1WarpedInterPredictorTests parameters, expectedScratch); - Av1InterPredictor.PredictWarped( + Av1WarpedInterPredictor.PredictWarped( source, sourceStride, new Point(padding, padding), @@ -131,11 +131,11 @@ public class Av1WarpedInterPredictorTests ushort[] actual = new ushort[destinationStride * height]; Array.Fill(expected, (ushort)0xDEAD); Array.Fill(actual, (ushort)0xDEAD); - short[] expectedScratch = new short[Av1InterPredictor.WarpedScratchLength]; - short[] actualScratch = new short[Av1InterPredictor.WarpedScratchLength]; + short[] expectedScratch = new short[Av1WarpedInterPredictor.WarpedScratchLength]; + short[] actualScratch = new short[Av1WarpedInterPredictor.WarpedScratchLength]; Point destinationPosition = subsampling == 0 ? new Point(32, 24) : new Point(16, 12); - Av1InterPredictor.PredictWarpedScalar( + Av1WarpedInterPredictor.PredictWarpedScalar( source, sourceStride, new Point(padding, padding), @@ -152,7 +152,7 @@ public class Av1WarpedInterPredictorTests parameters, expectedScratch); - Av1InterPredictor.PredictWarped( + Av1WarpedInterPredictor.PredictWarped( source, sourceStride, new Point(padding, padding), diff --git a/tests/ImageSharp.Tests/TestImages.cs b/tests/ImageSharp.Tests/TestImages.cs index 73749cfd4..54ae22fd2 100644 --- a/tests/ImageSharp.Tests/TestImages.cs +++ b/tests/ImageSharp.Tests/TestImages.cs @@ -1388,8 +1388,14 @@ public static class TestImages public const string Av1GlobalWarpSequenceAvif = "Heif/Av1/Conformance/libavif-rotating-grid-global-warp.avif"; public const string Av1GlobalWarpSequenceNativeReference = "Heif/Av1/Conformance/libavif-rotating-grid-global-warp-libaom.y4m"; public const string Av1GlobalWarpSequencePresentationReference = "Heif/Av1/Conformance/libavif-rotating-grid-global-warp-libavif.png"; + public const string Av1OfficialAllIntraSequence = "Heif/Av1/Conformance/libaom-av1-1-b8-02-allintra.ivf"; + public const string Av1OfficialAllIntraSequenceNativeReference = "Heif/Av1/Conformance/libaom-av1-1-b8-02-allintra-libaom.y4m"; + public const string Av1OfficialCdfUpdateSequence = "Heif/Av1/Conformance/libaom-av1-1-b8-04-cdfupdate.ivf"; + public const string Av1OfficialCdfUpdateSequenceNativeReference = "Heif/Av1/Conformance/libaom-av1-1-b8-04-cdfupdate-libaom.y4m"; public const string Av1OfficialMotionVectorSequence = "Heif/Av1/Conformance/libaom-av1-1-b8-05-mv.ivf"; public const string Av1OfficialMotionVectorSequenceNativeReference = "Heif/Av1/Conformance/libaom-av1-1-b8-05-mv-libaom.y4m"; + public const string Av1OfficialMotionFieldSequence = "Heif/Av1/Conformance/libaom-av1-1-b8-06-mfmv.ivf"; + public const string Av1OfficialMotionFieldSequenceNativeReference = "Heif/Av1/Conformance/libaom-av1-1-b8-06-mfmv-libaom.y4m"; public const string Av1Cdef8BitPayload = "Heif/Av1/Conformance/libaom-cdef-kodim23-8b.bit"; public const string Av1Cdef8BitReference = "Heif/Av1/Conformance/libaom-cdef-kodim23-8b-libaom.yuv"; public const string Av1Cdef8BitAvif = "Heif/Av1/Conformance/libavif-cdef-kodim23-8b.avif"; diff --git a/tests/Images/Input/Heif/Av1/Conformance/libaom-av1-1-b8-02-allintra-libaom.y4m b/tests/Images/Input/Heif/Av1/Conformance/libaom-av1-1-b8-02-allintra-libaom.y4m new file mode 100644 index 000000000..c52d56694 --- /dev/null +++ b/tests/Images/Input/Heif/Av1/Conformance/libaom-av1-1-b8-02-allintra-libaom.y4m @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1211ebefbc9ccef9ed19be4cce3f807d69fffe338e95cca1b5f4ca8023482175 +size 5930767 diff --git a/tests/Images/Input/Heif/Av1/Conformance/libaom-av1-1-b8-02-allintra.ivf b/tests/Images/Input/Heif/Av1/Conformance/libaom-av1-1-b8-02-allintra.ivf new file mode 100644 index 000000000..20073178e --- /dev/null +++ b/tests/Images/Input/Heif/Av1/Conformance/libaom-av1-1-b8-02-allintra.ivf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5fcd265fd9f9bdd0d3179340b4c4532f1422ca5e5d97741c7481b84cb5dc122f +size 1488725 diff --git a/tests/Images/Input/Heif/Av1/Conformance/libaom-av1-1-b8-04-cdfupdate-libaom.y4m b/tests/Images/Input/Heif/Av1/Conformance/libaom-av1-1-b8-04-cdfupdate-libaom.y4m new file mode 100644 index 000000000..6624f077e --- /dev/null +++ b/tests/Images/Input/Heif/Av1/Conformance/libaom-av1-1-b8-04-cdfupdate-libaom.y4m @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4fbff73ff0de2d9084dae557d1d4bd677b0486516525bf4d327d2d795d5a7779 +size 304178 diff --git a/tests/Images/Input/Heif/Av1/Conformance/libaom-av1-1-b8-04-cdfupdate.ivf b/tests/Images/Input/Heif/Av1/Conformance/libaom-av1-1-b8-04-cdfupdate.ivf new file mode 100644 index 000000000..a1923b372 --- /dev/null +++ b/tests/Images/Input/Heif/Av1/Conformance/libaom-av1-1-b8-04-cdfupdate.ivf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:14a3dbf537b6bf15efc003182d9916d61438c93624a8bd26e6e3ae7eaf33ea82 +size 36088 diff --git a/tests/Images/Input/Heif/Av1/Conformance/libaom-av1-1-b8-06-mfmv-libaom.y4m b/tests/Images/Input/Heif/Av1/Conformance/libaom-av1-1-b8-06-mfmv-libaom.y4m new file mode 100644 index 000000000..30415cd73 --- /dev/null +++ b/tests/Images/Input/Heif/Av1/Conformance/libaom-av1-1-b8-06-mfmv-libaom.y4m @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f7db607694818c19e62fd9a27f53e1a3e2d00b72c39c0430c1b26399cc76777d +size 608318 diff --git a/tests/Images/Input/Heif/Av1/Conformance/libaom-av1-1-b8-06-mfmv.ivf b/tests/Images/Input/Heif/Av1/Conformance/libaom-av1-1-b8-06-mfmv.ivf new file mode 100644 index 000000000..38d15c7e9 --- /dev/null +++ b/tests/Images/Input/Heif/Av1/Conformance/libaom-av1-1-b8-06-mfmv.ivf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b59bf9586d8546dfda81dfec4ee4e32ceb502c9d22412ab0b63a2abb534a1f14 +size 44964