From 02253984e76bb8bb0b98ff7272538577c8f2b395 Mon Sep 17 00:00:00 2001 From: James Jackson-South Date: Thu, 3 Sep 2026 23:21:48 +1000 Subject: [PATCH] Implement live AV1 8x8 partition search --- HEIF_IMPLEMENTATION_PLAN.md | 10 +- .../Heif/Av1/Entropy/Av1SymbolEncoder.cs | 49 + .../Av1/Pipeline/Av1EncoderBlockWorkspace.cs | 17 +- .../Heif/Av1/Pipeline/Av1FrameEncoder.cs | 6 +- ...traSuperblockEncoder.ChromaModeDecision.cs | 45 +- ...blockEncoder.IntraBlockCopyModeDecision.cs | 22 +- .../Av1IntraSuperblockEncoder.ModeDecision.cs | 843 +++++++++++++++--- .../Av1/Tiling/Av1EncoderPictureBuffer.cs | 6 +- .../Av1/Tiling/Av1TileWriter.BlockEncoding.cs | 29 + .../Formats/Heif/Av1/Tiling/Av1TileWriter.cs | 255 ++++-- .../Heif/Av1/Av1CoefficientsEntropyTests.cs | 3 +- .../Formats/Heif/Av1/Av1EncoderFrameTests.cs | 63 ++ .../Heif/Av1/Av1EncoderModeInfoBufferTests.cs | 3 +- .../Heif/Av1/Av1IntraBlockCopyTests.cs | 15 +- .../Av1/Av1IntraSuperblockEncoderTests.cs | 74 +- 15 files changed, 1179 insertions(+), 261 deletions(-) diff --git a/HEIF_IMPLEMENTATION_PLAN.md b/HEIF_IMPLEMENTATION_PLAN.md index aad4d0550d..30f443cf41 100644 --- a/HEIF_IMPLEMENTATION_PLAN.md +++ b/HEIF_IMPLEMENTATION_PLAN.md @@ -792,7 +792,7 @@ Writer primitives are not an encoder. The public encoder remains incomplete unti ### 5. Define and enforce the encoder contract -- [x] Use official libaom `main` at `a40ed1ea9e4ecc3df58a5bccb76623f2c94ae727` as the encoder syntax, probability-model, transform, quantization, filtering, and bitstream reference. +- [x] Use official libaom `main` at `d773924c767f7433d3838dbe1ba90978b10bf675` as the encoder syntax, probability-model, transform, quantization, filtering, and bitstream reference. - [x] Use the existing PNG, TIFF, and JPEG encoders as the ImageSharp architecture reference: generic `Image` input, encoder options taking precedence over converted format metadata and codec defaults, allocator-owned temporary storage, and deterministic disposal. - [x] Treat source pixel type, source alpha representation, and decoded source bit depth as conversion inputs, never as output-eligibility checks. Do not pre-scan pixels before encoding. - [x] Resolve output configuration once from explicit encoder options, converted `HeifMetadata`, and AV1 defaults in that order. Sanitize only combinations that cannot describe a legal requested output, and never write resolved values back to source metadata. @@ -825,13 +825,13 @@ Encoder verification contract: - [~] A production single-tile all-intra writer now walks raster superblocks, analyzes each immediately before entropy coding, reuses one decision workspace and one block workspace, and retains decoder-identical reconstructed references across the tile. Its closed byte and high-bit-depth operators feed the existing superblock boundary without runtime sample-type checks. A byte-exact test compares this composed path with an explicit superblock-then-tile-writer oracle, so producer and writer traversal or coefficient-area drift cannot pass unnoticed. A separate clipped 2x2-superblock regression proves global raster indexing by requiring all four coefficient segments and the bottom-right reconstruction to be populated. Multi-tile ownership and the complete frame/OBU operation remain. - [~] A non-owning encoder-frame view now separates visible conversion regions from coded regions and performs complete left, top, right, bottom, and corner extension across each bordered plane. Current libaom uses 8-sample-aligned coded dimensions, a 32-sample-aligned luma stride with chroma stride derived from it, and a 64-pixel luma border for non-resized all-intra encoding. One operation-ready frame owner now rents the aligned Y, U, and V storage contiguously, exposes non-owning `Buffer2D` plane views, and returns the rent exactly once. A 4K 4:2:0 frame occupies about 13.0 MiB at 8-bit or 26.0 MiB at 10/12-bit; source and reconstruction therefore remain distinct frame owners rather than adding a full-frame copy. The corrected tests use this real ownership path and verify the exact 54 KiB 64x64 4:2:0 rent. The frame-encoder operation now instantiates matching source and reconstruction owners with ordinary `using` lifetimes and converts packed pixels directly into the source owner before extension. - [~] Temporal delimiter, sequence header, frame header, combined-frame tile-group writing, and an internal reduced-still-picture frame operation now exist locally. The remaining required metadata, padding, multi-tile, option, and public encoder paths are not complete. -- [~] Implement superblock and partition analysis for every permitted block size and partition. The current baseline deliberately splits every in-frame node to 8x8 blocks and records decisions in current-libaom writer preorder; block-size selection and non-split partition analysis remain. +- [~] Implement superblock and partition analysis for every permitted block size and partition. Efforts zero through eight deliberately split every in-frame node to 8x8 blocks. Efforts nine and ten now perform live rate-distortion selection among `PARTITION_NONE`, `PARTITION_SPLIT`, `PARTITION_HORZ`, and `PARTITION_VERT` at each 8x8 node in current-libaom search order. Split trials visit four 4x4 leaves in raster order, rectangular trials visit two 8x4 or 4x8 leaves, and every non-final trial leaf publishes reconstructed mode, transform, and coefficient contexts for the next leaf without mutating entropy distributions. The reusable block owner grew by four integer elements for exact parent-edge restoration; no partition trial allocates or copies probability state. Production picture state retains the compact 8x8 mode allocation below effort nine and explicitly selects 4x4 allocation granularity when sub-8x8 partitions are enabled. Partition selection above 8x8, asymmetric and four-way partitions, and effort-dependent pruning remain. - [~] Implement intra mode search, palette, filter intra, chroma-from-luma, and intra-block copy decisions. Live luma search now covers all 13 zero-angle base modes and all six nonzero adjustments for each of the eight directional modes. Joint spatial chroma search covers the same 61 candidates, combines both chroma planes in one rate-distortion decision, and preserves the winning shared angle adjustment. Chroma-from-luma now searches the complete signed alpha alphabet from reconstructed luma and retains its joint U/V syntax. Filter-intra now searches all five predictors after ordinary luma modes. Palette entropy, retained state, production syntax, exhaustive luma and paired chroma palette selection, adaptive screen-content activation, and joint intra-block-copy mode selection are complete. - [ ] Implement inter mode search for bounded sequences, including reference selection and the decoder-supported inter tools. - [~] Current-libaom `av1_quantize_fp_no_qmatrix` arithmetic is implemented as a closed generic forward-quantizer family with Vector512, Vector256, Vector128, and scalar paths, raster-order output, coded 64-point coefficient limits, and scan-order EOB selection. Transform search, coefficient optimization, and lossless behavior remain. -- [~] Implement real rate-distortion selection and make quality and effort change work, size, and output quality. The complete luma and joint chroma candidate sets, including chroma-from-luma, filter-intra, palette, and intra-block copy, now perform live rate-distortion selection. Public quality mapping and effort tiers through exhaustive uniform luma mode/transform search are implemented; partition search, effort-dependent pruning, and the remaining sequence searches remain. -- [~] Frame effort now progressively expands the available current search: zero is DC-only, one adds every zero-angle spatial mode, two adds every legal directional adjustment, three refines the preliminary luma winner's transform type, four adds filter-intra and chroma-from-luma, and five adds adaptive palette and intra-block-copy analysis. Lower tiers do not signal unavailable sequence or frame tools, and tiers below five skip the whole-frame screen-content scan. Effort six enables `TX_MODE_SELECT` and compares the winning ordinary spatial or filter-intra luma mode as one 8x8 transform against four raster-ordered 4x4 transforms; each luma palette candidate owns that size comparison from effort six onward. Effort seven searches every legal 8x8 transform type inside every ordinary spatial candidate rather than refining only the preliminary winner. Effort eight also performs the 8x8-versus-four-4x4 comparison inside every ordinary spatial and filter-intra candidate, matching current libaom's per-candidate uniform-transform ownership. Values nine and ten currently share the effort-eight ceiling. Prediction and residual construction run once per mode and are reused across its legal transform types. Every 4x4 transform searches all legal types with live coefficient contexts and reconstructed intra references. The search reuses the aligned block workspace, preserves only global improvements, and performs no per-block or per-transform rent. Non-skipped intra-block copy writes and costs the current-libaom unsplit variable-transform root; skipped intra-block copy emits no transform-partition symbol. Partition search and effort-dependent model/transform pruning remain. Decoder-visible production cases now execute effort zero through eight and ten, inspect the emitted restrictions and frame state, and decode the produced streams. The complete non-HEVC HEIF/AV1 namespace passes 9,301 of 9,301 through one foreground net11 VSTest run. Current-main `aomdec` at `a40ed1ea9e4ecc3df58a5bccb76623f2c94ae727` accepts the generated effort-eight and effort-ten payloads as well as the existing palette and intra-block-copy payloads. The net11 Release build and Roslynk compiler pass report zero errors. -- [~] Encoder rate accounting converts the entropy writer's live inverse cumulative distributions into current-libaom fixed-point symbol costs without allocating or duplicating probability state. Read-only luma-mode, directional-delta, filter-intra, chroma-mode, block-skip, transform-size, transform-block-skip, and complete transform-coefficient queries share the exact distributions mutated by the subsequent entropy write. Complete coefficient costing follows current libaom's optimized shape: it returns immediately for an empty transform, uses the EOB-specific base-range context, fuses magnitude, sign, base-range, and Golomb accounting into one reverse traversal, and combines repeated full base-range chunks instead of replaying each emitted symbol. Tile-lifetime level and context scratch is reused, the one-coefficient path neither clears nor initializes the forward-neighbor level map, and steady-state queries allocate nothing. Transform-size writing and costing share one subdivision-depth calculation, while shared closed symbol operations keep the writer and cost mappings for transform skip, transform type, and EOB syntax identical without forcing the estimator through the writer's slower two-pass coefficient traversal. The current-libaom fixed-point RD combiner preserves 64-bit distortion and rounds the weighted 1/512-bit rate at the required boundary. Its key-frame multiplier follows libaom's squared DC-quantizer formula and exact 10/12-bit normalization. Live final-block selection evaluates all 61 legal 8x8 luma candidates: the 13 zero-angle base modes in current-libaom order, followed by six nonzero adjustments for each directional mode. Joint chroma selection evaluates the equivalent 61 spatial candidates, combines U and V distortion plus coefficient rate, and charges one live chroma-mode and shared-angle symbol over the actual subsampled 4x4, 4x8, or 8x8 geometry. Chroma-from-luma subsamples the reconstructed luma block once into fixed-stride Q3 stack scratch, subtracts the rounded mean, evaluates all 33 signed alpha values independently for each plane with complete transform RD, and combines the cached plane results across all 1,088 valid joint pairs with one live sign cost and the conditional U/V magnitude costs. This is the allocation-free equivalent of current libaom's exhaustive 33-value path: it requires 66 evaluation transforms rather than transforming every joint pair, preserves DC-before-CfL-before-spatial tie order, and fixes the implicit chroma transform to DCT-DCT. Filter-intra follows ordinary luma candidates, searches all five predictors in syntax order, and evaluates every legal transform while reusing one prepared prediction and source residual per filter mode. Every candidate includes its live mode, angle, filter mode, alpha, and coefficient rate plus normalized pixel-domain distortion. Each prepared reference edge retains the common-corner prefix and twice the transform dimension required by directional prediction. A shared encoder/decoder availability calculation selects reconstructed top-right and bottom-left extensions according to tile, frame, superblock, and block reconstruction order; unavailable extensions repeat the nearest coded endpoint. Missing top or left edges retain current libaom's perpendicular-sample and bit-depth-midpoint rules. Directional prediction applies the AV1 three-degree adjustment step and reuses transform workspace for zone-three transposition before the transform overwrites it, keeping candidate evaluation allocation-free. The winning luma and chroma signed adjustments are retained in the packed final-block state consumed by the tile writer. The tile writer invokes these reusable workspace-backed selectors after mapping current neighbors and immediately before writing each block, so later decisions see reconstructed samples, coefficient contexts, and CDF updates from every preceding block. Block skip is read only after the callback has combined every coded plane. Luma and chroma candidate scratch is partitioned from the encoder's single aligned reusable block workspace; transform-size search uses that owner for four retained 4x4 transform states, local coefficient contexts, and the compact trial reconstruction needed to preserve the best result. No candidate path rents a buffer per block or per transform. Only a newly winning candidate is copied into retained frame storage. Production fixtures force every luma base predictor, both extreme adjustments in all three directional zones, available top-right and bottom-left extensions, high-bit-depth adjustment propagation, exact signed luma and chroma angle-rate terms, joint U/V decisions, packed chroma state, and 4:2:0, 4:2:2, and 4:4:4 transform geometry. The CfL fixtures derive target chroma from a pilot production encode's actual reconstructed luma through an independent scalar Q3 oracle and prove exact positive/negative alpha syntax plus zero-residual DCT-DCT reconstruction for all three subsampling geometries at 8, 10, and 12 bits. The stable fixed-DC traversal comparison uses neutral samples for which both the baseline and live search are contractually DC and skipped, instead of relying on textured content to happen to select the baseline mode. Luma palette selection now evaluates dominant-color and one-dimensional K-means candidates for every legal size, snaps near-cache colors with the reference threshold and tie order, removes duplicate snapped colors, extends boundary maps from active samples, and performs complete transform rate-distortion search. Ordinary DC and filter-intra candidates pay the palette-disabled symbol whenever screen-content syntax is enabled. The exact net11 Release rebuild reports 1,992 test-project warnings and zero errors, all 58 intra-superblock cases pass, all 8,935 AVIF cases pass, and all 230 HEIF cases pass. Remaining mode decision work includes transform-size coverage for filter-intra and palette, broader joint mode/transform refinement, partition search, and effort-dependent pruning. Non-empty intra blocks deliberately remain non-skipped, matching current libaom; later inter mode selection owns its distinct skip-transform RD decision. +- [~] Implement real rate-distortion selection and make quality and effort change work, size, and output quality. The complete luma and joint chroma candidate sets, including chroma-from-luma, filter-intra, palette, and intra-block copy, now perform live rate-distortion selection. Public quality mapping and effort tiers through exhaustive uniform luma mode/transform search are implemented. Efforts nine and ten add exact 8x8 partition rate-distortion selection; partition search above 8x8, effort-dependent pruning, and the remaining sequence searches remain. +- [~] Frame effort now progressively expands the available current search: zero is DC-only, one adds every zero-angle spatial mode, two adds every legal directional adjustment, three refines the preliminary luma winner's transform type, four adds filter-intra and chroma-from-luma, and five adds adaptive palette and intra-block-copy analysis. Lower tiers do not signal unavailable sequence or frame tools, and tiers below five skip the whole-frame screen-content scan. Effort six enables `TX_MODE_SELECT` and compares the winning ordinary spatial or filter-intra luma mode as one 8x8 transform against four raster-ordered 4x4 transforms; each luma palette candidate owns that size comparison from effort six onward. Effort seven searches every legal 8x8 transform type inside every ordinary spatial candidate rather than refining only the preliminary winner. Effort eight also performs the 8x8-versus-four-4x4 comparison inside every ordinary spatial and filter-intra candidate, matching current libaom's per-candidate uniform-transform ownership. Efforts nine and ten additionally search every legal 8x8 partition in current-libaom order. Prediction and residual construction run once per mode and are reused across its legal transform types. Every 4x4 transform searches all legal types with live coefficient contexts and reconstructed intra references. The search reuses the aligned block workspace, preserves only global improvements, and performs no per-block, per-partition, or per-transform rent. Non-skipped intra-block copy writes and costs the current-libaom unsplit variable-transform root; skipped intra-block copy emits no transform-partition symbol. Partition search above 8x8 and effort-dependent model/transform pruning remain. Decoder-visible production cases now execute effort zero through eight and ten, inspect the emitted restrictions and frame state, and decode the produced streams. The complete non-HEVC HEIF/AV1 namespace passes 9,301 of 9,301 through one foreground net11 VSTest run. Current-main `aomdec` at `a40ed1ea9e4ecc3df58a5bccb76623f2c94ae727` accepts the generated effort-eight and effort-ten payloads as well as the existing palette and intra-block-copy payloads. The affected partition and mode-decision surface passes 231 of 231 through one foreground net11 Release VSTest run, including a production effort-nine stream that selects sub-8x8 rectangular blocks and decodes successfully. The net11 Release build and Roslynk compiler pass report zero errors. +- [~] Encoder rate accounting converts the entropy writer's live inverse cumulative distributions into current-libaom fixed-point symbol costs without allocating or duplicating probability state. Read-only luma-mode, directional-delta, filter-intra, chroma-mode, block-skip, transform-size, transform-block-skip, and complete transform-coefficient queries share the exact distributions mutated by the subsequent entropy write. Complete coefficient costing follows current libaom's optimized shape: it returns immediately for an empty transform, uses the EOB-specific base-range context, fuses magnitude, sign, base-range, and Golomb accounting into one reverse traversal, and combines repeated full base-range chunks instead of replaying each emitted symbol. Tile-lifetime level and context scratch is reused, the one-coefficient path neither clears nor initializes the forward-neighbor level map, and steady-state queries allocate nothing. Transform-size writing and costing share one subdivision-depth calculation, while shared closed symbol operations keep the writer and cost mappings for transform skip, transform type, and EOB syntax identical without forcing the estimator through the writer's slower two-pass coefficient traversal. The current-libaom fixed-point RD combiner preserves 64-bit distortion and rounds the weighted 1/512-bit rate at the required boundary. Its key-frame multiplier follows libaom's squared DC-quantizer formula and exact 10/12-bit normalization. Live final-block selection evaluates all 61 legal 8x8 luma candidates: the 13 zero-angle base modes in current-libaom order, followed by six nonzero adjustments for each directional mode. Joint chroma selection evaluates the equivalent 61 spatial candidates, combines U and V distortion plus coefficient rate, and charges one live chroma-mode and shared-angle symbol over the actual subsampled 4x4, 4x8, or 8x8 geometry. Chroma-from-luma subsamples the reconstructed luma block once into fixed-stride Q3 stack scratch, subtracts the rounded mean, evaluates all 33 signed alpha values independently for each plane with complete transform RD, and combines the cached plane results across all 1,088 valid joint pairs with one live sign cost and the conditional U/V magnitude costs. This is the allocation-free equivalent of current libaom's exhaustive 33-value path: it requires 66 evaluation transforms rather than transforming every joint pair, preserves DC-before-CfL-before-spatial tie order, and fixes the implicit chroma transform to DCT-DCT. Filter-intra follows ordinary luma candidates, searches all five predictors in syntax order, and evaluates every legal transform while reusing one prepared prediction and source residual per filter mode. Every candidate includes its live mode, angle, filter mode, alpha, and coefficient rate plus normalized pixel-domain distortion. Each prepared reference edge retains the common-corner prefix and twice the transform dimension required by directional prediction. A shared encoder/decoder availability calculation selects reconstructed top-right and bottom-left extensions according to tile, frame, superblock, and block reconstruction order; unavailable extensions repeat the nearest coded endpoint. Missing top or left edges retain current libaom's perpendicular-sample and bit-depth-midpoint rules. Directional prediction applies the AV1 three-degree adjustment step and reuses transform workspace for zone-three transposition before the transform overwrites it, keeping candidate evaluation allocation-free. The winning luma and chroma signed adjustments are retained in the packed final-block state consumed by the tile writer. The tile writer invokes these reusable workspace-backed selectors after mapping current neighbors and immediately before writing each block, so later decisions see reconstructed samples, coefficient contexts, and CDF updates from every preceding block. Block skip is read only after the callback has combined every coded plane. Luma and chroma candidate scratch is partitioned from the encoder's single aligned reusable block workspace; transform-size search uses that owner for four retained 4x4 transform states, local coefficient contexts, and the compact trial reconstruction needed to preserve the best result. No candidate path rents a buffer per block or per transform. Only a newly winning candidate is copied into retained frame storage. Production fixtures force every luma base predictor, both extreme adjustments in all three directional zones, available top-right and bottom-left extensions, high-bit-depth adjustment propagation, exact signed luma and chroma angle-rate terms, joint U/V decisions, packed chroma state, and 4:2:0, 4:2:2, and 4:4:4 transform geometry. The CfL fixtures derive target chroma from a pilot production encode's actual reconstructed luma through an independent scalar Q3 oracle and prove exact positive/negative alpha syntax plus zero-residual DCT-DCT reconstruction for all three subsampling geometries at 8, 10, and 12 bits. The stable fixed-DC traversal comparison uses neutral samples for which both the baseline and live search are contractually DC and skipped, instead of relying on textured content to happen to select the baseline mode. Luma palette selection now evaluates dominant-color and one-dimensional K-means candidates for every legal size, snaps near-cache colors with the reference threshold and tie order, removes duplicate snapped colors, extends boundary maps from active samples, and performs complete transform rate-distortion search. Ordinary DC and filter-intra candidates pay the palette-disabled symbol whenever screen-content syntax is enabled. The exact net11 Release rebuild reports 1,992 test-project warnings and zero errors, all 58 intra-superblock cases pass, all 8,935 AVIF cases pass, and all 230 HEIF cases pass. Remaining mode decision work includes transform-size coverage for filter-intra and palette, broader joint mode/transform refinement, partition search above 8x8, extended partition shapes, and effort-dependent pruning. Non-empty intra blocks deliberately remain non-skipped, matching current libaom; later inter mode selection owns its distinct skip-transform RD decision. - [~] The tile writer now publishes one packed coefficient context per covered 4x4 edge unit and derives luma/chroma skip plus DC-sign contexts from the complete transform edges using current-libaom units. Partition, transform, and coefficient neighbor state retains only the above and left context regions used by current libaom; the unused third top-left region, its granularity state, and its unused sentinel are removed. One picture owner now packs segmentation plus every tile's partition, luma, chroma, and transform edges into one clean byte allocation with typed non-owning views; together with the separately typed packed mode-information owner, the complete picture state uses two allocator rents rather than seven. Exact aligned lengths, clean initialization, and balanced exactly-once returns are covered in Release. Multi-tile payload ownership and verified CDF update behavior remain. - [~] Encoder mode information now uses a frame-owned integer alias grid over a packed 8-byte value allocation, matching current libaom's `mi_grid_base` and `mi_alloc` relationship without a managed object or reference per 4x4 entry. The visible dimensions are aligned to eight luma samples, the grid stride and allocated row count are aligned to 32 mode-information units, and optional 8x8 allocation granularity reduces the value store in both dimensions exactly as current libaom does. One clean ImageSharp byte owner contains both independently typed regions, reducing libaom's two allocation lifetimes to one without a copy. At 4K, the 4x4 layout occupies about 6.0 MiB in total; the 8x8 layout occupies about 3.0 MiB. Exact geometry, clean allocation, typed lengths, aligned mapping, untouched row padding, and exactly-once return pass 4 of 4 direct net11 VSTest cases in Release. Every coded 4x4 cell covered by square, rectangular, or clipped edge blocks maps to its owning allocation entry before context-dependent symbols are written. Packed syntax, relative neighbor lookup, full block mapping, writer traversal, entropy, and OBU coverage pass 1,947 of 1,947 direct net11 VSTest cases in Release; complete mode decision still remains. - [~] The final-block decision workspace uses one reusable 8.3 KiB ImageSharp allocator owner. It contains 1,024 explicitly packed 8-byte final-block entries and the 341 preorder partition bytes required by a complete 128x128-through-8x8 quadtree, replacing separate managed arrays. Palette colors now have their own current-block value and are copied only to the picture edges that later blocks can reference, so enabling palette mode does not add 50 bytes to every final-block entry. Construction and the explicit per-superblock reset initialize every syntax field, including the nonzero sentinel that disables filter-intra prediction; pooled quantizer, prediction, partition, and current-palette bytes cannot leak into the next decision pass. Complete mode decision still remains. diff --git a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolEncoder.cs b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolEncoder.cs index 8062fb7793..9a02cb479e 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolEncoder.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolEncoder.cs @@ -722,6 +722,15 @@ internal class Av1SymbolEncoder : IDisposable public int GetDisplacementVectorSearchCost(Av1MotionVector value, Av1MotionVector reference) => this.displacementVector.GetCost(this.writer, value, reference); + /// + /// Gets the current fixed-point cost of a complete block partition symbol. + /// + /// The partition type to measure. + /// The partition probability context. + /// The rate cost in 1/512-bit units. + public int GetPartitionTypeCost(Av1PartitionType partitionType, int context) + => Av1ProbabilityCost.GetSymbolCost(this.tilePartitionTypes[context], (int)partitionType); + /// /// Writes a complete block partition type using the selected partition context. /// @@ -747,6 +756,26 @@ internal class Av1SymbolEncoder : IDisposable w.WriteBoolean(value, frequency); } + /// + /// Gets the current fixed-point cost of the split-versus-horizontal boundary decision. + /// + /// The split or horizontal partition outcome. + /// The current block size. + /// The partition probability context. + /// The rate cost in 1/512-bit units. + public int GetSplitOrHorizontalCost(Av1PartitionType partitionType, Av1BlockSize blockSize, int context) + { + int frequency = (int)Av1SymbolDecoder.GetSplitOrHorizontalFrequency( + this.tilePartitionTypes, + blockSize, + context); + + return Av1ProbabilityCost.GetProbabilityCost( + partitionType == Av1PartitionType.Split + ? frequency + : Av1Distribution.ProbabilityTop - frequency); + } + /// /// Writes the split-versus-vertical boundary decision for a block clipped at the right tile edge. /// @@ -761,6 +790,26 @@ internal class Av1SymbolEncoder : IDisposable w.WriteBoolean(value, frequency); } + /// + /// Gets the current fixed-point cost of the split-versus-vertical boundary decision. + /// + /// The split or vertical partition outcome. + /// The current block size. + /// The partition probability context. + /// The rate cost in 1/512-bit units. + public int GetSplitOrVerticalCost(Av1PartitionType partitionType, Av1BlockSize blockSize, int context) + { + int frequency = (int)Av1SymbolDecoder.GetSplitOrVerticalFrequency( + this.tilePartitionTypes, + blockSize, + context); + + return Av1ProbabilityCost.GetProbabilityCost( + partitionType == Av1PartitionType.Split + ? frequency + : Av1Distribution.ProbabilityTop - frequency); + } + /// /// Encodes one transform block's coefficient syntax using scan-order probability contexts. /// diff --git a/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1EncoderBlockWorkspace.cs b/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1EncoderBlockWorkspace.cs index 717a9617ed..20f4d37d2e 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1EncoderBlockWorkspace.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1EncoderBlockWorkspace.cs @@ -31,7 +31,8 @@ internal sealed class Av1EncoderBlockWorkspace : IDisposable MaximumCoefficientCount + MaximumCoefficientCount + Av1TransformWorkspace.MaximumLength + - IntraBlockCopyStorageLength; + IntraBlockCopyStorageLength + + PartitionContextStorageLength; private const int ResidualStorageLength = MaximumResidualCount / 2; private const int TransformCoefficientOffset = ResidualStorageLength; @@ -64,6 +65,11 @@ internal sealed class Av1EncoderBlockWorkspace : IDisposable IntraBlockCopyResidualStorageLength + IntraBlockCopyCoefficientStorageLength; + private const int PartitionContextStorageOffset = + IntraBlockCopySampleStorageOffset + IntraBlockCopyStorageLength; + + private const int PartitionContextStorageLength = 4; + /// /// Owns the complete reusable block workspace in 32-bit elements so every transform region is naturally aligned. /// @@ -100,6 +106,15 @@ internal sealed class Av1EncoderBlockWorkspace : IDisposable public Span TransformWorkspace => this.owner.Memory.Span.Slice(TransformWorkspaceOffset, Av1TransformWorkspace.MaximumLength); + /// + /// Gets storage for the coefficient and transform edges restored after an 8x8 partition trial. + /// + public Span PartitionContexts + => MemoryMarshal.AsBytes( + this.owner.Memory.Span.Slice( + PartitionContextStorageOffset, + PartitionContextStorageLength)); + /// /// Gets the reusable storage used while comparing spatial, chroma-from-luma, filter-intra, and palette candidates. /// diff --git a/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1FrameEncoder.cs b/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1FrameEncoder.cs index 07756edf65..880d59dcdf 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1FrameEncoder.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1FrameEncoder.cs @@ -300,7 +300,8 @@ internal static class Av1FrameEncoder sequenceHeader, frameHeader, image.Width, - image.Height); + image.Height, + disallow4x4AllFrames: effort < 9); using Av1EncoderCoefficientBuffer coefficients = new( configuration, @@ -363,7 +364,8 @@ internal static class Av1FrameEncoder sequenceHeader, frameHeader, image.Width, - image.Height); + image.Height, + disallow4x4AllFrames: effort < 9); using Av1EncoderCoefficientBuffer coefficients = new( configuration, diff --git a/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1IntraSuperblockEncoder.ChromaModeDecision.cs b/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1IntraSuperblockEncoder.ChromaModeDecision.cs index 6ab322a888..d0e648eec2 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1IntraSuperblockEncoder.ChromaModeDecision.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1IntraSuperblockEncoder.ChromaModeDecision.cs @@ -46,6 +46,7 @@ internal static partial class Av1IntraSuperblockEncoder Av1MacroBlockModeInfo modeInfo, Point lumaOrigin, Point chromaOrigin, + Av1BlockSize blockSize, ushort tileIndex, Av1PredictionMode lumaMode, Av1TransformSize transformSize, @@ -59,7 +60,6 @@ internal static partial class Av1IntraSuperblockEncoder out sbyte selectedChromaFromLumaSigns, out long selectedCost) { - const Av1BlockSize BlockSize = Av1BlockSize.Block8x8; Av1EncoderModeDecisionWorkspace workspace = this.blockWorkspace.GetModeDecisionWorkspace(); @@ -73,11 +73,24 @@ internal static partial class Av1IntraSuperblockEncoder int modeInfoColumn = lumaOrigin.X >> Av1Constants.ModeInfoSizeLog2; bool hasLeft = macroBlock.IsLeftAvailable; bool hasAbove = macroBlock.IsUpAvailable; + + // Subsampled chroma belongs to the bottom-right luma unit in its shared 8x8 region. Its external + // references therefore begin before that region, not immediately beside the owning 4x4 luma block. + if (subsamplingX != 0 && blockSize.Get4x4WideCount() < Av1BlockSize.Block8x8.Get4x4WideCount()) + { + hasLeft = modeInfoColumn - 1 > macroBlock.Tile.ModeInfoColumnStart; + } + + if (subsamplingY != 0 && blockSize.Get4x4HighCount() < Av1BlockSize.Block8x8.Get4x4HighCount()) + { + hasAbove = modeInfoRow - 1 > macroBlock.Tile.ModeInfoRowStart; + } + bool rightAvailable = modeInfoColumn + (transformSize.Get4x4WideCount() << subsamplingX) < macroBlock.Tile.ModeInfoColumnEnd; bool bottomAvailable = modeInfoRow + (transformSize.Get4x4HighCount() << subsamplingY) < macroBlock.Tile.ModeInfoRowEnd; bool hasTopRight = Av1IntraReferenceAvailability.HasTopRight( this.picture.Sequence.SequenceHeader.SuperblockSize, - BlockSize, + blockSize, modeInfoRow, modeInfoColumn, hasAbove, @@ -91,7 +104,7 @@ internal static partial class Av1IntraSuperblockEncoder bool hasBottomLeft = Av1IntraReferenceAvailability.HasBottomLeft( this.picture.Sequence.SequenceHeader.SuperblockSize, - BlockSize, + blockSize, modeInfoRow, modeInfoColumn, bottomAvailable, @@ -139,7 +152,7 @@ internal static partial class Av1IntraSuperblockEncoder ReadOnlySpan blueLeft = blueLeftStorage.Slice(1, height * 2); ReadOnlySpan redAbove = redAboveStorage.Slice(1, width * 2); ReadOnlySpan redLeft = redLeftStorage.Slice(1, height * 2); - Av1BlockSize chromaBlockSize = BlockSize.GetSubsampled(colorConfig.SubSamplingX, colorConfig.SubSamplingY); + Av1BlockSize chromaBlockSize = blockSize.GetSubsampled(colorConfig.SubSamplingX, colorConfig.SubSamplingY); Av1TransformBlockContext blueContext = Av1TileWriter.GetTransformBlockContexts( Av1ComponentType.Chroma, this.picture.CbDcSignLevelCoefficientNeighbors[tileIndex], @@ -172,11 +185,13 @@ internal static partial class Av1IntraSuperblockEncoder { 0 => 1, 1 => baseModeCount, - _ => baseModeCount + (directionalModeCount * deltaCount) + _ when blockSize >= Av1BlockSize.Block8x8 => baseModeCount + (directionalModeCount * deltaCount), + _ => baseModeCount }; bool hasLumaPalette = paletteInfo.PaletteSizes[0] != 0; - int paletteDisabledCost = this.picture.Parent.FrameHeader.AllowScreenContentTools + int paletteDisabledCost = blockSize >= Av1BlockSize.Block8x8 && + this.picture.Parent.FrameHeader.AllowScreenContentTools ? writer.GetPaletteUvModeCost(false, hasLumaPalette) : 0; @@ -209,6 +224,7 @@ internal static partial class Av1IntraSuperblockEncoder lumaMode, chromaMode, angleDelta, + blockSize, chromaOrigin, transformSize, blueSource, @@ -257,7 +273,7 @@ internal static partial class Av1IntraSuperblockEncoder } } - bool chromaFromLumaAllowed = BlockSize.AllowsChromaFromLuma( + bool chromaFromLumaAllowed = blockSize.AllowsChromaFromLuma( this.picture.Parent.FrameHeader.LosslessArray[modeInfo.Block.SegmentId], colorConfig.SubSamplingX, colorConfig.SubSamplingY); @@ -265,9 +281,15 @@ internal static partial class Av1IntraSuperblockEncoder if (this.effort >= 4 && chromaFromLumaAllowed) { Span lumaQ3 = workspace.ChromaFromLumaSamples; + Point chromaLumaOrigin = new( + chromaOrigin.X << subsamplingX, + chromaOrigin.Y << subsamplingY); + + // CfL consumes the complete luma region represented by this chroma block, which starts before + // the bottom-right ownership point for subsampled 4x4 luma leaves. TOperator.PrepareChromaFromLuma( this.reconstruction.GetPlane(Av1Plane.Y), - lumaOrigin, + chromaLumaOrigin, lumaQ3, transformSize, colorConfig.SubSamplingX, @@ -345,7 +367,7 @@ internal static partial class Av1IntraSuperblockEncoder this.picture.Parent.FrameHeader, colorConfig, modeInfo, - BlockSize, + blockSize, lumaMode, Av1ChromaPredictionMode.ChromaFromLuma, 0); @@ -457,6 +479,7 @@ internal static partial class Av1IntraSuperblockEncoder } if (this.effort >= 5 && + blockSize == Av1BlockSize.Block8x8 && this.picture.Parent.FrameHeader.AllowScreenContentTools && this.SelectChromaPalette( writer, @@ -544,6 +567,7 @@ internal static partial class Av1IntraSuperblockEncoder Av1PredictionMode lumaMode, Av1ChromaPredictionMode chromaMode, int angleDelta, + Av1BlockSize blockSize, Point chromaOrigin, Av1TransformSize transformSize, Buffer2DRegion blueSource, @@ -564,7 +588,6 @@ internal static partial class Av1IntraSuperblockEncoder ref Av1EncoderTransformBlockState candidateBlueState, ref Av1EncoderTransformBlockState candidateRedState) { - const Av1BlockSize BlockSize = Av1BlockSize.Block8x8; Av1PredictionMode predictionMode = chromaMode.ToLumaMode(); // Intra chroma derives one transform type from the shared UV prediction mode. The type is not @@ -623,7 +646,7 @@ internal static partial class Av1IntraSuperblockEncoder this.picture.Parent.FrameHeader, this.picture.Sequence.SequenceHeader.ColorConfig, modeInfo, - BlockSize, + blockSize, lumaMode, chromaMode, angleDelta); diff --git a/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1IntraSuperblockEncoder.IntraBlockCopyModeDecision.cs b/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1IntraSuperblockEncoder.IntraBlockCopyModeDecision.cs index 5c244b07a0..cb21938f9e 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1IntraSuperblockEncoder.IntraBlockCopyModeDecision.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1IntraSuperblockEncoder.IntraBlockCopyModeDecision.cs @@ -20,13 +20,12 @@ internal static partial class Av1IntraSuperblockEncoder where TSample : unmanaged where TOperator : struct, IBlockEncodingOperator { - private void SelectIntraBlockCopy( + private long SelectIntraBlockCopy( Av1SymbolEncoder writer, Av1MacroBlockD macroBlock, Point blockOrigin, ushort tileIndex, - long regularModeCost, - int regularEmptyTransformRate, + long regularCost, ref Av1MacroBlockModeInfo modeInfo, ref Av1EncoderBlockStruct block, ref Av1EncoderPaletteInfo paletteInfo) @@ -99,21 +98,11 @@ internal static partial class Av1IntraSuperblockEncoder if (uniqueCandidateCount == 0) { - return; + return regularCost; } - // Mode-decision costs already contain the selected coefficient syntax but not the block's skip - // or IBC choice. When regular intra skips, replace its empty-coefficient rate with skip syntax. int skipContext = Av1TileWriter.GetSkipContext(macroBlock); - int regularRateAdjustment = writer.GetUseIntraBlockCopyCost(false) + - writer.GetSkipCost(modeInfo.Block.Skip, skipContext); - - if (modeInfo.Block.Skip) - { - regularRateAdjustment -= regularEmptyTransformRate; - } - - long bestCost = regularModeCost + Av1RateDistortion.GetCost(this.rateMultiplier, regularRateAdjustment, 0); + long bestCost = regularCost; bool hasSelectedCandidate = false; bool selectedSkip = false; Av1MotionVector selectedVector = default; @@ -369,7 +358,7 @@ internal static partial class Av1IntraSuperblockEncoder if (!hasSelectedCandidate) { - return; + return bestCost; } // Only the winning vector is now visible to later coding blocks. This single publication keeps @@ -440,6 +429,7 @@ internal static partial class Av1IntraSuperblockEncoder block.PredictionUnit.ChromaFromLumaSigns = 0; paletteInfo = default; this.picture.SetDisplacementVector(modeInfoPosition, selectedVector); + return bestCost; } private void EvaluateIntraBlockCopyPlane( diff --git a/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1IntraSuperblockEncoder.ModeDecision.cs b/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1IntraSuperblockEncoder.ModeDecision.cs index 18cbe1a46d..157c441cda 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1IntraSuperblockEncoder.ModeDecision.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Pipeline/Av1IntraSuperblockEncoder.ModeDecision.cs @@ -120,6 +120,7 @@ internal static partial class Av1IntraSuperblockEncoder private readonly int effort; private int codedAreaLuma; private int codedAreaChroma; + private long selectedBlockCost; /// /// Initializes a new instance of the struct. @@ -152,6 +153,214 @@ internal static partial class Av1IntraSuperblockEncoder this.effort = effort; this.codedAreaLuma = 0; this.codedAreaChroma = 0; + this.selectedBlockCost = 0; + } + + /// + public Av1PartitionType SelectPartition( + Av1SymbolEncoder writer, + Av1MacroBlockD macroBlock, + Point blockOrigin, + ushort tileIndex, + Av1BlockSize blockSize, + Av1PartitionType preparedPartition) + { + if (blockSize != Av1BlockSize.Block8x8) + { + return preparedPartition; + } + + Point modeInfoPosition = blockOrigin >> Av1Constants.ModeInfoSizeLog2; + bool hasRows = modeInfoPosition.Y + 1 < this.picture.Parent.Common.ModeInfoRowCount; + bool hasColumns = modeInfoPosition.X + 1 < this.picture.Parent.Common.ModeInfoColumnCount; + if (!hasRows || !hasColumns) + { + // A clipped 8x8 node must split because its missing half cannot be represented by PARTITION_NONE. + for (int childIndex = 0; childIndex < 4; childIndex++) + { + Point childOrigin = blockOrigin + new Size( + (childIndex & 1) << Av1Constants.ModeInfoSizeLog2, + (childIndex >> 1) << Av1Constants.ModeInfoSizeLog2); + + Point childPosition = childOrigin >> Av1Constants.ModeInfoSizeLog2; + if (childPosition.Y < this.picture.Parent.Common.ModeInfoRowCount && + childPosition.X < this.picture.Parent.Common.ModeInfoColumnCount) + { + this.SetBlockGeometry(childOrigin, Av1BlockSize.Block4x4, Av1PartitionType.None); + } + } + + return Av1PartitionType.Split; + } + + if (this.effort < 9) + { + return preparedPartition; + } + + int savedLumaArea = this.codedAreaLuma; + int savedChromaArea = this.codedAreaChroma; + this.SavePartitionTrialContexts(blockOrigin, tileIndex); + long bestCost = this.EvaluatePartitionCandidate( + writer, + macroBlock, + blockOrigin, + tileIndex, + blockSize, + Av1PartitionType.None); + + Av1PartitionType selectedPartition = Av1PartitionType.None; + this.ResetPartitionTrial(blockOrigin, tileIndex, savedLumaArea, savedChromaArea); + long splitCost = this.EvaluatePartitionCandidate( + writer, + macroBlock, + blockOrigin, + tileIndex, + blockSize, + Av1PartitionType.Split); + + if (splitCost < bestCost) + { + bestCost = splitCost; + selectedPartition = Av1PartitionType.Split; + } + + this.ResetPartitionTrial(blockOrigin, tileIndex, savedLumaArea, savedChromaArea); + long horizontalCost = this.EvaluatePartitionCandidate( + writer, + macroBlock, + blockOrigin, + tileIndex, + blockSize, + Av1PartitionType.Horizontal); + + if (horizontalCost < bestCost) + { + bestCost = horizontalCost; + selectedPartition = Av1PartitionType.Horizontal; + } + + this.ResetPartitionTrial(blockOrigin, tileIndex, savedLumaArea, savedChromaArea); + long verticalCost = this.EvaluatePartitionCandidate( + writer, + macroBlock, + blockOrigin, + tileIndex, + blockSize, + Av1PartitionType.Vertical); + + if (verticalCost < bestCost) + { + selectedPartition = Av1PartitionType.Vertical; + } + + this.ResetPartitionTrial(blockOrigin, tileIndex, savedLumaArea, savedChromaArea); + + // Trial reconstruction and mode entries need no copy-back. The selected branch is evaluated again + // in raster order, overwriting each trial-local value before a later selected leaf can consume it. + this.PreparePartitionGeometry(blockOrigin, blockSize, selectedPartition); + return selectedPartition; + } + + private long EvaluatePartitionCandidate( + Av1SymbolEncoder writer, + Av1MacroBlockD macroBlock, + Point blockOrigin, + ushort tileIndex, + Av1BlockSize blockSize, + Av1PartitionType partitionType) + { + int rate = Av1TileWriter.GetPartitionCost( + this.picture, + writer, + blockSize, + partitionType, + blockOrigin, + this.picture.PartitionContexts[tileIndex]); + + long cost = Av1RateDistortion.GetCost(this.rateMultiplier, rate, 0); + int leafCount = partitionType == Av1PartitionType.Split + ? 4 + : partitionType == Av1PartitionType.None + ? 1 + : 2; + + Av1BlockSize leafSize = partitionType.GetBlockSubSize(blockSize); + + // Child reconstruction and syntax contexts become input to the next child. Publishing only + // non-final leaves reproduces libaom's raster trial without writing entropy symbols. + for (int leafIndex = 0; leafIndex < leafCount; leafIndex++) + { + Point leafOrigin = GetPartitionLeafOrigin( + blockOrigin, + blockSize, + partitionType, + leafIndex); + + cost += this.EvaluatePartitionLeaf( + writer, + macroBlock, + leafOrigin, + tileIndex, + leafSize, + publishContexts: leafIndex < leafCount - 1); + } + + return cost; + } + + private void ResetPartitionTrial( + Point blockOrigin, + ushort tileIndex, + int savedLumaArea, + int savedChromaArea) + { + this.codedAreaLuma = savedLumaArea; + this.codedAreaChroma = savedChromaArea; + this.RestorePartitionTrialContexts(blockOrigin, tileIndex); + } + + private void PreparePartitionGeometry( + Point blockOrigin, + Av1BlockSize blockSize, + Av1PartitionType partitionType) + { + int leafCount = partitionType == Av1PartitionType.Split + ? 4 + : partitionType == Av1PartitionType.None + ? 1 + : 2; + + Av1BlockSize leafSize = partitionType.GetBlockSubSize(blockSize); + for (int leafIndex = 0; leafIndex < leafCount; leafIndex++) + { + Point leafOrigin = GetPartitionLeafOrigin( + blockOrigin, + blockSize, + partitionType, + leafIndex); + + this.SetBlockGeometry(leafOrigin, leafSize, Av1PartitionType.None); + } + } + + private static Point GetPartitionLeafOrigin( + Point blockOrigin, + Av1BlockSize blockSize, + Av1PartitionType partitionType, + int leafIndex) + { + int halfWidth = blockSize.GetWidth() >> 1; + int halfHeight = blockSize.GetHeight() >> 1; + return partitionType switch + { + Av1PartitionType.Horizontal => blockOrigin + new Size(0, leafIndex * halfHeight), + Av1PartitionType.Vertical => blockOrigin + new Size(leafIndex * halfWidth, 0), + Av1PartitionType.Split => blockOrigin + new Size( + (leafIndex & 1) * halfWidth, + (leafIndex >> 1) * halfHeight), + _ => blockOrigin + }; } /// @@ -164,21 +373,25 @@ internal static partial class Av1IntraSuperblockEncoder ref Av1EncoderBlockStruct block, ref Av1EncoderPaletteInfo paletteInfo) { - const Av1BlockSize BlockSize = Av1BlockSize.Block8x8; - const Av1TransformSize LumaTransformSize = Av1TransformSize.Size8x8; + Av1BlockSize blockSize = modeInfo.Block.BlockSize; + Av1PartitionType partitionType = modeInfo.Block.PartitionType; + Av1TransformSize maximumLumaTransformSize = blockSize.GetMaximumTransformSize(); int qIndex = this.quantization.QIndex[0]; modeInfo.Block = new Av1EncoderBlockModeInfo { - BlockSize = BlockSize, - PartitionType = Av1PartitionType.None, + BlockSize = blockSize, + PartitionType = partitionType, SegmentId = 0, - TransformSize = LumaTransformSize, + TransformSize = maximumLumaTransformSize, Mode = Av1PredictionMode.DC, UvMode = Av1ChromaPredictionMode.DC }; modeInfo.CdefStrength = 0; - block.HasChroma = !this.source.IsMonochrome; + Point modeInfoPosition = blockOrigin >> Av1Constants.ModeInfoSizeLog2; + block.HasChroma = !this.source.IsMonochrome && + Av1TileReader.HasChroma(this.picture.Sequence.SequenceHeader, modeInfoPosition, blockSize); + block.QuantizationIndex = qIndex; block.SegmentId = 0; @@ -194,6 +407,7 @@ internal static partial class Av1IntraSuperblockEncoder writer, macroBlock, blockOrigin, + blockSize, tileIndex, lumaCoefficients[this.codedAreaLuma..], retainedLumaStates, @@ -207,9 +421,11 @@ internal static partial class Av1IntraSuperblockEncoder block.FilterIntraMode = filterIntraMode; modeInfo.Block.TransformSize = lumaTransformSize; - // One 8x8 coding block retains either one 8x8 transform or four 4x4 transforms. The block-level - // skip decision is legal only when every transform selected by mode decision has an empty EOB. - int lumaTransformBlockCount = LumaTransformSize.GetSize2d() / lumaTransformSize.GetSize2d(); + // Mode decision retains one state for every uniform transform tile in the coding block. The block + // can skip coefficient syntax only when every retained transform has an empty end-of-block marker. + int lumaTransformBlockCount = + (blockSize.GetWidth() * blockSize.GetHeight()) / lumaTransformSize.GetSize2d(); + bool lumaTransformEmpty = true; for (int transformIndex = 0; transformIndex < lumaTransformBlockCount; transformIndex++) { @@ -224,7 +440,7 @@ internal static partial class Av1IntraSuperblockEncoder this.picture.LuminanceDcSignLevelCoefficientNeighbors[tileIndex], Av1ComponentType.Luminance, blockOrigin, - BlockSize, + blockSize, lumaTransformSize, modeInfo.Block.Mode, block.FilterIntraMode) @@ -236,29 +452,42 @@ internal static partial class Av1IntraSuperblockEncoder Av1TileWriter.GetSkipContext(macroBlock), emptyTransformRate); - if (this.picture.Parent.FrameHeader.AllowIntraBlockCopy) - { - this.SelectIntraBlockCopy( + bool allowIntraBlockCopy = blockSize == Av1BlockSize.Block8x8 && + this.picture.Parent.FrameHeader.AllowIntraBlockCopy; + + long regularCost = this.GetRegularBlockCost( + writer, + macroBlock, + lumaCost, + emptyTransformRate, + modeInfo.Block.Skip, + allowIntraBlockCopy); + + this.selectedBlockCost = allowIntraBlockCopy + ? this.SelectIntraBlockCopy( writer, macroBlock, blockOrigin, tileIndex, - lumaCost, - emptyTransformRate, + regularCost, ref modeInfo, ref block, - ref paletteInfo); - } + ref paletteInfo) + : regularCost; - this.codedAreaLuma += LumaTransformSize.GetSize2d(); + this.codedAreaLuma += blockSize.GetWidth() * blockSize.GetHeight(); return; } ObuColorConfig colorConfig = this.picture.Sequence.SequenceHeader.ColorConfig; int subsamplingX = colorConfig.SubSamplingX ? 1 : 0; int subsamplingY = colorConfig.SubSamplingY ? 1 : 0; - Point chromaOrigin = new(blockOrigin.X >> subsamplingX, blockOrigin.Y >> subsamplingY); - Av1TransformSize chromaTransformSize = BlockSize.GetMaxUvTransformSize( + Point chromaOrigin = Av1TileWriter.GetChromaBlockOrigin( + blockOrigin, + subsamplingX, + subsamplingY); + + Av1TransformSize chromaTransformSize = blockSize.GetMaxUvTransformSize( colorConfig.SubSamplingX, colorConfig.SubSamplingY); @@ -274,36 +503,43 @@ internal static partial class Av1IntraSuperblockEncoder ref Av1EncoderTransformBlockState blueState = ref blueTransformBlocks[chromaTransformIndex]; ref Av1EncoderTransformBlockState redState = ref redTransformBlocks[chromaTransformIndex]; - modeInfo.Block.UvMode = this.SelectChromaMode( - writer, - macroBlock, - modeInfo, - blockOrigin, - chromaOrigin, - tileIndex, - modeInfo.Block.Mode, - chromaTransformSize, - blueCoefficients[this.codedAreaChroma..], - redCoefficients[this.codedAreaChroma..], - ref blueState, - ref redState, - ref paletteInfo, - out int chromaAngleDelta, - out byte chromaFromLumaIndex, - out sbyte chromaFromLumaSigns, - out long chromaCost); - - block.PredictionUnit.AngleDelta[(int)Av1PlaneType.Uv] = (sbyte)chromaAngleDelta; - block.PredictionUnit.ChromaFromLumaIndex = chromaFromLumaIndex; - block.PredictionUnit.ChromaFromLumaSigns = chromaFromLumaSigns; + long chromaCost = 0; + if (block.HasChroma) + { + modeInfo.Block.UvMode = this.SelectChromaMode( + writer, + macroBlock, + modeInfo, + blockOrigin, + chromaOrigin, + blockSize, + tileIndex, + modeInfo.Block.Mode, + chromaTransformSize, + blueCoefficients[this.codedAreaChroma..], + redCoefficients[this.codedAreaChroma..], + ref blueState, + ref redState, + ref paletteInfo, + out int chromaAngleDelta, + out byte chromaFromLumaIndex, + out sbyte chromaFromLumaSigns, + out chromaCost); + + block.PredictionUnit.AngleDelta[(int)Av1PlaneType.Uv] = (sbyte)chromaAngleDelta; + block.PredictionUnit.ChromaFromLumaIndex = chromaFromLumaIndex; + block.PredictionUnit.ChromaFromLumaSigns = chromaFromLumaSigns; + } // Skip suppresses coefficient syntax for the entire coding block, not one plane independently. // Preserve normal coefficient coding when any selected luma or chroma transform is nonempty. - bool allTransformsEmpty = lumaTransformEmpty && blueState.EndOfBlock == 0 && redState.EndOfBlock == 0; + bool allTransformsEmpty = lumaTransformEmpty && + (!block.HasChroma || (blueState.EndOfBlock == 0 && redState.EndOfBlock == 0)); + int regularEmptyTransformRate = 0; if (allTransformsEmpty) { - Av1BlockSize chromaBlockSize = BlockSize.GetSubsampled( + Av1BlockSize chromaBlockSize = blockSize.GetSubsampled( colorConfig.SubSamplingX, colorConfig.SubSamplingY); @@ -312,30 +548,33 @@ internal static partial class Av1IntraSuperblockEncoder this.picture.LuminanceDcSignLevelCoefficientNeighbors[tileIndex], Av1ComponentType.Luminance, blockOrigin, - BlockSize, + blockSize, lumaTransformSize, modeInfo.Block.Mode, block.FilterIntraMode); - regularEmptyTransformRate += this.GetEmptyTransformRate( - writer, - this.picture.CbDcSignLevelCoefficientNeighbors[tileIndex], - Av1ComponentType.Chroma, - chromaOrigin, - chromaBlockSize, - chromaTransformSize, - modeInfo.Block.Mode, - Av1FilterIntraMode.AllFilterIntraModes); + if (block.HasChroma) + { + regularEmptyTransformRate += this.GetEmptyTransformRate( + writer, + this.picture.CbDcSignLevelCoefficientNeighbors[tileIndex], + Av1ComponentType.Chroma, + chromaOrigin, + chromaBlockSize, + chromaTransformSize, + modeInfo.Block.Mode, + Av1FilterIntraMode.AllFilterIntraModes); - regularEmptyTransformRate += this.GetEmptyTransformRate( - writer, - this.picture.CrDcSignLevelCoefficientNeighbors[tileIndex], - Av1ComponentType.Chroma, - chromaOrigin, - chromaBlockSize, - chromaTransformSize, - modeInfo.Block.Mode, - Av1FilterIntraMode.AllFilterIntraModes); + regularEmptyTransformRate += this.GetEmptyTransformRate( + writer, + this.picture.CrDcSignLevelCoefficientNeighbors[tileIndex], + Av1ComponentType.Chroma, + chromaOrigin, + chromaBlockSize, + chromaTransformSize, + modeInfo.Block.Mode, + Av1FilterIntraMode.AllFilterIntraModes); + } modeInfo.Block.Skip = Av1TileWriter.ShouldSkipCoefficients( writer, @@ -343,22 +582,364 @@ internal static partial class Av1IntraSuperblockEncoder regularEmptyTransformRate); } - if (this.picture.Parent.FrameHeader.AllowIntraBlockCopy) - { - this.SelectIntraBlockCopy( + bool allowColorIntraBlockCopy = blockSize == Av1BlockSize.Block8x8 && + this.picture.Parent.FrameHeader.AllowIntraBlockCopy; + + long regularColorCost = this.GetRegularBlockCost( + writer, + macroBlock, + lumaCost + chromaCost, + regularEmptyTransformRate, + modeInfo.Block.Skip, + allowColorIntraBlockCopy); + + this.selectedBlockCost = allowColorIntraBlockCopy + ? this.SelectIntraBlockCopy( writer, macroBlock, blockOrigin, tileIndex, - lumaCost + chromaCost, - regularEmptyTransformRate, + regularColorCost, ref modeInfo, ref block, - ref paletteInfo); + ref paletteInfo) + : regularColorCost; + + this.codedAreaLuma += blockSize.GetWidth() * blockSize.GetHeight(); + if (block.HasChroma) + { + Av1BlockSize chromaBlockSize = blockSize.GetSubsampled( + colorConfig.SubSamplingX, + colorConfig.SubSamplingY); + + this.codedAreaChroma += chromaBlockSize.GetWidth() * chromaBlockSize.GetHeight(); } + } - this.codedAreaLuma += LumaTransformSize.GetSize2d(); - this.codedAreaChroma += chromaTransformSize.GetSize2d(); + private long EvaluatePartitionLeaf( + Av1SymbolEncoder writer, + Av1MacroBlockD macroBlock, + Point blockOrigin, + ushort tileIndex, + Av1BlockSize blockSize, + bool publishContexts) + { + this.SetBlockGeometry(blockOrigin, blockSize, Av1PartitionType.None); + Point modeInfoPosition = blockOrigin >> Av1Constants.ModeInfoSizeLog2; + Av1TileWriter.SetModeInfoRowAndColumn( + this.picture, + macroBlock, + macroBlock.Tile, + modeInfoPosition, + blockSize, + this.picture.Parent.Common.ModeInfoStride, + this.picture.Parent.Common.ModeInfoRowCount, + this.picture.Parent.Common.ModeInfoColumnCount); + + ref Av1MacroBlockModeInfo modeInfo = ref this.picture.GetMacroBlockModeInfo(modeInfoPosition); + Av1EncoderBlockStruct block = default; + Av1EncoderPaletteInfo paletteInfo = default; + int lumaArea = this.codedAreaLuma; + int chromaArea = this.codedAreaChroma; + this.EncodeBlock( + writer, + macroBlock, + blockOrigin, + tileIndex, + ref modeInfo, + ref block, + ref paletteInfo); + + if (publishContexts) + { + this.PublishPartitionLeafContexts( + blockOrigin, + tileIndex, + lumaArea, + chromaArea, + modeInfo, + block); + } + + return this.selectedBlockCost; + } + + private void SetBlockGeometry( + Point blockOrigin, + Av1BlockSize blockSize, + Av1PartitionType partitionType) + { + Point modeInfoPosition = blockOrigin >> Av1Constants.ModeInfoSizeLog2; + ref Av1MacroBlockModeInfo modeInfo = ref this.picture.GetMacroBlockModeInfo(modeInfoPosition); + modeInfo.Block = new Av1EncoderBlockModeInfo + { + BlockSize = blockSize, + PartitionType = partitionType + }; + + this.picture.MapModeInfoBlock(modeInfoPosition, blockSize); + } + + private void PublishPartitionLeafContexts( + Point blockOrigin, + ushort tileIndex, + int lumaArea, + int chromaArea, + Av1MacroBlockModeInfo modeInfo, + Av1EncoderBlockStruct block) + { + Av1BlockSize blockSize = modeInfo.Block.BlockSize; + Av1TransformSize transformSize = modeInfo.Block.TransformSize; + Size blockDimensions = new(blockSize.GetWidth(), blockSize.GetHeight()); + Av1NeighborArrayUnit transformContexts = this.picture.TransformFunctionContexts[tileIndex]; + transformContexts.UnitModeWrite( + (byte)transformSize.GetWidth(), + blockOrigin, + blockDimensions, + Av1NeighborArrayUnit.UnitMask.Top); + + transformContexts.UnitModeWrite( + (byte)transformSize.GetHeight(), + blockOrigin, + blockDimensions, + Av1NeighborArrayUnit.UnitMask.Left); + + Span lumaStates = + this.coefficientBuffer.GetTransformBlockSpan(this.superblock.Index, Av1Plane.Y); + + Av1EncoderTransformBlockState lumaState = + lumaStates[lumaArea / Av1EncoderCoefficientBuffer.TransformBlockUnitCoefficientCount]; + + Span lumaCoefficients = this.coefficientBuffer.GetPlaneSpan(this.superblock.Index, Av1Plane.Y); + byte lumaContext = Av1SymbolContextHelper.GetCoefficientContext( + lumaCoefficients[lumaArea..], + transformSize, + lumaState.TransformType, + lumaState.EndOfBlock); + + this.picture.LuminanceDcSignLevelCoefficientNeighbors[tileIndex].UnitModeWrite( + lumaContext, + blockOrigin, + blockDimensions, + Av1NeighborArrayUnit.UnitMask.Top | Av1NeighborArrayUnit.UnitMask.Left); + + if (!block.HasChroma) + { + return; + } + + ObuColorConfig colorConfig = this.picture.Sequence.SequenceHeader.ColorConfig; + int subsamplingX = colorConfig.SubSamplingX ? 1 : 0; + int subsamplingY = colorConfig.SubSamplingY ? 1 : 0; + Point chromaOrigin = Av1TileWriter.GetChromaBlockOrigin( + blockOrigin, + subsamplingX, + subsamplingY); + + Av1BlockSize chromaBlockSize = blockSize.GetSubsampled( + colorConfig.SubSamplingX, + colorConfig.SubSamplingY); + + Av1TransformSize chromaTransformSize = blockSize.GetMaxUvTransformSize( + colorConfig.SubSamplingX, + colorConfig.SubSamplingY); + + int chromaStateIndex = + chromaArea / Av1EncoderCoefficientBuffer.TransformBlockUnitCoefficientCount; + + Span blueStates = + this.coefficientBuffer.GetTransformBlockSpan(this.superblock.Index, Av1Plane.U); + + Span redStates = + this.coefficientBuffer.GetTransformBlockSpan(this.superblock.Index, Av1Plane.V); + + Av1EncoderTransformBlockState blueState = blueStates[chromaStateIndex]; + Av1EncoderTransformBlockState redState = redStates[chromaStateIndex]; + Span blueCoefficients = this.coefficientBuffer.GetPlaneSpan(this.superblock.Index, Av1Plane.U); + Span redCoefficients = this.coefficientBuffer.GetPlaneSpan(this.superblock.Index, Av1Plane.V); + byte blueContext = Av1SymbolContextHelper.GetCoefficientContext( + blueCoefficients[chromaArea..], + chromaTransformSize, + blueState.TransformType, + blueState.EndOfBlock); + + byte redContext = Av1SymbolContextHelper.GetCoefficientContext( + redCoefficients[chromaArea..], + chromaTransformSize, + redState.TransformType, + redState.EndOfBlock); + + Size chromaDimensions = new(chromaBlockSize.GetWidth(), chromaBlockSize.GetHeight()); + this.picture.CbDcSignLevelCoefficientNeighbors[tileIndex].UnitModeWrite( + blueContext, + chromaOrigin, + chromaDimensions, + Av1NeighborArrayUnit.UnitMask.Top | Av1NeighborArrayUnit.UnitMask.Left); + + this.picture.CrDcSignLevelCoefficientNeighbors[tileIndex].UnitModeWrite( + redContext, + chromaOrigin, + chromaDimensions, + Av1NeighborArrayUnit.UnitMask.Top | Av1NeighborArrayUnit.UnitMask.Left); + } + + private void SavePartitionTrialContexts(Point blockOrigin, ushort tileIndex) + { + Span storage = this.blockWorkspace.PartitionContexts; + int offset = 0; + SaveNeighborEdges( + this.picture.LuminanceDcSignLevelCoefficientNeighbors[tileIndex], + blockOrigin, + Av1BlockSize.Block8x8.Get4x4WideCount(), + Av1BlockSize.Block8x8.Get4x4HighCount(), + storage, + ref offset); + + SaveNeighborEdges( + this.picture.TransformFunctionContexts[tileIndex], + blockOrigin, + Av1BlockSize.Block8x8.Get4x4WideCount(), + Av1BlockSize.Block8x8.Get4x4HighCount(), + storage, + ref offset); + + if (this.source.IsMonochrome) + { + return; + } + + ObuColorConfig colorConfig = this.picture.Sequence.SequenceHeader.ColorConfig; + int subsamplingX = colorConfig.SubSamplingX ? 1 : 0; + int subsamplingY = colorConfig.SubSamplingY ? 1 : 0; + Point chromaOrigin = Av1TileWriter.GetChromaBlockOrigin( + blockOrigin, + subsamplingX, + subsamplingY); + + Av1BlockSize chromaBlockSize = Av1BlockSize.Block8x8.GetSubsampled( + colorConfig.SubSamplingX, + colorConfig.SubSamplingY); + + SaveNeighborEdges( + this.picture.CbDcSignLevelCoefficientNeighbors[tileIndex], + chromaOrigin, + chromaBlockSize.Get4x4WideCount(), + chromaBlockSize.Get4x4HighCount(), + storage, + ref offset); + + SaveNeighborEdges( + this.picture.CrDcSignLevelCoefficientNeighbors[tileIndex], + chromaOrigin, + chromaBlockSize.Get4x4WideCount(), + chromaBlockSize.Get4x4HighCount(), + storage, + ref offset); + } + + private void RestorePartitionTrialContexts(Point blockOrigin, ushort tileIndex) + { + ReadOnlySpan storage = this.blockWorkspace.PartitionContexts; + int offset = 0; + RestoreNeighborEdges( + this.picture.LuminanceDcSignLevelCoefficientNeighbors[tileIndex], + blockOrigin, + Av1BlockSize.Block8x8.Get4x4WideCount(), + Av1BlockSize.Block8x8.Get4x4HighCount(), + storage, + ref offset); + + RestoreNeighborEdges( + this.picture.TransformFunctionContexts[tileIndex], + blockOrigin, + Av1BlockSize.Block8x8.Get4x4WideCount(), + Av1BlockSize.Block8x8.Get4x4HighCount(), + storage, + ref offset); + + if (this.source.IsMonochrome) + { + return; + } + + ObuColorConfig colorConfig = this.picture.Sequence.SequenceHeader.ColorConfig; + int subsamplingX = colorConfig.SubSamplingX ? 1 : 0; + int subsamplingY = colorConfig.SubSamplingY ? 1 : 0; + Point chromaOrigin = Av1TileWriter.GetChromaBlockOrigin( + blockOrigin, + subsamplingX, + subsamplingY); + + Av1BlockSize chromaBlockSize = Av1BlockSize.Block8x8.GetSubsampled( + colorConfig.SubSamplingX, + colorConfig.SubSamplingY); + + RestoreNeighborEdges( + this.picture.CbDcSignLevelCoefficientNeighbors[tileIndex], + chromaOrigin, + chromaBlockSize.Get4x4WideCount(), + chromaBlockSize.Get4x4HighCount(), + storage, + ref offset); + + RestoreNeighborEdges( + this.picture.CrDcSignLevelCoefficientNeighbors[tileIndex], + chromaOrigin, + chromaBlockSize.Get4x4WideCount(), + chromaBlockSize.Get4x4HighCount(), + storage, + ref offset); + } + + private static void SaveNeighborEdges( + Av1NeighborArrayUnit neighbors, + Point blockOrigin, + int width, + int height, + Span storage, + ref int offset) + { + neighbors.Top.Slice(neighbors.GetTopIndex(blockOrigin), width).CopyTo(storage[offset..]); + offset += width; + neighbors.Left.Slice(neighbors.GetLeftIndex(blockOrigin), height).CopyTo(storage[offset..]); + offset += height; + } + + private static void RestoreNeighborEdges( + Av1NeighborArrayUnit neighbors, + Point blockOrigin, + int width, + int height, + ReadOnlySpan storage, + ref int offset) + { + storage.Slice(offset, width).CopyTo(neighbors.Top[neighbors.GetTopIndex(blockOrigin)..]); + offset += width; + storage.Slice(offset, height).CopyTo(neighbors.Left[neighbors.GetLeftIndex(blockOrigin)..]); + offset += height; + } + + private long GetRegularBlockCost( + Av1SymbolEncoder writer, + Av1MacroBlockD macroBlock, + long modeCost, + int emptyTransformRate, + bool skip, + bool allowIntraBlockCopy) + { + int rateAdjustment = writer.GetSkipCost(skip, Av1TileWriter.GetSkipContext(macroBlock)); + if (allowIntraBlockCopy) + { + rateAdjustment += writer.GetUseIntraBlockCopyCost(false); + } + + if (skip) + { + // Mode search includes empty transform symbols, while block skip suppresses them from the bitstream. + rateAdjustment -= emptyTransformRate; + } + + return modeCost + Av1RateDistortion.GetCost(this.rateMultiplier, rateAdjustment, 0); } private int GetEmptyTransformRate( @@ -443,6 +1024,7 @@ internal static partial class Av1IntraSuperblockEncoder Av1SymbolEncoder writer, Av1MacroBlockD macroBlock, Point blockOrigin, + Av1BlockSize blockSize, ushort tileIndex, Span retainedCoefficients, Span retainedStates, @@ -452,8 +1034,9 @@ internal static partial class Av1IntraSuperblockEncoder out Av1TransformSize selectedTransformSize, out long selectedCost) { - const Av1BlockSize BlockSize = Av1BlockSize.Block8x8; - const Av1TransformSize TransformSize = Av1TransformSize.Size8x8; + Av1TransformSize transformSize = blockSize.GetMaximumTransformSize(); + int blockWidth = blockSize.GetWidth(); + int blockHeight = blockSize.GetHeight(); Av1EncoderModeDecisionWorkspace workspace = this.blockWorkspace.GetModeDecisionWorkspace(); @@ -463,17 +1046,17 @@ internal static partial class Av1IntraSuperblockEncoder bool hasAbove = macroBlock.IsUpAvailable; int modeInfoRow = blockOrigin.Y >> Av1Constants.ModeInfoSizeLog2; int modeInfoColumn = blockOrigin.X >> Av1Constants.ModeInfoSizeLog2; - bool rightAvailable = modeInfoColumn + TransformSize.Get4x4WideCount() < macroBlock.Tile.ModeInfoColumnEnd; - bool bottomAvailable = modeInfoRow + TransformSize.Get4x4HighCount() < macroBlock.Tile.ModeInfoRowEnd; + bool rightAvailable = modeInfoColumn + transformSize.Get4x4WideCount() < macroBlock.Tile.ModeInfoColumnEnd; + bool bottomAvailable = modeInfoRow + transformSize.Get4x4HighCount() < macroBlock.Tile.ModeInfoRowEnd; bool hasTopRight = Av1IntraReferenceAvailability.HasTopRight( this.picture.Sequence.SequenceHeader.SuperblockSize, - BlockSize, + blockSize, modeInfoRow, modeInfoColumn, hasAbove, rightAvailable, Av1PartitionType.None, - TransformSize, + transformSize, 0, 0, 0, @@ -481,13 +1064,13 @@ internal static partial class Av1IntraSuperblockEncoder bool hasBottomLeft = Av1IntraReferenceAvailability.HasBottomLeft( this.picture.Sequence.SequenceHeader.SuperblockSize, - BlockSize, + blockSize, modeInfoRow, modeInfoColumn, bottomAvailable, hasLeft, Av1PartitionType.None, - TransformSize, + transformSize, 0, 0, 0, @@ -500,12 +1083,14 @@ internal static partial class Av1IntraSuperblockEncoder if (hasAbove) { - reconstructionPlane.DangerousGetRowSpan(blockOrigin.Y - 1).Slice(blockOrigin.X, 8).CopyTo(above[..8]); + reconstructionPlane.DangerousGetRowSpan(blockOrigin.Y - 1) + .Slice(blockOrigin.X, blockWidth) + .CopyTo(above[..blockWidth]); } if (hasLeft) { - for (int row = 0; row < 8; row++) + for (int row = 0; row < blockHeight; row++) { left[row] = reconstructionPlane.DangerousGetRowSpan(blockOrigin.Y + row)[blockOrigin.X - 1]; } @@ -517,33 +1102,35 @@ internal static partial class Av1IntraSuperblockEncoder // available uses the asymmetric midpoint offsets that distinguish top from left. if (!hasAbove) { - above[..8].Fill(hasLeft ? left[0] : TOperator.CreateSample(midpoint - 1)); + above[..blockWidth].Fill(hasLeft ? left[0] : TOperator.CreateSample(midpoint - 1)); } if (!hasLeft) { - left[..8].Fill(hasAbove ? above[0] : TOperator.CreateSample(midpoint + 1)); + left[..blockHeight].Fill(hasAbove ? above[0] : TOperator.CreateSample(midpoint + 1)); } if (hasTopRight) { - reconstructionPlane.DangerousGetRowSpan(blockOrigin.Y - 1).Slice(blockOrigin.X + 8, 8).CopyTo(above[8..]); + reconstructionPlane.DangerousGetRowSpan(blockOrigin.Y - 1) + .Slice(blockOrigin.X + blockWidth, blockWidth) + .CopyTo(above.Slice(blockWidth, blockWidth)); } else { - above[8..].Fill(above[7]); + above.Slice(blockWidth, blockWidth).Fill(above[blockWidth - 1]); } if (hasBottomLeft) { - for (int row = 8; row < 16; row++) + for (int row = blockHeight; row < blockHeight * 2; row++) { left[row] = reconstructionPlane.DangerousGetRowSpan(blockOrigin.Y + row)[blockOrigin.X - 1]; } } else { - left[8..].Fill(left[7]); + left.Slice(blockHeight, blockHeight).Fill(left[blockHeight - 1]); } // Zone-two projection and Paeth address the common corner immediately before both prepared edges. @@ -563,24 +1150,26 @@ internal static partial class Av1IntraSuperblockEncoder Av1ComponentType.Luminance, this.picture.LuminanceDcSignLevelCoefficientNeighbors[tileIndex], blockOrigin, - BlockSize, - TransformSize); + blockSize, + transformSize); int transformSizeContext = Av1TileWriter.GetTransformSizeContext( this.picture.TransformFunctionContexts[tileIndex], macroBlock, blockOrigin, - BlockSize); + blockSize); int largestTransformRate = this.picture.Parent.FrameHeader.TransformMode == Av1TransformMode.Select - ? writer.GetTransformSizeCost(BlockSize, TransformSize, transformSizeContext) + && blockSize > Av1BlockSize.Block4x4 + ? writer.GetTransformSizeCost(blockSize, transformSize, transformSizeContext) : 0; int paletteDisabledCost = 0; - if (this.picture.Parent.FrameHeader.AllowScreenContentTools) + if (blockSize >= Av1BlockSize.Block8x8 && + this.picture.Parent.FrameHeader.AllowScreenContentTools) { Av1NeighborArrayUnit paletteContexts = this.picture.PaletteContexts[tileIndex]; - int blockSizeContext = Av1TileWriter.GetPaletteBlockSizeContext(BlockSize); + int blockSizeContext = Av1TileWriter.GetPaletteBlockSizeContext(blockSize); int neighborContext = Av1TileWriter.GetPaletteYModeContext( paletteContexts, macroBlock, @@ -600,7 +1189,7 @@ internal static partial class Av1IntraSuperblockEncoder Av1PredictionMode bestMode = Av1PredictionMode.DC; selectedAngleDelta = 0; selectedFilterIntraMode = Av1FilterIntraMode.AllFilterIntraModes; - selectedTransformSize = TransformSize; + selectedTransformSize = transformSize; int baseModeCount = LumaModeSearchOrder.Length; int deltaCount = AngleDeltaSearchOrder.Length; int directionalModeCount = (int)Av1PredictionMode.Directional67Degrees - (int)Av1PredictionMode.Vertical + 1; @@ -610,18 +1199,20 @@ internal static partial class Av1IntraSuperblockEncoder { 0 => 1, 1 => baseModeCount, - _ => baseModeCount + (directionalModeCount * deltaCount) + _ when blockSize >= Av1BlockSize.Block8x8 => baseModeCount + (directionalModeCount * deltaCount), + _ => baseModeCount }; bool useReducedTransformSet = this.picture.Parent.FrameHeader.UseReducedTransformSet; Av1TransformSetType transformSetType = Av1SymbolContextHelper.GetExtendedTransformSetType( - TransformSize, + transformSize, useReducedTransformSet); // Transform type and transform size are separate search axes. Splitting their effort thresholds // gives callers a useful intermediate tier without changing the fast default path. bool searchEveryTransformType = this.effort >= 7; - bool searchEveryTransformSize = this.effort >= 8; + bool searchEveryTransformSize = this.effort >= 8 && + transformSize == Av1TransformSize.Size8x8; // Zero-angle modes precede groups of six nonzero adjustments for each directional mode. // A single index preserves that tie-breaking order without duplicating candidate evaluation. @@ -655,7 +1246,7 @@ internal static partial class Av1IntraSuperblockEncoder mode, angleDelta, residual, - TransformSize, + transformSize, this.bitDepth); // Transform types are visited in AV1 enumeration order. A strict cost comparison below keeps @@ -664,7 +1255,7 @@ internal static partial class Av1IntraSuperblockEncoder ? Av1TransformType.DctDct : Av1SymbolContextHelper.GetDefaultIntraTransformType( mode, - TransformSize, + transformSize, useReducedTransformSet); Av1TransformType transformTypeLimit = searchEveryTransformType @@ -690,6 +1281,8 @@ internal static partial class Av1IntraSuperblockEncoder residual, mode, angleDelta, + blockSize, + transformSize, transformType, blockContext, paletteDisabledCost, @@ -708,14 +1301,14 @@ internal static partial class Av1IntraSuperblockEncoder reconstructionPlane, blockOrigin, retainedCoefficients, - TransformSize, + transformSize, candidateState, ref retainedStates[0]); bestCost = candidateCost; bestMode = mode; selectedAngleDelta = angleDelta; - selectedTransformSize = TransformSize; + selectedTransformSize = transformSize; } } @@ -782,7 +1375,7 @@ internal static partial class Av1IntraSuperblockEncoder bestMode, selectedAngleDelta, residual, - TransformSize, + transformSize, this.bitDepth); for (Av1TransformType transformType = Av1TransformType.DctDct; @@ -804,6 +1397,8 @@ internal static partial class Av1IntraSuperblockEncoder residual, bestMode, selectedAngleDelta, + blockSize, + transformSize, transformType, blockContext, paletteDisabledCost, @@ -820,7 +1415,7 @@ internal static partial class Av1IntraSuperblockEncoder reconstructionPlane, blockOrigin, retainedCoefficients, - TransformSize, + transformSize, candidateState, ref retainedStates[0]); @@ -829,7 +1424,8 @@ internal static partial class Av1IntraSuperblockEncoder } } - if (this.effort >= 4 && this.picture.Sequence.SequenceHeader.EnableFilterIntra) + if (this.effort >= 4 && + this.picture.Sequence.SequenceHeader.EnableFilterIntra) { // Each recursive filter prediction and its source residual are independent of transform type. // Prepare them once per filter mode so all legal transforms reuse the same samples. @@ -846,7 +1442,7 @@ internal static partial class Av1IntraSuperblockEncoder left, residual, filterIntraMode, - TransformSize, + transformSize, this.bitDepth); for (Av1TransformType transformType = Av1TransformType.DctDct; @@ -867,6 +1463,8 @@ internal static partial class Av1IntraSuperblockEncoder prediction, residual, filterIntraMode, + blockSize, + transformSize, transformType, blockContext, paletteDisabledCost, @@ -883,7 +1481,7 @@ internal static partial class Av1IntraSuperblockEncoder reconstructionPlane, blockOrigin, retainedCoefficients, - TransformSize, + transformSize, candidateState, ref retainedStates[0]); @@ -891,7 +1489,7 @@ internal static partial class Av1IntraSuperblockEncoder bestMode = Av1PredictionMode.DC; selectedAngleDelta = 0; selectedFilterIntraMode = filterIntraMode; - selectedTransformSize = TransformSize; + selectedTransformSize = transformSize; } } @@ -942,6 +1540,7 @@ internal static partial class Av1IntraSuperblockEncoder } if (this.effort >= 5 && + blockSize == Av1BlockSize.Block8x8 && this.picture.Parent.FrameHeader.AllowScreenContentTools && this.SelectLumaPalette( writer, @@ -970,6 +1569,7 @@ internal static partial class Av1IntraSuperblockEncoder // Efforts six and seven save work by testing transform size only for the global non-palette // winner. Effort eight and above already tested both sizes inside every candidate. if (this.effort >= 6 && + transformSize == Av1TransformSize.Size8x8 && !searchEveryTransformSize && this.picture.Parent.FrameHeader.TransformMode == Av1TransformMode.Select && paletteInfo.PaletteSizes[0] == 0) @@ -1477,6 +2077,8 @@ internal static partial class Av1IntraSuperblockEncoder ReadOnlySpan residual, Av1PredictionMode mode, int angleDelta, + Av1BlockSize blockSize, + Av1TransformSize transformSize, Av1TransformType transformType, Av1TransformBlockContext blockContext, int paletteDisabledCost, @@ -1485,9 +2087,6 @@ internal static partial class Av1IntraSuperblockEncoder Span candidateCoefficients, ref Av1EncoderTransformBlockState candidateState) { - const Av1BlockSize BlockSize = Av1BlockSize.Block8x8; - const Av1TransformSize TransformSize = Av1TransformSize.Size8x8; - // Prediction and subtraction were prepared by the owning mode loop. This stage performs only // transform, quantization, reconstruction, and distortion for the requested transform type. long distortion = TOperator.EncodePredictionCandidate( @@ -1497,9 +2096,9 @@ internal static partial class Av1IntraSuperblockEncoder prediction, residual, candidateReconstruction, - TransformSize.GetWidth(), + transformSize.GetWidth(), candidateCoefficients, - TransformSize, + transformSize, transformType, Av1Plane.Y, this.quantization.QIndex[0], @@ -1510,7 +2109,7 @@ internal static partial class Av1IntraSuperblockEncoder // Charge every block-level choice that distinguishes this spatial candidate before adding // coefficient syntax derived from the live neighboring-transform context. - int rate = Av1TileWriter.GetLumaModeCost(writer, macroBlock, BlockSize, mode, angleDelta); + int rate = Av1TileWriter.GetLumaModeCost(writer, macroBlock, blockSize, mode, angleDelta); rate += transformSizeRate; if (mode == Av1PredictionMode.DC) { @@ -1519,11 +2118,11 @@ internal static partial class Av1IntraSuperblockEncoder if (mode == Av1PredictionMode.DC && this.picture.Sequence.SequenceHeader.EnableFilterIntra) { - rate += writer.GetFilterIntraModeCost(Av1FilterIntraMode.AllFilterIntraModes, BlockSize); + rate += writer.GetFilterIntraModeCost(Av1FilterIntraMode.AllFilterIntraModes, blockSize); } rate += writer.GetCoefficientCost( - TransformSize, + transformSize, transformType, mode, candidateCoefficients, @@ -1545,6 +2144,8 @@ internal static partial class Av1IntraSuperblockEncoder ReadOnlySpan prediction, ReadOnlySpan residual, Av1FilterIntraMode filterIntraMode, + Av1BlockSize blockSize, + Av1TransformSize transformSize, Av1TransformType transformType, Av1TransformBlockContext blockContext, int paletteDisabledCost, @@ -1553,8 +2154,6 @@ internal static partial class Av1IntraSuperblockEncoder Span candidateCoefficients, ref Av1EncoderTransformBlockState candidateState) { - const Av1BlockSize BlockSize = Av1BlockSize.Block8x8; - const Av1TransformSize TransformSize = Av1TransformSize.Size8x8; long distortion = TOperator.EncodePredictionCandidate( this.blockWorkspace, sourcePlane, @@ -1562,9 +2161,9 @@ internal static partial class Av1IntraSuperblockEncoder prediction, residual, candidateReconstruction, - TransformSize.GetWidth(), + transformSize.GetWidth(), candidateCoefficients, - TransformSize, + transformSize, transformType, Av1Plane.Y, this.quantization.QIndex[0], @@ -1576,15 +2175,15 @@ internal static partial class Av1IntraSuperblockEncoder int rate = Av1TileWriter.GetLumaModeCost( writer, macroBlock, - BlockSize, + blockSize, Av1PredictionMode.DC, 0); rate += transformSizeRate; rate += paletteDisabledCost; - rate += writer.GetFilterIntraModeCost(filterIntraMode, BlockSize); + rate += writer.GetFilterIntraModeCost(filterIntraMode, blockSize); rate += writer.GetCoefficientCost( - TransformSize, + transformSize, transformType, Av1PredictionMode.DC, candidateCoefficients, diff --git a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1EncoderPictureBuffer.cs b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1EncoderPictureBuffer.cs index c968f85bea..2b2ed29afa 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1EncoderPictureBuffer.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1EncoderPictureBuffer.cs @@ -32,19 +32,21 @@ internal sealed class Av1EncoderPictureBuffer : IDisposable /// The frame header defining dimensions and tiles. /// The visible luma width. /// The visible luma height. + /// Whether each allocated mode-information value represents an 8x8 region. public Av1EncoderPictureBuffer( Configuration configuration, ObuSequenceHeader sequenceHeader, ObuFrameHeader frameHeader, int width, - int height) + int height, + bool disallow4x4AllFrames) { const int ContextAlignmentLog2 = Av1Constants.MaxSuperBlockSizeLog2 - Av1Constants.ModeInfoSizeLog2; this.modeInfo = new Av1EncoderModeInfoBuffer( configuration, width, height, - disallow4x4AllFrames: true); + disallow4x4AllFrames); int alignedModeInfoRowCount = Av1Math.AlignPowerOf2(this.modeInfo.ModeInfoRowCount, ContextAlignmentLog2); int lumaLeftLength = alignedModeInfoRowCount; diff --git a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileWriter.BlockEncoding.cs b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileWriter.BlockEncoding.cs index fe64bd6c2c..fd44f70736 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileWriter.BlockEncoding.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileWriter.BlockEncoding.cs @@ -15,6 +15,24 @@ internal partial class Av1TileWriter /// internal interface IBlockEncodingHandler { + /// + /// Selects the partition used for the current tree node. + /// + /// The live tile symbol encoder. + /// The tile-local macroblock state. + /// The absolute luma-sample origin. + /// The zero-based tile index. + /// The current square partition size. + /// The partition retained before live analysis. + /// The partition to encode. + Av1PartitionType SelectPartition( + Av1SymbolEncoder writer, + Av1MacroBlockD macroBlock, + Point blockOrigin, + ushort tileIndex, + Av1BlockSize blockSize, + Av1PartitionType preparedPartition); + /// /// Encodes one final block against the current reconstructed neighbors and live tile probabilities. /// @@ -37,6 +55,17 @@ internal partial class Av1TileWriter private readonly struct PrecomputedBlockEncodingHandler : IBlockEncodingHandler { + /// + public Av1PartitionType SelectPartition( + Av1SymbolEncoder writer, + Av1MacroBlockD macroBlock, + Point blockOrigin, + ushort tileIndex, + Av1BlockSize blockSize, + Av1PartitionType preparedPartition) + => preparedPartition; + + /// public void EncodeBlock( Av1SymbolEncoder writer, Av1MacroBlockD macroBlock, diff --git a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileWriter.cs b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileWriter.cs index 48b4531882..364e2b7940 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileWriter.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileWriter.cs @@ -149,7 +149,19 @@ internal partial class Av1TileWriter return; } - Av1PartitionType partition = (Av1PartitionType)superblock.CodingUnitPartitionTypes[partitionIndex++]; + int currentPartitionIndex = partitionIndex++; + Av1PartitionType preparedPartition = + (Av1PartitionType)superblock.CodingUnitPartitionTypes[currentPartitionIndex]; + + Av1PartitionType partition = blockEncoder.SelectPartition( + writer, + entropyCodingContext.MacroBlock, + blockOrigin, + tileIndex, + blockSize, + preparedPartition); + + superblock.CodingUnitPartitionTypes[currentPartitionIndex] = (byte)partition; Av1BlockSize subSize = partition.GetBlockSubSize(blockSize); int halfBlockSize = blockSize.GetWidth() >> 1; int quarterBlockSize = blockSize.GetWidth() >> 2; @@ -232,54 +244,89 @@ internal partial class Av1TileWriter break; case Av1PartitionType.Split: - WritePartitionTree( - pcs, - entropyCodingContext, - writer, - superblock, - coefficientBuffer, - tileIndex, - subSize, - blockOrigin, - ref partitionIndex, - ref finalBlockIndex, - ref blockEncoder); - WritePartitionTree( - pcs, - entropyCodingContext, - writer, - superblock, - coefficientBuffer, - tileIndex, - subSize, - blockOrigin + new Size(halfBlockSize, 0), - ref partitionIndex, - ref finalBlockIndex, - ref blockEncoder); - WritePartitionTree( - pcs, - entropyCodingContext, - writer, - superblock, - coefficientBuffer, - tileIndex, - subSize, - blockOrigin + new Size(0, halfBlockSize), - ref partitionIndex, - ref finalBlockIndex, - ref blockEncoder); - WritePartitionTree( - pcs, - entropyCodingContext, - writer, - superblock, - coefficientBuffer, - tileIndex, - subSize, - blockOrigin + new Size(halfBlockSize, halfBlockSize), - ref partitionIndex, - ref finalBlockIndex, - ref blockEncoder); + if (blockSize == Av1BlockSize.Block8x8) + { + // A split 8x8 node terminates in four 4x4 coding blocks. AV1 does not carry another + // partition symbol at that size, so the children are final blocks rather than tree nodes. + for (int childIndex = 0; childIndex < 4; childIndex++) + { + Point childOrigin = blockOrigin + new Size( + (childIndex & 1) * halfBlockSize, + (childIndex >> 1) * halfBlockSize); + + Point childModeInfoPosition = childOrigin >> Av1Constants.ModeInfoSizeLog2; + if (childModeInfoPosition.Y >= common.ModeInfoRowCount || + childModeInfoPosition.X >= common.ModeInfoColumnCount) + { + continue; + } + + WriteFinalBlock( + pcs, + entropyCodingContext, + writer, + superblock, + coefficientBuffer, + tileIndex, + childOrigin, + ref finalBlockIndex, + ref blockEncoder); + } + } + else + { + WritePartitionTree( + pcs, + entropyCodingContext, + writer, + superblock, + coefficientBuffer, + tileIndex, + subSize, + blockOrigin, + ref partitionIndex, + ref finalBlockIndex, + ref blockEncoder); + + WritePartitionTree( + pcs, + entropyCodingContext, + writer, + superblock, + coefficientBuffer, + tileIndex, + subSize, + blockOrigin + new Size(halfBlockSize, 0), + ref partitionIndex, + ref finalBlockIndex, + ref blockEncoder); + + WritePartitionTree( + pcs, + entropyCodingContext, + writer, + superblock, + coefficientBuffer, + tileIndex, + subSize, + blockOrigin + new Size(0, halfBlockSize), + ref partitionIndex, + ref finalBlockIndex, + ref blockEncoder); + + WritePartitionTree( + pcs, + entropyCodingContext, + writer, + superblock, + coefficientBuffer, + tileIndex, + subSize, + blockOrigin + new Size(halfBlockSize, halfBlockSize), + ref partitionIndex, + ref finalBlockIndex, + ref blockEncoder); + } break; case Av1PartitionType.HorizontalA: @@ -592,6 +639,47 @@ internal partial class Av1TileWriter edgeMask); } + /// + /// Gets the partition-symbol rate from the above and left contexts available at a block origin. + /// + /// The picture coding state. + /// The live tile symbol encoder. + /// The square parent block size. + /// The partition type to measure. + /// The block origin in samples. + /// The partition neighbor arrays for the tile. + /// The rate cost in 1/512-bit units. + public static int GetPartitionCost( + Av1PictureControlSet pcs, + Av1SymbolEncoder writer, + Av1BlockSize blockSize, + Av1PartitionType partitionType, + Point blockOrigin, + Av1NeighborArrayUnit partitionContexts) + { + int context = GetPartitionContext( + pcs, + blockSize, + blockOrigin, + partitionContexts, + out bool hasRows, + out bool hasColumns); + + if (!hasRows && !hasColumns) + { + return 0; + } + + if (hasRows && hasColumns) + { + return writer.GetPartitionTypeCost(partitionType, context); + } + + return !hasRows + ? writer.GetSplitOrHorizontalCost(partitionType, blockSize, context) + : writer.GetSplitOrVerticalCost(partitionType, blockSize, context); + } + /// /// Writes a partition symbol using the above and left partition contexts available at a block origin. /// @@ -616,33 +704,13 @@ internal partial class Av1TileWriter return; } - int halfBlockModeInfoCount = blockSize.Get4x4WideCount() >> 1; - Point modeInfoPosition = blockOrigin >> Av1Constants.ModeInfoSizeLog2; - bool has_rows = modeInfoPosition.Y + halfBlockModeInfoCount < pcs.Parent.Common.ModeInfoRowCount; - bool has_cols = modeInfoPosition.X + halfBlockModeInfoCount < pcs.Parent.Common.ModeInfoColumnCount; - - int partition_context_left_neighbor_index = partition_context_na.GetLeftIndex(blockOrigin); - int partition_context_top_neighbor_index = partition_context_na.GetTopIndex(blockOrigin); - - int context_index = 0; - - byte above_ctx = - (byte)(partition_context_na.Top[partition_context_top_neighbor_index].Above == byte.MaxValue - ? 0 - : partition_context_na.Top[partition_context_top_neighbor_index].Above); - byte left_ctx = - (byte)(partition_context_na.Left[partition_context_left_neighbor_index].Left == byte.MaxValue - ? 0 - : partition_context_na.Left[partition_context_left_neighbor_index].Left); - - int blockSizeLog2 = blockSize.Get4x4WidthLog2() - 1; - int above = (above_ctx >> blockSizeLog2) & 1, left = (left_ctx >> blockSizeLog2) & 1; - - Guard.IsTrue(blockSize.Get4x4WidthLog2() == blockSize.Get4x4HeightLog2(), nameof(blockSize), "Blocks need to be square."); - Guard.IsTrue(blockSizeLog2 >= 0, nameof(blockSizeLog2), "bsl needs to be a positive integer."); - - // Each square block-size level owns four contexts selected by the current split bit of its neighbors. - context_index = ((left * 2) + above) + (blockSizeLog2 * Av1Constants.PartitionProbabilitySet); + int context_index = GetPartitionContext( + pcs, + blockSize, + blockOrigin, + partition_context_na, + out bool has_rows, + out bool has_cols); if (!has_rows && !has_cols) { @@ -666,6 +734,39 @@ internal partial class Av1TileWriter return; } + private static int GetPartitionContext( + Av1PictureControlSet pcs, + Av1BlockSize blockSize, + Point blockOrigin, + Av1NeighborArrayUnit partitionContexts, + out bool hasRows, + out bool hasColumns) + { + int halfBlockModeInfoCount = blockSize.Get4x4WideCount() >> 1; + Point modeInfoPosition = blockOrigin >> Av1Constants.ModeInfoSizeLog2; + hasRows = modeInfoPosition.Y + halfBlockModeInfoCount < pcs.Parent.Common.ModeInfoRowCount; + hasColumns = modeInfoPosition.X + halfBlockModeInfoCount < pcs.Parent.Common.ModeInfoColumnCount; + int leftIndex = partitionContexts.GetLeftIndex(blockOrigin); + int topIndex = partitionContexts.GetTopIndex(blockOrigin); + byte aboveContext = partitionContexts.Top[topIndex].Above == byte.MaxValue + ? (byte)0 + : partitionContexts.Top[topIndex].Above; + + byte leftContext = partitionContexts.Left[leftIndex].Left == byte.MaxValue + ? (byte)0 + : partitionContexts.Left[leftIndex].Left; + + int blockSizeLog2 = blockSize.Get4x4WidthLog2() - 1; + int above = (aboveContext >> blockSizeLog2) & 1; + int left = (leftContext >> blockSizeLog2) & 1; + + Guard.IsTrue(blockSize.Get4x4WidthLog2() == blockSize.Get4x4HeightLog2(), nameof(blockSize), "Blocks need to be square."); + Guard.IsTrue(blockSizeLog2 >= 0, nameof(blockSizeLog2), "bsl needs to be a positive integer."); + + // Each square block-size level owns four contexts selected by the current split bit of its neighbors. + return ((left * 2) + above) + (blockSizeLog2 * Av1Constants.PartitionProbabilitySet); + } + /// /// Writes the segmentation, prediction, transform, coefficient, and filter syntax for one final coding block. /// @@ -1900,7 +2001,7 @@ internal partial class Av1TileWriter /// The horizontal chroma subsampling shift. /// The vertical chroma subsampling shift. /// The aligned origin in chroma samples. - private static Point GetChromaBlockOrigin(Point lumaOrigin, int subsamplingX, int subsamplingY) + public static Point GetChromaBlockOrigin(Point lumaOrigin, int subsamplingX, int subsamplingY) => new( (lumaOrigin.X >> (Av1Constants.ModeInfoSizeLog2 + subsamplingX)) << Av1Constants.ModeInfoSizeLog2, (lumaOrigin.Y >> (Av1Constants.ModeInfoSizeLog2 + subsamplingY)) << Av1Constants.ModeInfoSizeLog2); diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CoefficientsEntropyTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CoefficientsEntropyTests.cs index 1d3e815017..ba6c062c81 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CoefficientsEntropyTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1CoefficientsEntropyTests.cs @@ -368,7 +368,8 @@ public class Av1CoefficientsEntropyTests sequenceHeader, frameHeader, Width, - Height); + Height, + disallow4x4AllFrames: true); Av1PictureControlSet picture = pictureBuffer.Picture; Av1NeighborArrayUnit paletteContexts = Assert.Single(picture.PaletteContexts); diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1EncoderFrameTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1EncoderFrameTests.cs index 41854dd3b5..c26d3a4f78 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1EncoderFrameTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1EncoderFrameTests.cs @@ -153,6 +153,69 @@ public class Av1EncoderFrameTests } } + [Fact] + public void EncodeEffortNineSelectsSubEightPartition() + { + const int Size = 16; + using Image source = new(Size, Size); + for (int y = 0; y < Size; y++) + { + Span row = source.Frames.RootFrame.PixelBuffer.DangerousGetRowSpan(y); + for (int x = 0; x < Size; x++) + { + // The bottom-right 8x8 uses horizontal prediction on its left half and vertical prediction + // on its right half. Twelve source values keep a parent palette from reproducing both halves. + byte value; + if (x < 8 && y < 8) + { + value = 128; + } + else if (y < 8) + { + value = (byte)(16 + ((x - 8) * 20)); + } + else + { + value = x < 12 + ? (byte)(176 + ((y - 8) * 9)) + : (byte)(16 + ((x - 8) * 20)); + } + + row[x] = new Rgba32(value, value, value); + } + } + + using MemoryStream stream = new(); + _ = Av1FrameEncoder.Encode( + Configuration.Default, + source.Frames.RootFrame, + stream, + CreateColorConfig(Av1BitDepth.EightBit, Av1ColorFormat.Yuv400), + qIndex: 4, + effort: 9); + + byte[] payload = stream.ToArray(); + using Av1Decoder decoder = new(Configuration.Default); + using Image decoded = decoder.Decode(payload); + Av1FrameInfo frameInfo = Assert.IsType(decoder.FrameInfo); + Point[] leafPositions = + [ + new(2, 2), + new(3, 2), + new(2, 3), + new(3, 3) + ]; + + foreach (Point leafPosition in leafPositions) + { + Assert.Equal( + Av1BlockSize.Block4x8, + frameInfo.GetModeInfoAt(leafPosition).BlockSize); + } + + Assert.Equal(new Size(Size, Size), decoded.Size); + } + [Theory] [InlineData(EightBit)] [InlineData(TenBit)] diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1EncoderModeInfoBufferTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1EncoderModeInfoBufferTests.cs index 2ddcdcb480..3c4d6ce847 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1EncoderModeInfoBufferTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1EncoderModeInfoBufferTests.cs @@ -99,7 +99,8 @@ public class Av1EncoderModeInfoBufferTests sequenceHeader, frameHeader, Width, - Height)) + Height, + disallow4x4AllFrames: true)) { allocations = allocator.AllocationLog.ToArray(); Assert.Equal(2, allocations.Length); diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1IntraBlockCopyTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1IntraBlockCopyTests.cs index d1fa546ed0..d5fb5477c4 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1IntraBlockCopyTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1IntraBlockCopyTests.cs @@ -100,7 +100,8 @@ public class Av1IntraBlockCopyTests sequenceHeader, frameHeader, Width, - Height); + Height, + disallow4x4AllFrames: true); Av1PictureControlSet picture = buffer.Picture; Point candidatePosition = new(80, 12); @@ -161,7 +162,8 @@ public class Av1IntraBlockCopyTests sequenceHeader, frameHeader, Width, - Height); + Height, + disallow4x4AllFrames: true); Av1PictureControlSet picture = buffer.Picture; Point modeInfoPosition = new(80, 0); @@ -254,7 +256,8 @@ public class Av1IntraBlockCopyTests sequenceHeader, frameHeader, Width, - Height); + Height, + disallow4x4AllFrames: true); using Av1EncoderFrameBuffer source = new( Configuration.Default, @@ -339,7 +342,8 @@ public class Av1IntraBlockCopyTests sequenceHeader, frameHeader, Width, - Height); + Height, + disallow4x4AllFrames: true); using Av1EncoderFrameBuffer source = new( Configuration.Default, @@ -434,7 +438,8 @@ public class Av1IntraBlockCopyTests sequenceHeader, frameHeader, Width, - Height); + Height, + disallow4x4AllFrames: true); using Av1EncoderFrameBuffer source = new( Configuration.Default, diff --git a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1IntraSuperblockEncoderTests.cs b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1IntraSuperblockEncoderTests.cs index 9dd099f223..4c84ade06f 100644 --- a/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1IntraSuperblockEncoderTests.cs +++ b/tests/ImageSharp.Tests/Formats/Heif/Av1/Av1IntraSuperblockEncoderTests.cs @@ -244,7 +244,8 @@ public class Av1IntraSuperblockEncoderTests picture.Sequence.SequenceHeader, picture.Parent.FrameHeader, Width, - Height); + Height, + disallow4x4AllFrames: true); using Av1EncoderCoefficientBuffer tileCoefficients = new( Configuration.Default, @@ -320,7 +321,8 @@ public class Av1IntraSuperblockEncoderTests pictureTemplate.Sequence.SequenceHeader, pictureTemplate.Parent.FrameHeader, Width, - Height); + Height, + disallow4x4AllFrames: true); Av1PictureControlSet picture = pictureBuffer.Picture; using Av1EncoderCoefficientBuffer coefficients = new( @@ -393,7 +395,8 @@ public class Av1IntraSuperblockEncoderTests pictureTemplate.Sequence.SequenceHeader, pictureTemplate.Parent.FrameHeader, Width, - Height); + Height, + disallow4x4AllFrames: true); using Av1EncoderCoefficientBuffer liveCoefficients = new( Configuration.Default, @@ -517,7 +520,8 @@ public class Av1IntraSuperblockEncoderTests picture.Sequence.SequenceHeader, picture.Parent.FrameHeader, Width, - Height); + Height, + disallow4x4AllFrames: true); using Av1EncoderCoefficientBuffer tileCoefficients = new( Configuration.Default, @@ -572,7 +576,8 @@ public class Av1IntraSuperblockEncoderTests pictureTemplate.Sequence.SequenceHeader, pictureTemplate.Parent.FrameHeader, Width, - Height); + Height, + disallow4x4AllFrames: true); using Av1EncoderCoefficientBuffer coefficients = new( Configuration.Default, @@ -664,7 +669,8 @@ public class Av1IntraSuperblockEncoderTests sequenceHeader, frameHeader, Width, - Height); + Height, + disallow4x4AllFrames: true); using Av1EncoderCoefficientBuffer coefficients = new( Configuration.Default, @@ -921,7 +927,8 @@ public class Av1IntraSuperblockEncoderTests pictureTemplate.Sequence.SequenceHeader, pictureTemplate.Parent.FrameHeader, Width, - Height); + Height, + disallow4x4AllFrames: true); using Av1EncoderCoefficientBuffer coefficients = new( Configuration.Default, @@ -1133,7 +1140,8 @@ public class Av1IntraSuperblockEncoderTests pictureTemplate.Sequence.SequenceHeader, pictureTemplate.Parent.FrameHeader, Width, - Height); + Height, + disallow4x4AllFrames: true); using Av1EncoderCoefficientBuffer coefficients = new( Configuration.Default, @@ -1246,7 +1254,8 @@ public class Av1IntraSuperblockEncoderTests pictureTemplate.Sequence.SequenceHeader, pictureTemplate.Parent.FrameHeader, Width, - Height); + Height, + disallow4x4AllFrames: true); using Av1EncoderCoefficientBuffer coefficients = new( Configuration.Default, @@ -1406,7 +1415,8 @@ public class Av1IntraSuperblockEncoderTests pilotTemplate.Sequence.SequenceHeader, pilotTemplate.Parent.FrameHeader, Width, - Height); + Height, + disallow4x4AllFrames: true); using Av1EncoderCoefficientBuffer pilotCoefficients = new( Configuration.Default, @@ -1507,7 +1517,8 @@ public class Av1IntraSuperblockEncoderTests pictureTemplate.Sequence.SequenceHeader, pictureTemplate.Parent.FrameHeader, Width, - Height); + Height, + disallow4x4AllFrames: true); using Av1EncoderCoefficientBuffer coefficients = new( Configuration.Default, @@ -1747,7 +1758,8 @@ public class Av1IntraSuperblockEncoderTests pilotTemplate.Sequence.SequenceHeader, pilotTemplate.Parent.FrameHeader, Width, - Height); + Height, + disallow4x4AllFrames: true); using Av1EncoderCoefficientBuffer pilotCoefficients = new( Configuration.Default, @@ -1903,7 +1915,8 @@ public class Av1IntraSuperblockEncoderTests pictureTemplate.Sequence.SequenceHeader, pictureTemplate.Parent.FrameHeader, Width, - Height); + Height, + disallow4x4AllFrames: true); using Av1EncoderCoefficientBuffer coefficients = new( Configuration.Default, @@ -2101,7 +2114,8 @@ public class Av1IntraSuperblockEncoderTests pictureTemplate.Sequence.SequenceHeader, pictureTemplate.Parent.FrameHeader, Width, - Height); + Height, + disallow4x4AllFrames: true); using Av1EncoderCoefficientBuffer coefficients = new( Configuration.Default, @@ -2243,7 +2257,8 @@ public class Av1IntraSuperblockEncoderTests pictureTemplate.Sequence.SequenceHeader, pictureTemplate.Parent.FrameHeader, Width, - Height); + Height, + disallow4x4AllFrames: true); using Av1EncoderCoefficientBuffer coefficients = new( Configuration.Default, @@ -2370,7 +2385,8 @@ public class Av1IntraSuperblockEncoderTests pictureTemplate.Sequence.SequenceHeader, pictureTemplate.Parent.FrameHeader, Width, - Height); + Height, + disallow4x4AllFrames: true); using Av1EncoderCoefficientBuffer coefficients = new( Configuration.Default, @@ -2480,7 +2496,8 @@ public class Av1IntraSuperblockEncoderTests sequenceHeader, frameHeader, Width, - Height); + Height, + disallow4x4AllFrames: true); using Av1EncoderCoefficientBuffer coefficients = new( Configuration.Default, @@ -2738,7 +2755,8 @@ public class Av1IntraSuperblockEncoderTests pictureTemplate.Sequence.SequenceHeader, pictureTemplate.Parent.FrameHeader, width, - height); + height, + disallow4x4AllFrames: true); using Av1EncoderCoefficientBuffer coefficients = new( Configuration.Default, @@ -3035,6 +3053,16 @@ public class Av1IntraSuperblockEncoderTests /// public int Count { get; private set; } + /// + public readonly Av1PartitionType SelectPartition( + Av1SymbolEncoder writer, + Av1MacroBlockD macroBlock, + Point blockOrigin, + ushort tileIndex, + Av1BlockSize blockSize, + Av1PartitionType preparedPartition) + => preparedPartition; + /// public void EncodeBlock( Av1SymbolEncoder writer, @@ -3100,6 +3128,16 @@ public class Av1IntraSuperblockEncoderTests /// public int Count { get; private set; } + /// + public readonly Av1PartitionType SelectPartition( + Av1SymbolEncoder writer, + Av1MacroBlockD macroBlock, + Point blockOrigin, + ushort tileIndex, + Av1BlockSize blockSize, + Av1PartitionType preparedPartition) + => preparedPartition; + /// public void EncodeBlock( Av1SymbolEncoder writer,